Gnip is always looking for ways to improve its filtering capabilities and customer feedback plays a huge role in these efforts.  We are excited to announce enhancements to our PowerTrack product that allow for more precise filtering of the Twitter Firehose, a feature enhancement request that came directly from you, our customers.

Gnip PowerTrack rules now support OR and Grouping using ().  We have also loosened limitations on the number of characters and the number of clauses per rule. Specifically, a single rule can now include up to 10 positive clauses and up to 50 negative clauses (previously 10 total clauses).  Additionally, the character limit per rule has grown from 255 characters to 1024.

With these changes, we are now able to offer our customers a much more robust and precise filtering language to ensure you receive the Tweets that matter most to you and your business.  However, these improvements bring their own set of specific constraints that are important to be aware of.  Examples and details on these limitations are as follows:

OR and Grouping Examples

• apple OR microsoft
• apple (iphone OR ipad)
• apple computer –(fruit OR green)
• (apple OR mac) (computer OR monitor) new –fruit
• (apple OR android) (ipad OR tablet) –(fruit green microsoft)

Character Limitations

• A single rule may contain up to 1024 characters including operators and spaces.

Limitations

• A single rule must contain at least 1 positive clause
• A single rule supports a max of 10 positive clauses throughout the rule
• A single rule supports max of 50 negative clauses throughout the rule
• Negated ORs are not allowed. The following are examples of invalid rules:
• -iphone OR ipad
• ipad OR -(iphone OR ipod)

Precedence

• An implied “AND” takes precedence in rule evaluation over an OR

For example a rule of:

• android OR iphone ipad  would be evaluated as apple OR (iphone ipad)
• ipad iphone OR android would be evaluated as (iphone ipad) OR android

You can find full details of the Gnip Power Track filtering changes in our online documentation.

Know of another way we can improve our filtering to meet your needs?  Let us know in the comments below.

- Rob Begg, VP Marketing of Radian6

Updated: Coverage in GigaOM – Gnip and WordPress deepen ties, expand data partnership

Last night proved a great example of this as word of Huntsman’s withdrawal from the GOP presidential race crept out. Interestingly, the news was broken by Peter Hamby, a CNN Political Reporter–on Twitter. While CNN followed up on this news a few minutes later, it seems the reporter (or the network) realized the inherent ‘newswire’ value of breaking this news as fast as possible…and used Twitter as part of their strategy to do so!

This Tweet was followed with what we’ve begun to see as the normal ‘Twitter’ spike for breaking news – the chart below, built by our Data Scientist Scott, shows how quickly Huntsman withdrawl was retweeted and passed along. When looked at in comparison to an aggregate news feed (in this case, NewsGator’s Datawire Firehose, which is a content aggregator derived from crowdsourced rss feeds and contains many articles from traditional media providers), some interesting comparisons are brought to light.

Comparing tweets of “huntsman” and news articles breaking Jon Huntsman’s withdrawal from GOP primary race. The blue curves show the “Social Activity Pulse” that characterizes the growth and decay of media activity around this topic. By fitting the rate of articles or tweets to a function we can compare standard measure such as time-to-peak, store half-life etc. (More on this in a future post.) The peak in Twitter is reached about the same time as the first story arrives from NewsGator, over 10 minutes after the story broke on Twitter.

Both streams show a similar curve in story adoption, peak and tail. What’s different is the timeframe of the content. Twitter’s data spikes about 10 minutes earlier than NewsGator’s. NewsGator’s content is more in-depth, as it contains news stories and blog posts, but as we’ve seen in other cases, Twitter is the place where news breaks these days.

When Tim Tebow connected with Demaryius Thomas on an 80-yard touchdown pass on the first play of overtime, we saw a noticeable spike in the overall volume of social media messages flowing through the Gnip platform.

Tebow Time!

As we enter 2012, Facebook users are more optimistic than Twitter users.

You’re probably wondering how we can say that. Well, we looked at all of the public posts on Facebook and Tweets on Twitter that contained “Happy New Year.” For all of those posts and Tweets, we compared the use of positive words such as “better” and “good” to the use of negative words such as “worse” and “bad.” We found that Tweets with positive words appeared 8 times more frequently than Tweets with negative words. You might be thinking a ratio of 8 to 1 is pretty optimistic…

It may be, but posts on Facebook had a ratio of 40 to 1–such a huge difference lead us to speculate that Facebook is a more optimistic place than Twitter.

Interesting stuff. Could be a variety of reasons for the difference, from the mix of users on each service to the fact that Facebook is used to communicate with friends, while Twitter is user to broadcast to followers. We’ll leave the speculation up to you.

“66% of respondents who bought something on Black Friday did so as a direct result of social media interactions with friends and family.”

While that stat speaks to the impact social media has upon us as individuals, think more broadly about how powerful it is to analyze that data in aggregate, in real-time. Companies are leveraging data from WordPress blogs, Twitter mentions, Facebook likes and multiple other sources to inform critical realtime decisions for inventory management and operational planning, sales and marketing planning, revenue forecasting, and many others.

Example Scenario for Using Social Data: It’s holiday time, 2011. Your company begins to aggregate ‘mentions’ of a new product from Twitter, Facebook, WordPress blogs in realtime. You take that data and analyze it for # of mentions about the new product, geography of posts (where available), demographic information within user profiles (what keywords are most consistent within Twitter user profiles that mentioned your product?), etc.

You spread that data among multiple divisions, providing additional forecast, regional buying pattern, and customer habit data. Your teams use that to:

1. Manage supply chain: Redirect inventory to areas with highest potential sales and (depending on how far out you are) use as a data point in the S&OP system for manufacturing forecasts to keep ahead of the holiday demand.
2. Target marketing spend: Use regional buying patterns and customer habit data to inform what demographic you are, and aren’t, hitting. Do you need to reposition your marketing plan?
3. Incorporate product feedback: Are there consistent reasons why people are buying your product – or why they aren’t? Information on quality, packaging, price, etc will be incredibly valuable for future products.
4. Calibrate investor expectations: Inform your IR team of potential positive/negative performance feedback to give them running room ahead of any announcements.

Those are just some of the more common use cases we’re seeing. But new opportunities are popping up on a daily basis. We spotted this gem in a recent WSJ article about finding a parking space during crazy shopping times:

Bud Kleppe, a real-estate agent in St. Paul, Minn., watches Mall of America’s Twitter feed for parking updates. (The mall sends them out under the hash tag #moaparking.)

Imagine collecting data from update systems like this and using it measure parking turnover across prime shopping days. Now, overlay the turnover of spots in specific sections against a map of stores and you have some interesting potential for data on economic performance and forecasting. When incorporated with other traditional retail data and compared on a store-to-store basis, you’ve built a unique and realtime analysis tool.

You’re only limited by your imagination in how you can apply social media data to you business. The more software developers, corporations, and people use social media, and the more things they use it for (like parking updates!), the greater the possible use cases for analysis of that data and the more valuable it becomes.

Welcome to the Battle of the Pies!

For those of you that have been in a pie eating contest or had a pie in the face, you know this one will be a fight all the way down to the very last crumb. In one corner (well actually it is the Gnip Octagon so can you really have corners, oh well) we have The Traditionalist, pumpkin pie and in the opposite corner, The New Comer, pecan pie. Without further ado, Ladies and Gentleman, Let’s Get Ready to Rumble, wait wrong sport. Let’s Fight!

Six Social Media Sources, Two Words, One Winner . . . And the Winner Is . . .

Source	Pumpkin Pie	Pecan Pie	Winning Ratio   Pumpkin Pie to Pecan Pie
Twitter	X		4:1
Facebook	X		5:1
Google+	X		6:1
Newsgator	X		3:1
WordPress	X		5:1
WordPress Comments	X		2:1
Overall	+6 Winner!	+0

We looked at one week’s worth of data across six of the top social media sources and determined that pumpkin pie “takes the cake” (so to speak) across every source.

In this case, it is interesting to point out that in sources like Twitter, Facebook, Google+ and WordPress we see higher winning ratios, while sources that tend to have higher latency such as Newsgator and WordPress Comments were a little more even. Is this because, on further consideration, pecan pie sounds pretty good? Or is it that everyone will have to have two pies and, with pecan as the traditional second, it is highly discussed?

Top Pie Recipes

Even though pumpkin pie was our clear winner, we thought it would be fun to share a few of the most popular holiday pie recipes by social media source:

1. Twitter – Cook du Jour Gluten-Free Pumpkin Pie and Pecan Pie Video Recipe from joyofcooking.com
2. Facebook – Ben Starr’s Pumpkin Bourbon Pecan Pie Recipe
3. Newsgator – BlogHer’s Pumpkin Pecan Roulade with Orange Mascarpone Cream Pie Recipe
4. WordPress and WordPress Comments – Chocolate Bourbon Pecan Pie from allrecipes.com

Non-Traditional Thanksgiving Pies

Another interesting fact that came out of this Cagefight was the counts of non-traditional Thanksgiving pies that were mentioned across the social media sources we surveyed. Though we rarely find these useful for communicating numerical values effectively, you can’t not have a pie chart in this post.

Happy Thanksgiving!

• Launch of Gnip MarketStream: Our hedge fund clients have been quite vocal in their desire for a package incorporating the most relevant social media data streams into a single low-latency, high-volume solution. We’re proud to answer their needs with the launch of Gnip MarketStream, a realtime data solution that packages the incredibly rich and broad “voice of the market” Twitter stream with the uniquely deep and targeted “voice of the trader” StockTwits stream.

• Premium Partnership with StockTwits: An integral component of the Gnip MarketStream is StockTwits social media data. We’re thrilled to announce this partnership with StockTwits, the leading realtime financial platform for the investment community and creator of the $(TICKER) tag. The StockTwits stream is a curated, defined-demographic, realtime social data stream focused on investment decisions and analysis. Gnip now provides streaming access to the full StockTwits firehose of social data, and offers access to historical content as far back as 2009.

While the use of social media data by the investment community has included use of this data in news analysis and equity research, the primary adoption of this data across the last six months has been as a trading indicator. By combining the strengths of both the Twitter stream and the StockTwits stream, Gnip MarketStream provides investment professionals unparalleled access to relevant social data at time when social media has become an increasingly vital channel for news and market sentiment.

For more information about Gnip MarketStream or StockTwits data, contact trading@gnip.com.

These are topics with significant political/legislative dynamics, and the Chamber has given us, a local software firm, access to relevant forums in which we can get our point of view on the table; thank you.

Whether or not the Chamber has been providing this kind of relevant access all along, I don’t know (my perception is otherwise). I do know that the impact they’re having on us as a local software business, as well as the channel they’re giving Gnip to get its perspective heard in the broader (National) forum, is significant. I’d encourage other Boulder software/technology firms to support their efforts, contribute in their events, and help them build an agenda that in the end, helps us be more effective software/technical businesses.

Join us, in joining the Chamber.

As part of our daily processing, Gnip stores many terabytes of data in millions of keys on Amazon’s S3. Various aspects of serving our customers require that we pour over those keys and the data behind them, regularly.

As an example, every 24 hours we construct usage reports that provide visibility into how our customers are using our service. Are they consuming a lot or a little volume? Did their usage profile change? Are they not using us at all? So on and so on. We also have what we affectionately refer to as the “dude where’s my tweet” challenge; of the billion activities we deliver each day to our customers, inevitably someone says “hey, I didn’t receive Tweet ‘X’ what gives?” Answering that question requires that we store the ID of every Tweet a customer ever receives. Pouring over all this data every 24 hours is a challenge.

As we started on the project, it seemed like a good fit for Hadoop. It involves pulling in lots of small-ish files, doing some slicing, aggregate the results, and spitting them out the other end. Because we’re hosted in Amazon it was natural to use their Elastic MapReduce service (EMR).

Conceptually the code was straight forward and easy to understand. The logic fit the MapReduce programming model well. It requires a lot of text processing and sorts well into various stages and buckets. It was up and running quickly.

As the size of the input grew it started to have various problems, many of which came down to configuration. Hadoop options, JVM options, open file limits, number and size of instances, number of reducers, etc. We went through various rounds of tweaking settings and throwing more machines in the cluster, and it would run well for a while longer.

But it still occasionally had problems. Plus there was that nagging feeling that it just shouldn’t take this much processing power to do the work. Operational costs started to pop up on the radar.

So we did a small test to check the feasibility of getting all the necessary files from S3 onto a single EC2 instance and processing it with standard old *nix tools. After promising results we decided to pull it out of EMR. It took several days to re-write, but we’ve now got a simple Ruby script using various *nix goodies like cut, sort, grep and their friends. The script is parallel-ized via JRuby threads at various points that make sense (downloading multiple files at once and processing the files independently once they’ve been bucketed).

In the end it runs in less time than it did on EMR, on a single modest instance, is much simpler to debug and maintain, and costs far less money to run.

We landed in a somewhat counter-intuitive place. There’s great technology available these days to process large amounts of data; we continue to use Hadoop for other projects. But as we start to bring them into our tool-set we have to be careful not to forget the power of straight forward, traditional tools.

Simplicity wins.