Copyright © 2010 Gnip, inc.
Gnip makes it easy to build social media tracking tools.
Posts Tagged ‘gnip’
Enhanced Filtering for PowerTrack
February 2nd, 2012Gnip is always looking for ways to improve its filtering capabilities and customer feedback plays a huge role in these efforts. We are excited to announce enhancements to our PowerTrack product that allow for more precise filtering of the Twitter Firehose, a feature enhancement request that came directly from you, our customers.
Gnip PowerTrack rules now support OR and Grouping using (). We have also loosened limitations on the number of characters and the number of clauses per rule. Specifically, a single rule can now include up to 10 positive clauses and up to 50 negative clauses (previously 10 total clauses). Additionally, the character limit per rule has grown from 255 characters to 1024.
With these changes, we are now able to offer our customers a much more robust and precise filtering language to ensure you receive the Tweets that matter most to you and your business. However, these improvements bring their own set of specific constraints that are important to be aware of. Examples and details on these limitations are as follows:
OR and Grouping Examples
- apple OR microsoft
- apple (iphone OR ipad)
- apple computer –(fruit OR green)
- (apple OR mac) (computer OR monitor) new –fruit
- (apple OR android) (ipad OR tablet) –(fruit green microsoft)
Character Limitations
- A single rule may contain up to 1024 characters including operators and spaces.
Limitations
- A single rule must contain at least 1 positive clause
- A single rule supports a max of 10 positive clauses throughout the rule
- A single rule supports max of 50 negative clauses throughout the rule
- Negated ORs are not allowed. The following are examples of invalid rules:
- -iphone OR ipad
- ipad OR -(iphone OR ipod)
Precedence
- An implied “AND” takes precedence in rule evaluation over an OR
For example a rule of:
- android OR iphone ipad would be evaluated as apple OR (iphone ipad)
- ipad iphone OR android would be evaluated as (iphone ipad) OR android
You can find full details of the Gnip Power Track filtering changes in our online documentation.
Know of another way we can improve our filtering to meet your needs? Let us know in the comments below.
Twitter Shouts: Huntsman’s Out!
January 16th, 2012At Gnip, one of the most fascinating aspects of social media is ‘speed’ – specifically in regards to news stories. We continue to see a trend towards the ‘breaking’ of news stories on platforms like Twitter. Both the speed at which a story is broken as well as the speed at which that story catches on show the incredible power of this medium for information exchange. And as we’ve pointed out before, different social media streams offer different analytical value – Twitter versus a news feed for example.
Last night proved a great example of this as word of Huntsman’s withdrawal from the GOP presidential race crept out. Interestingly, the news was broken by Peter Hamby, a CNN Political Reporter–on Twitter. While CNN followed up on this news a few minutes later, it seems the reporter (or the network) realized the inherent ‘newswire’ value of breaking this news as fast as possible…and used Twitter as part of their strategy to do so!
This Tweet was followed with what we’ve begun to see as the normal ‘Twitter’ spike for breaking news – the chart below, built by our Data Scientist Scott, shows how quickly Huntsman withdrawl was retweeted and passed along. When looked at in comparison to an aggregate news feed (in this case, NewsGator’s Datawire Firehose, which is a content aggregator derived from crowdsourced rss feeds and contains many articles from traditional media providers), some interesting comparisons are brought to light.
Comparing tweets of “huntsman” and news articles breaking Jon Huntsman’s withdrawal from GOP primary race. The blue curves show the “Social Activity Pulse” that characterizes the growth and decay of media activity around this topic. By fitting the rate of articles or tweets to a function we can compare standard measure such as time-to-peak, store half-life etc. (More on this in a future post.) The peak in Twitter is reached about the same time as the first story arrives from NewsGator, over 10 minutes after the story broke on Twitter.
Both streams show a similar curve in story adoption, peak and tail. What’s different is the timeframe of the content. Twitter’s data spikes about 10 minutes earlier than NewsGator’s. NewsGator’s content is more in-depth, as it contains news stories and blog posts, but as we’ve seen in other cases, Twitter is the place where news breaks these days.
Boulder Chamber of Commerce: Why Gnip Joined
November 11th, 2011It took awhile, but Gnip’s now a Boulder Chamber of Commerce (@boulderchamber) Member. We joined after a pattern of clear value to our particular industry became clear. In August of this year they hosted an event on that put us face-to-face with a the U.S. Department of Commerce Under Secretary for International Trade (Francisco Sánchez) and Colorado Congressman (Jared Polis) where we discussed software patent issues, as well as immigration visa challenges the U.S. tech industry faces. Tonight I’m attending an event with Congressman Polis and a local software Venture Capitalist (Jason Mendelson) to talk about challenges surrounding the hiring of technical talent locally, and globally.
These are topics with significant political/legislative dynamics, and the Chamber has given us, a local software firm, access to relevant forums in which we can get our point of view on the table; thank you.
Whether or not the Chamber has been providing this kind of relevant access all along, I don’t know (my perception is otherwise). I do know that the impact they’re having on us as a local software business, as well as the channel they’re giving Gnip to get its perspective heard in the broader (National) forum, is significant. I’d encourage other Boulder software/technology firms to support their efforts, contribute in their events, and help them build an agenda that in the end, helps us be more effective software/technical businesses.
Join us, in joining the Chamber.
Simplicity Wins
November 10th, 2011It seems like every once in a while we all have to re-learn certain lessons.
As part of our daily processing, Gnip stores many terabytes of data in millions of keys on Amazon’s S3. Various aspects of serving our customers require that we pour over those keys and the data behind them, regularly.
As an example, every 24 hours we construct usage reports that provide visibility into how our customers are using our service. Are they consuming a lot or a little volume? Did their usage profile change? Are they not using us at all? So on and so on. We also have what we affectionately refer to as the “dude where’s my tweet” challenge; of the billion activities we deliver each day to our customers, inevitably someone says “hey, I didn’t receive Tweet ‘X’ what gives?” Answering that question requires that we store the ID of every Tweet a customer ever receives. Pouring over all this data every 24 hours is a challenge.
As we started on the project, it seemed like a good fit for Hadoop. It involves pulling in lots of small-ish files, doing some slicing, aggregate the results, and spitting them out the other end. Because we’re hosted in Amazon it was natural to use their Elastic MapReduce service (EMR).
Conceptually the code was straight forward and easy to understand. The logic fit the MapReduce programming model well. It requires a lot of text processing and sorts well into various stages and buckets. It was up and running quickly.
As the size of the input grew it started to have various problems, many of which came down to configuration. Hadoop options, JVM options, open file limits, number and size of instances, number of reducers, etc. We went through various rounds of tweaking settings and throwing more machines in the cluster, and it would run well for a while longer.
But it still occasionally had problems. Plus there was that nagging feeling that it just shouldn’t take this much processing power to do the work. Operational costs started to pop up on the radar.
So we did a small test to check the feasibility of getting all the necessary files from S3 onto a single EC2 instance and processing it with standard old *nix tools. After promising results we decided to pull it out of EMR. It took several days to re-write, but we’ve now got a simple Ruby script using various *nix goodies like cut, sort, grep and their friends. The script is parallel-ized via JRuby threads at various points that make sense (downloading multiple files at once and processing the files independently once they’ve been bucketed).
In the end it runs in less time than it did on EMR, on a single modest instance, is much simpler to debug and maintain, and costs far less money to run.
We landed in a somewhat counter-intuitive place. There’s great technology available these days to process large amounts of data; we continue to use Hadoop for other projects. But as we start to bring them into our tool-set we have to be careful not to forget the power of straight forward, traditional tools.
Simplicity wins.
Why Traders Use Social Media: Speed & Amplification
November 3rd, 2011Gnip’s asset and investment management clients are consistently impressed by two aspects of our social media data that differentiate this data from their other sources: Speed & Amplification.
Speed
Speed relates to the ability of social media content to be ‘instant’; an ability fueled by millions of global users who can break news and sentiment more immediately than traditional media sources always can.
A prime example is news of the death of Osama Bin Laden. Keith Urbahn, the former chief of staff for Don Rumsefeld, is widely credited with the breaking that story… through Twitter!
After Keith’s tweet, multiple retweets quickly followed. Within 19 tweets on this subject, a company called DataMinr had identified this as an important and breaking story. DataMinr, a “global sensor network for emerging events and consumer signals,” then issued a signal to their clients, alerting them to this important piece of information.
How does this play into the ‘speed’ characteristic? Because it would be over 20 minutes before that story appeared on traditional news sites. Access to a data stream that can beat traditional media sources by over 20 minutes requires no explanation as to its value for traders and investors.
Amplification
Amplification speaks to the ability of social media as a ‘crowd-sourced megaphone.’ The propensity of users to like, share, and retweet content from other users gives those consuming social media data an extremely easy mechanism to measure what content is most important to the world – and compare that content against other content in real time.
A prime example is the passing of Steve Jobs. We wrote about Steve Jobs’ passing a few weeks ago – that post is here – but there’s an important item to revisit:
The impact he had on us made his death that much more profound and the reaction on Twitter was immediate and immense. Word spread rapidly, peaking at 50,000 Tweets per minute within 30 minutes. At that point, Tweets about Jobs accounted for almost 25% of all Tweets being sent globally.
Access to Gnip’s social media data stream allowed our clients to measure, in the moment, the amplification of this story to measure the importance the world placed on this piece of news. While I doubt any of us needed to see those numbers to know Steve’s passing was an important piece of news, that’s a clear example of how ‘amplification’ works.
Our clients use amplification as a measure to weigh the importance of breaking news, upcoming events, market and product announcements, etc. against other stories. By capturing a realtime snapshot of what the market considers important – and what it doesn’t – they’re able to add an important factor to their existing algorithms.
None of this is to suggest that either social media data speed or amplification should be a sole factor in investing. But when the Gnip social media data stream provides clients with an additional factor to help understand or predict market fluctuations, the value is obvious.
Google+ Now Available from Gnip
October 27th, 2011
Gnip is excited to announce the addition of Google+ to its repertoire of social media data sources. Built on top of the Google+ Search API, Gnip’s stream allows its customers to consume realtime social media data from Google’s fast-growing social networking service. Using Gnip’s stream, customers can poll Google+ for public posts and comments matching the terms and phrases relevant to their business and client needs.
Google+ is an emerging player in the social networking space that is a great pairing with the Twitter, Facebook, and other microblog content currently offered by Gnip. If you are looking for volume, Google+ quickly became the third largest social networking platform within a week of its public launch and some are projecting it to emerge as the world’s second largest social network within the next twelve months. Looking to consume content from social network influencers? Google+ is where they are! (even former Facebook President Sean Parker says so).
By working with Gnip along with a stream of Google+ data (and the availability of an abundance of other social data sources), you’ll have access to a normalized data format, unwound URLs, and data deduplication. Existing Gnip customers can seamlessly add Google+ to their Gnip Data Collectors (all you need is a Google API Key). New to Gnip? Let us help you design the right solution for your social data needs, contact sales@gnip.com.
Gnip is Headed to Defrag
October 20th, 2011
Just wanted to give everyone an update on the Gnip events front. We are gearing up for our appearance at the upcoming Defrag Conference taking place in our neighboring town of Broomfield, Colorado at the Omni Interlocken Resort on November 9th and 10th. We are excited to be a sponsor of this year’s event that will focus on the exploration of the tools and technologies that intersect around the data deluge.
At the conference, Gnip’s very own Chris Moody (@chrismoodycom) will be featured as a keynote speaker, where he will discuss, Emerging Use Cases for Big Social Data. During this presentation Chris will highlight the ways that social data is being used to drive innovation across a variety of industries from Financial Services and Emergency Response to Local Business and Social CRM. Chris will also unveil the importance of having the right data for the right need with the debut of Gnip’s Social Data Analysis Grid. Be sure to catch Chris’ presentation on November 9th from 11:30am to 11:45am.
With the conference is just around the corner, we wanted to extend everyone a discounted registration code on behalf of Gnip. To receive a 20% discount on your registration, contact sales@gnip.com by November 1st.
Finally, Gnip’s Director of Sales, Fred Funke (@funkefred) and Data Scientist, Scott Hendrickson (@drskippy27) will also be at the conference.
Let us know if you will be attending, we’d love to chat. See you at Defrag!
The VMAs, Lady Gaga and Data Science
September 29th, 2011
Hi everyone. I’m the new Data Scientist here at Gnip. I’ll be analyzing the fascinating data that we have coming from all of our varied social data streams to pull out the stories, both impactful and trivial, that are flowing through social media conversations. I’m still getting up-to-speed but wanted to share one of the first social events that I’ve dug into, the 2011 MTV Video Music Awards.
Check out the info below and let me know in the comments what you think and what you’d like to see more of. And now, on with the show…
3.6M Tweets Mention “VMA”
The volume of tweets containing “VMA” rose steadily from a few hours before the VMA pre-show was broadcast, up to the starting of the pre-show at 8:00 PM ET (00:00 GMT) and remained fairly strong during the event. It trailed to low volume within the hour after the VMA broadcast ended at 11:15 PM ET (03:15 GMT). Tweets mentioning “VMA” totaled 3.6M during the 7 hours surrounding and including the VMA broadcast.
Lady Gaga Steals the “Tweet” Show
The largest volume of tweets for an individual artist are the mentions of “gaga.” Lady Gaga performed early in the show and the surge of tweets during her performance surpassed 35k tweets per minute for about 8 minutes. Again in the second half, Lady Gaga tweet volume briefly jumped above 50k per minute. Tweets mentioning “gaga” totaled 1.8M during the 7 hours surrounding and including the VMA broadcast.
As you can see in the chart below, other artists that garnered significant tweet volumes included Beyonce’, Justin Beiber, Chris Brown, Katy Perry and Kanye West. Perry, West and Brown got a lot of attention during their appearances, while Justin Bieber and Lady Gaga lead the counts in volume by maintaining a fairly steady stream of tweets during the broadcast.
| Term | Representation of Tweets Sampled |
| VMA | 44 % |
| Lady Gaga | 21 % |
| Beyonce | 16 % |
| Justin Bieber | 10 % |
| MTV | 9.2 % |
| Chris Brown | 8.0 % |
| Katy Perry | 5.6 % |
| Kanye West | 4.8 % |
| Jonas | 3.5 % |
| Taylor Swift | 2.1 % |
| Rihanna | 1.1 % |
| Eminem | 0.55 % |
| Michael Jackson | 0.18 % |
| Ke$ha | 0.17 % |
| Cher | 0.14 % |
| Paramore | 0.12 % |
Contrasting, it is interesting to note that Beyonce’ and Chris Brown gained most of their tweet attention around their performances with very larger surges in tweet volume. Beyonce’s volume–another Beyonce’ bump–continues after her performance as twitter users absorb the news of her pregnancy.
One surprise that emerges from looking for other artists connected to the VMAs was Michael Jackson’s tweet volume. While Jackson gleaned many Retweets after winning the King of the VMA poll, he also received a large number of natural tweets lamenting his passing and celebrating his past successes.
Methodology
The free-form text and limited length of twitter messages creates a number of challenges for monitoring an event via twitter comments. People refer to the event differently and focus on different parts of the event. There will be spelling variations and differences in idioms and nicknames used to describe people and performances. Do we search for “Bieber”,”Beiber” and “Justin”? Will tweeters use “Beyonce” or Beyonce’”? Knowledge of what we are monitoring is required; preparing tools to adapt things we learn during the events is also essential to getting good results.
One effective strategy is to use one or two tokens to identify tweets related to the event. The objective is to choose terms that we know are related to the event, that won’t be widely used outside the event, and that will give a representative sample–diverse and with sufficient volume. Once we have started to collect the event-focused twitter sample, we can look for relevant terms correlated with the filter term to find out what else people are tweeting about during the event.
Hope you enjoyed this first post. Look for more to come.
Archive
Recent Posts
- Enhanced Filtering for PowerTrack
- Gnip and Automattic Make Whole New Universe of Data Available
- Twitter Shouts: Huntsman’s Out!
- Tebow Time Scores in Social Media
- Are Facebook Users More Optimistic than Twitter Users?
Categories
- Corporate
- Customers
- Data
- Development
- Events
- Guest Bloggers
- Industry
- Just For Fun
- Marketing
- News
- Partners
- Product
- Sales
- Technology
Tags
activity data
activity streams
api
api rate limits
apis
boulder colorado
data aggregators
data collection
data consumers
data delivery
data feeds
data filtering
data filters
data normalization
data producers
data publishers
data sources
data streaming
data streams
delicious
digg
facebook
firehose
flickr
gnip
hiring
http
keyword filters
metadata
normalization
poll
polling
power track
push
realtime data
rest
rss
social data
social media
social media data
tweets
twitter
twitter data
xml
xmpp
Blogroll
- Facebook Developers
- FeldThoughts
- First Round Capital
- Foundry Group
- Klout
- O’Reilly Radar
- PeteSearch
- The Net-Savvy Executive
- Thought Experiments
- Twitter Engineering
Recent Tweets
- # {New Product Feature} Enhanced Filtering for PowerTrack http://t.co/zVgJUY6H More precise filtering options for the Twitter firehose!
- # Feasting on whale carcasses http://t.co/espZtpNL Twitter and Facebook, Why Twitter Might Be Worth More In The Long Run @pointsnfigures
- # You learn something new every day http://t.co/oWsf08om - 8 Crazy Things IBM Scientists Have Learned Studying Twitter
- # Full firehoses that ensure 100% coverage in realtime http://t.co/R03nlExx More details on our partnership with Automattic on the @gnip blog
- # Likes from WordPress & IntenseDebate now available http://t.co/kRoBM2W4 "Automattic is an important source in the social data mix" @radian6
Gnip and Automattic Make Whole New Universe of Data Available
January 17th, 2012-
Tags: automattic, blogs, comments, deep analysis, engagement, firehose, gnip, intensedebate, jetpack, likes, wordpress.com, wordpress.org
No CommentsPosted by Bill Adkins, Director of Business Development in Data, Partners, Product
“This new data from Automattic is a big addition and a testament to Gnip’s commitment to drive the social data economy forward. This is an important source to add to the social data mix, one that we know our customers will take full advantage of.”
- Rob Begg, VP Marketing of Radian6
Today, we’re excited to announce a major addition to our coverage of the conversations taking place on blogs around the world. We’re expanding our relationship with Automattic to make a whole new universe of blog and comment data available to the market for the first time anywhere.
For those who don’t know, Automattic is a network of web services including WordPress.com, VIP hosting and support, Polldaddy, IntenseDebate, and Jetpack. We’ve been delivering data from WordPress.com and IntenseDebate for about a year and a half and found that while our customers loved their data, they always wanted more.
As of today, we are now offering the full firehose of blog posts and comments from Jetpack-powered WordPress.org sites, as well as engagement streams of “likes” from WordPress.com and IntenseDebate. The new data from WordPress.org greatly increases the coverage available to those who are looking to do deep analysis of blog posts and comments. The new engagement streams enable companies to pull in reaction data to quickly understand sentiment, relevance and resonance. With this they can gauge the intensity of opinion around fast moving blog and comment conversations, helping prioritize critical response.
Being full firehoses, all of the streams from Automattic ensure 100% coverage in realtime giving customers the peace of mind that they can keep up the entire discussion on fast moving threads.
The scope of coverage offered by Automattic is pretty incredible. Check out some of these stats:
We’re thrilled to be able to offer these new data streams to our customers and can’t wait to see the amazing things they’ll be able to do with them.
Updated: Coverage in GigaOM – Gnip and WordPress deepen ties, expand data partnership