Archive for the ‘Partners’ Category
“This new data from Automattic is a big addition and a testament to Gnip’s commitment to drive the social data economy forward. This is an important source to add to the social data mix, one that we know our customers will take full advantage of.”
- Rob Begg, VP Marketing of Radian6
As social media data becomes more and more important across a range of businesses, our customers are asking for access to more data sources to give them a more complete picture of the social media conversations that are relevant to their businesses.
Today, we’re excited to announce a major addition to our coverage of the conversations taking place on blogs around the world. We’re expanding our relationship with Automattic to make a whole new universe of blog and comment data available to the market for the first time anywhere.
For those who don’t know, Automattic is a network of web services including WordPress.com, VIP hosting and support, Polldaddy, IntenseDebate, and Jetpack. We’ve been delivering data from WordPress.com and IntenseDebate for about a year and a half and found that while our customers loved their data, they always wanted more.
As of today, we are now offering the full firehose of blog posts and comments from Jetpack-powered WordPress.org sites, as well as engagement streams of “likes” from WordPress.com and IntenseDebate. The new data from WordPress.org greatly increases the coverage available to those who are looking to do deep analysis of blog posts and comments. The new engagement streams enable companies to pull in reaction data to quickly understand sentiment, relevance and resonance. With this they can gauge the intensity of opinion around fast moving blog and comment conversations, helping prioritize critical response.
Being full firehoses, all of the streams from Automattic ensure 100% coverage in realtime giving customers the peace of mind that they can keep up the entire discussion on fast moving threads.
The scope of coverage offered by Automattic is pretty incredible. Check out some of these stats:
We’re thrilled to be able to offer these new data streams to our customers and can’t wait to see the amazing things they’ll be able to do with them.
Updated: Coverage in GigaOM – Gnip and WordPress deepen ties, expand data partnership
While the market has been on its roller coaster ride across the past month, Gnip has kept its collective head down and stayed busy on behalf of our Investment Management clients (hedge funds, HFTs, asset managers, etc.). That hard work has paid off and we have two exciting announcements to make today.
- Launch of Gnip MarketStream: Our hedge fund clients have been quite vocal in their desire for a package incorporating the most relevant social media data streams into a single low-latency, high-volume solution. We’re proud to answer their needs with the launch of Gnip MarketStream, a realtime data solution that packages the incredibly rich and broad “voice of the market” Twitter stream with the uniquely deep and targeted “voice of the trader” StockTwits stream.
- Premium Partnership with StockTwits: An integral component of the Gnip MarketStream is StockTwits social media data. We’re thrilled to announce this partnership with StockTwits, the leading realtime financial platform for the investment community and creator of the $(TICKER) tag. The StockTwits stream is a curated, defined-demographic, realtime social data stream focused on investment decisions and analysis. Gnip now provides streaming access to the full StockTwits firehose of social data, and offers access to historical content as far back as 2009.
While the use of social media data by the investment community has included use of this data in news analysis and equity research, the primary adoption of this data across the last six months has been as a trading indicator. By combining the strengths of both the Twitter stream and the StockTwits stream, Gnip MarketStream provides investment professionals unparalleled access to relevant social data at time when social media has become an increasingly vital channel for news and market sentiment.
For more information about Gnip MarketStream or StockTwits data, contact trading@gnip.com.
The
Twitter Streaming API is designed to deliver limited volumes of data via two main types of realtime data streams: sampled streams and filtered streams. Many users like to use the Streaming API because the streaming nature of the data delivery means that the data is delivered closer to realtime than it is from the Search API (which I wrote about
last week). But the Streaming API wasn’t designed to deliver full coverage results and so has some key limitations for enterprise customers. Let’s review the two types of data streams accessible from the Streaming API.
The first type of stream is “sampled streams.” Sampled streams deliver a random sampling of Tweets at a statistically valid percentage of the full 100% Firehose. The free access level to the sampled stream is called the “Spritzer” and Twitter has it currently set to approximately 1% of the full 100% Firehose. (You may have also heard of the “Gardenhose,” or a randomly sampled 10% stream. Twitter used to provide some increased access levels to businesses, but announced last November that they’re not granting increased access to any new companies and gradually transitioning their current Gardenhose-level customers to Spritzer or to commercial agreements with resyndication partners like Gnip.)
The second type of data stream is “filtered streams.” Filtered streams deliver all the Tweets that match a filter you select (eg. keywords, usernames, or geographical boundaries). This can be very useful for developers or businesses that need limited access to specific Tweets.
Because the Streaming API is not designed for enterprise access, however, Twitter imposes some restrictions on its filtered streams that are important to understand. First, the volume of Tweets accessible through these streams is limited so that it will never exceed a certain percentage of the full Firehose. (This percentage is not publicly shared by Twitter.) As a result, only low-volume queries can reliably be accommodated. Second, Twitter imposes a query limit: currently, users can query for a maximum of 400 keywords and only a limited number of usernames. This is a significant challenge for many businesses. Third, Boolean operators are not supported by the Streaming API like they are by the Search API (and by Gnip’s API). And finally, there is no guarantee that Twitter’s access levels will remain unchanged in the future. Enterprises that need guaranteed access to data over time should understand that building a business on any free, public APIs can be risky.
–
The Search API and Streaming API are great ways to gather a sampling of social media data from Twitter. We’re clearly fans over here at Gnip; we actually offer Search API access through our Enterprise Data Collector. And here’s one more cool benefit of using Twitter’s free public APIs: those APIs don’t prohibit display of the Tweets you receive to the general public like premium Twitter feeds from Gnip and other resyndication partners do.
But whether you’re using the Search API or the Streaming API, keep in mind that those feeds simply aren’t designed for enterprise access. And as a result, you’re using the same data sets available to anyone with a computer, your coverage is unlikely to be complete, and Twitter reserves the right change the data accessibility or Terms of Use for those APIs at any time.
If your business dictates a need for full coverage data, more complex queries, an agreement that ensures continued access to data over time, or enterprise-level customer support, then we recommend getting in touch with a premium social media data provider like Gnip. Our complementary premium Twitter products include Power Track for data filtered by keyword or other parameters, and Decahose and Halfhose for randomly sampled data streams (10% and 50%, respectively). If you’d like to learn more, we’d love to hear from you at sales@gnip.com or 888.777.7405.
The
Twitter Search API can theoretically provide full coverage of ongoing streams of Tweets. That means it can, in theory, deliver 100% of Tweets that match the search terms you specify almost in realtime. But in reality, the Search API is not intended and does not fully support the repeated constant searches that would be required to deliver 100% coverage.
Twitter has indicated that the Search API is primarily intended to help end users surface interesting and relevant Tweets that are happening now. Since the Search API is a polling-based API, the rate limits that Twitter has in place impact the ability to get full coverage streams for monitoring and analytics use cases. To get data from the Search API, your system may repeatedly ask Twitter’s servers for the most recent results that match one of your search queries. On each request, Twitter returns a limited number of results to the request (for example “latest 100 Tweets”). If there have been more than 100 Tweets created about a search query since the last time you sent the request, some of the matching Tweets will be lost.
So . . . can you just make requests for results more frequently? Well, yes, you can, but the total number or requests you’re allowed to make per unit time is constrained by Twitter’s rate limits. Some queries are so popular (hello “Justin Bieber”) that it can be impossible to make enough requests to Twitter for that query alone to keep up with this stream. And this is only the beginning of the problem as no monitoring or analytics vendor is interested in just one term; many have hundreds to thousands of brands or products to monitor.
Let’s consider a couple examples to clarify. First, say you want all Tweets mentioning “Coca Cola” and only that one term. There might be fewer than 100 matching Tweets per second usually — but if there’s a spike (say that term becomes a trending topic after a Super Bowl commercial), then there will likely be more than 100 per second. If because of Twitter’s rate limits, you’re only allowed to send one request per second, you will have missed some of the Tweets generated at the most critical moment of all.
Now, let’s be realistic: you’re probably not tracking just one term. Most of our customers are interested in tracking somewhere between dozens and hundreds of thousands of terms. If you add 999 more terms to your list, then you’ll only be checking for Tweets matching “Coca Cola” once every 1,000 seconds. And in 1,000 seconds, there could easily be more than 100 Tweets mentioning your keyword, even on an average day. (Keep in mind that there are over a billion Tweets per week nowadays.) So, in this scenario, you could easily miss Tweets if you’re using the Twitter Search API. It’s also worth bearing in mind that the Tweets you do receive won’t arrive in realtime because you’re only querying for the Tweets every 1,000 seconds.
Because of these issues related to the monitoring use cases, data collection strategies relying exclusively on the Search API will frequently deliver poor coverage of Twitter data. Also, be forewarned, if you are working with a monitoring or analytics vendor who claims full Twitter coverage but is using the Search API exclusively, you’re being misled.
Although coverage is not complete, one great thing about the Twitter Search API is the complex operator capabilities it supports, such as Boolean queries and geo filtering. Although the coverage is limited, some people opt to use the Search API to collect a sampling of Tweets that match their search terms because it supports Boolean operators and geo parameters. Because these filtering features have been so well liked, Gnip has replicated many of them in our own premium Twitter API (made even more powerful by the full coverage and unique data enrichments we offer).
So, to recap, the Twitter Search API offers great operator support but you should know that you’ll generally only see a portion of the total Tweets that match your keywords and your data might arrive with some delay. To simplify access to the Twitter Search API, consider trying out Gnip’s Enterprise Data Collector; our “Keyword Notices” feed retrieves, normalizes, and deduplicates data delivered through the Search API. We can also stream it to you so you don’t have to poll for your results. (“Gnip” reverses the “ping,” get it?)
But the only way to ensure you receive full coverage of Tweets that match your filtering criteria is to work with a premium data provider (like us! blush…) for full coverage Twitter firehose filtering. (See our Power Track feed if you’d like for more info on that.)
Stay tuned for Part 3, our overview of Twitter’s Streaming API coming next week…
You may find yourself wondering . . . “What’s the best way to access the Twitter data I need?” Well the answer depends on the type and amount of data you are trying to access. Given that there are multiple options, we have designed a three part series of blog posts that explain the differences between the coverage the general public can access and the coverage available through Twitter’s resyndication agreement with Gnip. Let’s dive in . ..
Understanding Twitter’s Public APIs . . . You Mean There is More than One?
In fact, there are three Twitter APIs: the REST API, the Streaming API, and the Search API. Within the world of social media monitoring and social media analytics, we need to focus primarily on the latter two.
- Search API - The Twitter Search API is a dedicated API for running searches against the index of recent Tweets
- Streaming API – The Twitter Streaming API allows high-throughput, near-realtime access to various subsets of Twitter data (eg. 1% random sampling of Tweets, filtering for up to 400 keywords, etc.)
Whether you get your Twitter data from the Search API, the Streaming API, or through Gnip, only public statuses are available (and NOT protected Tweets). Additionally, before Tweets are made available to both of these APIs and Gnip, Twitter applies a quality filter to weed out spam.
So now that you have a general understanding of Twitter’s APIs . . . stay tuned for Part 2, where we will take a deeper dive into understanding Twitter’s Search API, coming next week…
In April 2010, the U.S. Library of Congress and Twitter announced that Twitter will donate its public Twitter conversation data to the Library of Congress archives, making it available for researchers to read and preserving it for generations to come. This was an historic moment: for the first time, a communication channel in the social media space had been identified as a relic of our culture, emblematic of the role technology has come to play in our interactions with one another.
Today we’re honored to announce that Gnip is playing a key role in the this project by delivering past and future Twitter data to the Library of Congress for historic preservation in the archives.
On a technical level, the magnitude of this project is enormous: with over a billion Tweets being generated weekly, the scale of data delivery involved in this project is historic. We’ve learned a lot from this project and it is clear that some of the outcomes from this project will benefit Gnip’s future product offerings. We look forward to sharing some of the technical details in the future at the appropriate time.
As a company, we couldn’t be more pleased to be a part of this momentous project. We’re eager to see what kind of research develops from the public archive made accessible by the Library of Congress. We’ve watched with awe as news of world-changing events like the Arab Spring and Osama Bin Laden’s death have flowed through our servers, and now we’re proud to be part of ensuring these historic events will be preserved for our posterity.
The archive of Twitter data which Gnip delivers to the Library of Congress will cater to the needs of researchers who wish to use limited amounts of public Twitter data for non-commercial purposes. Gnip will continue to serve those seeking realtime data, full-coverage data, and commercial use cases.
Thanks to Twitter and the Library of Congress for making this announcement possible. We couldn’t be more excited to be working with you both.
Not too long ago Gnip celebrated its third birthday. I am celebrating my one week anniversary with the company today. To say a lot happened before my time at Gnip would be the ultimate understatement, and yet it is easy for me to see the results produced from those three years of effort. Some of those results include:
The Product
Gnip’s social media API offering is the clear leader in the industry. Gnip is delivering over a half a billion social media activities daily from dozens of sources. That certainly sounds impressive, but how can I be so confident Gnip is the leader? Because the most important social media monitoring companies rely on our services to deliver results to their customers every single day. For example, Gnip currently works with 8 of the top 9 enterprise social media monitoring companies, and the rate we are adding enterprise focused companies is accelerating.
The Partners
Another obvious result is the strong partnerships that have been cultivated. Some of our partnerships such as Twitter and Klout were well publicized when the agreements were put in place. However, having strong strategic partners takes a lot more than just a signed agreement. It takes a lot of dedication, investment, and hard work by both parties in order to deliver on the full promise of the agreement. It is obvious to me that Gnip has amazing partnerships that run deep and are built upon a foundation of mutual trust and respect.
The People
The talent level at Gnip is mind blowing, but it isn’t the skills of the people that have stood out the most for me so far. It is the dedication of each individual to doing the right thing for our customers and our partners that has made the biggest impression. When it comes to gathering and delivering social media data, there are a lot of shortcuts that can be taken in order to save time, money, and effort. Unfortunately, these shortcuts can often come at the expense of publishers, customers, or both. The team at Gnip has no interest in shortcuts and that comes across in every individual discussion and in every meeting. If I were going to describe this value in one word, the word would be “integrity”.
In my new role as President & COO, I’m responsible for helping the company grow quickly and smoothly while maintaining the great values that have been established from the company’s inception. The growth has already started and I couldn’t be more pleased with the talent of the people who have recently joined the organization including: Bill Adkins, Seth McGuire, Charles Ince, and Brad Bokal who have all joined Gnip within the last week. And, we are hiring more! In fact, it is worth highlighting one particular open position for a Customer Support Engineer. I’m hard pressed to think of a higher impact role at our company because we consider supporting our customers to be such an important priority. If you have 2+ years of coding experience including working with RESTful Web APIs and you love delivering over-the-top customer service, Gnip offers a rare opportunity to work in an environment where your skills will be truly appreciated. Apply today!
I look forward to helping Gnip grow on top of a strong foundation of product, partners, and people. If you have any questions, I can be reached at chris [at] gnip.com.
We’re excited to announce two new enrichments today: we’ve partnered with Klout to deliver influence score data and we’ve enabled filtering by languages on our Twitter firehose-based premium data feeds. Combined with Gnip’s other enrichments (format normalization, URL expansion, etc.), we hope you’ll find it easier than ever to filter your Twitter feeds to precisely the data you want. (See all Gnip Enrichments)
Our latest partner, Klout, is known as “the standard for influence.” Our friends there analyze Twitter and other social media data to determine how influential (or not) different Twitter users are and assign “Klout Scores” to them accordingly. (Last we checked, @gnip’s Klout Score was 41 on Klout’s scale of 1 to 100.) Klout is a Gnip customer as well, so we’re particularly pleased to work with them to bring Klout Score metadata to other Gnip customers and share the love.
Now when you access premium Twitter data through Gnip, you can opt to have each user’s Klout Score appended to their Tweets. Klout filtering capabilities are also available via Gnip — for example, when you use our Power Track feed, you can choose to receive Tweets only from users whose Klout Score exceeds a certain number. Although Klout data has been available upon request to existing Gnip customers for some time, today marks the official start of our partnership and Klout enrichment on Gnip feeds. Welcome to the family, Klout!
Our other new enrichment feature today, language filtering, has long been requested a wide variety of Gnip customers (our international clients in particular!). Starting today, language filtering too is available on Gnip’s premium Twitter feeds for 11 languages: English, Dutch, French, German, Italian, Japanese, Korean, Norwegian, Portuguese, Spanish, and Swedish (with more to follow).
To filter for English Tweets only, for instance, just append “lang:EN” to each relevant rule you’re querying. You can also enter “lang:EN” as a rule on its own if you’d like to receive all Tweets that our algorithm has identified as English language Tweets. Our language filtering option is based on our recently announced language metadata, built from the open sourced JTCL, using n-gram frequencies to categorize Tweets into given languages.
With these two new filtering capabilities you can construct a whole new class of streams using Power Track, such as:
- All Tweets in German from users with Klout Score greater than @gnip (“lang:de klout_score:41”)
- All geocoded Tweets written in Dutch (“lang:NL has:geo”)
- Norwegian Tweets about Coca Cola from very influential users (“coca cola lang:no klout:50”)
- and lots of others that we’re sure our customers will surprise us with!
Although Klout Scores and language filtering are only available on premium Twitter feeds so far, many of Gnip’s data enrichments come included with every Gnip Data Collector. Contact us to learn more or try Gnip’s enrichments for yourself.
One of the more interesting components of Twitter streams are the links within the Tweets themselves. Not only are links one way to bridge from traditional web trend analysis, to social media, but they are also a window into what people are sharing.
Gnip provides three mechanisms to get at links in Tweets.
- Link Stream. The link stream provides you with 100% of the Tweets that contain links. Furthermore, Gnip enriches the stream with unwound URLs, so you don’t have to bother with an unwind-farm on your end.
- Power Track’s ‘has:links’ operator. Through Power Track, you can refine your complex queries (including substring matching) to collect only Tweets that contain links.
- Power Track’s ‘url_contains:’ operator. The ‘url_contains:’ operator allows you to filter the 100% Firehose for Tweets that have links and contain the substring you provide. It filters against both short, and long, URLs.
Happy filtering!
The response to the commercial Twitter streams we’ve made available has been outstanding. We’ve talked to hundreds of companies who are building growing businesses that analyze conversations on Twitter and other social media sites. As Twitter’s firehose continues to grow (now over 110 million Tweets per day), we’re hearing more and more requests for a way to filter the firehose down to the Tweets that matter.
Today, we’re announcing a new commercial Twitter product called Power Track. This is a keyword based filter of the full firehose that provides 100% coverage over a stream that you define. Power Track customers no longer have to deal with polling rate limits on the Search API and volume limits on the Streaming API.
In addition to keyword based filters, Power Track also supports boolean operators and many of the custom operators allowed on Twitter Search API. With Power Track, companies and developers can define the precise slice of the Twitter stream they need and be confident they’re getting every Tweet, without worrying about volume restrictions.
Currently we support operators for narrowing the stream to a set of users, matching against unwound URLs, filtering by location, and more. We’ll continue to add support for more ways for our customers to filter the content relevant to them in the future. Check the documentation to see the technical details of these operators and more.
Gnip is here to ensure the enterprise marketplace gets the depth, breadth, and reliability of social media data it requires. Please contact us at info@gnip.com to find out more.
Gnip and Automattic Make Whole New Universe of Data Available
January 17th, 2012-
Tags: automattic, blogs, comments, deep analysis, engagement, firehose, gnip, intensedebate, jetpack, likes, wordpress.com, wordpress.org
No CommentsPosted by Bill Adkins, Director of Business Development in Data, Partners, Product
“This new data from Automattic is a big addition and a testament to Gnip’s commitment to drive the social data economy forward. This is an important source to add to the social data mix, one that we know our customers will take full advantage of.”
- Rob Begg, VP Marketing of Radian6
Today, we’re excited to announce a major addition to our coverage of the conversations taking place on blogs around the world. We’re expanding our relationship with Automattic to make a whole new universe of blog and comment data available to the market for the first time anywhere.
For those who don’t know, Automattic is a network of web services including WordPress.com, VIP hosting and support, Polldaddy, IntenseDebate, and Jetpack. We’ve been delivering data from WordPress.com and IntenseDebate for about a year and a half and found that while our customers loved their data, they always wanted more.
As of today, we are now offering the full firehose of blog posts and comments from Jetpack-powered WordPress.org sites, as well as engagement streams of “likes” from WordPress.com and IntenseDebate. The new data from WordPress.org greatly increases the coverage available to those who are looking to do deep analysis of blog posts and comments. The new engagement streams enable companies to pull in reaction data to quickly understand sentiment, relevance and resonance. With this they can gauge the intensity of opinion around fast moving blog and comment conversations, helping prioritize critical response.
Being full firehoses, all of the streams from Automattic ensure 100% coverage in realtime giving customers the peace of mind that they can keep up the entire discussion on fast moving threads.
The scope of coverage offered by Automattic is pretty incredible. Check out some of these stats:
We’re thrilled to be able to offer these new data streams to our customers and can’t wait to see the amazing things they’ll be able to do with them.
Updated: Coverage in GigaOM – Gnip and WordPress deepen ties, expand data partnership