Tag Archives: halfhose

Guide to the Twitter API – Part 3 of 3: An Overview of Twitter’s Streaming API

Posted on by

0 comments

The Twitter Streaming API is designed to deliver limited volumes of data via two main types of realtime data streams: sampled streams and filtered streams. Many users like to use the Streaming API because the streaming nature of the data delivery means that the data is delivered closer to realtime than it is from the Search API (which I wrote about last week). But the Streaming API wasn’t designed to deliver full coverage results and so has some key limitations for enterprise customers. Let’s review the two types of data streams accessible from the Streaming API.The first type of stream is “sampled streams.” Sampled streams deliver a random sampling of Tweets at a statistically valid percentage of the full 100% Firehose. The free access level to the sampled stream is called the “Spritzer” and Twitter has it currently set to approximately 1% of the full 100% Firehose. (You may have also heard of the “Gardenhose,” or a randomly sampled 10% stream. Twitter used to provide some increased access levels to businesses, but announced last November that they’re not granting increased access to any new companies and gradually transitioning their current Gardenhose-level customers to Spritzer or to commercial agreements with resyndication partners like Gnip.)

The second type of data stream is “filtered streams.” Filtered streams deliver all the Tweets that match a filter you select (eg. keywords, usernames, or geographical boundaries). This can be very useful for developers or businesses that need limited access to specific Tweets.

Because the Streaming API is not designed for enterprise access, however, Twitter imposes some restrictions on its filtered streams that are important to understand. First, the volume of Tweets accessible through these streams is limited so that it will never exceed a certain percentage of the full Firehose. (This percentage is not publicly shared by Twitter.) As a result, only low-volume queries can reliably be accommodated. Second, Twitter imposes a query limit: currently, users can query for a maximum of 400 keywords and only a limited number of usernames. This is a significant challenge for many businesses. Third, Boolean operators are not supported by the Streaming API like they are by the Search API (and by Gnip’s API). And finally, there is no guarantee that Twitter’s access levels will remain unchanged in the future. Enterprises that need guaranteed access to data over time should understand that building a business on any free, public APIs can be risky.

–

The Search API and Streaming API are great ways to gather a sampling of social media data from Twitter. We’re clearly fans over here at Gnip; we actually offer Search API access through our Enterprise Data Collector. And here’s one more cool benefit of using Twitter’s free public APIs: those APIs don’t prohibit display of the Tweets you receive to the general public like premium Twitter feeds from Gnip and other resyndication partners do.

But whether you’re using the Search API or the Streaming API, keep in mind that those feeds simply aren’t designed for enterprise access. And as a result, you’re using the same data sets available to anyone with a computer, your coverage is unlikely to be complete, and Twitter reserves the right change the data accessibility or Terms of Use for those APIs at any time.

If your business dictates a need for full coverage data, more complex queries, an agreement that ensures continued access to data over time, or enterprise-level customer support, then we recommend getting in touch with a premium social media data provider like Gnip. Our complementary premium Twitter products include Power Track for data filtered by keyword or other parameters, and Decahose and Halfhose for randomly sampled data streams (10% and 50%, respectively). If you’d like to learn more, we’d love to hear from you at sales@gnip.com or 888.777.7405.

Top 10 Reasons Gnip Data Collectors Outperform Direct API Connections

Posted on by

0 comments

Sometimes we’re asked why it makes sense to access social data from Gnip and not through direct access to the publicly accessible APIs. (We usually get this question from people who have never tried to access data from various social media APIs; those who have tried it understand how tedious and time-intensive data collection is and they can’t wait to hand their social data collection over to Gnip to manage for them.)

So, if you’ve never tried collecting data from multiple social media APIs at once… why would you use Gnip instead of connecting directly to the publicly accessible APIs? Here are 10 of the reasons…

#10 – Customer Support
When you use most public APIs, development teams are often busy, so they’re tough (if not impossible) for most developers to reach with questions. At Gnip, we actually want to talk to you. We offer enterprise-level support so clients can contact us at all odd hours and receive a thoughtful, thorough response. And we work closely with a variety of sources, so we can reach out to them directly if necessary.

#9 – Reliability
Public APIs are not contractually guaranteed; data availability and access levels may change at any time, with or without warning to users. Many businesses worry about building their businesses on data that doesn’t come with contractual agreements. When you subscribe to premium data such as the premium Twitter feeds available through Gnip, we provide you with a formal agreement. This locks in your access level, price, service, and terms of use for the duration of your agreement.

#8 – Rate limit recommendations
Instead of having to figure out rate limits for the various sources on your own, Gnip can recommend rate limits based on our own extensive experience with the various APIs.

#7 – Delivery in your protocol of choice: never poll for data again
A lot of developers think polling for data is tedious… and unfortunately, most APIs are polling-based. So if you go to the sources directly, you have to poll their servers for the data. By using Gnip, you can choose between polling for your data or to having your data streamed to you.

#6 – New feed setup in seconds
Without Gnip, it can take many hours (or days) of a developer’s time to set up a new API connection, parse the new feed, and start bringing data into your system. With Gnip, it can take as little as 30 seconds and no dev effort at all to start consuming the data.

#5 – Gnip is the only source for some data
Gnip can offer access to some data that’s not available from any other source (eg. Premium Twitter volume-based feeds like our Decahose and Halfhose).

#4 – Established premium data partnerships
Established partnerships with premium data publishers (Twitter, BackType, WordPress, etc.) make it quick and easy for Gnip customers to test and add premium data feeds.

#3 – Established relationships with all publishers
Because we manage data collection for customers all day every day, we’re among the earliest to know when API changes happen and the fastest to make any necessary changes to keep your data flowing.

#2 – APIs are generally hard to manage
Publishers change their APIs sometimes. Some APIs change frequently and without warning or documentation (cough, Facebook, cough) while others change less frequently. But no matter what, change is inevitable. Gnip manages your social media data delivery over time so you can keep your data flowing smoothly and reliably with minimal effort.

#1 – Enrichments
A variety of enrichments, or added metadata and features, come included with feeds delivered through Gnip data collectors. Some of the most popular enrichments include format normalization across sources (so you only have to write one parser for all your social media data), Klout Score inclusion (currently available for premium Twitter feeds), and language detection and filtering via a proprietary Gnip algorithm. We add enrichments all the time, so look for lots more to come.

We think Gnip is pretty cool (yes, we’re biased)… but even we know that Gnip isn’t for everyone. If you only need 1 feed from 1 source, the data you need is available through a publicly accessible API, you have an engineer who can monitor and optimize your data consumption regularly, and you’re certain that you will never need any other feeds forever and ever, then Gnip probably isn’t the right choice for you.

But if you’d like to ensure you’re receiving top-quality premium data access without requiring your engineering team to invest lots of time in data collection, we’d like to invite you to give Gnip a try. We’ve got lots of happy customers already and we just might prove valuable to you, too.

Announcing Multiple Connections for Premium Twitter Feeds

Posted on by

0 comments

A frequent request from our customers has been the ability to open multiple connections to Premium Twitter Feeds on their Gnip data collectors. Our customers have asked and we have delivered!

While multiple connections to standard data feeds have been available for quite some time, we have only allowed one connection to our Premium Twitter Feeds.  Beginning today you will be able to open multiple mirrored connections to Power Track, Decahose, Halfhose, and all of our other Premium Twitter Feeds.  This feature will be helpful when testing connections to your Gnip data collector in different environments (such as staging or review) without having an impact on your production connection.

You may be saying “Sounds great Gnip, but will I be charged the standard Twitter licensing fee for the same tweet delivered across multiple connections?”. The answer is no!  You will pay a small flat fee per month for each additional connection.  If you’re interested in adding Multiple Connections to your Premium Twitter Feed please Contact Us.

New Gnip & Twitter Partnership

Posted on by

17 comments

We at Gnip have been waiting a long time to write the following sentence: Gnip and Twitter have partnered to make Twitter data commercially available through Gnip’s Social Media API. I remember consuming the full firehose back in 2008 over XMPP. Twitter was breaking ground in realtime social streams at a then mind-blowing ~6 (six) Tweets per second. Today we see many more Tweets and a greater need for commercial access to higher volumes of Twitter data.

There’s enormous corporate demand for better monitoring and analytics tools, which help companies listen to their customers on Twitter and understand conversations about their brands and products. Twitter has partnered with Gnip to sublicense access to public Tweets, which is great news for developers interested in analyzing large amounts of this data. This partnership opens the door to developers who want to use Twitter streams to create monitoring and analytics tools for the non-display market.

Today, Gnip is announcing three new Twitter feeds with more on the way:

  • Twitter Halfhose. This volume-based feed is comprised of 50% of the full firehose.
  • Twitter Mentionhose. This coverage-based feed provides the realtime stream of all Tweets that mention a user, including @replies and retweets. We expect this to be very interesting to businesses studying the conversational graph on Twitter to determine influencers, engagement, and trending content.
  • Twitter Decahose. This volume-based product is comprised of 10% of the full firehose. Starting today, developers who want to access this sample rate will access it via Gnip instead of Twitter. Twitter will also begin to transition non-display developers with existing Twitter Gardenhose access over to Gnip.

We are excited about how this partnership will make realtime social media analysis more accessible, reliable, and sustainable for businesses everywhere.

To learn more about these premium Twitter products, visit http://gnip.com/twitter, send us an email at info@gnip.com, or appropriately, find us on Twitter @gnip.