Archive for October, 2008

Update on Gnip Platform and Twitter Publisher

October 25th, 2008
No Comments

We wanted to provide an update on how people are able to interact with Twitter via the Gnip platform since people creating new filters will notice a change in behavior as of today.

Gnip has been working with Twitter for several months. This has given developers another option for how they access events and data from the Twitter APIs. During this time Twitter was making their meta-data available to a limited set of partners like Gnip, as they state in their blog post from earlier this week, this meta-data access was on an experimental basis. In addition, Twitter announced in their blog post that they are working on a solution that will better allow people to use a service like Gnip for data-driven applications, but the new solution is not ready. At Gnip we plan to continue to work with Twitter as they bring this new solution to market.

In the interim people who create new filters on the Gnip platform will notice that these filters only provide access to the Gnip Notifications features, and not the Gnip Data Streams features. This change was put in place for new filter creation until the updated solution for data becomes available from Twitter. Existing filters that use the Gnip Data Streams features will continue to function normally.

At Gnip we see this as yet another example of how we are focusing on delivering a standard way for developers to access all the web’s data. We will continue to work on with our partners and the industry to provide reliable tools and services that allow you to get real-time access to the data you need when and where you need it. However, in some cases we in the industry will experience growing pains as the social Internet continues to expand and innovate.

I’m Speaking on the Feeds Panel at the SF MusicTech Conference Today

October 20th, 2008
No Comments

Quite excited that Brian Zisk is giving me a chance to speak at the conference today.  Doubly excited that I just stumbled across the new Search.Me streaming music integration courtesy of iMeem’s APIs.  Nope, it’s not powered by Gnip, but it’s a rad example of the open web in action.

Web APIs of All Shapes and Sizes

October 13th, 2008
No Comments

Not all APIs have the same capabilities and therefore they provide different levels of access to events, procedures and data. Seems obvious, but you would not think that based on the normal questions we see from people. In fact we have found that APIs can be like a lot like apples and oranges. So, with the number of available APIs growing, at a rate that can be more than 60 per month we thought people would benefit from some simple way to think of API categorization based on how they expose events and data.

We work with a large variety of APIs from a variety of service providers and have noticed that most APIs fall into a few descriptive types based on how they expose events and data. The following are the main ways we are starting to look at APIs.

  • Fire hose or “full stream”. Identi.ca and Twitter are two examples, but Flickr also has a fire hose
  • User-based stream: These services do not directly expose a full stream, but instead give people a way to assemble an aggregate stream based on a list of users. Flickr again is a good example and there are many others.
  • Activity-based Tag-based and “other”: The main way to work with these services is usually some defined activity (tag, bookmark, etc) access to information or pre-defined streams based on feeds. An example would be Delicious, which allows multiple methods to access information by APIs and feeds.

This bi-frication in API types is something people should keep in mind when they want to access a service for some specific need. If you need to get events and data for a specific need then obviously the behavior of the API is going to impact your approach. And of course here at Gnip we are hard at work trying to provide consistent approaches across all types of APIs, so back to work!

Integrating Gnip Notifications

October 9th, 2008
1 Comment

Now that we’ve got some time under our belt, we’ve observed a pattern in Data Consumers who are using Gnip. Many Data Consumers already have event detection in place with polling infrastructure they’ve built. They poll various endpoints looking for changes, and when there’s a change, they consume all the associated data with that change, and digest everything into their application.

Gnip Notifications enhance this model, with very little effort on the Data Consumer’s part. As a Data Consumer, you can leave your existing infrastructure in place, un-touched, yet leverage Gnip’s latenency minimization. To do so, all you have to do is injest a Gnip Notification, then have that Notification bump a poll/crawl to the top of your existing stack. Customers have called this “hinting” or “accellerating.”

For example, today you have a queue/batch of jobs to poll for a user’s activities on serviceX. Let’s say you poll serviceX for that user every hour. Gnip can tell you precisely when that user made a change on serviceX, and what it does, you move that entry in your queue/batch to the top. Basically using Gnip as an acellerator to move various events in your system to a “high-priority queue.” That queue may indeed be a separate queue in your model, or a logical high-priority queue by just moving entries to the top of your stack.

Your users now see their actions (photo uploads, twitters, whatever) in near real-time, rather than potentially the eternity of 15 minutes (often longer).

Next I’ll blog about how to incorporate Gnip Full Data activities into your existing infrastructure.

What We Are Up to At Gnip

October 2nd, 2008
No Comments

As the newest member of the Gnip team I have noticed that people are asking a lot of the same questions about what we are doing at Gnip and what are the ways people can use our services in their business.

What we do

Gnip provides an extensible messaging platform that allows for the publishing or subscribing of events and data from across the Internet, which makes data portability exponentially less painful and more automatic once it is set up. Because Gnip is being built as a platform of capabilities and not a web application the core services are instantly useful for multiple scenarios, including data producers, data consumers and any custom web applications. Gnip already is being used with many of the most popular Internet data sources, including Twitter, Delicious, Flickr, Digg, and Plaxo.

How to use Gnip

So, who is the target user of Gnip? It is a developer, as the platform is not a consumer-oriented web application, but a set of services meant to be used by a developer or an IT department for a set of core use cases.

  • Data Consumers: You’ve built your pollers, let us tell you when and where to fire them. Avoid throttling and decrease latency from hours to seconds.
  • Data Producers: Push your data to us and reduce API traffic by an order of magnitude while increasing distribution through aggregators.
  • Custom web applications: You want to embed or publish content to be used in your own application or for a third-party application. Decide who, or what, you care about for any Publisher, give us an end-point, and we push the data to you so you can solve your business use cases, such as customer service websites, corporate websites, blogs, or any web application.

Get started now

By leveraging the Gnip APIs, developers can easily design reusable services, such as, push-based notifications, smart filters and data streams that can be used for all your web applications to make them better. Are you a developer? Give the new 2.0 version a try!

Follow Gnip

Archive

Recent Posts
Categories
Tags
activity data activity streams api api rate limits apis boulder colorado data aggregators data collection data consumers data delivery data feeds data filtering data filters data normalization data producers data publishers data sources data streaming data streams delicious digg facebook firehose flickr gnip hiring http keyword filters metadata normalization poll polling power track push realtime data rest rss social data social media social media data tweets twitter twitter data xml xmpp
Blogroll

Recent Tweets

  • # {New Product Feature} Enhanced Filtering for PowerTrack http://t.co/zVgJUY6H More precise filtering options for the Twitter firehose!
  • # Feasting on whale carcasses http://t.co/espZtpNL Twitter and Facebook, Why Twitter Might Be Worth More In The Long Run @pointsnfigures
  • # You learn something new every day http://t.co/oWsf08om - 8 Crazy Things IBM Scientists Have Learned Studying Twitter
  • # Full firehoses that ensure 100% coverage in realtime http://t.co/R03nlExx More details on our partnership with Automattic on the @gnip blog
  • # Likes from WordPress & IntenseDebate now available http://t.co/kRoBM2W4 "Automattic is an important source in the social data mix" @radian6