Archive for September, 2008

Gnip 2.0 is Here…

September 30th, 2008
4 Comments

Gnip 2.0 is in production! Gnip 1.0 is no longer accessible.

Open Wide: Full Data Is Here

We’re proud to introduce full data and expanded meta-data. You now have access to all the data associated with activities, as well as continued support for activity notifications. Full data provides you with additional meta-data about an activity, as well as the actual activity “body” (e.g. the comment text on a blog, or the text of a tweet). Gnip has been contributing to Diso in order to distill activity streams into a common format, and this release is our first attempt at getting this right.

Collections Are Dead; Long Live Filters!

Collections have been migrated to more feature rich “Filters.” You can now do more complex rule based filtering on data that comes through Gnip. For example, Publishers that support “tags” on their activities (e.g. delicious), can be filtered on specific tags/keywords). We currently only support the OR conditional, and not AND. AND introduces not only some interesting technical challenges, but also drives Gnip into some specific product directions we’re still evaluating.

Introducing XMPP

Data Producers can “publish” activities to Gnip via ATOM, REST, or XMPP. Data Consumers can get activities from Gnip via REST, and as of this release, XMPP. We’re just starting to dabble with XMPP for outbound activities, so please help test. Just give us your JID, and add our JID to your roster, and watch it flow. Note, this is not an implementation of XMPP PubSub; baby steps.

Migration Notes

     

  • Existing implementations are now broken if you haven’t updated to the new API.
  • Data Producers: please start pushing activities into Gnip using the new schema.
  • Data Consumers
       

    • Your accounts, and data, have been migrated to the new service.
    • REST endpoints have changed, so please update (API doc here).
    • Your collections have been migrated to the new Filter format as “actor” based Filters.
    • The schema has changed, so please update (new schema).
  • We’ve adopted the microblogging verb/noun of “notice” for microblogging service activity actions. e.g. a “tweet” is no longer a “tweet”, it’s a “notice.”

    •  

We’re Taking Part in the Boulder Job Fair — Would You Like a Free Trip to Check Out Boulder (and Gnip)?

September 23rd, 2008
2 Comments

Boulder has the highest per capita programmers in the country.  It also has the healthiest people on the planet (I can’t back this one up with stats, just anecdotal evidence of 60-year-old grandmothers zooming up and down the mountains).  Basically, Boulder is the land of the mathlete, and it’s awesome!

A ton of local startups are competing for the best developers in Boulder and we’ve come to a common conclusion — we need to expand the pool of applicants.  It’s time to give developers living in the Bay Area, Boston and Bentonville a taste of the Boulder lifestyle and simultaneously introduce them to some of the coolest companies Boulder has to offer.

Are you a badass developer?  Do you code PHP or Jave or C++ in you sleep?  Can you denormalize a database with your eyes closed or create elegant streams of CSS?  We’d like to meet you.  In fact, we’d like to fly you out all expense paid to Boulder for a couple of days to meet some awesome companies, including Gnip, to see if there’s a love connection.  You’ll fly out on day one and spend time checking out the town, spend day two meeting with 20 killer tech companies and then have a third day to follow up with companies you like the best and then fly home.  Not a bad way to spend the last week of October.

If you’d like to know more, check out the additional details at Boulder.Me and then click the button to apply.

We’re looking forward to meeting you in Boulder next month.  We think you’ll dig the town as much as we do, and the companies are pretty rad, too.

Gnip 2.0 is Coming…

September 23rd, 2008
No Comments

Friends,

Gnip’s first major update is right around the corner, and we want to ensure everyone has enough time to prepare for the upcoming changes.  Gnip 1.0′s launch a few months ago provided a long overdue optimization to the way Data Consumers access information.  We’ve saved the network billions of API calls/polls since our launch, providing Data Producers operational relief, and allowing Data Consumers to access multiple data sources in a “one stop shopping” manner.

As previously promised, Gnip 2.0 takes a giant step forward by adding full data to the activities that flow through the system.  With the upcoming release, you can simply tell us who (or what) you care about, provide us an endpoint, and we’ll push those full activities to you in real time.  Data aggregators are no longer required to build one-off integrations with each data source they are interested in.  When inspiration strikes, you can have Gnip push relevant data to you and instead focus your efforts on the vital front-end work that makes-or-breaks a new service.

We’re currently testing the following improvements.  If you are pushing data into the system or pulling it out, please make sure you check out the new API and get ready for the roll out Oct 1.  Improvements include:

  • XML schema format change to provide extended meta-data and full data payload. New schema is at http://groups.google.com/group/gnip-community/web/gnip2.xsd
  • Collections are now called Filters and have expanded rule based functionality.
  • In addition to having your Filter activities HTTP POSTed to you, you can provide a JID for delivery via XMPP.

We’re also pleased to announce that complete Public Timelines for several services, including Twitter, Delicious and Digg, will be available for testing at the same time.  You’ll still need to abide by each service’s Terms of Service, but now the data will flow to an endpoint of your choosing.  If you’d like to take part in the test, please email us with your company and contact info and we’ll reach out to you.  We’re finalizing the billing structure for delivering full data to you and we’ll post more about that this week and next. Notifications, as Gnip provides today, will always be free.

We’ve posted a more technically in-depth announcement on the Community Boards; please head over there if you’d like to get the complete listing of changes and the new API document.

Cheers!
Eric

G[oogle]Data API & Standards

September 9th, 2008
No Comments

We’re in the throws of re-visioning gnip.xsd and that’s led to pondering Google’s Data API. If you haven’t noticed, at the interface level (not at a service level), there is a high degree of overlap between Gnip’s API, and Google’s Data API. We both chose REST as the primary interface, and data moves through as XML. Google decided to support both RSS and ATOM, while Gnip has constructed it’s own XML. From a system efficiency standpoint, our own boiled down schema makes sense. We’re a message aggregator, and message transmission and processing have to be done at scale (RSS & ATOM are heavy). That said, we’ll be offering ATOM and RSS based formats in the future, as our internal view of data doesn’t always match how folks want to consume it.

As for adopting the Google Data API, we have other priorities at the moment. A GData interface to Gnip as a service definitely has its appeal. I could see Gnip using it as the stepping stone to accessing Gnip activities as RSS/ATOM. Selfishly, Gnip could leverage GData’s convenience libs, and any time you can aggregate use of convenience libraries, everyone wins.

Software Evolution

September 5th, 2008
1 Comment

Those of us who have been around for awhile constantly joke about how “I remember building that 10 years ago” everytime some big “new” trend emerges. It’s always a lesson in market readiness and timing for a given idea. The flurry around Google Chrome has rekindled the conversation around distributed apps. Most folks are tied up in the concept of a “new browser,” but Chrome is actually another crack at the age old “distrbuted/server-side application” problem; albeit an apparent good one. The real news in Chrome (I’ll avoid the V8 vs. TraceMonkey conversation for now) is native Google Gears support.

My favorite kind of technology is the kind that quietly gets built, then one day you wake up and it’s changed everything. Google Gears has that potential and if Chrome winds up with meaningful distribution (or Firefox adopts Gears) web apps as we know them will finally have mark-up-level access to local resources (read “offline functionality”). This kind of evolution is long overdue.

Another lacking component on the network is the age-old, CS101, notion of event-driven architectures. HTTP GET dominates web traffic, and poor ‘ol HTTP POST is rarely used. Publish and subscribe models are all but unused on the network today, and Gnip aims to change that. We see a world that is PUSH driven rather than PULL. The web has come a looooong way on GET, but apps are desperate for traditional flow paradigms such as local processor event loops. Our goal is to do this in a protocol agnostic manner (e.g. REST/HTTP POST, XMPP, perhaps some distributed queuing model)

Watching today’s web apps poll eachother to death is hard. With each new product that integrates serviceX, the latency of serviceX’s events propegating through the ecosystem degrades, and everyone loses. This is a broken model that if left unresolved, will drive our web apps back into the dark ages once all the web service endpoints are overburdened to the point of being uninteresting.

We’ve seen fabulous adoption of our API since launching a couple of months ago. We hope that more Data Producers and Data Consumers leverage it going forward.

Building A Platform: One Feature at a Time.

September 4th, 2008
No Comments

As Gnip evolves, we are amazed, almost daily, at what folks build using our API. My favorite thing each morning is reading the search.twitter.com feed for Gnip. Seeing how we’re being used is so cool, and much of the time we’re used in ways we didn’t anticipate. When that starts happening, you realize you’re building a Platform with a capital ‘P’.

A recent product that caught our eye was “tweetrush” (out of Cork, Ireland); http://tweetrush.com/. They’re using Gnip to build statistical information/visualization around Twitter. Look for more cool stuff from these guys; the current product is just the beginning!

The vast majority of products leveraging Gnip, do so to make existing aspects of their product better (e.g. lower the latency of activity notification), but it’s really cool when we see entire products built up around our offering.

As the wheel turns.

Follow Gnip

Archive

Recent Posts
Categories
Tags
activity data activity streams api api rate limits apis boulder colorado data aggregators data collection data consumers data delivery data feeds data filtering data filters data normalization data producers data publishers data sources data streaming data streams delicious digg facebook firehose flickr gnip hiring http keyword filters metadata normalization poll polling power track push realtime data rest rss social data social media social media data tweets twitter twitter data xml xmpp
Blogroll

Recent Tweets

  • # {New Product Feature} Enhanced Filtering for PowerTrack http://t.co/zVgJUY6H More precise filtering options for the Twitter firehose!
  • # Feasting on whale carcasses http://t.co/espZtpNL Twitter and Facebook, Why Twitter Might Be Worth More In The Long Run @pointsnfigures
  • # You learn something new every day http://t.co/oWsf08om - 8 Crazy Things IBM Scientists Have Learned Studying Twitter
  • # Full firehoses that ensure 100% coverage in realtime http://t.co/R03nlExx More details on our partnership with Automattic on the @gnip blog
  • # Likes from WordPress & IntenseDebate now available http://t.co/kRoBM2W4 "Automattic is an important source in the social data mix" @radian6