Archive for August, 2008
The Gnip API supports incremental collection updates. We’ve supported this for awhile, but we didn’t do a good job communicating when it came out. Several folks are taking advantage of it, but over the past few days it’s become clear not everyone knows the functionality exists. Please see “Collection Updates” in the API doc for details.
Eric and I just spent the past few days in Los Angeles. We were lucky enough to be invited as one of two presenters to Mahalo’s first hosted tech meetup; Google (Chris Schalk and Kevin Marks) represented the other slot with Open Social. Attendance of 175 left standing room only, and Jason Calacanis, Mark Jeffrey, and crew were great hosts; thanks for having us guys! You can view the event here. Lots of great business ideas in the LA area, but focus seems to be around media (no surprise) rather than technology. 75% of the attendees I talked to after the meetup were heavy technologists however, so clearly folks want more tech representation; hopefully Mahalo’s regular tech meetups can help facilitate.
While we were out there, we spent time with a dozen or so companies/people about Gnip. The discussions ranged from revenue opportunities, and integration details, to our product roadmap, and how folks want it to look. We left with renewed focus on our coming feature set, and the need to hire more great people.
Everyone wants full data (aka activity/message payload) in activities, extended meta-data (beyond the current “type”, “guid”, “uid”, and “at” fields), and meta-data normalization. On the activity normalization front, checkout the work going on at DiSo, and contribute where you can. Data Consumers obviously want a broader range of Data Producers as well, and that’s where Gnip’s polling infrastructure will come into play. We’re cranking to get as much of this done by end of this calendar quarter as we can. If you think you can help us, please send us a note!
Gnip now has Delicious v2 data flowing through it. The delicous bookmarking data flowing through the system now includes bookmarking/tagging done via delicious plugins/API tools (e.g. toolbar buttons). Nice, clean and pure stream of data from delicous now. Enjoy!
Gnip is an intermediary service for message flow across disparate network endpoints. Standing in the middle allows for a variety of value adds (Data Producers can “publish once, distribute to many,” Data Consumers can enjoy single service interaction rather than one-off’ing over and over again), but the quality of data that Data Producers push into the system is fundamental.
Only As Good As The Sum Of Our Parts
Gnip doesn’t control the quality of the data being published to it. Whether it comes in the form of XMPP messages, RSS, or ATOM, there are many issues that can come into play that can affect the data a Data Consumer receives.
- Bad transport/delivery – The source XMPP, RSS, ATOM, or REST, feed can go down. When this happens for a given Publisher, that source has vanished and Gnip doesn’t receive messages for that Publisher. We’re only as good as the data coming in. While Gnip can consume data from XMPP, RSS, ATOM, and other sources, our preferred inbound message delivery method is via our REST API. Firing off messages to Gnip directly, and not through yet another layer, minimizes delivery issues.
- Bad data – As any aggregator (Friend Feed, Social Thing, MoveableType Activity Streams…) can attest, the data coming across XMPP, RSS, and ATOM feeds today is a mess. From bad/illegal formatting, to bad/illegal data escaping, nearly every activity feed has unique issues that have to be handled on a case by case basis. There will be bugs. We will fix them as they arise. Once again, these issues can be minimized if Data Producers deliver messages directly to Gnip via our REST API.
- Bad policy – This one’s interesting. Gnip makes certain assumptions about the kind of data it receives. In our current implementation we advertise to Data Consumers that Data Producers push all public, per user, change notifications generated within their systems, to Gnip. This usually corresponds to the existing public API policies for said Data Producers. We will eventually offer finely tuned, Data Producer controlled, data policies, but for today’s public facing Gnip service, we do not want to see Data Producers creating publishing policies specific to Gnip. Doing so confuses the middle-ware dynamic we’re trying to create with our current product, and subsequently muddies the water for everyone. Imagine a Data Consumer interacting with a Data Producer directly under one policy, then interacting with Gnip under another policy; confusing. Again, we will, perhaps earlier than we think, cater to unique data policies on a per Data Producer basis, but, we’re not there yet.
While addressing all of these issues is part of our vision, they’re not all resolved out of the gate.