We always get some interesting requests for doing additional processing on data sources.   Some of these are addressed using Gnip filters, but others do not really fit the filter model.   In able to support richer or more complex data processsing we have built some additional features into the Gnip platform.   The first new publisher using some of these new features is “Digg-2000“.

Digg-2000 Publisher — What is it?

Lots of people submit stories to Digg and lots of other people Digg the stories which allows more popular information to rise to the top in being discoverable.    Several Gnip customers asked if we could make it possible for them to only receive stories that had a specific number of Diggs.   We asked Digg about the idea and they said it sounded great since they have a Twitter account that provides a similar type of feature, so the Digg-2000 Gnip publisher was born.

Digg-2000 Publisher — How it works?

On the Gnip platform we have set up a publisher that is listening to activities on Digg.

• When new stories are submitted to Digg we pick those new activites up along with the Digg count on the story and they are posted to the standard Gnip Digg publisher.
• With every new Digg on a story we increment our tracking of the story and when we hit 2000 diggs we re-post the original story to the Digg-2000 publisher.
• The configuration of the Digg-2000 publisher allows for us to turn two different dials.
  • The default configuration only will re-post stories that were first posted on Digg in the last two days.   This means we are looking for current active stories and not stories that were posted months ago and through a slow and steady interest finally hit 2000 Diggs.
  • The default configuration re-posts at 2000 Diggs.   This can be set to any number of Diggs — 100, 1000, 2000, etc.

With everything going on here at Gnip we want to try and regularly preview some of the new features we are working on so people can send us feedback and plan ahead. One feature that we know a lot of people are interested in us delivering is usage and operational reporting and analytics. The reasons for adding an analytics dashboard are many and the primary reason is that we believe it will help companies and developers better understand the richness and variability of the data streams they care about.

Below is one example of the analytics features that we are planning to provide in the near future. This image shows the Digg Data Stream summary view with individual diggs, comments and submissions per second being streamed by the Gnip platform.

Figure: Gnip — Digg Data Stream Activity View

Obviously we could pivot on the summary view to show different types of details depending on any number of variables that Gnip partners and customers find interesting. If your company has specific requests for analytics and reporting please let us know.