Guest Post, Rick Boykin: Gnip C# .NET Convenience Library

Microsoft .NETNow that the new Gnip convenience libraries have been published for a few weeks on GitHub, I’m going to tell you a bit about the libraries that I’m currently responsible for, the .NET libraries.  So, let’s dive in, shall we… The latest versions of the .NET libraries are heavily based on the previous version of the Java libraries, with a bit of .NET style thrown in. What that means is that I used Microsoft’s Java Language Conversion Assistant as a starting point, mixed in some shell scripting like Bash, Sed and Perl to fix the comments, and some of the messy parts that did not translate very well. I then made it more C# like by removing Java Annotations, adding .NET attributes, taking advantage of .NET native XML Serializer, utilizing System.Net.HttpWebRequest for communications, etc. It actually went fairly quick.  The next task was to start the Unit testing deep dive.

I have to say, I really didn’t know anything about the Gnip model, how it worked, or what it really was, at first. It just looked like an interesting project and some good folks. Unit testing, however, is one place where you learn about the details of how each little piece of a system really works. And since hardly any of my tests passed out of the gate (and I was not really even convinced that I even had enough tests in place,) I decided it was best to go at it till I was convinced. The library components are easy enough. The code is really separated into two parts. The first component is the Data Model, or Resources, which directly map to the Gnip XML model and live in the Gnip.Client.Resource namespace. The second component is the Data Access Layer or GnipConnection. The GnipConnection, when configured, is responsible for passing data to, and receiving data from, the Gnip servers.  So there are really only two main pieces to this code. Pretty simple: Resources and GnipConnection. The other code is just convenience and utility code to help make things a little more orderly and to reduce the amount of code.

So yeah, the testing… I used NUnit so folks could utilize the tests with the free version of VisualStudio, or even the command line if you want. I included a Gnip.build Nant file so that you can compile, run the tests, and create a zipped distribution of the code. I’ve also included an nunit project file in the Gnip.ClientTest root (gnip.nunit) that you can open with the NUnit UI to get things going. To help configure the tests, there is an App.config file in the root of the test project that is used to set all the configuration parameters.

The tests, like the code, are divided onto the Resource objects tests and the GnipConnection tests (and a few utility tests). The premise of the Resource object tests is to first ensure that the Resource objects are cool. These are simple data objects with very little logic built in (which is not to say that testing them thoroughly is not the utmost important.) There is a unit test for each one of the data objects and they proceed by ensuring that the properties work properly, the DeepEquals methods work properly, and that the marshalling to and from XML works properly. The DeepEquals methods are used extensively by the tests, so it is essential that we can trust them. As such, they are fairly comprehensive. The marshalling and un-marshalling tests are less so. They do a decent job; they just do not exercise every permutation of the XML elements and attributes. I do feel that they are sufficient enough to convince me that things are okay.

The GnipConnection is responsible for creating, retrieving, updating and deleting Publishers and Filters, and retrieving and publishing Activities and Notifications. There is also a mechanism built into the GnipConnection to get the Time from the Gnip server and to use that Time value to calculate the time offset between the calling client machine and the Gnip server. Since the Gnip server publishes activities and notifications in 1 minute wide addressable ‘buckets’, it is nice to know what the time is on the Gnip server with some degree of accuracy. No attempt is made to adjust for network latency, but we get pretty close to predicting the real Gnip time. That’s it. That little bit is realized in 25 or so methods on the GnipConnection class. Some of those methods are just different signatures of methods that do the same thing only with a more convenient set of parameters. The GnipConnection tests try to exercise every API call with several permutations of data. They are not completely comprehensive. There are a lot of permutations. But, I believe they hit every major corner case.

In testing all this, one thing I wanted to do was to run my tests and have the de-serialization of the XML validate against the XML Schema file I got from the good folks at Gnip. If I could de-serialize and then serialize a sufficiently diverse set of XML streams, while validating that those streams adhere to the XML Schema, then that was another bit of ammo for trusting that this thing works in situations beyond the test harness. In the Gnip.Client.Uti namespace there is a helper class icalled XmlHelper that contains a singleton of itself. There is a property called ValidateXml that can be reached like this XmlHelper.Instance.ValidateXml. Setting that to true will cause the XML to be validated anytime it is de-serialized, either in the tests or from the server. It is set to true in the tests. But, it doesn’t work with the stock Xsd distributed by Gnip.That Xsd does not include an element definition for each element at the top level which is required when validating against a schema. I had to create one that did. It is semantically identical to the Gnip version; it just pulls things out to the top level. You can find the custom version in the Gnip.Client/Xsd folder. By default it is compiled into the Gnip.Client.dll.

One of the last things I did, which had nothing really to do with testing, is to create the IGnipConnection interface. Use it if you want. If you use some kind of Inversion of Control container like Unity, or like to code to interfaces, it should come in handy.
That’s all for now. Enjoy!

Rick is a Software Engineer and Technical Director at Mondo Robot in Boulder, Colorado. He has been designing and writing software professionally since 1989, and working with .NET for the last 4 years. He is a regular fixture at the Boulder .NET user’s group meetings and the is a member of Boulder Digital Arts.

Gnip Beta 2 Launching Today, New www.gnip.com coming

Several new updates to Gnip websites are being released this week that will impact companies and people using the Gnip developer site and the Gnip corporate website.

1) We just started a maintenance on the demo system with a post on the forum and to the @gnipsupport Twitter account.

  • The reason for this maintenance period is that we are moving to the next stage of this release and entering Beta 2.  Once we come back online the Beta developer site that is currently hosted at demo.gnip.com will be available at a new location:  http://api.gnip.com.
  • We are migrating all the account information to the new site so if you have an existing account on the prior demo site it will be waiting for you at http://api.gnip.com.
  • If your integration to the demo Gnip API is being done using our convenience libraries then you will need to manually update your hostname to https://api-v21.gnip.com .
  • Documentation is available for the new Gnip version 2.1 API and schema at http://gnip.com/docs
  • The new URL, http://api.gnip.com, will be the final destination and future home for the version 2.1 version of the Gnip platform after we conclude Beta 2.  We are making this update now so that the integrations completed to api.gnip.com will contiue to work once we flip the switch to making this generally available sometime in the next few weeks.
  • For people using the current version of the Gnip platform (v2.0) we are continuing to provide support and will do so through the Beta 2 period and 30 days after the general release of version 2.1.

2) We are also doing an update to our corporate website.  Today the website is located at www.gnipcentral.com, and because we were able to acquire the gnip.com domain we are moving!  Everyone going to any gnipcentral.com based URL will be redirected to the apporpriate gnip.com URL.   So, look for the new website before the end of the week.

Here is a glimpse of the new home page for the curious…..

gnip.com

Solution Spotlight: Storytlr Using Gnip for Real-Time Social Data Integration

Storytlr

Who is Storytlr?
Storytlr provides a life streaming service that allows people to bring together their entire web 2.0 life and assemble their content to tell stories in a whole new way.  Learn more at their website, http://storytlr.com/, or their blog, http://blog.storytlr.com/.

Real-world results Storytlr says they are realizing from using Gnip
Storytlr is using Gnip to provide real-time data integration to Twitter, Digg, Delicious and Seesmic.  Since Storytrl starting using Gnip they have seen a reduction in the latency for the data integration of these social media activity streams (i.e. the time elapsed for a tweet, digg, or event notice to show up in the Storytlr service from a third-party is now real-time). Read more on how Storytlr added real-time integration using Gnip in their recent blog post.

We are looking forward to working more with the Storytlr team as we roll out more publishers that they can take advantage of in their business. 

Solution Spotlight: Soup.io is Now Using Gnip

Soup.io is now using the Gnip messaging platform for their web API data integration needs. Welcome Soup.io!

Who is Soup.io?
Soup.io provides a easy to use micro-blogging and lifestream service that serves as an aggregator for your public social media feeds. Visit their website at http://www.soup.io/or their blog at http://updates.soup.io/ to learn more.

Real-world results Soup.io says they are realizing from using Gnip
Soup.io is using Gnip to provide data integration to Twitter, and they have seen a reduction in the latency for their Twitter integration (i.e. the time elapsed for a tweet to show up in the Soup.io service) since moving to Gnip. Now Soup.io users should see their Twitter notices show up within a minute of them being sent on the Twitter service. Since Gnip also provides data streams from many other providers as well (Flickr, Delicious, etc) Soup.io is working to use Gnip as the way to access and integrate to these services in the future.

Solution Spotlight: Strands Now Using Gnip

Strands is the newest company using the Gnip messaging platform for their web API data integration needs. Welcome Strands and thank you to Aaron for sharing what the team is doing!

Who is Strands?
Strands develops technologies to better understand people’s taste and help them discover things they like and didn’t know about. Strands has created a social recommendation engine that is able to provide real-time recommendations of products and services through computers, mobile phones and other Internet-connected devices. This enables users to discover new things, based on their online, offline and mobile activities. The Strands.com website helps people discover new things from other people. Visit http://www.strands.com to learn more.

Real-world results Strands says they are realizing from using Gnip
Strands.com is now able to give people updates faster and more reliably. In addition, Strands has seen reduced load on their system by not having to poll for updates on sites like Twitter, Flickr, Delicious, and Digg. Gnip allows Strands to receive push data from several of these sites, and at a minimum receive notifications when a user on these sites has made an update.

More Examples of How Companies are Using Gnip

We have noticed that we are interacting with two distinct groups of companies; those who instantly understand what Gnip does and those that struggle with what we do, so we decided to provide a few detailed real-world examples of the companies we are actively working with to provide data integration and messaging services today.

First, we are not an end-user facing social aggregation application. (We repeat this often.) We see a lot of people wanting to put Gnip in that bucket along with social content aggregators like FriendFeed, Plaxo and many others. These content aggregators are destination web sites that provide utility to end users by giving them flexibility to bring their social graph or part of their graph together in one place. Also, many of these services are now providing web APIs that allow people to use an alternative client to interact with their core services around status updates and conversations as well other features specific to the service.

Gnip is an infrastructure service and specifically we provide an extensible messaging system that allows companies to more easily access, filter and integrate data from web based APIs. While someone could use Gnip as a way to bring content into a personal social media client they want to write for a specific social aggregator it is not something we are focused. Below are the company use cases we are focused:

  1. Social content aggregators: One of the main reasons we started Gnip was to solve the problems being caused by the point-to-point integration issues that were springing up with the increase of user generated content and corresponding open web APIs. We believe that any developer who has written a poller once, twice, or to their nth API will tell you how unproductive it is to write and maintain this code. However, writing one-off pollers has become a necessary evil for many companies since the content aggregators need to provide access to as many external services as possible for their end users. Plaxo, who recently integrated to Gnip as a way to support their Plaxo Pulse feature is a perfect example, as are several other companies.
  2. Business specific applications: Another main reason we started Gnip was that we believe more and more companies are seeing the value of integrating business and social data as a way to add additional compelling value to their own applications. There are a very wide set of examples, such as how Eventvue uses Gnip as a way to integrate Twitter streams into their online conference community solution, and the companies we have talked to about how they can use Gnip to integrate web-based data to power everything from sales dashboards to customer service portals.
  3. Content producers: Today, Gnip offers value to content producers by providing developers an alternative tool that can be used to integrate to their web APIs. We are working with many producers, such as Digg, Delicious, Identi.ca, and Twitter, and plan to continue to grow the producers available aggressively. The benefits that producers see from working with Gnip include off-loading direct traffic to their web apis as well as providing another channel to make their content available. We are also working very hard to add new capabilities for producers, which includes plans to provide more detailed analytics on how their data is consumed and evaluating publishing features that could allow producers to define their own filters and target service endpoints and web sites where they want to push relevant data for their own business needs.
  4. Market and brand research companies: We are working with several companies that provide market research and brand analysis. These companies see Gnip as an easy way to aggregate social media data to be included in their brand and market analysis client services.

Hopefully this set of company profiles helps provide more context on the areas we are focused and the typical companies we are working with everyday. If your company does something that does not fit in these four areas and is using our services please send me a note.

New Gnip Data Consumer: Retaggr

Just got word that the very cool London-based service retagger has started working with Gnip to reduce the number of calls they make to data providers.  Here’s what co-founder Nicholas Smit has to say:

Retaggr aggregates your online identity into an interactive embeddable business card. Contained within it is actual content from services like flickr, twitter and so on. We’re using Gnip to receive notifications about when our users publish data on these services, so we don’t have to poll them unnecessarily. This is great for efficiency, alleviates problems with API quotas, and helps us provide a consistent level of service to our users.

Welcome Nicholas and the Retaggr team!