HOW-TO: Twitter Search Publisher

There has been some confusion around how to leverage Gnip’s Twitter Search (“twitter-search”) Publisher. We have work to do in order to clarify this use case from a usability/documentation standpoint, but in the meantime hopefully the following clarifies things a bit.

First off, “twitter-search” is a Polled Publisher which means it is subject to high latencies, as well as gaps in coverage. Secondly, we overload the “keyword” rule type in Filters in order to provide a mechanism for you to enter your http://search.twitter.com compatible queries (see http://search.twitter.com/operators for more information). Any query you can run on http://search.twitter.com, can be added to your Gnip filter as a “keyword” rule.

For example, if you search Twitter for “Boulder, CO” (including the quotes), Twitter considers that a literal, case-insensitive, phrase search; and so will Gnip. “Boulder, CO” (excluding the quotes), yields an OR search on Twitter; and hence the same in Gnip. If you search for “cars AND trucks” you get Boolean search operator behavior in Twitter, and subsequently in Gnip as well.

In short, we pass through the literal queries/strings that you hand Gnip, straight on through to Twitter. The “keywords” are opaque to Gnip. The only trick is in ensuring your “keywords” are entered into Gnip appropriately.

Through Gnip’s web interface, you can add comma separated keywords to a Filter. This is usually straightforward, however in the twitter-search Publisher case, it takes extra care to get the results you want, especially when you want to include commas or quotes in your queries. As a result, the format of the keywords entered in a twitter-search Publisher Filter must conform to csv quoting to ensure your queries get executed properly.

Through Gnip’s REST interface, you encapsulate the keywords within XML <rule> elements, so the csv quoting rules can be ignored.

For some further examples of how to add twitter-search keywords, see the Gnip API documentation.

As a final note, the overload of “keyword” rule types in Filters is something we’re experimenting with and is subject to change.