Financial Markets in the Age of Social Media

When you think about it, the stock market is a pretty inspiring thing.

Over the past several centuries, humans have actually created an infrastructure that lets people put their money where their mouth is; an infrastructure that provides a mechanism for daily valuation of companies, currencies and commodities. It’s just unbelievable how far we’ve come and reflecting on the innovation that’s led us here brings to light a common but powerful denominator: Information.

  • When traders began gathering under a buttonwood tree at the foot of Wall Street in the late 1800’s, it was because proximity allowed them to gossip about companies.
  • When Charles Dow began averaging “peaks and flows” of multiple stocks in 1883, his ‘index’ became a new type of data with which to make decisions.
  • In 1975, when the sheer volume of paper necessary for trades became unmanageable, the SEC changed rules to permit electronic trading, allowing for an entirely new infrastructure.
  • And in the 1980’s, when Michael Bloomberg and his partners began building machines (the now ubiquitous Bloomberg Terminals), they tapped into an ever-growing need for more data.

Those are just some examples of the history that is exciting for us @Gnip, because of the powerful signal the market is sending us about social media. Here are some of the more recent signals we’ve seen:

  • The Bank of England announcing they were using Google search results as a means of informing their “nowcasts” detailing the state of the economy.
  • Derwent Capital Markets launching the first social-media based hedge fund this year.
  • The dedication of an entire panel to Social Media Hedge Fund Strategies at the Battle of the Quants conference in London last week.
  • Weekly news articles that describe how traders are using social data as a trading indicator (here’s one as an example).
  • Incorporation of social data into the algorithms of established hedge funds.

In other words, the market is tapping into a new and unique source of information as a means of making trading decisions. And the reason social media data is so exciting is because it offers an unparalleled view into the emotions, opinions and choices of millions of users. A stream of data this size, with this depth and range, has never really existed before in a format this immediate and accessible. And that access is changing how our clients analyze the world and make trades.

We’ve been privileged to see these use cases as we continue to serve a growing number of financial clients. Most exciting to us, as we respond to the market’s outreach for our services, is understanding our pivotal place in this innovation. As the premier source of legal, reliable and realtime data feeds from more than 30 sources of social media- including our exclusive agreement with Twitter- we’re at the center of how firms are integrating this data as an input. And that’s incredible stuff.

Are you in the financial market looking for a social media data provider? Contact us today to learn more! You can reach us at 888.777.7405 or by email.

Gnip. The Story Behind the Name

Have you ever thought “Gnip”. . . well that is a strange name for a company, what does it mean? As one of the newest members of the Gnip team I found myself thinking that very same thing. And as I began telling my friends about this amazing new start-up that I was going to be working for in Boulder, Colorado they too began to inquire as to the meaning behind the name.

Gnip, pronounced (guh’nip), got its name from the very heart of what we do, realtime social media data collection and delivery. So let’s dive in to . . .

Data Collection 101

There are two general methods for data collection, pull technology and push technology. Pull technology is best described as a data transfer in which the request is initiated by the data consumer and responded to by the data publisher’s server. In contrast, push technology refers to the request being initiated by the data publisher’s server and sent to the data consumer.

So why does this matter . . .

Well most social media publishers use the pull method. This means that the data consumer’s system must constantly go out and “ping” the data publisher’s server asking, “do you have any new data now?” . . . “how about now?” . . . “and now?” And this can cause a few issues:

  1. Deduplication – If you ping the social media server one second and then ping it again a second later and there were no new results, you will receive the same results you got one second ago. This would then require deduplication of the data.
  2. Rate Limiting – every social media data publisher’s server out there sets different rate limits, a limit used to control the number of times you can ping a server in a given time frame. These rate limits are constantly changing and typically don’t get published. As such, if your server is set to ping the publisher’s server above the rate limit, it could potentially result in complete shut down of your data collection, leaving you to determine why the connection is broken (Is it the API . . . Is it the rate limit . . . What is the rate limit)?

So as you can see, pull technology can be a tricky beast.

Enter Gnip

Gnip sought to provide our customers with the option: to receive data in either the push model or the pull model, regardless of the native delivery from the data publisher’s server. In other words we wanted to reverse the “ping” process for our customers. Hence, we reversed the word “ping” to get the name Gnip. And there you have it, the story behind the name!

Get your Hack On! Gnip Helps Power an App Developed at the 2011 TechCrunch Disrupt Hackathon

Over 500 individuals recently gathered in New York City for this year’s TechCrunch Disrupt Hackathon. This annual event, fueled by pizza, beer, and Red Bull, features teams of die-hard techies that spend 20 hours, many without sleep (hence the Red Bull), developing and coding the next big idea. Participants compete in a lightning round of pitches in front of a panel of judges with the winners receiving an opportunity to pitch on the main stage at the TechCrunch Disrupt Conference in front of more than 1,000 venture capitalists and industry insiders.

We are excited that one of the apps that was developed at the 2011 Hackathon was powered by Gnip data! We love it when our customers find new and creative ways to use the data we provide.

Edward Kim (@edwkim) and Eric Lubow (@elubow) from SimpleReach (@SimpleReach), which provides next generation social advertising for brands, put a team together to develop LinkCurrent, an app powered by Gnip data and designed to measure the current and future social value of a specific URL. When fully developed, the LinkCurrent app will provide the user with a realtime dashboard illustrating various measures of a URL’s worth — featuring an overall social score, statistics on the Klout Scores of people who have Tweeted the URL, how many times the URL has been Liked on Facebook and posted on Twitter, and geo-location information to provide insight into the content’s reach. Call it influence-scoring for web content.

The hackathon team also included Russ Bradberry (@devdazed) and Carlos Zendejas (@CLZen), also of SimpleReach, Jeff Boulet (@properslang) of EastMedia/Boxcar (@eastmedia/@boxcar), Ryan Witt (@onecreativenerd) of Opani (@TheOpanis), and Michael Nutt (@michaeln3) of Movable Ink (@movableink)– Congratulations to everyone who participated! You created an amazing app in less than 20 hours and developed a creative new use for Gnip data. I highly encourage all of you to check it out: www.linkcurrent.co

Have fun and creative way you have used data delivered by Gnip? We would love to hear about it and you could be featured in our next blog. Drop us an email or give us a call at 888.777.7405.

Why You Should Join Gnip

Gnip’s business is growing heartily. As a result, we need to field current demand, refine our existing product offering, and expand into completely new areas in order to deliver the web’s data. From a business standpoint we need to grow our existing sales team in order to capture as much of our traditional market as possible, as fast as possible. We also need to leverage established footholds in new verticals, and turn those into businesses as big as, or hopefully bigger than, our current primary market. The sales and business-line expansion at Gnip is in full swing, and we need more people on the sales and business team to help us achieve our goals.

From a technical standpoint I don’t know where to begin. We have a large existing customer base that we need to keep informed, help optimize, and generally support; we’re hiring technical support engineers. Our existing system scales just fine, but software was meant to iterate, and we have learned a lot about handling large volumes of real-time data streams, across many protocols and formats, for ultimate delivery to large numbers of customers. We want to evolve the current system to even better leverage computing resources, and provide a more streamlined customer experience. We’ve also bit off a historical data set indexing challenge that is well… of true historical proportion. The historical beast needs feeding, and it needs big brains to feast on. We need folks who know Java very well, have search, indexing, and large data-set management backgrounds.

On the system administration side of things… if you like to twiddle IP tables, tune MTUs for broad geographic region high-bandwidth data flow optimization, handle high-volume/bandwidth streaming content, then we’d like to hear from you. We need even more sys admin firepower.

Gnip is a technical product, with a technical sale. Our growth has us looking to offload a lot of the Sales Engineering support that the dev team currently takes on. Subsequently we’re looking to hire a Sales Engineer as well.

Gnip has a thriving business. We have a dedicated, passionate, intelligent team that knows how to execute. We’re building hard technology that has become a critical piece of the social media ecosystem. Gnip is also located in downtown Boulder, CO.

http://gnip.com/careers

25 Million Free Tweets on Power Track

Last week we announced Twitter firehose filtering. This week we’re celebrating the news with Free Tweets for all. Sign up by February 28th to enjoy no licensing fees on your first 25 million Tweets in your first 60 days using Power Track. 

Power Track offers powerful filtering of the Twitter firehose, guaranteeing 100% Tweet delivery. For instance, filter by keyword or username to access all Tweets that match the criteria you care about and have all of the matching results delivered to you in realtime via API. Power Track supports Boolean operators, can match your filtering criteria even within expanded URLs, and has no query volume or traffic limitations, helping you access all of the data you want. And it’s only available from Gnip, currently the only authorized distributor of Twitter data via API.

The licensing fee for Power Track is $.10 per 1,000 Tweets, but we’re waiving that fee for the first 25 million Tweets in 60 days for Power Track customers who sign up by February 28th. 1-year agreement and Gnip data collector fee still required.

Learn More or Contact Us to start testing Power Track for firehose filtering. Cheers!

Strata, Big Data & Big Streams

Gnip had a great time at O’Reilly’s Strata 2011 conference in California last week. We signed up several months ago as a big sponsor without knowing exactly how things were going to come together. The bet paid off and Strata was a huge success for us, and the industry at large. We were blown away with the relevance of the topics discussed and the quality of the attendees and discussions that were sparked. I was amazed at how much knowledge everyone now has surrounding big data set analysis and processing. Technologies that were immature and new just a few years ago, are now baked into the ecosystem and have become tools of the trade (e.g. Hadoop). All very cool to see.

 

That said, there remains a distinct gap between big data set handling and high-volume/real-time data stream handling. We’ve come a long way in handling monster data set processing in batch or offline modes, but we have a long way to go when it comes to handling large streaming data set challenges. Hillary Mason, of bit.ly, hit this point squarely in her “What Data Tells Us” talk at Strata. We can open sourcely fan out ungodly amounts of processing… like piranha on fresh meat. However, blending that processing, and high-latency transactions, into real-time streams of thousands of activities per second is not as refined and well understood. Frankly, I’m shocked at the number of engineers I run into that simply don’t understand asynchronous programming at all.

 

The night before the conference started, Pete Warden drove BigDataCamp @Strata, where Mike Montano from BackType gave a high-level overview of their infrastructure. He laid out a few tiers and described the “speed” tier as something that did a lot of work on high-volume streams, and a “batch” tier that did stuff in a more offline manner. The blend of approaches was an interesting teaser into how Big Stream challenges can be handled. Gnip’s own infrastructure has had to address these challenges of course, and we launched into a thread of detail in our Expanding The Twitter Firehose post awhile back.

 

Big Stream handling occupies a good part of my brain. I’d like to see Big Data discussion start to unravel Big Stream challenges as well.

New Gnip & Twitter Partnership

We at Gnip have been waiting a long time to write the following sentence: Gnip and Twitter have partnered to make Twitter data commercially available through Gnip’s Social Media API. I remember consuming the full firehose back in 2008 over XMPP. Twitter was breaking ground in realtime social streams at a then mind-blowing ~6 (six) Tweets per second. Today we see many more Tweets and a greater need for commercial access to higher volumes of Twitter data.

There’s enormous corporate demand for better monitoring and analytics tools, which help companies listen to their customers on Twitter and understand conversations about their brands and products. Twitter has partnered with Gnip to sublicense access to public Tweets, which is great news for developers interested in analyzing large amounts of this data. This partnership opens the door to developers who want to use Twitter streams to create monitoring and analytics tools for the non-display market.

Today, Gnip is announcing three new Twitter feeds with more on the way:

  • Twitter Halfhose. This volume-based feed is comprised of 50% of the full firehose.
  • Twitter Mentionhose. This coverage-based feed provides the realtime stream of all Tweets that mention a user, including @replies and retweets. We expect this to be very interesting to businesses studying the conversational graph on Twitter to determine influencers, engagement, and trending content.
  • Twitter Decahose. This volume-based product is comprised of 10% of the full firehose. Starting today, developers who want to access this sample rate will access it via Gnip instead of Twitter. Twitter will also begin to transition non-display developers with existing Twitter Gardenhose access over to Gnip.

We are excited about how this partnership will make realtime social media analysis more accessible, reliable, and sustainable for businesses everywhere.

To learn more about these premium Twitter products, visit http://gnip.com/twitter, send us an email at info@gnip.com, or appropriately, find us on Twitter @gnip.