Who Knew First: Steve Jobs or Aron Pinson?

While Steve Jobs’ resignation yesterday had investors anxiously watching how $AAPL fared in trading, we at Gnip were having fun watching a different ticker- the realtime Twitter feed.

As you can see from the graph below (these represent the number of “Steve Jobs” mentions per minute*), Twitter showed an incredible spike almost immediately. Apple-specific activity peaked 11 minutes after the news broke, showing how quickly the word spread. Honors for first tweet go to @AronPinson, who must have some blazing fast typing skills.

Once again, it’s incredible to see how social media is quickly becoming a trusted means of accessing and delivering realtime information.

*For more details on how we conducted this search across the millions of real-time tweets we have access to, contact us!

Can Social Media Data Offset Market Volatility?

It’s been a volatile time for the markets the last few weeks. Today especially – the Dow closed down 635 points; S&P, -80; NASDAQ, -175. While there’s no shortage of opinions on how/why the market will/will not recover, one thing is for certain – having the right data to make decisions is more important than ever.

Part of the reason for this is that the markets are clamoring for trends – definitive information on stock trends and market sentiment. Which is why it’s exciting to see how our finance clients are using the Gnip realtime social media data feeds. In a time of increased volatility, our hedge fund (and other buy-side) clients are leveraging social media data as a new source of analysis and trend identification. With an ever-growing number of users, and Tweets, per day, Twitter is exploding, and market-leading funds are looking at the data we provide as a way to more accurately tap into the voice of the market. They’re looking at overall trend data from millions of Tweets to predict the sentiment of consumers as well as researching specific securities based on what’s being said about them online.

While the early-adopters of this data have been funds, this type of analysis is available to individuals as well. Check out some start-ups doing incredible things at the intersection of finance and social media:

  • Centigage provides analytics and intelligence designed to enable financial market participants to use social media in their investment decision-making process
  • SNTMNT offers an online tool that gives daily insights into online consumer sentiment surrounding 25 AEX funds and the index itself

For the first time in history, access to (literally) millions of voices is at our fingertips. As the market continues its volatility, those voices gain resonance as a source of pertinent information.

Handling High-Volume, Realtime, Big Social Data

The social ecosystem has become the pulse of the world. From delivering breaking news like the death of Osama Bin Laden before it hit mainstream media to helping President Obama host the first Twitter Town Hall, the realtime social web is flooded with valuable information just waiting to be analyzed and acted upon. With millions of users and billions of social activities passing through the ever-growing realtime social web each day, it is no wonder that companies need to reevaluate their traditional business models to take advantage of this big social data.But with the exponentially ever-growing social web, massive amounts of data are pouring into and out of social media publishers’ websites and APIs every second. In a talk I gave at GlueCon a couple of months ago, I ran down some math to put things into perspective. The numbers are a little dated, but the impact is the same. At that time there were approximately 155,000,000 Tweets per day and the average size of a Tweet was approximately 2,500 Bytes (keep in mind this could include Retweets).

A Little Bit of Arithmetic

155,000,000 Tweets/day   2,500 Bytes = 387,500,000,000 Bytes/day

387,500,000,000 Bytes/day  24 Hours = 16,145,833,333 Bytes/hour

16,145,833,333 Bytes/hour 60 minutes = 269,097,222 Bytes/minute

269,097,222 Bytes/minute 60 second = 4,484,953 Bytes/second

4,484,953 Bytes/second  1,048,576 Bytes/megabyte = 4.2 Megabytes/second

And in terms of data transfer rates . . .

1 Megabyte/second = 8 Megabits/second

So . . .

4.2 Megabytes/second  8 Megabits/Megabyte = 33.8 Megabits/second

That’s a Lot of Data

So what does this mean for the data consumers, the companies wanting to reevaluate their traditional business models to take advantage of vast amounts of Twitter data? At Gnip we’ve learned that some of the collective industry data processing tools simply don’t work at this scale: out-of-the-box HTTP servers/configs aren’t sufficient to move the data, out-of-the-box config’d TCP stacks can’t deliver this much data, and consumption via typical synchronous GET request handling isn’t applicable. So we’ve built our own proprietary data handling mechanisms to capture and process mass amounts of realtime social data for our clients.

Twitter is just one example. We’re seeing more activity on today’s popular social media platforms and a simultaneous increase in the number of popular social media platforms. We’re dedicated to seamless social data delivery to our enterprise customer base and we’re looking forward to the next data processing challenge.

Financial Markets in the Age of Social Media

When you think about it, the stock market is a pretty inspiring thing.

Over the past several centuries, humans have actually created an infrastructure that lets people put their money where their mouth is; an infrastructure that provides a mechanism for daily valuation of companies, currencies and commodities. It’s just unbelievable how far we’ve come and reflecting on the innovation that’s led us here brings to light a common but powerful denominator: Information.

  • When traders began gathering under a buttonwood tree at the foot of Wall Street in the late 1800’s, it was because proximity allowed them to gossip about companies.
  • When Charles Dow began averaging “peaks and flows” of multiple stocks in 1883, his ‘index’ became a new type of data with which to make decisions.
  • In 1975, when the sheer volume of paper necessary for trades became unmanageable, the SEC changed rules to permit electronic trading, allowing for an entirely new infrastructure.
  • And in the 1980’s, when Michael Bloomberg and his partners began building machines (the now ubiquitous Bloomberg Terminals), they tapped into an ever-growing need for more data.

Those are just some examples of the history that is exciting for us @Gnip, because of the powerful signal the market is sending us about social media. Here are some of the more recent signals we’ve seen:

  • The Bank of England announcing they were using Google search results as a means of informing their “nowcasts” detailing the state of the economy.
  • Derwent Capital Markets launching the first social-media based hedge fund this year.
  • The dedication of an entire panel to Social Media Hedge Fund Strategies at the Battle of the Quants conference in London last week.
  • Weekly news articles that describe how traders are using social data as a trading indicator (here’s one as an example).
  • Incorporation of social data into the algorithms of established hedge funds.

In other words, the market is tapping into a new and unique source of information as a means of making trading decisions. And the reason social media data is so exciting is because it offers an unparalleled view into the emotions, opinions and choices of millions of users. A stream of data this size, with this depth and range, has never really existed before in a format this immediate and accessible. And that access is changing how our clients analyze the world and make trades.

We’ve been privileged to see these use cases as we continue to serve a growing number of financial clients. Most exciting to us, as we respond to the market’s outreach for our services, is understanding our pivotal place in this innovation. As the premier source of legal, reliable and realtime data feeds from more than 30 sources of social media- including our exclusive agreement with Twitter- we’re at the center of how firms are integrating this data as an input. And that’s incredible stuff.

Are you in the financial market looking for a social media data provider? Contact us today to learn more! You can reach us at 888.777.7405 or by email.

Announcing Multiple Connections for Premium Twitter Feeds

A frequent request from our customers has been the ability to open multiple connections to Premium Twitter Feeds on their Gnip data collectors. Our customers have asked and we have delivered!

While multiple connections to standard data feeds have been available for quite some time, we have only allowed one connection to our Premium Twitter Feeds.  Beginning today you will be able to open multiple mirrored connections to Power Track, Decahose, Halfhose, and all of our other Premium Twitter Feeds.  This feature will be helpful when testing connections to your Gnip data collector in different environments (such as staging or review) without having an impact on your production connection.

You may be saying “Sounds great Gnip, but will I be charged the standard Twitter licensing fee for the same tweet delivered across multiple connections?”. The answer is no!  You will pay a small flat fee per month for each additional connection.  If you’re interested in adding Multiple Connections to your Premium Twitter Feed please Contact Us.

New Premium Feed: Twitter Link Stream

Last month we announced a partnership with Twitter and three commercial Twitter feeds. Today we’re excited to announce a new premium Twitter feed: the Twitter Link Stream is a feed of all Tweets that contain any URL.

Marketers are always asking for better ways to track the influence of their campaigns. Businesses always want to know more about how their users are discovering and reacting to their brands. The Link Stream represents a tremendous opportunity for business that want to track the viral nature of many different webpages at once and for businesses hoping to understand how the social web influences its participants’ actions.

Like our other commercial Twitter data feeds, the Link Stream is available today for licensing by companies that don’t display Tweets to the general public (though displaying a few Tweets to your paying customers is generally fine). Contact us at info@gnip.com for more information.

What Facebook Data is Available from Gnip's Social Media API?

Facebook is among the most in-demand (but also among the most challenging) social media sources to access. Most Facebook conversation data is private and so it’s not accessible via Facebook’s API or any of Gnip’s feeds. Facebook data availability is also pretty confusing to understand and the rules keep changing. So, let’s clarify what kinds of Facebook information we can offer through our social media API today. 

What Facebook Data is Available from Gnip?
Within the realm of publicly accessible data only, we provide:

  • User page content: status updates, wall posts, comments
  • Fan page content: wall posts, comments (probably more than you’ll find from any service), “Like” count, historical data up to 90 days


How Can You Get the Data?

Instead of a firehose of Facebook data, you enter parameters indicating what you want to find:

  • Keyword search
    You provide a list of keywords. We’ll return public mentions of those keywords.
  • Username search
    You provide a list of usernames. We’ll return publicly accessible posts generated by those users.
  • Fan page search
    You provide a list of fan pages. We’ll return publicly available posts and comments on those fan pages.

    While these lists are vastly simplified, we hope they’ll clarify what kinds of Facebook data most businesses can access legally, and exactly what Facebook data Gnip provides.

    Oh, and one last thing. We’re sometimes asked how we feel about Facebook’s privacy policies. At Gnip, we don’t make the rules — we just play by them. Our job is to facilitate access to the social data that publishers (like Facebook) officially make accessible to our customers.

    Best wishes to you with your Facebook data collection! If you think we might be able to help, please drop us a note.