Big Boulder: Creating the Social Data Ecosystem with Twitter

Ryan Sarver and Doug Williams of the Twitter platform discuss the launch of commercial public social data nearly two years ago and how the Twitter firehose has evolved.

Doug Williams and Ryan Sarver at Big Boulder

Ryan and Doug of Twitter have been there three years and describe the experience as “learning as you go.” What excited Doug was for the first time was that there was an open data source on the web, and he became the API evangelist. During the last six years, Twitter now has 140 million users. During that incredible growth page, the scaling has been tough for the company yet having that many users has created many opportunities for people to build products on top of that social data.

Several years ago, Twitter decided to change how data was syndicated. Twitter was built to serve consumers, so at the time it was hard to take resources away from that to support the API. Two years ago, Twitter was receiving multiple requests and weren’t able to provide enterprise-level support for features. At the time, Joe Fernandez of Klout was in the same building and was making multiple requests because Klout felt they could create much cooler features if they had access to more types of metadata. Wanting to focus on serving the consumers that were using the product but still wanting to support the API, Twitter decided to work with outside companies such as Gnip to provide its social data to provide that enterprise level support. Twitter decided to select a small number of companies to provide the data because they wanted to know where the data was going and how it was being used. Doug Williams called working with Gnip one of the most successful partnerships that Twitter has ever had. But it was important to Twitter that by providing companies with data, they wanted to create value for both directions. The companies represented at Big Boulder are helping to create a better audience, encouraging companies to invest time and resources into Twitter.

One of the most frequent questions that Twitter is asked if they’re going to do analytics. What Ryan and Doug talked about is that they’re going to continue to build out baseline features,  but they’ll rely on other companies to provide the features that companies need to do business and interact on Twitter. They want companies to get the analytics they need and recognize that the analytics that other social media companies provide add value to Twitter. When people get the analytics they need, they understand the value that Twitter provides and it powers their decisions to use Twitter and advertise on the Twitter platform. They also talked about how Twitter is firmly committed to providing Twitter data and are invest in by making sure companies receive all the tweets they need.

The conversation segued to the firehose, which Gnip still receives requests to have access to the entire firehose but most companies are realizing that they really don’t need it. Receiving the entire firehose can be prohibitive because it can be too much to consume and expensive. Ryan and Doug talked about how they will continue to license the firehose when it makes sense but the overall trend is to service those use cases less and less. Each case is evaluated on a case by case basis. Twitter is committed to ensuring that businesses have a clear path to getting the social data they need, as they recognize businesses are being built around the Twitter data.

Last week Twitter announced expandable tweets, or as they call it internally, In Tweet Media. This is an exciting advancement for the platform because it provides more options to chose how your content on Twitter gets consumed. Publishers want to have control on how that social data gets syndicated. Since news is frequently breaking on Twitter, publishers want to be able to tell their stories on the platform, allowing them to drive greater distribution with Twitter.

So exactly what is Twitter doing about spam? A lot. According to Ryan, the spam prevention team is one the largest at Twitter, and they’ve made several acquisitions around it. As they pointed out, spam is still a problem for email which has been around for many years. Spam is an ongoing battle that they’ll have to fight for the existence of the program. Doug talked about how when people test the Gnip platform, they often start a new account and then the tweet can be marked as spam and doesn’t make it through. Doug said to the delight of Jud and Chris, “messages marked as spam that you sent from a test account is Twitter’s issue, not Gnip’s issue.” The takeaway is not to create a new account to test Twitter.

Twitter now has 140 million users, and 400 millions tweets every couple of days. Yet the team still sees lots of room for growth, and that they’d love to see everyone with a phone using Twitter. In fact, they talked about during the Q&A with how often Twitter is used during protests that they put much consideration into making sure that Twitter can be used without a smart phone and that Tweets are quickly delivered across the world. As they’re trying to encourage more use, they recognize that much of the world knows about Twitter, but the gap is to help people understand why they need to be part of the platform.

Big Boulder is the world’s first social data conference. Follow along at #BigBoulder, on the blog under Big BoulderBig Boulder on Storify and on Gnip’s Facebook page.