Evaluating: In-house or Outsourcing Solutions

Infomart, an industry leader in traditional media monitoring, was struggling with its legacy technology when it came to social media monitoring. The company, which has been around as a media monitoring company since 1986, had what it refers to as “Frankenstein” technology where they were adding on hodgepodge solutions to meet current needs. Users of their system had to employ multiple tools for traditional news and social media monitoring.

Getting to the Right Product

They came to a point where they had to decide if they wanted to do social media monitoring in-house or not. Ultimately they moved towards moving it out of house and found the the perfect solution by using the Infochimps cloud paired with Gnip data, which allowed them to ramp up quickly on the front end and focus on their other strengths and consulting.

During the process, Infomart also realized that their customers had to always do custom analysis to get the data they needed, which led them to their current solution. Their current solution focuses on three aspects: portfolios, integration and collaboration. Portfolios gives customers a collection point for information, tools and work about a subject, project, crisis, department, client, issue or campaign. Tools to work with data. Integration is about the ability to work on any content piece with the same tools, regardless of source/type of content. Infomart is rare in that it has the ability to integrate with both traditional types of media, as well as social media from Gnip including Twitter, Facebook, YouTube, blogs and forums. Infomart also allows for collaboration within its platform so you can email content or assign tasks right away.

As Informart says, “Data is the shining star of the platform.”

Behind the Product

Ultimately, what made Infomart’s new solution so flawless was the ability to build on top of Infochimps solution. Infochimps is a suite of enterprise class cloud services that makes it simple and quick to deploy big data applications. Infochimps makes big data, which is traditionally slow and cumbersome, much easier and less complex. Another advantage of Infochimps is its ability to do real-time analytics, ad hoc analytics and batch analytics.

To learn more how all three organizations create returns in real time, you can see the rest of the webinar here.

1. You entered many Kaggle contests before you started working for Kaggle. What were some of the biggest lessons you learned?

Indeed, many years back I competed in the Netflix prize. As looking at spreadsheets goes, it was a thrilling experience (albeit also quite humbling). I took out a $3,000 loan from my parents to buy a computer with enough RAM to even load the data. A few years later, I was in the final throes of my doctorate when Kaggle was founded. I made it a side hobby and spent my evenings trying to port what I researched in my biomedical engineering day job to all sorts of crazy problems.

The fact that I was able to get anywhere is evidence that domain expertise can be overstated when working within different fields. If I can price bonds, it’s not that I understand bond pricing; it’s that I can learn how bonds were priced in the past. This is not to say that domain expertise is not important or necessary to make progress, but that there is a set of statistical skills that support all data problems.

What are these skills? People make them sound more fancy than they really are. It’s not about knowing the latest, greatest, machine learning methods. Will it help? Sure, but you don’t need to train a gigantic deep learning net to solve problems. The lesson Kaggle reinforced for me was the importance of the scientific method applied to data. It was really basic, embarrassing things: e.g. When you do something many times, the results need to be the same. When you add a bit of noise to the input, the output shouldn’t change too much. If two models tell you the same thing, but a little differently, then you can blend them and do better. If two models tell you something completely different, then you have a bug–or even better, a massive flaw in your entire understanding of what you’re doing. Training on a lot of different perspectives of the data is better than training on one perspective of all the data. Look at pictures of what you’re doing! Write down the things that you try, because you will forget a few hours later! The competition format forces you to do all of these basic things right, more so than having them lectured at you, or reading them in a paper.

I’m also happy to report that I have paid back the loan to my parents, though the jury is still out on whether I’m any wiser in the face of data. Humility is one of most used tools in my arsenal!

2. Wired recently called attention to the fact that PhDs were leaving academia to become data scientists. At Kaggle, do you see this pattern? What kind of backgrounds do the 85,000 data scientists in the Kaggle community have?

It’s not hard to believe that PhDs are leaving to join data science positions. Academia is brutally competitive, and the difficulty is compounded by a dearth in grant funding. In data science, the disparity is flipped; companies are clamoring to hire data-literate people.

The information we have is mostly self-reported, so it’s difficult to make any real quantitative statement about a mass migration from academia to industry. What we can say about our userbase is that many thousands of them have PhDs, and that they are coming from all kinds of backgrounds. Physics, engineering, bioinformatics, actuarial science, you name it.

3. Do you see Kaggle as democratizing data science? We interviewed one of the winners of a Kaggle contest, and he was a student from Togliatti, Russia and was taking classes in data science on Coursera. I was blown away by him.

This is the fun part of sitting in the middle of a data labor market. I get to work with people who make me—presumably a not-unintelligent person…I hope…on my good days—realize how much I didn’t even know I don’t know.

Your question also brings up a controversial point. People have an understandable misconception about Kaggle’s democracy. Our critics are fond of saying that we are solving billion-dollar problems five times over and paying people a wooden nickel to do it. I think this reaction is partly a fear that smart people from anywhere, regardless of credentials, are given equal access to data problems, but I also think it’s a criticism that mistakes what our deliverable really represents. The fear over democratizing data science more parallels the old open-source software fallacy (how will we make money writing code if others give it out for free?!) than it does an outsourcing analogy.

Let’s take the problem of solving flight delay prediction. People immediately think “well that’s worth billions of dollars and if MyConsultingCorp were to solve that problem it would be for tens of millions in fees.” This stance is out of touch with what is really happening in these competitions. To wit:

• People are solving singular problems for one company in one sector
• The devil is in the implementation details
• There are no constraints on absolute model performance, just relative rankings
• The crowd always (p < 0.05) outperforms on accuracy, so when a business wants to optimize on accuracy, crowdsourcing gets chosen because it works well

Our asset is our community, not an outsourcing value proposition. To this end, we believe our efforts will actually increase the scope and amount of work available for people in analytics. Is this democracy? I think so. We sell in to companies, convince them of the merits of machine learning, isolate their problems, and open them up to the world.

The alternative is that DataDinosaur Corp. sells them on their proprietary Hadoop platform, cornering them into a big data pipe dream and leeching money via support contracts. The phrase “actionable intelligence” has never meant less than it does right now. It’s a scary, fake world out there in big-data land!

4. What data do you wish would be made available for a Kaggle contest?

I have a cancer research background. Much of the data from medical experiments is extremely shrouded in privacy fears. A lot of this fear is justified–it’s certainly nonnegotiable that we preserve patient privacy—but I believe the majority reason is that saying no means less work & bureaucracy and saying yes means new approvals & lawsuit risk. There is a tragic amount of health and pharmaceutical data that goes to waste because it lives (dies?) in institutional silos.

Access to data for health researchers is not a new problem, but I think the tragedy is especially exacerbated given what I’ve seen Kagglers do with data for other industries.

5. What is your favorite problem that a Kaggle contest has solved?

We ran a competition with Marinexplore and Cornell University to identify the sound of the endangered North American Right whale in audio recordings. Researchers have a network of buoys positioned around the ocean, constantly recording with underwater microphones. When a whale is detected in the vicinity of the buoy, they warn shipping traffic to steer clear of the area.

Not only did the participants come up with an algorithm that was very close to perfect, but how about the opportunity to work on a data science problem with such a clear and unquestionably positive goal? We spend a lot of time as data scientists thinking about optimizing ads, predicting ratings, marketing widgets, etc. These are economically important and still quite interesting, but they lack the feel-good factor of sitting at your keyboard thousands of miles away and knowing that your work might trickle down to save the life of an endangered whale.

Gnip is hiring for a data scientist position if you’re looking to do your own cutting-edge work. 

Thanks to Will for taking the time to interview with Gnip! If you have suggestions for another data story interview, please leave it in the comments.

Previous Data Stories:

• Harper Reed, former CTO of Obama for America
• Lada Adamic of Michigan on information networks
• Hilary Mason, Chief Data Scientist of bitly
• Blake Shaw, Data Scientist of Foursquare on product development with data science
• Simon Rogers, data journalist at The Guardian 
• Mel Hogan of CU Boulder on digital archiving 
• Liv Buli of Next Big Sound, the world’s first music data journalist
• Annicka Campbell of SapientNitro on the Digital Love Project 
• Gabriel Banos of ZauberLabs on predicting the election with social data

Photo courtesy of Liz Philips

Big Boulder was 16 sessions of social data goodness with more than 200 attendees coming together to learn, collaborate, network and maybe get in some hiking. Creating a space where the leaders of the industry can get together in an intimate setting is what we set out to create, and it’s rewarding to hear that others agreed. My favorite piece of feedback was hearing from Adam Laiacano, the data scientist from Tumblr, was that it was “the most Baller conference I’ve ever attended.”

A highlight for me was to see the introduction of the Big Boulder Initiative, whose mission is — “To establish the foundation for the long-term success of the social data industry.” I love that it allows everyone to gather more than once a year to collaborate on what is still a nascent industry. If you’re interested, you can check it at BigBoulderInitiative.com.

It was also fun to see how social data is expanding outside the United States. Every single publisher interviewed on stage was focused on international growth, especially in Brazil, India, Europe, China and Japan. One of my favorite takeaways was from the session on Social Data in China about how people constantly look for the Twitter or Facebook of China but you can’t make that kind of comparison.

If you missed a session or couldn’t make it to the conference, don’t worry! We have blog recaps of each and every session (see below) summing up the highlights. We also have pictures of the conference on our Facebook page, a Storify recap, and you can always catch up with the hashtag #BigBoulder.

During Big Boulder, we held a contest to see who could create the best Vine. We awarded the mini iPad to Carmen Sutter for having the best Vine. We had a hard time choosing so be sure to check out some of the other Vines showing unique Boulder culture, take a spin around Big Boulder, see some of the more creative breakfast options and go for a ride.

You can see the 2013 Big Boulder recap video, which has a great summary of the highlights from the conference.

 

Big Boulder 2013 from Gnip on Vimeo.

Day 1

• Accelerating the Social Data Ecosystem (A talk by Chris Moody from Gnip)
• Building the Location Layer of the Internet (An Interview with Mike Harkey from Foursquare)
• Unleashing the Creative Expression of 100 Million People (An interview with Derek Gottfrid and Danielle Strle of Tumblr)
• Social Data in Brazil (An Interview with Daniel Heise and Gabriel Baños)
• Tweets, Texts and Tickers (A Discussion with the NYSE, NASDAQ, Thomson Reuters and Markit)
• Social Data in China (An Interview with Ken Hu and Mike Smith from Soshio)
• Mining Consumer Opinion in Comments (An Interview with Daniel Ha and Steve Roy from Disqus)
• Social Data in Academic Research (Sherry Emery, Abe Kazemzadeh and Jaime Settle Discuss the Role of Social Data in Their Research)
• Driving Citizen Outcomes (An Interview with Jason Thomas from Thomson Reuters)

Day 2

• Twitter Certified Program and International Expansion (An Interview with Conway Chen and Zach Hofer-Shall from Twitter)
• Measuring Impact on Facebook (An Interview with Daniel Slotwiner from Facebook)
• Social Data in Japan (An Interview with Koki Uchiyama from Hottolink)
• Creating and Sharing Content on WordPress (An Interview with Paul Maiorana from Automattic)
• Social Data and Primetime TV (An Interview with Maya Harris from GetGlue)
• Data Science and Social Data (Data Scientists from Pinterest, Twitter, Tumblr Discuss the Challenges and Opportunities in Social Data)
• Social Analytics and Business Intelligence (Susan Etlinger, Shawn Rogers and John Lovett Discuss What’s Next for Social and BI)

 

Where Does Social Data Live Within an Organization? Where Should It?

Marketing, public relations and communications groups have a firm grip on social data, and they’re holding on tight because other parts of the business enterprise want access to the information. That’s a good (and exciting!) thing. Other departments want this data available to the decision-makers and stakeholders, which could be an indicator of the maturing of the social data environment. Social data becomes really impactful when it resides in other departments. The CFO, for instance, can make data sacred and turn it into metrics that really matter. Social data is starting to actively move into other departments such as customer support, human resources, finance, legal, risk, sales, IT services, operations, market research, etc., because stakeholders are now wanting to understand the implications of that data.

Convergence
Enterprise wants to understand the data, but they don’t know where to start and, even if they did, they don’t know how to communicate the data. Social data firms can help markets understand the social data at hand and the solutions this data can provide (or the problems they can help reveal) in context through a deep technical integration. When we look at data, what are we actually seeing? Data needs to be pervasive through a business. But then we begin to look at unstructured data like images and video — how are businesses supposed to capture and interpret this kind of rich media social data from its consumers? How long should companies hold on to any kind of social data?

We are still counting things: How many Fans/Comments/Likes/RTs? We are still focused on the volume-based metrics and asking the wrong questions about social data. There is a need to think deeply and ask what are outcomes we as businesses are trying to drive on social and work around those. And even that is not enough — you also need to know if it’s effective and energizing your communities.

Where Are The Opportunities for Business Intelligence?
Right now, analytics can determine a brand score and sentiment, but let’s start to tie solutions to better decision-making. For example, when a flight is cancelled, an airline should know that a customer is angrily tweeting about it in real time and should be able to do something about it. Currently customers take to two separate customer experience channels — tweeting about their unhappiness in real time and filing a complaint via the company’s traditional customer service channels later on when the airline can do little to improve that specific experience. There is a big value payoff when companies bring in data to operational workflows that truly drive business. Action goes beyond insight. There could be privacy issues as companies try to stitch together a customer profile based on transactional and social data, but imagine if businesses leveraged and marketed that knowledge?

Drill, Baby, Drill

A holding company for a restaurant chain, through social data platforms, noticed very serious incidences of complaints about the steak quality in a geographical region. By integrating the data of the supply chain with social data from the Twitter stream, the company was able to drill down through the data and pinpoint the problem to a distributor where six shipments of meat had been compromised. If the company had not done this, who knows how long this problem could have gone undetected or if the true issue would have been identified and corrected.

A company that supplies commercial kitchen equipment to restaurant chains was exploring ways to use social data — could a complaint about a meal at Chili’s mean something was amiss with one of their fryers? If you listen to the social signal — a customer complaint or comment — and correlate that to  a location point, can you figure out trends if equipment is having issues, etc.? The company wanted to explore this, and more and more businesses are looking for ways to decipher and decode social data. Current data sets are isolated, and so companies are not always seeing the big picture on their customer’s experiences and possible opportunities to beat out competitors. By incorporating these data streams together, they can understand what happens and how the customer experience is impacted.

Another case: Kraft was using social data to see how consumers liked the taste and flavor of a reformulated salad dressing. But social data was showing that keywords associated with the dressing were “blood” and “thumb.” Turns out, the company had also redone the way the bottled opened, and customers were getting hurt with this new design. Consumers were taking to the social channels to complain, not calling the 1-800 customer service phone number to Kraft. The company, through social data analysis, was able to discover the problem within a few months, and fix the bottle cap design.

Are We There Yet?

Business intelligence has a long way to go in using and understanding social data and its implications. We need to do a better job of understanding and educating people about what exactly these social platforms are used for, how consumers use them, and what value do companies receive by using them, in order to quantify the value to make sense to the C-suite. There is a level of ignorance about the power of these social channels, and some work in better understanding their capabilities, as well as limitations, is needed for the metrics to be more meaningful and insightful.

Industry leaders have explored creating social media standards under the simple objective to better understand social media metrics: What is reach? What is an impression? Define influence. Define engagement. But debate is fierce, as every vendor for data has a different opinion about these definitions, which currently remain too vague. Departments can’t even agree on what a “customer” is defined as, as it means something different to operations than it does to marketing, thanks to differing objectives of each stakeholder. Also, as social media channels evolve, so too does the meaning of a Like or Retweet or Comment, and what that means for businesses. We will need correct ways to talk about the social data and ask scientific questions in ways that are consistent. 

Big Boulder is the world’s first social data conference. Follow along at #BigBoulder, on the blog under Big Boulder, Big Boulder on Storify and on Gnip’s Facebook page.

GetGlue is a social entertainment phenomenon to be reckoned with. TV is intrinsically social, and GetGlue is leading the social TV movement. People used to physically gather around the television to watch a show or sporting event, and “social” TV meant gathering around the proverbial water cooler to talk about last night’s episode or game. Now, audiences are cutting ties with traditional cable television and increasingly turning to streaming video and time-shifted video watching via Hulu and Netflix. But social networks are allowing networks and studios to connect and engage with fans, as well as fans to connect and engage with each other, in an efficient way and on a greater scale.

How Are Consumers Using GetGlue?
Users check in on GetGlue about what they’re watching on TV, movies, and sports, and they can earn rewards along the way. Based on a user’s actions, GetGlued puts together a taste profile, which fuels recommendations and a personal guide in calendar format which shows users what they like to watch, what GetGlue thinks they might like to watch, as well as friend recommendations and trends. GetGlue can also customize recommendations tailored to the user’s preferences and entertainment experiences, i.e. if they prefer HD viewing or “I don’t have HBO so don’t tease me with shows I can’t watch,” etc. Also, its second screen after check-in mashes up multimedia content form other channels like YouTube and Twitter. GetGlue is also refining its commentary platform, where comments of friends will show ahead of total strangers, despite timing of the comments.

GetGlue by the Numbers

• 25 percent of 18-34 year olds comment about what they like and don’t like while watching TV.
• More than 50 percent of users want to connect with a fellow fan — they are looking for their niche entertainment community.
• GetGlue partners with 75 networks and 25 studios to create incentives (stickers, for example) to reward fan loyalty.
• 50 percent of stickers are shared to Facebook and Twitter = a genuine fan endorsement and viral marketing tool across highly visible social channels.
• 20 percent of consumers start watching a show because of a social impression.
• 70 percent of users are U.S.-based. International growth at this time is organic and concentrated in Great Britain, Canada, and Australia.
• GetGlue found that 7 out of 8 people, when they sit down to watch TV, have no idea what they want to watch.
• 50 percent of users share what they are watching to other social channels like Facebook and Twitter

What Differentiates GetGlue From Other Social Networks?
There are numerous online channels to discuss TV, so how does GetGlue stand out? It provides a focused place for fans to engage, which is important. GetGlue believes that fans “deserve a place to focus on what they want to say” about TV, movies and sports. Scripted TV is the bread and butter of GetGlue. When a fan wants to comment on Game of Thrones or The Mindy Project, they may not want to share with all their Facebook or Twitter friends; fans want to share with people who are also involved and care about the show and actively doing same thing, watching and commenting, in the same moment. This rich commentary is one of the fine points of the social data GetGlue is curating from its users.

Fun fact: Fans of the AMC show The Walking Dead crashed the GetGlue system one night; it was the top scripted cable show on GetGlue for 8 weeks.

And the Winner Is …

GetGlue has had a starring role at the GRAMMYs, an age-old event that is embracing this new world of social TV. This year GRAMMY viewers were rewarded for checking in during the live broadcast and the website; there was also a prize giveaway, which included tickets to the invite-only event. The GRAMMYs via GetGlue saw 140,000 total activity but through the social networks Facebook and Twitter reached an audience of 40 million. The amplification effect is where the value is found. Also, data mimicked peak times through the live broadcast, from the opening act of Taylor Swift to the repeated wins for Adele. GetGlue was the second driver of all traffic to the GRAMMY website.

Why Should Networks and Brands Care About Social TV?
Because viewers are talking about your content. And if you aren’t listening, you can’t partake in the conversations. Networks understand that this is important and are focusing more resources to figure out what the value is. Studies show that players in the social TV space have correlated check ins on GetGlue with Nielsen ratings; networks have told GetGlue that check ins are the best predictive element for what the ratings will be. Networks are moving beyond the total check ins, tweets, RTs, comments and likes and are now digging deeper and analyzing the conversations and what value lies within. Increasing amounts of social chatter contribute to ratings and revenue. It’s important because people are not only tuning in, but the fan base is engaged.

Fun fact: The rabid fan base and fan acitivty on GetGlue single handedly kept The CW from cutting the show “Nikita.”
Brands are just starting to look at what’s available in this social data space and are spending money in linear ads in order to monitor what’s being said in a real time basis about them (and their competitors) in order to optimize current campaigns and longer-term brand messaging in order to market to the consumer better.

Stay tuned
There is a wealth of data. Consider the number of eyeballs watching a TV show and evaluate the engagement: check ins, comments, replies, likes, votes … studios, networks and brands can start to get a great picture of the activity around their shows, which opens great opportunities to communicate with fans and engage with them elsewhere. If a fan is using GetGlue or other social networks, there is a higher likelihood that they will adopt their other digital platforms and engage there, too.

Big Boulder is the world’s first social data conference. Follow along at #BigBoulder, on the blog under Big Boulder, Big Boulder on Storify and on Gnip’s Facebook page.

As Gnip’s own data scientist Dr. Skippy was joined on stage by three data scientists representing three prolific social networks, Big Boulder Master of Ceremonies Lindsay Campbell couldn’t help herself gushing to the crowd, “This is by far the sexiest panel this year”. (Which was a reference to the Harvard Business Review naming data science the sexiest profession of the 21st century.)

Physical appearance aside, there could hardly be a truer statement to Big Boulder attendees: a legion of self-proclaimed data nerds.

Scott Hendrickson, better known as Dr. Skippy, Data Scientist at Gnip was joined on stage by Mohammad Shahangian of Pinterest, Kostas Tsioutsiouliklis of Twitter, and Adam Laiacano of Tumblr.

A Look at the Data Science Departments

The conversation began with each guest sharing the size of data science teams and roles at their respective organizations.

The data science team at Twitter is currently comprised of 7-8 people, looking to build to team of 20 in the near future (see open positions here). Data scientists at Twitter fall into two departments: a business intelligence and insights team of data scientists and individual data scientists who are embedded into teams. Data scientists embedded into teams become key stakeholders in improving and evolving the product.

The business intelligence team works collaboratively to explore ideas and create reports, even if it is not always favorable to the company. As Kostas explains, data scientists are trusted at Twitter. It’s ok to report the truth.

At Pinterest, there are 8 full-time data scientists on the team. The primary goal for data scientists is to understand what users are doing, to put pinners first- a strong company value. Much like Twitter, Pinterest data scientists are integrated into other engineering teams. This blend of engineers and data scientists on the same team enables nimble product iterations. Since adding data scientists to the mix at Pinterest teams are now requesting deeper and deeper metrics to measure success and plan product.

Tumblr’s team of data scientists is also eight strong in two roles, first a search and discovery team six strong and second, a two person, very self reflective business intelligence team. The search and discovery team is tasked to maintain the quality of the data and build products that can make the data usable, and ensure the end product is something users enjoy. The business intelligence team of two people is highly self-reflective investigating actions users take to determine which actions are indicatory of long term success.The outcome of which is most frequently is reporting.

Data Science Impact on Product

At Tumblr, there is a significant amount of testing around registration and onboarding, what users see when they land at Tumblr.com. However, Adam is quck to add that Tumblr has a unique view on their research, stating, “You don’t have to do as much research on your product when you use it yourself”.

Data scientists at Twitter report metrics all the way to the top. The CEO and the executives are asking questions about the data around launch of a new product and value the input of data scientists.

By sharing data with product teams, Pinterest engineers are being driven by the data. Mohammad shares, “After exposing metrics to people, the first instinct is to want to make the metrics better. This brings a culture of people who come to the data science team and seek their input. They take the ideas of product and run some queries to see if the data validates it. We’ve made it very easy for product teams to set up experiments, we don’t even call them experiments anymore.” Expounding on this fact, he shares an anecdote from a recent rewrite of the entire website. When launched, scientists noticed a dip in follows. Investigation from the team lead to understanding that the enhanced speed of the rewritten website had eliminated a small lag which followed a users like. A lag of time in which users had been following pinners on the site. By correcting the lag, follows went back up.

Who You Callin’ Sexy?

As Dr. Skippy joked about the popularity, ahem sexiness, of the data science title, conversation turned to the lack of an industry standard definition for the role, noting there is often confusion and a lack of differentiation from business analysts and business intelligence roles.

Kostas began noting that data science is not about analyzing but about prediction. Twiter data scientists are also engineers. Backgrounds of Twitter data scientists include statistics, data mining, machine learning, and engineering.

Further delineating from data analysts, Mohammad points out that role isn’t pulling their own data. Continuing on he added, “If you can’t pull your own data, how can you figure out what you want? A data scientist is skeptical. If results seem too good to be true, they will investigate. Question the data. Analysts will take the data as the data.”

Adam relates a good scientist as individual who can get data in any format and clean it up, can take weird, fuzzy forms and see the layout of the information is available. To connect the puzzle and build the data set that is useful.

The Future For Analysis of Social Data

Much of data science to date has been ad hoc, but the panelists agree that as you look closely at what data scientists do, it’s templates and patterns. Over time this work will become progressively more standardized. With new, faster tools it will move away from ad hoc processes. Teams will build models and tools to solve recurring problems.

Adam of Twitter added optimistically that the future is the work data scientists will do as they collect data across platforms and across multiple streams. It’s up to those developing third-party tools and resources to innovate using all the data.

Lastly, Mohammad chimed in that machine learning and prediction modeling is the sexy amongst the sexy. Adding, “That’s what we’re all waiting for”.

Big Boulder is the world’s first social data conference. Follow along at #BigBoulder, on the blog under Big Boulder, Big Boulder on Storify and on Gnip’s Facebook pag

There are a lot of names for the WordPress/Automattic group, so it’s important to distinguish who is who. WordPress, who just celebrated their 10 year anniversary in May is an open source platform, free to use and free to download. Automattic (named for its founder, Matt Mullenweg) is the organization providing services around WordPress and handling its infrastructure. Lastly, Jetpack is the plugin used to add features to a WordPress site, powered by the cloud infrastructure.

Paul Maiorana, Automattic’s VP of Platform Services dove into the WordPress.com VIP, a solution for large media organizations and enterprises. You can run WordPress anywhere in the world, and Automattic is the largest user of and contributor to the open source platform. They’ve built a significant amount of knowledge around scaling the product and now provide this knowledge to enterprises. Huge organizations like Turner Broadcasting, federal agencies and a wide spectrum of other groups are customers.

“Biggest Home of Users on the Web”

WordPress has a philosophy when building their open source software – the idea of the independent web. Paul says they like to think of WordPress as a digital hub and your home on the web. At the end of the day, they try to give you (the user) the tools to create and export content and put it where you want. The user will always own WordPress as much as the company does. “A place on the web you can call your own, where you own the data, you own the experience,” says Paul, is part of the DNA at WordPress. More than 18% of the top 10 million website are WordPress, and 70 million WordPress websites are hosted between WordPress.com and other sources.

Blogging and Enterprise

While WordPress’ roots have always been in blogging, they see themselves as more of a content management system. This perception has persisted because of reputation. But over the last couple years, they’ve expanded on this to bring tools to customize user sites and take advantage of it to be more than just a blog. More and more organizations are using WordPress as a CMS these days instead of just a blog. On an enterprise level, major websites like CBS are using WordPress for CMS. It’s a testament to how the tool has evolved over the recent years.

Product Roadmap

Paul says product decisions have an interesting in relationship with the open source portion of WordPress. At the end of the day, WordPress has little control over what happens on that side. Unlike other CMS platforms, WordPress updates three times a year. It is updated without breaks to make it seamless for people to use the best WordPress there is. Within Automattic, they’ve built a lot of enterprise solutions and open source solutions to help make WordPress better for everyone.

Mobile is also a huge focus of what they’re currently focusing on, and how they will continue to shape their roadmap. For now, it’s a big initiative in two ways: from a front-end user experience and from a dashboard admin experience. The past three releases have focused a default theme that is responsive, and they will continue to do so. For the admin experience, mobile is perfect for “of the moment” publishing. With apps for IOS, Android, Blackberry, and Windows, more content publishers will have the ability to publish on the go efficiently. They’ve seen real world use cases too, with reporters catching stories first because they were able to use the mobile publishing.

WordPress and Social

Blogging is inherently social and it’s not an accident comments are an important part of the WordPress software. The conversation is an important part of publishing on the web.  Paul said WordPress spends a lot of time thinking about additional social features they can add (likes, re-blogging, following, subscribing to updates). Looking forward, they’re hoping to expose the idea of consuming content within WordPress. They’re experimenting with reader interface and giving users ability to subscribe to content they like from topics or specific blogs and then see it all in one place and interact with it socially.

Big Boulder is the world’s first social data conference. Follow along at #BigBoulder, on the blog under Big Boulder, Big Boulder on Storify and on Gnip’s Facebook page.

Japan Is Social Data Rich

Gnip researchers estimate that 10 to 15 % of the Twitter firehouse is in Japanese characters. In addition, Japan is in the top 5 countries for Foursquare content (excluding the US). More than 32 million Japanese users have written blog articles.

Platforms

Japanese have been engaging on social channels since early 2000. Early platforms included bulletin boards and question and answer sites similar to Ask Yahoo!. Blogs continue to be a popular Japanese social medium. Twitter & Facebook continue to gain traction with Japanese users, but more exciting is the Japanese social media platform Line. Available in over 20 countries, Line originated as a text messaging application and has grown to include features including voice calls, group messaging and the use of stamp based character communications. Many Japanese users choose to leave text communication behind and share their thoughts, emotions and feelings through images the network calls stamps. In January, Line crossed 100 million users. At only 19 months old, this growth trajectory exceeds that of Facebook. A favored platform beyond Japan, Line is also popular in Taiwan, Thailand, East Asia, the Middle East and Mexico.

Japanese Social Media Adoption

The 2011 earthquake can be attributed for fueling adoption for two reasons. First, the robustness of social media as a communication tool. Following the earthquake phone lines were dead, but access to the internet remained available. It became the primary channel to communicate with family and colleagues. In fact, use as a primary communication tool drove adoption by nontraditional, older users. Second is the power of social media to provide localized information. Regional areas utilized social media to publish their specific needs and demands, rather than rely on myopic media focus.

Social Media Informed Elections

The Japanese election cycle is only 2 weeks. This short term precludes politicians from using social media as a push campaign marketing strategy. Instead, social media has emerged as an invaluable political strategy tool.

In 2009, social media analysis predicted 80% of the lower-house elections correctly. Political teams realized they could use the data as a listening tool to understand the political climate and needs of a region, even influencing the selection of candidates.

Listening Tools Make Mass Media a Dialogue

Using social data in tandem with mass media communications provided further campaign insight. If a party holds a press conference in the morning, the feedback from social data can be interpreted by noon and campaign strategy or topic stance can be altered to reflect citizen opinion by the evening television appearance.

Brand Use of Social Media in Japan

Koki was quick to point out that brand use in Japan is very similar to the US. In early 2000 brands began utilizing social media for risk monitoring. This was followed by promotions on blogs, then Twitter and after that Facebook.

Currently, brands use social platforms to publish company information, perform market research through listening to drive product development and for engagement and support of customers, though this is not as prevalent as the US.

Koki diagrams the focus of business into a 5 category pyramid. Listed from top down the categories are mission, strategy, product development, marketing and customer support. Currently many brands are reporting that efficiency of social campaigns as less than expected, something Koki attributes to a failure to engage all parts of the pyramid.

If brands fail to utilize social platforms for engagement and dialogue, they will never influence to the top layer, and hence influence focus. By listening and engaging in dialogue, social media will drive all components of successful business.

Big Boulder is the world’s first social data conference. Follow along at #BigBoulder, on the blog under Big Boulder, Big Boulder on Storify and on Gnip’s Facebook page.

“There are a lot of misconceptions about Facebook and data,” Chris Moody eloquently opened the interview with Daniel Slotwiner, Head of Measurement Solutions Group for Facebook. For Daniel’s team, their job is to build tools and methods of analysis to highlight the value of Facebook’s media business. But as Daniel explained, it’s not a win for a brand to measure a brand campaign by the CTR it gets. Instead, he emphasized the importance of working with an advertiser who is defining objectives and setting the right measurement program alongside. The Measurement Solutions Group not only tries to build the tools the industry can use, but also educate and work with them to get the most out of the ecosystem. The hope is that the ecosystem will be self-sufficient.

Partnerships

Last year Facebook announced their partnership with Datalogix, initially for measurement. However, with Datalogix’s comprehensive roster of US households, Facebook realized the impact of the information they could provide to advertisers. Datalogix and Facebook have been able to append data of frequent shoppers with consumer purchase decisions. This has aided in analyzing the impact of Facebook in driving offline sales. With more than 80 campaigns executed with these tools, Facebook can see which segments are responding to the advertising and make smarter campaign. At the end of the day, the value of this data is just to calculate ROI, but rather the scale allows for in-depth analysis and huge learnings for not only Facebook, but also advertisers.

Scale 

If the unique advantage of Twitter is that everything is public, Facebook’s advantage is knowing who is saying what. The uniqueness of this data is two fold: scale and concept of identity (demographically and geographically).  If advertisers can understand the value of this data, they have a fantastic starting point.

It’s hard to argue Facebook isn’t doing a good job of scaling their users. “Obviously we love new users,” David said, and it’s still a huge focus for Facebook, as it expands internationally. And they’re prioritizing serving everyone in the world, especially through segmenting. When it comes to the level of use, Facebook has found light users are more receptive to advertising in comparison to heavy users. As  advertisers, understanding this user segmentation can help shape campaigns and execution on the social network. Facebook is intent building these insight back into the advertising systems to help advertisers make better decisions.

Value in Multi-Point Attribution

The world of influencing consumers is only getting more complex. In one sense it’s because there’s so many touch points. Facebook is focused on making sure the measurement systems are keeping pace with the world, but this is virtually impossible. There are a lot of approaches, but Facebook is pretty focused on multi-touch attribution systems to measure. One way they can look into this is through mobile.

Because almost all users access Facebook using mobile, they get to observe a lot and measure they information around mobile usage. This is information Facebook eventually wants to share with the industry. The platform allows for see the different paths to purchases because Facebook has so much visibility into the touch points. Facebook is in a excellent position to observe how many devices people have and how content is distributed across them.

At the end of the day, there’s a lot of data that can be utilized from Facebook. However, Daniel urges the proper use cases of the data. Research, for example is a huge opportunity given the quality of the data. Daniel cautions against the use of the data for its prediction. While a brand may use the discussion online to respond to an emergency or to participate in the conversation, it’s not clear if they should use it as an objective to drive more sales online.

Big Boulder is the world’s first social data conference. Follow along at #BigBoulder, on the blog under Big Boulder, Big Boulder on Storify and on Gnip’s Facebook page.

As Chris Moody sat down with Conway Chen and Zach Hofer-Shall of Twitter this morning, the conversation began with shared optimism on increased talk about Twitter data. All panelists were quick to praise the recent conversation of Twitter CEO Dick Costello on All Things D, where the Twitter data stream was the star of the conversation.

Conway explained this emerging interest in data with an anecdote around Twitter’s early expectations when opening the data stream- expectations that were little to none. Instead, it is the innovation built using the data that is making Twitter infinitely more valuable.

Twitter Data Is Special

4 things set Twitter data apart:

1. It is real time

2. It is public

3. It is conversational, people aren’t just speaking into the ether the conversation goes both ways

4. It is distributed

Honor Thy User

It is a delicate balance to simultaneously respect users creating the data while also wanting to get data out there and ensure it is monetizable. Zach is quick to mention strict adherence and support of a Twitter core values: Defend and respect the users voice. He continues by stating that if this goes wrong, the whole system falls apart.

Twitter has mindfully created a structure that honors this, a key component of which is data resellers. Data resellers enable Twitter to maintain values and still be able to scale. These partnerships have allowed Twitter to encourage and foster innovation in ways they would not have been able to.

Sustainability and Long-term Growth

Conway- we are absolutely committed to the success of Twitter data and the ecosystem around it. Continuing to look at is the data we are pushing out correct? Is the way we are pushing out helping resellers and developers to innovate and build on it? Twitter data and the strategy around Twitter data is pivotal in how Twitter sees their growth.

Data is a core part of the business that wasn’t always seen as a core part of the business. We are so invested in the success of Twitter data long term that we are committed to seeing it scale. And a key part of that is improving efficiency.

There is an understanding now that Twitter data is important- this speaks volume to the sustainability of the system. People don’t need a sell on the access to the data, they are instead interested in how resellers can make that data useful to them.

Twitter Certified Partner Program

Zach defines the Twitter Certified Partner Program as the answer to skeptics that Twitter doesn’t like their ecosystem. The program was established to help the ecosystem grow, help them succeed and grant providers their seal of approval.

The program ultimately acts as a tool to empower innovation on the Twitter stream. Twitter does not have the capacity to create these tools and resources independently. Less than a year old, the program has been adding 5 to 10 strategic companies each quarter. Factors when selecting certified partners include innovative uses of the data (beyond analytics and engagement) and strategic international partnerships.

Certified partners benefit from instant credibility provided through membership in the program when talking to investors and customers, access to prioritized developer support and promotion from the Twitter sales team. Twitter sales team members are trained and knowledgeable of certified partner products. As the team sells promoted content, they are also able to suggest and recommend partners to fill needs Twitter cannot.

International Growth: Not Just Language Localization

Conway identifies two areas of growth that are current bright spots: Europe and Japan. In identifying new markets, Twitter is looking for existing ecosystems where then can bolster and support what’s already happening. Brazil, Japan, South Korea and India are four regions appealing to Twitter now.

Localization isn’t just localization in terms of language, there is localization of analytics and data types as well.

International tools looking to join the Twitter Certified Partner Program need to match the same high standards of other partners. Twitter works with products in new markets to bring them to their standards.

Advertising

Conway calls for service providers to develop tools to empower advertisers to move to ROI driven decisions. He encourages developers to focus on tools to provide actionable insights to inform ad-spend.

The Future of Twitter Data

In a word: Media. In the last year Twitter has blossomed beyond the 140 to the media hung off those characters. Innovation in the data will include tapping into what is attached to the Tweet. Not just the Tweet itself.

GeoData

Self-defining as a mobile-first company, Conway identifies explaining why geodata remains so low as one of his biggest pain points. The balance to respect user’s privacy first while acknowledging delivering a better consumer experience depends on the inclusion of geodata. Ultimately, Conway categorizes it as a product side problem: to get users to opt-in to share their data.

Big Boulder is the world’s first social data conference. Follow along at #BigBoulder, on the blog under Big Boulder, Big Boulder on Storify and on Gnip’s Facebook page.