Every Activity Matters

One of Gnip’s founding principles is that every activity matters. We’ve spent years building redundant, reliable, infrastructure with a feature set that minimizes loss. While most SaaS companies today view their platform in common web infrastructure light (lossy, stateless, nascent connections), we do not. Losslessly delivering public social activities in real-time with sub-second latency is hard to do, but our infrastructure is guided by those principles, because our customers demand it.

Gnip could significantly reduce its costs by relaxing the requirements in this area. Our lives would be so much easier if we could tell a customer that we tolerate inconsistent, lossy, data delivery. “You might not get every activity that you asked for” is an unacceptable statement to make to a customer or prospect who is building an application that is rooted in business-critical, timely activity receipt. Nearly all of our customers have built these kinds of applications. Multi-million dollar marketing campaigns can hinge on a single user-generated public social activity. If that activity does not get to the right place at the right time, our customers business is put at significant risk.

Of course, after more than four years in this space, we learned this the hard way. Issues still arise, and we’re not foolish enough to use the “100% reliability” phrase. However, Gnip is dedicated to staying as close to that figure as humanly & technically possible. If you think you missed data with Gnip, we have processes and systems in place to do root-cause analysis to get to the bottom of any delivery issue that may arise. We will turn a feature iteration on a dime to dig into a customer issue, and dedicate our most senior/experienced staff to ferret out a single “lost” activity. You will never hear us say “sorry, you shouldn’t expect to see everything.”

Each Monday we sit down to start a new iteration, and each Monday the plan is prioritized such that Quality of Service related stories sit at the top of the list. With a finite amount of time and energy, it can be hard to watch sexier feature work get pushed down the stack, but our deliberate focus providing a bullet proof service is hard to debate. As a side note, we’re hiring in order to increase overall feature velocity; join us.

Gnip knows high quality of service is a critical piece in your business, and we treat you, and our software to ensure it. We’re not just another web-app. We are the business-end of social media, and we treat our software/infrastructure investment that way.

Big Software, Big Jobs, Big Impact

Gnip moves billions of real-time public social activities to its customers everyday. Doing that efficiently, accurately, and reliably is an awesome challenge. Gnip also has a lot of product roadmap to build-out. We’re hiring across the board, but I wanted to provide some insight into the software side of things to give any software construction minded readers of this post a sense of the interesting technical challenges you’d get to work on if you were on our development team.

Big Streams: Long-lived, Stateful, Variable Throughput TCP Connections
While our web-app (Rails) looks/feels much like a typical web-app, looks can be deceiving. Our core system (Java) has uncommon TCP (HTTP for the most part) connection challenges that we’re constantly applying operational and business logic creativity to. We often describe the connectivity scenario as akin to a video streaming system (Netflix for example). The load balancing, tuning, connection handling logic (often in code (Java and/or C)), restart, buffering, flushing games we get to play keep one’s brain thoroughly engaged. Imagine getting to write code that runs in a pipeline moving at several hundred Mbps (sustained).

Big CPU: Filtering
We’ve built a real-time equivalent parallel to age-old SQL. Something powerful and efficient, yet simple and intuitive to use. The world has grown up with SQL on their minds, yet it generally doesn’t apply to the world of high-volume real-time nature of public social data streams. We spend a lot of time and energy crafting the language, as well as building the infrastructure underneath to ensure it can operate on a message (from small Tweets, to large blog posts) in ones of milliseconds. Very powerful. Very fun. We’d love your help in evolving this part of the system with us.

Big Blending: Enrichments
As data moves through Gnip’s infrastructure we do a variety of things to it. As a couple examples, we will enrich it with Klout scores and typed language classification scores. We’ll also unwind all those opaque URLs (and allow you to filter against the result) for our customers, so they don’t have to stand up horizontally scaled/parallelized infrastructure to do so on their end. We have a long list of enrichments we’re in the process of adding, and we’d like to do so with as minimal latency impact as technically possible. Speed matters. There’s amazing opportunity in the industry to blend other datasets into “the stream.” Help us do this.

Big Data: Historical
There are three organizations on Earth that have the complete public Twitter corpus on their servers. Gnip, Twitter & the Library of Congress. Gnip has dipped our toes into the “historical” offering with our 30-day reply product. Doing so has allowed us to meet many of the business critical backfill requirements of our customers. That said, there is still a huge opportunity in the historical space. We leverage a variety of parallelizable data access, query, and filter technologies at Gnip, but want more horsepower here. We’d love to have more people on the team with practical experience around map-reduce based data access models. Gnip personifies “Big Data” challenges and solutions. Show us what you’ve got (beyond textbook and academic understanding of the latest trends in this space)!

Big Customers: Impact
We’ve been delivering public social data for four years and serve the biggest, most demanding customers in the space. From 8 of the 9 largest social media monitoring companies to some of the largest hedge funds in the world, Gnip is expected to provide a bulletproof solution. Through our customers, Gnip serves more than 90% of the Fortune 500. We’re not just about “Big Ideas,” we’re about “Big Impact.” Want to work in a place where what you do matters? This is it.

Our Approach
We take an “every message matters” approach to our products. As a result, our development team is tightly coupled (one-and-the-same for the most part) with the production operation of the system. We don’t have a wall between the software being built and that software being run. Gnippers who write code also operate that code throughout the local, review, staging, and production environments. We believe this approach yields a better service for our customers; higher quality, better reliability, better consciousness. There is a strong sense of collective code ownership, as opposed to folks siloing into “their area.” This results in everyone’s creative and talented mind having an opportunity to impact the entire system, and subsequently yields better, more consistent, product.

If you’re interested in Gnip engineering jobs, please email jobs@gnip.com.