When open source streaming service Apache Kafka was created at LinkedIn in 2011, it was a different world. Most companies were still on-premises. The concept of cloud computing was just beginning to emerge. WarpStream, an early-stage startup, recognized the value of streaming in a cloud-native context and built a new solution from the ground up based on the Apache Kafka protocol and designed for the cloud era.
Today, the company announced a $20 million investment.
WarpStream CEO Richard Artoul and co-founder CTO Ryan Worl said they realized in their previous jobs that moving data to Kafka was complex and expensive, and they wanted to change that. “If we were building something similar to Apache Kafka today, as we did with WarpStream, we would be taking a completely different approach than we did in 2011, when Kafka was first designed. “We think this is a really good time for us to build something new that can actually meaningfully reduce costs and operational burden for people,” Artur told TechCrunch.
The way to do this is to take advantage of today's cloud environments and separate compute from storage using object storage services like Amazon S3. According to the company, by taking this approach, they were able to reduce interzone networking costs, which account for more than 80% of the total cost of running large-scale Kafka workloads.
“By moving and storing data in cloud object storage, you avoid all the network fees that are a problem when big data systems are moved to the cloud,” he said. “Many of the toughest data durability and replication problems Kafka had to solve on its own by copying and replicating data three times to ensure data was never lost, but I “We can offload those issues to the system. Hardening the object storage layer itself makes the system much easier and cheaper to operate.”
Artoul and Worl worked together at Datadog when they helped develop a storage system called Husky. Today, when Datadog customers search application logs, they are actually using Husky. Datadog was also a big user of Kafka. “Based on our experience building a kind of storage system on top of the object storage we built with Datadog, we felt that a streaming system should work the same way. So last year we left his Datadog and added I started working on it,” he said.
They take a two-pronged approach. One is basically allowing customers to bring their own cloud and install her WarpStream, and the other is offering a fully managed serverless option. The BYOC version will be available starting today. The company has added a calculator to its pricing page to calculate how much it costs to run WarpStream.
The founders brought in some of the people who helped build Husky to build the new system. Currently, the company has nine employees. The good news is that they are hiring and hope to double their number of employees by the end of the year.
The $20 million investment was led by Greylock and Amplify Partners, with angel investment from some of the biggest names in the data industry.