articles•February 10th, 2021
Transforming Log Management with EraSearch
EraSearch allows you to finally run your own log management infrastructure without the hardware costs, operational complexity, and performance pitfalls of Elasticsearch.
The Current Log Management Landscape
Log management is not a new problem. Syslog has been around since the 1980s, about two decades before the enterprise log management market was born, which itself has been around almost another two decades. But despite having these 40 years of opportunities for product innovation, there appears to be a lack of respect for the technical complexities associated with managing vast amounts of machine-generated log data.
When Splunk was founded in 2003, the concept of using a purpose-built tool for managing and exploring logs was far from an obvious choice. The product that Splunk ended up building was heavily influenced by early users who needed to make sense out of an emerging, but rapidly growing volume of logs coming from the systems and servers these users were beginning to operate at scale. They didn't know they needed a special-purpose tool until they got their hands on it, but there was no turning back once they had experienced it.
Splunk did a great job defining a new product category, and in the decade or so after its launch we saw a number of competitors emerged with the goal of competing with Splunk—Loggly, Sumo Logic, and Papertrail, just to name a few. And while these companies brought interesting interfaces and convenient SaaS offerings, instead of building their own storage technologies they largely all chose to delegate that infrastructure to an existing open-source project. And that project was Elasticsearch.
What's wrong with Elasticsearch?
The first version of Elasticsearch was released in 2010 and was a rewrite of Shay Banon's earlier project called Compass. The goal was to write a distributed wrapper around Apache Lucene that allowed easy storage and retrieval of JSON objects and was easy to access via HTTP.
It was a very appealing value proposition for users wanting to add a search service to their web applications—a use case that had previously been notoriously difficult to get right. As Elasticsearch grew in popularity, it began to be used with success in many new ways. One of those use cases was solidified by a developer at DreamHost named Jordan Sissel, when he added Elasticsearch support to a log shipping tool he had built called Logstash. By 2012, Logstash had been acquired by Elastic and, along with another acquisition called Kibana, the ELK stack was born. From this point on, log management would grow to become one of the largest use cases for Elasticsearch.
Initially, users were very happy with this solution for log management. It rapidly became the de facto open-source choice for storing logs, which led to its adoption by all of the aforementioned log management vendors. Unfortunately, as Elastic matured as a company and Elasticsearch continued to add dozens of feature to support new use cases, we also moved into a world where average log volumes exploded from gigabytes per day to terabytes per day and what began as simple, single-node deployments of Elasticsearch grew into complex clusters that had skyrocketing hardware costs and burdensome operational demands. In short, it became clear that Elasticsearch had never been designed to be an optimal solution for the high-volume demands of the log management use case.
So what now?
Elasticsearch is still a hugely-popular tool and is chosen for new log management workloads every day. At the same time, users are battling the costs of running Elasticsearch at scale but find themselves confronted with a lack of alternatives. We believe that there exists a massive opportunity to deliver a product with an Elasticsearch-compatible interface that is built from the ground up to be optimized for ingesting, indexing, and storing logs while also leveraging the best properties of a cloud-native architecture. We've built EraSearch to realize that dream.
By supporting the Elasticsearch API, we allow users to leverage an ecosystem of tools that has emerged over the past 11 years. We believe that EraSearch can deliver the greatest value by reimagining Elasticsearch from the inside out and bringing transformational efficiency, performance, and operational simplicity to the log management market.