EraStreams is a no-code data pipeline that lets users integrate, transform, and route observability data to EraSearch as well as third-party monitoring tools.
Modern IT and security teams have an urgent operational problem. With the rapid proliferation of observability data from applications and workloads built on microservices, containers, orchestration tools, and cloud infrastructure, operational complexity and costs are increasing faster than organizations can manage them. Siloed monitoring tools from the pre-cloud era are at a breaking point, unable to keep up with the massive data volumes created by modern cloud environments. The underlying weakness of these tools is their inability to make modern IT and security operations simple, scalable, and cost-effective.
Two years ago, we built our observability platform from scratch to solve today’s scale and cost issues. Our initial focus was on log management, centralizing observability data in EraSearch for fast ingest and query.
Today, we announced EraStreams in limited availability (EraStreams beta) to help teams reduce observability costs without sacrificing data fidelity. EraStreams is a no-code data pipeline that lets users integrate, transform, and route observability data to EraSearch as well as third-party monitoring tools. Together, EraSearch and EraStreams create a foundation that provides speed, scale, and flexibility to help teams turn observability data into actionable insights within seconds.
Figure 1. Era Software observability data management
EraStreams complements EraSearch to optimize cost and performance, and it flexibly integrates into existing DevOps workflows and tools to help teams manage observability costs and improve troubleshooting effectiveness. That means IT and security teams can continue to use monitoring tools they rely on, while controlling the volume of data that gets routed to these tools to optimize data usage and cost efficiency. Teams also have the option to route any logs to EraSearch for low-cost storage and fast, petabyte-scale queries.
Figure 2. EraStreams no-code observability data pipeline
Reduce costs and observability data volume
EraStreams provides a powerful set of features that offers multiple ways to reduce observability costs. We’ve observed that most log data includes extraneous or duplicate information that can be easily cleaned up by dropping or deduplicating fields within each payload. Additionally, you can use real-time filters to reduce the overall volume of data that is currently being sent to existing tools. For example, teams already using Splunk or other expensive log management solutions for IT intelligence or security operations can manage log data costs and performance challenges by routing only a filtered subset of data to Splunk. They can then optionally store a full copy of the unfiltered data in a more cost-effective log management tool such as EraSearch.
Take action on data in flight
Teams can also take action on data in flight to transform, enrich, and deliver optimized data where they need it. For customers handling sensitive data, EraStreams allows them to add useful metadata that labels a field containing PII. By marking these fields, you can allow a user to search for data in an index, but not allow access to the fields that contain PII. You can redact sensitive fields by selecting Drop Fields as part of the pipeline configuration in the UI. You can also mask PII before writing it to data storage.
Additional transformations users can perform in the EraStreams UI include:
You can route all logs to EraSearch or to other storage or tools of choice to keep costs low, and route a subset of transformed data to Splunk and Elasticsearch based on the filtering rules you configure.
Effortlessly manage data at scale
EraStreams was designed with ease-of-use and reliability in mind to help teams see how observability data flows through pipelines and to manage data at scale. Challenges with data flows stem from system failures and data intake variation. When networks, disks, and software fail or slow down, the chain of processes that handle the data must be able to handle backpressure, or data gets dropped. Data intake can be uneven and sometimes results in capacity or memory overload. How do you handle a sudden increase in the rate of ingestion?
With dynamic backpressure management, EraStreams provides better handling of traffic spikes and dealing with failure modes in such a way that you minimize data loss. Its dynamic reconfiguration capability also helps to minimize downtime and to maximize availability by allowing you to make changes without needing to stop a running pipeline and restart it. To maintain data quality at every step of the pipeline process, built-in validation mechanisms take the friction and overhead out of working with data.
Some of the common use cases that EraStreams supports include:
Get early access
We’re offering early access to EraStreams for a limited number of beta customers. If you’re interested in getting your hands on the integration, transformation, and routing capabilities in EraStreams, join the EraStreams beta. EraStreams will change the way you work with data.