EraSearch is a log-management platform for storing, exploring, and managing large amounts of data. Use it to simplify your log-management setup, lower costs, and get the most out of your data.
This page describes EraSearch and ways to use it. If you want to start using EraSearch now, visit the getting started series.
What EraSearch isCopyCopied!
EraSearch is a log-management platform. With all EraSearch instances, you can:
- Store and manage lots of data with fewer resources
- Use the REST API to interact with data
- Explore data with an Elasticsearch-like language
- Write and act on data with several integrations
Store and manage lots of data with fewer resourcesCopyCopied!
EraSearch stores data in cold storage and keeps a hot cache for faster queries. It separates storage and compute to help you:
- Use fewer resources
- Handle more data
- Lower your costs
Use the REST API to interact with dataCopyCopied!
Connect and work with EraSearch using the EraSearch REST API. With the API, you can write and query data with HTTP requests using your preferred language or framework.
Here's an example of using the API with cURL to write data to an EraSearch instance on EraCloud:
$ curl -XPOST 'https://db-eraeraera123123123eraera.e12.eradb.com/_bulk' \
-H 'Authorization: Bearer abcdefghijklmnop12345678910' \
-d '{"index":{"_index":"my_era_logs"}}
{"_line": "my first log line"}'
Explore data with an Elasticsearch-like languageCopyCopied!
EraSearch supports most of Elasticsearch's query string syntax. Use the same syntax to write queries, and explore data based on keywords, ranges, booleans, and wildcards.
Here's an example of a query to EraSearch.
It requests all data where _line contains the word verified, trail exists, and elevation is greater than one.
_line:verified AND _exists_:trail AND elevation:>1
Write and act on data with several integrationsCopyCopied!
EraSearch integrates with several popular tools, helping you fit EraSearch into your existing setup. Some of those integrations include Grafana, Vector, Telegraf, Logstash, Fluentd, and Cloudflare.
To see all EraSearch integrations, visit the lists of write integrations and explore integrations.
Ways to use EraSearchCopyCopied!
There are two ways to use EraSearch: EraSearch on EraCloud and self-hosted EraSearch.
EraSearch on EraCloudCopyCopied!
This is Era Software’s SaaS product. It offers hosted EraSearch fully managed by Era Software. To start using EraSearch on EraCloud, visit the getting started series.
Self-hosted EraSearchCopyCopied!
Self-hosted EraSearch is when you run and manage EraSearch on your own cloud infrastructure. Reach out to us at Era Software to get started with self-hosted EraSearch.
EraCloud vs. self-hosted EraSearchCopyCopied!
This table outlines some of the differences between EraCloud and self-hosted EraSearch:
| EraSearch on EraCloud | Self-hosted EraSearch | |
|---|---|---|
| Setup and installation | Era Software's sign-up UI | Self-install in your environment |
| Managing and hosting | Era Software on Amazon Web Services (AWS) | You, on your own cloud provider (AWS, Google Cloud Platform, or Microsoft Azure) |
| Authentication | API key | Basic auth |
| Authorization | Coming soon | Role-based access control (RBAC) |
| Data exploration and alerting | EraSearch UI or integrations | Integrations |
Note: As of October 2021, EraSearch on EraCloud supports AWS only. Future releases will include Google Cloud Platform and Microsoft Azure.
Note: As of January 2022, only EraCloud comes with the EraSearch UI. Future self-hosted EraSearch releases will support the UI. Until then, self-hosted EraSearch users can use Grafana to visualize and interact with their data.
Common workflowsCopyCopied!
Here's a common workflow for EraSearch on EraCloud, and some documentation to get you started:
- Collecting real-time data with Vector
- Storing data in EraSearch on EraCloud
- Viewing and querying data in the EraSearch UI
- Alerting on data in the EraSearch UI
Here's a common workflow for self-hosted EraSearch, and some documentation to get you started:
- Collecting real-time data with Telegraf
- Storing data in self-hosted EraSearch
- Querying and visualizing data in Grafana
- Managing users and roles with EraSearch RBAC
How EraSearch worksCopyCopied!
This section describes EraSearch's database architecture. If you're a self-hosted user, this information is helpful for installing and managing EraSearch. If you're an EraCloud user, you don't need this information to work with EraSearch on EraCloud. But, feel free to continue reading if you want to learn more!
EraSearch is made up of internal services. With this service-based architecture, you can customize EraSearch to meet your needs by:
- Configuring, managing, and updating services without impacting other services
- Adding resources to specific services without having to scale the whole database
Key servicesCopyCopied!
There are four internal EraSearch services, and every service has a specific role. The sections below list the services and what they do.
API ServiceCopyCopied!
The API Service receives and handles all client requests, including writes and queries. You can run several API Services to scale EraSearch, increasing how many write and query requests you can make to the database.
The API Service delegates incoming queries to other services and then responds with a single combined result. The diagram section below goes into more detail about that workflow.
Cache ServiceCopyCopied!
The Cache Service handles the in-database hot cache, including:
- Writing data to local storage
- Compacting data to maximize query performance
- Servicing query results from local storage
You can run several Cache Services to scale your database.
Coordinator ServiceCopyCopied!
The Coordinator Service generates and works with object IDs (OIDs). OIDs are unique identifiers that EraSearch uses to store and retrieve data.
The Coordinator Service stores OIDs in Redis.
Storage ServiceCopyCopied!
The Storage Service works with object storage (for example, S3 in AWS). The service's main roles are to:
- Batch data for long-term storage
- Help you manage object storage costs
- Optimize object-storage communication with minimal networking overhead
Architecture diagramCopyCopied!
This diagram shows how EraSearch's services work together to form the database. It also outlines how writes and queries flow through the system. The sections below go into more detail about the write and query flows.
Write flowsCopyCopied!
Here's what happens when you send a write request to EraSearch:
- Clients send a write request to the API Service.
Write requests can have one or more documents
document
A document is a JSON object made up of data. In EraSearch, all documents have a unique identifier (
_id) and a timestamp (_ts). Most documents include additional fields. Here's an example of a document:.CopyCopied!{"_id":4248176661010579457,"_line":"access","response":200,"_ts":1634060854000} - The API Service requests OIDs from the Coordinator Service. The Coordinator Service creates one OID for each document.
- The API Service sends the OIDs and documents to the Storage Service which batches the data.
- The Storage Service sends the batched data to object storage for long-term storage.
- The Cache Service receives the OIDs and documents, and it compacts the data for future queries.
- The API Service sends a response to the client, acknowledging the write.
Query flowsCopyCopied!
Here's what happens when you send a query to EraSearch:
- Clients send a query using Elasticsearch's query string syntax to the API Service.
- The API Service computes the query results by:
- Sending the query to all Cache Services.
- Merging the results it gets from the Cache Services into one response.
- The API Service returns the query results to the client.
Next stepsCopyCopied!
To start using EraSearch on EraCloud, visit the getting started series. If you're interested in self-hosted EraSearch, reach out to us at Era Software.
To learn more about getting data into EraSearch, visit the Era Software blog and these documents:
- Writing bulk data
- Writing data with Cloudflare
- Writing data with Fluentd
- Writing data with Fluent Bit
- Writing data with Logstash
- Writing data from Node.js
- Writing data with Telegraf
- Writing data with Vector
To learn more about exploring data in EraSearch, visit: