Era Software

No results match your query

EraSearch overview

Estimated reading time: 6 minutes
  • explanatory
  • eracloud
  • self-hosted

EraSearch is a log-management platform for storing, exploring, and managing large amounts of data. Use it to simplify your log-management setup, lower costs, and get the most out of your data.

This page describes EraSearch and ways to use it. If you want to start using EraSearch now, visit the getting started series.

What EraSearch is
Copy
Copied!

EraSearch is a log-management platform. With all EraSearch instances, you can:

  • Store and manage lots of data with fewer resources
  • Use the REST API to interact with data
  • Explore data with an Elasticsearch-like language
  • Write and act on data with several integrations

Store and manage lots of data with fewer resources
Copy
Copied!

EraSearch stores data in cold storage and keeps a hot cache for faster queries. It separates storage and compute to help you:

  • Use fewer resources
  • Handle more data
  • Lower your costs

Use the REST API to interact with data
Copy
Copied!

Connect and work with EraSearch using the EraSearch REST API. With the API, you can write and query data with HTTP requests using your preferred language or framework.

Here's an example of using the API with cURL to write data to an EraSearch instance on EraCloud:

Copy
Copied!
$ curl -XPOST 'https://db-eraeraera123123123eraera.e12.eradb.com/_bulk' \
  -H 'Authorization: Bearer abcdefghijklmnop12345678910' \
  -d '{"index":{"_index":"my_era_logs"}}
  {"_line": "my first log line"}'

Explore data with an Elasticsearch-like language
Copy
Copied!

EraSearch supports most of Elasticsearch's query string syntax. Use the same syntax to write queries, and explore data based on keywords, ranges, booleans, and wildcards.

Here's an example of a query to EraSearch. It requests all data where _line contains the word verified, trail exists, and elevation is greater than one.

Copy
Copied!
_line:verified AND _exists_:trail AND elevation:>1

Write and act on data with several integrations
Copy
Copied!

EraSearch integrates with several popular tools, helping you fit EraSearch into your existing setup. Some of those integrations include Grafana, Vector, Telegraf, Logstash, Fluentd, and Cloudflare.

To see all EraSearch integrations, visit the lists of write integrations and explore integrations.

Ways to use EraSearch
Copy
Copied!

There are two ways to use EraSearch: EraSearch on EraCloud and self-hosted EraSearch.

EraSearch on EraCloud
Copy
Copied!

This is Era Software’s SaaS product. It offers hosted EraSearch fully managed by Era Software. To start using EraSearch on EraCloud, visit the getting started series.

Self-hosted EraSearch
Copy
Copied!

Self-hosted EraSearch is when you run and manage EraSearch on your own cloud infrastructure. Reach out to us at Era Software to get started with self-hosted EraSearch.

EraCloud vs. self-hosted EraSearch
Copy
Copied!

This table outlines some of the differences between EraCloud and self-hosted EraSearch:

EraSearch on EraCloudSelf-hosted EraSearch
Setup and installationEra Software's sign-up UISelf-install in your environment
Managing and hostingEra Software on Amazon Web Services (AWS)You, on your own cloud provider (AWS, Google Cloud Platform, or Microsoft Azure)
AuthenticationAPI keyBasic auth
AuthorizationComing soonRole-based access control (RBAC)
Data exploration and alertingEraSearch UI or integrationsIntegrations

Note: As of October 2021, EraSearch on EraCloud supports AWS only. Future releases will include Google Cloud Platform and Microsoft Azure.

Note: As of January 2022, only EraCloud comes with the EraSearch UI. Future self-hosted EraSearch releases will support the UI. Until then, self-hosted EraSearch users can use Grafana to visualize and interact with their data.

Common workflows
Copy
Copied!

Here's a common workflow for EraSearch on EraCloud, and some documentation to get you started:

Here's a common workflow for self-hosted EraSearch, and some documentation to get you started:

How EraSearch works
Copy
Copied!

This section describes EraSearch's database architecture. If you're a self-hosted user, this information is helpful for installing and managing EraSearch. If you're an EraCloud user, you don't need this information to work with EraSearch on EraCloud. But, feel free to continue reading if you want to learn more!

EraSearch is made up of internal services. With this service-based architecture, you can customize EraSearch to meet your needs by:

  • Configuring, managing, and updating services without impacting other services
  • Adding resources to specific services without having to scale the whole database

Key services
Copy
Copied!

There are four internal EraSearch services, and every service has a specific role. The sections below list the services and what they do.

API Service
Copy
Copied!

The API Service receives and handles all client requests, including writes and queries. You can run several API Services to scale EraSearch, increasing how many write and query requests you can make to the database.

The API Service delegates incoming queries to other services and then responds with a single combined result. The diagram section below goes into more detail about that workflow.

Cache Service
Copy
Copied!

The Cache Service handles the in-database hot cache, including:

  • Writing data to local storage
  • Compacting data to maximize query performance
  • Servicing query results from local storage

You can run several Cache Services to scale your database.

Coordinator Service
Copy
Copied!

The Coordinator Service generates and works with object IDs (OIDs). OIDs are unique identifiers that EraSearch uses to store and retrieve data.

The Coordinator Service stores OIDs in Redis.

Storage Service
Copy
Copied!

The Storage Service works with object storage (for example, S3 in AWS). The service's main roles are to:

  • Batch data for long-term storage
  • Help you manage object storage costs
  • Optimize object-storage communication with minimal networking overhead

Architecture diagram
Copy
Copied!

This diagram shows how EraSearch's services work together to form the database. It also outlines how writes and queries flow through the system. The sections below go into more detail about the write and query flows.

EraSearch architecture diagram

Write flows
Copy
Copied!

Here's what happens when you send a write request to EraSearch:

  1. Clients send a write request to the API Service. Write requests can have one or more documents

    document

    A document is a JSON object made up of data. In EraSearch, all documents have a unique identifier (_id) and a timestamp (_ts). Most documents include additional fields. Here's an example of a document:

    Copy
    Copied!
    {"_id":4248176661010579457,"_line":"access","response":200,"_ts":1634060854000}
    
    .
  2. The API Service requests OIDs from the Coordinator Service. The Coordinator Service creates one OID for each document.
  3. The API Service sends the OIDs and documents to the Storage Service which batches the data.
  4. The Storage Service sends the batched data to object storage for long-term storage.
  5. The Cache Service receives the OIDs and documents, and it compacts the data for future queries.
  6. The API Service sends a response to the client, acknowledging the write.

Query flows
Copy
Copied!

Here's what happens when you send a query to EraSearch:

  1. Clients send a query using Elasticsearch's query string syntax to the API Service.
  2. The API Service computes the query results by:
    • Sending the query to all Cache Services.
    • Merging the results it gets from the Cache Services into one response.
  3. The API Service returns the query results to the client.

Next steps
Copy
Copied!

To start using EraSearch on EraCloud, visit the getting started series. If you're interested in self-hosted EraSearch, reach out to us at Era Software.

To learn more about getting data into EraSearch, visit the Era Software blog and these documents:

To learn more about exploring data in EraSearch, visit: