This page shows how to use Vector to write real-time data to EraSearch. In this guide, you'll:

• Use Vector to generate sample log data
• Configure Vector to collect, transform, and write the logs to EraSearch
• View the logs in EraSearch

While the steps below use log data stored in files, you can customize the setup to use any Vector source, including Fluent, AWS Kinesis Firehose, and Kubernetes logs.

Before you begin

Copy

Copied!

This content is intended for engineers and developers using EraSearch on EraCloud or self-hosted EraSearch:

• If you're using EraSearch on EraCloud, you need your service URI and API key. To get started with EraSearch on EraCloud, set up an account.
• If you're using self-hosted EraSearch, you need your EraSearch URL. To get started with self-hosted EraSearch, contact us at Era Software.

This page also assumes you've installed Vector.

Instructions

Copy

Copied!

Step 1: Configure the "demo_log" source

Copy

Copied!

Vector's demo_logs source generates sample log events, useful for getting started with and testing Vector. Follow these steps to set it up:

1. Open or create your Vector configuration (typically called vector.toml)
2. Paste in this content:

# Generate sample logs
[sources.sample_logs]
type = "demo_logs"
format = "apache_common"

# Parse the log body before sending to any sinks
[transforms.parse_logs]
type = "remap"
inputs = ["sample_logs"]
source = '''
. = parse_apache_log!(string!(.message), "common")
'''

Step 2: Configure the EraSearch output sink

Copy

Copied!

For EraSearch on EraCloud

Copy

Copied!

To configure Vector to send data to EraSearch, add the content below into your Vector configuration file, replacing:

• YOUR_SERVICE_URI with your EraCloud service URI
• YOUR_INDEX_NAME with the target EraSearch index

  index

  An index is a group of similar documents. With EraSearch, you can query documents in one or more indexes to optimize your searches.

  -- EraSearch creates the index for you
• YOUR_API_KEY with your EraCloud API key

[sinks.erasearch]
type="elasticsearch"
inputs=["parse_logs"]
endpoint="YOUR_SERVICE_URI"
healthcheck.enabled = false
request.concurrency = "adaptive"
bulk.index = "YOUR_INDEX_NAME"

request.headers.Authorization = "Bearer YOUR_API_KEY"

For self-hosted EraSearch

Copy

Copied!

To configure Vector to send data to EraSearch, add the content below into your Vector configuration file, replacing:

• YOUR_ERASEARCH_URL with your EraSearch URL

  Example: http://localhost:9200

• YOUR_INDEX_NAME with the target EraSearch index

  index

  An index is a group of similar documents. With EraSearch, you can query documents in one or more indexes to optimize your searches.

  -- EraSearch creates the index for you

• YOUR_USERNAME and YOUR_PASSWORD with your EraSearch credentials

[sinks.erasearch]
type="elasticsearch"
inputs=["parse_logs"]
endpoint="YOUR_ERASEARCH_URL"
healthcheck.enabled = false
request.concurrency = "adaptive"
bulk.index = "YOUR_INDEX_NAME"

auth.strategy = "basic"
auth.user = "YOUR_USERNAME"
auth.password = "YOUR_PASSWORD"

Note: The configurations above use the Elasticsearch sink to let Vector work with EraSearch. That workflow is possible because the EraSearch REST API supports much of the Elasticsearch API.

Step 3: Start Vector

Copy

Copied!

In the same directory, enter this command to start Vector:

$ vector --config ./vector.toml

When successful, your terminal outputs several INFO logs about Vector.

Step 4: View your data in EraSearch's UI

Copy

Copied!

For EraSearch on EraCloud

Copy

Copied!

Access EraSearch's UI by visiting your EraCloud account and clicking Search. Your logs are in the index you specified above.

For self-hosted EraSearch

Copy

Copied!

Use the EraSearch REST API to query the data in EraSearch. Paste the command below in your terminal, replacing:

• YOUR_ERASEARCH_URL with your EraSearch URL

  Example: http://localhost:9200

• YOUR_INDEX_NAME with the EraSearch index you specified above

$ curl 'YOUR_ERASEARCH_URL/YOUR_INDEX_NAME/_search?q=_lid:*'

The response shows information about your data and API request, including:

• took - The time, in milliseconds, EraSearch took to serve the query request
• _id - A unique, auto-generated numerical identifier for documents

  document

  A document is a JSON object made up of data. In EraSearch, all documents have a unique identifier (_id) and a timestamp (_ts). Most documents include additional fields. Here's an example of a document:

  Copy

  Copied!

  {"_id":4248176661010579457,"_line":"access","response":200,"_ts":1634060854000}
  

Next steps

Copy

Copied!

You're all set. Your EraSearch instance is now receiving real-time log data. For more information about Vector, including what logs you can collect and how to configure the Elasticsearch sink, visit these pages:

• Vector documentation
• Vector's Source reference
• Vector's Elasticsearch sink reference

For other ways to get data into your database, visit the list of write integrations. To learn more about exploring, querying, and visualizing your data in EraSearch, visit these pages:

• Alerting with PagerDuty (for EraCloud users only)
• Alerting with Slack (for EraCloud users only)
• Connecting EraSearch to Grafana
• Exploring data in EraSearch's UI (for EraCloud users only)
• List of explore integrations