Era Software

No results match your query

Write data

Writing data with Logstash

Estimated reading time: 4 minutes
  • guide
  • logstash
  • eracloud
  • self-hosted

This page shows how to use Logstash to write data to EraSearch. In this guide, you'll:

  • Use Logstash to collect data about specific files
  • Configure Logstash to write the data to EraSearch
  • View the data in EraSearch

While the steps below use log data stored in files, you can customize the setup to use any Logstash input plugin, including Elastic Beats, AWS Kinesis, Kafka, and more.

Before you begin
Copy
Copied!

This content is intended for engineers and developers using EraSearch on EraCloud or self-hosted EraSearch:

This guide also assumes you've installed Logstash.

Instructions
Copy
Copied!

Step 1: Configure the Logstash file input
Copy
Copied!

Logstash's file input plugin tails a target file on the host filesystem and emits a log message per line found in that file. Follow these steps to set it up:

  1. Open or create your Logstash configuration (typically called logstash.conf)
  2. Paste in the content below
  3. Replace YOUR_FILE_PATH with the path to any file (or directory) you'd like Logstash to tail
Copy
Copied!
input {
  file {
    path => "YOUR_FILE_PATH"
  }
}

Step 2: Configure the EraSearch output plugin
Copy
Copied!

For EraSearch on EraCloud
Copy
Copied!

To configure Logstash to send data to EraSearch, paste the content below into your Logstash configuration file, replacing:

  • YOUR_SERVICE_URI with your EraCloud service URI
  • YOUR_INDEX_NAME with the target EraSearch index

    index

    An index is a group of similar documents. With EraSearch, you can query documents in one or more indexes to optimize your searches.

     -- EraSearch creates the index for you
  • YOUR_API_KEY with your EraCloud API key
Copy
Copied!
output {
  elasticsearch {
    hosts => "YOUR_SERVICE_URI"
    index => "YOUR_INDEX_NAME"

    custom_headers => {
        "Authorization" => "Bearer YOUR_API_KEY"
    }
  }
}

For self-hosted EraSearch
Copy
Copied!

To configure Logstash to send data to EraSearch, paste the content below into your Logstash configuration file, replacing:

  • YOUR_ERASEARCH_URL with your EraSearch URL

    Example: localhost:9200

  • YOUR_INDEX_NAME with the target EraSearch index

    index

    An index is a group of similar documents. With EraSearch, you can query documents in one or more indexes to optimize your searches.

    -- EraSearch creates the index for you

  • YOUR_USERNAME and YOUR_PASSWORD with your EraSearch credentials

Copy
Copied!
output {
  elasticsearch {
    hosts => "YOUR_ERASEARCH_URL"
    index => "YOUR_INDEX_NAME"

    # Uncomment the following line if you're using SSL
    # ssl => true

    user => "YOUR_USERNAME"
    password => "YOUR_PASSWORD"
  }
}

Note: The configurations above use the Elasticsearch output plugin to let Logstash work with EraSearch. That workflow is possible because the EraSearch REST API supports much of the Elasticsearch API.

Step 3: Start Logstash and confirm your configuration
Copy
Copied!

Start Logstash with the relevant command. When Logstash loads the file input and elasticsearch output, it shows output like this:

Copy
Copied!
[INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.16.2", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.13+8 on 11.0.13+8 +indy +jit [linux-aarch64]"}
[...]
[INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["YOUR_ERASEARCH_URL"]}
[INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[INFO ][filewatch.observingtail  ][main][XXXX] START, creating Discoverer, Watch with file and sincedb collections

With Logstash started, you should start to see log data from the file start to be persisted in EraSearch.

Note: To generate some data for Logstash to send, use this command to write text to the file you configured above:

Copy
Copied!
$ cat "Hello, EraSearch!" >> YOUR_FILE_PATH

Step 4: View your data EraSearch
Copy
Copied!

For EraSearch on EraCloud
Copy
Copied!

Access EraSearch's UI by visiting your EraCloud account and clicking Search. Your data is in the index you specified above. You may need to refresh the UI if the index is new.

For self-hosted EraSearch
Copy
Copied!

Use the EraSearch REST API to query the data in EraSearch. Paste the command below in your terminal, replacing:

  • YOUR_ERASEARCH_URL with your EraSearch URL

    Example: http://localhost:9200

  • YOUR_INDEX_NAME with the EraSearch index you specified above

Copy
Copied!
$ curl 'YOUR_ERASEARCH_URL/YOUR_INDEX_NAME/_search?q=_lid:*'

The response shows information about your data and API request, including:

  • took - The time, in milliseconds, EraSearch took to serve the query request
  • _id - A unique, auto-generated numerical identifier for documents

    document

    A document is a JSON object made up of data. In EraSearch, all documents have a unique identifier (_id) and a timestamp (_ts). Most documents include additional fields. Here's an example of a document:

    Copy
    Copied!
    {"_id":4248176661010579457,"_line":"access","response":200,"_ts":1634060854000}

Next steps
Copy
Copied!

You're all set! You're now using Logstash to send log data to your EraSearch instance. For more information about Logstash, visit these pages:

For other ways to get data into your database, visit the list of write integrations. To learn more about exploring, querying, and visualizing your data in EraSearch, visit these pages: