Era Software

No results match your query

Writing data with Fluentd

Estimated reading time: 3 minutes
  • guide
  • fluentd
  • eracloud
  • self-hosted

This page shows how to use Fluentd to write real-time data to EraSearch. In this guide, you'll do the following:

  • Use Fluentd to generate sample logs
  • Configure Fluentd to write logs to EraSearch
  • View the logs in EraSearch

While the steps below use Fluentd's sample input plugin, you can customize the setup to use any Fluentd input, including windows_eventlog, http, monitor_agent, and more.

Before you begin
Copy
Copied!

This content is intended for engineers and developers using EraSearch on EraCloud or self-hosted EraSearch:

This guide also assumes you've installed Fluentd.

Instructions
Copy
Copied!

Step 1: Configure the sample input
Copy
Copied!

Fluentd's sample input plugin generates sample log events useful for getting started with and testing Fluentd. Follow these steps to set it up:

  1. Open or create your Fluentd configuration (typically called fluentd.conf)
  2. Paste in this content:
    Copy
    Copied!
    <source>
      @type sample
      sample {"hello":"world"}
      tag sample
    </source>
    

Step 2: Configure the EraSearch output
Copy
Copied!

For EraSearch on EraCloud
Copy
Copied!

Paste the content below in your Fluentd configuration file, replacing:

  • YOUR_SERVICE_URI with your EraSearch service URI. Don't include https://.

    Example: db-abcdefghi1234567.abc.eradb.com.

  • YOUR_API_KEY with your EraSearch API key.

  • YOUR_INDEX_NAME with the target EraSearch index

    index

    An index is a group of similar documents. With EraSearch, you can query documents in one or more indexes to optimize your searches.

    -- EraSearch creates the index for you.

Copy
Copied!
<match td.apache.access>
  @type elasticsearch
  host YOUR_SERVICE_URI
  port 443
  scheme https
  custom_headers "Authorization: Bearer YOUR_API_KEY"
  index_name YOUR_INDEX_NAME
</match>

For self-hosted EraSearch
Copy
Copied!

Paste the content below in your Fluentd configuration file, replacing:

  • YOUR_HOSTNAME with your EraSearch hostname. Don't include http:// or https://.

    Example: localhost.

  • YOUR_INDEX_NAME with the target EraSearch index

    index

    An index is a group of similar documents. With EraSearch, you can query documents in one or more indexes to optimize your searches.

    -- EraSearch creates the index for you.

  • YOUR_USERNAME and YOUR_PASSWORD with your EraSearch credentials.

Copy
Copied!
<match td.apache.access>
  @type elasticsearch
  host YOUR_HOSTNAME
  port 9200
  index_name YOUR_INDEX_NAME
  user YOUR_USERNAME
  password YOUR_PASSWORD
</match>

Note: The configurations above use the Elasticsearch output to let Fluentd work with EraSearch. That workflow is possible because the EraSearch REST API supports much of the Elasticsearch API.

Step 3: Start Fluentd and check your configuration
Copy
Copied!

Start Fluentd with the relevant command. For information on starting Fluentd, visit their installation guides.

Next, run this command to make sure Fluentd is running with your new configuration. The output is similar to the example below.

Copy
Copied!
$ tail /var/log/td-agent/td-agent.log
[...]
2022-01-21 18:04:51 +0000 [info]: adding source type="sample"
2021-12-03 14:38:40 -0500 [info]: #0 fluentd worker is now running worker=0

Step 4: View your data in EraSearch
Copy
Copied!

For EraSearch on EraCloud
Copy
Copied!

Access EraSearch's UI by visiting your EraCloud account and clicking Search. Your data is in the index you specified above. You may need to refresh the UI if the index is new.

For self-hosted EraSearch
Copy
Copied!

Use the EraSearch REST API to query the data in EraSearch. Paste the command below in your terminal, replacing:

  • YOUR_ERASEARCH_URL with your EraSearch URL

    Example: http://localhost:9200

  • YOUR_INDEX_NAME with the EraSearch index you specified above

Copy
Copied!
$ curl 'YOUR_ERASEARCH_URL/YOUR_INDEX_NAME/_search?q=_lid:*'

The response shows information about your data and API request, including:

  • took - The time, in milliseconds, EraSearch took to serve the query request
  • _id - A unique, auto-generated numerical identifier for documents

    document

    A document is a JSON object made up of data. In EraSearch, all documents have a unique identifier (_id) and a timestamp (_ts). Most documents include additional fields. Here's an example of a document:

    Copy
    Copied!
    {"_id":4248176661010579457,"_line":"access","response":200,"_ts":1634060854000}
    

Next steps
Copy
Copied!

You're all set! You're now using Fluentd to send real-time log data to your EraSearch instance. For more information about Fluentd, visit these pages:

For other ways to get data into your database, visit the list of write integrations. To learn more about exploring, querying, and visualizing your data in EraSearch, visit these pages: