Era Software

Writing data with Fluentd

Estimated reading time: 3 minutes

This page shows how to use Fluentd to write real-time data to EraSearch. In this guide, you'll do the following:

  • Use Fluentd to read logs from a file
  • Configure Fluentd to write logs to EraSearch
  • View logs in the EraSearch UI

While the steps below use Fluentd's tail input plugin, you can customize the setup to use any Fluentd input, including windows_eventlog, http, and monitor_agent.

Before you begin

This guide assumes you have the following:

For EraCloud users, you need your EraSearch Service URI and API key to complete the steps below. If you don't have an EraCloud account yet, visit the Getting started series.

For self-hosted EraSearch users, you need the hostname of your EraSearch database instance.

Instructions

Step 1: Configure the tail input

Use tail to read logs from the end of apache.log and tag them with td.apache.access. To set it up, open your Fluentd configuration file. If you installed Fluentd with td-agent, the file is at /etc/td-agent/td-agent.conf. For other file locations, see Fluentd's config file documentation.

Next, paste in the content below, replacing YOUR_FILE_PATH with the path to your apache.log file.

## Tail input
<source>
  @type tail
  path YOUR_FILE_PATH/apache.log
  pos_file /var/log/td-agent/httpd-access.log.pos
  tag td.apache.access
  <parse>
    @type apache2
  </parse>
</source>

Step 2: Configure the EraSearch output

For EraCloud users, paste the content below in your Fluentd configuration file, replacing:

  • YOUR_SERVICE_URI with your EraSearch Service URI. Don't include https://, for example, db-abcdefghi1234567.abc.eradb.com.
  • YOUR_API_KEY with your EraSearch API key.
  • YOUR_INDEX_NAME with the target EraSearch index -- EraSearch creates the index for you.
<match td.apache.access>
  @type elasticsearch
  host YOUR_SERVICE_URI
  port 443
  scheme https
  custom_headers "Authorization: Bearer YOUR_API_KEY"
  index_name YOUR_INDEX_NAME
</match>

For self-hosted EraSearch users, paste the content below in your Fluentd configuration file, replacing:

  • YOUR_HOSTNAME with your EraSearch hostname. Don't include https://, for example, localhost.
  • YOUR_INDEX_NAME with the target EraSearch index -- EraSearch creates the index for you.
  • YOUR_USERNAME and YOUR_PASSWORD with your EraSearch credentials.
<match td.apache.access>
  @type elasticsearch
  host YOUR_HOSTNAME
  port 9200
  index_name YOUR_INDEX_NAME
  user YOUR_USERNAME
  password YOUR_PASSWORD
</match>

Note: This step uses the Elasticsearch output to let Fluentd work with EraSearch. That workflow is possible because the EraSearch REST API supports much of the Elasticsearch API.

Step 3: Start Fluentd and check your configuration

Start Fluentd with the relevant command. For information on starting Fluentd, visit their installation guides.

Next, run this command to make sure Fluentd is running with your new configuration. The output is similar to the example below.

$ tail /var/log/td-agent/td-agent.log
[...]
2021-12-03 14:38:40 -0500 [info]: #0 following tail of YOUR_FILE_PATH/apache.log
2021-12-03 14:38:40 -0500 [info]: #0 fluentd worker is now running worker=0

Step 4: View data in the EraSearch UI

Go to the EraSearch UI:

  • For EraCloud users, sign in to your EraCloud account and click EraSearch UI.
  • For self-hosted EraSearch users, visit the EraSearch UI in your environment.

Next, click the FILTERS tab, and select the relevant index. You may need to refresh the UI if the index is new.

The UI displays the log data organized by time. Each document has a unique numerical identifier (_lid) and stores data in several field key-value pairs.

Next steps

You're all set! You're now using Fluentd to send real-time log data to your EraSearch instance. To learn more about writing data to EraSearch, visit these guides:

For more information about Fluentd, visit these pages: