Connecting Fluentd to EraSearch for Log Management

Using Fluentd for log collection? It's easy to utilize EraSearch as your log management solution!

image of Connecting Fluentd to EraSearch for Log Management

If you’re already using Fluentd for log collection, you can use our Fluentd integration to forward your log data to EraSearch. EraSearch is an observability and analytics platform optimized for log management. Forwarding your logs to EraSearch allows DevOps, SREs, platform, and infrastructure teams to take advantage of EraSearch’s performance, enabling faster troubleshooting and better scaling of services - all without the operational complexity and cost of running your own log management infrastructure.

Why use Fluentd with EraSearch for logs?

Fluentd is an open-source data collector which unifies data collection and consumption.

Specifically for this use case, Fluentd takes data from the Windows Event Log and sends it to the datastore of your choice. Logs are imperfect records of real-world events. The consolidation of logs facilitates analysis across events and systems to help you identify correlations and take action where it’s most needed. Consolidating logging systems is costly, and introduces management complexity, creating one of the many trade-offs organizations face in their log management strategy. Cost and complexity are two of the pain points Era Software is solving. 

At Era.co we believe logging should be easy and should reduce your management burden. We provide both a SaaS offering, EraCloud, and the ability to run EraSearch in your own Kubernetes cluster, so however you’ve chosen to architect your systems, EraSearch can support the processes you’ve established. We have developed EraSearch to be compatible with popular APIs for ease of integration. One of our APIs is an Elasticsearch-like interface optimized for ingesting, indexing, storing, and querying high-volume log data for a variety of use cases. Using the Elasticsearch API, our customers can use the tools they already know and love to access and retain more of their data, but less expensively.

Common scenarios for log ingest

We are seeing two common approaches to implementing Fluentd. The first is by having Fluentd push directly from each host to the storage system. This requires a Fluentd configuration on each host that connects and writes to EraSearch as the underlying datastore. The second method involves adding Fluentd to all hosts in a data center or availability zone, then forwarding data to a centralized Fluentd host that acts as a consolidator before writing to EraSearch. This method allows you to collate logs at a single node, simplify your log ingestion stream and, if you’re using EraSearch via our SaaS service, allows you to secure only one point of egress. 

Both approaches have pros and cons that have been debated and discussed extensively; we will focus on the latter configuration with EraSearch. But no matter which way you choose to connect Fluentd to EraSearch, the setup is straightforward.

Fluentd setup for the Windows Event Log

Ok, let’s set up Fluentd with the Windows Event Log connecting to EraSearch in a demo scenario. If you have questions, drop us a line here: [email protected].

Fluentd is distributed through an MSI for Windows systems. In order for Fluentd to read the setup or security events in the Windows log, you need to have admin privileges to those channels. It’s a simple install process, and by default it installs in C:\opt\td-agent. The configurations are located in C:\opt\td-agent\etc and the binaries are in C:\opt\td-agent\bin. For more information, please take a look at the documentation by Fluentd

Configuring Fluentd for EraSearch

Once you have Fluentd installed, you are ready to configure the connection to EraSearch. Edit C:\opt\td-agent\etc\td-agent.conf with your favorite text editor to add the following lines. The channels line designates which event log channels you want to send to EraSearch. The parse_description flag will make content easier to read and search; it takes the string_inserts and turns them into key-value pairs. You can find more details in the fluent-plugin-windows-eventlog GitHub repo.

<source>

  @type windows_eventlog

  @id windows_eventlog

  channels application,system,security,setup

    parse_description

  tag winevt.raw

  <storage>

    @type local

    persistent true

    path C:\opt\td-agent\winevt.pos

  </storage>

</source>

Now we can move on to configuring the outputs. We’re using env variables here, again for easy setup; this is another place you’ll want to be sure you’re following your security guidelines for your environment.

Configuring EraCloud as a Fluentd output

If you’re running in the EraCloud environment, use the following sample output configuration with an API key retrieved from the Cloud dashboard:

<match winevt.raw>

  @type elasticsearch

  host "#{ENV['ERACLOUD_HOST']}"

  port 443

  scheme https

  # Insert EraCloud bearer authentication token

  custom_headers "Authorization: Bearer #{ENV['ERACLOUD_API_KEY']}"

  index_name fluentd

</match>

Configuring a self-hosted EraSearch environment as a Fluentd output

If you’re running in a self-hosted EraSearch environment, your configuration will look similar to the following:

<match winevt.raw>

  @type elasticsearch

  host "#{ENV['ERASEARCH_HOST']}"

  port 443

  scheme https

  #You may have to handle user and password differently

  user "#{ENV['ERASEARCH_USER']}"

  password "#{ENV['ERASEARCH_PASSWORD']}"

  index_name fluentd

  ssl_verify false

</match>

There you have it. Our Elasticsearch API lets us leverage Fluentd’s built-in Elasticsearch type for a quick and easy way to send your logs to EraSearch. 

EraSearch is the perfect observability and analytics platform for high-volume log use cases.

If you’re not a customer, sign up for a 60-day free trial of EraCloud.

Tags