Era

Connecting Fluentd to EraSearch

Using Fluentd for your logs? It's easy to utilize EraSearch as your log data store!

image of Connecting Fluentd to EraSearch

What is Fluentd and why would I choose it?

Fluentd is an open source data collector, which lets you unify the data collection and consumption for a better use and understanding of data.

Specifically for this use case, Fluentd allows you to take data from the Windows Event Log and send it to the datastore of your choice. This allows for consolidation of windows event logs to a platform that the rest of your logs are in. This consolidation will allow for correlation across events such systems. The more log consolidation the better the correlation. This is one of the trade-offs many organization make when collecting logs. The cost of consolidating logs through these logging systems is high, not to mention management complexity. These two constraints are why Era.co was started.

Why use EraSearch for your Logs?

We at Era.co believe that consolidation of logs across the enterprise, be it on-prem cloud or public cloud or both, provides the ability to gain more knowledge from these logs. We believe the logging should be easy and reduce your management costs. We provide both a SaaS offering as well as the ability to run EraSearch in your own Kubernetes cluster.

We have developed EraDB to allow for any number of API's to be developed on top of it, and our first API is an "Elasticsearch-like interface that is built from the ground up to be optimized for ingesting, indexing, and storing logs while also leveraging the best properties of a cloud-native architecture. We’ve built EraSearch to realize that dream."

Using the Elasticsearch API, our customers can use the tools they already know and love and keep more of the data at a lower cost.

Common scenarios for log ingest

We are seeing two common approaches to utilizing Fluentd. The first is to add Fluentd to all hosts in a DataCenter or availability zone and forward to another Fluentd host as a consolidator. This consolidation allows you to possibly reduce your security surface area of writing logs especially if you plan on using our SaaS service. To be able to consolidate down to a few collectors with access to Era.co's SaaS platform reduces the number of firewall rules that you will have to keep up with.

The other way we see Fluentd implemented is a directly from client to the storage system. This creates a config on each system to push directly to the storage tier, EraSearch. Both have pros and cons that have been debated and discussed extensively, so we will stay focused on the configuration with EraSearch. No matter which way you choose to implement this, the EraSearch setup is pretty straight forward.

fluentd-setup.png

Fluentd setup for windows and specifically the Windows Event Log

We will be focusing this blog on setting up Fluentd with the Windows Event Log connecting to EraSearch in a demo scenario. In our demo environments we have chosen to utilize Basic Authentication mechanism for ease of setup. For a production implementation, we would recommend a SAML approach, but that will be a separate post for another time.

Installing Fluentd on Windows

Fluentd is distributed through an MSI for Windows systems. Simple install process and by default it installs in c:/opt/td-agent/. The configurations are located c:/opt/td-agent/etc the binaries are in c:/opt/td-agent/bin . For more information, please take a look at the great documentation by Fluentd. In order for Fluentd to read the setup or security events in the Windows log, the user will need to have admin privileges to those channels

Configuring Fluentd for EraSearch

Once you have Fluentd installed you are ready to configure. Edit c:/opt/td-agent/etc/td-agent.conf with your favorite editor. Add code the below to your config file the channels line is what you want to send to EraSearch. The parse_description will make things easier to look at and search, it takes the string_inserts and turns them into key:value pairs. You can find more details in the fluent-plugin-windows-eventlog GitHub repo

<source>
  @type windows_eventlog
  @id windows_eventlog
  channels application,system,security,setup
    parse_description
  tag winevt.raw
  <storage>
    @type local
    persistent true
    path C:\opt\td-agent\winevt.pos
  </storage>
</source>

Once we have the source defined, we have to add a few fields to each record. EraSearch requires an _ts and a _lid. _ts is a timestamp as a number (I'd recommend in milliseconds as that's what most visualization tools are expecting. _lid is a tie breaker, so I use nanoseconds for that. The below required my 11 year old son to help me with the decimal places. Just kidding, but I did show him how his 5th grade math is relevant when he grows up.

<filter winevt.*>
  @type record_transformer
  enable_ruby
 <record>
  #Time.now.to_f produces 1270968744.77658
  #so mutipling by a million gives us nanoseconds
    _lid ${t = Time.now; ((Time.now.to_f * 1000000).to_i)}
  #Time.now.to_f produces 1270968744.77658
  #so mutipling by a thousand gives us milliseconds
    _ts ${t = Time.now; ((Time.now.to_f * 1000).to_i)}
  </record>
</filter>

Now that we have the record fixed up, we can move on to the outputs. I'm using env variables here, but most folks will want to do what they deem secure for their environment.

<match winevt.raw>
  @type elasticsearch
  host #{ENV['OUTPUT_HOST']}"
  port 443
  #You may have to handle User and password differently
  user "#{ENV['OUTPUT_USER']}"
  password "#{ENV['OUTPUT_PASSWORD']}"
  scheme https
  index_name fluentd
  ssl_verify false
</match>

There you have it, we can use the Elasticsearch type in Fluentd to send our logs to Erasearch. This is what's great about using our Elasticsearch-like API.