Era Software

No results match your query

Writing data with Fluent Bit

Estimated reading time: 3 minutes
  • guide
  • fluent bit
  • self-hosted

This page shows how to use Fluent Bit to write data to EraSearch. In this guide, you'll:

  • Use Fluent Bit to generate dummy log events
  • Configure Fluent Bit to write the data to EraSearch
  • Query the data using the EraSearch REST API

While the steps below use dummy log data, you can customize the setup to use any Fluent Bit input plugin, including StatsD, Docker, Windows Event Logs, and more.

Before you begin
Copy
Copied!

This content is intended for engineers and developers using self-hosted EraSearch, and assumes you've installed Fluent Bit. You also need your EraSearch URL and authentication credentials to complete the steps below.

Instructions
Copy
Copied!

Step 1: Configure the Fluent Bit dummy input
Copy
Copied!

Fluent Bit's dummy input plugin generates fake/dummy log events, which can be useful for getting started with and testing Fluent Bit. Follow these steps to set it up:

  1. Open or create your Fluent Bit configuration (typically called fluent-bit.conf)
  2. Paste in the content below
Copy
Copied!
[INPUT]
    Name   dummy
    Tag    dummy.log

Step 2: Configure the EraSearch output plugin
Copy
Copied!

To configure Fluent Bit to send data to EraSearch, paste the content below into your Fluent Bit configuration file, replacing:

  • YOUR_ERASEARCH_HOSTNAME with your EraSearch hostname

    Examples: localhost and erasearch.example.com

  • YOUR_ERASEARCH_PORT with your EraSearch port

    Examples: 9200 and 443

  • YOUR_USERNAME and YOUR_PASSWORD with your EraSearch credentials

  • YOUR_INDEX_NAME with the target EraSearch index

    index

    An index is a group of similar documents. With EraSearch, you can query documents in one or more indexes to optimize your searches.

    -- EraSearch creates the index for you

Copy
Copied!
[OUTPUT]
    Name  es
    Match *
    Host  YOUR_ERASEARCH_HOSTNAME
    Port  YOUR_ERASEARCH_PORT
    # if TLS/SSL is needed
    tls On

    Index YOUR_INDEX_NAME
    Time_Key _ts

    # for HTTP authentication
    HTTP_User YOUR_USERNAME
    HTTP_Password YOUR_PASSWORD

Note: This step uses the Elasticsearch output plugin to let Fluent Bit work with EraSearch. That workflow is possible because the EraSearch REST API supports much of the Elasticsearch API.

Step 3: Start Fluent Bit and confirm your configuration
Copy
Copied!

Start Fluent Bit with the relevant command. When Fluent Bit starts up, it should show output like this:

Copy
Copied!
Fluent Bit v1.8.11
* Copyright (C) 2019-2021 The Fluent Bit Authors
* Copyright (C) 2015-2018 Treasure Data
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[ info] [engine] started (pid=1)
[ info] [storage] version=1.1.5, initializing...
[ info] [storage] in-memory
[ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[ info] [cmetrics] version=0.2.2
[ info] [http_server] listen iface=0.0.0.0 tcp_port=2020
[ info] [sp] stream processor started

With Fluent Bit started, you should start to see log data from the file start to be persisted in EraSearch.

Step 4: Query your data in EraSearch
Copy
Copied!

Use the EraSearch REST API to query the Fluent Bit data in EraSearch. Paste this cURL command in your terminal, replacing:

  • YOUR_ERASEARCH_URL with your EraSearch URL

    Example: http://localhost:9200

  • YOUR_INDEX_NAME with the EraSearch index you specified above

Copy
Copied!
$ curl 'YOUR_ERASEARCH_URL/YOUR_INDEX_NAME/_search?q=_lid:*' | jq

The response shows information about your data and API request, including:

  • took - The time, in milliseconds, EraSearch took to serve the query request
  • _id - A unique, auto-generated numerical identifier for documents

    document

    A document is a JSON object made up of data. In EraSearch, all documents have a unique identifier (_id) and a timestamp (_ts). Most documents include additional fields. Here's an example of a document:

    Copy
    Copied!
    {"_id":4248176661010579457,"_line":"access","response":200,"_ts":1634060854000}
    
  • count - The number of files in the specified directory
Copy
Copied!
{
  "took": 33,
  "timed_out": false,
  "hits": {
    "total": {
      "value": 1,
      "relation": "eq"
    },
    "max_score": null,
    "hits": [
      {
        "_index": "YOUR_INDEX_NAME",
        "_id": "2528005658872578048",
        "_score": 1,
        "_source": {
          "_ts": "2022-01-20T16:28:14.225Z",
          "message": "dummy",
          "_lid": 2528005658872578000
        }
      }
    ]
  }
}

Next steps
Copy
Copied!

You're all set! You're now using Fluent Bit to send log data to your EraSearch instance. For more information about Fluent Bit, visit these pages:

For other ways to get data into your database, visit the list of write integrations. To learn more about exploring, querying, and visualizing your data in EraSearch, visit these pages: