Era Software

No results match your query

Write data

Writing data with Telegraf

Estimated reading time: 4 minutes
  • guide
  • telegraf
  • eracloud
  • self-hosted

This page shows how to use Telegraf to write real-time data to EraSearch. In this guide, you'll do the following:

  • Use Telegraf to collect data about specific files
  • Configure Telegraf to write the data to EraSearch
  • View the data in EraSearch

While the steps below use Telegraf's Filecount input plugin, you can customize the setup to use any Telegraf input plugin.

Before you begin
Copy
Copied!

This content is intended for engineers and developers using EraSearch on EraCloud or self-hosted EraSearch:

This page also assumes you've installed Telegraf version 1.21.4+ and jq, a JSON parser for the command line.

Instructions
Copy
Copied!

Step 1: Configure the Filecount input plugin
Copy
Copied!

Telegraf's Filecount input plugin tracks the size and number of files in directories. Follow these steps to set it up:

  1. Open your Telegraf configuration file
  2. Paste in the content below
  3. Replace YOUR_FILE_PATH with the path to the directory you want to monitor
Copy
Copied!
## Filecount input plugin
[[inputs.filecount]]
  directories = ["YOUR_FILE_PATH"]

Note: If you don't have a Telegraf configuration file, go to your telegraf directory and enter the command below.

The command outputs telegraf.conf including the two plugins you need for this guide: Filecount and Elasticsearch.

Copy
Copied!
$ telegraf -sample-config --input-filter filecount --output-filter elasticsearch > telegraf.conf

Step 2: Configure the EraSearch output plugin
Copy
Copied!

For EraSearch on EraCloud
Copy
Copied!

To configure Telegraf to send data to EraSearch, paste the content below in your Telegraf configuration file, replacing:

  • YOUR_SERVICE_URI with your EraCloud service URI
  • YOUR_API_KEY with your EraCloud API key
  • YOUR_INDEX_NAME with the target EraSearch index

    index

    An index is a group of similar documents. With EraSearch, you can query documents in one or more indexes to optimize your searches.

     -- EraSearch creates the index for you
Copy
Copied!
[[outputs.elasticsearch]]
  urls = [ "YOUR_SERVICE_URI" ]
  timeout = "5s"
  enable_sniffer = false
  enable_gzip = false
  health_check_interval = "0s"
  auth_bearer_token = "YOUR_API_KEY"

  index_name = "YOUR_INDEX_NAME"
  manage_template = false

For self-hosted EraSearch
Copy
Copied!

To configure Telegraf to send data to EraSearch, paste the content below in your Telegraf configuration file, replacing:

  • YOUR_ERASEARCH_URL with your EraSearch URL

    Example: http://localhost:9200

  • YOUR_USERNAME and YOUR_PASSWORD with your EraSearch credentials

  • YOUR_INDEX_NAME with the target EraSearch index

    index

    An index is a group of similar documents. With EraSearch, you can query documents in one or more indexes to optimize your searches.

    -- EraSearch creates the index for you

Copy
Copied!
[[outputs.elasticsearch]]
  urls = [ "YOUR_ERASEARCH_URL" ]
  timeout = "5s"
  enable_sniffer = false
  enable_gzip = false
  health_check_interval = "0s"
  username = YOUR_USERNAME
  password = YOUR_PASSWORD

  index_name = "YOUR_INDEX_NAME"
  manage_template = false

Note: The configurations above use the Elasticsearch output plugin to let Telegraf work with EraSearch. That workflow is possible because the EraSearch REST API supports much of the Elasticsearch API.

Step 3: Start Telegraf and confirm your configuration
Copy
Copied!

Start Telegraf with the relevant command. When Telegraf loads the Filecount and Elasticsearch plugins, it shows output like this:

Copy
Copied!
2021-12-22T19:43:54Z I! Starting Telegraf x.x.x
2021-12-22T19:43:54Z I! Loaded inputs: filecount
2021-12-22T19:43:54Z I! Loaded aggregators:
2021-12-22T19:43:54Z I! Loaded processors:
2021-12-22T19:43:54Z I! Loaded outputs: elasticsearch
2021-12-22T19:43:54Z I! Tags enabled: host=Era-Software.local
2021-12-22T19:43:54Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:"Era-Software.local", Flush Interval:10s
2021-12-22T19:43:54Z I! Elasticsearch version: 7.10.2

Step 4: View your data in EraSearch
Copy
Copied!

For EraSearch on EraCloud
Copy
Copied!

Access EraSearch's UI by visiting your EraCloud account and clicking Search. Your logs are in the index you specified above.

For self-hosted EraSearch
Copy
Copied!

Use the EraSearch REST API to query the Telegraf data in EraSearch. Paste this cURL command in your terminal, replacing:

  • YOUR_ERASEARCH_URL with your EraSearch URL

    Example: http://localhost:9200

  • YOUR_INDEX_NAME with the EraSearch index you specified above

Copy
Copied!
$ curl 'YOUR_ERASEARCH_URL/YOUR_INDEX_NAME/_search?q=_lid:*' | jq

The response shows information about your data and API request, including:

  • took - The time, in milliseconds, EraSearch took to serve the query request
  • _id - A unique, auto-generated numerical identifier for documents

    document

    A document is a JSON object made up of data. In EraSearch, all documents have a unique identifier (_id) and a timestamp (_ts). Most documents include additional fields. Here's an example of a document:

    Copy
    Copied!
    {"_id":4248176661010579457,"_line":"access","response":200,"_ts":1634060854000}
  • count - The number of files in the specified directory
Copy
Copied!
{
  "took": 0,
  "timed_out": false,
  "hits": {
    "total": {
      "value": 1,
      "relation": "eq"
    },
    "max_score": null,
    "hits": [
      {
        "_index": "my_era_files",
        "_id": "8157407908185636864",
        "_score": 1,
        "_source": {
          "@timestamp": "2021-12-22T14:12:20-06:00",
          "filecount": {
            "count": 3905,
            "size_bytes": 1407199699
          },
          "measurement_name": "filecount",
          "tag": {
            "directory": "/File/path",
            "host": "Era-Software.local"
          },
          "_ts": 1640203941294,
          "_lid": 8157407908185637000
        }
      }
    ]
  }
}

Next steps
Copy
Copied!

You're all set! You're now using Telegraf to send real-time log data to EraSearch. For more information about Telegraf, visit these pages:

For other ways to get data into your database, visit the list of write integrations. To learn more about exploring, querying, and visualizing your data in EraSearch, visit these pages: