Era Software

Writing bulk data from files

To start managing and learning from your logs, you need to get them into your database. With EraSearch, you can write data by working directly with the EraSearch REST API or by integrating with other tools.

This page covers how to write data from a file to your database, using the EraSearch REST API. The content is intended for engineers and developers using EraSearch on EraCloud. If you haven't already, get started with an EraCloud account and learn the basics.

Before you begin

Make sure you have your Service URI and API key for your EraCloud account. You can find that information on your account’s Overview page.

The steps below also assume you've installed jq, a JSON parser for the command line.

Writing data in bulk

Step 1: Prepare your data

EraSearch accepts bulk data in JSON Lines text file format. When formatting your JSON Lines file, separate lines with newlines and specify the index before every document.

For example, the bulkdata.json file below has three documents for the my_era_logs index. Each document includes these field keys: _line, response, and _ts.

$ cat bulkdata.json
{"index":{"_index":"my_era_logs"}}
{"_line":"health check","response":404,"_ts":1633632261000}
{"index":{"_index":"my_era_logs"}}
{"_line":"health check","response":503,"_ts":1633628661000}
{"index":{"_index":"my_era_logs"}}
{"_line":"access","response":200,"_ts":1634060854000}

Note: _line and _ts are recognized field keys in EraSearch.

_line
EraSearch auto-parses _line values and stores them as distinct strings for future queries. For example, EraSearch stores the field value "health check" as ["health","check"]. By default, EraSearch doesn't auto-parse the values of other field keys such as line or _logline.

_ts
EraSearch recognizes _ts as the document's timestamp, where the field value is in epoch time in milliseconds. If you don't include _ts in your file, EraSearch generates the field for you. In those cases, the value is the time EraSearch writes the data to your database.

Step 2: Write your file to EraSearch

To write bulk data to your database, enter the command below in your terminal, replacing YOUR_SERVICE_URI and YOUR_API_KEY with your own information. The example assumes the data is in bulkdata.json and the current directory.

$ curl -XPOST 'YOUR_SERVICE_URI/_bulk' \
  -H 'Authorization: Bearer YOUR_API_KEY' \
  --data-binary "@bulkdata.json"

The response from EraSearch has information about the bulk write. For this example, it has:

  • "errors" : false - The boolean showing all writes succeeded
  • "_id" : "XXX" - The unique numerical identifier EraSearch generated for each document
  • "status" : 201 - The HTTP status code for each write
  • "took" : 490 - The time, in milliseconds, it took EraSearch to complete the request
{
   "errors" : false,
   "items" : [
      {
         "index" : {
            "_id" : "4248176661010579456",
            "_index" : "my_era_logs",
            "_shards" : {
               "failed" : 0,
               "successful" : 1,
               "total" : 1
            },
            "_type" : "_doc",
            "status" : 201
         }
      },
      {
         "index" : {
            "_id" : "4248176661010579457",
            "_index" : "my_era_logs",
            "_shards" : {
               "failed" : 0,
               "successful" : 1,
               "total" : 1
            },
            "_type" : "_doc",
            "status" : 201
         }
      },
      {
         "index" : {
            "_id" : "4248176661010579458",
            "_index" : "my_era_logs",
            "_shards" : {
               "failed" : 0,
               "successful" : 1,
               "total" : 1
            },
            "_type" : "_doc",
            "status" : 201
         }
      }
   ],
   "took" : 490
}

Step 3: Query your data in EraSearch

To view your data in EraSearch, enter the command below in your terminal, replacing YOUR_SERVICE_URI and YOUR_API_KEY with your own information. The request targets only the my_era_logs index, and it uses Elasticsearch's query string syntax.

$ curl 'YOUR_SERVICE_URI/my_era_logs/_search?q=_line:*' \
  -H 'Authorization: Bearer YOUR_API_KEY'|jq

The response shows the three documents in your database:

{
   "hits" : {
      "hits" : [
         {
            "_id" : "4248176661010579458",
            "_index" : "my_era_logs",
            "_score" : 1,
            "_source" : {
               "_lid" : 4248176661010579458,
               "_line" : "access",
               "_ts" : 1634060854000,
               "response" : 200
            }
         },
         {
            "_id" : "4248176661010579456",
            "_index" : "my_era_logs",
            "_score" : 1,
            "_source" : {
               "_lid" : 4248176661010579456,
               "_line" : "health check",
               "_ts" : 1633632261000,
               "response" : 404
            }
         },
         {
            "_id" : "4248176661010579457",
            "_index" : "my_era_logs",
            "_score" : 1,
            "_source" : {
               "_lid" : 4248176661010579457,
               "_line" : "health check",
               "_ts" : 1633628661000,
               "response" : 503
            }
         }
      ],
      "max_score" : null,
      "total" : {
         "relation" : "eq",
         "value" : 3
      }
   },
   "timed_out" : false,
   "took" : 6
}

Next steps

Visit the Era Software blog to see how EraSearch integrates with tools such as Amazon CloudWatch, Fluentd, and Datadog. To learn more about exploring and querying data, visit Exploring data in the EraSearch UI.