Era Software

No results match your query

Writing data from Node.js

Estimated reading time: 4 minutes
  • guide
  • nodejs
  • application logs
  • eracloud
  • self-hosted

This page shows how to write Node.js application logs into EraSearch. In this guide, you'll:

  • Create a sample Node.js application with Winston as a logging framework
  • Write data to EraSearch using the Elasticsearch transport
  • View the logs in EraSearch

Before you begin
Copy
Copied!

This content is intended for engineers and developers using EraSearch on EraCloud or self-hosted EraSearch:

This page also assumes you've installed Node.js and jq, a JSON parser for the command line.

Instructions
Copy
Copied!

Step 1: Create a sample Node.js application
Copy
Copied!

  1. In your terminal, enter this command to initialize a node project in a new sample directory:

    Copy
    Copied!
    $ mkdir sample && cd sample && npm init -y
    

    The command returns this output:

    Copy
    Copied!
    Wrote to .../sample/package.json:
    
    {
      "name": "sample",
      "version": "1.0.0",
      "description": "",
      "main": "index.js",
      "scripts": {
        "test": "echo \"Error: no test specified\" && exit 1"
      },
      "keywords": [],
      "author": "",
      "license": "ISC"
    }
    
  2. Create a small server by adding this code to a new file named app.js:

    Copy
    Copied!
    require("http")
      .createServer((_, res) => {
        res.end("Hello World");
      })
      .listen(3000);
    
  3. Start the server:

    Copy
    Copied!
    $ node app.js
    
  4. To verify your setup, go to a separate terminal window and enter the command below. If the server is up and running, the command returns Hello World.

    Copy
    Copied!
    $ curl localhost:3000
    

To stop the server, press Ctrl + C.

Step 2: Configure Winston
Copy
Copied!

  1. In the sample directory, add Winston to your node project:

    Copy
    Copied!
    $ npm install winston
    
  2. To create a Winston logger and emit your first log message, update app.js to look like the following:

    Copy
    Copied!
    const logger = require("winston").createLogger();
    
    require("http")
      .createServer((_, res) => {
        logger.info("Message received");
        res.end("Hello World");
      })
      .listen(3000);
    
  3. To verify your configuration, restart the server and make the same $ curl localhost:3000 request.

    The server's log output shows that it received a document

    document

    A document is a JSON object made up of data. In EraSearch, all documents have a unique identifier (_id) and a timestamp (_ts). Most documents include additional fields. Here's an example of a document:

    Copy
    Copied!
    {"_id":4248176661010579457,"_line":"access","response":200,"_ts":1634060854000}
    
    but doesn't have a logging backend. You'll set up the backend in step 3.

    Copy
    Copied!
    [winston] Attempt to write logs with no transports {"message":"Message received","level":"info"}
    

Step 3: Configure an EraSearch transport
Copy
Copied!

  1. In your sample directory, add the winston-elasticsearch transport to your node project:

    Copy
    Copied!
    $ npm install winston-elasticsearch
    
  2. Configure the transport and add it to Winston.

    For EraSearch on EraCloud
    Copy
    Copied!

    Add the following to the top of app.js, replacing YOUR_SERVICE_URI and YOUR_API_KEY with your EraCloud account information:

    Copy
    Copied!
    const {
      ElasticsearchTransport,
      ElasticsearchTransformer,
    } = require("winston-elasticsearch");
    
    const esTransport = new ElasticsearchTransport({
      transformer: (logData) => {
        const { ["@timestamp"]: _, ...transformed } = ElasticsearchTransformer(logData);
        return { ...transformed, _ts: Date.now() };
      },
      clientOpts: {
        node: "YOUR_SERVICE_URI",
        auth: {
          bearer: "YOUR_API_KEY",
        },
      },
    });
    

    For self-hosted EraSearch
    Copy
    Copied!

    Add the following to the top of app.js, replacing YOUR_ERASEARCH_URL with your EraSearch URL (for example, http://localhost:9200):

    Copy
    Copied!
    const {
      ElasticsearchTransport,
      ElasticsearchTransformer,
    } = require("winston-elasticsearch");
    
    const esTransport = new ElasticsearchTransport({
      transformer: (logData) => {
        const { ["@timestamp"]: _, ...transformed } = ElasticsearchTransformer(logData);
        return { ...transformed, _ts: Date.now() };
      },
      clientOpts: {
        node: "YOUR_ERASEARCH_URL",
      },
    });
    
  3. Next, in app.js, update the existing logger declaration to tell Winston to use the new transport:

    Copy
    Copied!
    const logger = require("winston").createLogger({
    transports: [esTransport],
    });
    

Note: The configuration above uses the Elasticsearch transport. That workflow is possible because EraSearch supports much of the Elasticsearch API. The example above replaces Elasticsearch's timestamp field (@timestamp) with EraSearch's timestamp field

field

A field is data stored as a key and value pair.

(_ts).

Step 4: View your data in EraSearch
Copy
Copied!

  1. Restart the server:

    Copy
    Copied!
    $ node app.js
    
  2. In a separate terminal window, send your server a request:

    Copy
    Copied!
    $ curl localhost:3000
    
  3. View your data.

    For EraSearch on EraCloud
    Copy
    Copied!

    Access EraSearch's UI by visiting your EraCloud account and clicking Search. Your logs are in the index

    index

    An index is a group of similar documents. With EraSearch, you can query documents in one or more indexes to optimize your searches.

    called logs-YYYY.MM.DD.

    For self-hosted EraSearch
    Copy
    Copied!

    Enter the command below in your terminal, replacing YOUR_ERASEARCH_URL with your information and YYYY.MM.DD with today's date.

    The request targets the logs-YYYY.MM.DD index

    index

    An index is a group of similar documents. With EraSearch, you can query documents in one or more indexes to optimize your searches.

    , and returns one document with the field "message": "Message received".

    Copy
    Copied!
    $ curl 'YOUR_ERASEARCH_URL/logs-YYYY.MM.DD/_search?q=_lid:*' | jq
    

Next steps
Copy
Copied!

You can extend your existing configuration to do the following:

  • Add custom fields to your log messages, for example, logger.info({message: "abc123", customField: "value1"})
  • Configure a static index name by adding index: "myIndex" alongside transformer in your transport options

For other ways to get data into your database, visit the list of write integrations. To learn more about exploring, querying, and visualizing your data in EraSearch, visit these pages: