Era Software

Collecting Datadog Logs with EraSearch

How-to send Datadog logs to both Datadog and EraSearch

image of Collecting Datadog Logs with EraSearch

Collecting Datadog Logs with EraSearch

Do you keep all of your logs in Datadog and wish you could keep them longer but for less money? Do you want to keep years worth of data in a platform that can be queried easily in realtime?

Whatever your reason, below are some quick instructions on how to simultaneously send data to both EraSearch and Datadog.

Step 1: Setup up the Datadog agent to collect Logs

Look for the Log Collection Configuration section of the /etc/datadog-agent/datadog.yaml. There are a few config's that need to be turned on:

logs_enabled: true
logs_config:
  logs_dd_url: '<vector-host-name>:<vector-port>'
  ## logs_no_ssl / use_http - set both to true if the vector host name or IP
  ## address does not support SSL. Datadog expects a valid x509 cert
  ## using server names otherwise.
  logs_no_ssl: true
  use_http: true

Step 2: configuring a logging source (Optional)

You may need to configure a logging source, so below is how that can be done with the Datadog agent. We will configure Datadog to collect journald logs.

We need to start in /etc/datadog-agent/conf.d/journald.d. You will need to create a conf.yamland add lines below:

logs:
  - type: journald
    container_mode: true

Restart the Datadog agent to pick up these changes, and you should be all set.

sudo service datadog-agent restart

Step 3: Configure Vector to accept the Datadog logs

First you will need to add datadog_logs type source to your Vector configuration as shown below. The address below should match the Datadog logs_dd_urlfrom Step 1.

# dd-source
[sources.dd-logs]
type = "datadog_logs" # required
address = "0.0.0.0:2000"
store_api_key = true # This allows vector to pass through the API key
# tls.enabled = true
# tls.crt_file = "/var/lib/vector/cert/cert.pem"
# tls.key_file = "/var/lib/vector/cert/key.pem"

Step 4: Configure Vector to send logs to Datadog

Step 4 and Step 5 are where we duplicate the log and send it to both Datadog and EraSearch. This configuration is where we send the log line to Datadog

[sinks.dd-sink]
type = "datadog_logs"
inputs = [ "dd-logs" ]
default_api_key = "<Datadog API Key>"
compression = "gzip"
encoding.codec = "json"
encoding.timestamp_format = "rfc3339"
healthcheck.enabled = true

Step 5: Configure Vector to send logs to EraSearch

Below is the EraSearch configuration:

## Parse Syslog logs
## See the Vector Remap Language reference for more info: https://vrl.dev
## This adds _lid and _ts to each log line
[transforms.parse_logs]
type = "remap"
inputs = ["dd-logs"]
source = '''
. = parse_json!(.message)
._lid = to_unix_timestamp(now(),unit: "nanoseconds")
._ts =to_unix_timestamp(now(),unit: "milliseconds")
'''

## Use the elasticsearch type in vector to send data to EraSearch
[sinks.elastic_1]
type = "elasticsearch"
inputs = ["parse_logs*"]

endpoint = "https://<EraSearch Host>:443"
auth.user = "<EraUser>"
auth.password = "<EraPassword>"
auth.strategy = "basic"
healthcheck.enabled = false
## Index name the insert the logs into
index = "dd-logs"

Once we have a config file ready, we can start Vector on the same host where we configured the Datadog agent.

Implementation in larger environment

While this example is for a small proof-of-concept, we could do this at a much larger scale either in a Kubernetes cluster, or with a load balancer and a cluster of Vector nodes as shown below.

/images/blog/dd-arch-large.png

This will give you high availability and allow you to split streams to both EraSearch and Datadog at the same time.