Era Software

Shipping logs from Kubernetes to EraSearch via Vector

Using a Kubernetes DaemonSet will make it trivial for you to ship logs to EraSearch

Installing Vector with Helm as a DaemonSet

What is Vector?

Vector is a high speed data pipeline. Vector is written in Rust and has many ingestion sources as well as a powerful transformation engine and router. Read more about it on Vector's website.

What is Helm?

Helm is a package manager for Kubernetes. Helm was created in 2015 at Deis, which was later acquired by Microsoft. What is now known as Helm Classic was introduced at the inaugural KubeCon that November. In January 2016, Helm Classic was merged with Google’s Deployment Manager for Kubernetes into the repository that is now the main Helm project. Read more about Helm

What is a DaemonSet?

A DaemonSet ensures that all (or some) Nodes run a copy of a Pod. As Nodes are added to the cluster, Pods are added to them. As Nodes are removed from the cluster, their Pods are garbage collected. Deleting a DaemonSet will clean up the Pods it created.

Why would you do logging with a DaemonSet?

What's nice about using a DaemonSet is you do it once at the cluster level and then every new node has a logging daemon automatically. This isn't the only way to gather logs, you could also do a sidecar — and we will create an installation guide for that as well.

Assumptions on installation

This document assumes that you have an understanding of Kubernetes and Helm charts. It assumes that you have those installed and are just looking to add a Vector DaemonSet to push logs to EraSearch.

Installation

Prepare the Helm release configuration

We will be using the official Vector Helm chart to create a new Helm release, Helm's term for a deployed application instance, running the Vector agent. The first thing we need to do is prepare that release's configuration in a file named values.yaml. There are a lot of ways that you could customize this, but we used the below:

# The Vector Kubernetes integration automatically defines a
# kubernetes_logs source that is made available to you.
# You do not need to define a log source.
transforms:
   parse_logs:
     type: remap
     inputs: ["kubernetes_logs"]
     source: |
       . = flatten(.)
       # The next 2 lines add a _ts and _lid which is required at this time for EraSearch.
       # This configuration assumes a `timestamp` field in the log line being inserted.
       ._lid = to_unix_timestamp(to_timestamp!(.timestamp), unit: "nanoseconds")
       ._ts = to_unix_timestamp(to_timestamp!(.timestamp), unit: "milliseconds")

sinks:
  # Adjust as necessary. By default we use the console sink
  # to print all data. This allows you to see Vector working.
  # /docs/reference/sinks/
  es_kube:
    type: elasticsearch
    inputs: ["parse_logs"]
    endpoint: ${ERASEARCH_HOST}
    index: kube_logs
    auth:
      user: "${ERASEARCH_USERNAME}"
      password: "${ERASEARCH_PASSWORD}"
      strategy: basic
    healthcheck:
      enabled: false
env:
   - name: ERASEARCH_USERNAME
     value: <nameOfUser>
   - name: ERASEARCH_PASSWORD
     valueFrom:
        secretKeyRef:
          name: <nameOfKey>
          key: password
   - name: ERASEARCH_HOST
     value: https://<hostName>:443

Create a Kubernetes Secret

Now that we have the above Helm release configuration, we next need to create a Kubernetes secret to store the ERASEARCH_PASSWORD value. We don't want to store it in a clear text password in the values.yaml, so instead we can use the great Secret resource built into Kubernetes. It's pretty easy to use once you understand how it works.

First, if you want a more in-depth look at Kubernetes secrets, you can find some great documentation at Kubernetes.io.

Second, you will want to setup encryption at rest as well as RBAC, but we are assuming you have already done that. After that you will only need 3 things:

  1. Name of the key, which is really anything but needs to match the config in the Helm chart.
  2. The name of the Namespace, we use "vector" in this example. The new password secret will only be accessible in this namespace.
  3. The literal password that EraSearch gave you for your authentication.
kubectl create secret generic <nameOfKey> -n <namespace> --from-literal password=<ESPWD>

Install the Helm Chart

The namespace can be anything that suits your needs. The values.yaml is the release configuration file we prepared above. Make sure you have replaced the <nameOfUser>, <nameOfKey>, and <hostName> placeholders in that configuration to match your environment.

helm upgrade --install --namespace vector --create-namespace vector timberio/vector-agent --values values.yaml

Operational Commands

Viewing the Vector Logs in Kubernetes

kubectl logs --namespace vector daemonset/vector-agent

Restarting Vector agents

kubectl rollout restart --namespace vector daemonset/vector-agent