Shipping Kubernetes Logs to EraSearch Using Vector

EraSearch provides the fastest and most cost-effective way to tackle Kubernetes logging correctly.

image of Shipping Kubernetes Logs to EraSearch Using Vector

Kubernetes has become the de facto industry standard for automating deployment, scaling, and management of containerized applications. Kubernetes makes it easy for development teams to deploy their applications in a microservices architecture by creating an abstraction layer on top of a group of hosts to automatically manage activities for applications such as resource consumption, load balancing, and storage management. 

These benefits also come with challenges to Kubernetes adoption. Given its highly distributed and dynamic nature, Kubernetes can be difficult to manage - from DNS outages, to a cascading failure of distributed systems. As with other systems, logging and observing your Kubernetes environments are among the first major challenges you’ll need to address as you begin your Kubernetes journey. If you don’t have visibility inside your Kubernetes cluster, how can you troubleshoot applications and cluster issues efficiently?

How EraSearch supports your Kubernetes journey

EraSearch is an observability and analytics platform optimized for ingesting, indexing, storing, and querying high-volume log data. Because of its multi-tier architecture and zero-schema design, EraSearch allows teams to easily collect multiple layers of logs generated by Kubernetes clusters - regardless of the log data type and characteristics. Working with Kubernetes and deploying microservices in cloud environments requires ultra-fast data ingest, queries, and alerting to fully understand the performance of your clusters and workloads, ensuring you can troubleshoot any issues quickly and easily.

EraSearch provides the fastest and most cost-effective way to tackle Kubernetes logging correctly. If you’re battling the costs of running Elasticsearch at scale and need to consolidate tools to handle logging, EraSearch’s support for the Elasticsearch API makes it ideal for organizations looking to lower costs and avoid the operational toil of managing Elasticsearch deployments.

Overview of Vector and Kubernetes logging architecture

Before we start configuring Vector to collect Kubernetes logs and forward them to EraSearch, let’s establish an understanding of Vector, Helm in Kubernetes, and Kubernetes DaemonSets, and how they work together to support logging.

What is Vector?

Vector is a lightweight, ultra-fast tool for collecting, transforming, and routing all of your logs. Vector is already a popular tool for data collection, and we’re likely to run into this technology more often in the future. You can read more about how to use Vector to write real-time data to EraSearch here.

What is Helm?

Helm is a package manager for Kubernetes which streamlines installing and managing Kubernetes applications. Helm uses a packaging format called a chart. A chart is a collection of files that describe a related set of Kubernetes resources. A single chart can be used to deploy everything from a simple Pod to something complex, like a full web app stack with HTTP servers, databases, or more. Read more about Helm here.

What is a DaemonSet?

A DaemonSet ensures that all (or some) Nodes in a cluster run a copy of a Pod. As Nodes are added to the cluster, Pods are added to the Nodes. As Nodes are removed from the cluster, those Pods are garbage collected. Deleting a DaemonSet will clean up the Pods it created. One of the typical uses of a Kubernetes DaemonSet is to ensure that every node running in your cluster has a copy of a logging agent Pod.

Using a DaemonSet for cluster-level logging 

There are several ways to gather Kubernetes logs, but in this tutorial, we’ll focus on using a Vector agent to collect all logs. Vector runs as a DaemonSet in Kubernetes, which means that you install it at the cluster level, and the DaemonSet controller will automatically ensure every Node runs a Vector agent. 

Get started collecting Kubernetes logs using Vector

Now, we’ll show you how to add a Vector agent to a cluster to push logs to EraSearch. We assume that you have Kubernetes and Helm installed in your environment as they are prerequisites for this tutorial. 

Add the Vector repo

If you haven’t already, start by adding the Vector Helm chart repo:

helm repo add vector http://helm.vector.dev
helm repo update

Prepare the Vector agent release configuration

We will be combining the official Vector agent chart and some configuration to create a new Vector agent release. After adding the Vector repo, prepare the release configuration with the values.yaml file. There are many ways to customize this configuration, and here’s an example you can use:

sinks:
  es_kube:
    type: elasticsearch
    inputs: ["kubernetes_logs"]
    endpoint: ${ERASEARCH_HOST}
    index: kube_logs
    auth:
      user: "${ERASEARCH_USERNAME}"
      password: "${ERASEARCH_PASSWORD}"
      strategy: basic
    healthcheck:
      enabled: false
    # If you’re using EraCloud, use API key authentication instead
    # request:
    #   headers:
    #     Authorization: "Bearer ${ERACLOUD_API_TOKEN}”
env:
   - name: ERASEARCH_USERNAME
     value: <nameOfUser>
   - name: ERASEARCH_PASSWORD
     valueFrom:
        secretKeyRef:
          name: <nameOfSecret>
          key: password
   - name: ERASEARCH_HOST
     value: https://<hostName>:443
   # If in EraCloud, use API key authentication instead
   # - name: ERACLOUD_API_TOKEN
   #   valueFrom:
   #      secretKeyRef:
   #        name: <nameOfSecret>
   #        key: password

Before moving on, make sure that you have replaced the <nameOfUser>, and <hostName> placeholders in the configuration to match your self-hosted EraSearch or EraCloud environment. Note that <nameOfSecret> must match the value you will pass in the kubectl command in the next section.

Create a Kubernetes Secret

Now that we’ve prepared the Vector agent release configuration, we can move on to creating a Kubernetes Secret to store the ERASEARCH_PASSWORD (or ERACLOUD_API_TOKEN) value. Instead of storing the password in clear text in the values.yaml, use a Kubernetes Secret to contain your password. You may also want to set up encryption at rest as well as RBAC if you haven’t already done so. 

The following variable values are passed in the kubectl command to create a Kubernetes Secret:

  • nameOfSecret needs to match value in the Vector agen release configuration 

  • namespace is the name of the Namespace. We use vector for the namespace in this tutorial. The new password secret will be accessible only in this namespace.

  • password is the password or token that EraSearch provided for your authentication.

kubectl create secret generic <nameOfSecret> -n <namespace> --from-literal password=<password>

Install the Vector agent

Now, let’s install the Vector agent into the cluster by running the helm command below. The namespace should be vector; values.yaml is the release configuration file that you prepared earlier. 

helm upgrade --install --namespace vector --create-namespace vector vector/vector-agent --values values.yaml

View the logs

To view the logs of your newly created Vector deployment, run the following command:

kubectl logs --namespace vector daemonset/vector-agent

With Vector now configured with an EraSearch sink, you should see Kubernetes logs starting to flow into EraSearch. We saw how to easily implement cluster-level logging using Vector agents deployed via a DaemonSet. Vector is one of the best tools for collecting Kubernetes logs because of its high-performing transforming capabilities.

Next steps

To restart the Vector agents for any reason (for example, after updating your configuration), run the command:

kubectl rollout restart --namespace vector daemonset/vector-agent

To get the most value out of your Kubernetes and other logs at a very low cost, you’ll need an observability and analytics platform like EraSearch. As noted above, EraSearch is the fastest and easiest way to manage the complexity of Kubernetes and cloud environments. 

If you’re not a customer, sign up for a 60-day free trial of EraCloud.

If you have questions about EraSearch, contact us here.

Tags

Stay Informed