Era Software

No results match your query

Writing data with Cloudflare

Estimated reading time: 3 minutes
  • guide
  • cloudflare
  • eracloud
  • self-hosted

This integration is not yet generally available. Please reach out if you would like to use this integration or have any questions.

This page shows how to use Cloudflare's Logpush feature to write logs to EraSearch. In this guide, you'll:

  • Configure Cloudflare to push logs to EraSearch using the Splunk HEC format
  • View the logs in EraSearch

Before you begin
Copy
Copied!

This content is intended for engineers and developers using EraSearch on EraCloud or self-hosted EraSearch:

You also need administrative access to a Cloudflare Enterprise account to configure the Logpush integration.

Instructions
Copy
Copied!

Step 1: Go to the Logpush job configuration page in Cloudflare
Copy
Copied!

From the Cloudflare dashboard, after selecting a domain you want to use with Logpush:

  1. Navigate to the Analytics section
  2. From Analytics, choose the Logs sub-section
  3. From Logs, select Connect a Service to create the integration

Step 2: Configure the data fields for your Logpush job
Copy
Copied!

From the Select Data Set screen, choose the HTTP requests option.

From the Select Data Fields screen, select all available fields using the checkbox at the top.

Note: If you already know what data fields you want to include or exclude from the integration, feel free to only select those items. If you are new to this integration, we recommend starting with all options.

Step 3: Configure the destination for your Logpush job
Copy
Copied!

From the Select Destination screen, choose the Splunk option.

Note: This guide uses the Splunk option to integrate with EraSearch. That workflow is possible because the EraSearch REST API supports ingesting data in Splunk's HEC format.

From the Enter Destination Info screen, fill in the following details:

  • Splunk raw HTTP Event Collector URL - This is your EraSearch URL with the suffix /services/collector/raw. For example, if your EraSearch URL is https://era.example.com, set this option to https://era.example.com/services/collector/raw.

  • Channel ID - A random UUID or string identifier used to identify this stream of data. You'll need to generate this yourself using a random string or UUID generator.

  • Auth Token - This is the token set in the Authorization header of the incoming requests. To have EraSearch accept the data, this needs to be set to a valid authorization header.

    • For EraSearch on EraCloud, use Bearer%20${YOUR_ERACLOUD_TOKEN}, where ${YOUR_ERACLOUD_TOKEN} is your EraCloud API key.
    • For self-hosted EraSearch, insert a valid basic HTTP authorization header.
  • Source Type - The source type, set to cloudflare:json.

Step 4: Enable your Logpush job
Copy
Copied!

With the destination information configured, all that's left to do is enable the integration by selecting Push.

Once completed, data should now be flowing from Cloudflare into your EraSearch database. In the next section, you'll view your data.

Step 5: View your data in EraSearch
Copy
Copied!

For EraSearch on EraCloud
Copy
Copied!

Access EraSearch's UI by visiting your EraCloud account and clicking Search. Your logs are in the logs-cloudflare index

index

An index is a group of similar documents. With EraSearch, you can query documents in one or more indexes to optimize your searches.

. You may need to refresh the UI if the index is new.

For self-hosted EraSearch
Copy
Copied!

Use the EraSearch REST API to query the logs in EraSearch. Paste this command in your terminal, replacing YOUR_ERASEARCH_URL with your EraSearch URL, for example, http://localhost:9200.

Copy
Copied!
$ curl 'YOUR_ERASEARCH_URL/logs-cloudflare/_search?q=_lid:*'

The response shows information about your data and API request, including:

  • took - The time, in milliseconds, EraSearch took to serve the query request
  • _id - A unique, auto-generated numerical identifier for documents

    document

    A document is a JSON object made up of data. In EraSearch, all documents have a unique identifier (_id) and a timestamp (_ts). Most documents include additional fields. Here's an example of a document:

    Copy
    Copied!
    {"_id":4248176661010579457,"_line":"access","response":200,"_ts":1634060854000}
    

Next steps
Copy
Copied!

You're all set. Your EraSearch instance is now receiving real-time log data. For more information about Cloudflare, including what logs you can collect, visit these pages:

For other ways to get data into your database, visit the list of write integrations. To learn more about exploring, querying, and visualizing your data in EraSearch, visit these pages: