Enterprise Logging Integrations

ELK Stack - Filebeat Integration

  • Updated on Apr 25, 2022
  • 10 minutes to read
  • Contributors

Overview

Banyan supports integrations with the ELK Stack (which includes three open-source projects: Elasticsearch, Logstash, and Kibana) for log consumption. We do this by integrating with Filebeat, using Banyan’s API for system logs. Banyan’s integration provides the system log events, shown in the System Log in the console. The API used to integrate with the ELK stack uses the same terminology and definitions found in our system logs. All syntax is defined here.

Pre-requisites

  • Filebeat server access
  • Rights to store API keys and make configuration changes to the filebeat.yaml file

Steps

Step 1: Create an API key in Banyan’s Cloud Command Center with ReadOnly Access

1.1 In Banyan’s Cloud Command Center, navigate from Settings > API Keys.

1.2 Add an API Key, assigning a Name, a Description, and a ReadOnly scope.

Step 2: Store API key in Filebeat store

2.1 Log into the Filebeat system.

2.2 Enter the following in your CLI:

(a) filebeat keystore create

Note: This is only needed if the filebeat key store does not exist.

(b) filebeat keystore add BANYAN_API_KEY

(c) the string generated for the ReadOnly API key from Step 1

Step 3: Update filebeat.inputs section of the filebeat.yaml file

3.1 Enter the following into the filebeat.inputs section:

  `- type: httpjson
    config_version: 2
    interval: 1m
    request.url: 'https://net.banyanops.com/api/v1/events'
    request.transforms:
    - set:
        target: header.Authorization
        value: 'Bearer ${BANYAN_API_KEY}'
    - append:
        target: url.params.after
        value: '[[ .cursor.last_created_at ]]'
        default: '[[ (now (parseDuration "-5m")).UnixMilli ]]'
    - append:
        target: url.params.order
        value: 'ASC'
    - append:
        target: url.params.severity
        value: 'INFO'
    - append:
        target: url.params.limit
        value: '1000'
  response.split:
    target: body.data
  cursor:
    last_created_at:
      value: '[[ printf "%d" (add (toInt (index .last_event "created_at")) 1) ]]'
  fields_under_root: true
  fields:
    event.dataset: banyan`

3.2 Save your file.

3.3 Restart your Filebeat service.

Note: For more information on how to start Filebeat on each platform type, visit here.

Additional Information

The filebeat.yaml file contains editable fields. We recommend the code block above (in Step 3.1). Some of these fields are described below:

type: httpjson

  • Note: Our API for event logs returns in JSON format.

interval: 1m

  • Note: Pulling once per minute prevents being rate limited from the API; If you require a shorter interval, please contact support.

value: 'Bearer ${BANYAN_API_KEY}

  • Note: BANYAN_API_KEY is the name of the key stored in Step 1. We recommend entering the key into a secure space (i.e., Filebeat keystore) rather than entering the plaintext of the key in the filebeat.yaml file.

value: '[[ .cursor.last_created_at ]]'

  • Note: Using our last_created_at value in the events JSON will provide the last time the events database consumed data.

default: '[[ (now (parseDuration "-5m")).UnixMilli ]]'

  • Note: If the last_created_at field is unavailable (most often occurs during the first launch of integration), we recommend only obtaining the last 5 minutes of data so that you don’t exceed the request size limit. If you require more, please contact support.

event.dataset: banyan

  • Note: We recommend calling the events something specific for easy searching and filtering (e.g., ‘banyan’).
Can’t find what you’re looking for?

We’re happy to help. Contact us here.