Many organizations have standardized on Splunk for logging operations. This procedure describes how to integrate a Splunk Enterprise configuration with a Konvoy cluster.
Splunk Prerequisites
Before your begin, you need the following information specific to your Splunk configuration:
- IP address of your Splunk Enterprise installation reachable from your Konvoy cluster
Create an index for the Konvoy Cluster
Use the Splunk console to create an index into the kubernetes logs. Go to Settings -> Indexes. Enter information to create a new index. Select the Save button to save your index settings.
Create a new HTTP Event Collector
Go to Settings -> Data Inputs. Select +Add New for HTTP Event Collector. Enter information to create a new HTTP Event Collector. Select the Next button.
Take note of the displayed Token Value, it will be used for the Fluentbit configuration.
Go to Global Settings.
For the sake of simplifying this tutorial, disable SSL by unchecking Enable SSL. Take note of the HTTP Port Number, it will be used for the Fluentbit configuration.
In the Input Settings dialog box, select the name of the recently created index to associate that index with this HTTP Event Collector.
Disable default Konvoy cluster monitoring
In your config.yaml
, for each Konvoy cluster, edit the addons section and disable the Elastic Search Components:
- name: elasticsearch
enabled: false
- name: elasticsearch-curator
enabled: false
- name: elasticsearchexporter
enabled: false
- name: kibana
enabled: false
Configure Fluentbit to send logs to Splunk
In your cluster.yaml
file, for each Konvoy cluster, edit the Fluentbit section. Add the supplied host IP address, port and HTTP Event Collector token of your Splunk configuration.
- name: fluentbit
enabled: true
values: |
# add value for splunk
backend:
splunk:
host: 192.168.10.6
port: 8088
token: "fc13d8bd-a511-48f9-831a-3820bb1c34cc"
send_raw: "off"
tls: "off"
tls_verify: "off"
tls_debug: 1
message_key: "kubernetes"
type: splunk
Start your Konvoy cluster. Konvoy starts and uses your updated configuration.
Check the Fluentbit pod for health
Ensure the Fluentbit pods are running successfully.
kubectl get pods -n kubeaddons |grep fluent
fluentbit-kubeaddons-fluent-bit-cbbgj 1/1 Running 0 23m
fluentbit-kubeaddons-fluent-bit-cln4p 1/1 Running 0 20m
fluentbit-kubeaddons-fluent-bit-fqffj 1/1 Running 0 4s
fluentbit-kubeaddons-fluent-bit-pxb7w 1/1 Running 0 22m
fluentbit-kubeaddons-fluent-bit-tnhgd 1/1 Running 0 19s
fluentbit-kubeaddons-fluent-bit-z8fbp 1/1 Running 0 4m27s
Examine the logs for the Fluentbit container
kubectl logs -n kubeaddons fluentbit-kubeaddons-fluent-bit-fqffj
Fluent Bit v1.3.2
Copyright (C) Treasure Data
[2020/02/28 00:35:58] [ info] [storage] initializing...
[2020/02/28 00:35:58] [ info] [storage] in-memory
[2020/02/28 00:35:58] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[2020/02/28 00:35:58] [ info] [engine] started (pid=1)
[2020/02/28 00:35:58] [ info] [filter_kube] https=1 host=kubernetes.default.svc port=443
[2020/02/28 00:35:58] [ info] [filter_kube] local POD info OK
[2020/02/28 00:35:58] [ info] [filter_kube] testing connectivity with API server...
[2020/02/28 00:35:58] [ info] [filter_kube] API server connectivity OK
[2020/02/28 00:35:58] [ info] [http_server] listen iface=0.0.0.0 tcp_port=2020
[2020/02/28 00:35:58] [ info] [sp] stream processor started
Access your Konvoy logs from Splunk
Go to the Splunk user console to do a search. Use the index name you associated with your HTTP Event Collector. Enter the search string:
index=“konvoy_logs” *
When this integration is complete your Konvoy logs display in your Splunk console.
References
Splunk Operator Github: https://splunk.github.io/splunk-operator/
Fluentbit Docs: https://fluentbit.io/documentation/0.13/output/splunk.html