SOC4Kafka collector

The new SOC4Kafka collector, built on OpenTelemetry, enables the collection of Kafka messages and forwards these events to Splunk. It serves as a replacement for the existing Splunk Connect for Kafka (kafka-connect-splunk).

For the installation instructions and advanced settings check the documentation.

Requirements

1. Kafka version 3.7.0 and above.
   • Tested with following versions: 3.7.0, 3.8.0, 3.9.0, 4.0.0
2. A Splunk environment of version 9.x and above, configured with valid HTTP Event Collector (HEC) token.

NOTE: HEC Acknowledgements are not supported in SOC4Kafka

Support technologies

Splunk OTel Collector for Kafka lets you subscribe to a Kafka topic and stream the data to the Splunk HTTP event collector on the following technologies:

• Apache Kafka
• Amazon Managed Streaming for Apache Kafka (Amazon MSK)
• Confluent Platform

Key differences to Splunk Connect for Kafka

Not supported features which are available in previous version of Splunk Connect for Kafka but are not available in SOC4Kafka collector:

• Acknowledgment support - Not supported
• Protobuf encoding - Not supported

How to start with SOC4Kafka?

Choose an installation method that fits your environment:

• Kubernetes (Helm): Use the Helm chart to deploy SOC4Kafka on Kubernetes. See the chart Installation Guide for install and upgrade steps.
• Automated (Ansible): See the Quickstart Guide for automated installation.
• OCI Streaming: See the OCI Streaming for Oracle Cloud Infrastructure installation.
• Manual: Follow the steps below to run the collector from a downloaded package and config file.

Download Splunk OTel Collector package

The SOC4Kafka base package is the Splunk OpenTelemetry Collector, offering multiple installation methods to suit different needs. Get the newest release (prefixed with v) using this link, download the package suited for your platform.

For instance, if you are using Linux on an AMD64 architecture, you can execute the following wget command:

wget https://github.com/signalfx/splunk-otel-collector/releases/download/v0.159.0/otelcol_linux_amd64

Create a minimal config template

receivers:
  kafka:
    brokers: [<Brokers>]
    logs:
      topics:
        - <Topic>
      encoding: <Encoding>

processors:
  resourcedetection:
    detectors: ["system"]
    system:
      hostname_sources: ["os"]

exporters:
  splunk_hec:
    token: "<Splunk HEC Token>"
    endpoint: <Splunk HEC Endpoint>
    source: <Source>
    sourcetype: <Sourcetype>
    index: <Splunk index>
    tls:
      insecure_skip_verify: false
    splunk_app_name: "soc4kafka"
    sending_queue:
      enabled: true
      num_consumers: 10
      queue_size: 10000
      block_on_overflow: true
      sizer: items
      batch:
        min_size: 1000

service:
  pipelines:
    logs:
      receivers: [kafka]
      processors: [resourcedetection]
      exporters: [splunk_hec]

Configuration Table

Mind that this is just a minimal configuration. You can customize it further based on your requirements by referring to the official documentation linked in the Component column.

Category	Component	Parameter	Description	Required	Default Value
Receivers	kafka	brokers	Kafka broker addresses for message consumption.	Yes	N/A
		logs.topics	Kafka list of topics to subscribe to for receiving messages.	Yes	N/A
		logs.encoding	Encoding format of the Kafka messages.	No	"text"
Processors	resourcedetection		Sets a host field based on a machine's information.	No	N/A
Exporters	splunk_hec	token	Splunk HEC token for authentication.	Yes	N/A
		endpoint	Splunk HEC endpoint URL for sending data.	Yes	N/A
		source	Source metadata for events sent to Splunk.	No	"otel"
		sourcetype	Sourcetype metadata for events sent to Splunk.	No	"otel"
		index	Splunk index where the logs will be stored.	Yes	N/A
		tls.insecure_skip_verify	Whether to skip checking the certificate of the HEC endpoint when sending data over HTTPS.	No	false
		sending_queue.queue_size	Maximum number of queued items waiting to be exported.	No	10000
		sending_queue.block_on_overflow	Applies backpressure instead of immediately rejecting data when the exporter queue is full.	No	true
		sending_queue.sizer	Counts queue capacity by items.	No	items
		sending_queue.batch	Enables exporter-level batching before requests are sent to Splunk HEC.	No	enabled
		sending_queue.batch.min_size	Minimum number of items to batch before sending a request.	No	1000
Service		pipelines.logs.receivers	Specifies the receiver(s) for the log pipeline.	Yes	N/A
		pipelines.logs.processors	Specifies the processor(s) for the log pipeline.	No	[] (empty)
		pipelines.logs.exporters	Specifies the exporter(s) for the log pipeline.	Yes	N/A

Example configuration

receivers:
  kafka:
    brokers: ["kafka-broker-1:9092", "kafka-broker-2:9092", "kafka-broker-3:9092"]
    logs:
      topics: 
       - "example-topic"
      encoding: "text"

processors:
  resourcedetection:
    detectors: ["system"]
    system:
      hostname_sources: ["os"]

exporters:
  splunk_hec:
    token: "your-splunk-hec-token"
    endpoint: "https://splunk-hec-endpoint:8088/services/collector"
    source: my-kafka
    sourcetype: kafka-otel
    index: kafka_otel
    tls:
      insecure_skip_verify: false
    splunk_app_name: "soc4kafka"
    sending_queue:
      enabled: true
      num_consumers: 10
      queue_size: 10000
      block_on_overflow: true
      sizer: items
      batch:
        min_size: 1000

service:
  pipelines:
    logs:
      receivers: [kafka]
      processors: [resourcedetection]
      exporters: [splunk_hec]

Fill the file with your data and save it with a .yaml extension. For example config.yaml.

Run Splunk OTel Collector package with config file

To run SOC4Kafka Connect, use the base package along with a completed configuration template.

./<otel_package> --config <config_file>

NOTE: Ensure the file has executable permissions before running the command. On Linux-based systems you can add executable permissions using the following command:

chmod a+x <otel_package>

Example: For Linux on AMD64 architecture:

chmod a+x otelcol_linux_amd64
./otelcol_linux_amd64 --config config.yaml

For the detailed instructions and advanced settings check the documentation.