Rsyslog Project . Using imkafka

One of these modules is the imkafka module, which enables Rsyslog to receive log messages from Apache Kafka.Apache Kafka is a distributed messaging system that is widely used for streaming data in real-time. It is used in various applications for collecting and analyzing large volumes of data. Rsyslog's imkafka module allows it to integrate with Kafka and receive log messages in real-time.

To use the imkafka module, you need to first configure Rsyslog to receive log messages from Kafka. This involves setting up a Kafka topic for the log messages and configuring Rsyslog to subscribe to that topic. Once the configuration is set up, Rsyslog can start receiving log messages from Kafka.

The imkafka module supports various configuration options, which allow you to fine-tune the way Rsyslog receives log messages from Kafka. For example, you can configure the maximum message size, the number of messages to fetch in a single request, and the polling interval for fetching messages.

Once Rsyslog receives log messages from Kafka, it can process them using various filters and output them to various destinations. For example, you can filter log messages based on their severity or source, and output them to different files or databases.

To install imkafka module  :

yum install rsyslog-kafka -y

or alternatively you can try this as well

yum install librdkafka -y

An example configuration would be some thing like :

module(load="imkafka")
input(type="imfile"
  File="/var/log/messages"
  Tag="messages"
  Facility="local7"
  Severity="info"
  ruleset="kafka")
  
ruleset(name="kafka") {
  action(
    broker=["localhost:9092"]
    topic="mytopic"
    type="omkafka"
    template="json-template"
    partition="auto"
    queue.type="linkedlist"
    queue.size="10000"
    queue.highwatermark="8000"
    queue.lowwatermark="2000"
  )
}

template(name="json-template" type="list") {
  constant(value="{\"message\":\"")
  property(name="msg")
  constant(value":"\"}")
}

References:

1. https://www.rsyslog.com/doc/configuration/modules/imkafka.html