Apache Kafka

Apache Kafka is an open-source distributed event-streaming platform. It ingests, stores, and processes high-throughput streams of records across topics, powering messaging, log aggregation, and real-time data pipelines. Modern Kafka runs in KRaft mode, so it no longer needs a separate ZooKeeper cluster.

Software included

Component	Version
Apache Kafka	4.3.1
OpenJDK	21 (JRE)
Ubuntu	24.04 LTS

Getting started

1. Connect to your VM

ssh ubuntu@<your-vm-ip>

2. Wait for first-boot configuration

On the first boot, a setup script configures advertised.listeners, formats the KRaft storage, and starts Kafka. Track progress:

journalctl -u kafka-first-boot.service -f

The login message (MOTD) confirms when Kafka is ready.

3. Verify Apache Kafka is running

systemctl status kafka

List the available topics to verify the broker:

sudo -u kafka /opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server localhost:9092 --list

4. Connect to the broker

The bootstrap server listens on:

<your-vm-ip>:9092

You can review the connection guidance written at first boot:

sudo cat /root/.credentials/kafka.txt

Managing Apache Kafka

# Check service status
systemctl status kafka

# Restart
sudo systemctl restart kafka

# View logs
sudo journalctl -u kafka -f

Path	Purpose
`/opt/kafka/config/server.properties`	Broker and KRaft configuration
`/opt/kafka/data/`	Kafka data

Security

Port 9092 is accessible on the VM’s network interface. UFW is enabled and allows SSH (port 22) only by default.

To allow broker access from a specific IP:

sudo ufw allow from <trusted-ip> to any port 9092

The advertised listener address is generated from the VM’s local IP on first boot. To connect without opening the firewall, point the advertised listener at localhost on the VM, restart Kafka, then open an SSH tunnel:

# On the VM: advertise localhost instead of the VM's IP, then restart Kafka
sudo sed -i 's|^advertised.listeners=.*|advertised.listeners=PLAINTEXT://localhost:9092|' /opt/kafka/config/server.properties
sudo systemctl restart kafka

# Run this on your local machine
ssh -L 9092:localhost:9092 ubuntu@<your-vm-ip>

Kafka uses plaintext transport and has no authentication in this single-node image. Configure authentication and encrypted listeners before using it for production workloads, and restrict port 9092 to trusted clients.