Running Apache Kafka on a VPS: Event Streaming for High-Volume Applications

What is Apache Kafka?

Apache Kafka is a distributed event streaming platform. Think of it as a highly scalable, fault-tolerant message queue. Producers write events (messages) to topics. Consumers read events from topics. Events are retained for a configurable period, allowing multiple consumers to process the same events independently.

Installing Kafka

wget https://downloads.apache.org/kafka/3.7.0/kafka_2.13-3.7.0.tgz
tar -xzf kafka_2.13-3.7.0.tgz
cd kafka_2.13-3.7.0

# Start Zookeeper
bin/zookeeper-server-start.sh config/zookeeper.properties &

# Start Kafka
bin/kafka-server-start.sh config/server.properties &

Creating Topics and Producing Messages

bin/kafka-topics.sh --create --topic orders --bootstrap-server localhost:9092 --partitions 3 --replication-factor 1
echo "Order #12345" | bin/kafka-console-producer.sh --topic orders --bootstrap-server localhost:9092

FAQ

How much RAM does Kafka need?

Kafka itself needs 4-6 GB of JVM heap. Total VPS RAM should be 16+ GB for production use — Kafka performs much better when the OS can cache frequently accessed log segments in the page cache.