When I first heard the word "Kafka" in a technical meeting at Ericsson, I nodded confidently while quietly Googling it under the table. A few months later, I was designing Avro schemas, building consumer groups, and debugging lag metrics in production.

This is everything I wish someone had told me when I started.

🎯 What Is Apache Kafka ?

Imagine a post office. Instead of letters, services send events things that happened, like "user logged in" or "notification dispatched". Instead of delivering directly to recipients, every event goes into a central post box (Kafka). Any service that cares about that event can pick it up whenever it's ready.

This is event-driven architecture. And Kafka is the most battle-tested way to build it.

The technical version: Apache Kafka is a distributed event streaming platform that lets you:

• Publish events from producer services

• Store events reliably and durably

• Subscribe to events from consumer services

• Process streams of events in real time

At Ericsson, every customer notification: SMS, push, email, in-app is flown through a Kafka-based pipeline. Getting this right was critical. Getting it wrong meant missed messages at scale.

🏗️ Core Concepts You Must Understand

Topics

A topic is a named category for events. Think of it like a database table, but append-only.

Enter fullscreen mode

Exit fullscreen mode

Producers

A producer is any service that writes events to a topic.

private final KafkaTemplate kafkaTemplate; private static final String TOPIC = "notification-events";

public NotificationProducer(KafkaTemplate kafkaTemplate) { this.kafkaTemplate = kafkaTemplate; }

public void sendNotification(NotificationEvent event) { kafkaTemplate.send(TOPIC, event.getUserId(), event); log.info("Notification event sent for userId: {}", event.getUserId()); } }`

Enter fullscreen mode

Exit fullscreen mode

Consumers

A consumer is any service that reads events from a topic.

private final NotificationProcessor processor;

@KafkaListener(topics = "notification-events", groupId = "notification-service") public void consume(ConsumerRecord record) { log.info("Received event for userId: {}", record.key()); processor.process(record.value()); } }`

Enter fullscreen mode

Exit fullscreen mode

Consumer Groups

Multiple consumers can form a group to share the work of processing events. Kafka automatically distributes partitions across group members, this is how you scale horizontally.

Enter fullscreen mode

Exit fullscreen mode

Partitions

Topics are split into partitions, this is what enables parallelism. Events with the same key always go to the same partition, preserving order.

Enter fullscreen mode

Exit fullscreen mode

📦 Avro Schemas — Why They Matter

Raw JSON Kafka messages are flexible but dangerous at scale. One typo in a field name can break every downstream consumer silently.

Avro schemas solve this by defining the exact structure of every message, enforced at the Schema Registry level before a message is even sent.

Define your schema (notification-event.avsc):

Enter fullscreen mode

Exit fullscreen mode

⚡ Setting Up Kafka Locally

The fastest way to run Kafka locally for development:

docker-compose.yml:

version: '3.8'

Enter fullscreen mode

Exit fullscreen mode

Start everything:

docker-compose up -d

Enter fullscreen mode

Exit fullscreen mode

Verify Kafka is running:

docker ps

You should see zookeeper, kafka, and schema-registry all running`

Enter fullscreen mode

Exit fullscreen mode

🔍 Monitoring Kafka Lag

Consumer lag is the number of unprocessed messages sitting in a partition. High lag = your consumers are falling behind.

At Ericsson we monitored this constantly.

Output:

Enter fullscreen mode

Exit fullscreen mode

Partition 1 has a lag of 5 minor, but worth watching. If lag grows consistently, you need more consumer instances.

Add a second consumer instance by simply running another instance of your service, Kafka automatically rebalances partitions across the group.

🐛 Common Mistakes I Made (So You Don't Have To)

1. Not setting a partition key

// ✅ Correct — keyed by userId, preserves order per user kafkaTemplate.send(TOPIC, event.getUserId(), event);`

Enter fullscreen mode

Exit fullscreen mode

2. Creating ObjectMapper inside the consumer loop

// ✅ Correct — inject as Spring Bean @Autowired private ObjectMapper mapper;`

Enter fullscreen mode

Exit fullscreen mode

3. Not handling deserialization errors

Enter fullscreen mode

Exit fullscreen mode

4. Forgetting idempotency

Kafka guarantees at least once delivery, your consumer may receive the same message twice. Always make processing idempotent:

Enter fullscreen mode

Exit fullscreen mode

📊 What I Learned in Production

After months of running Kafka in production at Ericsson:

Lesson Detail

Partition key matters Always key by the entity that needs ordering (userId, orderId)

Monitor lag daily Lag growth is an early warning sign

Dead letter queues Always have a DLQ for failed messages

Schema evolution Add fields as nullable with defaults — never remove fields

Consumer group naming Use descriptive group IDs — notification-service not group1

Replication factor Always 3 in production for fault tolerance

🚀 Getting Started Checklist

If you're setting up Kafka for the first time:

• Run Kafka locally with Docker Compose (see above)

• Create your first topic

• Write a simple producer in Spring Boot

• Write a simple consumer with @KafkaListener

• Define an Avro schema for your messages

• Add error handling and a dead letter queue

• Monitor consumer lag

• Test with multiple consumer instances

🔮 What's Next

Kafka is one of those technologies where the basics are approachable but the depth is enormous. Once you're comfortable with the fundamentals, explore:

• Kafka Streams — real-time stream processing

• KSQL — SQL-like queries over Kafka streams

• Exactly-once semantics — guaranteeing no duplicates

• Kafka Connect — integrating Kafka with databases and external systems

The investment in learning Kafka properly pays dividends across your entire engineering career. Event-driven architecture is how modern distributed systems are built and Kafka is at the center of it.

Thanks for reading! I'm Zaina, a Software Engineer based in Perth, Australia, working with Java microservices, Apache Kafka, and cloud-native technologies at Ericsson. Connect with me on LinkedIn or check out my portfolio.

Found this useful? Drop a ❤️ and share it with a fellow engineer who's just getting started with Kafka!