← All cheatsheets

Data

Kafka — topics, producers, consumers, and lag

Fast-lookup reference for Apache Kafka: the topic/partition/offset mental model, producer keys and acks, consumer groups and rebalancing, offset commits, and consumer lag — the #1 health signal — with the CLI to diagnose and drain a backlog.

Docs-checkedkafkastreamingtopicsconsumer-grouplag

Quick reference — reach for these first

List all topicskafka-topics.sh --bootstrap-server localhost:9092 --listEnumerate topics on the cluster (add --describe TOPIC for partitions/replicas).
Describe a group → LAGkafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group GROUPThe health check: shows CURRENT-OFFSET, LOG-END-OFFSET, LAG per partition + consumer owner.
Consume from a topickafka-console-consumer.sh --bootstrap-server localhost:9092 --topic TOPIC --from-beginning --property print.key=trueTail/replay messages to inspect payloads (drop --from-beginning to read only new records).
Produce a test messagekafka-console-producer.sh --bootstrap-server localhost:9092 --topic TOPIC --property parse.key=true --property key.separator=:Type KEY:VALUE lines; the key determines the target partition.
Reset offsets (DESTRUCTIVE)kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group GROUP --topic TOPIC --reset-offsets --to-earliest --executeRewinds/skips a group's committed position. Group must have no active members. --dry-run first.

Topic → partitions → offsets, and how lag is measured

Kafka topic, partitions, offsets, consumer group, and lagA producer writes keyed records into a topic of three append-only partitions; a consumer group reads one partition each; lag is log-end-offset minus committed offset.Producerkey → partitionTopic: orders (append-only log)P0P1P2012345committedlog-endLAG = log-end − committed→ newer→ newerConsumer groupConsumer C0Consumer C1Consumer C2parallelism = #partitions1 partition → 1 consumer

Text equivalent: A producer sends keyed records into a topic split into three partitions, each an append-only log of offset-numbered cells. A consumer group with three consumers reads the partitions one-to-one. On partition 0, the consumer's committed offset sits behind the log-end offset, and the gap between them is labeled LAG.

Common tasks

Inspect · Operate · Recover

InspectOperateRecover
List topics on the clusterkafka-topics.sh --bootstrap-server localhost:9092 --listExpected topic names print; cluster is reachable.
Inspect a topic's partitions and replicationkafka-topics.sh --bootstrap-server localhost:9092 --describe --topic TOPICPartitionCount, ReplicationFactor, Leader, Isr columns shown; Isr size == ReplicationFactor.
Create a topic with N partitionskafka-topics.sh --bootstrap-server localhost:9092 --create --topic TOPIC --partitions 6 --replication-factor 3'Created topic TOPIC.' — partition count caps consumer-group parallelism.
Increase partitions on an existing topickafka-topics.sh --bootstrap-server localhost:9092 --alter --topic TOPIC --partitions 12PartitionCount rises on --describe. NOTE: changes key→partition mapping; cannot decrease partitions.
Produce a keyed test messageprintf 'user-42:hello\n' | kafka-console-producer.sh --bootstrap-server localhost:9092 --topic TOPIC --property parse.key=true --property key.separator=:Same key always lands on the same partition (hash of key mod partitions).
Consume and print keys/offsetskafka-console-consumer.sh --bootstrap-server localhost:9092 --topic TOPIC --from-beginning --property print.key=true --property print.offset=trueRecords appear with offset and key; Ctrl-C to stop.
List all consumer groupskafka-consumer-groups.sh --bootstrap-server localhost:9092 --listYour application's group id appears.
Read lag for a group (primary signal)kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group GROUPLAG column is near 0 and not trending up; every partition has a CONSUMER-ID.
See group members and partition assignmentkafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group GROUP --members --verboseActive member count ≤ partition count; no partition is unassigned.
Check the group's rebalance/coordinator statekafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group GROUP --stateSTATE = Stable (not PreparingRebalance/CompletingRebalance flapping).
Dry-run an offset reset before executingkafka-consumer-groups.sh --bootstrap-server localhost:9092 --group GROUP --topic TOPIC --reset-offsets --to-earliest --dry-runPrints the NEW-OFFSET it would set; no change applied yet.
Skip a poison message by shifting offsets (DESTRUCTIVE)kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group GROUP --topic TOPIC:2 --reset-offsets --shift-by 1 --executeGroup has NO active members first; CURRENT-OFFSET advances past the stuck record.
Inspect retention on a topickafka-configs.sh --bootstrap-server localhost:9092 --describe --topic TOPIC --all | grep retentionretention.ms / retention.bytes shown (default retention is cluster log.retention.hours = 168h).
Delete a topic (DESTRUCTIVE, PROD-CHANGING)kafka-topics.sh --bootstrap-server localhost:9092 --delete --topic TOPICAll data and offsets for the topic are removed; requires delete.topic.enable=true. Confirm topic is unused first.

Consumers are falling behind — diagnose and drain growing lag

troubleshooting

1
Read the group's lag. The LAG column (LOG-END-OFFSET − CURRENT-OFFSET) is climbing across partitions.
kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group orders-processor
# TOPIC      PARTITION  CURRENT-OFFSET  LOG-END-OFFSET   LAG      CONSUMER-ID
# orders     0          4820133         6060251          1240118  consumer-1-abc
# orders     1          4790021         6058900          1268879  -
# orders     2          4805554         6061002          1255448  consumer-1-abc
2
Check members vs partitions. Partition 1 shows CONSUMER-ID '-' (unowned), and one member owns multiple partitions — the group is under-provisioned, not just slow.
kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group orders-processor --members --verbose
# CONSUMER-ID       HOST          #PARTITIONS  ASSIGNMENT
# consumer-1-abc    /10.0.0.5     2            orders(0,2)
# (only 1 active member for a 3-partition topic → 1 partition queued with no owner)
3
Decide down vs slow. A missing owner on P1 means a consumer is down (or never scaled to partition count). If every partition were owned but LAG still rose, the cause would be a slow handler instead.
4
Scale consumers up to the partition count (3), so each partition gets its own consumer. Adding more than 3 would leave the extras idle (parallelism = partitions).
kubectl scale deploy/orders-processor --replicas=3   # or start 2 more consumer instances
# Rebalance assigns orders(0),(1),(2) to three distinct members.
5
Watch LAG drain. Re-describe every ~15s; CURRENT-OFFSET should climb faster than LOG-END-OFFSET and every partition now has an owner.
watch -n 15 'kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group orders-processor'
# orders 0 ... LAG 612004  consumer-1-abc
# orders 1 ... LAG 598110  consumer-2-def   ← now owned, draining
# orders 2 ... LAG 604887  consumer-3-ghi
6
Verify recovery: LAG settles near 0 and stays flat across several polls, STATE is Stable. If a single partition's LAG instead froze while others drained, suspect a poison message and jump to that runbook.
kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group orders-processor --state
# STATE: Stable  #MEMBERS: 3   — LAG ~0 and steady: healthy.
# (Tip: an LLM such as Claude Opus 4.x / Sonnet can summarize the --describe output into an on-call alert, but the LAG trend is the ground truth.)

Troubleshooting — symptom → first check → next check

SymptomFirst checkEstablishesNext check
Consumer lag is large and growingkafka-consumer-groups.sh --describe --group GROUP — watch the LAG column over 30sConsumers can't keep up: either a consumer is down (partition unowned) or the handler is slower than the produce rate.If partitions are unassigned, restart/scale consumers up to the partition count; if all owned but LAG still climbs, speed up the handler (batch, async I/O) or raise max.poll.records / add partitions.
Group keeps rebalancing (PreparingRebalance loops, 'storms')--describe --group GROUP --state and broker logs for repeated JoinGroup; check consumer poll loop timingMembers are being evicted because they miss heartbeats or exceed max.poll.interval.ms (slow processing between polls).Lower per-poll work (max.poll.records) or raise max.poll.interval.ms; tune session.timeout.ms/heartbeat.interval.ms; enable cooperative-sticky assignor to avoid stop-the-world rebalances.
Under-replicated or offline partitionskafka-topics.sh --describe --under-replicated-partitions (Isr smaller than Replicas)A broker is down or lagging, so some replicas fell out of the in-sync set — durability and (for min.insync.replicas) acks=all writes are at risk.Bring the failed broker back or reassign replicas; check disk/network on the lagging broker; verify min.insync.replicas vs replication.factor so acks=all can still be satisfied.
One partition's lag is stuck; consumer makes no progress (poison message)--describe --group GROUP — one partition's CURRENT-OFFSET is frozen while LOG-END-OFFSET rises; check consumer error logsA record repeatedly throws in the handler, so the offset never commits and the partition blocks.Fix the handler and/or route failures to a dead-letter topic; as a stopgap stop the group and --reset-offsets --shift-by 1 on that TOPIC:PARTITION to skip the record (DESTRUCTIVE — you drop that message).
Throughput is capped no matter how many consumers you addkafka-topics.sh --describe --topic TOPIC — compare PartitionCount to active group membersParallelism in a group equals partition count; extra consumers beyond that sit idle with no partitions.Increase partitions (kafka-topics --alter --partitions N) to raise max parallelism — note it rekeys future messages and never reorders within a key; size partitions ahead of peak.
Messages for the same key land out of order or on different partitionsConfirm the producer sets a key and that PartitionCount hasn't changed recentlyOrdering is guaranteed only within a partition; a null key uses sticky partitioning (Kafka 2.4+, KIP-480) — batched onto one partition then rotated, not per-record round-robin — and adding partitions changes key→partition mapping.Always set a stable partition key for ordered streams; avoid altering partition count on order-sensitive topics, or plan a migration.

References

Classic Kafka CLI shipped with Apache Kafka (kafka-*.sh wrappers) against a broker at localhost:9092; placeholders TOPIC/GROUP in CAPS. Partitioning assumes the default partitioner (hash of key mod partition count). Offset/lag semantics follow committed offsets stored in __consumer_offsets. Retention default cited as the cluster log.retention.hours = 168h (7 days) unless a per-topic retention.ms/retention.bytes override is set.