Preamble
Kafka is often introduced for decoupling, durability, and replay. In polyglot stacks, Python services commonly pair with Java peers and Avro/Protobuf in a schema registry; broker semantics are the same whichever language you use. This post walks through Python end-to-end, then trade-offs, then design patterns worth adopting when Kafka becomes part of your architecture.
Step-by-step: Kafka with Python
1. Choose a client
Two common options:
confluent-kafka(librdkafka): best throughput and operational parity with other production stacks; use when you care about latency, metrics, and stability at scale.kafka-python(pure Python): simpler to reason about for small scripts; often slower and less aligned with how large Kafka deployments are run.
Below uses confluent-kafka; install with pip install confluent-kafka (requires a working build environment for the native extension).
2. Prerequisites
You need brokers reachable from the app (bootstrap.servers), and topics created (via CLI, Terraform, or admin API). For local learning, Docker Compose with Kafka (or Redpanda) is enough; production adds TLS, SASL, and ACLs.
3. Producer: send a message
Configure bootstrap servers, produce with an optional key (for partition stickiness), and flush so shutdown does not drop in-flight sends:
from confluent_kafka import Producer
conf = {"bootstrap.servers": "localhost:9092"}
producer = Producer(conf)
def delivery_report(err, msg):
if err is not None:
print(f"delivery failed: {err}")
else:
print(f"delivered {msg.topic()} p={msg.partition()} o={msg.offset()}")
producer.produce(
"orders.created",
key="user-123".encode("utf-8"),
value=b'{"order_id":"o1"}',
callback=delivery_report,
)
producer.poll(0)
producer.flush(timeout=10)
Ordering is per partition: same key → same partition (for a stable partitioner). Random or missing keys spread load but give no cross-key ordering.
4. Consumer: read in a group
Subscribe with a group id; partitions in the topic are divided among group members. Process, then commit offsets after successful work if you want at-least-once semantics (or rely on auto-commit only if duplicate processing is safe):
from confluent_kafka import Consumer
conf = {
"bootstrap.servers": "localhost:9092",
"group.id": "order-workers",
"enable.auto.commit": False,
"auto.offset.reset": "earliest",
}
consumer = Consumer(conf)
consumer.subscribe(["orders.created"])
try:
while True:
msg = consumer.poll(1.0)
if msg is None:
continue
if msg.error():
print(msg.error())
continue
# process msg.value()
consumer.commit(asynchronous=False)
finally:
consumer.close()
Lag (difference between log end and committed offset) is the main health signal: alert when processing falls behind ingest.
5. Serialization and schemas
Raw bytes are the transport contract. In teams that share topics across services, use Avro or Protobuf with a schema registry so compatibility is enforced and Python/Java agree on field evolution. Breaking schemas without policy is a common source of incidents.
6. Async and web stacks
For asyncio, aiokafka is a common choice; the same ideas apply: partitions, groups, and commit timing. Propagate trace context into consumers when possible—see OpenTelemetry Traces Across Python and Java.
Benefits of using Kafka
- Durability and replay: retained logs let new consumers catch up and support reprocessing for fixes or new projections.
- Decoupling: producers do not need to know who consumes; add readers without changing writers (within schema constraints).
- Backpressure-friendly ingest: consumers pull at their own rate; lag makes overload visible.
- High throughput: partitioning scales horizontal read and write for suitable workloads.
Downsides and costs
- Operational complexity: brokers, ZooKeeper or KRaft, upgrades, disk, rebalancing, and monitoring are real ownership costs versus a single database or queue.
- Semantics are subtle: ordering, duplicates, and failure modes differ from “RPC then commit”; teams must design for at-least-once (or invest in exactly-once where supported and justified).
- Topic and schema governance: without conventions, topic sprawl and breaking schema changes hurt everyone.
- Python client trade-offs: native clients need build/deploy care; CPU-bound serialization in-process can matter at extreme rates compared to JVM ecosystems.
Design patterns when introducing Kafka
- Outbox pattern: write business data and an outbox row in the same DB transaction, then a relay publishes to Kafka—avoids “DB committed but event never sent” or the reverse.
- Idempotent or deduplicating consumers: assume duplicate delivery; use natural keys, idempotency keys, or dedupe stores so retries are safe.
- Bounded contexts and topic ownership: one team owns a topic’s contract (schema, semantics, SLA); others consume read-only. Prefer event names that mean something happened (
OrderPlaced) over vague pipes (updates). - Explicit partition strategy: choose keys so related events co-locate when ordering matters; avoid assuming global order.
- Dead-letter or quarantine topics: invalid or repeatedly failing messages go to a DLQ with metadata instead of blocking the main partition forever.
- Consumer group sizing: scale consumers up to partition count; more instances than partitions idle—plan partition count with expected parallelism.
- Schema evolution rules: enforce backward/forward compatibility in the registry; coordinate deploy order when schemas change.
- Choreography vs orchestration: Kafka favors choreography (many services reacting to events); for strict sagas, combine events with clear compensations or a dedicated orchestrator where complexity demands it.
Debugging
Read broker logs and client metrics together. Language is the wrapper; lag, rebalances, and ISR health tell the story.
Conclusion
Kafka pays off when you treat event streams as architecture: clear ownership, schemas, idempotent handling, and operational discipline. For Python services, pick a client that matches your scale, wire serialization to your registry, and design partitions and groups before traffic grows.