Kafka Partitions are the wrong ordering abstraction. Keys are.

So, what’s the deal here?

I spent years building event-driven and event-sourcing systems before Conduktor. The kind of systems where the partition is a deployment detail and the key: the entity id, whatever your domain calls it, is the entire universe (customer_id, order_id, etc.). I doubled down on his post:

Partitions are the wrong ordering abstractions TBF, keys are, and this library was “fixing” Kafka. Parallelism + Keys + Offset commit correctness is the real deal. Super useful for event-driven system and event-sourcing where the key is all that matters. You want parallelism across keys and no head-of-line blocking because one key or one external call is slow.

Almost every “we need more parallelism on this consumer” problem in an event-driven system is really a “we’re using the wrong abstraction” problem.

Let me walk through it.

What is Parallel Consumer?

Parallel Consumer is a library that wraps the vanilla Kafka consumer and lets a single consumer instance process records in parallel across configurable units of granularity:

• KEY: the precious: per-key order. Records with the same key are processed sequentially in the application, and records with different keys can run in parallel. This is the default, and what you want.
• UNORDERED: no ordering guarantee. Maximum parallelism. Just crunch through the records in parallel.
• PARTITION: spin up worker threads for each partition to dispatch the work, instead of multiplexing records from N partitions sequentially

The challenge it’s solving is to parallelize processing in the application while keeping offset commit correctness, as Kafka does not support ack-per-message but only commit-per-batches.

Why is this fixing Kafka?

Because it parallelizes within the partitions already assigned to that consumer instance. It doesn’t steal partitions from other members of the group, and it doesn’t touch the rebalance protocol at all.

One instance assigned 10 partitions can process each batch in parallel: 10, 100, 1000, whatever you set as maxConcurrency, and the lib will keep ordering everything by key/timestamp.

You decouple your processing concurrency from your partition count.

This is the entire pitch of this lib.

Why Share Consumers are not the replacement

Share Consumers (KIP-932, “Queues for Kafka”) are great and production-ready since 4.2 (February 2026). They give Kafka a queue-like consumption model on top of the same topic you used to consume.

Each record is individually acquired by a consumer, then it is processed then acknowledge. In terms of actions: ACCEPT, RELEASE (put it back for retry), or REJECT (dead-letter it).

The records lifecycle becomes: Available → Acquired → Acknowledged.

It’s per-message acks.

The big difference with parallel-consumer: there is no per-partition order and no per-key order. The KIP: “The records in a share-partition can be delivered out of order to a consumer.” Can = Will.

• For job-queue use cases (process this PDF, send this email, run this image transformation), Share Consumers are excellent. Each record is an independent task, order doesn’t matter, you want max throughput and per-record retry.
• For event-driven systems, event sourcing? Ordering per key is the entire point.
• For AI agents, it depends on their logic and context needs

If UserUpdated(user=42) arrives at offset 10 and UserDeleted(user=42) arrives at offset 50, you'd better process them in that order. Otherwise you've just resurrected a deleted user.

Telling people to migrate from Parallel Consumer to Share Consumers is like telling people to migrate from PostgreSQL to Redis because both store data. Sure, but the semantics and behavior is vastly different.

Partitions are an infrastructure concern. Keys are the domain concern.

The partition is a storage and assignment unit. It exists because Kafka needs a way to shard data across brokers and distribute consumption across instances. It’s a side effect of horizontal scaling. The partition count is a deployment parameter : you pick a number, hope it’s enough, and live with the consequences.

The key, on the other hand, is what your domain actually cares about. user_id, order_id, account_id. The business contract is: “all events about the same entity must be processed in order”. Nobody in the business cares whether user_42 lives on partition 7 or partition 23. They care that the sequence of events on user_42 is respected.

The partition is the wrong abstraction for ordering , it's a leaky one that couples your domain's ordering guarantee to a deployment constraint. If you bump partitions to chase throughput, you’ll enjoy reporting processing errors and correctness issues downstream.

Another consequence of this partition model is the head-of-line blocking issue: Slow records block other records that have no reason to be blocked. If processing user_42 takes 4 second, all records following on the same partition will have to wait 4 seconds, just because they're behind user_42 in line.

The vanilla consumer model is: one partition = one unit of work = one thread. It falls apart the moment your processing is IO-bound: external HTTP call, database write, ML inference, external enrichment, RAG retrieval, LLM call (yep), anything synchronous. And let me tell you, that’s VERY common.

So the question becomes:

How do you process records in parallel across keys, without breaking per-key ordering, without leaking ordering bugs into your domain, and without committing offsets wrong?

That’s exactly what Parallel Consumer was built for.

The hard part: offset commit correctness

Clearly underestimated. Parallel processing across keys is the easy part, just fan out by key into a thread pool, right? Not so fast.

Imagine you’re processing partition 7. The records come in at offsets 100, 101, 102, 103, ..., 200. You're processing them in parallel, by key.

• Record at offset 105 (key A) takes 4 seconds because it's slow.
• Records 106, 107, 108 (keys B, C, D) finish in 100ms each.

What offset do you commit?

Two wrong answers:

• If you commit 108 and if the consumer crashes, you'll lose record 105 on restart.
• If you wait for 105 to finish before committing anything, you've reintroduced head-of-line blocking.

The right (complicated) answer:

You keep an in-memory map of completed offsets per partition, you commit only the highest contiguous offset that has been fully processed, and you keep the “gaps” (completed but blocked-by-earlier-incomplete records) in memory so the system can recover correctly on restart.

This is non-trivial, and you need to think about memory, backpressure, rebalancing situation, key fairness. Parallel Consumer gets all of this right. It stuffed this state into the offset commit metadata string in Kafka (that nobody ever used).

“Just use virtual threads”

Java 21 brought virtual threads, and people have been (rightly) excited about them for IO-bound workloads. You can have millions of them, and they don’t block OS threads on IO. So you can spin up a virtual thread per partition or per key. Well..

Virtual threads solve the concurrency cost problem, and avoids blocking an OS thread on a synchronous HTTP call. They do not solve the ordering and offset-commit problem on their own.

Spawning virtual thread per partition barely improved on the vanilla consumer, it would achieve the same as with a regular thread pool. If you spawn virtual thread per key on the other hand, you need:

• A way to track which keys are in flight (so you don’t dispatch two threads for the same key)
• A completed-offset tracker — including that 4 KB metadata-encoding dance from the previous section, or your restarts reprocess or lose records
• Backpressure
• Rebalance handling
• Bounded memory

It’s exactly what Parallel Consumer is doing. There’s a new library called kpipe (https://github.com/eschizoid/kpipe) that wraps virtual threads on top of the vanilla Kafka consumer and gives you some of this. I haven't run it yet.

TLDR: This should be in Apache Kafka

The Kafka consumer should ship with key parallelism out of the box. Partition parallelism is a terrible default when you work with event-driven architecture and AI agents payloads. There should be a KIP for first-class parallel processing in org.apache.kafka.clients.consumer:

If you’re wondering whether someone already proposed this: sort of, and the story is telling. This is not a new need, I discover there is even an old KIP-attempt from 2019 “KIP-X: a cooperative consumer processing semantic”.

So what should I do tomorrow?

If you’re running event-driven workloads with synchronous processing — billing, payments, enrichment, anything where a single record takes more than a few milliseconds and you’re not using Parallel Consumer (or an equivalent), you’re almost certainly wasting partitions or time.

Remember this:

Another path: Client-specific virtual partitioning

What if the platform team could help without changing the application code? With a Kafka proxy, a slow consumer does not have to see Kafka exactly as it is.

For one specific app, the proxy could expose more [virtual] partitions than the topic really has, then map them back to the real partitions behind the scenes. The app gets more horizontal scaling room. The platform team keeps control. No library migration. No app rewrite.

When you sit in the protocol path, you can bend the Kafka protocol to add the missing abstractions Kafka never gave you.

Conduktor is a Kafka Operations PLatform providing Console and Gateway (Kafka proxy) for streaming data infra.