How We Built Automated Capacity Testing for Kafka Consumers

Kafka makes it easy to distribute work across consumer instances. It is much harder to prove, safely and repeatedly, how those instances behave when the distribution changes and one of them has to carry more than its usual share.

For teams that run Kafka at scale, this is a practical reliability question: how much load can a consumer instance actually handle? We had automated capacity testing for HTTP services, but Kafka consumers were still tested with manual drills. Those drills could tell us something, but they were disruptive, difficult to reproduce, and risky precisely when the system was close to its limit.

We wanted a controlled way to answer three questions:

• What is the maximum sustainable throughput of a consumer instance?
• If an instance or failure domain disappears, can the remaining consumers absorb the reassigned work?
• Are we overprovisioning resources because we do not know the real limit?

The result was an extension to our capacity-testing platform that turns Kafka partition assignment into a safe, measurable load-control mechanism.

Why HTTP capacity testing did not translate

Our existing platform was designed for request-response services behind a load balancer. A scheduled test selects one instance, routes an increasing share of traffic to it, runs health checks after each step, and records the highest ratio the instance can sustain. After the test, traffic returns to its normal distribution and the result is reported to the service owner.

Kafka has no equivalent traffic knob. Consumers pull records, and the unit of parallelism is the partition. Within a consumer group, each partition is owned by one consumer at a time. If a topic has 12 partitions and four equally loaded instances, each instance owns about three. When one instance disappears, a rebalance gives the survivors more partitions — and the extra work arrives as a step change, not as a smooth increase from a load balancer.

The key translation: for an HTTP service, increase load by changing the traffic ratio. For a Kafka consumer, increase load by changing the partition ratio.

Before automation, teams simulated this by stopping consumer instances or removing an entire failure domain and watching the remaining instances. The method reproduced the right failure mode, but it did so by creating a real failure. There was no controlled ramp, no consistent stopping rule, and no automatic recovery if the test controller failed.

Turning partition assignment into load control

We added a Kafka-specific layer between the generic test scheduler and the consumer application. The architecture has three responsibilities, kept deliberately separate:

• Scheduler : decides when to run, which instance to target, and how far to increase the requested load. It records the last healthy result and handles reporting.
• Kafka capacity API : translates an abstract load ratio into concrete partition assignments, coordinates scale and reset commands, and aggregates health signals.
• Client library : runs with the consumer, exposes the hooks needed to inspect and change assignment, and provides a custom partition assignor.

The client library is the important Kafka-specific piece. During normal operation, its assignor distributes partitions conventionally. During a test, the control plane gives one target instance a higher priority, so that instance receives more partitions than its peers. This reproduces the load concentration caused by a failure without killing a healthy consumer.

A capacity test, step by step

The scheduler still thinks in ratios, which keeps Kafka details out of the generic platform. The Kafka layer converts the requested ratio into the nearest assignment that the topic can actually support. A typical run looks like this:

1. Read the current assignment and establish a healthy baseline.
2. Select one target instance and request the next load step.
3. Translate the requested ratio into one or more additional partitions and trigger a rebalance.
4. Wait for the assignment to settle, then evaluate the composite health check.
5. Repeat while the target is healthy and the configured maximum has not been reached.
6. Record the last healthy assignment and restore the original strategy and partition layout.

Only one target is tested at a time. A Kafka rebalance affects the whole consumer group, so sequential execution keeps the result attributable and avoids overlapping experiments.

Health is more than a 200 OK

Moving partitions is easy; deciding whether the consumer is still safe is the real test. After each step, the Kafka layer combines several signals:

• Liveness, Is the target instance still responding?
• Resource headroom, Are CPU and memory below service-specific safety limits?
• Consumer lag , Does the configured percentile of record age remain below the service-defined threshold?
• Throughput and errors, where available , Does processing rise with the added partitions, and does the error rate remain acceptable?

A test step passes only when all required checks pass. If any signal becomes unhealthy, the test stops and the last successful step becomes the measured capacity. Importantly, safe CPU and memory thresholds are configured with headroom; reaching 100 percent is not a useful success condition.

Recovery is part of the design

A load test that depends on a healthy controller to undo its changes is unsafe. Every scale command therefore carries a time to live. The client library applies the temporary assignment only while that lease remains valid. A successful run ends with an explicit reset; if the scheduler or API disappears mid-test, the lease expires and the library returns to the normal assignment automatically.

This fail-safe changes the operational posture of the test. The system can increase load aggressively enough to find a limit, while still having a bounded path back to the steady state.

The partition math and its limits

Suppose a topic has six partitions and three consumer instances. Under an even assignment, each instance owns two partitions. A test can move the target to three partitions (50 percent of the topic), then four, five, and finally all six if the service remains healthy. Each successful step answers a concrete question about the load the instance can sustain.

The same mechanism also exposes an unavoidable limitation: the load curve is discrete. With P partitions, the smallest possible step is roughly 1/P of the topic. Five partitions mean 20-percent steps. Ten partitions mean 10-percent steps. The API can accept a requested ratio, but the effective ratio must be rounded to a real number of partitions, and the result should report that effective ratio rather than pretending the ramp was smoother than it was.

Single-partition topics are the extreme case. There is no gradual ramp: the test can observe only whether the target owns the partition or does not. If a finer capacity curve is required, the topic needs more partitions or the service needs a different controlled load mechanism, such as a producer-rate test in an isolated environment.

What we learned while making it repeatable

• Fast and slow health signals need different treatment. Local checks react quickly; lag and throughput telemetry may arrive later. The controller must observe long enough to catch degradation without leaving an unhealthy assignment in place.
• Rebalances are group-wide. Serialize tests for a consumer group and treat assignment settle time as part of the experiment.
• Onboarding is coordinated. Group members must agree on the assignment strategy, so the custom assignor needs a deliberate rollout and tested rollback.
• Capacity is not a single metric. CPU may remain healthy while memory, lag, processing time, or errors become the binding constraint.
• Preserve the evidence. Store requested and effective ratios, assignments, timestamps, health results, and throughput so the result remains explainable.

From a risky drill to continuous evidence

Once onboarded, a consumer can be tested on a schedule with consistent inputs and stopping rules. Results feed a status view, throughput graphs, and notifications, showing the highest partition load each instance sustained and whether it meets the resilience target.

One useful policy is N-1 resilience: if one of N instances disappears, can the remaining N-1 carry the workload without breaching health limits? Measured partition ratios make that policy testable and expose headroom for careful right-sizing.

Automation does not remove Kafka constraints: partition count still sets the resolution, rebalances affect the group, and health signals have observation delays. It changes the evidence ; the test becomes controlled, repeatable, automatically reversible, and comparable over time.

A more resilient future

We started with a gap between two worlds. Our capacity platform knew how to increase HTTP traffic, while Kafka expressed load through partition ownership. A small translation layer, a priority-aware client library, and a composite health check connected those worlds without turning the generic scheduler into a Kafka-specific system.

The larger lesson is that capacity testing does not have to imitate an HTTP load generator. It needs a safe control surface for the system being tested. For Kafka consumers, that surface is partition assignment. Once we could manipulate it deliberately and reset it automatically, we could replace guesswork with a capacity result that service owners can use.