GitHub - dcminter/kafkaesque: A mock Kafka server presenting wire-compatible APIs

A library for mocking Apache Kafka dependencies in a realistic way.

Kafkaesque is compatible with the Kafka TCP wire-protocol but without the startup overhead required to launch the real Kafka brokers. You need to provide your preferred version of kafka-clients as a dependency when using Kafkaesque. Kafkaesque requires Java 11 or later.

Status & Performance

I'd call this a "potentially useful beta" - give it a whirl if you think it might be handy!

A very quick and dirty benchmark (using the kafkaesque-speed-test repository) gives the following performance numbers:

This was run on underwhelming (2017 edition) hardware; the relative performance of the underlying tools is the interesting part. Obviously at the end of the day Kafkaesque isn't Kafka so there's stuff it's not doing that the others must, but that's kind of the point!

Why not just use real Kafka?

While running Kafka itself (perhaps within TestContainers) is a perfectly reasonable approach, it does have some drawbacks - depending on how you configure and launch it, it can be slow, perhaps taking multiple seconds to start up in a naiive configuration. If you're currently using Kafka in your integration tests and have no problems, then Kafkaesque is probably not the tool for you.

If, however, you're finding your Kafka tests are very slow (particularly if they launch large numbers of Kafka instances during the test lifecycle), or you want more control over the exact behaviours you're testing for, then Kafkaesque might be a good fit. It also might work for you if running Kafka inside testcontainers creates a dependency on Docker that would otherwise be unnecessary.

Note that if your tests are very slow because you're inserting sleep statements into otherwise fragile tests of asynchronous behaviour, then you might alternatively/additionally want to investigate the excellent Awaitility library. Also, if you need 100% guaranteed compatibility with real Kafka in your integration tests, you should stick with real Kafka - Kafkaesque cannot (and doesn't try to) be 100% compatible in every way.

Examples

Kafkaesque provides direct support for JUnit 4 and JUnit 5. The following examples assume an existing application under test that consumes from an orders topic and then publishes a confirmation to a notifications topic.

JUnit 5 (Jupiter)

Here's how you might write the test for JUnit 5 using Kafkaesque:

@Kafkaesque(topics = {
    @KafkaesqueTopic(name = "orders"),
    @KafkaesqueTopic(name = "notifications")
})
class OrderNotificationServiceTest {

    @Test
    void shouldSendNotificationWhenOrderIsPlaced(
            final KafkaesqueServer kafkaesque,
            @KafkaesqueProducer final KafkaProducer<String, String> producer) throws Exception {

        // Start the application under test, pointed at our mock Kafka
        var application = new OrderNotificationService(kafkaesque.getBootstrapServers());
        application.start();

        // Simulate an incoming order
        producer.send(new ProducerRecord<>("orders", "order-123",
                """
                { "customer": "Alice", "item": "Kafka In Action", "quantity": 1}
                """)).get();

        // Verify that the service produced a notification in response
        await().atMost(Duration.ofSeconds(5)).untilAsserted(() -> {
            var notifications = kafkaesque.getRecordsByTopic("notifications");
            assertThat(notifications).hasSize(1);
            assertThat(notifications.get(0).key()).isEqualTo("order-123");
            assertThat(notifications.get(0).value()).contains("Alice");
        });

        application.stop();
    }
}

The @Kafkaesque annotation spins up an in-process mock that speaks the real Kafka wire protocol
The topics and producer are created via Kafkaesque's annotations
Kafkaesque exposes a getRecordsByTopic(...) method on the server that lets you inspect what was published without wiring up a consumer
The application under test uses standard Kafka clients and has no knowledge of Kafkaesque

JUnit 4

The same scenario using JUnit 4's @ClassRule:

public class OrderNotificationServiceTest {

    @ClassRule
    public static KafkaesqueRule kafkaesqueRule = KafkaesqueRule.builder()
            .topics(
                    new TopicDefinition("orders"),
                    new TopicDefinition("notifications"))
            .build();

    @Test
    public void shouldSendNotificationWhenOrderIsPlaced() throws Exception {
        var application = new OrderNotificationService(kafkaesqueRule.getBootstrapServers());
        application.start();

        KafkaProducer<String, String> producer = kafkaesqueRule.createProducer();
        producer.send(new ProducerRecord<>("orders", "order-123",
                "{ \"customer\": \"Alice\", \"item\": \"Kafka In Action\", \"quantity\": 1}")).get();

        await().atMost(Duration.ofSeconds(5)).untilAsserted(() -> {
            var notifications = kafkaesqueRule.getServer().getRecordsByTopic("notifications");
            assertThat(notifications).hasSize(1);
            assertThat(notifications.get(0).key()).isEqualTo("order-123");
            assertThat(notifications.get(0).value()).contains("Alice");
        });

        application.stop();
    }
}

KafkaesqueRule is a JUnit 4 TestRule and can be used as an @Rule (per-method) or @ClassRule (per-class)
Topics can be configured via a builder pattern on the KafkaesqueRule
Factory methods (createProducer(), createConsumer(...)) are provided to make it simpler to create clients connected to the mock server

Standalone Docker Container

Kafkaesque can also run as a standalone service via Docker, useful during local development or in CI pipelines. A pre-built image is published to GitHub Container Registry on each release:

docker pull ghcr.io/dcminter/kafkaesque:latest
docker run -p 9092:9092 ghcr.io/dcminter/kafkaesque:latest

Your Kafka clients can then connect to localhost:9092. See the standalone guide for configuration options and Docker Compose examples.

Features

Support for versions 1.x through 4.x of the Apache Kafka client (kafka-clients)
Produce and fetch using standard Kafka clients
Multi-partition topics with configurable partition counts
Transactions (including READ_COMMITTED / READ_UNCOMMITTED)
Consumer groups with partition rebalancing
JUnit 5 (@Kafkaesque server instantiation per-class or per-method, @KafkaesqueTopic topic creation, @KafkaesqueProducer, @KafkaesqueConsumer injection)
JUnit 4 (KafkaesqueRule usable as @Rule or @ClassRule, builder for topic configuration, factory method for producer/consumer creation)
Direct record inspection (no consumer required)
Admin client operations
Log compaction / retention
Auto-topic-creation (optional)
Event listeners (record published, topic created, transaction completed)
Record headers
Server-side compression (gzip, snappy, lz4, zstd) configurable per topic
Server-side idempotency for producers (dedupe on retry, per-partition sequence tracking)
Offset commit and reset (earliest, latest)
Standalone executable Jar or Docker container

Installation

Kafkaesque is published to Maven Central. Add the dependency for your test framework along with your own version of kafka-clients - to avoid incompatibilities with your own test suite and application, Kafkaesque does not bring one in transitively:

JUnit 5 (Jupiter):

<dependency>
    <groupId>eu.kafkaesque</groupId>
    <artifactId>kafkaesque-junit5</artifactId>
    <version>2.0.0</version>
    <scope>test</scope>
</dependency>
<!-- Bring your own Kafka client (1.x through 4.x) -->
<dependency>
    <groupId>org.apache.kafka</groupId>
    <artifactId>kafka-clients</artifactId>
    <version>${your.kafka.version}</version>
    <scope>test</scope>
</dependency>

JUnit 4:

<dependency>
    <groupId>eu.kafkaesque</groupId>
    <artifactId>kafkaesque-junit4</artifactId>
    <version>2.0.0</version>
    <scope>test</scope>
</dependency>
<!-- Bring your own Kafka client (1.x through 4.x) -->
<dependency>
    <groupId>org.apache.kafka</groupId>
    <artifactId>kafka-clients</artifactId>
    <version>${your.kafka.version}</version>
    <scope>test</scope>
</dependency>

Or use the BOM for dependency management:

<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>eu.kafkaesque</groupId>
            <artifactId>kafkaesque-bom</artifactId>
            <version>2.0.0</version>
            <type>pom</type>
            <scope>import</scope>
        </dependency>
    </dependencies>
</dependencyManagement>

<!-- You still need to add kafka-clients explicitly -->
<dependencies>
    <dependency>
        <groupId>org.apache.kafka</groupId>
        <artifactId>kafka-clients</artifactId>
        <version>${your.kafka.version}</version>
        <scope>test</scope>
    </dependency>
</dependencies>

Building and testing

The build tool is Maven and we're using Maven Wrapper so to build and run the test suite:

The build is gated on both Checkstyle and SpotBugs: style violations fail the build at the validate phase, and SpotBugs static analysis findings fail the build at the verify phase. SpotBugs runs at effort Max; the threshold is Medium for the JUnit-support and standalone modules and Low for kafkaesque-core, where the wire-protocol implementation lives. The fb-contrib detector plugin is enabled to add concurrency, NIO, and methodology checks beyond the stock SpotBugs detectors; analysis also runs over the integration-test harness in kafkaesque-it.

JSR-305 nullness annotations (@ParametersAreNonnullByDefault and @ReturnValuesAreNonnullByDefault from spotbugs-annotations) are applied at the package level in kafkaesque-core so that interprocedural null analysis flows through the parsing pipeline; opt out per field, parameter, or return with @edu.umd.cs.findbugs.annotations.Nullable.

Lombok-generated code is excluded from SpotBugs analysis via spotbugs-exclude.xml; hand-written code in Lombok-annotated classes remains in scope. Genuine false positives may be silenced with a tightly scoped @edu.umd.cs.findbugs.annotations.SuppressFBWarnings carrying a non-trivial justification.

To test against a different kafka-clients version (see the multi-version guide):

$ ./mvnw clean verify -Dkafka.clients.test.version=2.8.2 -Dkafka.api.level=1

Or to install into your local artifact repository:

To build the Docker container and run it locally from source:

docker build -t kafkaesque .
docker run -p 9092:9092 kafkaesque

Further documentation

See the JUnit 5 guide for full details of the JUnit 5 (Jupiter) annotations and extension.
See the JUnit 4 guide for full details of the JUnit 4 rule, builder, and Vintage engine compatibility.
See the standalone guide for running Kafkaesque as a Docker container or executable JAR.
See the listener documentation for details of how to get various callbacks without using Kafka client libraries.
See the event storage summary for details of the internal representation of events etc.
See the multiversion strategy summary for details of how Kafkaesque supports multiple kafka-clients versions while still using that library internally.
See the future directions documentation for a sketch of features I plan to add to Kafkaesque.
See BUGS.md for known issues and contract inconsistencies that are tracked for future fixes.

License & Development

The software is licensed under the Apache License, Version 2.0

This software is designed to support projects making extensive use of Apache Kafka. It embeds (shaded) Apache Kafka libraries internally for its wire-protocol types, and it therefore makes sense to release it under the same license.

Random note on language

I bounce a bit between saying "I" and "We" in this documentation. Some of that is habit from the day job where "we" is usually the most accurate, some of that is because I'm often working with the cats on my desk, and some of it is because (see next section) Claude was a contributor to this project.

AI Declaration

Large parts of this software were developed using Claude Code