Exactly-Once Delivery Is a Spectrum, Not a Checkbox: Part 1

“Exactly once” is one of the most overloaded phrases in data infrastructure.

• A source can say it emits every change once.
• Kafka can claim to support exactly-once semantics.
• Flink can say it checkpoints state exactly once.
• BigQuery and Snowflake can say they support exactly-once streaming writes.
• A lakehouse table can say commits are atomic.

All of those statements can be true, but your dashboard can still double-count revenue.

The reason is simple: exactly once is not one guarantee. It is a chain of guarantees. The chain crosses the source database, the capture connector, the transport layer, the stream processor, the destination connector, the destination storage engine, and finally the business operation you care about.

This series of articles provides a practical model for exploring exactly-once guarantees and how much failure they tolerate.

We will start with the first two links in the chain: source guarantees and transport guarantees. Then we will move into stream processing, where input progress and processing state have to recover together. From there, we will get to the place where correctness becomes truly visible: materialization into a table, file, index, lakehouse, warehouse, or external API.

The duplicate no one owns

Let’s kick off with a common pipeline:

Postgres -> CDC connector -> Kafka -> stream job -> warehouse

Now imagine a customer places an order:

INSERT INTO orders (id, customer_id, amount, status)
VALUES (123, 42, 100.00, 'paid');

The source database commits the transaction, then the CDC connector reads the write-ahead log, and Kafka accepts the record. On the other side, a worker reads it and writes it to a warehouse table.

Then this happens:

t1 worker reads order_id = 123
t2 worker writes order_id = 123 to destination
t3 destination commit succeeds
t4 worker crashes before saving its checkpoint
t5 worker restarts from old checkpoint
t6 worker reads order_id = 123 again
t7 worker writes order_id = 123 again

Do you see the issue? No component “lost” data or necessarily violated its own contract. But, the source replayed from a safe position and the worker retried after a crash. After all that, the destination accepted a valid write.

But the business effect happened twice, so if the destination table is append-only, you now have this:

order_id | amount | source_lsn
 - - - - -+ - - - - + - - - - - -
123 | 100.00 | 0/16B6C50
123 | 100.00 | 0/16B6C50

If a dashboard runs this:

SELECT SUM(amount) AS revenue
FROM orders;

the answer is obviously wrong.

This sneaky bug lives in the gap between delivery and effect.

• Delivery asks: did the event arrive?
• Effect asks: did the durable state change exactly once?

Those are very different questions.

Exactly-once is scoped

Every “exactly-once” claim has a scope.

Kafka’s idempotent producer prevents producer retries from creating duplicate log entries in a Kafka partition, and Kafka transactions can atomically send records to multiple partitions and topics. But Kafka’s own producer documentation also says application-level resends cannot be deduplicated by producer idempotence, and idempotence is guaranteed only within a producer session.

Postgres logical replication slots normally emit each change once, but after a crash a slot can return to an earlier LSN and resend recent changes. The Postgres documentation explicitly says logical decoding clients are responsible for avoiding bad effects from handling the same message more than once.

BigQuery Storage Write API supports exactly-once semantics through stream offsets, but that guarantee is scoped to messages with the same offset within a write stream when the client provides offsets. The default stream is at-least-once.

Snowpipe Streaming uses channels and offset tokens to support recovery without duplicate ingestion, but its docs also make clear that offset tokens are user-defined identifiers and the external system must use them to track progress correctly.

Those might seem like contradictions, but they are, in fact, scoped guarantees.

A useful pipeline model looks like this:

“Exactly once” can apply at any of those boundaries, but keep in mind that it rarely applies to all of them automatically.

The spectrum

Think of delivery guarantees as a spectrum.

A pipeline with at-least-once transport and a truly idempotent destination write can be more correct than a pipeline with Kafka transactions and a non-idempotent warehouse append.

The weakest link will define the actual guarantee.

Five guarantees hiding behind one phrase

When someone says “exactly once,” ask which of these five guarantees they mean.

1. Source guarantee: Can the source replay from a precise position?
2. Transport guarantee: Can the stream/broker commit records and offsets atomically?
3. Processing guarantee: Can state and input progress recover together?
4. Materialization guarantee: Can destination data and destination progress commit together?
5. Business effect guarantee: Is the operation safe to retry?

Most duplicate bugs happen because developers verify one layer and assume the rest.

• They check Kafka exactly-once.
• They forget the warehouse MERGE.
• They check Flink checkpointing.
• They forget the HTTP API side effect.
• They check CDC source offsets.
• They forget that SUM(amount) is not idempotent.

Source guarantees: a replay position is not a miracle

A source system usually gives you a position, but it does not give you end-to-end correctness.

The position just tells you that you can resume from here. It does not say that “every downstream effect is now exactly once”. Those are different contracts.

Postgres: LSNs and replication slots

Postgres logical decoding reads committed changes from the write-ahead log and exposes them through logical replication slots. A slot represents a stream of changes from a single database, persists independently of the client connection, and is crash-safe. In normal operation, a logical slot emits each change once. But Postgres persists a slot’s current position only at checkpoint time. After a crash, the slot can return to an earlier LSN, causing recent changes to be sent again.

That is a fairly honest guarantee.

Postgres is telling you a few things:

• “I can give you an ordered change stream.”
• “I can keep WAL needed by the slot.”
• “I can let you resume.”
• “But if I resend a recent message, your client must handle it.”

A CDC event from Postgres should carry source identity:

{
"source": "postgres",
"database": "app",
"schema": "public",
"table": "orders",
"primary_key": { "id": 123 },
"op": "insert",
"commit_lsn": "0/16B6C50",
"transaction_id": 742991,
"transaction_order": 3
}

A destination should not treat this merely as “an order row.” It should treat it as a specific source change.

For many CDC systems, a practical event identity is something like:

event_id =
source_cluster +
database +
schema +
table +
commit_position +
transaction_position +
row_position

The exact fields depend on the source connector, but the principle is stable: carry the source position all the way to the sink.

MySQL: GTIDs protect replication, not every downstream system

MySQL Global Transaction Identifiers are source transaction identifiers. A GTID is unique across a replication topology and has the form:

GTID = source_id:transaction_id

MySQL uses GTIDs to prevent the same transaction from being applied more than once on a replica. After a transaction with a given GTID has committed on a server, a later attempt to execute the same GTID is ignored by that server.

That is powerful inside MySQL replication.

But if you turn the transaction into a JSON event and send it to a warehouse, MySQL’s replica auto-skip logic no longer protects you because the warehouse does not know the GTID unless you write it there. It cannot skip a duplicate event unless the materialization logic uses the GTID, or some derived event identity, as part of its idempotency strategy.

The same event can be safe in one context and unsafe in another:

1. MySQL source -> MySQL replica: GTID auto-skip can prevent duplicate application.
2. MySQL source -> Kafka -> Snowflake: GTID is just metadata unless the sink uses it.
3. MySQL source -> append-only S3 files: GTID does not dedupe files unless the writer/table format records it.

The important learning here is that source identity must survive translation.

MongoDB: resume tokens are restart handles

MongoDB change streams are resumable with resume tokens. The token can be passed to resumeAfter or startAfter when opening a new cursor. MongoDB also helpfully warns that the same pipeline and options should be used when resuming, and the oplog must still contain enough history for the token or timestamp.

You might have guessed, that gives you a restart mechanism, but it does not automatically make every downstream write exactly once.

A robust MongoDB CDC event should carry at least:

{
"resume_token": "…",
"operationType": "update",
"clusterTime": "Timestamp(…)",
"ns": {
"db": "app",
"coll": "customers"
},
"documentKey": {
"_id": "…"
}
}

The resume token helps the capture process know where to restart and the document key helps the destination know which entity to update, but they are not the same identity.

This is an important distinction, because two updates to the same MongoDB document have the same document key but different event identities. If you dedupe only by document key, you may drop a real update. If you append both events without event identity, you may double-count downstream aggregates.

Debezium: at-least-once is the honest default

Debezium’s documentation is refreshingly direct: Debezium provides at-least-once delivery guarantees, which means no changes are missed, but a record may be delivered more than once. It also notes that exactly-once may be required in scenarios where duplicates are problematic.

This is not really a flaw. It is the default shape of reliable CDC.

1. A CDC connector that never retries can lose data.
2. A CDC connector that retries can duplicate data.
3. A CDC connector that retries and carries stable source offsets gives the downstream system enough information to be correct.

The destination still has to use that information.

Source rule

A source guarantee usually gives you three things:

1. Ordering
2. Replay
3. Position

It does not automatically give you:

1. Destination deduplication
2. Aggregate correctness
3. API idempotency
4. Cross-system atomicity

A replayable source is necessary for a robust system in all environments.

Transport guarantees: Kafka exactly-once is real, but bounded

Kafka is often where exactly-once discussions get confused. Make no mistake, Kafka’s exactly-once features are real, but they are also scoped. Let’s take a look at that deeper.

Kafka producer idempotence prevents producer retries from creating duplicate entries in the Kafka log. Kafka transactions allow messages to multiple partitions and topics to be committed atomically. With transactional producers, consumers also need to read only committed messages for end-to-end transactional visibility inside Kafka.

Confluent’s documentation explains the boundary clearly: Kafka supports exactly-once delivery in Kafka Streams and for transferring and processing data between Kafka topics. It also warns that claims of exactly-once semantics may not account for failures of producers or consumers outside the system.

That boundary is everything.

Kafka can make this loop exactly-once:

Kafka topic A -> process -> Kafka topic B

It cannot, by itself, make this external side effect exactly-once:

Kafka topic A -> process -> charge credit card

or this:

Kafka topic A -> process -> UPDATE warehouse table

or this:

Kafka topic A -> process -> call webhook

External systems need their own commit or idempotency protocol.

Kafka consume-transform-produce

A common exactly-once Kafka pattern looks like this:

producer.initTransactions();
while (true) {
    ConsumerRecords<K, V> records = consumer.poll(Duration.ofMillis(1000));
    producer.beginTransaction();
    for (ConsumerRecord<K, V> record : records) {
        Output output = transform(record);
        producer.send(new ProducerRecord<>("output-topic", output.key(), output.value()));
    }
    producer.sendOffsetsToTransaction(
        offsetsFor(records),
        consumer.groupMetadata()
    );
    producer.commitTransaction();
}

This works fine because output records and consumed offsets are committed as one Kafka transaction, but if the transaction aborts, the output records are not visible to read_committed consumers, and the consumer offset does not advance.

The atomic unit is:

input offsets + output Kafka records

Strong contract, but what happens if we change the code?

producer.initTransactions();
while (true) {
    ConsumerRecords<K, V> records = consumer.poll(Duration.ofMillis(1000));
    producer.beginTransaction();
    for (ConsumerRecord<K, V> record : records) {
        warehouse.insert(transform(record));   // external side effect
        producer.send(new ProducerRecord<>("audit-topic", record.key(), "done"));
    }
    producer.sendOffsetsToTransaction(
        offsetsFor(records),
        consumer.groupMetadata()
    );
    producer.commitTransaction();
}

The warehouse insert is not part of the Kafka transaction, meaning if warehouse.insert() succeeds and producer.commitTransaction() fails, the Kafka offset rolls back but the warehouse row remains. On retry, the same input record is processed again.

In the end, Kafka did its job, but the application did not establish an atomic contract with the warehouse.

Kafka Connect source exactly-once is also a contract

Kafka Connect added exactly-once support for source connectors in Kafka 3.3. To use it, a source connector must provide meaningful source offsets for each record and must be able to resume from the external system at the exact position corresponding to those offsets without dropping or duplicating messages.

The worker config exactly.once.source.support enables this by using transactions to write source records and their source offsets together, and by fencing old task generations.

That is important, but notice the shape of the guarantee:

external source -> Kafka topic

It helps source connectors write into Kafka exactly once, but it does not automatically solve:

Kafka topic -> external destination

For sink connectors, the hard problem remains the same: can the destination write and the consumed offset be committed together, or can the destination apply be made idempotent?

This conundrum appears in Kafka Connect’s connector development guide: sink tasks can store offset information in the destination system to provide exactly-once delivery.

And that is the materialization problem, which we’ll explore in a future chapter.

Transport rule

So Kafka’s exactly-once guarantees are not magic. It can protect Kafka records, Kafka offsets, Kafka transactions, and Kafka Streams state. It cannot automatically protect an external database, warehouse, object store, API, or search index.

So always keep this boundary in mind:

• Inside Kafka: transactions can compose records and offsets.
• Outside Kafka: the sink needs transactions, idempotency, or reconciliation.

In the next chapter

So far, we have looked at the first two boundaries: the source and the transport layer.

The source can give you a replayable position: an LSN, GTID, resume token, binlog offset, or Kafka offset. The transport layer can give you stronger delivery semantics: idempotent producers, transactions, committed offsets, and fencing.

But that still does not prove the pipeline is correct.

The next boundary is the stream processor.

This is where things get a little more subtle. A processor does not just move records. It also keeps state: counts, windows, joins, dedupe sets, latest versions, and intermediate results. If that state is not checkpointed with the input position, the pipeline can become wrong before the destination ever sees a write.

In the next chapter, we will look at processing guarantees: how state and input progress recover together, why Flink checkpoints are a useful model, and why exactly-once processing still does not mean exactly-once effects unless the sink participates in the same commit protocol.