One fill, two deliveries, one ledger entry

To count a fill once, the consumer needs to remember that it has already applied it. I put that processed-fill marker and the holding update in one SQLite transaction, then killed the consumer on either side of the commit. Both restarts recovered to one unit. Committing those writes separately left either zero or two.

These are recorded runs with a real local Kafka broker, process kills and database writes. The fill, account and instrument are constructed.

Four crash points, one expected holding

The difference is what the replacement can infer from finding the marker. If the marker and holding commit together, “already processed” means the fill actually changed the holding. Separating those commits lets one fact survive without the other.

Why Kafka delivered the fill again

The database and Kafka recorded different things. SQLite held the position and processed-fill identities. Kafka held the consumer group’s progress through the records it was reading. Finishing the database write did not update that progress.

Kafka stores records in topic partitions. An offset identifies a record’s position within one partition; a group’s committed offset is where its next read should start. Here, F1 was at offset 0. Committing offset 1 would move the group past it.

In each crash case, the consumer died before making that Kafka commit. The replacement joined the same group and received offset 0 again. This is the at-least-once processing window: doing the work before committing consumption progress allows the work to be delivered again after a crash. Kafka’s delivery semantics describe this ordering.

Every replacement eventually committed offset 1, including the two that left the wrong holding. Advancing the consumption offset showed progress through Kafka; the database’s commit boundary determined whether the ledger was correct.

Keeping the marker and the effect together

The working variant used one SQLite transaction for the identity check, the processed-fill marker and the position update. The marker’s primary key was (source, account, fill_id). Kafka’s offset was not part of that identity.

For a fill not already recorded, the relevant sequence was:

BEGIN IMMEDIATE
  check the fill identity
  insert its processed marker
  add its quantity to the position
COMMIT

commit the Kafka offset

BEGIN IMMEDIATE acquires SQLite’s write transaction before the check. The identity lookup and both writes remain inside it. If the identity already exists, the consumer checks that the payload matches and ends the transaction without increasing the position. A different payload under the same identity raises an error instead of being silently discarded. SQLite’s transaction documentation describes the write-transaction behavior used here.

The transaction leaves two useful restart states: neither write committed, so apply F1; or both committed, so recognize it and skip the update. In the before-commit run, process death rolled back both writes. In the after-commit run above, both survived. The replacement did not need to infer whether the previous process had reached the end of its function.

A fill’s identity can cross Kafka offsets

Redelivery of offset 0 is only one way to see F1 twice. An upstream application can also submit the same business event in two separate producer calls.

The fifth case did exactly that with Kafka producer idempotence enabled. Both sends succeeded: one at offset 0, another at offset 1. The consumer applied the first, recognized the second by its source, account and fill ID, and finished with holding 1 and committed offset 2.

Producer idempotence protects against duplicates arising from the producer’s retries. It does not interpret two intentional application sends as the same trade because their payloads happen to match. The result makes the distinction concrete: these were two Kafka records describing one fill. A deduplication key based only on topic, partition and offset would give them different identities.

Kafka also supports transactional processing across Kafka records and consumption offsets. Extending a guarantee to an external destination requires that destination’s cooperation; enabling Kafka transactions alone does not include an arbitrary SQLite write. Those scopes are described in the same Kafka design account. This experiment uses ordinary explicit consumer commits and a separate database transaction.

The business identity has a lifetime, too. This experiment retains its markers. Deleting F1’s marker while F1 can still be replayed would remove the evidence the consumer uses to skip it. Deciding when that evidence can be retired belongs with the replay policy, including intentional reprocessing from older records.

Inspect or reproduce the runs

The reproduction bundle includes the executed source, process receipts and database exports. The recorded results keep the offsets and holdings behind the figures on this page.

The experiment grew out of the trading replay’s Read F1 again action. That browser example illustrates the unchanged holding; these runs examine the consumer-crash windows behind that behavior. They cover selected local cases, not production reliability.