In Monafy, one service publishes price events and other services consume them through Kafka. Applying an event updates more than a price: it also advances sequence information used by downstream parts of the product. The boundary between writing that data and marking a message as consumed therefore matters.
Inspecting the client library's error behavior showed that a handler failure could still be followed by a committed consumer position. We decided to store the authoritative position in PostgreSQL, in the same transaction as the data it describes.
The gap between applying data and recording progress
If a process stops after writing data but before recording progress, the message can arrive again. If progress advances before the data is safely stored, the application can miss the update. Downstream components using sequence numbers need that relationship to be explicit rather than inferred from process order.
We did not replace the entire Kafka client. We kept its consumption loop while taking control of error propagation and the point at which progress is recorded. A failed handler must not look like a successfully completed message.
Committing the position with the data
The application applies a batch and updates its consumer position inside one database transaction. Committing makes both durable; rolling back leaves both at their previous state. A restarted consumer reads its starting position from that application-owned record.
The Kafka position is still updated after the database commit so monitoring tools can observe it. That secondary update is useful operationally, but it is not the application's authority for where processing should resume.
This is a transaction boundary with a specific scope. It coordinates data and progress inside PostgreSQL. It does not automatically make unrelated external side effects exactly-once operations, and it should not be described as a universal delivery guarantee.
This simplified example illustrates the transaction boundary around the data and its recorded progress:
1DB::transaction(function () use ($batch, $position) {
2 applyBatch($batch);
3 savePosition($position);
4});A rejected row can hold up the batch
Atomic application has a deliberate cost. If the database rejects one row, the rest of that batch rolls back too. The consumer keeps returning to the same position until the cause is resolved. Continuing by silently skipping the row would defeat the correctness property the design was intended to establish.
That makes observability a separate requirement. A running process does not prove that the consumer is advancing. Changes in its recorded position and the distance from the incoming stream need to be monitored independently of basic process health.
Testing the restart boundary
The useful failure scenario interrupts processing and checks where a replacement consumer resumes. It examines repeated application, missing records and position advancement together rather than checking only the successful insert path.
Correct recovery and low latency are different concerns. I discuss where time is spent on the same pipeline in the article on measuring Kafka price-stream latency.