Where latency builds up in a Kafka price pipeline

In Monafy, a price reaching the screen late does not by itself establish that Kafka is slow. The event is stored, waits for publication, travels through Kafka and is applied by a consumer. Any of those steps can add waiting time to the result the reader sees.

I started by breaking the single idea of latency into separate measurements. Upstream publication delay, time inside our own services and delivery to the browser have different boundaries. Establishing those boundaries made it clearer which part of the delay our changes could actually address.

Defining the event at both ends of a measurement

I matched timestamps for the same price across the pipeline. Clock precision matters here: a record with whole-second timestamps cannot reliably show a change smaller than a second. The start and finish also need to describe the same event rather than two nearby updates for the same symbol.

I looked at distributions rather than relying on an average alone. A median describes the typical event, while higher percentiles expose prices that waited longer. Adding medians from individual steps does not produce the end-to-end median, so the complete path needs its own measurement.

Waking the publisher when work is ready

The first useful finding was a publisher that slept when no pending records were available. A price stored immediately after its check could wait until the next poll even though there was already work to do. This was a scheduling decision rather than the cost of moving a message through Kafka.

PostgreSQL NOTIFY became a wake-up signal for the publisher. The notification does not carry the durable price payload; storage and Kafka retain that responsibility. Periodic polling remains available when a notification is missed, so the stream does not depend entirely on a transient signal.

Transaction scope affects when data becomes visible

Waking earlier does not help if the stored rows are still inside an uncommitted transaction. The writer also updated an auxiliary table inside the batch transaction, delaying visibility of prices that had already been prepared for publication.

That auxiliary update did not need to share the same atomic boundary. Committing the price batch first and updating the other table separately let the publisher see the primary data earlier. The useful question was which work belonged in one transaction, before asking how to optimize an individual query.

Choosing batch waits for the workload

The consumer also waited after the first event arrived so that a batch could grow. That can make sense for large historical imports, but on a current-price stream the wait becomes part of the user-visible delay. One setting was serving workloads with different priorities.

We evaluated the live-price consumers separately from bulk-data consumers. Low-traffic conditions matter too: when a batch contains one event, waiting can dominate even if the processing itself is quick. Measuring only a busy session would miss that behavior.

Evaluating speed alongside recovery behavior

Publishing earlier should not weaken the order in which data becomes durable. I discuss the decision to store consumer progress with the data in the Kafka and PostgreSQL case study. Latency changes need to be evaluated around that correctness boundary.

Sending prices more frequently also has a system-wide cost. More events can mean more database work and more client updates. Before removing a wait, I examine what it currently achieves and which additional work the change creates along the rest of the pipeline.