Kafka I/O Timeout

Metrics ingestion in my Kubernetes cluster started failing in a way that wasn't immediately obvious from the outside. Dashboards went stale or showed data arriving with extreme latency. The only consistent signal pointing to a problem was intermittent Kafka I/O timeout errors and high p99 latency, with everything else in the stack appearing deceptively healthy.

Triaging

The first concrete signal was Kafka I/O timeouts appearing across multiple components, specifically both the Mimir ingesters and distributors. This immediately ruled out a single workload or deployment issue, since it was affecting both the write and read paths on different sets of pods.

Given the symmetry across roles and nodes, the working assumption shifted toward Kafka itself being the common point of failure rather than anything in the consumers. What made this more suspicious was that the issue appeared “sticky”: it consistently followed the same pods with the highest ingestion lag. Even when those pods were rescheduled onto different nodes, the problem persisted. The only reliable way to clear it was manually advancing the consumer offsets to the most current position, after which the affected pods would recover.

The Smoking Gun

The first real clue came from the Mimir distributor p99 latency, which consistently reported around 1.67 minutes. That number was oddly specific and didn't match anything in the system's expected behavior or configured timeouts.

Breaking it down, 1.67 minutes maps almost exactly to ~100 seconds. One possible way to arrive at 100 seconds is a pattern of ten sequential 10-second Kafka I/O timeouts, rather than a single long delay. Not-so-coincidentally, that's exactly how the Mimir distributor and ingest storage pipeline is configured. That lined up with Kafka itself showing relatively low CPU usage (around ~300m requested vs 900m limit per broker), which initially made it look underutilized rather than overloaded.

Taken together, the symptoms at first pointed towards stalled network connections: long-lived connections that weren't failing outright, but were instead stalling and repeatedly hitting the same timeout boundary until they eventually surfaced as latency spikes upstream.

Root Cause

The underlying issue turned out to be CPU throttling driven by Completely Fair Scheduler (CFS) behavior, rather than anything Kafka-specific.

The mismatch came from how CPU time was being allocated versus how system behavior was being observed. Metrics were being collected at a 15-second interval, while the default CFS scheduling window operates at roughly 100ms granularity. That gap matters: short-lived bursts inside the 100ms window were effectively invisible at the metrics layer, even though they were dominating actual CPU scheduling behavior.

What this resulted in was a consistent pattern where Kafka would consume a large portion of available CPU early in each CFS window, then get throttled for the remainder of that window. This cycle repeated continuously. From the outside, it looked like Kafka was underutilizing CPU overall, but in reality it was repeatedly hitting short bursts of contention followed by enforced starvation—leading to stalled I/O progression and the cascading timeout behavior seen upstream.

Remediation

The fix ended up being straightforward: the CPU limit on the Kafka brokers was removed entirely.

There was minimal risk in doing this because the Kafka nodes were dedicated and already had substantial CPU headroom. The original limit was effectively introducing artificial throttling rather than providing any meaningful protection against contention.

After removing the limit, the throttling behavior disappeared and the downstream symptoms stopped immediately—Kafka I/O timeouts cleared, and Mimir ingestion latency returned to normal.

An expected side effect was a change in resource profile: Kafka's memory usage increased, while reported CPU usage remained largely unchanged. This aligned with removing the periodic CPU starvation pattern that had previously been shaping its runtime behavior.