Metrics Ingestion Performance Tuning
In a previous post, I described a metrics ingestion pipeline sustaining ~25k samples/sec and ~450k active series.
Since then, the observability footprint expanded significantly. The original configuration became a bottleneck under increased load, resulting in dropped samples and visibility gaps. This write-up covers the changes that increased capacity to ~700k active series at ~40k samples/sec without additional compute provisioning.
Dashboard
Mimir Tuning
Ingesters
The baseline deployment of Mimir uses three availability zones to support read quorum requirements. However, in an ingest-storage architecture backed by Kafka, this constraint becomes largely redundant. Kafka is already configured with min.insync.replicas=2, and Mimir explicitly documents that a two-zone ingester topology is more resilient against random ingester failures.
The first adjustment was therefore reducing the ingester topology from three zones to two, while retaining six total replicas. This preserved aggregate CPU allocation at the Kubernetes level, but effectively increased per-zone throughput capacity by ~33%, while improving fault tolerance against zone-level disruption.
Distributors
The primary change in the distributor layer was removal of CPU limits.
Historically, CPU caps were introduced as a mitigation for retry storms under elevated ingestion load, where Mimir's backpressure handling was insufficient. While this stabilized the cluster at the node level, it introduced artificial throttling and reduced distributor stability under sustained pressure.
Removing CPU limits restored predictable scheduling behavior and eliminated throttling-induced amplification effects during peak ingestion.
Kafka Tuning
The second area of optimization was the Kafka write path.
The Mimir distributor architecture maps one Kafka partition per shard per availability zone. Previously, with two shards per zone, the system operated with two active partitions.
After increasing shard count from 2 to 3 per zone, the active partition count also increased to 3.
This change improved broker-level load distribution. With three brokers, each broker can serve as the leader for a single partition, eliminating the prior concentration where multiple partitions frequently converged on the same“hot”brokers. This resulted in more balanced steady-state throughput characteristics and improved capacity predictability.
Additional gains came from JVM configuration tuning. Setting both -Xms and -Xmx to 4g ensured heap preallocation, reducing allocation churn and improving GC stability. Combined with a 7Gi memory request, this effectively guaranteed the runtime operated with a largely pre-warmed heap under steady load.
Alloy Tuning
The final area of optimization was Alloy's metric scraping and sending to the Mimir distributors.
Alloy uses sharding to write to Prometheus-compatible endpoints. In the case of using Mimir, it's simple: more shards means more distributor connections, which means more Kafka Connections.
Therefore, I opted to use the following queue configuration within the write endpoint:
queue_config {
capacity = 500000
max_samples_per_send = 10000
max_shards = 15
}
Massively increasing the queue capacity means allocating more RAM for each Alloy instance. Decreasing the number of shards puts less pressure on Kafka, trading sustained network throughput and CPU for lower spikes in CPU usage.
Results
The net outcome was a substantial increase in efficiency and headroom.
The system now sustains ~40k samples/sec with ~750k in-memory series (~67% increase) while consuming ~30% less CPU and ~40% less memory. Ingestion p99 latency stabilized below 2s, down from ~9s prior to the changes. On the other hand, WAL latency in Alloy increased from 0s to ~5s, which was expected given the decrease in available shards.
While still above the recommended 200ms p99 target, the trade-off remains acceptable given the cost profile (~$300 total infrastructure cost) and sustained 100% write acceptance rate.
The additional headroom also enabled ingestion of exemplars from Grafana Tempo. This instrumentation quickly isolated a bottleneck in util.ParseProtoReader, which intermittently exhibits high latency during buffer reads independent of configured buffer size.
Given the underlying infrastructure characteristics (Contabo-hosted environments with observed iowait and CPU steal variability), the root cause may lie in host-level contention rather than application behavior. That investigation remains open.
Backlinks