Prometheus Internals: The Clever Compression Behind Time Series Storage

When we think about Prometheus, we often imagine it as “the metrics database” scraping, storing, and querying endless time series from our apps. But under the hood, it’s doing something surprisingly clever that makes it much more efficient than you’d expect.

The Core Insight

Prometheus assumes that most metric data looks boring and predictable, then exploits that fact ruthlessly.

This approach is inspired by a 2015 paper published by Facebook titled “Gorilla: A Fast, Scalable, In-Memory Time Series Database” (PDF).

Here’s the trick: Prometheus doesn’t actually store full timestamps or raw values for every single metric sample. Instead, it uses a smart compression technique based on deltas and XOR encoding.

The Problem with Naive Storage

Let’s say Prometheus scrapes a metric every 15 seconds. A naive approach would store each sample like this:

[timestamp: 64 bits] [value: 64 bits]

That’s 16 bytes per sample. For millions of time series scraped every few seconds, the storage cost explodes.

The Delta + XOR Magic

Prometheus takes advantage of two simple observations about metrics:

👉 Timestamps are Predictable

Scrapes usually happen at fixed intervals (e.g., every 15s). Instead of storing the full timestamp, Prometheus stores the difference (delta) between consecutive timestamps.

Example:
For timestamps like 1694000000, 1694000015, 1694000030
Store: 1694000000 (first one raw), +15, +15

👉 Values Don’t Change Drastically

Prometheus stores the first value as is, but every subsequent value is XOR’d with the previous one. If values are close (like CPU usage moving from 0.51 to 0.52), the XOR result has lots of leading zeros. Those zeros compress very well.

The Payoff

Because of this scheme, Prometheus can often store 2-3 bytes per sample instead of 16.

That’s why we can comfortably run Prometheus on a single node with millions of metrics without needing a giant cluster behind it.