How Prometheus Scrapes and Indexes Metrics

The Prometheus Architecture

Prometheus is a time-series monitoring system engineered around a pull model. Unlike push-based metrics pipelines that require client nodes to continuously push statistics to a central database, Prometheus regularly connects to configured target endpoints via HTTP to scrape raw metric payloads.

graph TD
    Prom[Prometheus Scraper Engine] --> Target1[Target Node /metrics Endpoint]
    Prom --> Target2[Exporter /metrics Endpoint]
    Prom --> TSDB[TSDB Storage Block]
    TSDB --> WAL[Write-Ahead Log]
    TSDB --> MemChunks[In-Memory Chunks]
    MemChunks --> Block[On-Disk 2h Storage Blocks]

---

1. The Scraping Cycle (Pull Model)

At configured intervals (e.g. every 15 seconds), the Prometheus scraper engine executes the following loop:

1. Service Discovery: Prometheus resolves target endpoints dynamically using Kubernetes, AWS EC2 APIs, or static DNS profiles.
2. Scrape Execution: Sends an HTTP GET request to the target’s /metrics path.
3. Data Parsing: The target replies with plain-text lines conforming to the OpenMetrics specification:

   # HELP http_requests_total Total number of HTTP requests.
   # TYPE http_requests_total counter
   http_requests_total{method="post",handler="/login"} 1043 1709420000000

   Each line identifies the metric name, label dimensions, current float value, and epoch millisecond timestamp.

---

2. The TSDB Memory Pipeline

When metric data lands in Prometheus, it is routed immediately to the Time Series Database (TSDB) write pipeline:

• Write-Ahead Log (WAL): To prevent data loss from sudden system failures, raw metrics are appended immediately to a disk-based WAL.
• In-Memory Head Block: Chunks are kept in memory for up to two hours. To maximize memory efficiency, Prometheus uses Gorilla compression, which calculates double-delta differences between consecutive metric timestamps and values, shrinking the VRAM/RAM footprint of time-series samples by up to 10x.

---

3. On-Disk Block Compaction

Every two hours, the TSDB flushes the in-memory chunks to disk as immutable blocks containing:

1. Chunk Files: Directory containing actual time-series raw data chunks (10,000s of samples per chunk).
2. Metadata (meta.json): Stores block ranges, source statistics, and compaction versions.
3. The Index Block: Maps metric names and label keys directly to the physical byte coordinates inside the chunk files, allowing Prometheus to execute instant metric lookups without scanning entire database blocks.