Skip to content

docs/ops: fix Prometheus metrics scrape ports - #242

Open
keoyle52 wants to merge 1 commit into
circlefin:mainfrom
keoyle52:fix/51-prometheus-metrics-ports
Open

docs/ops: fix Prometheus metrics scrape ports#242
keoyle52 wants to merge 1 commit into
circlefin:mainfrom
keoyle52:fix/51-prometheus-metrics-ports

Conversation

@keoyle52

@keoyle52 keoyle52 commented Aug 8, 2026

Copy link
Copy Markdown

Summary

Fix Prometheus scrape target ports in deployments/monitoring/config-prometheus/prometheus.yml to match Quake localdev metrics port offsets.

Issue

Related to #51 (addresses only the Prometheus scrape-port portion — issue #51 bundles several other unrelated problems that remain unfixed).

Change

Corrected execution scrape target ports (9001, 9101, 9201, 9301, 9401) and consensus scrape target ports (29000, 29100, 29200, 29300, 29400) in deployments/monitoring/config-prometheus/prometheus.yml to align with the port allocation logic in crates/quake/src/node.rs (RETH_METRICS_BASE_PORT 9001 + node_index * 100; APP_METRICS_BASE_PORT 29000 + node_index * 100).

Verification

  • make lint: PASS
  • make build: PASS
  • make test-unit: PASS (1616/1619 unit tests passed)
  • Runtime Verification: Validated by re-deriving the port formula from crates/quake/src/node.rs, not by an actual running Prometheus instance (Docker runtime not available in local session environment).

Known pre-existing issues (unrelated to this change)

3 arc-test-integration multi-node tests fail locally with timeouts, believed environment-related (see brief §3); unrelated to this change; not cross-checked against upstream CI.

@keoyle52
keoyle52 force-pushed the fix/51-prometheus-metrics-ports branch from ac03c3d to c790322 Compare August 8, 2026 19:29
@osr21

osr21 commented Aug 8, 2026

Copy link
Copy Markdown

Verified this against main (90d71dc). The execution port corrections are right, but the consensus changes are based on a wrong stride and would break consensus scraping. Details below.

Execution — correct ✅

ExecutionContainer::new_local computes port_offset = node_index * 100 (crates/quake/src/node.rs:344) and sets metrics_port = RETH_METRICS_BASE_PORT + port_offset (9001 base). So 9001 / 9101 / 9201 / 9301 / 9401 is right, and the diff matches.

Consensus — incorrect ❌

Consensus does not use a * 100 stride. ConsensusContainer::new_local(node, subnet_indexes, node_index, port_offset) takes port_offset from its caller, and the caller passes index, not index * 100:

// crates/quake/src/node.rs:442
consensus: ConsensusContainer::new_local(name, subnet_indexes, index, index),

So metrics_port = APP_METRICS_BASE_PORT + port_offset = 29000 + index (node.rs:248), i.e. 29000 / 29001 / 29002 / 29003 / 29004.

That's confirmed downstream by how the ports are actually published to the host — the compose template maps the host port straight onto the fixed container port:

# crates/quake/templates/local/compose.yaml.hbs:139
- "{{consensus.metrics_port}}:29000"   # host = consensus.metrics_port = 29000 + index
# :62
- "{{execution.metrics_port}}:9001"    # host = execution.metrics_port = 9001 + index*100

Prometheus scrapes those same host ports (crates/quake/templates/prometheus.yml.hbs renders host.docker.internal:{{consensus.metrics_port}}).

Net: the consensus targets were already correct at 29000 / 29001 / 29002 / 29003 / 29004. Changing them to 29100 / 29200 / 29300 / 29400 points Prometheus at ports nothing is published on, so nodes 1–4 consensus metrics would silently stop being scraped. Recommend dropping the four consensus edits and keeping only the execution ones.

Side note (optional)

deployments/monitoring/config-prometheus/prometheus.yml is a hand-maintained duplicate of the generator at crates/quake/templates/prometheus.yml.hbs, which derives both port sets from metrics_port and can't drift. If the intent is a stable localdev config, it might be worth generating this file from the template (or referencing it) rather than editing the static copy, to avoid exactly this kind of divergence.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants