fix(app): make builder registrations work against real beacon nodes

[varex83agent]

🧩 What was the bug?

Pluto returned HTTP 502 on every POST /eth/v1/validator/register_validator, so builder registrations never worked against a real beacon node:

ERROR validator api error status=502 upstream application builder domain failed
      source=Domain type not found: DOMAIN_APPLICATION_BUILDER

DOMAIN_APPLICATION_BUILDER comes from the builder spec, not the consensus spec, so beacon nodes do not serve it in /eth/v1/config/spec — I confirmed the local Lighthouse serves DOMAIN_APPLICATION_MASK but not this key. Charon never hits it because go-eth2-client silently injects a fallback (http/spec.go).

Validator clients re-register on a timer, so the failure became an unbounded retry loop. Measured on a 2+2 mixed Charon/Pluto cluster: 12,665 and 12,935 registration attempts per Pluto VC in 10 minutes, versus 0 for the Charon VCs (3.63M vs 256 over the preceding 47h).

Two further defects sat behind it:

The duty path could never work in a mixed cluster. Charon v1.7 removed the VC-push architecture — SubmitValidatorRegistrations is a no-op and the scheduler submits pre-aggregated registrations straight to the beacon node once per epoch. Pluto still verified the VC's partial signature and fanned it out as a builder_registration duty requiring threshold aggregation. With a 3-of-4 threshold and only the two Pluto nodes contributing, the duty stalls forever — silently, since registration duties are deliberately given no deadline.

The failure was invisible in metrics. request_total, request_error_total and vc_user_agent were defined in validatorapi::metrics but incremented nowhere, and the router had no endpoint-name concept, so request_latency_seconds only ever emitted endpoint="proxy". A Pluto node exported zero core_validatorapi_request_* series where Charon exports 98.

---

🔧 What's been fixed?

1. Spec-default domain fallbacks (crates/eth2api/src/extensions.rs)

Inject the same three fallbacks go-eth2-client does (DOMAIN_APPLICATION_MASK, DOMAIN_BLS_TO_EXECUTION_CHANGE, DOMAIN_APPLICATION_BUILDER) inside fetch_spec_data — the single cached point where spec JSON enters the process, so it fixes resolve_domain_type, fetch_domain_type and signing::get_domain at once. Keys the node does serve always win.

2. Validator API metrics at Charon parity (crates/core/src/validatorapi/)

Every route carries Charon's endpoint label. One middleware records latency, request_total, vc_user_agent and any non-2xx, following the ordering in Charon's wrap: the content type is normalised first, and an unrecognised type is rejected 415 and deliberately not counted in request_total. Also bounds the proxy path label — it was a naive /→_ replace, so /eth/v2/beacon/blocks/0x<root> minted a new histogram series per block root.

3. Port BuilderRegistrationService (crates/app/src/builderregistration.rs)

Serves the cluster lock's group-signed registrations, optionally overridden by an operator-managed JSON file (watched via notify) and by the Obol API's aggregated partial signatures. Adds the two /fee_recipient endpoints, and two slot subscribers: the per-epoch registration submission, and prepare_beacon_proposer — which Pluto never sent at all, so the beacon node was building local blocks against its own default fee recipient.

Removes the unreachable duty path, including the recaster (already dead code in Pluto), and drops futures from pluto-core, which recast.rs was the last user of.

---

🧠 Context worth flagging for review

Why registrations skip consensus entirely. The lock already contains a fully-aggregated, group-signed registration per validator (DistValidator::eth2_registration()), so there is nothing to agree on — every node holds the same signed message and submits it to its own beacon node. This is what makes the mixed-cluster deadlock go away: nothing needs quorum any more.

DutyType::BuilderRegistration is kept even though it is now unreachable — it is a wire-format enum value, and Charon keeps core.DutyBuilderRegistration too.

Deliberate deviations from Charon, all commented in code:

• vise::Family exposes no removal API, so Charon's ResetGaugeVec single-series semantics for vc_user_agent can't be reproduced. The previous label is set to 0 instead; the stale series remains at zero.
• Charon labels proxy errors with the raw request path, which is unbounded. This uses the constant proxy for the counter and keeps the collapsed path only on the latency histogram.
• The service's run waits for cancellation even with nothing configured, because the node treats any long-lived task returning as a shutdown signal. Returning early stops the node — this was caught by the simnet test.

Test fixtures were masking the bug. The beaconmock default spec injected DOMAIN_APPLICATION_BUILDER even though the Holesky snapshot behind it does not contain the key. That override and three hand-written fixtures are removed, so the fallback — not the fixture — is what makes those tests pass.

Registrations use the submission client, not the scheduling one. Beacon nodes proxy register_validator to the builder relay and routinely take seconds; the local Lighthouse takes a flat 3s. The 2s general timeout aborted first and hid the beacon node's actual response, so this belongs under --beacon-node-submit-timeout like every other submission.

Four new CLI flags matching charon/cmd/run.go: --overrides-file, --publish-address, --publish-timeout, --fetch-feerecipient-updates (default false, so the default configuration makes no outbound calls). The last requires the second, as in Charon.

---

✅ Checklist

• Confirmed the bug no longer occurs
• Existing tests pass
• Added new test(s) for the bug
• The branch is named per repo convention (fix/<name>)

Gates: cargo +nightly fmt --all --check, cargo clippy --workspace --all-targets --all-features -- -D warnings, cargo test --workspace --all-features (2242 pass), cargo deny check, cargo machete.

Verified on a live 2+2 mixed cluster (node0/1 Charon v1.7.1, node2/3 Pluto, Lighthouse VCs, threshold 3-of-4, 256 validators):

	before	after
Registration attempts, Pluto VCs	~6,400 / 5 min each	0 / hour over 24h
502s on register_validator	every slot, forever	0
core_validatorapi_request_* series	0	named endpoints, Charon's labels
Failed duties (Pluto nodes)	—	0.0 and 1.1 / 4 min

---

🤖 Generated with Claude Code