Skip to content

fix(network): reconcile eRPC overlay replacements - #815

Draft
bussyjd wants to merge 1 commit into
mainfrom
fix/erpc-overlay-replace
Draft

fix(network): reconcile eRPC overlay replacements#815
bussyjd wants to merge 1 commit into
mainfrom
fix/erpc-overlay-replace

Conversation

@bussyjd

@bussyjd bussyjd commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

Summary

  • treat the persisted eRPC operator overlay as desired state for overlay-owned networks and upstreams
  • remove or restore keys owned by the previous overlay when a replacement omits them
  • preserve union provenance before changing the cluster so interrupted replacements can be retried safely
  • retain the overlay and provenance files when erpc reset cannot clean the live ConfigMap
  • add regressions for disjoint replacement, restoration of chart-owned entries, and failed-reset recovery

Problem

obol network erpc set replaced the host-side overlay file but only merged the new document into the existing eRPC ConfigMap.

Given overlay A followed by disjoint overlay B, the resulting state was effectively:

saved overlay: B
live config:   chart base + A + B

A subsequent reset read only B, removed only B-owned entries, and deleted provenance. Entries from A remained live without any durable ownership record, so an old RPC upstream could continue receiving traffic after the operator believed it had been replaced and reset.

Reset had a related failure mode: cluster cleanup was best-effort, but the durable overlay and provenance files were deleted even when cleanup failed. That made a transient Kubernetes failure permanently discard the information needed for a later retry.

Implementation

Replacement reconciliation now follows this model:

current live - retired overlay keys + desired overlay = desired live

Before merging the desired overlay, reconciliation:

  1. identifies provenance-tracked network and upstream keys absent from the desired document
  2. removes keys originally added by the overlay or restores chart/recorded entries that were replaced
  3. captures provenance for newly owned keys against the restored base state
  4. persists union provenance before writing the ConfigMap
  5. writes the desired live configuration
  6. prunes retired provenance after the cluster write succeeds

Persisting union provenance first makes the intermediate states recoverable. If the process or cluster write fails, the saved desired overlay and union ownership record allow the next stack-up reconcile to complete the transition.

For overlays created before provenance tracking, SetERPC records legacy drop markers before replacing the old document so their keys are not forgotten.

Reset now returns an error and retains both durable files when live cleanup fails.

Scope

This changes replacement and reset semantics for overlay-owned networks and upstreams, the entries that control routing. The existing intentional behavior for rate limiters and cache policies is unchanged.

Validation

  • go test ./internal/network -count=1
  • go test -race ./internal/network -count=1
  • go vet ./internal/network
  • go test ./cmd/obol ./internal/stack -count=1
  • go test ./... -count=1
  • just build
  • git diff --check

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant