Skip to content

feat(gateway): DNS credential verification and cert attempt observability - #1062

Draft
Leechael wants to merge 2 commits into
nextfrom
feat/gateway-admin-observability
Draft

feat(gateway): DNS credential verification and cert attempt observability#1062
Leechael wants to merge 2 commits into
nextfrom
feat/gateway-admin-observability

Conversation

@Leechael

@Leechael Leechael commented Aug 17, 2026

Copy link
Copy Markdown
Collaborator

Summary

Two admin-facing observability additions for the gateway's certificate lifecycle:

  • VerifyDnsCredential admin RPC — read-only check of a DNS credential against the provider API. Reports the Cloudflare token status (active/expired/disabled/invalid), the granted permission groups, whether issuance requirements (Zone Read, DNS Write) are covered, and zone resolution for every domain referencing the credential (directly or via the default-credential fallback). Today an invalid or under-privileged token only surfaces as a log line during the next renewal attempt.
  • Per-domain cert attempt status — certificate issuance/renewal runs as a background task whose failures only reach the logs. DistributedCertBot now keeps a node-local, in-memory record per domain (last attempt time and node, last success, consecutive failures, truncated error chain of the last failure). Exposed by extending ZtDomainCertStatus, so existing GetZtDomain/ListZtDomains return it without a new RPC. try_renew and init_domain both record outcomes; nodes that skip because a peer holds the renew lock do not write.

Design notes

  • Token permissions are queried, not probed: GET /user/tokens/verify yields the token id, then GET /user/tokens/{id} returns the granted permission groups. No DNS records are created or removed.
  • Attempt status is in-memory (node-local) by design: current deployments are single-node. The response shape is storage-agnostic, so the record can later move into WaveKV for multi-node visibility without API changes. If this lands after feat(gateway): upgrade to wavekv 2.0 delta-state sync #1031, no schema interaction is needed since nothing new is written to WaveKV.
  • Error chains are truncated to 16 KiB keeping head and tail — upstream gateway errors (e.g. Cloudflare 502 pages) are full HTML documents.

Test plan

  • cargo test -p dstack-gateway -p certbot — 161 passed, including new unit tests for permission matching, error truncation, and attempt-status tracking
  • End-to-end against tools/mock-cf-dns-api (extended with /user/tokens/verify and /user/tokens/{id}, plus MOCK_TOKEN_STATUS/MOCK_TOKEN_PERMISSIONS knobs): token verify, permission group parsing, zone resolution, and unknown-zone failure all verified
  • cargo clippy clean, prek run --all-files clean
  • Live gateway against real Cloudflare token

Read-only check of a DNS credential against the provider API: reports the
Cloudflare token status (active/expired/disabled/invalid), the granted
permission groups, whether issuance requirements (Zone Read, DNS Write)
are covered, and zone resolution for every domain referencing the
credential. Today an invalid or under-privileged token only surfaces as a
log line during the next renewal attempt.

certbot gains verify_cloudflare_token() (GET /user/tokens/verify followed
by GET /user/tokens/{id}) and resolve_cloudflare_zone(); the mock CF API
learns the matching token endpoints, with MOCK_TOKEN_STATUS and
MOCK_TOKEN_PERMISSIONS env knobs for failure testing.
Certificate issuance and renewal run as a background task whose failures
only reach the logs, so an expiring certificate is the first visible
signal of a broken renewal. DistributedCertBot now keeps a node-local,
in-memory record per domain: last attempt time and node, last success,
consecutive failures, and the (16 KiB head+tail truncated) error chain of
the last failed attempt. Both the periodic/manual renewal path
(try_renew) and startup issuance (init_domain) record their outcome;
nodes that skip because a peer holds the renew lock do not write.

The record is exposed on the existing ZtDomainCertStatus, so GetZtDomain
and ListZtDomains show it without a new RPC. In-memory storage targets
the current single-node deployments; multi-node visibility can later move
the record into WaveKV without changing the response shape.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant