Skip to content

Add configurable trusted client-IP header support for Fastly - #1048

Open
prk-Jr wants to merge 29 commits into
mainfrom
fix/trusted-client-ip-header
Open

Add configurable trusted client-IP header support for Fastly#1048
prk-Jr wants to merge 29 commits into
mainfrom
fix/trusted-client-ip-header

Conversation

@prk-Jr

@prk-Jr prk-Jr commented Aug 19, 2026

Copy link
Copy Markdown
Collaborator

Summary

On Fastly, Trusted Server takes the client IP from req.get_client_ip_addr(), which Fastly defines as the immediate peer. Behind a fronting Fastly or CDN service that peer is the fronting edge node, so geolocation, EC identity derivation, and consent jurisdiction all describe the POP instead of the reader. It fails silently — pages render, ads serve, nothing errors. Measured evidence is in #1040.

Reading Fastly-Client-IP on its own would be worse than the bug: Fastly does not protect it at ingress, so any caller could choose the address that feeds EC identity, geo, and bot classification.

This PR adds an opt-in [trusted_client_ip] section. The forwarded address is accepted only when the same request carries a shared secret that only the front door knows. Every other outcome falls back to the peer address without rejecting the request.

Closes #1041
Fixes #1040


What changes for existing deployments

With no [trusted_client_ip] section, client-IP resolution is byte-identical to today. One behavior change lands regardless of configuration:

fastly-client-ip is added to SPOOFABLE_FORWARDED_HEADERS, so the Fastly adapter strips it at request entry. Trusted Server never read that header, so nothing internal changes — but the origin request reuses the inbound header set, so Trusted Server no longer forwards an inbound Fastly-Client-IP to the publisher origin. Confirm whether the origin reads it for geolocation, fraud checks, or logging before merging.

Deploy order

Warning

Settings uses deny_unknown_fields. Once a pushed config blob contains [trusted_client_ip], any binary predating this PR fails to load it and returns its startup-error response.

  • Rolling out: deploy the code to every instance first, then ts config push.
  • Rolling back: restore a config without the section first, then roll the binary back.

Getting this order wrong takes the service down rather than degrading it. The unconfigured direction is safe — skip_serializing_if = "Option::is_none" keeps the key out of the blob entirely, and a compat test pins that against a model of the pre-change schema.

How resolution works

Per request, in resolve_and_sanitize_client_ip:

  1. Read auth_header. Must be exactly one UTF-8 value.
  2. Compare against shared_secret in constant time (SHA-256 + subtle::ct_eq).
  3. Read ip_header. Must be exactly one UTF-8 value.
  4. Parse as a bare IPv4 or IPv6 address. No trimming, no port, no zone suffix.
  5. Strip both trust headers plus the static spoofable list, then route.

Resolution and sanitization are fused into one function so the ordering cannot be reversed by a future edit — resolution has to observe the headers before sanitization removes them.

Request state Address used
One matching auth value and one bare IP value Forwarded reader
Auth value missing, empty, wrong, duplicated, or not UTF-8 Immediate peer
IP value missing, duplicated, not UTF-8, or not a bare IP Immediate peer
Request bypassed the front door Immediate peer
No [trusted_client_ip] section Immediate peer

No combination rejects the request. A rotated secret, renamed header, or misconfigured front door degrades to current behavior. Fallbacks taken while a config is present log at debug — not warn, so a direct caller cannot drive log volume with junk headers. Neither the secret nor the address is logged.

Front-door requirement

Code configuration alone does not enable this. The fronting VCL service must set both headers, and must remove client-supplied copies. Full setup in docs/guide/fastly.md; the shape is:

sub vcl_recv {
  if (fastly.ff.visits_this_service == 0) {
    # Client-supplied copies never survive, on any route.
    unset req.http.X-TS-Client-IP;
    unset req.http.X-TS-Client-IP-Auth;

    # Stamp only on the Trusted Server route, so the secret never reaches
    # another backend.
    if (req.http.host == "www.example.com") {
      set req.http.X-TS-Client-IP = client.ip;
      set req.http.X-TS-Client-IP-Auth =
        table.lookup(ts_private_config, "trusted_client_ip_secret");
    }
  }
}

Four things this shape gets right, all explained in the docs:

  • A dedicated x- header name rather than Fastly-Client-IP. fastly-client-ip is still accepted and suits a dedicated service, but on a shared service a dedicated name means the front door never modifies Fastly-Client-IP, so security rules, rate limiters, logging formats, and vendor snippets that read it keep working unchanged.
  • unset before set. Trusted Server ignores the forwarded address whenever either header carries more than one value. Without the unset, a reader can send its own copy, force the fallback, and keep its real address out of geolocation and bot classification. This is anti-evasion, not tidiness.
  • The route condition wraps only the set lines. Client-supplied values die on every route; the secret is added only on the route that reaches Trusted Server, so it never reaches another backend or another backend's logs.
  • No req.restarts guard. A restart can change which backend a request reaches. Re-running the block each pass re-evaluates the route condition, so a stamp made before a restart cannot follow the request to a different backend.

The secret comes from a private (write-only) edge dictionary, not inline VCL — inline VCL is readable by anyone with service-configuration access and is preserved in every version diff.

Confirm the topology first

Topology client.ip at the front door Result
Reader → fronting Fastly service → Trusted Server The reader Correct reader address
Reader → another CDN → Fastly → Trusted Server That CDN's node Wrong address, and authenticated as valid
Reader → Trusted Server directly Not applicable Falls back to the immediate peer
Fastly no-code request routing No injection point Mechanism unavailable

Row 2 is the only topology that fails with a wrong value instead of falling back. If another CDN precedes Fastly, restrict direct access to the Fastly front door and derive ip_header from that CDN's protected reader-IP value rather than from client.ip.

Fastly no-code request routing provides no point to inject headers, so this mechanism cannot help that topology.

Validation

Startup rejects, without exposing the secret in the error:

  • shared_secret shorter than 32 bytes, or containing any byte outside !~ (whitespace, tab, control, DEL, non-ASCII)
  • the documented placeholder value, via reject_placeholder_secrets
  • ip_header that is neither fastly-client-ip nor x--prefixed; auth_header that is not x--prefixed
  • the two header names being equal, or either reusing a Trusted Server internal header name

Redacted<String> keeps the secret out of debug output and validation errors.

Secret storage

Redaction does not move the value into a platform secret store. With the current configuration architecture, ts config push serializes shared_secret into the Trusted Server application-config blob, so access to that config store must be restricted — it is the only gate on forging the reader address.

Fastly recommends Secret Store rather than Config Store for sensitive values. Migrating this field requires a shared-schema change to hold a secret-store reference and resolve it in the Fastly request path. The broader migration of existing Trusted Server passphrases and secrets is tracked by #846 and is outside this PR.

The fronting copy is a separate concern: a Fastly VCL service cannot read a Compute Secret Store, so it must obtain the identical value through a VCL-accessible mechanism such as a write-only edge dictionary.

Scope notes

  • Only Fastly resolves client IP from this section. Cloudflare, Spin, and Axum validate it and strip the configured headers before routing — so a shared multi-adapter config cannot leak the authentication value into publisher or integration handlers — but keep using their own runtime client address.
  • X-Forwarded-For is unchanged. An earlier partial hardening was removed after review because it covered Fastly and Spin but not Cloudflare and Axum. Consistent reconstruction from authoritative ClientInfo should be handled separately.

Changes

File Change
crates/trusted-server-core/src/settings.rs Validated, redacted [trusted_client_ip] config; constant-time authentication; 32-byte ASCII-graphic minimum; placeholder rejection; skip_serializing_if for blob compatibility.
crates/trusted-server-core/src/http_util.rs Treat fastly-client-ip as spoofable after authenticated consumption; shared sanitizer for configured trust headers.
crates/trusted-server-adapter-fastly/src/platform.rs resolve_client_ip: exactly one authenticated IPv4/IPv6 value, peer fallback, debug-level fallback categories with no secret or address logged.
crates/trusted-server-adapter-fastly/src/compat.rs resolve_and_sanitize_client_ip fuses resolution ahead of stripping so the order cannot be reversed.
crates/trusted-server-adapter-fastly/src/main.rs Resolve once; propagate through authoritative ClientInfo.
crates/trusted-server-adapter-fastly/src/middleware.rs Use authoritative ClientInfo for response geolocation, including authoritative absence.
crates/trusted-server-adapter-fastly/src/app.rs Assert the resolved address reaches request-scoped services and the EC finalization context.
crates/trusted-server-adapter-{axum,cloudflare,spin}/src/middleware.rs New SanitizeRequestMiddleware that strips the configured trust headers. Client-IP resolution unchanged.
crates/trusted-server-adapter-{axum,cloudflare,spin}/src/app.rs Register SanitizeRequestMiddleware as the outermost middleware, so nothing else observes the authentication header.
trusted-server.example.toml Commented [trusted_client_ip] example.
docs/guide/configuration.md Fields, validation, deploy-order warning, env overrides, fallback table, cross-adapter sanitization, config-blob storage, origin-forwarding note.
docs/guide/fastly.md Front-door setup, dedicated header-name guidance, per-line VCL rationale, failure-mode and topology tables, verification procedure, no-code routing limitation.
docs/superpowers/specs/…-design.md, docs/superpowers/plans/… Security and data-flow design; implementation and verification plan.

Test plan

  • cargo test-fastly, cargo test-axum, cargo test-cloudflare, cargo test-spin
  • cargo clippy-fastly, -axum, -cloudflare, -cloudflare-wasm, -spin-native, -spin-wasm
  • cargo fmt --all -- --check
  • JS tests and format (npx vitest run, npm run format) on pinned Node 24.12.0
  • Docs format (cd docs && npm run format)
  • WASM release build: cargo build --package trusted-server-adapter-fastly --release --target wasm32-wasip1
  • Local Viceroy: authenticated forwarded IP selects the configured Sydney geo fixture; missing or incorrect authentication falls back to the peer-IP San Francisco fixture without rejecting the request.
  • Live Fastly service-chain validation against the two-hop topology. Not yet run — the checks above do not prove the handoff end to end. Verification procedure is in docs/guide/fastly.md: compare x-geo-city / x-geo-coordinates through the front door against a direct request, then replay with duplicate and junk trust headers to confirm the front door's unset holds.
  • Confirm whether the publisher origin reads Fastly-Client-IP (see What changes for existing deployments).

Checklist

  • Changes follow CLAUDE.md conventions
  • No unwrap() in production code
  • Uses repository log macros, not println!
  • New code has tests
  • No real secrets or credentials committed

@prk-Jr prk-Jr self-assigned this Aug 19, 2026
prk-Jr added 3 commits August 19, 2026 23:32
Reject secrets shorter than the 32-character minimum already applied to
ec.passphrase, and route the secret through reject_placeholder_secrets so
the placeholder published in the example config and guides fails startup.
The secret is the only gate on forging the client address that geolocation,
EC identity derivation, and bot protection consume.
The shared proxy code forwards an inbound X-Forwarded-For to publisher
origins, and no adapter has a trustworthy upstream forwarded-for chain, so
a client could choose the address the origin attributes the request to.
The Spin adapter already stripped it; moving the rule into the shared
spoofable-header list closes the same gap on Fastly without changing Spin
behavior. Integrations that need the address keep injecting their own value
from the resolved client IP.
Fold resolution and forwarded-header sanitization into
resolve_and_sanitize_client_ip so resolution cannot be reordered after the
sanitization that removes the headers it reads. Log every fallback taken
while a configuration is present at debug level, without the secret or the
address, so a rotated secret or renamed header stops failing silently.
Document the front door requirement to leave exactly one value per trust
header, since duplicates select the peer address.
@prk-Jr prk-Jr added this to the 202608 milestone Aug 20, 2026
@prk-Jr prk-Jr changed the title Trust client IPs behind Fastly service chains Add configurable trusted client-IP header support for Fastly Aug 20, 2026
@aram356
aram356 marked this pull request as draft August 20, 2026 16:07
@prk-Jr
prk-Jr marked this pull request as ready for review August 21, 2026 11:29

@ChristianPavilonis ChristianPavilonis left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

Requesting changes for one high-severity config rollout and rollback compatibility issue. The trusted client-IP runtime behavior and validation otherwise looked sound at the reviewed revision.

Comment thread crates/trusted-server-core/src/settings.rs Outdated
prk-Jr added 2 commits August 22, 2026 12:01
`#[serde(default)]` only affects deserialization, so an unconfigured
`trusted_client_ip` was still written as `"trusted_client_ip": null` into
every blob produced by `ts config push`. `Settings` uses
`deny_unknown_fields`, so pushing an otherwise unchanged config before
upgrading all instances - or rolling back after such a push - made older
instances reject the blob even though the feature was never enabled.

Skip serialization when the field is `None`, matching the existing
`AuctionConfig` rollback-compatibility pattern, and document that a
configured section requires restoring a compatible blob before rollback.

Add regression tests proving a default payload omits the key and stays
readable by a schema matching the base revision.
Resolve two conflicts:

- `settings.rs` imports: keep both `std::time::Duration` (cache policy
  from main) and `subtle::ConstantTimeEq` (shared-secret comparison).
- `configuration.md` key-sections table: keep both the `[cache]` row from
  main and the `[trusted_client_ip]` row from this branch.

Also add `cache` to the `BaseRevisionSettings` test schema, since main
adds that key to `Settings` and the base revision this branch rolls back
to now knows it.

@aram356 aram356 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

Adds authenticated trusted client-IP forwarding for Fastly with peer fallback, cross-adapter trust-header stripping, and validated configuration. No blocking defects found in the resolution path: authentication precedes IP parsing, the comparison is constant-time over fixed-size digests, malformed or duplicated input fails closed to the peer address, no secret or header value is logged, and the config-blob rollback hazard is regression-tested against a modeled base-revision schema. The findings below are hardening and maintainability items.

2 of the inline comments below carry a one-click GitHub suggestion. Use Commit suggestion (or Add suggestion to batch) to apply them as commits on the PR branch. The remaining comments describe concerns in prose because the change is design-level or spans files outside the diff.

Non-blocking

♻️ refactor

  • Reserve all internal trust headers, not just the TLS bridge pair - see inline at crates/trusted-server-core/src/settings.rs:2672

⛏ nitpick

  • Duplicate /.worktrees/ entry - see inline at .gitignore:55

🤔 thinking

  • Trust-header sanitization depends on unstated middleware ordering - see inline at crates/trusted-server-adapter-axum/src/middleware.rs:40

🌱 seedling

  • No operator-visible signal when the trust handshake degrades - see inline at crates/trusted-server-adapter-fastly/src/platform.rs:730

CI Status

  • cargo fmt: PASS (required)
  • cargo test: PASS (required)
  • format-typescript: PASS (required)
  • format-docs: PASS (required)
  • cargo test (axum native): PASS
  • cargo test (ts CLI, native): PASS
  • cargo test (cross-adapter parity): PASS
  • cargo check (cloudflare native + wasm32-unknown-unknown): PASS
  • cargo check/build/test (spin native + wasm32-wasip1): PASS
  • integration tests: PASS
  • integration tests (Fastly EC lifecycle): PASS
  • browser integration tests: PASS
  • prepare integration artifacts: PASS
  • vitest: PASS
  • CodeQL: PASS
  • Analyze (rust): PASS
  • Analyze (javascript-typescript): PASS (both runs)
  • Analyze (actions): PASS

Comment thread crates/trusted-server-core/src/settings.rs
Comment thread .gitignore Outdated
Comment thread crates/trusted-server-adapter-axum/src/middleware.rs Outdated
Comment thread crates/trusted-server-adapter-fastly/src/platform.rs
Reserve every Trusted Server internal header name for the trusted
client-IP configuration instead of only the two TLS bridge names, so a
configured trust header cannot collide with an internal signal such as
x-forwarded-for or x-geo-info-available.

Move the trust-header strip out of FinalizeResponseMiddleware into a
dedicated SanitizeRequestMiddleware on the Cloudflare, Spin, and Axum
adapters, registered outermost with the ordering requirement stated at
each registration site. The security invariant no longer hides inside a
response-finalization middleware.

Drop the duplicate /.worktrees/ gitignore entry added by this branch;
line 41 already ignores that path.
@prk-Jr
prk-Jr requested a review from aram356 August 24, 2026 05:04

@ChristianPavilonis ChristianPavilonis left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

Reviewed 3329711dbd7724d0c16178d2ec9d79518852ef90 against 2c1e0825d248361331441f312b7728d4c3041006. The authenticated resolution, peer fallback, ClientInfo propagation, and cross-adapter sanitization paths looked sound. Approving with one low-severity documentation correction inline.

Comment on lines +68 to +70
through `http::HeaderName`. The two Fastly-injected TLS bridge fields
(`x-ts-tls-protocol` and `x-ts-tls-cipher`) are forbidden for either setting
because the entry point owns and re-injects them after sanitization. These rules

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔧 P3: Design spec understates the reserved-header restriction

The final validator rejects every name in INTERNAL_HEADERS, not only the two TLS bridge fields described here. A maintainer following this new design document could choose x-forwarded-for, x-geo-info-available, or x-ts-ec and get a startup failure even though the document appears to allow it. The public configuration guide and trusted_client_ip_rejects_reserved_internal_headers already describe and test the broader contract.

Suggested change
through `http::HeaderName`. The two Fastly-injected TLS bridge fields
(`x-ts-tls-protocol` and `x-ts-tls-cipher`) are forbidden for either setting
because the entry point owns and re-injects them after sanitization. These rules
through `http::HeaderName`. Every name in `INTERNAL_HEADERS` is forbidden for
either setting. This includes `x-forwarded-for`, `x-geo-info-available`,
`x-ts-ec`, `x-ts-tls-protocol`, and `x-ts-tls-cipher`.

@ChristianPavilonis

ChristianPavilonis commented Aug 24, 2026

Copy link
Copy Markdown
Collaborator

@prk-Jr to confirm how vcl path works / clarify language in pr description

prk-Jr added 2 commits August 25, 2026 20:54
Recommend a dedicated x- header name for ip_header instead of
fastly-client-ip. On a fronting VCL service that carries other traffic, a
dedicated name means the front door never modifies Fastly-Client-IP, so
security rules, rate limiters, logging formats, and vendor snippets that
read it keep working unchanged. fastly-client-ip stays supported and
still suits a service dedicated to Trusted Server.

Replace the pasteable VCL example with a route-scoped, restart-safe form.
The previous example set the authentication header unconditionally, so on
a shared service the shared secret reached every backend and its access
logs. The unset lines now run on every route while the set lines run only
on the Trusted Server route. Drop the req.restarts guard: a restart can
change which backend a request reaches, and re-evaluating the route
condition on each pass stops a stamp made before the restart from
following the request to a different backend.

State why each guard exists, including that keeping the unset lines
inside the fastly.ff.visits_this_service check lets shielded requests
preserve the values stamped on first entry.

Add failure-mode and topology tables. Every misconfiguration falls back
to the immediate peer address except one: another CDN in front of Fastly
makes client.ip that CDN's node, so the forwarded address is wrong and
authenticated as valid.

Document that the Fastly adapter strips fastly-client-ip at request entry
whether or not the section is configured, so Trusted Server no longer
forwards an inbound Fastly-Client-IP to the publisher origin.

Promote the deploy-order requirement to a warning. Settings rejects
unknown fields, so a blob carrying this section fails to load on a binary
that predates it. Add a verification procedure using the geolocation
response headers, including a duplicate-header request that proves the
front door's unset holds.

Resolve a pre-existing contradiction in configuration.md, where the prose
already advised dedicated x- names while the example used
fastly-client-ip.
One conflict, in the Fastly middleware test module's imports. This branch
added ClientInfo for the authoritative client-IP geo tests; main added
apply_inactive_ad_stack_browser_cache_policy for the ad-template cache
policy tests. Both test sets stay, so both imports stay.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

3 participants