Add configurable trusted client-IP header support for Fastly - #1048
Add configurable trusted client-IP header support for Fastly#1048prk-Jr wants to merge 29 commits into
Conversation
Reject secrets shorter than the 32-character minimum already applied to ec.passphrase, and route the secret through reject_placeholder_secrets so the placeholder published in the example config and guides fails startup. The secret is the only gate on forging the client address that geolocation, EC identity derivation, and bot protection consume.
The shared proxy code forwards an inbound X-Forwarded-For to publisher origins, and no adapter has a trustworthy upstream forwarded-for chain, so a client could choose the address the origin attributes the request to. The Spin adapter already stripped it; moving the rule into the shared spoofable-header list closes the same gap on Fastly without changing Spin behavior. Integrations that need the address keep injecting their own value from the resolved client IP.
Fold resolution and forwarded-header sanitization into resolve_and_sanitize_client_ip so resolution cannot be reordered after the sanitization that removes the headers it reads. Log every fallback taken while a configuration is present at debug level, without the secret or the address, so a rotated secret or renamed header stops failing silently. Document the front door requirement to leave exactly one value per trust header, since duplicates select the peer address.
ChristianPavilonis
left a comment
There was a problem hiding this comment.
Summary
Requesting changes for one high-severity config rollout and rollback compatibility issue. The trusted client-IP runtime behavior and validation otherwise looked sound at the reviewed revision.
`#[serde(default)]` only affects deserialization, so an unconfigured `trusted_client_ip` was still written as `"trusted_client_ip": null` into every blob produced by `ts config push`. `Settings` uses `deny_unknown_fields`, so pushing an otherwise unchanged config before upgrading all instances - or rolling back after such a push - made older instances reject the blob even though the feature was never enabled. Skip serialization when the field is `None`, matching the existing `AuctionConfig` rollback-compatibility pattern, and document that a configured section requires restoring a compatible blob before rollback. Add regression tests proving a default payload omits the key and stays readable by a schema matching the base revision.
Resolve two conflicts: - `settings.rs` imports: keep both `std::time::Duration` (cache policy from main) and `subtle::ConstantTimeEq` (shared-secret comparison). - `configuration.md` key-sections table: keep both the `[cache]` row from main and the `[trusted_client_ip]` row from this branch. Also add `cache` to the `BaseRevisionSettings` test schema, since main adds that key to `Settings` and the base revision this branch rolls back to now knows it.
aram356
left a comment
There was a problem hiding this comment.
Summary
Adds authenticated trusted client-IP forwarding for Fastly with peer fallback, cross-adapter trust-header stripping, and validated configuration. No blocking defects found in the resolution path: authentication precedes IP parsing, the comparison is constant-time over fixed-size digests, malformed or duplicated input fails closed to the peer address, no secret or header value is logged, and the config-blob rollback hazard is regression-tested against a modeled base-revision schema. The findings below are hardening and maintainability items.
2 of the inline comments below carry a one-click GitHub
suggestion. Use Commit suggestion (or Add suggestion to batch) to apply them as commits on the PR branch. The remaining comments describe concerns in prose because the change is design-level or spans files outside the diff.
Non-blocking
♻️ refactor
- Reserve all internal trust headers, not just the TLS bridge pair - see inline at
crates/trusted-server-core/src/settings.rs:2672
⛏ nitpick
- Duplicate
/.worktrees/entry - see inline at.gitignore:55
🤔 thinking
- Trust-header sanitization depends on unstated middleware ordering - see inline at
crates/trusted-server-adapter-axum/src/middleware.rs:40
🌱 seedling
- No operator-visible signal when the trust handshake degrades - see inline at
crates/trusted-server-adapter-fastly/src/platform.rs:730
CI Status
- cargo fmt: PASS (required)
- cargo test: PASS (required)
- format-typescript: PASS (required)
- format-docs: PASS (required)
- cargo test (axum native): PASS
- cargo test (ts CLI, native): PASS
- cargo test (cross-adapter parity): PASS
- cargo check (cloudflare native + wasm32-unknown-unknown): PASS
- cargo check/build/test (spin native + wasm32-wasip1): PASS
- integration tests: PASS
- integration tests (Fastly EC lifecycle): PASS
- browser integration tests: PASS
- prepare integration artifacts: PASS
- vitest: PASS
- CodeQL: PASS
- Analyze (rust): PASS
- Analyze (javascript-typescript): PASS (both runs)
- Analyze (actions): PASS
Reserve every Trusted Server internal header name for the trusted client-IP configuration instead of only the two TLS bridge names, so a configured trust header cannot collide with an internal signal such as x-forwarded-for or x-geo-info-available. Move the trust-header strip out of FinalizeResponseMiddleware into a dedicated SanitizeRequestMiddleware on the Cloudflare, Spin, and Axum adapters, registered outermost with the ordering requirement stated at each registration site. The security invariant no longer hides inside a response-finalization middleware. Drop the duplicate /.worktrees/ gitignore entry added by this branch; line 41 already ignores that path.
ChristianPavilonis
left a comment
There was a problem hiding this comment.
Summary
Reviewed 3329711dbd7724d0c16178d2ec9d79518852ef90 against 2c1e0825d248361331441f312b7728d4c3041006. The authenticated resolution, peer fallback, ClientInfo propagation, and cross-adapter sanitization paths looked sound. Approving with one low-severity documentation correction inline.
| through `http::HeaderName`. The two Fastly-injected TLS bridge fields | ||
| (`x-ts-tls-protocol` and `x-ts-tls-cipher`) are forbidden for either setting | ||
| because the entry point owns and re-injects them after sanitization. These rules |
There was a problem hiding this comment.
🔧 P3: Design spec understates the reserved-header restriction
The final validator rejects every name in INTERNAL_HEADERS, not only the two TLS bridge fields described here. A maintainer following this new design document could choose x-forwarded-for, x-geo-info-available, or x-ts-ec and get a startup failure even though the document appears to allow it. The public configuration guide and trusted_client_ip_rejects_reserved_internal_headers already describe and test the broader contract.
| through `http::HeaderName`. The two Fastly-injected TLS bridge fields | |
| (`x-ts-tls-protocol` and `x-ts-tls-cipher`) are forbidden for either setting | |
| because the entry point owns and re-injects them after sanitization. These rules | |
| through `http::HeaderName`. Every name in `INTERNAL_HEADERS` is forbidden for | |
| either setting. This includes `x-forwarded-for`, `x-geo-info-available`, | |
| `x-ts-ec`, `x-ts-tls-protocol`, and `x-ts-tls-cipher`. |
|
@prk-Jr to confirm how vcl path works / clarify language in pr description |
Recommend a dedicated x- header name for ip_header instead of fastly-client-ip. On a fronting VCL service that carries other traffic, a dedicated name means the front door never modifies Fastly-Client-IP, so security rules, rate limiters, logging formats, and vendor snippets that read it keep working unchanged. fastly-client-ip stays supported and still suits a service dedicated to Trusted Server. Replace the pasteable VCL example with a route-scoped, restart-safe form. The previous example set the authentication header unconditionally, so on a shared service the shared secret reached every backend and its access logs. The unset lines now run on every route while the set lines run only on the Trusted Server route. Drop the req.restarts guard: a restart can change which backend a request reaches, and re-evaluating the route condition on each pass stops a stamp made before the restart from following the request to a different backend. State why each guard exists, including that keeping the unset lines inside the fastly.ff.visits_this_service check lets shielded requests preserve the values stamped on first entry. Add failure-mode and topology tables. Every misconfiguration falls back to the immediate peer address except one: another CDN in front of Fastly makes client.ip that CDN's node, so the forwarded address is wrong and authenticated as valid. Document that the Fastly adapter strips fastly-client-ip at request entry whether or not the section is configured, so Trusted Server no longer forwards an inbound Fastly-Client-IP to the publisher origin. Promote the deploy-order requirement to a warning. Settings rejects unknown fields, so a blob carrying this section fails to load on a binary that predates it. Add a verification procedure using the geolocation response headers, including a duplicate-header request that proves the front door's unset holds. Resolve a pre-existing contradiction in configuration.md, where the prose already advised dedicated x- names while the example used fastly-client-ip.
One conflict, in the Fastly middleware test module's imports. This branch added ClientInfo for the authoritative client-IP geo tests; main added apply_inactive_ad_stack_browser_cache_policy for the ad-template cache policy tests. Both test sets stay, so both imports stay.
Summary
On Fastly, Trusted Server takes the client IP from
req.get_client_ip_addr(), which Fastly defines as the immediate peer. Behind a fronting Fastly or CDN service that peer is the fronting edge node, so geolocation, EC identity derivation, and consent jurisdiction all describe the POP instead of the reader. It fails silently — pages render, ads serve, nothing errors. Measured evidence is in #1040.Reading
Fastly-Client-IPon its own would be worse than the bug: Fastly does not protect it at ingress, so any caller could choose the address that feeds EC identity, geo, and bot classification.This PR adds an opt-in
[trusted_client_ip]section. The forwarded address is accepted only when the same request carries a shared secret that only the front door knows. Every other outcome falls back to the peer address without rejecting the request.Closes #1041
Fixes #1040
What changes for existing deployments
With no
[trusted_client_ip]section, client-IP resolution is byte-identical to today. One behavior change lands regardless of configuration:fastly-client-ipis added toSPOOFABLE_FORWARDED_HEADERS, so the Fastly adapter strips it at request entry. Trusted Server never read that header, so nothing internal changes — but the origin request reuses the inbound header set, so Trusted Server no longer forwards an inboundFastly-Client-IPto the publisher origin. Confirm whether the origin reads it for geolocation, fraud checks, or logging before merging.Deploy order
Warning
Settingsusesdeny_unknown_fields. Once a pushed config blob contains[trusted_client_ip], any binary predating this PR fails to load it and returns its startup-error response.ts config push.Getting this order wrong takes the service down rather than degrading it. The unconfigured direction is safe —
skip_serializing_if = "Option::is_none"keeps the key out of the blob entirely, and a compat test pins that against a model of the pre-change schema.How resolution works
Per request, in
resolve_and_sanitize_client_ip:auth_header. Must be exactly one UTF-8 value.shared_secretin constant time (SHA-256 +subtle::ct_eq).ip_header. Must be exactly one UTF-8 value.Resolution and sanitization are fused into one function so the ordering cannot be reversed by a future edit — resolution has to observe the headers before sanitization removes them.
[trusted_client_ip]sectionNo combination rejects the request. A rotated secret, renamed header, or misconfigured front door degrades to current behavior. Fallbacks taken while a config is present log at debug — not warn, so a direct caller cannot drive log volume with junk headers. Neither the secret nor the address is logged.
Front-door requirement
Code configuration alone does not enable this. The fronting VCL service must set both headers, and must remove client-supplied copies. Full setup in
docs/guide/fastly.md; the shape is:Four things this shape gets right, all explained in the docs:
x-header name rather thanFastly-Client-IP.fastly-client-ipis still accepted and suits a dedicated service, but on a shared service a dedicated name means the front door never modifiesFastly-Client-IP, so security rules, rate limiters, logging formats, and vendor snippets that read it keep working unchanged.unsetbeforeset. Trusted Server ignores the forwarded address whenever either header carries more than one value. Without theunset, a reader can send its own copy, force the fallback, and keep its real address out of geolocation and bot classification. This is anti-evasion, not tidiness.setlines. Client-supplied values die on every route; the secret is added only on the route that reaches Trusted Server, so it never reaches another backend or another backend's logs.req.restartsguard. A restart can change which backend a request reaches. Re-running the block each pass re-evaluates the route condition, so a stamp made before a restart cannot follow the request to a different backend.The secret comes from a private (write-only) edge dictionary, not inline VCL — inline VCL is readable by anyone with service-configuration access and is preserved in every version diff.
Confirm the topology first
client.ipat the front doorRow 2 is the only topology that fails with a wrong value instead of falling back. If another CDN precedes Fastly, restrict direct access to the Fastly front door and derive
ip_headerfrom that CDN's protected reader-IP value rather than fromclient.ip.Fastly no-code request routing provides no point to inject headers, so this mechanism cannot help that topology.
Validation
Startup rejects, without exposing the secret in the error:
shared_secretshorter than 32 bytes, or containing any byte outside!–~(whitespace, tab, control, DEL, non-ASCII)reject_placeholder_secretsip_headerthat is neitherfastly-client-ipnorx--prefixed;auth_headerthat is notx--prefixedRedacted<String>keeps the secret out of debug output and validation errors.Secret storage
Redaction does not move the value into a platform secret store. With the current configuration architecture,
ts config pushserializesshared_secretinto the Trusted Server application-config blob, so access to that config store must be restricted — it is the only gate on forging the reader address.Fastly recommends Secret Store rather than Config Store for sensitive values. Migrating this field requires a shared-schema change to hold a secret-store reference and resolve it in the Fastly request path. The broader migration of existing Trusted Server passphrases and secrets is tracked by #846 and is outside this PR.
The fronting copy is a separate concern: a Fastly VCL service cannot read a Compute Secret Store, so it must obtain the identical value through a VCL-accessible mechanism such as a write-only edge dictionary.
Scope notes
X-Forwarded-Foris unchanged. An earlier partial hardening was removed after review because it covered Fastly and Spin but not Cloudflare and Axum. Consistent reconstruction from authoritativeClientInfoshould be handled separately.Changes
crates/trusted-server-core/src/settings.rs[trusted_client_ip]config; constant-time authentication; 32-byte ASCII-graphic minimum; placeholder rejection;skip_serializing_iffor blob compatibility.crates/trusted-server-core/src/http_util.rsfastly-client-ipas spoofable after authenticated consumption; shared sanitizer for configured trust headers.crates/trusted-server-adapter-fastly/src/platform.rsresolve_client_ip: exactly one authenticated IPv4/IPv6 value, peer fallback, debug-level fallback categories with no secret or address logged.crates/trusted-server-adapter-fastly/src/compat.rsresolve_and_sanitize_client_ipfuses resolution ahead of stripping so the order cannot be reversed.crates/trusted-server-adapter-fastly/src/main.rsClientInfo.crates/trusted-server-adapter-fastly/src/middleware.rsClientInfofor response geolocation, including authoritative absence.crates/trusted-server-adapter-fastly/src/app.rscrates/trusted-server-adapter-{axum,cloudflare,spin}/src/middleware.rsSanitizeRequestMiddlewarethat strips the configured trust headers. Client-IP resolution unchanged.crates/trusted-server-adapter-{axum,cloudflare,spin}/src/app.rsSanitizeRequestMiddlewareas the outermost middleware, so nothing else observes the authentication header.trusted-server.example.toml[trusted_client_ip]example.docs/guide/configuration.mddocs/guide/fastly.mddocs/superpowers/specs/…-design.md,docs/superpowers/plans/…Test plan
cargo test-fastly,cargo test-axum,cargo test-cloudflare,cargo test-spincargo clippy-fastly,-axum,-cloudflare,-cloudflare-wasm,-spin-native,-spin-wasmcargo fmt --all -- --checknpx vitest run,npm run format) on pinned Node 24.12.0cd docs && npm run format)cargo build --package trusted-server-adapter-fastly --release --target wasm32-wasip1docs/guide/fastly.md: comparex-geo-city/x-geo-coordinatesthrough the front door against a direct request, then replay with duplicate and junk trust headers to confirm the front door'sunsetholds.Fastly-Client-IP(see What changes for existing deployments).Checklist
unwrap()in production codelogmacros, notprintln!