From 8520e9edf7716025dd7382fcce612076cd4e9618 Mon Sep 17 00:00:00 2001 From: Elmehdi Aitbrahim Date: Tue, 25 Aug 2026 16:40:51 -0400 Subject: [PATCH] feat(web): delete the renderer -- the server serves files and JSON, and nothing else (#540) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `keel/web/render.py` is gone: 987 lines of server-side HTML, the seven page handlers that called it, and its inline stylesheet. The client moved from `/static/` to `/` and is the application now. Net -896 lines. ── THE WRITE SURFACE MOVED, AND THAT IS WHAT MADE A LAYER REAL ──────────────── `/setup/*` was an HTML form. It is `POST /api/setup/` now, and the action SET is a 0-line diff: `keel.commands.setup.ACTIONS`, still only idempotent, non-destructive, `MECHANICAL` steps, still asserted disjoint from the eleven capability-increasing actions. A browser can set a deployment up; it still cannot arm a rule, attest an asset or enable autonomy. What changed is that `X-Keel-Client` now covers the write path. `_api_client_header_ok` recorded that widening as "#536's call to make, once (and only once) the forms it replaces are gone" -- because a `
` cannot set a header, and gating it would have refused every legitimate submission with no terminal to fall back to on the desktop bundle. The forms are gone. The CSRF token moved into `X-Keel-CSRF` for the same reason: in a body it would prove only that the sender could READ it; in a header it also proves the sender could set one, which a cross-origin form cannot do at all. ── A REVERSAL, RECORDED RATHER THAN QUIETLY APPLIED ─────────────────────────── `payload.setup_payload` used to argue against ever putting the token in a GET response: "minting a live write credential into a GET would put it into every cached copy, every proxy log and every paste". Two of those were already answered -- `/api/*` is `no-store` and #538's worker refuses to cache it, and the server binds loopback. The third does not survive the observation that settles it: this token authorises NOTHING without the session cookie, and anyone holding that cookie can mint it themselves. The old argument is quoted in place. ── TWO BUGS THE SUITE COULD NOT HAVE FOUND ──────────────────────────────────── **A refused write poisoned the next request on the same connection.** HTTP/1.1 keep-alive; a POST refused before its body was read left those bytes in the socket, and the stdlib parsed them as the next request line -- 501, on the request AFTER the one that was correctly refused. It became possible in this commit: moving the token to a header put the refusal in front of the body read. Every test in the suite opens a fresh connection, so none could see it. Found by driving a browser through five refusals in a row. `_drain_request_body` fixes it and disconnects rather than draining an oversized body, and a test now drives two requests down one socket. **An action's outcome was never visible.** The first version wrote the server's message into the card and then repainted the view, destroying it. The second restored it after the action's own repaint -- and the 15-second poll wiped it mid-sentence. Outcomes are session state now, re-applied in `rebuildInto`, which every path that replaces the view goes through. ── THE PIN TRANSFERRED, AND IT IS BIGGER THAN BEFORE ────────────────────────── `test_console_thinness` named `render` explicitly so its deletion would fail LOUDLY rather than shrink the scanned set. It did exactly that. The property -- "a THIRD renderer over the same reports, never a second place that computes them" -- belongs to the API layer now: `api`, `events`, `security`, `server` and `staticfiles` are pinned where three were. `keel/web/security.py`'s five layers are untouched. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01T6yA5khYnJ2qzheArRToQ2 --- keel/web/api.py | 46 +- keel/web/payload.py | 40 +- keel/web/render.py | 987 ------------------------ keel/web/security.py | 12 + keel/web/server.py | 550 ++++++------- keel/web/static/css/keel.css | 40 + keel/web/static/index.html | 40 +- keel/web/static/js/api.js | 67 ++ keel/web/static/js/main.js | 165 +++- keel/web/static/js/render.js | 115 ++- keel/web/static/manifest.webmanifest | 14 +- keel/web/static/sw.js | 39 +- keel/web/staticfiles.py | 23 +- tests/commands/test_console_thinness.py | 15 +- tests/web/test_api.py | 83 +- tests/web/test_client_assets.py | 101 ++- tests/web/test_doc_links.py | 54 +- tests/web/test_palette_contrast.py | 31 +- tests/web/test_pwa.py | 52 +- tests/web/test_render.py | 125 --- tests/web/test_server.py | 618 ++++++--------- tests/web/test_staticfiles.py | 55 +- 22 files changed, 1188 insertions(+), 2084 deletions(-) delete mode 100644 keel/web/render.py delete mode 100644 tests/web/test_render.py diff --git a/keel/web/api.py b/keel/web/api.py index 7f0da22..587681a 100644 --- a/keel/web/api.py +++ b/keel/web/api.py @@ -40,6 +40,7 @@ from typing import TYPE_CHECKING, Any from keel.web import payload +from keel.web.security import csrf_token if TYPE_CHECKING: # pragma: no cover - typing only from keel.web.server import ServeConfig @@ -157,7 +158,8 @@ def read_status(cfg: ServeConfig, _query: Query, _state: Any, now_ts: int) -> di def read_setup(cfg: ServeConfig, _query: Query, state: Any, _now_ts: int) -> dict[str, Any]: """The first-run checklist -- the ONE deployment-reading endpoint that must answer when there is no deployment, because it is the thing that says how to make one. `needs_database=False` - for that reason, and `server.needs_database` serves the same page in HTML for the same one. + for that reason: an endpoint that refused to answer until a deployment existed would refuse + precisely the operator who has none. `state` is the `DeploymentState` the envelope already read for `engine`, passed in rather than re-inspected: one 3.6 ms probe per response, not two.""" @@ -165,7 +167,15 @@ def read_setup(cfg: ServeConfig, _query: Query, state: Any, _now_ts: int) -> dic from keel.commands.setup import ACTIONS, NOT_AUTOMATED_YET return payload.setup_payload( - state, actions=ACTIONS, not_automated=NOT_AUTOMATED_YET, job=jobs.status() + state, + actions=ACTIONS, + not_automated=NOT_AUTOMATED_YET, + job=jobs.status(), + # The write token for this session, on the one endpoint whose view performs writes. Scoped + # to that endpoint rather than put on `/api/config` (which every view reads at boot) so it + # travels only to the page that needs it -- see `payload.setup_payload` for why it is in a + # body at all, which was a reversal. + csrf=csrf_token(cfg.token), ) @@ -607,6 +617,38 @@ def refusal_document(status: int, title: str, detail: str) -> dict[str, Any]: return payload.error_envelope(int(time.time()), status=status, title=title, detail=detail) +def action_document(result: Any) -> dict[str, Any]: + """One completed setup action, as JSON (#540). + + **`changed` is the field the client actually needs**, and it is not a success flag: it is the + difference between "created" and "already there". `keel.commands.setup`'s own note on it calls + it "the property that makes a double-click safe" -- every action is idempotent, so a repeated + submission succeeds and reports `changed: false`, which is a true statement about the + deployment rather than a soft failure. + + Judged here, in the serialiser, exactly as every other payload value is: the client is handed + `display` and `state` and never decides what a result means. + """ + return payload.envelope( + int(time.time()), + # `running=True` is a statement of fact rather than a probe: this document is built only + # after an action has RUN in this process, so the engine's presence is not in question and + # re-inspecting the deployment to say so would be one disk probe spent on a known answer. + running=True, + data={ + "step_key": str(getattr(result, "step_key", "")), + "changed": payload.flag( + bool(getattr(result, "changed", False)), + on="done", + off="already done — nothing to change", + on_state=payload.GOOD, + off_state=payload.NEUTRAL, + ), + "message": payload.label(str(getattr(result, "message", ""))), + }, + ) + + def sortable_columns() -> Mapping[str, Sequence[str]]: """The declared sort surface, for a test to read rather than restate.""" return {path: route.sortable for path, route in API_ROUTES.items() if route.sortable} diff --git a/keel/web/payload.py b/keel/web/payload.py index ce190c5..b7dff07 100644 --- a/keel/web/payload.py +++ b/keel/web/payload.py @@ -1479,6 +1479,16 @@ def _action_input_payload(field: Any) -> dict[str, Any]: # a form that can be submitted safely and one that leaks its own contents into browser # history. The client is told the answer rather than deriving it from the field's name. "secret": flag(field.secret, on="never echoed back", off="shown as typed"), + # The SAME fact as `secret`, in the form the client needs to act on rather than to show. + # + # Two keys for one boolean looks like duplication and is the opposite: `secret` is a + # `Field`, and a `Field` is a thing the client DISPLAYS -- Rule 3 of the client's pins + # forbids `render.js` from reading `.value` at all, precisely so no view can start + # deciding what a payload means. Choosing between `type="password"` and `type="text"` is + # not a judgement about a value, it is a rendering instruction, and the server is the one + # that gives it. Without this the client would have had to read `secret.value`, which is + # the whole rule. + "kind": "secret" if field.secret else "text", # A closed set of answers, rendered with NOTHING pre-selected: an action that could fill in # a field the operator left blank is one that could record something they never supplied. "choices": list(field.choices), @@ -1526,22 +1536,42 @@ def setup_payload( actions: Sequence[Any] = (), not_automated: Mapping[str, str] | None = None, job: Any = None, + csrf: str = "", ) -> dict[str, Any]: """`keel.commands.setup.inspect`'s `DeploymentState`, as JSON. - **No CSRF token, and that is the point rather than an omission.** `render_setup` takes one - because it emits ``; this issue ships reads only, and minting a live write - credential into a GET response would put it into every cached copy, every proxy log and every - paste of "here is what the API returned". The token is `csrf_token(session)` and it stays where - the write is. + **`csrf` is here since #540, and an earlier revision of this docstring argued it should not + be.** That argument is recorded rather than deleted, because reversing it was a decision: + + "No CSRF token, and that is the point rather than an omission. `render_setup` takes one + because it emits ``; this issue ships reads only, and minting a live + write credential into a GET response would put it into every cached copy, every proxy log + and every paste of 'here is what the API returned'." + + Two of those three concerns were already answered by the time the form was deleted: `/api/*` + is `Cache-Control: no-store` and #538's service worker refuses to cache it, so there is no + cached copy; and the server binds loopback, so there is no proxy. The third -- an operator + pasting API output into an issue -- is real, and it is what settles the question rather than + what blocks it: **this token is not a credential on its own.** It authorises nothing without + the session cookie, and anyone holding that cookie can read this endpoint and mint the same + token for themselves. A paste of it grants exactly what a paste of a random hex string grants. + + What the alternative would have cost is the reason not to be clever here: delivering it in a + second, script-readable cookie would take a live token out of a body and put it into + `document.cookie`, which is a strictly worse place for it on an origin whose session cookie is + deliberately `HttpOnly`. `actions` and `not_automated` are read from `keel.commands.setup`'s own closed registries and copied, never filtered here: an action appears only where the registry carries one, which is what stops "attest this asset" appearing because somebody edited a front-end. + """ from keel.commands.setup import Stage return { + # A bare string, not a `Field`: it is a credential the client SENDS, never a value it + # displays, and giving it a `display` would invite exactly that. + "csrf": str(csrf), "root": str(state.root), "config_path": str(state.config_path), "db_path": str(state.db_path), diff --git a/keel/web/render.py b/keel/web/render.py deleted file mode 100644 index 36a6158..0000000 --- a/keel/web/render.py +++ /dev/null @@ -1,987 +0,0 @@ -"""HTML for the read surface. Every function here is PURE -- a report in, a string out. - -The split matters more than it looks. `keel/commands/*` already returns frozen report dataclasses -and renders them to lines separately (`gather_status` / `render_human`, `build_insights_report` / -`render_summary`). This module is a THIRD renderer over the same reports, never a second place -that computes them -- which is what keeps `tests/commands/test_console_thinness.py` able to pin -the web layer with the rules it already applies to the console layer. - -Rendering from the dataclasses rather than wrapping the terminal lines in `
` is deliberate.
-`
` would have been a day's work instead of three, but it would freeze an 80-column terminal
-layout into a medium that has no columns, and it would make the web UI a screenshot of the TUI
-rather than a view of the data -- so every later improvement would have to start by undoing it.
-
-No JavaScript, no external assets, no CDN. The page is one document with an inline stylesheet.
-That is partly a packaging property -- D5 freezes this into a signed app bundle, and a build with
-no asset pipeline is a build with nothing to go wrong -- and partly the same argument the rest of
-the project makes: a page you can read the whole source of is a page you can audit.
-"""
-
-from __future__ import annotations
-
-import html
-import time
-from collections.abc import Iterable, Sequence
-from decimal import Decimal
-from typing import Any
-from urllib.parse import quote
-
-#: The published documentation root. Spelled here and in `static/js/docs.js`, and
-#: `tests/web/test_doc_links.py` pins that the two agree.
-DOCS_URL = "https://keeltrading.com/en/docs/"
-
-#: Nav order, and the labels. `/` first because the status page is the answer to "is it alive".
-#:
-#: **The eighth entry is an OUTBOUND link, and used to be a page (#539).** `/glossary` rendered
-#: `docs/glossary.md` read from the working directory -- which no installed deployment has, since
-#: `docs/` sits at the repository root, outside the `keel/` module `uv_build` packages. Every
-#: install therefore rendered an empty glossary, and `help_console.load_glossary`'s docstring said
-#: so. It is a link now, for the same reason the client's is: linking is the only form of this
-#: that reaches an installed deployment at all.
-NAV: tuple[tuple[str, str], ...] = (
-    ("/", "Status"),
-    ("/setup", "Setup"),
-    ("/activity", "Activity"),
-    ("/insights", "Insights"),
-    ("/rules", "Rules"),
-    ("/venues", "Venues"),
-    ("/gates", "Gates"),
-    (DOCS_URL, "Docs"),
-)
-
-_STYLE = """
-:root {
-  /* #532: `--good` and `--bad` were `#1f5f4f`/`#96322a`, luminances 0.0904/0.0893 -- a 1.01:1
-     ratio, i.e. profit and loss were told apart by hue alone (WCAG 1.4.1). The direction of
-     the fix matters: on a light background, moving a colour DOWN in luminance moves it AWAY
-     from the background (more contrast) while moving it UP moves it toward the background
-     (less contrast) -- so separating two dark colours by moving one of them lighter buys
-     separation by SPENDING contrast, while moving one of them darker buys separation and
-     contrast in the same move. `--bad` had contrast to spend (7.22:1 on `--bg`, comfortably
-     past the 7:1 AAA line); `--good` did not (7.17:1, already barely AAA), so `--bad` is the
-     one that moves. `--good` stays `#1f5f4f`, untouched, still 7.17:1 / 7.48:1 AAA.
-
-     REJECTED: lightening `--good` toward `--bg` instead of darkening `--bad` away from it. An
-     earlier draft of this fix did exactly that (`--good` -> `#237e38`) and reached a 0.0663
-     separation, but paid for it by moving `--good` DOWN to 4.89:1 -- AA, not AAA, a grade this
-     palette did not need to spend since `--bad` had the same separation available for free.
-     Caught by review, not by the contrast-ratio tests below: every ratio in the rejected draft
-     still cleared its WCAG floor, because "still passes AA" and "did not lose a grade it
-     already had" are different properties, and only the latter is what this repo's
-     documentation standard would call a decision made in the wrong direction. See
-     `test_no_text_pair_grade_drops_below_its_pinned_floor` in
-     tests/web/test_palette_contrast.py, added specifically because ratio-floor tests alone
-     could not have caught this mistake.
-
-     REJECTED, second time: darkening `--bad` all the way to `#4d1711` (luminance 0.0223).
-     That cleared AAA against `--bg`/`--card` (13.92:1 / 14.52:1) and a 0.0681 delta from
-     `--good`, but traded one photometric collision for another: `#4d1711` sits almost on top
-     of `--fg` (`#1c1b19`, luminance 0.0110) -- 1.19:1 against it, down from the original
-     `#96322a`'s 2.28:1. Every unhighlighted number in the same table renders in `--fg`, so in
-     greyscale, on e-ink, or for a red-green colour-deficient reader, that draft made a LOSS
-     indistinguishable from a neutral cell, which is the same shape of bug #532 exists to fix,
-     just moved to a different pair of tokens. `--bg` (0.9566) and `--fg` (0.0110) sit at
-     opposite ends of the luminance scale, and `--good` (0.0904) already occupies nearly the
-     only luminance band that is simultaneously AAA-against-`--bg` and clearly separated from
-     `--fg` -- there is no second, equally dark value that fits both properties AND stays far
-     from `--good`. `--bad` settles at `#7b2915` (luminance 0.0585): AAA on both surfaces
-     (9.28:1 `--bg` / 9.68:1 `--card`), 1.78:1 against `--fg` (real separation, clearly past
-     the 1.19:1 collision, short of `#96322a`'s coincidental 2.28:1 -- which is coincidental
-     precisely because `#96322a` sat almost on top of `--good`, the bug this whole fix exists
-     to remove), and a 0.0319 delta from `--good` -- smaller than `#4d1711`'s 0.0681 but 29x
-     the original 0.0011, and no longer a second collision. tests/web/test_palette_contrast.py
-     pins a floor against BOTH regressions now: `_MIN_GOOD_BAD_LUMINANCE_DELTA` for good/bad,
-     `_MIN_SIGNAL_FG_RATIO` for every signal token against `--fg`.
-
-     `--accent` was also byte-identical to `--good` in both themes, so a link and a gain
-     rendered the same colour; it gets its own blue, `#1a5578`, dark enough to clear AAA too
-     (7.70:1 `--bg` / 8.03:1 `--card`) rather than settle for AA now that it no longer has to
-     equal `--good`. */
-  --bg: #fbfaf8; --fg: #1c1b19; --muted: #6b6862; --line: #e3dfd8;
-  --card: #ffffff; --accent: #1a5578; --warn: #8a5a00; --bad: #7b2915; --good: #1f5f4f;
-  /* #532: `.field input, .field select` puts the control's background on `--bg` (the page
-     background), so `--line` at 1.27:1 was the only thing marking a form control's boundary --
-     below WCAG 1.4.11's 3:1 floor for non-text UI components. Raising `--line` itself was
-     rejected: `--line` also draws table rules, the footer border and card edges, which SC
-     1.4.11 explicitly exempts as decorative, and raising it would have widened all of those
-     for no accessibility gain. `--control-line`, `#84817c` here (3.72:1 on `--bg`), is scoped
-     to interactive control boundaries only. */
-  --control-line: #84817c;
-}
-:root:not([data-theme="light"]) { color-scheme: light dark; }
-@media (prefers-color-scheme: dark) {
-  :root:not([data-theme="light"]) {
-    /* Mirror image of the light-mode fix, and the direction flips with the background: dark
-       mode's background is dark, so moving a colour UP in luminance is the move away from it.
-       `--good` (`#6fbf9f`) had contrast to spend, 8.39:1 on `--bg`, past AAA with room; `--bad`
-       (`#e07a6a`) did not -- 6.24:1, already only AA -- so `--good` is the one that moves this
-       time: `#6fbf9f` -> `#83d3b2`, luminance 0.4314 -> 0.5463, now 10.39:1 on `--bg` / 9.70:1
-       on `--card` (AAA, up from AAA). Delta from `--bad` is 0.2382 (was 0.1234). `--bad` stays
-       `#e07a6a`, untouched, still 6.24:1 / 5.83:1 AA.
-
-       REJECTED: darkening `--bad` toward `--bg` instead -- the dark-mode mirror of the light
-       draft rejected above, for the same reason: `--bad` is already sitting on the AA floor it
-       cannot afford to spend, while `--good` has AAA headroom to give.
-
-       Lightening `--good` has one side effect, milder than light mode's `--bad`/`--fg` mistake
-       above but the same shape: `--good` is now closer to `--fg` (`#ecead5`), so `--good` on
-       `--fg` drops from 1.80:1 (at `#6fbf9f`) to 1.45:1 (at `#83d3b2`). Left as-is rather than
-       re-picked, because 1.45:1 is real, visible separation -- nothing like light mode's
-       1.19:1, which was nearly a collision -- and `_MIN_SIGNAL_FG_RATIO["dark"]` in
-       tests/web/test_palette_contrast.py pins 1.4 (a small margin under the measured 1.45) as
-       the floor going forward, so a future change that pushes it lower fails the build instead
-       of drifting.
-
-       `--accent` gets its own blue, distinct from `--good` (green) and `--bad` (salmon) as
-       in light mode -- separated from `--good` by HUE, not luminance (their luminance delta is
-       0.125, and that is fine: only `--good`/`--bad` need a luminance floor, because that pair
-       is what a red-green colour-deficient reader cannot otherwise tell apart; blue-against-
-       green carries no such risk, so no luminance floor is pinned between `--accent` and
-       `--good`). `#7aa8e0` was tried first and clears AAA on `--bg` (7.41:1) but only AA on
-       `--card` (6.92:1) -- the pairing that actually renders as button text
-       (`color: var(--card)` on `background: var(--accent)`).
-
-       REJECTED: darkening `#7aa8e0` further to try to reach AAA on `--card`. This repeats the
-       exact mistake the `--good`/`--bad` fix above exists to avoid: dark mode's background is
-       dark, so darkening a colour moves it TOWARD the background and loses contrast, not
-       toward some other hue -- darkening a blue keeps it blue, it just gets less readable.
-       `--accent` has headroom to spend the same way `--good` did: LIGHTENING it moves away
-       from `--bg` and gains contrast on both surfaces at once. `#86b1e5` -- lighter, still
-       unmistakably blue -- reaches 8.22:1 on `--bg` and 7.68:1 on `--card`, AAA on both, with
-       zero grades spent anywhere in either theme. */
-    --bg: #16150f; --fg: #ecead5; --muted: #9a968a; --line: #2f2d25;
-    --card: #1d1c15; --accent: #86b1e5; --warn: #d9a441; --bad: #e07a6a; --good: #83d3b2;
-    --control-line: #706d66;
-  }
-}
-* { box-sizing: border-box; }
-body { margin: 0; background: var(--bg); color: var(--fg);
-  font: 15px/1.55 ui-sans-serif, -apple-system, "Segoe UI", Roboto, sans-serif; }
-header { border-bottom: 1px solid var(--line); padding: 0.85rem 1.25rem; display: flex;
-  flex-wrap: wrap; gap: 0.4rem 1.1rem; align-items: baseline; }
-header .brand { font-weight: 650; letter-spacing: 0.02em; margin-right: 0.6rem; }
-header a { color: var(--muted); text-decoration: none; padding: 0.15rem 0;
-  border-bottom: 2px solid transparent; }
-header a:hover { color: var(--fg); }
-header a.on { color: var(--fg); border-bottom-color: var(--accent); }
-main { max-width: 62rem; margin: 0 auto; padding: 1.5rem 1.25rem 4rem; }
-h1 { font-size: 1.4rem; margin: 0 0 0.25rem; }
-h2 { font-size: 1.05rem; margin: 2rem 0 0.6rem; }
-.sub { color: var(--muted); margin: 0 0 1.5rem; font-size: 0.9rem; }
-.card { background: var(--card); border: 1px solid var(--line); border-radius: 10px;
-  padding: 0.9rem 1.1rem; margin: 0 0 1rem; }
-.grid { display: grid; grid-template-columns: repeat(auto-fit, minmax(11rem, 1fr)); gap: 0.75rem; }
-.kv { display: flex; flex-direction: column; gap: 0.15rem; }
-.kv .k { color: var(--muted); font-size: 0.78rem; text-transform: uppercase;
-  letter-spacing: 0.05em; }
-.kv .v { font-size: 1.05rem; font-variant-numeric: tabular-nums; }
-.tablewrap { overflow-x: auto; }
-table { border-collapse: collapse; width: 100%; font-size: 0.9rem; }
-th, td { text-align: left; padding: 0.42rem 0.7rem 0.42rem 0; border-bottom: 1px solid var(--line);
-  white-space: nowrap; }
-th { color: var(--muted); font-weight: 550; font-size: 0.76rem; text-transform: uppercase;
-  letter-spacing: 0.05em; }
-td.num { text-align: right; font-variant-numeric: tabular-nums; padding-right: 1.1rem; }
-.pill { display: inline-block; padding: 0.05rem 0.5rem; border-radius: 999px; font-size: 0.78rem;
-  border: 1px solid var(--line); }
-.good { color: var(--good); } .warn { color: var(--warn); } .bad { color: var(--bad); }
-.muted { color: var(--muted); }
-.empty { color: var(--muted); padding: 1.5rem 0; }
-.note { color: var(--muted); font-size: 0.85rem; margin: 0.4rem 0 0; }
-dl.terms dt { font-weight: 600; margin-top: 1rem; }
-dl.terms dd { margin: 0.2rem 0 0; }
-dl.terms dd.src { color: var(--muted); font-size: 0.82rem; }
-footer { border-top: 1px solid var(--line); color: var(--muted); font-size: 0.8rem;
-  padding: 1rem 1.25rem; }
-pre { white-space: pre-wrap; word-break: break-word; margin: 0; font-size: 0.85rem; }
-.job pre { margin-top: 0.6rem; max-height: 22rem; overflow-y: auto; }
-form { margin: 0.5rem 0 0; }
-.field { display: flex; flex-direction: column; gap: 0.2rem; margin: 0.6rem 0; max-width: 26rem; }
-.field span { font-size: 0.8rem; color: var(--muted); }
-.field input, .field select { font: inherit; padding: 0.4rem 0.6rem; border-radius: 7px;
-  border: 1px solid var(--control-line); background: var(--bg); color: var(--fg); }
-.field em { font-style: normal; font-size: 0.78rem; color: var(--muted); }
-button { font: inherit; font-weight: 550; padding: 0.35rem 0.9rem; border-radius: 7px;
-  border: 1px solid var(--accent); background: var(--accent); color: var(--card);
-  cursor: pointer; }
-button:hover { filter: brightness(1.08); }
-"""
-
-
-def esc(value: Any) -> str:
-    """Everything reaching the page goes through here. Rule names, product ids and adapter error
-    strings all originate outside this process; none of them is trusted markup."""
-    return html.escape("" if value is None else str(value), quote=True)
-
-
-def utc(ts: float | int | None, *, fmt: str = "%Y-%m-%d %H:%M:%S") -> str:
-    """UTC, always, and labelled as such wherever it is shown.
-
-    keel's day boundaries are UTC everywhere -- gates, scoping, the activity feed. Rendering in
-    local time is what made the activity feed show a stale date (#381): the gate said one day and
-    the rendering said another, and a "today" view could be permanently empty as a result."""
-    if ts is None:
-        return "--"
-    try:
-        return time.strftime(fmt, time.gmtime(float(ts)))
-    except (OverflowError, OSError, ValueError):
-        return "--"
-
-
-def age(ts: float | int | None, now_ts: float | int | None) -> str:
-    """How long ago, coarsely. A timestamp answers "when"; only this answers "is it stale"."""
-    if ts is None or now_ts is None:
-        return ""
-    delta = int(float(now_ts) - float(ts))
-    if delta < 0:
-        return "in the future"
-    if delta < 90:
-        return f"{delta}s ago"
-    if delta < 5400:
-        return f"{delta // 60}m ago"
-    if delta < 172800:
-        return f"{delta // 3600}h ago"
-    return f"{delta // 86400}d ago"
-
-
-def money(value: Decimal | None, *, places: int = 2) -> str:
-    """Display only -- never arithmetic. Rule 3 of the thinness pin forbids operating on a
-    `Decimal` in this layer, and the reports hand over every figure already computed."""
-    if value is None:
-        return "--"
-    return f"{value:,.{places}f}"
-
-
-def pct(value: Decimal | float | None, *, places: int = 2) -> str:
-    if value is None:
-        return "--"
-    return f"{value:.{places}f}%"
-
-
-def pnl_cell(value: Decimal | None) -> str:
-    """The journal's P&L cell: a glyph, the signed amount, and a colour class -- in that order
-    of how much of the meaning each one carries alone.
-
-    #532: colour alone failed WCAG 1.4.1 -- `--good`/`--bad` were once photometrically
-    identical in light mode (1.01:1), and even corrected they are separated by luminance a
-    red-green colour-deficient reader may still not resolve reliably by hue. `money()` already
-    prints the sign (`-12.34`, never `12.34` with an implied minus), so the SIGN survived colour
-    removal already; the GLYPH is what this function adds, because a sign is one character a
-    skimmed table row can miss where `▲`/`▼` at the start of the cell cannot. Strip every
-    colour from the page (greyscale, e-ink, `prefers-contrast`) and `▲ 120.00` / `▼ 45.00`
-    still read as gain and loss; strip the glyph instead and only the minus sign is left to
-    carry it, which is exactly the "distinguished by one easily-missed detail" state #532 was
-    filed to fix for colour."""
-    if value is None:
-        # "--", never "0.00": a trade with no recorded net is not a break-even trade.
-        return "--"
-    tone = "good" if value >= 0 else "bad"
-    glyph = "▲" if tone == "good" else "▼"
-    return f'{glyph} {esc(money(value))}'
-
-
-def kv(key: str, value: str, *, tone: str = "") -> str:
-    cls = f' class="v {tone}"' if tone else ' class="v"'
-    return f'
{esc(key)}{value}
' - - -def table(headers: Sequence[tuple[str, bool]], rows: Iterable[Sequence[str]]) -> str: - """`headers` is `(label, numeric)`; cell values are ALREADY escaped by the caller, because - several columns are deliberately markup (a tone span, a pill).""" - head = "".join( - f'{esc(label)}' if numeric else f"{esc(label)}" - for label, numeric in headers - ) - body_rows = [] - for row in rows: - cells = "".join( - f'{cell}' if numeric else f"{cell}" - for cell, (_, numeric) in zip(row, headers, strict=False) - ) - body_rows.append(f"{cells}") - if not body_rows: - return "" - return ( - '
' - + head - + "" - + "".join(body_rows) - + "
" - ) - - -def page( - *, - title: str, - path: str, - body: str, - build: str = "", - version: str = "", - refresh_sec: int | None = None, -) -> str: - """The document shell. `refresh_sec` emits a `` -- a zero-JS - auto-update, which keeps the "no scripts at all" property that makes this page auditable and - trivially freezable. The cost is a full reload rather than a patch; on a page that is a few - kilobytes of local HTML, that is not a cost.""" - nav_items = [] - for href, label in NAV: - on = ' class="on"' if href == path else "" - # An outbound entry is navigation, not a subresource, so `default-src 'none'` does not - # reach it. `noopener` still does matter: a new tab opened without it holds a - # `window.opener` handle back to a trading console on a token-bearing origin. - # - # `?v=` carries the running build, exactly as `static/js/docs.js` does for the client: - # the site pins `main` while an operator runs a tagged release, so a linked page can - # describe behaviour their build does not have. That skew is made VISIBLE rather than - # solved -- the build ends up in the URL bar of the page they are reading. `quote`, not - # an f-string, because a full version is `0.11.2+c1634a3fa17f` and a raw `+` in a query - # string decodes to a space. - # - # `version`, NOT `build`, and the two are different strings: `build` is the footer's - # human-readable LINE -- `keel 0.11.2+c1634a3fa17f (DIRTY) [checkout]` -- and the first - # spelling of this used it, putting that whole sentence percent-encoded into the query. - # Caught by looking at the rendered href, not by a test: both forms are non-empty - # strings and every assertion about "the link carries a version" passed. - away = "" - target = href - if href.startswith("https://"): - away = ' target="_blank" rel="noopener noreferrer"' - if version: - target = href + "?v=" + quote(version, safe="") - nav_items.append(f'{esc(label)}') - nav = "".join(nav_items) - meta_refresh = ( - f'' if refresh_sec else "" - ) - refresh_note = ( - f"this page reloads every {int(refresh_sec)}s" if refresh_sec else "read-only view" - ) - return ( - f"{esc(title)} - keel" - '' - '' - f"{meta_refresh}" - f"" - f'
keel{nav}
' - f"
{body}
" - f"
{esc(refresh_note)}" - + (f" · {esc(build)}" if build else "") - + " · keel does not give financial advice
" - ) - - -def _tone_for_rail(status: str) -> str: - lowered = (status or "").lower() - if "breach" in lowered or "halt" in lowered or "trip" in lowered: - return "bad" - if "warn" in lowered or "near" in lowered: - return "warn" - return "good" - - -def render_status(report: Any) -> str: - """The `keel status` report, as the landing page -- the same `StatusReport` the CLI renders - to lines and `--json` serialises, never a re-gather.""" - autonomy = report.autonomy - autonomy_text = "on" if getattr(autonomy, "enabled", False) else "off" - parts = [ - f'

Status

{esc(utc(report.now_ts))} UTC

', - '
', - kv("mode", esc(report.mode)), - kv( - "kill switch", - "ENGAGED" if report.kill_switch_engaged else "clear", - tone="bad" if report.kill_switch_engaged else "good", - ), - kv("autonomy", esc(autonomy_text), tone="warn" if autonomy_text == "on" else "muted"), - kv("rail 11", esc(report.rail11_status), tone=_tone_for_rail(report.rail11_status)), - "
", - '
', - kv("high water mark", esc(money(report.high_water_mark))), - kv("drawdown (total)", esc(pct(report.drawdown_total_pct))), - kv("drawdown (weekly)", esc(pct(report.drawdown_weekly_pct))), - kv("max total dd", esc(pct(report.max_total_dd_pct))), - kv("max weekly dd", esc(pct(report.max_weekly_dd_pct))), - kv("paper cash", esc(money(report.paper_cash_usdc))), - "
", - ] - - attestation = report.withdrawal_attestation - expired = bool(getattr(attestation, "expired", False)) - parts.append( - '
' - + kv( - "withdrawal attestation (rail 17)", - "EXPIRED" if expired else "fresh", - tone="bad" if expired else "good", - ) - + "
" - ) - - parts.append("

Open positions

") - position_rows = [ - ( - esc(getattr(pos, "product_id", "")), - esc(getattr(pos, "rule_name", "") or "--"), - esc(money(getattr(pos, "qty", None), places=8)), - esc(money(getattr(pos, "entry_fill", None))), - esc(utc(getattr(pos, "opened_at", None), fmt="%Y-%m-%d %H:%M")), - ) - for pos in report.open_positions - ] - parts.append( - table( - ( - ("product", False), - ("rule", False), - ("qty", True), - ("entry", True), - ("opened (UTC)", False), - ), - position_rows, - ) - or '

No open positions.

' - ) - - parts.append("

Rules

") - counts = " · ".join( - f'{esc(name)} {esc(count)}' - for name, count in sorted(report.rule_counts.items()) - ) - parts.append(f'

{counts or "no rules"}

') - live_rows = [ - ( - esc(getattr(rule, "name", "")), - esc(getattr(rule, "status", "")), - esc(getattr(rule, "kind", "")), - ) - for rule in report.live_rules - ] - parts.append( - table((("live rule", False), ("status", False), ("kind", False)), live_rows) - or '

No live rules.

' - ) - - parts.append("

Data freshness

") - fresh_rows = [ - ( - esc(getattr(row, "product_id", "")), - esc(getattr(row, "granularity", "") or "--"), - esc(utc(getattr(row, "last_ts", None), fmt="%Y-%m-%d %H:%M")), - esc(age(getattr(row, "last_ts", None), report.now_ts)), - ) - for row in report.data_freshness - ] - parts.append( - table( - ( - ("product", False), - ("granularity", False), - ("last candle (UTC)", False), - ("age", False), - ), - fresh_rows, - ) - or '

No market data yet.

' - ) - - parts.append("

Subscriptions

") - sub_rows = [ - ( - esc(getattr(row, "venue", "") or getattr(row, "name", "")), - esc(getattr(row, "status", "")), - esc(utc(getattr(row, "attested_ts", None), fmt="%Y-%m-%d")), - ) - for row in report.subscriptions - ] - parts.append( - table((("venue", False), ("status", False), ("attested (UTC)", False)), sub_rows) - or '

No subscription attestations.

' - ) - return "".join(parts) - - -def render_activity(feed: Any) -> str: - """The activity feed. Everything shown here is `ActivityFeed`'s own vocabulary -- including - the non-`ok` statuses, which are rendered as prose rather than suppressed: `missing` is the - commonest state on a fresh install and is not an error, and hiding it would leave a user - staring at a blank panel with nothing to act on.""" - explain = { - "missing": "No log file yet. This is normal before the first cycle -- it also happens when " - "keel is run from a directory that is not the deployment folder.", - "empty": "The log exists but the window held no records.", - "unparseable": "Lines were read, but none of them was a JSON record.", - "oversized": "The bounded tail read landed inside a single record, so nothing whole " - "survived it.", - "unreadable": "The log could not be read.", - } - parts = [ - '

Activity

' - + esc(feed.source or "no source") - + " · " - + esc(feed.scope) - + " · UTC

" - ] - if feed.status != "ok": - detail = f" {esc(feed.detail)}" if feed.detail else "" - parts.append( - f'
{esc(feed.status)} ' - f'{esc(explain.get(feed.status, ""))}{detail}
' - ) - - rows = [] - for cycle in feed.cycles: - tone = "muted" if cycle.is_quiet else "" - label = cycle.cycle_id or "uncorrelated" - rows.append( - ( - f'{esc(utc(cycle.started_ts, fmt="%m-%d %H:%M:%S"))}', - esc(label[:12]), - esc(cycle.mode or "--"), - esc(", ".join(cycle.products) or "--"), - esc(cycle.signals), - esc(cycle.blocked), - esc(cycle.entered), - esc(cycle.exited), - f'{esc(cycle.errors)}' if cycle.errors else "0", - esc("; ".join(cycle.highlights)), - ) - ) - parts.append( - table( - ( - ("started (UTC)", False), - ("cycle", False), - ("mode", False), - ("products", False), - ("signals", True), - ("blocked", True), - ("entered", True), - ("exited", True), - ("errors", True), - ("highlights", False), - ), - rows, - ) - or '

Nothing in this scope.

' - ) - - notes = [] - if feed.cycles_out_of_scope: - notes.append(f"{feed.cycles_out_of_scope} cycle(s) hidden by the scope") - if not feed.scope_fully_covered: - notes.append( - "the bounded read did not prove it reached the scope boundary, so an empty view here " - "does not mean the scope was quiet" - ) - if feed.lines_skipped: - notes.append(f"{feed.lines_skipped} unusable line(s) skipped") - if feed.window_truncated: - notes.append("the log window was truncated") - if feed.cycles_dropped: - notes.append(f"{feed.cycles_dropped} cycle(s) beyond the display cap") - if feed.last_cycle_before_scope is not None: - notes.append( - "last cycle before this scope: " - + utc(feed.last_cycle_before_scope.started_ts, fmt="%Y-%m-%d %H:%M") - ) - if notes: - parts.append('

' + " · ".join(esc(note) for note in notes) + "

") - return "".join(parts) - - -def render_insights(report: Any, journal: Any) -> str: - account = report.account - parts = [ - f'

Insights

{esc(utc(report.now_ts))} UTC · ' - f"{esc(report.closed_trade_count)} closed trade(s)

", - '
', - kv("mode", esc(account.mode)), - kv("rail 11", esc(account.rail11_status), tone=_tone_for_rail(account.rail11_status)), - kv("high water mark", esc(money(account.high_water_mark))), - kv("drawdown (total)", esc(pct(account.drawdown_total_pct))), - kv("drawdown (weekly)", esc(pct(account.drawdown_weekly_pct))), - "
", - "

Rule track records

", - ] - rows = [] - for rule in report.rules: - significance = ( - 'n≥30' - if rule.significant - else 'below the n=30 floor' - ) - rows.append( - ( - esc(rule.rule_name), - esc(rule.status), - esc(rule.n_trades), - esc(f"{rule.win_rate:.1f}%"), - esc(money(rule.expectancy, places=4)), - esc(money(rule.profit_factor, places=2)), - esc(money(rule.max_drawdown)), - significance, - ) - ) - parts.append( - table( - ( - ("rule", False), - ("status", False), - ("trades", True), - ("win rate", True), - ("expectancy", True), - ("profit factor", True), - ("max dd", True), - ("sample", False), - ), - rows, - ) - or '

No rules with a track record yet.

' - ) - parts.append( - '

Below 30 closed trades a win rate is not yet distinguishable from ' - "random entry, which is why the sample column says so rather than leaving the number to " - "speak for itself.

" - ) - - parts.append( - f'

Journal

{esc(journal.total_count)} closed trade(s) total.

' - ) - journal_rows = [] - for entry in journal.entries: - journal_rows.append( - ( - esc(utc(entry.closed_at, fmt="%Y-%m-%d %H:%M")), - esc(entry.product_id), - esc(entry.rule_name or "--"), - esc(money(entry.qty, places=8)), - esc(money(entry.entry_fill)), - esc(money(entry.exit_fill)), - pnl_cell(entry.pnl_net), - esc(money(entry.fees, places=4)), - esc(entry.outcome), - ) - ) - parts.append( - table( - ( - ("closed (UTC)", False), - ("product", False), - ("rule", False), - ("qty", True), - ("entry", True), - ("exit", True), - ("net p&l", True), - ("fees", True), - ("outcome", False), - ), - journal_rows, - ) - or '

No closed trades.

' - ) - return "".join(parts) - - -def render_rules(rows: Sequence[dict[str, Any]]) -> str: - parts = ['

Rules

read-only · promotion happens in the CLI

'] - table_rows = [ - ( - esc(row.get("id")), - esc(row.get("kind")), - esc(row.get("status")), - f"
{esc(row.get('params'))}
", - ) - for row in rows - ] - parts.append( - table((("id", True), ("kind", False), ("status", False), ("params", False)), table_rows) - or '

No rules.

' - ) - return "".join(parts) - - -def render_venues(infos: Sequence[Any]) -> str: - """Capability rows, exactly as `keel brokers list` declares them -- what the ADAPTER says it - can do, never an inference about the operator's keys. A row here is not a claim that the venue - is configured or reachable (#233).""" - parts = [ - '

Venues

what each installed adapter declares — not whether ' - "it is configured

" - ] - rows = [] - for info in infos: - if info.error: - rows.append( - ( - esc(info.name), - 'adapter failed to construct', - f'{esc(info.error)}', - "", - "", - "", - ) - ) - continue - rows.append( - ( - esc(info.name), - esc(info.venue), - esc(info.deployment), - esc(", ".join(info.asset_classes) or "--"), - esc(", ".join(info.supported_orders) or "--"), - esc(info.package_version or "--"), - ) - ) - parts.append( - table( - ( - ("adapter", False), - ("venue", False), - ("deployment", False), - ("asset classes", False), - ("orders", False), - ("version", False), - ), - rows, - ) - or '

No adapters installed.

' - ) - return "".join(parts) - - -#: How each kind of setup step is introduced. The wording carries the whole argument of #437 -- -#: what a wizard may do for you, what it may only collect, and what it cannot touch at all. -_STEP_KIND_NOTE: dict[str, str] = { - "mechanical": "keel can do this for you.", - "judgement": ( - "Yours to decide. keel can record it; it must never choose it for you, and an " - "attestation without a cited source is refused exactly like a missing one." - ), - "off_venue": ( - "Happens in the venue's own dashboard, and keel cannot verify it -- the venue's API " - "does not expose it. Never shown as done here, because a green check that verifies " - "nothing turns an open risk into a false assurance." - ), -} - - -def _job_panel(job: Any) -> str: - """A running, finished or failed background job, as a panel. - - The progress lines are shown NEWEST LAST, unscrolled, exactly as the CLI prints them -- an - operator who has run `keel fetch` in a terminal should recognise what they are looking at - rather than have to learn a second vocabulary for the same thing. - - A failure stays on screen. The whole point of running something in the background is that - nobody was watching when it broke.""" - tone = {"running": "warn", "done": "good", "failed": "bad"}.get(job.state, "muted") - elapsed = f"{int(job.elapsed_sec)}s" - parts = [ - '
', - f'
{esc(job.key)}' - f'{esc(job.state)}
', - f'

{esc(elapsed)} elapsed

', - ] - if job.error: - parts.append(f'

{esc(job.error)}

') - if job.lines: - parts.append("
" + esc("\n".join(job.lines)) + "
") - elif job.is_running: - parts.append('

starting…

') - parts.append("
") - return "".join(parts) - - -def render_setup( - state: Any, - *, - actions: Sequence[Any] = (), - not_automated: dict[str, str] | None = None, - csrf: str = "", - ran: str = "", - job: Any = None, -) -> str: - """The first-run checklist, with a button for each MECHANICAL step and a command for every - other one. - - A button appears ONLY where `actions` carries one, and `actions` is - `keel.commands.setup.ACTIONS` -- a closed set that cannot contain a judgement or off-venue - step without failing a test. So this function cannot grow a button for "attest this asset" by - someone adding markup here: it would have to be added to the registry first, where the test - is.""" - by_key = {action.key: action for action in actions} - not_automated = not_automated or {} - parts = [ - "

Setup

", - f'

{esc(state.root)}

', - ] - if ran: - item = next((s for s in state.states if s.step.key == ran), None) - if item is not None: - done = item.done is True - parts.append( - f'
' - f"{esc(item.step.title)} " - f'{esc(item.detail)}
' - ) - if state.is_new: - parts.append( - '
There is no deployment here yet. ' - 'Nothing below has been done, which is exactly what a first ' - "run looks like. Work down the list; the paper stage places no orders at all." - "
" - ) - if job is not None: - parts.append(_job_panel(job)) - - nxt = state.next_step - if nxt is not None: - parts.append( - '
next' - f'{esc(nxt.step.title)}
' - f'

{esc(nxt.step.how)}

' - ) - else: - parts.append('
Nothing outstanding.
') - - for stage, heading, blurb in ( - ( - "paper", - "To run in paper", - "Evaluates rules against real market data and places nothing.", - ), - ("live", "To go live", "Everything the go-live runbook adds before real money moves."), - ): - items = [item for item in state.states if item.step.stage.value == stage] - parts.append(f"

{esc(heading)}

") - parts.append(f'

{esc(blurb)}

') - rows = [] - for item in items: - if item.done is True: - mark = 'done' - elif item.done is False: - mark = 'to do' - else: - mark = 'not determined' - note = _STEP_KIND_NOTE.get(item.step.kind.value, "") - body = ( - f"{esc(item.step.title)}" - f'
{esc(item.detail)}
' - ) - action = by_key.get(item.step.key) - # A running job owns its step: offering the button again would invite a second start - # that the job slot refuses anyway, which reads as the page ignoring the click. - running_here = job is not None and job.is_running and job.key == item.step.key - if running_here: - body += '
running — see the panel above
' - elif item.blocking and action is not None and csrf: - body += _action_form(action, csrf) - elif item.blocking: - body += f"
{esc(item.step.how)}
" - if item.step.key in not_automated: - body += ( - '
Not a button, deliberately: ' - f"{esc(not_automated[item.step.key])}
" - ) - body += f'
{esc(item.step.why)} {esc(note)}
' - rows.append((mark, f'{esc(item.step.kind.value)}', body)) - parts.append(table((("", False), ("kind", False), ("step", False)), rows)) - return "".join(parts) - - -def _action_field(field: Any) -> str: - """One input. A `choices` field renders as a select whose FIRST option is an empty, - disabled, selected placeholder -- so the form opens with no valid answer chosen. - - That placeholder is the whole point for a judgement. A checkbox for "does it pay yield?" - would open unticked, and unticked is `no`, which is the PERMISSIVE answer -- a form whose - default is the compliant one attests on the operator's behalf. A select that starts on - "choose…" and is `required` cannot be submitted without someone answering. - """ - label = f"{esc(field.label)}" - hint = f"{esc(field.hint)}" if getattr(field, "hint", "") else "" - choices = getattr(field, "choices", ()) - if choices: - options = '' + "".join( - f"" for choice in choices - ) - control = f'' - else: - kind = "password" if field.secret else "text" - extra = ' autocomplete="off" spellcheck="false"' if field.secret else "" - control = f'' - return f'' - - -def _action_form(action: Any, csrf: str) -> str: - """One action key, one write token, and only the fields the action itself declares. - - A field marked `secret` renders as `type="password"` and is NEVER given a `value` -- not even - on a re-render after a failure. Pre-filling a secret field puts the secret in the page source, - where it survives a screenshot, a "view source", and anything that saves the page. The cost is - that a failed submission must be retyped; that is the correct cost. - - NO field is ever given a `value`, secret or not, and no select opens on a valid option. An - action over a judgement step records what the operator supplied and nothing else, and a - pre-filled form is the shortest route to recording something they did not. - """ - fields = "".join(_action_field(field) for field in getattr(action, "inputs", ())) - return ( - f'' - f'' - f'
{esc(action.detail)}
' - f"{fields}" - f'' - "" - ) - - -def render_gates(gates: Sequence[Any], capabilities: Sequence[Any]) -> str: - """The capability inventory (#436), rendered from `keel/capabilities.py`. - - This page is the reason a browser view can be honest about its own limits. The read surface - here cannot reach a single one of these actions -- the server implements no write verb at all - -- and the page says so, next to the list of what it cannot do and who can.""" - parts = [ - '

Gates

every action that increases what keel can do without ' - "asking again

", - '
This view cannot perform any of them. ' - 'The server answers GET and HEAD and implements no other verb, so ' - "there is no request it can accept that changes anything. Each action below needs a " - "human at a terminal.
", - ] - for gate in gates: - covered = [cap for cap in capabilities if cap.gate == gate.name] - parts.append(f"

{esc(gate.name)} · {esc(len(covered))} action(s)

") - parts.append( - '
evidence required' - f'{esc(gate.evidence)}
' - f'

Fails closed against {esc(gate.fails_closed_against)}.

' - f'

Implemented once, at {esc(gate.implementation)}.

' - "
" - ) - rows = [] - for cap in covered: - mirror = ( - f'mirrors {esc(cap.mirrors[1])}' if cap.mirrors else "" - ) - rows.append( - ( - f'{esc(cap.surface)}', - f"{esc(cap.invocation)} {mirror}", - esc(cap.increases), - f"{esc(cap.module)}.{esc(cap.function)}", - ) - ) - parts.append( - table( - (("surface", False), ("action", False), ("grants", False), ("call site", False)), - rows, - ) - ) - return "".join(parts) - - -def render_message(heading: str, detail: str) -> str: - return f'

{esc(heading)}

{esc(detail)}

' diff --git a/keel/web/security.py b/keel/web/security.py index f52da20..36298b3 100644 --- a/keel/web/security.py +++ b/keel/web/security.py @@ -58,6 +58,18 @@ #: a name that silently disables the cookie is worse than a plain name that works. SESSION_COOKIE = "keel_session" +#: The request header carrying the CSRF token on a write (#540). +#: +#: A HEADER rather than a body field, and the difference is the whole reason this layer still +#: earns its place now that the write surface is JSON. The token used to ride in a `
` as a +#: hidden input, where its job was to prove the submission came from a page keel rendered. There +#: is no form any more -- so putting it in the JSON body would prove only that the sender could +#: read the token, while putting it in a header ALSO proves the sender could set a header, which +#: a cross-origin form cannot do at all and a cross-origin `fetch` cannot do without surviving a +#: preflight. The same request now clears `X-Keel-Client` and this by the same mechanism, which +#: is redundancy rather than duplication: they fail independently. +CSRF_HEADER = "X-Keel-CSRF" + #: Hostnames that mean "this machine" and are therefore acceptable in a `Host:` header when the #: server is bound to a loopback address. Anything else -- including a hostname that RESOLVES to #: 127.0.0.1 -- is rejected, which is the entire point of checking the header at all. diff --git a/keel/web/server.py b/keel/web/server.py index 4031740..6533141 100644 --- a/keel/web/server.py +++ b/keel/web/server.py @@ -40,20 +40,19 @@ from __future__ import annotations -import functools import json import socket import sys -import time from collections.abc import Callable from dataclasses import dataclass from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer from pathlib import Path from typing import Any -from urllib.parse import parse_qs, quote, urlsplit +from urllib.parse import parse_qs, urlsplit -from keel.web import api, events, render, staticfiles +from keel.web import api, events, staticfiles from keel.web.security import ( + CSRF_HEADER, SESSION_COOKIE, HostPolicy, csrf_token, @@ -69,7 +68,7 @@ #: Cap on a form body. `rfile.read(n)` with an attacker-supplied `n` is a memory-exhaustion #: primitive and there is no proxy in front of this server to impose a limit. A setup form carries #: an action key and a token. -_MAX_FORM_BYTES = 8 * 1024 +_MAX_BODY_BYTES = 8 * 1024 #: How often the setup page reloads WHILE a background job runs. Shorter than the dashboards' #: 15s: someone watching a fetch wants to see it moving, and the page is a few kilobytes of local @@ -150,161 +149,8 @@ def ensure_schema(db_path: str) -> None: conn.close() -def page_setup(cfg: ServeConfig, query: dict[str, list[str]]) -> tuple[str, str, int | None]: - from keel.commands import jobs - from keel.commands.setup import ACTIONS, NOT_AUTOMATED_YET - - job = jobs.status() - return ( - "Setup", - render.render_setup( - api.deployment_state(cfg), - actions=ACTIONS, - not_automated=NOT_AUTOMATED_YET, - csrf=csrf_token(cfg.token), - ran=(query.get("ran") or [""])[0], - job=job, - ), - # Auto-refresh ONLY while something is running. A finished page that kept reloading would - # fight a reader, and the zero-JS meta refresh is the only progress mechanism available - # to a page that ships no scripts. - _JOB_REFRESH_SEC if job is not None and job.is_running else None, - ) - - -def needs_database( - page: Callable[[ServeConfig, dict[str, list[str]]], tuple[str, str, int | None]], -) -> Callable[[ServeConfig, dict[str, list[str]]], tuple[str, str, int | None]]: - """Serve the checklist instead of building a page that has no database to build from. - - Every page below this reads tables, and `sqlite3.connect` CREATES the file it cannot find -- - so without this a first-run user clicking "Activity" would get a 500 *and* leave an empty - `keel.db` behind, brought into existence by a read-only view being looked at. Found by - smoke-testing an empty directory; the unit tests missed it because they only exercised the - landing page. - - The guard is on the whole set rather than on the landing page alone for the same reason the - thinness pin globs a directory: a page added later gets the behaviour by construction, not by - its author remembering.""" - - @functools.wraps(page) - def guarded(cfg: ServeConfig, query: dict[str, list[str]]) -> tuple[str, str, int | None]: - if not api.deployment_state(cfg).has_usable_database: - # The full setup page, not a bare checklist: someone who lands here has nothing set - # up, and the actions are the reason they are being shown this instead of a 500. - return page_setup(cfg, query) - return page(cfg, query) - - return guarded - - -def page_status(cfg: ServeConfig, _query: dict[str, list[str]]) -> tuple[str, str, int | None]: - from keel.commands.status import gather_status - - repo = api.open_repo(cfg.db_path) - try: - config = api.load_config(cfg.config_path) - report = gather_status(repo, config, now_ts=int(time.time())) - finally: - api.close_repo(repo) - return "Status", render.render_status(report), _REFRESH_SEC - - -def page_activity(cfg: ServeConfig, query: dict[str, list[str]]) -> tuple[str, str, int | None]: - from keel.commands.activity import ( - apply_scope, - feed_from_lines, - normalise_scope, - read_log_window, - resolve_log_path, - ) - - scope = normalise_scope((query.get("scope") or [""])[0]) - config = api.load_config(cfg.config_path) - path = resolve_log_path(config) - window = read_log_window(path) - feed = feed_from_lines(window.lines, source=str(path), truncated=window.truncated) - if window.status != "ok" and feed.status == "empty": - # A read that failed and a window that held nothing are different facts; the reader's - # status is the more specific one and must not be flattened into "empty". - feed = feed_from_lines((), source=str(path)) - feed = apply_scope(feed, scope, now_ts=time.time()) - body = render.render_activity(feed) - links = " · ".join( - ( - f"{render.esc(name)}" - if name == scope - else f'{render.esc(name)}' - ) - for name in ("today", "7d", "all") - ) - return "Activity", body + f'

scope: {links}

', _REFRESH_SEC - - -def page_insights(cfg: ServeConfig, _query: dict[str, list[str]]) -> tuple[str, str, int | None]: - from keel.commands.insights import build_insights_report, build_journal_report - from keel.commands.status import gather_status - - repo = api.open_repo(cfg.db_path) - try: - config = api.load_config(cfg.config_path) - now_ts = int(time.time()) - status_report = gather_status(repo, config, now_ts=now_ts) - insights = build_insights_report(repo, config, status_report, now_ts) - journal = build_journal_report(repo, status_report, now_ts, limit=_JOURNAL_LIMIT) - finally: - api.close_repo(repo) - return "Insights", render.render_insights(insights, journal), None - - -def page_rules(cfg: ServeConfig, _query: dict[str, list[str]]) -> tuple[str, str, int | None]: - repo = api.open_repo(cfg.db_path) - try: - rows = repo.get_rules(None) - finally: - api.close_repo(repo) - return "Rules", render.render_rules(rows), None - - -def page_venues(_cfg: ServeConfig, _query: dict[str, list[str]]) -> tuple[str, str, int | None]: - from keel.commands.brokers import list_installed_brokers - - return "Venues", render.render_venues(list_installed_brokers()), None - - -def page_gates(_cfg: ServeConfig, _query: dict[str, list[str]]) -> tuple[str, str, int | None]: - """Read from `keel.capabilities`, which is a pure declaration -- no config, no database, no - network. It describes the binary that is serving the page.""" - from keel.capabilities import CAPABILITIES, GATES - - return "Gates", render.render_gates(GATES, CAPABILITIES), None - - -ROUTES: dict[str, Callable[[ServeConfig, dict[str, list[str]]], tuple[str, str, int | None]]] = { - # First-run detection (#437): every page that reads the database serves the checklist when - # there is no database to read, rather than a 500 whose real cause is that the user has not - # set anything up yet. `/venues` and `/gates` are not wrapped -- neither touches the - # deployment, and both are useful before one exists. - "/": needs_database(page_status), - "/setup": page_setup, - "/activity": needs_database(page_activity), - "/insights": needs_database(page_insights), - "/rules": needs_database(page_rules), - "/venues": page_venues, - "/gates": page_gates, -} - - -#: The write surface, in full, today. A path here maps to one `keel.commands.setup.Action`; there -#: is no other way into this handler, and no other verb. -SETUP_ACTION_PREFIX = "/setup/" - -#: The JSON API (#534). `GET` under this prefix routes through `keel/web/api.py`'s own table -- -#: reads only, one bounded read per endpoint. `POST` under it is unchanged from #535: it clears -#: `_api_client_header_ok` (the third CSRF layer, scoped to this prefix specifically -- see that -#: method's docstring for why it must NOT also gate `SETUP_ACTION_PREFIX`) and then meets the same -#: 404 every unmapped path gets, because there is still no JSON write surface and this issue added -#: none. +#: The JSON API's prefix (#534). Everything under it answers in `application/json`, admitted or +#: refused; nothing under it is a browsing context, so nothing under it carries a CSP. API_PREFIX = "/api/" #: The one path under `API_PREFIX` that is a STREAM rather than a document (#537). @@ -312,13 +158,30 @@ def page_gates(_cfg: ServeConfig, _query: dict[str, list[str]]) -> tuple[str, st #: It is not in `api.API_ROUTES` because it does not have that table's shape: every entry there #: maps to `(status, document)` through `api.respond`, and an `EventSource` connection is a #: response that never finishes. Bolting a "this one streams" flag onto `ApiRoute` would have put -#: a branch into the one function whose uniformity is the reason #536's `fetch` wrapper needs no -#: per-endpoint branch of its own. +#: a branch into the one function whose uniformity is the reason the client's `fetch` wrapper +#: needs no per-endpoint branch of its own. #: -#: It is still a GET, still behind `_admitted()`, and still answers no POST -- `do_POST` refuses -#: everything under `API_PREFIX` before it ever looks at a path. +#: It is a GET, behind `_admitted()`, and answers no POST: `do_POST` reaches only +#: `API_SETUP_PREFIX`. EVENTS_PATH = "/api/events" +#: The write surface, in full, today. A path here maps to one `keel.commands.setup.Action`; there +#: is no other POST this server answers, and `keel/web/__init__.py` is the file to read before +#: adding one. +#: +#: **It moved under `/api/` at #540**, from `/setup/`, when the HTML form that used to submit to it +#: was deleted along with the rest of the rendered pages. The move is what let `X-Keel-Client` -- +#: a header a plain form can never set -- finally cover the write path too: `_api_client_header_ok` +#: recorded that widening as "#536's call to make, once (and only once) the forms it replaces are +#: gone", and they are gone. +#: +#: The action SET is unchanged by all of this, and that is the invariant that matters more than the +#: path: `keel.commands.setup.ACTIONS` still contains only idempotent, non-destructive, +#: `MECHANICAL` steps, and a test still asserts it is disjoint from the eleven capability- +#: increasing actions in `keel/capabilities.py`. A browser can set a deployment up. It still cannot +#: arm a rule, attest an asset or enable autonomy. +API_SETUP_PREFIX = "/api/setup/" + def run_setup_action(cfg: ServeConfig, key: str, form: dict[str, str]) -> Any: """Perform one declared setup action. Returns its `ActionResult`, or `None` for a key that is @@ -338,39 +201,16 @@ def run_setup_action(cfg: ServeConfig, key: str, form: dict[str, str]) -> Any: # -- the handler ------------------------------------------------------------------------------- -#: Sent on every response, success or refusal. +#: The header set for the static tree -- which, since #540, is the whole application. #: -#: `default-src 'none'` with only `style-src 'unsafe-inline'` added states exactly what the page -#: is: markup and one inline stylesheet. No scripts, no images, no fonts, no connections. If a -#: future change smuggles in a script tag, the browser refuses it and the omission is visible -#: rather than silent. `frame-ancestors 'none'` (and the legacy `X-Frame-Options`) keep the page -#: out of an iframe on a hostile origin, which is the other half of the DNS-rebinding defence. -_SECURITY_HEADERS: tuple[tuple[str, str], ...] = ( - ( - "Content-Security-Policy", - # `form-action 'self'`, not `'none'`: the setup form posts back here, and `'none'` - # would have the browser silently refuse it. `'self'` is still the tightest value that - # works -- a form on this page cannot be made to submit anywhere else, which is what the - # directive is for. - "default-src 'none'; style-src 'unsafe-inline'; frame-ancestors 'none'; " - "form-action 'self'; base-uri 'none'", - ), - ("X-Content-Type-Options", "nosniff"), - ("X-Frame-Options", "DENY"), - ("Referrer-Policy", "no-referrer"), - ("Cache-Control", "no-store, max-age=0"), -) - - -#: The header set for `/static/*` (#535), separate from `_SECURITY_HEADERS` above because the -#: two routes need different values for the SAME header, not merely an additional one. -#: `_SECURITY_HEADERS`'s `default-src 'none'` is correct for the rendered pages -- they ship no -#: script, no style file, no image, nothing to permit -- but #536's client is exactly the thing -#: `'none'` forbids: its own JS, its own CSS, its own icons, all same-origin. `'self'` is the -#: tightest policy that still allows that, and `connect-src 'self'` on top of it is the specific -#: guarantee the design spec asks for: the interface is provably incapable of sending positions, -#: equity or trade history anywhere but this local process, checkable in the response headers -#: rather than merely promised. +#: There used to be a second set beside this one (`_SECURITY_HEADERS`) for the server-rendered +#: pages, whose `default-src 'none'` was correct for markup that shipped no script, no style file +#: and no image. Those pages are deleted and it went with them. What is left is the client, which +#: is exactly what `'none'` forbids: its own JS, its own CSS, its own icons, all same-origin. +#: `'self'` is the tightest policy that still allows that, and `connect-src 'self'` on top of it +#: is the specific guarantee the design spec asks for: the interface is provably incapable of +#: sending positions, equity or trade history anywhere but this local process, checkable in the +#: response headers rather than merely promised. #: #: `X-Frame-Options`, `Referrer-Policy` and `X-Content-Type-Options` are unconditional -- all #: three are meaningful (and harmless) on any content type, exactly as they are for the rendered @@ -404,9 +244,8 @@ def run_setup_action(cfg: ServeConfig, key: str, form: dict[str, str]) -> Any: #: `` is not a connection and would have sailed #: through, an injected `` could retarget every relative URL #: on the page, and the page could still be framed by a hostile origin -- the exact DNS-rebinding -#: half `_SECURITY_HEADERS` above already closes for the rendered pages -#: (`frame-ancestors 'none'` there is called "the other half of the DNS-rebinding defence"; this -#: is the same half, for the route that will host all of #536's JavaScript). +#: half the unconditional headers above already close (`frame-ancestors 'none'` is the other half +#: of the DNS-rebinding defence, and this is the same half for the route that hosts the client). _STATIC_CSP = ( "default-src 'self'; connect-src 'self'; form-action 'self'; base-uri 'none'; " "frame-ancestors 'none'" @@ -459,22 +298,6 @@ def run_setup_action(cfg: ServeConfig, key: str, form: dict[str, str]) -> Any: _JSON_CONTENT_TYPE = "application/json; charset=utf-8" -def _docs_version(cfg: ServeConfig) -> str: - """The build the nav's documentation link should report, or `""`. - - Read off `build_info` rather than off `cfg.build`, because those are different strings and - only one of them is a version: `cfg.build` is the footer's human-readable LINE - (`keel 0.11.2+c1634a3fa17f (DIRTY) [checkout]`), and putting it in a query string produced - `?v=keel%200.11.2%2B...%20%28DIRTY%29%20%5Bcheckout%5D`. This is the same field `/api/config` - hands the client for the same purpose (`payload.config_document`'s `"build"`), so both - front-ends report the identical string while both exist. - - `""` when there is no build info at all -- an unversioned link is honest, and a link claiming - `?v=unknown` is not. - """ - return str(getattr(cfg.build_info, "full_version", "") or "") - - def _static_headers(content_type: str) -> tuple[tuple[str, str], ...]: """`_STATIC_BASE_HEADERS` plus CSP, but ONLY when `content_type` is one of `_CSP_CONTENT_TYPES` -- see the comments on `_STATIC_BASE_HEADERS` and `_CSP_CONTENT_TYPES` @@ -497,6 +320,14 @@ class KeelHandler(BaseHTTPRequestHandler): #: Set by `build_server`. cfg: ServeConfig + #: Whether this request's body has already been read off the socket. + #: + #: Per REQUEST, not per connection, and reset at the top of each verb: a handler instance + #: serves every request on a keep-alive connection, so a flag left set by one write would make + #: the next refusal skip its drain and re-create the desynchronisation this exists to prevent. + #: `_drain_request_body` reads it; `_read_json_object` sets it. + body_consumed: bool = False + # -- logging -- def log_message(self, fmt: str, *args: Any) -> None: """Overridden to a near-silence, and NEVER with the query string. @@ -512,14 +343,21 @@ def _send( code: int, body: str, *, - content_type: str = "text/html; charset=utf-8", + content_type: str = "text/plain; charset=utf-8", extra: tuple[tuple[str, str], ...] = (), ) -> None: payload = body.encode("utf-8") self.send_response(code) self.send_header("Content-Type", content_type) self.send_header("Content-Length", str(len(payload))) - for name, value in _SECURITY_HEADERS: + # `_STATIC_BASE_HEADERS`, not a set of its own: the two responses this method now sends + # are the token-exchange redirect and a plain-text refusal, and neither is a browsing + # context. The header set that used to live here carried a CSP written for the rendered + # pages (`style-src 'unsafe-inline'`, for their inline stylesheet), and keeping a policy + # that describes a page nobody serves any more would be a comment pretending to be a + # defence. The shell keeps its CSP through `_static_headers`, which applies one to + # `text/html` -- and that is where the shell is served from now. + for name, value in _STATIC_BASE_HEADERS: self.send_header(name, value) for name, value in extra: self.send_header(name, value) @@ -531,10 +369,10 @@ def _send_json(self, code: int, document: dict[str, Any]) -> None: """One JSON response, with its own headers. Writes them itself rather than going through `_send` for the same reason `_serve_static` - does: `_send` puts `_SECURITY_HEADERS` on every response, and one of those is a CSP that - has no meaning on `application/json` (see `_API_HEADERS`). Sharing the method would have - meant a parameter with a default, and a default on a shared sender is how a header set - silently changes for a route nobody was thinking about. + does: the two senders need different values for the SAME headers, and sharing one method + would have meant a parameter with a default -- which is how a header set silently changes + for a route nobody was thinking about. `no-store` matters more here than anywhere else on + this server (see `_API_HEADERS`). A plain `json.dumps`: `keel/web/payload.py` normalises every leaf to a string before it gets here, so there is nothing for an encoder to convert -- and the encoder a hurried @@ -559,30 +397,71 @@ def _send_json(self, code: int, document: dict[str, Any]) -> None: def _refuse(self, code: int, heading: str, detail: str) -> None: """A refusal, in the media type the caller asked for by the path it used. - **Path-scoped, not method-scoped, and not content-negotiated.** An HTML error page handed - to a `fetch()` client's `res.json()` is a parse error in the client, which is a strictly - worse diagnostic than the 403 it is hiding -- so everything under `API_PREFIX` refuses in - JSON, including the POST that still 404s there. The GATE in front of that POST - (`_api_client_header_ok`) is untouched by this: same trigger, same status, same ordering; - only the body's media type follows the path. + **Path-scoped, not method-scoped, and not content-negotiated.** A refusal handed to a + `fetch()` client's `res.json()` in some other format is a parse error in the client, which + is a strictly worse diagnostic than the 403 it is hiding -- so everything under + `API_PREFIX` refuses in JSON. + + **Everything else refuses in PLAIN TEXT, since #540.** It used to refuse in HTML, through + `render.page`, and that module is gone: the server generates no markup at all now. Plain + text rather than JSON for these because the requests that land here are NAVIGATIONS -- a + person has opened `http://127.0.0.1:8765/` in a browser without the token, and the useful + answer is the sentence telling them to use the URL keel printed. A JSON envelope would + wrap that sentence in punctuation for a reader who is not a program. It is not HTML and + there is no template: two lines joined by a blank one. `Accept`-based negotiation was the alternative and it is worse here: a client that forgets - the header would get HTML from a JSON endpoint, and the one thing this server can be - certain about is which path was requested. + the header would get the wrong media type from a JSON endpoint, and the one thing this + server can be certain about is which path was requested. """ + # **Before anything is written.** See `_drain_request_body`: a refusal that leaves a + # request body unread poisons the next request on the same connection. + self._drain_request_body() if urlsplit(self.path).path.startswith(API_PREFIX): self._send_json(code, api.refusal_document(code, heading, detail)) return - self._send( - code, - render.page( - title=heading, - path="", - body=render.render_message(heading, detail), - build=self.cfg.build, - version=_docs_version(self.cfg), - ), - ) + self._send(code, f"{heading}\n\n{detail}\n") + + def _drain_request_body(self) -> None: + """Consume an unread request body, or close the connection rather than consume it. + + **This is keep-alive correctness, and its absence was a real bug.** `protocol_version` is + HTTP/1.1, so a connection is reused by default. A POST refused before its body is read + leaves those bytes sitting in the socket, and the stdlib then parses them as the NEXT + request line -- which is not a request line, so the next request dies as a 501. The symptom + is the worst kind: the refusal itself is correct, and the request AFTER it fails, with a + status that has nothing to do with either. + + It could not happen before #540 because the CSRF token was a field in the form body, so + every write read its body before it could refuse. Moving the token into a header (which is + what let it prove something a body cannot) put the refusal in front of the read. Found by + driving a browser, which reuses connections; every test in the suite opened a fresh one and + could not have seen it. + + **An oversized body is not drained, it is disconnected.** Reading it to be polite would + hand an attacker exactly the memory-exhaustion primitive `_MAX_BODY_BYTES` exists to deny. + Closing costs the client one reconnection and costs this server nothing. + """ + if self.body_consumed: + # Already off the socket -- `do_POST` reads the body before it can refuse for a reason + # that depends on the body's CONTENT (an unknown action key, an unreadable object). + # Reading it a second time blocks until the client's timeout, which is a hang rather + # than an error and was exactly what the first version of this did. + return + try: + length = int(self.headers.get("Content-Length") or 0) + except ValueError: + self.close_connection = True + return + if length <= 0: + return + if length > _MAX_BODY_BYTES: + self.close_connection = True + return + try: + self.rfile.read(length) + except OSError: # pragma: no cover - a client that vanished mid-body + self.close_connection = True # -- shared admission -- def _admitted(self) -> bool: @@ -635,25 +514,24 @@ def _sec_fetch_site_ok(self) -> bool: return True def _api_client_header_ok(self) -> bool: - """The third CSRF layer (#535), and scoped to `API_PREFIX` ONLY -- never to - `SETUP_ACTION_PREFIX`. This was gated at the top of `do_POST`, over every write, in an + """The third CSRF layer (#535). Scoped to `API_PREFIX`, which since #540 includes + the write path. This was gated at the top of `do_POST`, over every write, in an earlier version of this change, and that was a defect, not a stricter check: the shipped - UI's entire write surface IS a plain HTML `` - (`render.py`'s `_action_form`), and `_SECURITY_HEADERS` ships no `script-src` at all -- - there is no code path by which that form can set a custom request header. Gating - `/setup/*` on it would have refused every legitimate submission the shipped client can - make, with no fallback: the desktop bundle has no terminal. - `test_a_browser_form_post_succeeds_without_the_api_client_header` pins that this route - stays reachable from exactly what ships. + UI's entire write surface WAS a plain HTML ``, and + the rendered pages shipped no `script-src` at all -- so there was no code path by which + that form could set a custom request header, and gating it here would have refused every + legitimate submission the shipped client could make, with no fallback: the desktop bundle + has no terminal. + + **That form is deleted (#540) and this check now covers the write path**, which is what + the last paragraph below said should happen "once, and only once, the forms it replaces + are gone". `X-Keel-Client: 1` is a real defence where it CAN apply: a custom header forces a CORS preflight a hostile origin cannot satisfy, closing the one gap `SameSite=Strict` and the HMAC CSRF token both assume shut -- a plain form POST, which is never preflighted, in any - browser. But "a custom header" and "a `fetch()` client" are the same requirement, and - `/setup/*` has no `fetch()` client today. `API_PREFIX` is reserved for #533/#534's JSON - API, which #536's client speaks over `fetch()` -- that is where this check belongs, and - widening it onto `/setup/*` is #536's call to make, once (and only once) the forms it - replaces are gone.""" + browser. "A custom header" and "a `fetch()` client" are the same requirement, and every + write now comes from a `fetch()` client.""" if self.headers.get("X-Keel-Client") != "1": self._refuse( 403, @@ -748,16 +626,21 @@ def do_HEAD(self) -> None: # noqa: N802 - stdlib's naming, not ours self.do_GET() def do_POST(self) -> None: # noqa: N802 - stdlib's naming, not ours - """The ENTIRE write surface (#437). Read `keel/web/__init__.py` before extending it. - - Admission, then `Sec-Fetch-Site` (both apply to every POST), then a PATH-SCOPED branch: - `API_PREFIX` additionally requires `X-Keel-Client` before falling through to the 404 no - route there answers yet; `SETUP_ACTION_PREFIX` does not, and reads its own CSRF token - instead (see `_api_client_header_ok`'s docstring for why the two paths differ -- it is - not an oversight, it is the fix for one). Either way there is no dynamic dispatch here, - no getattr on a user-supplied name, and no path that reaches keel other than - `keel.commands.setup.ACTIONS` -- which contains three idempotent, non-destructive, - `MECHANICAL` steps and cannot contain anything else without failing a test.""" + """The ENTIRE write surface (#437, moved under `/api/` at #540). Read + `keel/web/__init__.py` before extending it. + + Five checks, in this order, and every one of them applies to every write now -- which is + the change #540 brought. While the write surface was an HTML form at `/setup/*`, the + `X-Keel-Client` gate could not cover it (a form cannot set a header) and the two paths had + to differ; `_api_client_header_ok`'s docstring called that "the fix for one" defect and + said the widening was to happen "once, and only once, the forms it replaces are gone". + They are gone. + + There is no dynamic dispatch here, no getattr on a user-supplied name, and no path that + reaches keel other than `keel.commands.setup.ACTIONS` -- which contains only idempotent, + non-destructive, `MECHANICAL` steps and cannot contain anything else without failing a + test.""" + self.body_consumed = False parsed = urlsplit(self.path) if not self._admitted(): return @@ -765,70 +648,95 @@ def do_POST(self) -> None: # noqa: N802 - stdlib's naming, not ours if not self._sec_fetch_site_ok(): return - if parsed.path.startswith(API_PREFIX): - if not self._api_client_header_ok(): - return - # No JSON API write surface exists yet (#533/#534 land the reads; a write is - # further out still) -- this is a 404 like any other unmapped path, not a stub - # success. Checked here, ahead of that surface existing, so its first action does - # not have to remember to add the gate. + if not parsed.path.startswith(API_PREFIX): + # Not "method not allowed" -- there is no write surface at this path at all, and + # saying so is both true and less informative to someone probing. self._refuse(404, "No such action", f"Nothing accepts a POST at {parsed.path}.") return - if not parsed.path.startswith(SETUP_ACTION_PREFIX): - # Not "method not allowed" -- there is no write surface at this path at all, and - # saying so is both true and less informative to someone probing. + if not self._api_client_header_ok(): + return + + if not parsed.path.startswith(API_SETUP_PREFIX): self._refuse(404, "No such action", f"Nothing accepts a POST at {parsed.path}.") return - body = self._read_form() - if not tokens_match(body.get("csrf"), csrf_token(self.cfg.token)): - # `SameSite=Strict` already stops a cross-site POST in any current browser. This is - # the layer that does not depend on the browser being current. + # The HMAC layer, "the layer that does not depend on the browser being current". It rides + # in a HEADER rather than in the body, so a request that reaches this line has proved the + # same thing twice -- a form can set neither header, and a cross-origin `fetch` cannot get + # past the preflight that either of them triggers. + if not tokens_match(self.headers.get(CSRF_HEADER), csrf_token(self.cfg.token)): self._refuse( 403, "Refused", - "That form did not carry this session's write token. Reload the page and try " + "That request did not carry this session's write token. Reload the page and try " "again.", ) return - key = parsed.path[len(SETUP_ACTION_PREFIX) :] + values = self._read_json_object() + if values is None: + self._refuse(400, "Unreadable request", "The body was not a JSON object of fields.") + return + + key = parsed.path[len(API_SETUP_PREFIX) :] try: - result = run_setup_action(self.cfg, key, body) + result = run_setup_action(self.cfg, key, values) except Exception as exc: + # A failed step is a stated 500 with a JSON body, never a traceback down the socket + # and never a 200 whose body says it went fine. self._refuse(500, "That step could not be completed", f"{type(exc).__name__}: {exc}") return if result is None: self._refuse(404, "No such action", f"{key!r} is not a setup step keel performs.") return - # POST/redirect/GET: a browser reload must not re-submit. The actions are idempotent, so - # a re-submission would be harmless -- but "harmless" is not a reason to leave a - # re-submitting page in a setup flow someone is clicking nervously. - # - # The Location carries the step KEY and nothing else. A submitted value in a redirect URL - # is a secret in browser history, in the Referer header of anything the page later loads, - # and in any proxy log between here and nowhere -- which is the whole reason the form is a - # POST in the first place. - self._send(303, "", extra=(("Location", f"/setup?ran={quote(result.step_key)}"),)) - - def _read_form(self) -> dict[str, str]: - """The urlencoded body, bounded. - - Bounded because `rfile.read(n)` with an attacker-supplied `n` is a memory-exhaustion - primitive, and this server has no proxy in front of it to impose a limit. A setup form - carries an action key and a token; anything past a few kilobytes is not one.""" + # No POST/redirect/GET any more, and nothing to redirect TO: the page that used to be + # reloaded is a client view that re-reads `/api/setup` itself. The actions are idempotent, + # so a repeated submission is harmless by construction rather than by a redirect. + self._send_json(200, api.action_document(result)) + + def _read_json_object(self) -> dict[str, str] | None: + """The JSON request body as `{field: string}`, or `None` for anything else. + + Bounded, because `rfile.read(n)` with an attacker-supplied `n` is a memory-exhaustion + primitive and this server has no proxy in front of it to impose a limit. A setup action + carries a handful of named fields; anything past a few kilobytes is not one. + + **Every value is coerced to `str`, and a nested object or array makes the whole body + `None`.** `run_setup_action` hands these to an `Action.run`, and a field that arrived as a + list or a dict would reach code written for a string -- the shape of bug that shows up as + a `TypeError` deep inside a step that has already half-run. JSON replaced urlencoded form + bodies at #540 along with the form that sent them; this is the one place that difference + can be exploited, so it is the one place it is refused. + """ try: length = int(self.headers.get("Content-Length") or 0) except ValueError: + return None + if length < 0 or length > _MAX_BODY_BYTES: + return None + if length == 0: + # An action with no declared inputs is submitted with an empty body, and that is a + # legitimate request rather than a malformed one. return {} - if length <= 0 or length > _MAX_FORM_BYTES: - return {} + self.body_consumed = True raw = self.rfile.read(length).decode("utf-8", "replace") - return {key: values[0] for key, values in parse_qs(raw, keep_blank_values=True).items()} + try: + parsed = json.loads(raw) + except ValueError: + return None + if not isinstance(parsed, dict): + return None + values: dict[str, str] = {} + for name, value in parsed.items(): + if isinstance(value, (dict, list)): + return None + values[str(name)] = "" if value is None else str(value) + return values def do_GET(self) -> None: # noqa: N802 - stdlib's naming, not ours + self.body_consumed = False parsed = urlsplit(self.path) query = parse_qs(parsed.query, keep_blank_values=True) @@ -885,41 +793,15 @@ def do_GET(self) -> None: # noqa: N802 - stdlib's naming, not ours self._send_json(code, document) return - if parsed.path.startswith(staticfiles.STATIC_PREFIX): - # Same admission as every rendered page (never weakened): a static asset is not - # exempted from the loopback-plus-session model just because it holds no secrets - # today. #536's client is what actually reads these, and it authenticates the same - # way any other fetch from this origin does -- the session cookie already on the - # request. - self._serve_static(parsed.path) - return - - handler = ROUTES.get(parsed.path) - if handler is None: - self._refuse(404, "No such page", f"Nothing is served at {parsed.path}.") - return - - try: - title, body, refresh = handler(self.cfg, query) - except Exception as exc: # a broken page must not take the server down - self._refuse( - 500, - "That page could not be built", - f"{type(exc).__name__}: {exc}", - ) - return - - self._send( - 200, - render.page( - title=title, - path=parsed.path, - body=body, - build=self.cfg.build, - version=_docs_version(self.cfg), - refresh_sec=refresh, - ), - ) + # Everything else is the client: a file under the static root, or one of the seven names + # `CLIENT_ROUTES` answers with the shell. `_serve_static` 404s anything that is neither, + # which is what keeps a mistyped asset a 404 rather than a 200 carrying HTML. + # + # This is the last branch because the prefix stopped narrowing anything when #540 moved + # the mount to `/`: `staticfiles.STATIC_PREFIX` matches every path now, so the ORDER of + # these branches -- events, then `/api/`, then the client -- is what separates them, not + # the prefixes themselves. + self._serve_static(parsed.path) class KeelServer(ThreadingHTTPServer): diff --git a/keel/web/static/css/keel.css b/keel/web/static/css/keel.css index de68061..1bb56c4 100644 --- a/keel/web/static/css/keel.css +++ b/keel/web/static/css/keel.css @@ -524,3 +524,43 @@ footer { transition-duration: 0.01ms !important; } } + +/* ── action forms (#540) ────────────────────────────────────────────────────────────────────── + The client performs setup actions itself now; the server-rendered page that used to hold these + forms is deleted. The rules below are the ones that page carried, moved rather than reinvented. + + `--control-line`, NOT `--line`. WCAG 1.4.11 asks 3:1 for the boundary of a component you can + interact with, and `--line` is 1.27:1 against the page -- fine for a decorative divider, which + is exempt, and not fine for the only thing marking where an input is. A control whose border is + the only cue that it IS a control, drawn at a ratio a divider may use, is the exact failure + `test_field_input_border_uses_the_control_line_token_not_line` exists to catch. */ +.action-form { margin: 0.75rem 0 0; } +.field { display: flex; flex-direction: column; gap: 0.25rem; margin: 0 0 0.75rem; } +.field label { color: var(--muted); font-size: 0.85rem; } +.field input, +.field select { + background: var(--card); + color: var(--fg); + border: 1px solid var(--control-line); + border-radius: 4px; + padding: 0.4rem 0.5rem; + font: inherit; + max-width: 32rem; +} +.field input:disabled, +.field select:disabled { opacity: 0.6; } +button.run { + background: var(--accent); + color: var(--bg); + border: 1px solid var(--accent); + border-radius: 4px; + padding: 0.45rem 0.9rem; + font: inherit; + cursor: pointer; +} +button.run:disabled { cursor: progress; opacity: 0.7; } +/* Empty until an action finishes, and it must occupy no space while empty: this is an + `aria-live` region that has to exist in the DOM before the text it announces is put in it, so + it is present on every card from the moment the card is drawn. */ +.action-outcome { margin: 0.5rem 0 0; } +.action-outcome:empty { margin: 0; } diff --git a/keel/web/static/index.html b/keel/web/static/index.html index f6ec75c..6ac1615 100644 --- a/keel/web/static/index.html +++ b/keel/web/static/index.html @@ -7,16 +7,14 @@ - + - + - - - + + + @@ -64,13 +62,13 @@ keel