diff --git a/CHANGELOG.md b/CHANGELOG.md index afee8f006..961524dd6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,6 +1,29 @@ # Changelog -## 1.0.2-beta.9 — 2026-08-27 +## 1.0.2 — 2026-08-27 + +The stable cut of the `1.0.2-beta.*` line, which stays documented in its own +sections underneath. The entries below shipped in no beta — `next` stopped at +`1.0.2-beta.8` — so they reach users for the first time here. + +Worth stating plainly for anyone arriving from 1.0.1: **`policies` is now the +one noun**, and the package ships NO policies of its own. `policy`, `pack` and +`p` still reach the same command, but a policy set is something you add +(`failproofai policies add FailproofAI/policies`) or publish +(`failproofai publish`) rather than something the install carries. A pack is +pinned by the digest of its artifact and by the commit it was built from, and a +pack that cannot be loaded DENIES within the scope it declared rather than +disappearing quietly. + +### Docs + +- The documentation is rewritten against the shipped 1.0.2 CLI. An audit of all 68 English pages plus the README found 219 factual errors, and the whole of `start/` documented a setup path that no longer exists: bare `failproofai config` — the verb that installs the daemon and wires every supported agent CLI — appeared on no page, and neither did `failproofai policies add FailproofAI/policies`, so a reader who followed the quickstart end to end finished with hook entries, no daemon, and one enforcing policy. The retired `pack`/`policy` spellings, `pack add core`, `--bundled` and `config --connect --token ` are gone; the local no-account path (the dashboard on `localhost:8020`, local findings from `failproofai audit`, and separately documented telemetry controls) is documented for the first time; and `start/integrations.mdx`, which five pages linked to and nothing in the navigation reached, is wired in (#756) +- Observe mode is documented as what it is: the policy is EVALUATED for real, under the same timeout and error handling as an enforcing one, and its verdict recorded — only the enforcement is withheld. Both layers that set it are named, `failproofai publish --effect observe` for a pack and `fp fleet deploy --add :observe` for a Cloud deployment, and every page showing `--add` now says that a bare one ENFORCES immediately, which is how a shadow rollout turns into a production incident. `policies/deploy.mdx` had described the workflow entirely as dashboard clicks and named no `fp` command at all; the CLI lane — `fp fleet list/show/deploy/diff/history/rollback`, `fp guardrails summary|timeline`, and `fp policies test`, which decides a policy locally with no server and no auth — is now covered (#756) +- Enforcement is no longer described as uniform across the twelve harnesses. `docs/index.mdx` claimed "the same events, the same policies, and the same session history apply to every one" — the exact claim `src/hooks/enforcement-capability.ts` exists to stop drifting. A `PreToolUse` deny is verified to stop the tool on all twelve; a `Stop` deny is verified on eight, is `observe` on Pi, and is unverified on OpenCode, Hermes and Goose. Goose has no `Stop` event and we install none on Hermes, so the five `require-*-before-stop` builtins never fire on either. A pair with no row in that file now reads as unverified rather than as blocking (#756) +- The install no longer claims that policies come with it. "39 built-in policies activate immediately" and "eleven are on when you accept the defaults" were both false: `configure-wizard.ts` enables nothing, so `enabledPolicies` is empty and `builtin-policies.ts` registers only what carries `alwaysOn` — exactly one policy, `block-failproofai-commands`. The eleven `defaultEnabled` flags are catalog metadata that decide nothing until a pack is installed (#756) +- The Policy Hub at befailproof.ai/policy-hub is documented, including that `failproofai publish` does not set the `failproofai-policies` topic the hub indexes on — so publishing is publish-then-tag-by-hand until it does (#756) +- Two claims copied out of the CLI's own `--help` are corrected, because the help is wrong. Scope support is not "Codex, Copilot, Cursor, OpenCode and Pi take user or project only": `types.ts` makes Hermes and OpenClaw **user-only** and gives `local` to Claude alone. And `backfill --help` names `~/.failproofai/config.toml`, a layout-2 file no current build writes; the file is `config.json` (#756) +- Six fabricated commands and flags were caught by an adversarial verification pass before they shipped, among them `fp fleet rename --name` (the command takes two positionals and declares no options), a `tool` filter on the local dashboard's activity view (`HookActivityFilters` has six keys and tool is not one), and `failproofai config --token --machine-label ` as a setup one-liner — which sets nothing up, since any invocation carrying `--machine-label` without `--connect` routes straight to the rename path and exits 1 on a machine that is not yet connected (#756) ### Fixes diff --git a/Cargo.lock b/Cargo.lock index ebde6aa63..37522615d 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -202,7 +202,7 @@ dependencies = [ [[package]] name = "failproofaid" -version = "1.0.2-beta.10" +version = "1.0.2" dependencies = [ "fpai-collect", "fpai-ipc", @@ -265,7 +265,7 @@ dependencies = [ [[package]] name = "fpai-collect" -version = "1.0.2-beta.10" +version = "1.0.2" dependencies = [ "notify", "reqwest", @@ -280,7 +280,7 @@ dependencies = [ [[package]] name = "fpai-ipc" -version = "1.0.2-beta.10" +version = "1.0.2" dependencies = [ "libc", "proptest", diff --git a/Cargo.toml b/Cargo.toml index e7ef09ff2..36372af46 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -3,7 +3,7 @@ resolver = "3" members = ["crates/*"] [workspace.package] -version = "1.0.2-beta.10" +version = "1.0.2" edition = "2024" license-file = "LICENSE" repository = "https://github.com/FailproofAI/failproofai" diff --git a/README.md b/README.md index 636d2666b..d2be7306a 100644 --- a/README.md +++ b/README.md @@ -14,11 +14,11 @@ **Translations:** [简体中文](./docs/i18n/README.zh.md) · [日本語](./docs/i18n/README.ja.md) · [한국어](./docs/i18n/README.ko.md) · [Español](./docs/i18n/README.es.md) · [Português](./docs/i18n/README.pt-br.md) · [Deutsch](./docs/i18n/README.de.md) · [Français](./docs/i18n/README.fr.md) · [Русский](./docs/i18n/README.ru.md) · [हिन्दी](./docs/i18n/README.hi.md) · [Türkçe](./docs/i18n/README.tr.md) · [Tiếng Việt](./docs/i18n/README.vi.md) · [Italiano](./docs/i18n/README.it.md) · [العربية](./docs/i18n/README.ar.md) · [עברית](./docs/i18n/README.he.md) -**Observability and enforcement for every harness your agents run in.** -Wherever your agents run, we see it — and we can say no. Failproof hooks 12 agent -harnesses — coding CLIs like Claude Code and Codex, chat gateways like Hermes, -self-hosted assistants like OpenClaw — capturing every run and blocking dangerous -tool calls before they execute. 39 built-in policies. Zero latency. Runs locally. +**See what your agents do. Stop known failures before they repeat.** +Failproof AI works wherever your agents run: coding tools like Claude Code and +Codex, chat gateways like Hermes, self-hosted assistants like OpenClaw, and agents +you instrument yourself. It records each run and can block dangerous tool calls +before they execute. @@ -30,9 +30,9 @@ tool calls before they execute. 39 built-in policies. Zero latency. Runs locally ## Supported harnesses -Twelve harnesses in two classes — ten coding CLIs, and two chat and assistant -gateways (Hermes, OpenClaw). Same events, same policies, same session history, -whichever one your agent runs in. +Twelve harnesses in two classes are supported: ten coding CLIs, plus two +gateways: Hermes, OpenClaw. The policy API and session history are shared; which +events can block varies by harness. Agents that run in none of them report through the [Python SDK](https://docs.befailproof.ai/reference/custom-agents), which gives you tracing, sessions and audits. Enforcement there needs a hook in @@ -134,13 +134,31 @@ your own runtime — [talk to us](mailto:support@befailproof.ai) and we'll map i ## Install +Give a compatible agent the Failproof AI skill if you want it to guide setup, +inspect the machine, and route policy, audit, session, and Cloud work correctly: + +```sh +npx skills add FailproofAI/skills +``` + +This installs the umbrella skill and its specialist siblings. To install only the +umbrella, add `--skill failproofai`. Skills supply operating instructions; install +and configure the product itself with: + ```sh npm install -g failproofai -failproofai policies --install # or just run `failproofai` and accept the first-run prompt -failproofai +failproofai config +failproofai policies add FailproofAI/policies +failproofai # dashboard on localhost:8020 ``` -39 built-in policies activate immediately. Dashboard at `localhost:8020`. Disable the first-run prompt with `FAILPROOFAI_NO_FIRST_RUN=1`. +Setup connects supported agents and installs the background service. It chooses no +policy pack: before you add one, only `block-failproofai-commands` runs to stop an +agent disabling Failproof AI. + +Connect Cloud without prompts with `failproofai config --token `. On a +shared machine or in CI, set `FAILPROOFAI_CLOUD_TOKEN` and run `failproofai config` +so the key does not appear in command history. --- @@ -157,8 +175,8 @@ failproofai | `block-rm-rf` | Recursive file deletion | | `block-force-push` / `block-push-master` | `git push --force`, direct pushes to `main` | -The first five apply to any agent that can call a tool. The last three are the -developer favourites — coding CLIs are the harness class we cover deepest. +These policies protect files, credentials, infrastructure, databases, and agent +workflows. Exact enforcement support varies by harness and event. → [All 39 built-in policies](https://docs.befailproof.ai/policies/builtin) @@ -195,6 +213,33 @@ Three decisions available to every policy: --- +## Policy packs + +A policy pack is a versioned set of policies published from a public GitHub +repository. Inspect one before installing it: + +```sh +failproofai policies show FailproofAI/policies +failproofai policies add FailproofAI/policies +``` + +Anything with a slash is a pack source; anything without one is a policy name. +You can install selected categories or policies, and pin a release when needed. + +```sh +failproofai policies add FailproofAI/policies --category git,database +failproofai policies add owner/repo@a1b2c3d4e5f6 +``` + +Browse published packs in the [Policy Hub](https://befailproof.ai/policy-hub/), or +run `failproofai publish --init` to start your own. Observe mode lets a pack record +what it would have done without blocking: `failproofai publish --effect observe`. + +→ [Policy packs](https://docs.befailproof.ai/policies/packs) · +[Publish a pack](https://docs.befailproof.ai/policies/publish-a-pack) + +--- + ## Observability Enforcement is one half. The other half is seeing what the agent actually did. diff --git a/docs/admin/keys-and-permissions.mdx b/docs/admin/keys-and-permissions.mdx index 82ca36dc2..b95df0130 100644 --- a/docs/admin/keys-and-permissions.mdx +++ b/docs/admin/keys-and-permissions.mdx @@ -1,74 +1,54 @@ --- title: "Keys and permissions" -description: "Create scoped API keys for machines, automation, and operators." +description: "Create credentials for people, automation, and machines." icon: "key-round" --- -API keys belong to an organization and carry explicit permissions. Use separate keys for agent ingestion, policy delivery, evaluators, CI automation, and administrative scripts. +Use a separate key for each person, machine, or automation job. Grant only the permissions it needs. ## Create and rotate a key - - - 1. Go to **Administration → Keys**, select **new key**, and enter a workload name. - 2. Choose a permission set and adjust individual permissions only when the preset is insufficient. - 3. Create the key and copy its one-time secret immediately. - 4. Open the key later to update grants, disable it, or regenerate the secret. +Use **Admin → Keys**, or the Cloud CLI: - The creation drawer is where you choose the narrowest grants required by the workload. +```bash +fp keys list +fp keys create "audit automation" --add audits:read +fp keys disable "audit automation" +``` - ![The new API key drawer with permission presets and individual grants.](/images/dashboard/key-create.png) +Store the secret when it is created; it is not shown again. Rotate by creating a replacement, updating the consumer, then disabling the old key. - After creation, the Keys page shows the persistent metadata and management actions. The one-time secret is not shown again. +## Machine keys - ![The API Keys page showing key permissions, creation time, and regenerate and disable actions.](/images/dashboard/api-keys.png) +A machine key connects the local service to Cloud: - Use this list to review grants regularly and disable keys that no longer map to an active workload. - - - ```bash - fp keys create production-agents \ - --add events:add \ - --add policies:pull - fp keys show production-agents - fp keys update production-agents --add events:read - fp keys regenerate production-agents --yes - fp keys disable production-agents - ``` +```bash +export FAILPROOFAI_CLOUD_TOKEN="" +failproofai config +``` - Redirect or capture create/regenerate output securely; the secret is returned once. - - +A connected machine may need two capabilities: -The two permissions required by a connected Failproof AI machine are independent: +- `policies:pull` to receive Cloud-managed policies. +- `events:add` to send decisions and sessions. -- `events:add` sends events and session data. -- `policies:pull` retrieves assigned policy deployments. +Status reports these separately because one may work while the other does not. -Key secrets are shown when created or regenerated. Store them in a secret manager and rotate them without reusing an operator's interactive credentials. +## Common permissions -## Permission catalog - -| Area | Permissions | +| Permission | Allows | | --- | --- | -| Events | `events:add`, `events:read` | -| Keys | `keys:create`, `keys:read`, `keys:disable`, `keys:regenerate`; `keys:update` is human-session only | -| Users | `users:create`, `users:read`, `users:update`, `users:delete` | -| Evaluations | `evaluations:read`, `evaluations:trigger` | -| Dashboards | `dashboards:read`, `dashboards:write`, `dashboards:delete` | -| Queries | `queries:read`, `queries:write`, `queries:delete`, `queries:run` | -| Assistant | `agent:use` | -| Settings | `settings:read`, `settings:write` | -| Alerts | `alerts:read`, `alerts:write` | -| Issues | `issues:read`, `issues:create`, `issues:close` | -| Audits | `audits:read`, `audits:write` | -| Policies | `policies:read`, `policies:write`, `policies:pull` | -| Usage | `usage:read` | - -`orgs:admin` is reserved for the instance operator and cannot be granted to an organization key or ordinary member. Retired `incidents:*` and `alerts:ack` tokens are accepted for compatibility and normalize to current `issues:*` permissions. +| `events:add` | Send events | +| `events:read` | Read events and errors | +| `evaluations:read` | Read sessions and evaluations | +| `audits:read` / `audits:write` | Review or manage audits | +| `policies:read` / `policies:write` | Review or deploy policies | +| `policies:pull` | Pull machine policy assignments | +| `keys:create` / `keys:disable` | Create or disable keys | +| `orgs:admin` | Instance-level organization administration; not assignable to an organization key | -Builtin permission sets are `read-only`, `standard`, and `admin`. `standard` adds evaluation triggering, query execution, issue response, and assistant use to read permissions. Key creation strips human-only grants even when a permission set contains them. +Some administrative `fp fleet` and `fp guardrails` commands require a signed-in user session rather than an API key. Their help text states this before making a request. - Instance-scoped keys can select an organization with the `X-AgentEye-Org` header. Set it explicitly on multi-organization deployments; omission may select the default organization. + Never reuse ingest credentials such as `AGENTEYE_KEY` or `AGENTEYE_API_KEY` as an `FP_API_KEY`. They serve different systems and permissions. diff --git a/docs/admin/overview.mdx b/docs/admin/overview.mdx index 83125f00e..04c476d70 100644 --- a/docs/admin/overview.mdx +++ b/docs/admin/overview.mdx @@ -1,22 +1,30 @@ --- title: "Administration" -description: "Operate access, usage, organizations, and security without mixing them into the reliability workflow." +description: "Operate Cloud access, membership, settings, and consumption without mixing them into the reliability workflow." icon: "settings-2" --- -Administration contains the controls needed to run Failproof AI across a team. Most readers can stay in Sessions, Audits, and Policies; administrators use this section to manage access and operating boundaries. +Administration is the Cloud half of Failproof AI. Organizations, members, API keys, org settings, and metered consumption all need a Cloud connection. + + + Failproof AI also runs with no account at all. A local machine enforces policy, keeps its session history on disk, serves the [local dashboard](/reference/local-dashboard) on `localhost:8020`, and runs `failproofai audit` without sending anything anywhere. See [machine configuration](/reference/events-and-configuration#machine-configuration) for the local `collector` keys. Skip the rest of this section if you are not using Cloud. + + +## Where the controls live - Use the Cloud sidebar's **Admin** section for **policy editor**, **enforcement**, **usage**, **keys**, **users**, and **settings**. A locked item shows that your account is missing its read permission. + The Cloud sidebar's **admin** group holds **policy editor**, **enforcement**, **usage**, **keys**, **users**, and **settings**. - ![The Admin section of the Cloud sidebar with policy editor, enforcement, usage, keys, users, and settings.](/images/dashboard/admin-navigation.png) + ![The admin group of the Cloud sidebar with policy editor, enforcement, usage, keys, users, and settings.](/images/dashboard/admin-navigation.png) + Install the Cloud CLI as an isolated tool, then sign in: + ```bash + uv tool install fp-cloud-cli + fp login fp whoami - fp orgs current - fp orgs perms fp usage ``` @@ -24,17 +32,43 @@ Administration contains the controls needed to run Failproof AI across a team. M +## Signed-in human, or API key + +`fp` runs in one of two authentication modes, and the split decides which commands work. A command an API key cannot perform refuses before it opens a connection and names the reason, rather than letting a 401 or a 403 stand in for it. + +| Command | API key | Permission | +| --- | --- | --- | +| `fp whoami` | Yes | none | +| `fp usage` | Yes | `usage:read` | +| `fp keys list` / `show` | Yes | `keys:read` | +| `fp keys create` | Yes | `keys:create` | +| `fp keys disable` | Yes | `keys:disable` | +| `fp keys regenerate` | Yes | `keys:regenerate` | +| `fp keys update` | No | `keys:update`, which no key can hold | +| `fp users list` / `show` | Yes | `users:read` | +| `fp users create` | Yes | `users:create` | +| `fp users update` | Yes | `users:update` | +| `fp users disable` / `enable` | Yes | `users:delete` | +| `fp settings list` / `schema` | Yes | `settings:read` | +| `fp settings set` | Yes | `settings:write` | +| `fp orgs list` / `switch` / `current` / `perms` | No | signed-in session only | + +`fp usage` is the command to give a reporting script: it runs unattended under a key and needs only `usage:read`. + - - Inspect billing windows and organization consumption. - - Give machines and automation only the permissions they require. + Mint the machine credential, and give every workload only the grants it needs. - Manage membership, defaults, and organization boundaries. + Manage membership, and keep each organization's data and actions scoped. - Configure operational settings, data handling, and deployment security. + Set sign-in and alerting values, and decide what agent data leaves a machine. + + + Read what the organization metered during its current 30-day window. + + + See which machines are enrolled, how they are identified, and what they enforce. diff --git a/docs/admin/settings-and-security.mdx b/docs/admin/settings-and-security.mdx index 5809a4c9f..672ae7d37 100644 --- a/docs/admin/settings-and-security.mdx +++ b/docs/admin/settings-and-security.mdx @@ -1,68 +1,37 @@ --- title: "Settings and security" -description: "Configure operational settings and make deliberate choices about agent data." -icon: "lock-keyhole" +description: "Control organization defaults, data handling, and machine credentials." +icon: "shield" --- -Use Settings for deployment-specific operational values and model context-window overrides. Inspect the settings schema before changing a value through the API or CLI. +Organization settings affect everyone in that Cloud organization. Change them from **Admin → settings** or with `fp settings`. -## Change an organization setting +```bash +fp settings list +fp settings set +``` - - - 1. Go to **Administration → Settings**, find the setting group, and read its description and current source. - 2. Change the value and save it. - 3. For model context windows, add or update the model override and confirm the effective limit. - 4. Recheck sessions and metrics that depend on the changed value. +## Data handling - ![The Settings page showing organization authentication and operational settings.](/images/dashboard/settings.png) - - - ```bash - fp settings list - fp settings schema - fp settings set --value - fp settings set alerts.email_default_recipients \ - --json-value '["oncall@example.com"]' - ``` +A connected machine sends policy decisions and full session transcripts by default. To send decisions only, set `collector.sessions` to `false` in the machine's `~/.failproofai/config.json` after connecting. The current `--no-transcripts` setup flag does not apply that setting. - Run `fp settings set --help` for the value type and confirmation flags used by the installed CLI. - - +Credentials are scrubbed on the machine before upload, but redaction is a safety floor, not a substitute for access control. -## Data handling choices +Local-only use needs no account. Session history and audits stay on the machine, except scheduled local audits send machine identity and a limited digest after you opt in. Anonymous CLI telemetry can be disabled with `FAILPROOFAI_TELEMETRY_DISABLED=1`. -Connecting the Failproof AI CLI sends transcripts by default because traces and audits depend on their content. Use `--no-transcripts` when prompts, file contents, or terminal input must remain local; hook activity and policy decisions can still be reported. +## Credentials -Local ingest credentials are stored separately from non-secret daemon settings and written with restrictive permissions. API keys should still be managed as production secrets. +Machine tokens are stored under `~/.failproofai/` with owner-only permissions. They are not written into the service definition. -## Organization setting reference +Use separate keys for people, automation, and machines. Give each only the permissions it needs, rotate it when ownership changes, and revoke it when the machine or workflow is retired. -All dashboard-editable settings are organization-scoped. Deployment-wide behavior remains server environment configuration. +## Checklist -| Key | Default | Purpose | -| --- | --- | --- | -| `allowed_sign_ins` | `[]` | Restrict existing members to exact emails or `*@domain`; an empty list means no additional restriction. | -| `session_ttl_secs` | `86400` | Dashboard session lifetime; accepted range is 60 seconds to 30 days. | -| `otp_ttl_secs` | `600` | OTP and magic-link lifetime; accepted range is 60–1800 seconds. | -| `alerts.email_default_recipients` | `[]` | Default recipients when an alert email channel does not override them. | -| `alerts.slack_default_webhook` | empty | Default Slack incoming-webhook URL. | -| `alerts.webhook_default_url` | empty | Default generic JSON webhook URL. | -| `alerts.webhook_signing_secret` | empty | HMAC-SHA256 key used for the `X-AgentEye-Signature` header; reads are masked. | -| `alerts.enabled_channels` | email, Slack, webhook | Organization-wide channel kinds that alert rules may dispatch. | -| `default_user_permissions` | `standard` | Named permission set preselected for new invitations. | +- Require the narrowest useful permissions. +- Set `collector.sessions` to `false` where full content is unnecessary. +- Review who can deploy enforcing policies. +- Test policies in observe mode before enforcement. +- Keep machine labels clear and machine IDs stable. +- Run `failproofai config --status` after changing connection settings. -`allowed_sign_ins` is a filter, not a grant: a person must already be an organization member. Use an empty list to allow every member; the bare value `*` is rejected. Alert URLs must use HTTPS except for loopback development addresses. - -## Security checklist - -- Use HTTPS for Cloud connections. -- Scope keys to the smallest permission set. -- Separate production and non-production environments. -- Review transcript and redaction settings before rollout. -- Audit user, key, and organization changes. -- Test backup, retention, and incident response requirements for your deployment. - - - Turning transcript capture off changes what audits and investigations can prove. Record the decision and its expected limitations. - +See [keys and permissions](/admin/keys-and-permissions) for the permission catalog. diff --git a/docs/admin/usage.mdx b/docs/admin/usage.mdx index 056179495..d2a524fca 100644 --- a/docs/admin/usage.mdx +++ b/docs/admin/usage.mdx @@ -1,21 +1,24 @@ --- title: "Usage" -description: "Inspect organization consumption and the active billing window." +description: "Inspect organization consumption for the current 30-day metering window." icon: "chart-no-axes-combined" --- -Usage shows consumption for the current organization and its billing windows. Use it to understand how production rollout, transcript volume, evaluations, and audit cadence affect your plan. +Usage shows what the current organization has metered during its current 30-day window. The window is fixed and anchored per organization, and usage is read-only: it applies and displays no limits, quotas, or plan thresholds. Both surfaces say so — the CLI panel prints "read-only usage, no limits applied". ## Review usage - 1. Go to **Administration → Usage**. - 2. Confirm the organization and metering window. - 3. Review ingest, sessions, evaluations, metrics, audits, findings, alerts, users, and keys. - 4. Compare started and completed audit work when audit usage looks unexpected. + 1. Go to **admin → usage**. + 2. Confirm the organization and the metering window — the header shows the window's start and end, the day within it, and how much has elapsed. + 3. Read the hero block: events ingested, sessions, agents, and environments. + 4. Read the two pipelines: evaluations (with scores and metrics) and audits (with issues and alerts), each showing completed runs against started runs. + 5. Read the workspace and access blocks: saved queries, dashboards, alerts, unique issues, members, and API keys. - ![The Usage page showing the metering window, captured events, evaluation and audit pipelines, workspace resources, members, and API keys.](/images/dashboard/usage-overview.png) + The figures are cached, not live. Use the refresh control in the top right after a change you expect to see. + + ![The usage page showing the metering window, events captured, the evaluation and audit pipelines, workspace resources, members, and API keys.](/images/dashboard/usage-overview.png) ```bash @@ -23,15 +26,52 @@ Usage shows consumption for the current organization and its billing windows. Us fp --json usage fp --org reliability-team --json usage ``` + + `fp usage` needs `usage:read` and runs under an API key, which makes it the right command for a reporting cron. + + With `--json` it returns the dashboard response unchanged: `org_id`, `billing_anchor`, `window`, `usage`, `calculated_at`, and `stale_after`. `calculated_at` and `stale_after` are how you tell how old a figure is. + + `fp usage` takes no flags of its own — only the global `--json` and `--org`. It reports the current window and nothing before it, so keep your own history by running it on a schedule and storing the `--json` payload rather than expecting to query an earlier window later. + + ```bash + fp --json usage | jq '.usage.events_ingested' + ``` +### Metric keys + +Everything under `usage` in the JSON payload, so a script can name a field instead of parsing a panel. + +| Group | Keys | +| --- | --- | +| Telemetry | `events_ingested`, `sessions`, `agents`, `environments` | +| Evaluations | `evaluation_runs`, `evaluation_finishes`, `evaluations`, `metrics` | +| Audits | `audit_runs`, `audit_finishes`, `issues_created`, `alerts_created` | +| Workspace | `queries_created`, `dashboards_created` | +| Access | `users_active`, `users_created`, `keys_active`, `keys_created` | + +Each pipeline reports started and finished counts separately, so a gap between `audit_runs` and `audit_finishes` means work that began and did not complete — worth checking before you read anything into the issue counts beneath it. + ## Investigate a change -1. Confirm the active organization and window. -2. Compare the increase with session volume by environment. -3. Check whether a new integration started sending transcripts. -4. Review evaluator and audit cadence changes. -5. Compare against the limits on the current pricing plan. +Ingest volume is decided per machine, not centrally. Start from the organization and window, then work through what changed on the machines reporting into it. + +| Cause | Where to check | What to change | +| --- | --- | --- | +| A machine is sending session transcripts | `collector.sessions` in that machine's `~/.failproofai/config.json` | Set it to `false` to send decisions only | +| Hook activity is being sent in full | `collector.hooks_verbosity` in the same file | `decisions` aggregates allows per minute; `off` stops hook events entirely | +| A new capture location was added | `failproofai harness list` | Remove it with `failproofai harness remove-path ` | +| History was re-sent | The machine's `failproofai backfill` runs | Re-sends deduplicate on a content hash and collapse into rows already there, so they do not double-count | +| A new integration started reporting | The `agents` and `environments` counts, and the sessions list | Scope the new machine's collector settings before rolling it out | + +Usage compares nothing against a plan. Do that outside the product. -The CLI returns the same summary for scripts. + + + The per-machine data-handling settings behind these numbers. + + + The full `fp` surface, including installation and sign-in. + + diff --git a/docs/admin/users-and-organizations.mdx b/docs/admin/users-and-organizations.mdx index 2ff1dea77..5340f724f 100644 --- a/docs/admin/users-and-organizations.mdx +++ b/docs/admin/users-and-organizations.mdx @@ -4,18 +4,13 @@ description: "Control membership and keep each organization's data and actions s icon: "users" --- -Organizations isolate sessions, evaluations, audits, issues, alerts, queries, dashboards, users, and keys. Confirm the active organization before changing administrative resources. +Organizations isolate sessions, evaluations, audits, issues, alerts, queries, dashboards, users, and keys. Confirm the active organization before you change any administrative resource. -## Manage members and organizations +## Choose the organization - 1. Use the organization switcher at the top of the Cloud sidebar to change organizations. - 2. Go to **Administration → Users** to search members or filter by active state and role. - 3. Select **new user**, enter the email, choose a permission set, and adjust overrides if needed. - 4. Open a user later to update grants, disable sign-in, or enable the account again. - - ![The Users page showing member email addresses, permission sets, and edit and disable controls.](/images/dashboard/users.png) + Use the organization switcher at the top of the Cloud sidebar. Everything below it — including every page in **admin** — then reads and writes that organization. ```bash @@ -23,22 +18,62 @@ Organizations isolate sessions, evaluations, audits, issues, alerts, queries, da fp orgs switch reliability-team fp orgs current fp orgs perms + ``` + + `fp orgs switch` with no slug opens an arrow-key picker starting on your current organization; a non-interactive run falls back to a numbered prompt, and under `--json` a slug is required. The choice persists to `~/.failproofai/fpcli/cli-auth.json`, so later commands send it as the active tenant. Override it for one command with `--org ` or `FP_ORG`. + + + + + Every `fp orgs` command needs a signed-in human and refuses under an API key, because org membership belongs to a person and a key already acts for one organization. The `fp users` commands below work either way. + + +## Manage members + + + + 1. Go to **admin → users** and search members by email, or narrow the list with the `protected`, `admin`, `standard`, and `read-only` filter chips. + 2. Select **new user**, enter the email, choose a permission set, and adjust per-member overrides if needed. + 3. Open a member later to change grants, disable sign-in, or enable the account again. + ![The users page with the email search box, the filter chips, and a member card showing its grants grouped by resource.](/images/dashboard/users.png) + + The grant chips in this frame predate the `incidents:*` to `issues:*` rename; the [common permissions](/admin/keys-and-permissions#common-permissions) list is current. + + + ```bash + fp users list + fp users list --active-only fp users create engineer@example.com --permission-set standard fp users show engineer@example.com fp users update engineer@example.com --add audits:write fp users disable engineer@example.com fp users enable engineer@example.com ``` + + Members are addressed by email and resolved case-insensitively, so `fp users show Alice.Chen@Example.com` and `fp users create alice.chen@example.com` refer to the same person. -Administrators can create, update, disable, and re-enable users, then assign the permission set appropriate for their role. The API uses a delete operation for disabling, but it does not remove the account or its membership record. +### Grants, and what each verb needs + +Member grants use the same arithmetic as keys: effective grants are `(set ∪ added) − removed`, and `--add` / `--remove` take compact `slug:action.action` tokens where dotted actions expand. See [keys and permissions](/admin/keys-and-permissions). + +| Verb | Permission | Notes | +| --- | --- | --- | +| `fp users list` / `show` | `users:read` | `--active-only` hides disabled members. | +| `fp users create` | `users:create` | `--permission-set` seeds the role; `--add` / `--remove` layer overrides. | +| `fp users update` | `users:update` | `--permission-set` **replaces** the member's per-member overrides; `--add` / `--remove` alone are incremental against their current grants. | +| `fp users disable` / `enable` | `users:delete` | Both, including enable — grant it deliberately. | + +Disabling refuses two cases outright, as a Forbidden error rather than a validation message: a **protected** member cannot be disabled, and you cannot disable your own account. Disabling an already-disabled member, or enabling an already-active one, is a calm no-op. The API uses a delete operation for disabling, but it removes neither the account nor its membership record. + +Two organization settings govern membership: `default_user_permissions` preselects the permission set for a new invitation, and `allowed_sign_ins` gates which addresses can receive a sign-in code at all. Manage them from the [settings page](/admin/settings-and-security). Disabling a user blocks that identity from signing in to every organization, not only the organization currently selected. Re-enabling restores global sign-in and the member's permissions in this organization. - Give service accounts descriptive names tied to a workload and owner. Avoid sharing keys between organizations or between people and machines. + Give service accounts descriptive names tied to a workload and an owner. Do not share a key between organizations, or between a person and a machine. diff --git a/docs/audits/alerts.mdx b/docs/audits/alerts.mdx index d5b0cce8a..9ee07d610 100644 --- a/docs/audits/alerts.mdx +++ b/docs/audits/alerts.mdx @@ -1,10 +1,10 @@ --- title: "Alerts" -description: "Detect recurrence and route an incident to the right responders." +description: "Detect recurrence and route an issue to the right responders." icon: "bell-ring" --- -Alerts watch a measurable condition and create an incident when it triggers. Use them when a failure should produce a timely response, whether or not a policy can block it. +An alert watches a measurable condition and opens an issue when it fires. Use one when a failure should produce a timely response, whether or not a policy can block it. ## Create and test an alert @@ -13,7 +13,7 @@ Alerts watch a measurable condition and create an incident when it triggers. Use 1. Go to **Analyze → Alerts** and select **new alert**. You can also start from the bell on a representative error. 2. Enter the name, severity, trigger kind, condition, evaluation interval, breach count, window, and channels. 3. Save the alert, open its detail page, and run **test**. - 4. Go to **Analyze → Issues** to acknowledge, assign, discuss, subscribe to, and resolve incidents created by the alert. + 4. Go to **Analyze → Issues** to acknowledge, assign, discuss, subscribe to, and resolve the issues the alert opens. The first part of the form identifies the alert and the signal that should trigger it. @@ -23,31 +23,67 @@ Alerts watch a measurable condition and create an incident when it triggers. Use ![The second part of the new-alert form with breach logic, schedule, severity, delivery channels, and create action.](/images/dashboard/alert-new-routing.png) - After saving, use the Alerts list to confirm the rule is enabled and that its trigger, window, severity, and channels match what you intended. + After saving, use the Alerts list to confirm the rule is enabled and that its trigger, window, severity, and channels match what you intended. A card that has open issues shows the count; a card with none omits the line entirely, so its absence tells you nothing. To read the figure for every rule including the zeroes, use `fp alerts show `, which always prints it. ![The Alerts page showing alert rules with their triggers, evaluation windows, channels, and severity.](/images/dashboard/alerts.png) - - Test the alert before depending on it for production response. ```bash fp alerts create high-errors \ --trigger-kind metric_threshold \ --severity warning \ - --trigger-spec '{"metric":"error_count","op":">","value":50,"window_secs":900}' + --trigger-spec '{"metric":"error_count","op":">","value":50,"window_secs":900}' \ + --eval-interval-secs 300 \ + --min-breaches 2 \ + --eval-window 3 fp alerts show high-errors fp alerts test high-errors fp alerts update high-errors --severity critical --yes ``` - Use `fp alerts list` to review rules and `fp alerts delete ` to remove one. + New alerts start **enabled**, and a name collision is rejected before anything is created. + + `fp alerts update` replaces the whole definition server-side, so the CLI re-reads the alert and layers your flags on top — a flag-only update needs `alerts:read` **as well as** `alerts:write`, and it confirms unless you pass `--yes`. + + Use `fp alerts list` to review rules and `fp alerts delete ` to remove one. Delete previews the rule — including its open-issue count — and confirms before acting, because it cannot be undone. + + Delete is not the way to silence a noisy rule. `DELETE /alerts/{id}` cascades: every issue the alert ever opened goes with it, taking its comments, subscribers, and activity history. The CLI's confirmation box words this as the open issues being "orphaned"; the API contract is a cascading delete, so treat the history as gone. To stop a rule firing and keep what it recorded, turn it off instead — the dashboard's alert form has the enabled toggle, and neither `fp alerts create` nor `fp alerts update` exposes an `--enabled` flag, so the CLI route is `fp alerts update --file` with a full definition carrying `enabled: false`. See the [`fp alerts` reference](/reference/cloud-cli#alerts) for the complete alert command set. -Alert conditions can be based on errors, evaluation scores, evaluation combinations, or custom SQL. Add recipients, test the rule, and open the resulting incident to acknowledge, assign, comment, subscribe, and resolve it. + + **test** delivers real notifications. It sends to the alert's real email, Slack, and webhook channels and opens a synthetic issue, so it can page whoever is on call. It confirms first on an interactive terminal; `--yes`, `--json`, and a redirected stdin all skip that prompt. The server also reports success as soon as it dispatches, so a green test proves the message left, not that it arrived. + + +## Define the trigger + +There are five trigger kinds: + +| `--trigger-kind` | Fires when | +| --- | --- | +| `metric_threshold` | A metric crosses a threshold — errors, latency, cost, or anything else you can measure. | +| `per_event` | A single matching event arrives. | +| `evaluation_score` | An evaluator's score crosses a threshold. | +| `eval_compound` | Several score conditions combine, such as two of three scores failing. | +| `custom_sql` | A query you write returns a breach. | + +The condition itself goes in `--trigger-spec`, shaped for the kind you chose. + +## Set the evaluation numbers + +The dashboard form and the CLI ask for the same four values. + +| Setting | Flag | Accepted values | +| --- | --- | --- | +| Severity | `--severity` | `info`, `warning`, `critical` | +| Evaluation interval | `--eval-interval-secs` | 30–86,400 seconds | +| Breach count | `--min-breaches` | At least 1, and never more than the window | +| Evaluation window | `--eval-window` | At least 1, counted in **intervals**, not seconds | + +So `--eval-interval-secs 300 --min-breaches 2 --eval-window 3` means "evaluate every five minutes, and fire when two of the last three evaluations breached". ## Good alert design @@ -56,7 +92,12 @@ Alert conditions can be based on errors, evaluation scores, evaluation combinati - Set a window and threshold that avoid reacting to one harmless event. - Include a link or query that leads responders to sessions. - Assign an owner before enabling the rule. +- Check `fp alerts show ` for the open-issue count before adding another rule for the same symptom. The dashboard card prints that line only when the count is non-zero. After resolving an audit finding, add an alert when the same failure could recur outside the policy's coverage. + + + Acknowledge, assign, comment, subscribe, and resolve — the workflow every firing alert lands in. + diff --git a/docs/audits/cadence.mdx b/docs/audits/cadence.mdx index 96ac55b48..522d29cf1 100644 --- a/docs/audits/cadence.mdx +++ b/docs/audits/cadence.mdx @@ -11,17 +11,19 @@ Use recurring audits for failure modes that can return as agents, prompts, tools 1. Go to **Analyze → Audits** and open the audit. - 2. Open its settings and change the enabled state, interval, UTC anchor, window mode, or lookback. - 3. Save the audit and confirm the next-run time on the audit card. - 4. Use **run now** once after a major scope or context change. + 2. Select **edit settings** and change the cadence, window, sensitivity, or findings per run. + 3. Save the audit and confirm the next-run time in the audit header. + 4. Pause and resume the schedule from the same header, beside **run now**. + 5. Use **run now** once after a major scope or context change. - ![The audit editor showing cadence, sweep window, scope, analysis settings, context, and notification channels.](/images/dashboard/audit-edit.png) + ![An audit header showing its paused state, open findings, last run, next run, sweep window, and sensitivity, with run now, edit settings, and resume controls.](/images/dashboard/audit-detail.png) ```bash fp audits edit checkout-reliability \ --schedule-interval-secs 86400 \ --schedule-anchor 2026-08-15T09:00:00Z \ + --window-mode since_last \ --lookback-window-secs 86400 \ --yes @@ -29,21 +31,50 @@ Use recurring audits for failure modes that can return as agents, prompts, tools fp audits edit checkout-reliability --enabled --yes ``` + `fp audits edit` confirms before changing anything, which is why every example here passes `--yes`. The server replaces the whole definition on every edit, so the CLI re-sends the current audit with your changes applied — a flag-only edit needs `audits:read` **as well as** `audits:write`. Scope a CI key for both. + See the [`fp audits` reference](/reference/cloud-cli#audits) for schedule bounds, window behavior, and all audit commands. -Choose cadence from the speed and cost of the risk: +## The two schedule bounds + +Every recommendation below has to fit inside these. The ranges are the server's, and `fp` mirrors them client-side, so a value outside one exits 2 locally rather than costing a 422 round-trip. + +| Field | Flag | Range | Default | +| --- | --- | --- | --- | +| Schedule interval | `--schedule-interval-secs` | 3,600–604,800 seconds (1 hour to 7 days) | 86,400 (daily) | +| Lookback window | `--lookback-window-secs` | 3,600–7,776,000 seconds (1 hour to 90 days) | 604,800 (7 days) | + +Seven days is the ceiling on cadence. There is no monthly Cloud audit. + +## Choose a cadence + +Pick from the speed and cost of the risk: | Risk pattern | Starting cadence | | --- | --- | -| High-impact production action | Daily | +| High-impact production action | Daily, or hourly while a risky change is landing | | Workflow or model regression | Weekly | -| Governance or access review | Monthly | -| One-time release investigation | Run once | +| Governance or access review | Weekly — the longest interval available | + +Align the lookback window with the cadence so runs neither leave gaps nor repeatedly examine an unnecessarily large population. + +There is no one-shot audit; every audit carries a schedule interval. For a release investigation, create it enabled, let the first run fire — the first run queues immediately on create, regardless of the anchor — then pause it with `fp audits edit --disabled --yes`. Re-enable before using **run now**: a disabled audit has no queue row, and a manual run against one is refused with a 409. + +## Window mode and anchor + +`--window-mode` decides which window each run sweeps: + +| Mode | Behavior | +| --- | --- | +| `since_last` | Continue from the end of the last fully analyzed window, so a skipped run leaves no gap. | +| `fixed` | Re-inspect a rolling `lookback_window_secs` on every run, whatever the last run covered. | + +`--schedule-anchor` fixes the phase rather than the frequency: runs land on `anchor + N × interval`. Omit it and the anchor defaults to the next 09:00 UTC; an anchor more than 365 days out is rejected. Changing the interval or the anchor takes effect at the next reschedule, not on the run already queued, and a failed run does not move the anchor. -Align the lookback window with the cadence so runs neither leave gaps nor repeatedly examine an unnecessarily large population. After changing an audit's goal or context, run it manually once before relying on the next scheduled result. +After changing an audit's goal or context, run it manually once before relying on the next scheduled result. - Local scheduled audits are configured on the machine and scan local agent history. Cloud audit schedules operate on Cloud sessions. Treat their results and ownership separately. + This page is about Cloud audit schedules, which operate on Cloud sessions. The local audit has its own timer: `failproofai audit --schedule [days]` takes 1–90 days and defaults to 7, scans the agent history on that one machine, and is the only surface with a monthly option. Its results and ownership are separate from anything here — see [Audit local agent history](/audits/local-audit). diff --git a/docs/audits/findings-and-issues.mdx b/docs/audits/findings-and-issues.mdx index 728815f80..49c1a0a14 100644 --- a/docs/audits/findings-and-issues.mdx +++ b/docs/audits/findings-and-issues.mdx @@ -6,13 +6,22 @@ icon: "clipboard-check" A finding is the audit's evidence-backed statement about a failure. An issue is the durable workflow for responding to it. +The two objects use different vocabularies, and they sit next to each other, so it is worth fixing them in mind first: + +| | Finding | Issue | +| --- | --- | --- | +| States | `open`, `recurring`, `resolved`, `dismissed`, `muted` | `firing`, `acknowledged`, `resolved` | +| Filter flag | `--status` | `--state` | +| Handle | Finding id | Issue id | +| Default listing | The live set: open plus recurring | Newest opened first | + ## Triage and assign the work 1. Open **Analyze → Audits**, choose a completed run, and select a finding to inspect its analysis, recommendation, sessions, and evidence queries. 2. Acknowledge, assign, dismiss, mute, resolve, or reopen the finding after checking its evidence. - 3. Go to **Analyze → Issues** and filter the durable inbox by status, severity, or assignee. + 3. Go to **Analyze → Issues** and filter the durable inbox by state, severity, or assignee. 4. Open the issue to assign it, add comments or subscribers, and resolve it after the fix is verified. Start with the finding summary. Confirm that the failure description, recommended response, severity, and ranking agree with the sessions you expected the audit to examine. @@ -37,15 +46,24 @@ A finding is the audit's evidence-backed statement about a failure. An issue is fp audits finding fp audits ack --reason "owner assigned" fp audits assign --to engineer@example.com + fp audits mute --reason "expected in staging" --yes + fp audits dismiss --reason "false positive" --yes + fp audits resolve --yes + fp audits reopen - fp issues list + fp issues list --state firing fp issues show + fp issues ack fp issues assign --assignee engineer@example.com fp issues comment-add --body "policy is in observe mode" fp issues resolve --yes ``` - Use `fp issues subscribe `, `fp issues unsubscribe `, and `fp issues subscribers ` to manage watchers. + `mute`, `dismiss` and `resolve` suppress or close a finding, so each confirms first — pass `--yes` in scripts. `ack`, `reopen` and `assign` are reversible bookkeeping and act immediately. `ack`, `mute` and `dismiss` take `--reason`, and it is worth passing: it is kept as durable feedback on the finding, not written to a log and forgotten. `resolve`, `reopen` and `assign` take no reason. + + Assignment works differently on each object. `fp audits assign` requires `--to ` and sets one owner; re-running it reassigns. `fp issues assign` takes a repeatable `--assignee` and **replaces** the whole list, so omitting it clears every assignee. + + Use `fp issues comment-list `, `fp issues count --state firing`, and `fp issues subscribe`/`unsubscribe`/`subscribers ` for the rest of the issue surface. See the [Cloud CLI audit and issue reference](/reference/cloud-cli#audits) for audit findings and [`fp issues`](/reference/cloud-cli#issues) for issue management. @@ -53,7 +71,20 @@ A finding is the audit's evidence-backed statement about a failure. An issue is ## Review a finding -Confirm that it contains: +Two fields decide what to do with it before anything else does. + +**`kind`** separates a `failure` (something went wrong) from a `policy` violation (a rule that was broken) and an `improvement` (the work could be done better). It is rendered as a badge on the finding. A `policy` finding is the one a policy can close; the other two usually need a workflow change, an alert, or a human. + +**`priority`** is a 0–1 score, recomputed per run, and it is the order the finding queue is sorted in. The finding page breaks it down under **why it ranks here**, as value × weight: + +| Factor | Weight | +| --- | --- | +| Coverage | 0.30 | +| Severity | 0.25 | +| Magnitude | 0.25 | +| Recency | 0.20 | + +Then confirm the finding contains: - A stable failure mode, not only a one-off title - Severity and operational impact @@ -61,9 +92,13 @@ Confirm that it contains: - Enough context to reproduce the behavior - A proposed response that matches the evidence +To reproduce a finding, pull it whole: `fp --json audits finding ` returns the full record with its `evidence`, `evidence_queries` and `scope` untouched, which is what the analysis actually ran. `--json` is a global option, so it goes before the command. + ## Use an issue to manage the response -Create or link an issue when the finding needs assignment, discussion, status changes, comments, or subscribers. Issues can also represent alert incidents and manually reported problems, which is why they live under audit response rather than in the primary navigation. +Create or link an issue when the finding needs assignment, discussion, status changes, comments, or subscribers. An issue's `source` records where it came from — `audit`, `alert`, or `manual` — which is why the dashboard gives issues their own **Issues** view rather than nesting them under an audit. + +Subscription is partly automatic. People are subscribed when they acknowledge an issue, comment on it, are assigned to it, or open it — `fp issues subscribers` lists active subscribers only, so it is not just the list of manual subscribes. A comment emails every active subscriber, and commenting needs only `issues:read`, so a read-only reviewer is never a silent observer. Resolve the issue when remediation is deployed and verified. Resolve the finding when the failure mode has been addressed for the audit population. Those moments may differ. @@ -83,11 +118,19 @@ Resolve the issue when remediation is deployed and verified. Resolve the finding ```bash fp issues show - fp audits finding + fp --json audits finding fp events --session-id --full --all ``` - Policy candidacy, Cloud publication, and fleet deployment are dashboard workflows. Use `failproofai policies --install --custom ` when you want to validate equivalent policy source locally first. + The dashboard is one lane; the CLI is the other. `fp policies compose ""` drafts a policy, `fp policies publish ./policy.mjs` mints a version (publishing deploys nothing), and `fp fleet deploy --add :observe` puts it on a machine in shadow. The `:observe` suffix is required — a bare `--add ` enforces immediately. Read what it would have done with `fp guardrails summary --since 24h`, then promote with `--add :enforce`. + + To try equivalent policy source on your own machine first, point the local CLI at the file: + + ```bash + failproofai policies -i -c ./checkout-policies.js + ``` + + A file whose name ends in `policies.js`, `policies.mjs` or `policies.ts`, dropped into `.failproofai/policies/` in the project or under `~/`, is loaded on every hook event with no flags at all. diff --git a/docs/audits/local-audit.mdx b/docs/audits/local-audit.mdx index 24c3ce48c..b9299e13b 100644 --- a/docs/audits/local-audit.mdx +++ b/docs/audits/local-audit.mdx @@ -1,74 +1,56 @@ --- title: "Audit local agent history" -description: "Scan supported agent CLI histories offline and review risky or wasteful behavior locally." +description: "Scan agent history on this machine for risky or wasteful behavior." icon: "laptop-minimal-check" --- -Use a local audit for an immediate, private review before connecting a machine to Failproof AI Cloud. It scans the agent histories already stored on your machine, replays tool activity through builtin policies, and opens a local results dashboard. +A local audit reads agent history already on this machine and opens results at `http://localhost:8020/audit`. It needs no account. -## Run an interactive audit +```bash +failproofai audit +``` - - - The local audit starts from the CLI because it must discover histories on the current machine. Run `failproofai audit`; after the scan, Failproof AI starts the bundled dashboard and opens **http://localhost:8020/audit**. +The scan covers history from all supported harnesses it finds. It replays the 39-policy catalog built into this version; installed packs and custom policies are not included. - In the audit view, review the number of sessions, tool calls, projects, and policy hits. Start with frequent findings, then inspect the affected project and agent history before enabling enforcement. +## Read the result - The local audit dashboard is separate from **Analyze → Audits** in Failproof AI Cloud. Local audits remain on the machine and require no account or network connection. - - - ```bash - npm install -g failproofai - failproofai audit - ``` +Start with the most frequent findings, then open the affected session before enabling enforcement. - The command performs a full scan of every supported history it finds. The current command does not accept filters such as `--since`, `--cli`, `--project`, `--port`, or `--no-open`. +Three limits matter: - Keep the process running while using the local dashboard. Press Ctrl+C when you are finished. - - +- The audit reconstructs tool events, not `Stop`, so `require-*-before-stop` policies do not appear. +- Eight “Audit-only” checks identify wasteful patterns without an exact matching policy. +- `warn-repeated-tool-calls` is skipped because replaying it would modify transcript-side state. -The current audit adapters can read histories from Claude Code, Codex, GitHub Copilot CLI, Cursor, OpenCode, Pi, Hermes, OpenClaw, Factory, Devin, Antigravity, and Goose. Only locally available histories are scanned. +For an enforceable finding, use the policy command shown in the result. Check [harness enforcement support](/reference/harnesses#enforcement-capability) before relying on it. -## Schedule recurring local audits +The cached result lives at `~/.failproofai/audit/dashboard.json` and expires after seven days. - - - Open **Settings** in the local dashboard, enable scheduled audits, choose the interval, and set the email address that should receive findings. The dashboard and CLI update the same machine configuration. - - - Enable a weekly audit and send findings to the named address: +## Schedule scans - ```bash - failproofai audit --schedule 7 --email reliability@example.com - failproofai audit --status - ``` +```bash +failproofai audit --schedule 7 +failproofai audit --status +failproofai audit --no-schedule +``` - The interval accepts 1–90 days and defaults to 7 when omitted. The first setup signs you in when needed; `--email` supplies the report address without prompting. +The interval accepts 1–90 days and defaults to 7. First-time scheduling requires an interactive sign-in. The background service must be running. - Stop the schedule without deleting local audit history: +### What leaves the machine - ```bash - failproofai audit --no-schedule - ``` - - +An interactive audit sends no findings. Anonymous CLI telemetry is enabled by default; disable it with `FAILPROOFAI_TELEMETRY_DISABLED=1`. -The daemon runs scheduled scans in the background, refreshes the cached result used by the local dashboard, and emails the configured report. Use `failproofai audit` when you want to run an interactive scan immediately. +After you opt into scheduling, each scheduled scan sends the machine ID, label, platform, and scan window. If harmful patterns are found, it also sends a limited digest with counts, timestamps, and up to three redacted examples. Clean scans send no findings and no email. -## Move from local evidence to Cloud operations - -A local audit is a fast baseline. Connect the machine to Failproof AI Cloud when you need shared traces, recurring population audits, findings and issues, alerts, organization-wide policy deployment, or fleet health. + + Audit results are evidence for review, not proof that every flagged action is unsafe. Check the session before turning a finding into blocking enforcement. + - - Define a recurring goal, population, evidence window, and response channels. + + Run shared, recurring audits across agents and machines. - - Validate locally, publish deliberately, and deploy to a narrow machine group first. + + Add reviewed policies after confirming a finding. - - - Audit output is evidence for review, not proof that every flagged action is unsafe. Confirm context before turning a finding into blocking enforcement. - diff --git a/docs/audits/overview.mdx b/docs/audits/overview.mdx index 6850c1b7a..04746a9ca 100644 --- a/docs/audits/overview.mdx +++ b/docs/audits/overview.mdx @@ -1,56 +1,64 @@ --- title: "Audits" -description: "Review a defined population of sessions for failures that traces alone will not reveal." +description: "Search agent runs for repeated failures, risky behavior, and wasted effort." icon: "scan-search" --- -An audit searches a selected set of sessions for a stated failure goal. It combines trace evidence, evaluation results, policy hits, and reference context to produce findings you can act on. +An audit reviews many agent runs against a goal and returns evidence-backed findings.
-See how an audit moves from a scheduled run to evidence-backed failures you can fix. +## Local or Cloud -## Open audits +| | Local audit | Cloud audit | +| --- | --- | --- | +| Command | `failproofai audit` | `fp audits` | +| Reads | Agent history on this machine | Sessions in your Cloud organization | +| Produces | Local results at `localhost:8020/audit` | Shared findings, issues, and alerts | +| Requires | No account | Cloud connection and API key | - - - Go to **Analyze → Audits**. The page shows scheduled state, open findings, last run, next run, cadence, and whether the audit has a brief or reference pages. Select a card for settings and run history; select **new audit** to create one. +Use a local audit for a quick review of one machine. Use Cloud when a team needs a recurring audit across agents and machines. - ![The Audits page showing recurring audits with their schedule, sensitivity, and run state.](/images/dashboard/audits.png) - - - ```bash - fp audits list - fp audits list --enabled-only --show-id - fp audits show - fp audits findings --status open --limit 20 - ``` - - +## Questions an audit can answer -Use an audit when you need to answer a population-level question such as: - -- Where do agents abandon tasks without escalating? +- Where do agents stop without escalating? - Which tool failures lead to ineffective retries? - Are agents accessing data outside the intended workflow? -- What changed after a model, prompt, or tool release? +- What changed after a model, prompt, or tool update? + +## Open Cloud audits + +Go to **Analyze → Audits**, or use: + +```bash +fp audits list +fp audits show +fp audits findings --status open --limit 20 +``` + +Audit commands use the audit name or full ID. Findings are referenced by their own ID. -## Audit response flow +## From finding to prevention ```text Session → Audit → Finding → Issue → Policy - ↘ Alert for recurrence + ↘ Alert ``` -A finding should name the failure mode and point to evidence. An issue owns the remediation. A policy prevents a known action pattern; an alert detects recurrence when prevention is not possible or needs monitoring. +A finding points to evidence. An issue owns the response. A policy prevents a known action pattern; an alert watches for recurrence. - - - Define the goal, population, and reference context before the first run. +Not every finding should become a policy. A `policy` finding describes an enforceable rule. A `failure` or `improvement` may need a workflow, prompt, model, or tool change instead. + + + + Scan local history with no account. + + + Choose the goal, sessions, and schedule. - - State what each agent must produce and what it must never do. + + Triage evidence and assign a response. - + diff --git a/docs/audits/recipes.mdx b/docs/audits/recipes.mdx index 9935ee777..20e79fabc 100644 --- a/docs/audits/recipes.mdx +++ b/docs/audits/recipes.mdx @@ -8,7 +8,9 @@ Use these as starting goals, then add your agent, environment, and expected work - Go to **Analyze → Audits → New audit**, copy one recipe into the description or brief, then add the relevant environment, agent, lookback, and reference URLs. Create the audit and inspect the first run before scheduling it. + Go to **Analyze → Audits → New audit**. A recipe line from this page is the **description** — what you expect this audit to catch. The workflow rules that make the goal judgeable go in **your brief**, in the **what it knows** card: background the model reads before it looks at a single event, capped at 8,192 characters, added to what the audit already looks for. A brief is never a replacement for the goal and never evidence for a finding. + + Then set scope in **what it reads**, because half of these recipes are noise without it. Environments and agents narrow the population; **errors to ignore** lists the error types you expect and handle by design, so they stop counting as failures. It takes error type names only, and it is what keeps the retry-loop recipe from flooding. The new audit form turns a recipe into an executable failure check by adding scope, context, cadence, and notifications. @@ -16,8 +18,6 @@ Use these as starting goals, then add your agent, environment, and expected work After creation, confirm the audit appears in the list with the expected status and schedule before relying on recurring runs. - ![The Audits page where a workflow-specific audit recipe appears after creation.](/images/dashboard/audits.png) - Open the first run and refine the recipe if its findings are broader or narrower than the intended failure mode. @@ -25,30 +25,69 @@ Use these as starting goals, then add your agent, environment, and expected work ```bash fp audits create retry-loop-review \ + --description "Find agents that repeat a failing tool call without changing anything" \ --scope '{"environments":["production"]}' \ + --ignore-error-type RateLimitRetried \ --text-file ./retry-loop-audit.txt \ --schedule-interval-secs 86400 ``` + + `--text` and `--text-file` are two ways to send the same brief; pass one, not both. Add reference pages with `--url`, up to five, public `https://` only. Send all of it with the create request: a new enabled audit is due immediately, so context written in a second call can miss the first run. +## Judge the recipe, not just the wording + +Two settings move the result more than the phrasing does. **Sensitivity** (`low`, `medium`, `high`, default `medium`) decides how eagerly a run flags a pattern; **findings per run** (`--top-k`, default 50) caps how many it keeps. Raise sensitivity for a question where a miss costs more than a false positive, and lower it for a volume pattern that would otherwise fill the queue. Each recipe below names a starting point — change it after reading the first run, not before. + +Task abandonment and missed human escalation are judgements against what an agent was supposed to do, not against an error it raised. Write that agent's [agent context](/audits/agent-contracts) first, or the analysis has no standard to grade against. + Find sessions where the agent repeats the same failing tool call without changing input, selecting an alternative tool, or escalating to a human. + + Start at `medium`, and list the error types you retry on purpose under **errors to ignore**. Find sessions where the chosen tool does not match the stated task, or where tool input violates the workflow's required preconditions. + + Start at `medium`. Put the preconditions themselves in the brief; without them the analysis has no rule to check. Find sessions that read, write, or transmit sensitive data outside the approved paths and services for this agent. + + Start at `high`. A missed exposure costs more than a false positive you dismiss once. Find sessions that end without the requested outcome, a clear error, or an explicit handoff to a human. + + Start at `medium`, and write the agent's **Done when** section first — this recipe is a judgement against it. Find sessions whose model, tool, or total duration exceeds the expected budget, and identify the event pattern responsible. + + Start at `low` and raise it if the first run is thin. Budgets belong in the brief, and any error type a slow path raises by design belongs in **errors to ignore**. Find sessions where confidence, repeated failure, or policy guidance required a human decision but the agent continued autonomously. + + Start at `high`, and state the escalation rule in the agent's **Must not** section so the analysis grades against it. + +## Get a first answer before you create anything + +Several of these patterns already have offline detectors in the local audit, which reads the agent histories on your own machine and needs no account: + +| Detector | Pattern it counts | +| --- | --- | +| `sleep-polling-loop` | A long `sleep`, or a `while … sleep … done` polling loop | +| `reread-after-edit` | A read of a file the agent just edited or wrote | +| `find-from-root` | `find` against `/` or another high-level directory | +| `redundant-cd-cwd` | `cd` into the directory the shell is already in | +| `prefer-edit-over-read-cat` | `cat`, `head`, `tail`, `less`, or `more` on a single source file | +| `prefer-edit-over-sed-awk` | In-place edits through `sed -i` or `awk … > file` | +| `prefer-write-over-heredoc` | Multi-line content written through a heredoc or `echo > file` | +| `git-commit-no-verify` | `git commit --no-verify`, skipping hooks | + +These detectors count; they do not block, and none of them measures cost or latency. Run `failproofai audit` to see which wasteful and risky shell patterns your agents already produce before you pay for a Cloud audit of the same ground — see [Audit local agent history](/audits/local-audit). diff --git a/docs/audits/run.mdx b/docs/audits/run.mdx index 05c67f343..9688eca62 100644 --- a/docs/audits/run.mdx +++ b/docs/audits/run.mdx @@ -25,22 +25,57 @@ Run an audit after its goal and population are specific enough that another oper fp audits findings --audit checkout-reliability ``` + `fp audits run` queues the audit; it does not wait for it. Follow it with `fp audits runs ` and read the findings once a run has finished. `--limit` never returns more than the 50 most recent runs, whatever you ask for. + + Triage acts on a finding id, not on the audit: + + ```bash + fp audits finding + fp audits ack --reason "owner assigned" + fp audits assign --to engineer@example.com + fp audits mute --reason "expected in staging" --yes + fp audits dismiss --reason "false positive" --yes + fp audits resolve --yes + fp audits reopen + ``` + + `mute`, `dismiss`, and `resolve` confirm before acting, so pass `--yes` in scripts. `ack`, `assign`, and `reopen` act immediately. + See the [`fp audits` reference](/reference/cloud-cli#audits) for run history, findings, and triage commands. +**run now** can be refused. A run already in progress and a disabled audit both answer `409`, with the reason in the response's `error` field; an audit that does not exist, or that belongs to another organization, answers `404`. + +| Refusal | What it means | What to do | +| --- | --- | --- | +| Unknown audit | The audit does not exist, or it belongs to another organization. | Confirm the name with `fp audits list`. | +| The audit is disabled | A disabled audit has no queue row, so there is nothing to make due. A paused audit still renders a **run now** control. | Resume it first, or `fp audits edit --enabled --yes`. | +| A run is already in progress | One run per audit at a time; the current one must finish before another is queued. | Check it with `fp audits runs `. | + ## Before you run - Confirm sessions exist in the selected time window. - Verify the environment and agent filters. -- Check that reference context is current. +- Check that reference context is current. Every run re-reads the audit's pages and falls back to the stored copy when one cannot be reached, so a page that has been moved or taken down keeps serving stale text until you fix the URL. - Make sure the goal describes a failure mode, not a desired conclusion. ## Review the run -Start with run status, session coverage, and whether model analysis ran. Then inspect each finding's severity, description, evidence sessions, supporting queries, and suggested prevention path. +The audit detail page opens with five tiles: **open findings** awaiting triage, **last run**, **next run**, **window** (how far back each run reads), and **sensitivity** (how eagerly a run flags a pattern: `low`, `medium`, `high`). The at-most-N-findings-per-run cap is a separate setting, findings per run (`--top-k`, default 50). Those five answer the coverage question faster than opening a run does. + +Then inspect each finding's severity, description, evidence sessions, supporting queries, and suggested prevention path. Findings are ordered by a **priority** score between 0 and 1, ranked per run, and each finding shows the four weighted factors behind it: + +| Ranking factor | Weight | +| --- | --- | +| Coverage | 0.30 | +| Magnitude | 0.25 | +| Severity | 0.25 | +| Recency | 0.20 | -Use finding status to acknowledge, mute, dismiss, resolve, reopen, or assign work. Preserve the evidence even when the finding is dismissed; it explains why the decision was made. +A finding is in exactly one of five statuses: `open`, `recurring`, `resolved`, `dismissed`, or `muted`. A `findings` query with no status filter returns the live set — `open` plus `recurring`. A finding you mute, dismiss, or resolve leaves that set; `ack` and `assign` leave the status alone, so the finding stays in the queue — deprioritized or owned, not gone. Ask for a status explicitly to see the ones that left. + +Use finding status to acknowledge, mute, dismiss, resolve, reopen, or assign work. Preserve the evidence even when the finding is dismissed; it explains why the decision was made. [Findings and issues](/audits/findings-and-issues) covers what each verb does to future runs. ## Interpret an empty or delayed run @@ -52,9 +87,13 @@ Use finding status to acknowledge, mute, dismiss, resolve, reopen, or assign wor | No analysis capacity is immediately available | The audit remains queued and retries instead of skipping the population. | Wait for capacity or spread audit anchors. Self-hosted operators should scale the audit-agent replicas and matching dispatcher capacity. | | Capacity remains unavailable for the retry window | The run gives up with zero findings and sends a failure notification when email delivery is available. | Check whether the audit fleet is saturated or repeatedly restarting. | +The last three rows describe behavior of the Cloud API server and its dispatcher. On managed Cloud they are Failproof AI's to fix; on a self-hosted deployment they are yours. + When analysis does not run, `since_last` audits keep that unanalysed window open for the next successful run. Existing findings are not retired because a skipped analysis is not evidence that the failure disappeared. -## Understand failure notifications +## Understand notifications + +A successful run notifies only when it finds something **new**. Silence from a healthy audit is the normal case, not a sign that nothing ran — check **last run** on the audit page, or `fp audits runs `, to see that it did. An audit with no channels selected files its findings and notifies nobody. A failed run or failed model-analysis step uses the audit's email recipients. If the audit has no email channel, Failproof AI falls back to the organization's `alerts.email_default_recipients` setting so a silently broken audit still has an escalation path. diff --git a/docs/audits/setup.mdx b/docs/audits/setup.mdx index 787a4e82f..e81e7f5a1 100644 --- a/docs/audits/setup.mdx +++ b/docs/audits/setup.mdx @@ -10,12 +10,15 @@ The quality of an audit starts with its scope. A broad request such as “find p - 1. Go to **Analyze → Audits → New audit** and enter the name and description. - 2. Set the cadence, time window, agent/environment scope, ignored errors, sensitivity, and maximum findings. - 3. In **agents**, add or review agent context, then add the operator brief and any public HTTPS reference URLs. - 4. Choose notification channels and select **create audit**. The first run queues immediately. + 1. Go to **Analyze → Audits → New audit** and enter the name and description. The name is unique per organization; the description records what you expect this audit to catch. + 2. In **when it runs**, set the cadence and the window. In **what it reads**, narrow the population with environments, agents, and the error types to ignore — an empty field includes everything. In **how it judges**, set sensitivity and findings per run. + 3. In **what it knows**, write the operator brief and add the pages the audit reads. The brief is background the model reads before it looks at a single event: it is added to what this audit already looks for, never a replacement, and never evidence for a finding. + 4. Open the **agents** drawer to add or review each agent's context. It saves independently of the audit, and its header counts how many of your agents already have one. See [agent context](/audits/agent-contracts). + 5. Choose notification channels and select **create audit**. A new audit starts enabled, and its first run queues immediately. ![The new audit form with name, description, cadence, window, environment and agent scope, ignored errors, analysis settings, brief, reference pages, and notification channels.](/images/dashboard/audit-new.png) + + **agents** in the **what it reads** card is a scope filter — it decides which sessions each run sweeps. What an agent is *for* lives in the separate agents drawer. ```bash @@ -29,16 +32,44 @@ The quality of an audit starts with its scope. A broad request such as “find p --url https://runbooks.example.com/checkout ``` - The first run queues immediately. Include the brief and reference URLs during creation so that run receives them. + Every field except the name has a server default, so a bare `fp audits create nightly` is already a valid daily audit. A name that is taken is rejected up front, before anything is created. New audits start enabled unless you pass `--disabled`, and the first run of an enabled audit queues immediately. + + Base a definition on saved JSON with `--file audit.json` and layer flags on top of it. That is the reproducible path when audit definitions are reviewed or kept in version control. See the complete [`fp audits create` reference](/reference/cloud-cli#audits). +## Ranges and defaults + +The numeric settings are validated on both sides, so an out-of-range value is a usage error rather than a rejected request. + +| Setting | CLI flag | Accepted values | Default | +| --- | --- | --- | --- | +| Cadence | `--schedule-interval-secs` | 3600–604800 (1 hour to 7 days) | 86400 (daily) | +| Schedule anchor | `--schedule-anchor` | ISO 8601 UTC; an anchor more than 365 days out is rejected | The next 09:00 UTC | +| Window | `--window-mode` | `fixed`, `since_last` | `since_last` | +| Lookback | `--lookback-window-secs` | 3600–7776000 (1 hour to 90 days) | 604800 (7 days) | +| Sensitivity | `--sensitivity` | `low`, `medium`, `high` | `medium` | +| Findings per run | `--top-k` | 1 or more | 50 | + +The anchor fixes the phase of the schedule: runs land on `anchor + N * interval`, so a slow run or a manual **run now** cannot drift the cadence. The first run is queued immediately on create regardless of the anchor. + +## Brief and reference pages + +The form shows both limits as counters, and the CLI enforces the same two: + +- The brief is capped at 8,192 characters (`--text`, or `--text-file` to read it from a file — pass one or the other, not both). +- An audit references at most five pages, public `https://` only (`--url`, repeated). + +Reference URLs are validated as you save. Private, loopback, and cloud-metadata addresses are refused, and a refused URL fails the whole create — no half-made audit is left behind. Accepted pages are fetched in the background, so a slow site never blocks the save. Every run re-reads them and falls back to the stored snapshot when one cannot be reached, and snapshots refresh on their own weekly. Use `fp audits context-refresh ` when you know a page changed and want it picked up before the next run. + +Send the brief and the URLs with the create request rather than in a second call. A new enabled audit is due the instant its row commits, so context written afterwards can be overtaken by the dispatcher and miss the first run — the run you are watching. Change it later with `fp audits context-set `, which replaces whichever half you name and leaves the other alone. + Start with a known failed session and several normal sessions. This gives the audit both a positive example and a comparison set. - Audit context is stored as its own resource so editing an audit does not accidentally remove the reference material. + Audit context is stored as its own resource. The definition endpoint refuses to write it on update, so an ordinary unrelated edit to the audit can never overwrite the brief or the reference pages. diff --git a/docs/docs.json b/docs/docs.json index 81034e0a4..7c237c8a3 100644 --- a/docs/docs.json +++ b/docs/docs.json @@ -78,10 +78,10 @@ "start/quickstart", "reference/harnesses", "sessions/assistant", - "start/first-audit", - "start/first-policy", "start/setup", "start/concepts", + "start/first-audit", + "start/first-policy", { "group": "Starter templates", "expanded": false, @@ -121,6 +121,7 @@ { "group": "Plug in your agent", "pages": [ + "start/integrations", "start/integrations/custom-agents", "start/integrations/langchain", "start/integrations/crewai", @@ -197,7 +198,6 @@ "group": "Connect your agents", "pages": [ "reference/overview", - "reference/harnesses", "reference/custom-agents", "reference/evaluator-sdk", "reference/policy-sdk", @@ -206,7 +206,6 @@ }, { "group": "Reference", - "expanded": false, "pages": [ "reference/failproof-cli", "reference/local-dashboard", @@ -240,10 +239,10 @@ "zh/start/quickstart", "zh/reference/harnesses", "zh/sessions/assistant", - "zh/start/first-audit", - "zh/start/first-policy", "zh/start/setup", "zh/start/concepts", + "zh/start/first-audit", + "zh/start/first-policy", { "group": "Starter templates", "expanded": false, @@ -283,6 +282,7 @@ { "group": "Plug in your agent", "pages": [ + "zh/start/integrations", "zh/start/integrations/custom-agents", "zh/start/integrations/langchain", "zh/start/integrations/crewai", @@ -357,7 +357,6 @@ "group": "Connect your agents", "pages": [ "zh/reference/overview", - "zh/reference/harnesses", "zh/reference/custom-agents", "zh/reference/evaluator-sdk", "zh/reference/policy-sdk", @@ -366,7 +365,6 @@ }, { "group": "Reference", - "expanded": false, "pages": [ "zh/reference/failproof-cli", "zh/reference/local-dashboard", @@ -394,10 +392,10 @@ "ja/start/quickstart", "ja/reference/harnesses", "ja/sessions/assistant", - "ja/start/first-audit", - "ja/start/first-policy", "ja/start/setup", "ja/start/concepts", + "ja/start/first-audit", + "ja/start/first-policy", { "group": "Starter templates", "expanded": false, @@ -437,6 +435,7 @@ { "group": "Plug in your agent", "pages": [ + "ja/start/integrations", "ja/start/integrations/custom-agents", "ja/start/integrations/langchain", "ja/start/integrations/crewai", @@ -511,7 +510,6 @@ "group": "Connect your agents", "pages": [ "ja/reference/overview", - "ja/reference/harnesses", "ja/reference/custom-agents", "ja/reference/evaluator-sdk", "ja/reference/policy-sdk", @@ -520,7 +518,6 @@ }, { "group": "Reference", - "expanded": false, "pages": [ "ja/reference/failproof-cli", "ja/reference/local-dashboard", @@ -548,10 +545,10 @@ "ko/start/quickstart", "ko/reference/harnesses", "ko/sessions/assistant", - "ko/start/first-audit", - "ko/start/first-policy", "ko/start/setup", "ko/start/concepts", + "ko/start/first-audit", + "ko/start/first-policy", { "group": "Starter templates", "expanded": false, @@ -591,6 +588,7 @@ { "group": "Plug in your agent", "pages": [ + "ko/start/integrations", "ko/start/integrations/custom-agents", "ko/start/integrations/langchain", "ko/start/integrations/crewai", @@ -665,7 +663,6 @@ "group": "Connect your agents", "pages": [ "ko/reference/overview", - "ko/reference/harnesses", "ko/reference/custom-agents", "ko/reference/evaluator-sdk", "ko/reference/policy-sdk", @@ -674,7 +671,6 @@ }, { "group": "Reference", - "expanded": false, "pages": [ "ko/reference/failproof-cli", "ko/reference/local-dashboard", @@ -702,10 +698,10 @@ "es/start/quickstart", "es/reference/harnesses", "es/sessions/assistant", - "es/start/first-audit", - "es/start/first-policy", "es/start/setup", "es/start/concepts", + "es/start/first-audit", + "es/start/first-policy", { "group": "Starter templates", "expanded": false, @@ -745,6 +741,7 @@ { "group": "Plug in your agent", "pages": [ + "es/start/integrations", "es/start/integrations/custom-agents", "es/start/integrations/langchain", "es/start/integrations/crewai", @@ -819,7 +816,6 @@ "group": "Connect your agents", "pages": [ "es/reference/overview", - "es/reference/harnesses", "es/reference/custom-agents", "es/reference/evaluator-sdk", "es/reference/policy-sdk", @@ -828,7 +824,6 @@ }, { "group": "Reference", - "expanded": false, "pages": [ "es/reference/failproof-cli", "es/reference/local-dashboard", @@ -856,10 +851,10 @@ "pt-br/start/quickstart", "pt-br/reference/harnesses", "pt-br/sessions/assistant", - "pt-br/start/first-audit", - "pt-br/start/first-policy", "pt-br/start/setup", "pt-br/start/concepts", + "pt-br/start/first-audit", + "pt-br/start/first-policy", { "group": "Starter templates", "expanded": false, @@ -899,6 +894,7 @@ { "group": "Plug in your agent", "pages": [ + "pt-br/start/integrations", "pt-br/start/integrations/custom-agents", "pt-br/start/integrations/langchain", "pt-br/start/integrations/crewai", @@ -973,7 +969,6 @@ "group": "Connect your agents", "pages": [ "pt-br/reference/overview", - "pt-br/reference/harnesses", "pt-br/reference/custom-agents", "pt-br/reference/evaluator-sdk", "pt-br/reference/policy-sdk", @@ -982,7 +977,6 @@ }, { "group": "Reference", - "expanded": false, "pages": [ "pt-br/reference/failproof-cli", "pt-br/reference/local-dashboard", @@ -1010,10 +1004,10 @@ "de/start/quickstart", "de/reference/harnesses", "de/sessions/assistant", - "de/start/first-audit", - "de/start/first-policy", "de/start/setup", "de/start/concepts", + "de/start/first-audit", + "de/start/first-policy", { "group": "Starter templates", "expanded": false, @@ -1053,6 +1047,7 @@ { "group": "Plug in your agent", "pages": [ + "de/start/integrations", "de/start/integrations/custom-agents", "de/start/integrations/langchain", "de/start/integrations/crewai", @@ -1127,7 +1122,6 @@ "group": "Connect your agents", "pages": [ "de/reference/overview", - "de/reference/harnesses", "de/reference/custom-agents", "de/reference/evaluator-sdk", "de/reference/policy-sdk", @@ -1136,7 +1130,6 @@ }, { "group": "Reference", - "expanded": false, "pages": [ "de/reference/failproof-cli", "de/reference/local-dashboard", @@ -1164,10 +1157,10 @@ "fr/start/quickstart", "fr/reference/harnesses", "fr/sessions/assistant", - "fr/start/first-audit", - "fr/start/first-policy", "fr/start/setup", "fr/start/concepts", + "fr/start/first-audit", + "fr/start/first-policy", { "group": "Starter templates", "expanded": false, @@ -1207,6 +1200,7 @@ { "group": "Plug in your agent", "pages": [ + "fr/start/integrations", "fr/start/integrations/custom-agents", "fr/start/integrations/langchain", "fr/start/integrations/crewai", @@ -1281,7 +1275,6 @@ "group": "Connect your agents", "pages": [ "fr/reference/overview", - "fr/reference/harnesses", "fr/reference/custom-agents", "fr/reference/evaluator-sdk", "fr/reference/policy-sdk", @@ -1290,7 +1283,6 @@ }, { "group": "Reference", - "expanded": false, "pages": [ "fr/reference/failproof-cli", "fr/reference/local-dashboard", @@ -1318,10 +1310,10 @@ "ru/start/quickstart", "ru/reference/harnesses", "ru/sessions/assistant", - "ru/start/first-audit", - "ru/start/first-policy", "ru/start/setup", "ru/start/concepts", + "ru/start/first-audit", + "ru/start/first-policy", { "group": "Starter templates", "expanded": false, @@ -1361,6 +1353,7 @@ { "group": "Plug in your agent", "pages": [ + "ru/start/integrations", "ru/start/integrations/custom-agents", "ru/start/integrations/langchain", "ru/start/integrations/crewai", @@ -1435,7 +1428,6 @@ "group": "Connect your agents", "pages": [ "ru/reference/overview", - "ru/reference/harnesses", "ru/reference/custom-agents", "ru/reference/evaluator-sdk", "ru/reference/policy-sdk", @@ -1444,7 +1436,6 @@ }, { "group": "Reference", - "expanded": false, "pages": [ "ru/reference/failproof-cli", "ru/reference/local-dashboard", @@ -1472,10 +1463,10 @@ "hi/start/quickstart", "hi/reference/harnesses", "hi/sessions/assistant", - "hi/start/first-audit", - "hi/start/first-policy", "hi/start/setup", "hi/start/concepts", + "hi/start/first-audit", + "hi/start/first-policy", { "group": "Starter templates", "expanded": false, @@ -1515,6 +1506,7 @@ { "group": "Plug in your agent", "pages": [ + "hi/start/integrations", "hi/start/integrations/custom-agents", "hi/start/integrations/langchain", "hi/start/integrations/crewai", @@ -1589,7 +1581,6 @@ "group": "Connect your agents", "pages": [ "hi/reference/overview", - "hi/reference/harnesses", "hi/reference/custom-agents", "hi/reference/evaluator-sdk", "hi/reference/policy-sdk", @@ -1598,7 +1589,6 @@ }, { "group": "Reference", - "expanded": false, "pages": [ "hi/reference/failproof-cli", "hi/reference/local-dashboard", @@ -1626,10 +1616,10 @@ "tr/start/quickstart", "tr/reference/harnesses", "tr/sessions/assistant", - "tr/start/first-audit", - "tr/start/first-policy", "tr/start/setup", "tr/start/concepts", + "tr/start/first-audit", + "tr/start/first-policy", { "group": "Starter templates", "expanded": false, @@ -1669,6 +1659,7 @@ { "group": "Plug in your agent", "pages": [ + "tr/start/integrations", "tr/start/integrations/custom-agents", "tr/start/integrations/langchain", "tr/start/integrations/crewai", @@ -1743,7 +1734,6 @@ "group": "Connect your agents", "pages": [ "tr/reference/overview", - "tr/reference/harnesses", "tr/reference/custom-agents", "tr/reference/evaluator-sdk", "tr/reference/policy-sdk", @@ -1752,7 +1742,6 @@ }, { "group": "Reference", - "expanded": false, "pages": [ "tr/reference/failproof-cli", "tr/reference/local-dashboard", @@ -1780,10 +1769,10 @@ "vi/start/quickstart", "vi/reference/harnesses", "vi/sessions/assistant", - "vi/start/first-audit", - "vi/start/first-policy", "vi/start/setup", "vi/start/concepts", + "vi/start/first-audit", + "vi/start/first-policy", { "group": "Starter templates", "expanded": false, @@ -1823,6 +1812,7 @@ { "group": "Plug in your agent", "pages": [ + "vi/start/integrations", "vi/start/integrations/custom-agents", "vi/start/integrations/langchain", "vi/start/integrations/crewai", @@ -1897,7 +1887,6 @@ "group": "Connect your agents", "pages": [ "vi/reference/overview", - "vi/reference/harnesses", "vi/reference/custom-agents", "vi/reference/evaluator-sdk", "vi/reference/policy-sdk", @@ -1906,7 +1895,6 @@ }, { "group": "Reference", - "expanded": false, "pages": [ "vi/reference/failproof-cli", "vi/reference/local-dashboard", @@ -1934,10 +1922,10 @@ "it/start/quickstart", "it/reference/harnesses", "it/sessions/assistant", - "it/start/first-audit", - "it/start/first-policy", "it/start/setup", "it/start/concepts", + "it/start/first-audit", + "it/start/first-policy", { "group": "Starter templates", "expanded": false, @@ -1977,6 +1965,7 @@ { "group": "Plug in your agent", "pages": [ + "it/start/integrations", "it/start/integrations/custom-agents", "it/start/integrations/langchain", "it/start/integrations/crewai", @@ -2051,7 +2040,6 @@ "group": "Connect your agents", "pages": [ "it/reference/overview", - "it/reference/harnesses", "it/reference/custom-agents", "it/reference/evaluator-sdk", "it/reference/policy-sdk", @@ -2060,7 +2048,6 @@ }, { "group": "Reference", - "expanded": false, "pages": [ "it/reference/failproof-cli", "it/reference/local-dashboard", @@ -2088,10 +2075,10 @@ "ar/start/quickstart", "ar/reference/harnesses", "ar/sessions/assistant", - "ar/start/first-audit", - "ar/start/first-policy", "ar/start/setup", "ar/start/concepts", + "ar/start/first-audit", + "ar/start/first-policy", { "group": "Starter templates", "expanded": false, @@ -2131,6 +2118,7 @@ { "group": "Plug in your agent", "pages": [ + "ar/start/integrations", "ar/start/integrations/custom-agents", "ar/start/integrations/langchain", "ar/start/integrations/crewai", @@ -2205,7 +2193,6 @@ "group": "Connect your agents", "pages": [ "ar/reference/overview", - "ar/reference/harnesses", "ar/reference/custom-agents", "ar/reference/evaluator-sdk", "ar/reference/policy-sdk", @@ -2214,7 +2201,6 @@ }, { "group": "Reference", - "expanded": false, "pages": [ "ar/reference/failproof-cli", "ar/reference/local-dashboard", @@ -2242,10 +2228,10 @@ "he/start/quickstart", "he/reference/harnesses", "he/sessions/assistant", - "he/start/first-audit", - "he/start/first-policy", "he/start/setup", "he/start/concepts", + "he/start/first-audit", + "he/start/first-policy", { "group": "Starter templates", "expanded": false, @@ -2285,6 +2271,7 @@ { "group": "Plug in your agent", "pages": [ + "he/start/integrations", "he/start/integrations/custom-agents", "he/start/integrations/langchain", "he/start/integrations/crewai", @@ -2359,7 +2346,6 @@ "group": "Connect your agents", "pages": [ "he/reference/overview", - "he/reference/harnesses", "he/reference/custom-agents", "he/reference/evaluator-sdk", "he/reference/policy-sdk", @@ -2368,7 +2354,6 @@ }, { "group": "Reference", - "expanded": false, "pages": [ "he/reference/failproof-cli", "he/reference/local-dashboard", diff --git a/docs/index.mdx b/docs/index.mdx index 97984e9bc..1f249d615 100644 --- a/docs/index.mdx +++ b/docs/index.mdx @@ -1,41 +1,58 @@ --- title: "Make your agent failproof" -description: "Observability and enforcement for every harness your agents run in — coding CLIs, chat gateways, self-hosted assistants, and your own instrumented agents." +description: "See what your agents do, find failures, and stop them happening again." icon: "shield-check" --- -Failproof AI helps teams understand what agents did, find where they failed, and deploy safeguards before the same behavior happens again. +Failproof AI helps anyone running agents understand what happened, find failures, and prevent them from repeating. -A **harness** is whatever your agent actually runs inside. Failproof AI hooks 12 of them — coding CLIs like Claude Code and Codex, chat gateways like Hermes, self-hosted assistants like OpenClaw — and the same events, the same policies, and the same session history apply to every one. Agents with no harness report in through the [Python SDK](/reference/custom-agents), which traces and audits them; enforcing a policy there needs a hook in your own runtime. +It works with 12 common agent environments, including Claude Code, Codex, Hermes, OpenClaw, and Goose. Agents built with LangChain, CrewAI, LlamaIndex, Pydantic AI, or your own runtime can report through the [Python SDK](/reference/custom-agents). - - Use the skill to instrument your project, connect it, and verify that agent logs arrive. + + Install Failproof AI, connect your agents, and choose what to enforce. - - Analyze, query, build dashboards, and run audits in natural language on your agent logs. + + Query sessions, investigate failures, and build dashboards in natural language. +## Start locally or connect Cloud + + + + Open the dashboard at `localhost:8020` and run `failproofai audit`. Your agent history stays on this machine. + + + See sessions across machines and manage policies for your team. + + + +Setup deliberately chooses no policy pack. Add ours after setup: + +```bash +failproofai policies add FailproofAI/policies +``` + +Until then, only `block-failproofai-commands` runs. It stops an agent disabling Failproof AI. + +## What you can do + - - Follow model calls, tools, errors, human input, latency, and policy decisions in one session. + + Follow an agent run through model calls, tools, errors, and policy decisions. - - Audit a defined set of sessions, review evidence-backed findings, and track remediation as issues. + + Review evidence across one run or many. - Turn a known failure mode into a policy, observe its impact, and deploy it across your fleet. + Observe a safeguard on real activity, then enforce it when ready. -> **Session → Audit → Finding → Issue → Policy**
-> Trace what happened, find the failure, manage the response, then prevent the same behavior in future runs. - -## Start here - -If you are deploying your first instrumented agent, start with the [quickstart](/start/quickstart). If data is already arriving, open [Sessions](/sessions/overview) and inspect a real run before configuring audits or policies. +> **Session → Audit → Finding → Issue → Policy** +> See what happened, find the failure, own the response, then prevent it. - - Complete the end-to-end workflow from capture to a safely deployed policy. - + + Setup supports Linux and macOS. See [supported harnesses](/reference/harnesses) for what each agent environment can observe or block. + diff --git a/docs/policies/builtin-catalog.mdx b/docs/policies/builtin-catalog.mdx index 75029a507..87d1c1c7f 100644 --- a/docs/policies/builtin-catalog.mdx +++ b/docs/policies/builtin-catalog.mdx @@ -1,97 +1,134 @@ --- title: "Builtin policy catalog" -description: "Review every builtin Failproof AI policy, its trigger, recommended state, and configurable parameters." +description: "Review every builtin Failproof AI policy, its trigger, default state, and configurable parameters." icon: "list-checks" --- -The installed package is the source of truth for policy availability. Run `failproofai policies` after every upgrade because catalog entries and behavior can change with the package version. +38 of the 39 builtin policies are delivered as the `FailproofAI/policies` pack; `block-failproofai-commands` ships compiled into the package because a pack may not declare `alwaysOn`. The installed pack is the source of truth for what this machine can enforce: -## Recommended baseline +```bash +failproofai policies show FailproofAI/policies # the catalog, as published +failproofai policies # what is enabled here +``` + +`failproofai policies` lists custom files, convention files, installed packs and Cloud assignments. It has no builtin section, so it cannot answer "which builtins exist" — `policies show` and the [Policy Hub](https://befailproof.ai/policy-hub/FailproofAI/policies/) are where that question is answered. -The guided setup's recommended selection currently enables secret sanitizers, environment protections, self-protection, catastrophic-command guards, and protected-branch safety: +## Defaults, and how to select -```text -sanitize-jwt sanitize-api-keys -sanitize-connection-strings sanitize-private-key-content -sanitize-bearer-tokens protect-env-vars -block-env-files block-secrets-write -block-failproofai-commands block-sudo -block-curl-pipe-sh block-rm-rf -block-push-master block-force-push +A bare `failproofai policies add FailproofAI/policies` switches on the pack's own defaults — the 10 rows marked **on** below. `block-failproofai-commands` is on regardless and is not part of that selection. `--all` takes everything; `--category ` and `--policy ` take a slice. The slug beside each heading is what `--category` matches: + +```bash +failproofai policies add FailproofAI/policies --category git,database ``` -`block-failproofai-commands` is **always on**. It is listed above for -completeness, but it registers on every evaluation whether or not it appears in -your enabled set, and it cannot be disabled or paused — a guard against the -agent switching off enforcement that the agent can switch off is not a guard. +`block-failproofai-commands` is **always on**. It registers on every evaluation whether or not it appears in your selection, and it cannot be disabled or paused — a guard against the agent switching off enforcement that the agent can switch off is not a guard. A pack may not declare `alwaysOn`, which is why this one policy ships compiled into the package rather than in the pack. -Recommended is deliberately narrower than **Everything**. Infrastructure and workflow policies can interrupt valid work and should be enabled for the repositories and machines that need them. +## Sanitize — `sanitize` -## Secrets and environment +These run on `PostToolUse`, after the tool has already run. They **detect and refuse the tool result**; they do not redact a substring and hand back the rest. -| Policy | Trigger | Outcome | -| --- | --- | --- | -| `sanitize-jwt` | `PostToolUse` | Redact JWTs from tool output before the model sees them. | -| `sanitize-api-keys` | `PostToolUse` | Redact common OpenAI, Anthropic, GitHub, AWS, Stripe, and Google keys. | -| `sanitize-connection-strings` | `PostToolUse` | Redact database connection strings containing credentials. | -| `sanitize-private-key-content` | `PostToolUse` | Redact PEM private-key bodies. | -| `sanitize-bearer-tokens` | `PostToolUse` | Redact authorization bearer tokens. | -| `protect-env-vars` | `PreToolUse` on shell tools | Block commands that dump environment variables. | -| `block-env-files` | `PreToolUse` | Block reads and writes of `.env` files. | -| `block-read-outside-cwd` | `PreToolUse` on read, glob, grep, or shell tools | Keep reads inside the session working directory. | -| `block-secrets-write` | `PreToolUse` on write tools | Block writes to common secret-key and credential filenames. | - -## Dangerous commands and infrastructure - -| Policy | Trigger | Outcome | -| --- | --- | --- | -| `block-sudo` | `PreToolUse`, `PermissionRequest` | Block `sudo` unless an allow pattern matches. | -| `block-curl-pipe-sh` | `PreToolUse` | Block downloaded scripts piped directly to a shell. | -| `block-rm-rf` | `PreToolUse` | Block catastrophic recursive deletion patterns. | -| `block-failproofai-commands` | `PreToolUse`, `PermissionRequest` | **Always on, cannot be disabled.** Block every Failproof AI CLI invocation, self-pause, and package-manager uninstall. | -| `block-kubectl` | `PreToolUse` | Gate Kubernetes commands. | -| `block-terraform` | `PreToolUse` | Gate Terraform and OpenTofu commands. | -| `block-aws-cli` | `PreToolUse` | Gate AWS CLI commands. | -| `block-gcloud` | `PreToolUse` | Gate Google Cloud CLI commands. | -| `block-az-cli` | `PreToolUse` | Gate Azure CLI commands. | -| `block-helm` | `PreToolUse` | Gate Helm commands. | -| `block-gh-pipeline` | `PreToolUse` | Gate mutating GitHub CLI workflow, run, merge, release, cache, and secret operations. | - -## Git and database safety - -| Policy | Trigger | Outcome | -| --- | --- | --- | -| `block-push-master` | `PreToolUse` | Block direct pushes to configured protected branches. | -| `block-force-push` | `PreToolUse` | Block force-pushes; `--force-with-lease` remains allowed by the current implementation. | -| `block-work-on-main` | `PreToolUse` | Block commits and merges on protected branches. | -| `warn-git-amend` | `PreToolUse` | Warn before rewriting a commit with `--amend`. | -| `warn-git-stash-drop` | `PreToolUse` | Warn before permanently dropping or clearing stashes. | -| `warn-all-files-staged` | `PreToolUse` | Warn on broad `git add -A`, `git add .`, or `git add --all`. | -| `warn-destructive-sql` | `PreToolUse` | Warn on `DROP`, `TRUNCATE`, and `DELETE` without `WHERE` through recognized database clients. | -| `warn-schema-alteration` | `PreToolUse` | Warn on recognized `ALTER TABLE` column and rename operations. | - -## Packages, system behavior, and agent loops - -| Policy | Trigger | Outcome | -| --- | --- | --- | -| `warn-package-publish` | `PreToolUse` | Warn before publishing to package registries. | -| `warn-global-package-install` | `PreToolUse` | Warn before global package installation. | -| `prefer-package-manager` | `PreToolUse` | Instruct the agent to use an allowed package manager. | -| `warn-large-file-write` | `PreToolUse` on write tools | Warn above the configured file-size threshold. | -| `warn-background-process` | `PreToolUse` | Warn on detached or long-lived background-process patterns. | -| `warn-repeated-tool-calls` | `PreToolUse` | Warn after three or more identical tool calls. | +A refusal here only reaches the model on the harnesses that consume a `PostToolUse` verdict — **codex** and **copilot**, where the reason replaces the whole tool result. On claude, cursor, opencode, pi, hermes, openclaw, factory, devin, antigravity and goose, `PostToolUse` is observe-only: the detection is recorded and the output still reaches the model. + +| Policy | Trigger | Default | Outcome | +| --- | --- | --- | --- | +| `sanitize-jwt` | `PostToolUse` | on | Refuse a tool result containing a JWT. | +| `sanitize-api-keys` | `PostToolUse` | on | Refuse a tool result containing an OpenAI, Anthropic, GitHub, AWS, Stripe or Google key. | +| `sanitize-connection-strings` | `PostToolUse` | on | Refuse a tool result containing a database connection string with embedded credentials. | +| `sanitize-private-key-content` | `PostToolUse` | on | Refuse a tool result containing PEM private-key content. | +| `sanitize-bearer-tokens` | `PostToolUse` | on | Refuse a tool result containing an `Authorization: Bearer` token. | + +## Environment — `environment` + +| Policy | Trigger | Default | Outcome | +| --- | --- | --- | --- | +| `protect-env-vars` | `PreToolUse` on `Bash` | on | Block commands that read environment variables. | +| `block-env-files` | `PreToolUse` | on | Block reads and writes of `.env` files. | +| `block-read-outside-cwd` | `PreToolUse` on `Read`, `Glob`, `Grep`, `Bash` | off | Keep file reads inside the session working directory. | + +## Dangerous commands — `dangerous-commands` + +| Policy | Trigger | Default | Outcome | +| --- | --- | --- | --- | +| `block-sudo` | `PreToolUse`, `PermissionRequest` on `Bash` | on | Block `sudo` unless an allow pattern matches. | +| `block-curl-pipe-sh` | `PreToolUse` on `Bash` | on | Block downloaded scripts piped directly to a shell. | +| `block-failproofai-commands` | `PreToolUse`, `PermissionRequest` on `Bash`, `Write`, `Edit`, `NotebookEdit` | **always on** | Block every Failproof AI CLI invocation, self-pause and uninstallation. | +| `block-rm-rf` | `PreToolUse` on `Bash` | off | Block catastrophic recursive deletion patterns. | +| `block-secrets-write` | `PreToolUse` on `Write` | off | Block writes to common secret-key and credential filenames. | + +## Infra commands — `infra-commands` + +All seven are off by default: they gate tools that legitimate work uses constantly, so they belong to the repositories and machines that need them. + +| Policy | Trigger | Default | Outcome | +| --- | --- | --- | --- | +| `block-kubectl` | `PreToolUse` on `Bash` | off | Gate `kubectl` cluster mutations. | +| `block-terraform` | `PreToolUse` on `Bash` | off | Gate `terraform` and `tofu` commands. | +| `block-aws-cli` | `PreToolUse` on `Bash` | off | Gate `aws` CLI commands. | +| `block-gcloud` | `PreToolUse` on `Bash` | off | Gate `gcloud` commands. | +| `block-az-cli` | `PreToolUse` on `Bash` | off | Gate `az` commands. | +| `block-helm` | `PreToolUse` on `Bash` | off | Gate `helm` commands. | +| `block-gh-pipeline` | `PreToolUse` on `Bash` | off | Gate mutating `gh` operations: workflow run, run rerun and cancel, pr merge, release create and delete, cache delete, secret set and delete. Read-only subcommands such as `gh pr view` are not matched. | -## End-of-task workflow +## Git — `git` -These policies require a harness that emits a compatible `Stop` event. +| Policy | Trigger | Default | Outcome | +| --- | --- | --- | --- | +| `block-push-master` | `PreToolUse` on `Bash` | on | Block direct pushes to configured protected branches. | +| `block-force-push` | `PreToolUse` on `Bash` | off | Block force-pushes. `--force-with-lease` and `--force-if-includes` remain allowed. | +| `block-work-on-main` | `PreToolUse` on `Bash` | off | Block commits and merges on protected branches. | +| `warn-git-amend` | `PreToolUse` on `Bash` | off | Warn before rewriting a commit with `--amend`. | +| `warn-git-stash-drop` | `PreToolUse` on `Bash` | off | Warn before permanently dropping or clearing stashes. | +| `warn-all-files-staged` | `PreToolUse` on `Bash` | off | Warn on broad `git add -A`, `git add .` or `git add --all`. | -| Policy | Outcome | +## Database — `database` + +| Policy | Trigger | Default | Outcome | +| --- | --- | --- | --- | +| `warn-destructive-sql` | `PreToolUse` on `Bash` | off | Warn on `DROP`, `TRUNCATE`, and `DELETE` without `WHERE` through recognized database clients. | +| `warn-schema-alteration` | `PreToolUse` on `Bash` | off | Warn on recognized `ALTER TABLE` column and rename operations. | + +## Packages and system — `packages-system` + +| Policy | Trigger | Default | Outcome | +| --- | --- | --- | --- | +| `warn-package-publish` | `PreToolUse` on `Bash` | off | Warn before publishing to npm, PyPI, crates.io, RubyGems and similar registries. | +| `warn-global-package-install` | `PreToolUse` on `Bash` | off | Warn before global package installation. | +| `prefer-package-manager` | `PreToolUse` on `Bash` | off | Block a non-preferred package manager and instruct the agent to use an allowed one. | +| `warn-large-file-write` | `PreToolUse` on `Write` | off | Warn above the configured file-size threshold. | +| `warn-background-process` | `PreToolUse` on `Bash` | off | Warn on detached or background-process patterns. | + +## AI behavior — `ai-behavior` + +| Policy | Trigger | Default | Outcome | +| --- | --- | --- | --- | +| `warn-repeated-tool-calls` | `PreToolUse` | off | Warn when the same tool is called three or more times with identical parameters. | + +## Workflow — `workflow` + +All five run on `Stop` and are off by default. + +| Policy | Default | Outcome | +| --- | --- | --- | +| `require-commit-before-stop` | off | Refuse completion while tracked work remains uncommitted. | +| `require-push-before-stop` | off | Refuse completion while commits remain local-only. | +| `require-pr-before-stop` | off | Require a pull request for the current branch. | +| `require-no-conflicts-before-stop` | off | Require a clean merge against the configured base branch. | +| `require-ci-green-before-stop` | off | Require CI checks on the current HEAD commit to pass, ignoring stale runs on prior commits. | + +These need a harness whose `Stop` verdict is consumed. That is not every harness: + +| Harness | `Stop` | | --- | --- | -| `require-commit-before-stop` | Refuse completion while tracked work remains uncommitted. | -| `require-push-before-stop` | Refuse completion while commits remain local-only. | -| `require-pr-before-stop` | Require a pull request for the current branch. | -| `require-no-conflicts-before-stop` | Require a clean merge against the configured base branch. | -| `require-ci-green-before-stop` | Require current-HEAD CI checks to complete successfully. | +| claude, codex, copilot, cursor, openclaw, factory, devin, antigravity | Verified to block: the refusal forces another turn | +| pi | Observe. The reason is carried into the next turn as an instruction, not a gate | +| goose, hermes | No `Stop` hook is installed, so these five never fire | +| opencode | Not verified. `Stop` is not among the events opencode consumes a verdict from | + +Cursor Cloud Agent VMs run no stop hooks at all, so a Cursor session there is uncovered even though local Cursor is not. + + + The `warn-*` policies return `instruct`, not `deny`. (`prefer-package-manager` is the exception among the non-`block-*` names: it returns `deny`, so it blocks on every harness that consumes a `PreToolUse` verdict — which is all twelve.) On Hermes and Goose, on Pi, OpenClaw and Factory outside the `Stop` channel, and on Antigravity outside `Stop` and `UserPromptSubmit`, `instruct` degrades to allow plus a note on stderr — the agent is not told. Antigravity's `UserPromptSubmit` is a real second channel: the instruction is injected as a transient message before the model runs. + ## Parameter reference @@ -115,7 +152,6 @@ Configure parameters under the selected scope's `policyParams` object. Types are ```json { - "enabledPolicies": ["block-sudo", "block-push-master"], "policyParams": { "block-sudo": { "allowPatterns": ["sudo systemctl status"] @@ -127,6 +163,10 @@ Configure parameters under the selected scope's `policyParams` object. Types are } ``` + + A bare policy name as a `policyParams` key is honoured for `FailproofAI/policies` only. For any other pack the key is `pack///` — a stranger's pack declaring the same policy name gets the schema defaults, not your parameters. + + An allow pattern broadens what an agent may do. Test the exact tokenization and command variants on the target harness before deploying it across a fleet. diff --git a/docs/policies/builtin.mdx b/docs/policies/builtin.mdx index 1dd24e61b..c9a430fc8 100644 --- a/docs/policies/builtin.mdx +++ b/docs/policies/builtin.mdx @@ -1,57 +1,100 @@ --- title: "Builtin policies" -description: "Enable maintained guardrails for common agent failure modes." +description: "Take the maintained guardrails for common agent failure modes, and turn on the ones you want." icon: "library" --- -Builtin policies cover secret handling, environment files, destructive shell commands, protected branches, cloud and infrastructure tools, package publishing, repeated calls, and end-of-task workflow checks. +The builtins are 39 maintained policies covering nine categories of agent failure. All but one are **delivered as a pack**, `FailproofAI/policies`, the same way anybody else's policies are delivered. The package ships no pack of its own, so a fresh install enforces nothing until you take it: -## Enable and verify a builtin policy +```bash +failproofai policies add FailproofAI/policies +``` - - - 1. Install the policy on a connected machine with the local CLI. - 2. Run a safe test action in the instrumented agent. - 3. Go to **Observe → policy** and filter by the policy name, machine environment, or decision. - 4. Open the linked session to confirm the matched tool input and returned reason. +That switches on the pack's own defaults — 10 of the 38 policies the pack carries. The 39th is `block-failproofai-commands`, which is on regardless: it ships compiled into the package, registers on every evaluation, and cannot be disabled or paused. A pack may not declare `alwaysOn`, which is why that one guard does not travel the pack lane. - +## What the nine categories cover + +| Category | `--category` slug | Policies | +| --- | --- | --- | +| Sanitize | `sanitize` | 5 | +| Environment | `environment` | 3 | +| Dangerous Commands | `dangerous-commands` | 5 | +| Infra Commands | `infra-commands` | 7 | +| Git | `git` | 6 | +| Database | `database` | 2 | +| Packages & System | `packages-system` | 5 | +| AI Behavior | `ai-behavior` | 1 | +| Workflow | `workflow` | 5 | + +Those are the compiled catalog's counts, and they total 39. The pack carries 38 of them, because `block-failproofai-commands` is `alwaysOn` and never travels the pack lane — so `--category dangerous-commands` selects the other four. + +Take a slice instead of the defaults: + +```bash +failproofai policies add FailproofAI/policies --category git,database +failproofai policies add FailproofAI/policies --policy block-rm-rf +failproofai policies add FailproofAI/policies --all +``` + +## Read the catalog before you take it + + ```bash + failproofai policies show FailproofAI/policies failproofai policies - failproofai policy add block-rm-rf --cli claude --scope project - failproofai config --status ``` - Remove it with `failproofai policy remove block-rm-rf --cli claude --scope project`. + `policies show` reads the published manifest — every policy, grouped by category, marked default or opt-in — without downloading or importing the pack's code. `failproofai policies` lists what is enabled on this machine: custom files, convention files, installed packs and Cloud assignments. It has no builtin section, so it answers "what is on here", never "what exists". + + + Browse the same catalog in a browser, without the CLI, at [befailproof.ai/policy-hub](https://befailproof.ai/policy-hub/). Each pack has a page at `/policy-hub///` and each policy a page at `/policy-hub////`. + + + 1. Install the policy on a connected machine with the local CLI. + 2. Run a safe test action in the instrumented agent. + 3. Go to **Observe → policy** and filter by the policy name, machine environment, or decision. + 4. Open the linked session to confirm the matched tool input and returned reason. -List the policies available in your installed version: +## Turn one on or off ```bash -failproofai policies +failproofai policies add block-rm-rf --cli claude --scope project +failproofai policies remove block-rm-rf --cli claude --scope project ``` -Enable one policy for a project: - -```bash -failproofai policy add block-rm-rf --scope project -``` - -Enable several policies for selected harnesses: +`policies add` and `policies remove` take exactly **one** policy name. Run them with no name at all and you get a picker, with what is already on ticked. For several names at once, use the install form: ```bash failproofai policies --install block-sudo block-force-push \ --cli claude codex --scope project ``` -Some policies accept parameters or are marked beta. Review the description, match scope, and default behavior before rollout. A policy that protects one workflow may block valid operations in another. + + On a machine with no pack installed, `failproofai policies add ` fetches `FailproofAI/policies` from its GitHub release to satisfy the name — so that first command needs the network. + + +## Not every policy enforces on every harness + +A policy only changes behavior where the harness consumes the verdict for its event. Two cases are worth checking before you rely on a builtin: + +| Event | Where a deny is verified to change behavior | +| --- | --- | +| `PreToolUse` | All 12 harnesses | +| `Stop` | claude, codex, copilot, cursor, openclaw, factory, devin, antigravity. No `Stop` hook is installed on goose or hermes, pi carries the reason into the next turn instead, and opencode is not verified | + +So the five `require-*-before-stop` policies in the Workflow category can be enabled on a machine and still never fire, depending on which agent runs there. Cursor Cloud Agent VMs run no stop hooks at all. + +The `warn-*` and `prefer-*` policies return `instruct` rather than `deny`. On Hermes and Goose, and outside the `Stop` channel on Pi, OpenClaw, Factory and Antigravity, `instruct` degrades to allow plus a note on stderr: the operator sees it in the logs, the agent does not. + +Some policies accept parameters. Review the description, match scope and default before rollout — a policy that protects one workflow may block valid operations in another. - Review all 40 current policies, their triggers, recommended baseline, and parameters. + All 39 builtin policies, by category, with their triggers, defaults and parameters. - Prefer project scope for repository-specific expectations and user scope for machine-wide safety requirements. + `failproofai config` wires every supported agent at user scope and chooses no policies — it has no `--scope` flag. Pass `--scope project` to `failproofai policies add` or `failproofai policies --install` only when the expectation genuinely belongs to one repository. diff --git a/docs/policies/custom.mdx b/docs/policies/custom.mdx index 61ea52783..d56d98db8 100644 --- a/docs/policies/custom.mdx +++ b/docs/policies/custom.mdx @@ -1,72 +1,51 @@ --- title: "Custom policies" -description: "Write a policy for a failure mode unique to your agent workflow." -icon: "shield-plus" +description: "Write a rule for behavior specific to your agent or workflow." +icon: "code-2" --- -Create a file ending in `policies.js`, `policies.mjs`, or `policies.ts` under `.failproofai/policies/`. Convention files load automatically at project and user scope. +Check the [builtin catalog](/policies/builtin-catalog) before writing a policy. A reviewed rule is usually safer than a new one. -## Test the policy before Cloud publication +## Start from a working policy - - - 1. Install the custom policy on one test machine and trigger both a matching action and a legitimate non-match. - 2. Go to **Observe → policy** and compare the two decisions. - 3. Open each linked session and verify the event payload contains enough evidence for the rule. - 4. When the behavior is correct, move the reviewed source into **Admin → policy editor** and publish a version. - - - - ```bash - failproofai policies --install --custom ./security.policies.ts \ - --cli claude --scope project - failproofai policies - ``` +```bash +failproofai publish --init guards.mjs +failproofai policies -i -c ./guards.mjs +``` - Convention files under `.failproofai/policies/` load without `--custom`. Keep an explicit install command in CI when validation should fail on a broken module. - - +The starter blocks `git push --force`. Edit it, ask your agent to try the blocked action, and inspect **Policies → Activity**. -```ts +```js import { customPolicies, allow, deny } from "failproofai"; customPolicies.add({ - name: "protect-production-paths", - description: "Block writes to production configuration", - match: { events: ["PreToolUse"] }, - fn: async (ctx) => { - if (ctx.toolName !== "Write" && ctx.toolName !== "Edit") return allow(); - const path = String(ctx.toolInput?.file_path ?? "").replaceAll("\\", "/"); - if (path.split("/").includes("production")) { - return deny("Writes to production configuration require approval."); - } - return allow(); - }, + name: "protect-production", + description: "Production changes need a human", + match: { events: ["PreToolUse"], toolNames: ["Bash"] }, + fn: async (ctx) => + String(ctx.toolInput?.command ?? "").includes("production") + ? deny("Ask a human before changing production.") + : allow(), }); ``` -This matches `production/config.yml`, `/srv/production/config.yml`, `/srv/production`, and `C:\\production\\config.yml` for both `Write` and `Edit`. It does not match names such as `production-backup` because `production` must be a complete path segment. +A policy returns `allow()`, `deny(message)`, or `instruct(message)`. Use `deny` when the action must stop; not every harness can deliver an instruction back to the agent. -Validate and install an explicit file: +## Load automatically -```bash -failproofai policies --install --custom ./security.policies.ts -``` +Put files named `*policies.js`, `*policies.mjs`, or `*policies.ts` under: -The policy context includes the event type, normalized payload, tool name and input, session metadata, parameters, and source CLI when available. +- `.failproofai/policies/` for one project. +- `~/.failproofai/policies/` for your user. ## Test failure paths -Run validation after changing the entry file or any local module it imports: +Test both the unsafe action and legitimate work that looks similar. Confirm the decision came from your policy rather than another rule. + +Remove explicit test files with: ```bash -failproofai policies --install --custom ./security.policies.ts --scope project +failproofai policies -u -c ``` -The strict CLI path fails for missing files, syntax errors, unresolved imports, top-level exceptions, and module-load timeouts. At enforcement time, a broken custom file is logged and skipped so builtin policies can continue. Treat any load warning as a loss of expected enforcement and alert on it in production logs. - -Use globally unique names across explicit, convention, and Cloud-managed policies. Keep policy functions deterministic, bound external calls with short timeouts, and return an intentional `allow`, `instruct`, or `deny` on every path. - - - A custom policy is enforcement code. Test missing fields, alternate tool names, and malformed input—not only the expected match. - +When the policy is ready, [publish a pack](/policies/publish-a-pack) or [deploy it through Cloud](/policies/deploy). diff --git a/docs/policies/deploy.mdx b/docs/policies/deploy.mdx index c918125dc..2bf970a53 100644 --- a/docs/policies/deploy.mdx +++ b/docs/policies/deploy.mdx @@ -1,51 +1,54 @@ --- title: "Deploy policies" -description: "Roll out a reviewed policy version to the intended machines." +description: "Observe a policy on real agent activity, then enforce or roll it back." icon: "cloud-upload" --- -A deployment connects one or more policy versions to a target set of enrolled machines. +Deploy a reviewed policy to one machine at a time. Start in observe mode. -## Apply a deployment +## Roll out from the CLI - - - 1. Go to **Admin → enforcement**, find the machine, and expand its row. - 2. Select **edit**, add the reviewed policy version, and choose **observe** or its enforcing effect. - 3. Apply the change, then wait for the machine's next check-in and confirm its deployment and coverage state. - 4. Go to **Observe → policy** to inspect live decisions. +```bash +fp policies test ./rule.mjs --tool Bash --command "git push --force" --expect deny +fp policies publish no-force-push ./rule.mjs +fp fleet deploy --add no-force-push:observe +fp guardrails summary --since 24h --machine +fp fleet deploy --add no-force-push:enforce +``` - ![The machine deployment editor with policy versions, enforce and observe effects, and the apply deployment action.](/images/dashboard/enforcement-editor.png) - - - Deploy from the CLI with `fp fleet`. Review the resulting set before applying it — `deploy` prints the full plan and asks **only on an interactive terminal without `--json`**. Under `--json`, with `--yes`, or with stdin redirected (a CI step, a script, an agent shelling out) it applies immediately with no plan and no prompt — so run `fp fleet show ` first if you want review: +Observe mode evaluates the real policy and records non-allow decisions, but does not block the agent. - ```bash - fp fleet list - fp fleet show - fp fleet deploy --add no-force-push - ``` + + A bare `--add no-force-push` enforces immediately. Add `:observe` for a shadow rollout. + - `fp fleet diff ` shows intent vs delivery (a machine reads as `behind` until it next polls), `fp fleet history ` lists the generations, and `fp fleet rollback ` reinstates one — it refuses if that generation names a policy since disabled or deleted. +If enforcement causes problems: - Check the machine itself with `failproofai config --status`, and use `fp sessions --env production --since 24h` and `fp events --event-type hook_completed` after deployment to verify activity reaches Cloud. - - +```bash +fp fleet history +fp fleet rollback +``` - - - Deploy a reviewed version, not a mutable draft, starting with a non-production machine or small cohort whose sessions you can inspect. - - - Review matches, reasons, affected tools, and false positives without blocking work. - - - Promote after observed matches separate unsafe actions from valid ones, then confirm every intended machine has pulled the deployment and is reporting decisions. - - +## Deploy from the dashboard -Machines need the `policies:pull` capability. Event reporting is separately controlled by `events:add`; verify both when you expect Cloud analysis and enforcement. +1. Go to **Admin → enforcement**. +2. Open the target machine. +3. Add the policy version with effect **observe**. +4. Apply the deployment. +5. Review results under **Observe → policy**. +6. Promote the same version to **enforce** when the matches are correct. - - Enforcement management is an administrative Cloud workflow. Do not treat root-only enforcement routes as ordinary customer `/v1` API endpoints. - +![The machine deployment editor with policy versions, enforce and observe effects, and the apply deployment action.](/images/dashboard/enforcement-editor.png) + +## Replace or pre-stage a set + +- `--remove ` removes one policy. +- `--set ...` replaces the complete policy set. +- `--create` prepares a deployment before a machine first checks in. +- `fp fleet diff ` compares intended and applied state. + + + Without Cloud, publish a pack with `failproofai publish --effect observe` and inspect decisions in the local dashboard. + + +Machines need `policies:pull` to receive deployments and `events:add` to report decisions. diff --git a/docs/policies/failure-behavior.mdx b/docs/policies/failure-behavior.mdx index 15865cd3c..c9597bd1d 100644 --- a/docs/policies/failure-behavior.mdx +++ b/docs/policies/failure-behavior.mdx @@ -1,67 +1,48 @@ --- -title: "Failure behavior" -description: "Understand what happens when policy evaluation or the local daemon is unavailable." +title: "Policy failure behavior" +description: "Understand why Failproof AI denies when evaluation cannot run." icon: "shield-alert" --- -Failproof AI is designed so an enforcement failure is visible rather than silently allowing risky work. +After setup, the local `failproofaid` service is the only evaluator. If it cannot answer, protected actions are denied rather than silently allowed. -## Diagnose a failure-closed block +## Diagnose a machine-wide block - - - 1. Go to **Admin → enforcement** and open the machine. - 2. Check its last check-in, assigned deployment, and reported deployment. - 3. Go to **Observe → policy** and open the denied decision's session. - 4. Confirm whether the reason reports daemon reachability, version skew, or the policy itself. - - - - ```bash - failproofai config --status - npm install -g failproofai@latest - failproofai config - ``` - - Rerunning `failproofai config` updates and restarts the daemon after a package upgrade. - - - -On a machine configured to use `failproofaid`, the daemon is the only evaluator. If it is unreachable or its protocol version does not match the CLI, hook evaluation fails closed. The action is denied with a reason that directs the operator to check or update the daemon. +```bash +failproofai config --status +systemctl status failproofaid@$USER # Linux +sudo launchctl print system/ai.failproof.failproofaid.$USER # macOS +``` -Before daemon configuration, hooks evaluate policies in process. Once daemon configuration is recorded, Failproof AI does not silently fall back to a second evaluator when the daemon fails. +Run `failproofai config` to repair the service or install the matching version. -## Respond to a failure-closed decision +Two failures can look similar: -1. Run `failproofai config --status`. -2. If versions differ, rerun `failproofai config` after updating the package. -3. If the daemon is unreachable, inspect its service state and local logs. -4. Resume agent work only after a known policy evaluation path is healthy. +| Failure | Meaning | +| --- | --- | +| Service unreachable | The socket has no working evaluator behind it | +| Version mismatch | The CLI and service disagree on the protocol | - - Do not repeatedly retry the blocked action. A failure-closed response means the system could not establish that the action was safe. - +Both deny, but status reports them separately. ## A pack will not load -A machine that was told to enforce a pack, and cannot run it, denies rather than continuing quietly. The trigger is a **recorded expectation**, never an empty one: a machine with no packs installed is silent, while a pack that is declared and will not resolve — or that registers less than its manifest declares — denies. +A selected pack that is missing, changed, or invalid denies the events covered by its selected policies. This prevents a broken pack from disappearing while the machine appears protected. -The deny is **narrow**, unlike an unreachable daemon. A daemon that cannot be reached means no evaluation happened at all, so nothing can be known safe. A pack that will not load has an enumerable set of missing guards, because every declared policy carries its own `match` — so it denies only the events and tools those policies covered, and everything else proceeds. +`failproofai policies` names the pack and the load error. -It does not fire for: +```bash +failproofai policies +failproofai policies remove owner/repo +failproofai policies add owner/repo@ +``` -- an `observe` pack, which evaluates and discards by construction -- policies you never took, or explicitly turned off -- a pack the loader never received, where "no registrations" cannot be told apart from a deliberate skip -- an active session pause -- a load timeout, which is transient — one slow disk moment must not deny until a human intervenes +Unreadable metadata widens the safe deny instead of narrowing it from data that could not be trusted. -`UserPromptSubmit` **instructs** instead of denying, whatever the missing policy declared. A blanket deny would take it along and lock you out of the agent that could fix the problem. +## What remains available -### What to do +The local dashboard and status commands still work while policy evaluation is failing. Use them to identify the service, pack, or version that needs repair. -```bash -failproofai pack list -``` - -It names any installed pack that will not load, says why, and exits non-zero. Then either reinstall it (`failproofai pack add `) or remove it (`failproofai pack remove `) — removing it withdraws the expectation, and the deny stops with it. + + Do not work around a failure-closed decision by deleting service or pack files. Repair the service, reinstall the pack, or remove the assignment through the CLI so the machine returns to a known state. + diff --git a/docs/policies/fleet.mdx b/docs/policies/fleet.mdx index e4f337092..f9b96eb32 100644 --- a/docs/policies/fleet.mdx +++ b/docs/policies/fleet.mdx @@ -1,52 +1,61 @@ --- -title: "Deploy policies to machines" -description: "Know which machines are enrolled, current, and enforcing the intended policy versions." +title: "Fleet coverage" +description: "See which machines received and applied the intended policies." icon: "network" --- -Fleet coverage answers whether a policy exists where the risk exists. Track machines by stable ID and a human-readable label, then compare their assigned and reported deployment state. +Fleet coverage compares what Cloud assigned with what each machine last applied. -## Check coverage +## Check a machine - - - 1. Go to **Admin → enforcement** and review the enforcing and observing totals. - 2. Search for a machine by ID or label, or filter for machines missing a policy. - 3. Expand a row to compare assigned policies, reported deployment, last check-in, and history. - 4. Refresh after the machine's polling interval when an applied deployment remains pending. +```bash +failproofai config --status +failproofai flush --wait --timeout 120 +``` - ![The Enforcement fleet showing policy coverage, machine deployment state, and observe and enforce assignments.](/images/dashboard/enforcement-fleet.png) - - - ```bash - failproofai config --status - failproofai config --machine-label checkout-runner-03 - failproofai flush --wait - ``` +From Cloud: + +```bash +fp fleet list +fp fleet diff +fp fleet show +``` - Use `fp events --agent-id --since 24h` to confirm the machine's agent activity reaches Cloud. - - +`fp fleet` requires a signed-in session, not an API key. -Use coverage views to find: +| Command | What it does | +| --- | --- | +| `fp fleet list` | List machines and deployment state | +| `fp fleet show ` | Show one machine's policy set | +| `fp fleet deploy --add ` | Add or update one policy | +| `fp fleet deploy --remove ` | Remove one policy | +| `fp fleet deploy --set ...` | Replace the whole set | +| `fp fleet diff [machine]` | Compare intended and applied versions | +| `fp fleet history ` | List deployment generations | +| `fp fleet rollback ` | Restore a generation | +| `fp fleet rename " +## Pause is not a rollback + +A pause suspends every locally-installed policy for **one session**, and always expires on its own — packs, custom files, convention files and builtins alike. Only Cloud-managed policies and the always-on `block-failproofai-commands` guard keep enforcing. So a pause is not a workaround for a bad Cloud deployment, and it is a wider blast radius than it reads like: a pack is where essentially all local enforcement lives, and the pack fail-closed guard is skipped for the duration too. + +| Command | Effect | +|---|---| +| `failproofai config --pause` | This directory's newest agent session, for 30 minutes | +| `failproofai config --pause 10m` | A given time. Maximum 8h; `s`/`m`/`h` suffixes, a bare number means minutes | +| `failproofai config --pause --session ` | Target a specific session | +| `failproofai config --resume` | End the pause early | +| `failproofai config --resume --all` | End every active pause | +| `failproofai config --status` | What is paused, and when it lifts | + ## When to roll back - A policy blocks an expected production action. @@ -34,7 +52,16 @@ Rollback changes the deployed version or removes a policy assignment; it does no - A policy depends on fields that an integration does not provide. - A new version changes behavior outside the intended failure mode. -After rollback, open the affected sessions and identify the condition that caused the false positive. Create a new version, test both the unsafe and legitimate cases, then repeat the observe phase. +After rollback, open the affected sessions and identify the condition that caused the false positive. Create a new version, test both the unsafe and the legitimate case locally, then go back through observe before you enforce again: + +```bash +fp policies test ./policy.mjs --tool Bash --command '' --expect allow +fp policies publish my-policy ./policy.mjs +fp fleet deploy ci-runner-01 --add my-policy:observe +fp guardrails summary --since 24h --machine ci-runner-01 +``` + +`:observe` is what makes that a shadow rollout. A bare `fp fleet deploy ... --add my-policy` **enforces immediately** — an omitted effect resolves to the effect already deployed, and then to `enforce` — which on a policy you have just rolled back means reinstating the incident. Observe mode still evaluates the policy for real and records every non-allow verdict; only the enforcement is withheld, so `fp guardrails summary` measures the new version against the same traffic that broke on the old one. Promote with `--add my-policy:enforce` when the numbers say so. Pausing enforcement can be appropriate during an incident, but it widens exposure for every active policy in that scope. Prefer rolling back the specific policy version when possible. diff --git a/docs/reference/cloud-cli.mdx b/docs/reference/cloud-cli.mdx index 946200cc5..a26357c2a 100644 --- a/docs/reference/cloud-cli.mdx +++ b/docs/reference/cloud-cli.mdx @@ -6,15 +6,19 @@ icon: "cloud-cog" Use `fp` to inspect Cloud telemetry, manage cloud-managed enforcement (policies, fleet deployments, guardrail decisions), and manage audits, findings, issues, alerts, keys, users, queries, and settings. Use [`failproofai`](/reference/failproof-cli) for local hooks, policies, capture, and machine enrollment. -Install the released Cloud CLI as an isolated tool: +Install the released Cloud CLI as an isolated tool. It needs Python 3.10 or newer. ```bash uv tool install fp-cloud-cli fp version ``` +The distribution is `fp-cloud-cli` and the installed command is `fp` — they differ because `fp` was already taken on PyPI. + ## Sign in +`fp login` emails you a six-digit code and saves the session. Against a dashboard with a self-signed certificate, add `--insecure` at login time. + ```bash fp login fp whoami @@ -191,7 +195,7 @@ Permission tokens use `resource:action`, such as `events:add`. Repeat `--add`, c | `fp query create NAME` | Save a query. | `--sql `; `--description` | | `fp query update NAME` | Update or rename a query. | `--name`; `--sql`; `--description`; `--yes`, `-y` | | `fp query delete NAME` | Delete a saved query. | `--yes`, `-y` | -| `fp query run [NAME]` | Run a saved query or ad-hoc SQL. | `--sql`; `--limit`; `--all`; `--arg`, `--param` | +| `fp query run [NAME]` | Run a saved query or ad-hoc SQL. `--limit` caps the **table** view only, at 50 by default; `--json` always returns every row. | `--sql`; `--limit`; `--all`; `--arg`, `--param` | | `fp query schema [TABLE]` | List queryable tables or inspect one table. | — | ### Users @@ -240,7 +244,7 @@ Alert severities are `info`, `warning`, and `critical`. Trigger kinds are `metri | `fp audits context-show NAME` | Show the brief and reference URL fetch state. | — | | `fp audits context-set NAME` | Change the brief or reference URLs. | `--text`; `--text-file`; `--url`; `--clear-urls` | | `fp audits context-refresh NAME` | Re-fetch reference URLs. | — | -| `fp audits findings` | List findings. | `--audit`; `--run-id`; `--status`; `--limit`, `-n`; `--offset`; `--show-id` | +| `fp audits findings` | List findings. `--limit` defaults to 100 here, not the 50 used elsewhere, and the server caps it at 500. | `--audit`; `--run-id`; `--status`; `--limit`, `-n`; `--offset`; `--show-id` | | `fp audits finding FINDING_ID` | Show one finding and its evidence. | — | | `fp audits ack FINDING_ID` | Acknowledge a finding. | `--reason` | | `fp audits mute FINDING_ID` | Suppress a recurring pattern. | `--reason`; `--yes`, `-y` | @@ -321,32 +325,77 @@ Valid issue states are `firing`, `acknowledged`, and `resolved`. Standalone issu ### Policies -Cloud-managed policy versions. **Session-only** — every command here exits `2` under an API key, before any request, because these are root-only write routes deliberately absent from `/v1`. +Cloud-managed policy versions. **Session-only** — every command here except `fp policies test` exits `2` under an API key, before any request, because these are root-only write routes deliberately absent from `/v1`. `fp policies test` is the deliberate exception: it runs entirely locally against `node` and talks to no server, so it carries neither an auth requirement nor a key-mode refusal and works logged out and under an API key alike. | Command | Purpose | Options | | --- | --- | --- | -| `fp policies list` | List policy versions. | `--json` | +| `fp policies list` | List policy versions, newest of each policy first. | — | | `fp policies show POLICY_ID` | Show one policy, with its source. | — | -| `fp policies publish NAME PATH` | Mint a version from a local `.mjs`. | `--description`; `--no-verify` | +| `fp policies publish POLICY_ID [SOURCE]` | Mint a new version; it never edits one in place. `SOURCE` is a path, `@path`, or `-` for stdin — omit it to paste. | `--description`; `--no-verify` | | `fp policies enable POLICY_ID` | Add it back to every deployment it was removed from, minting a new generation on each. | `--yes`, `-y` | | `fp policies disable POLICY_ID` | Remove it from every deployment carrying it, minting a new generation on each. | `--yes`, `-y` | | `fp policies delete POLICY_ID` | Delete a policy version. | `--yes`, `-y` | -| `fp policies test PATH` | Run a policy locally against a synthetic context. Applies each policy's `match` filter, so one that does not cover the given event/tool is reported `skipped` rather than run. | `--event`; `--tool`; `--command`; `--file-path`; `--expect` | -| `fp policies compose PROMPT` | Draft a policy with the assistant. Needs `policies:write`. | — | +| `fp policies test [SOURCE]` | Run a policy locally against a synthetic context. Applies each policy's `match` filter, so one that does not cover the given event/tool is reported `skipped` rather than run. Needs `node` on PATH. | `--event`; `--tool`; `--command`; `--file`; `--expect` | +| `fp policies compose PROMPT` | Draft a policy with the assistant. Prints the draft and does nothing else by default. Needs `policies:write`. | `--out `; `--publish ` | ### Fleet -Which machines run which policies. **Session-only**, same reason as above. +Which machines run which policies. **Session-only** — every command here, with no exception, exits `2` under an API key. | Command | Purpose | Options | | --- | --- | --- | | `fp fleet list` | List enrolled machines and their deployment generation. | — | | `fp fleet show MACHINE_ID` | The policy set a machine currently runs. | — | -| `fp fleet deploy MACHINE_ID` | **Replaces the machine's whole policy set.** Prints the plan and asks only on an interactive terminal without `--json`. | `--add`; `--remove`; `--set`; `--create`; `--yes`, `-y` | -| `fp fleet diff MACHINE_ID` | Compare a machine against another deployment. | — | -| `fp fleet history MACHINE_ID` | Past deployments for a machine. | — | -| `fp fleet rollback MACHINE_ID` | Restore a previous deployment. | `--yes`, `-y` | -| `fp fleet rename MACHINE_ID` | Give a machine a readable name. | required `--name` | +| `fp fleet deploy MACHINE_ID` | Change what a machine enforces. Prints the full resulting set first, and asks only on an interactive terminal without `--json`. | `--add`; `--remove`; `--set`; `--create`; `--yes`, `-y` | +| `fp fleet diff [MACHINE_ID]` | Intent versus delivery — what a machine is told to run against what it last pulled. Omit the id for the whole fleet. | — | +| `fp fleet history MACHINE_ID` | Every generation for a machine, newest first. | — | +| `fp fleet rollback MACHINE_ID GENERATION` | Reinstate a past generation's policy set. The generation number comes from `fp fleet history MACHINE_ID`. | `--yes`, `-y` | +| `fp fleet rename MACHINE_ID LABEL` | Give a machine a readable label. The id itself never changes. | — | + +```bash +fp fleet history ci-runner-01 +fp fleet rollback ci-runner-01 3 +fp fleet rename ci-runner-01 "CI runner (eu-west)" +``` + +#### Policy references + +`--add` and `--set` take a policy reference, not a bare id: + +| Form | Means | +| --- | --- | +| `id` | Current version, current effect | +| `id@3` | Version 3, current effect | +| `id:observe` | Current version, recorded but not enforced | +| `id@3:observe` | Version 3, recorded but not enforced | + +The effect is `enforce` or `observe`. Resolution is: an explicit effect wins; otherwise the effect already deployed for that policy; otherwise `enforce`. + + + A bare `--add` **enforces immediately**. `fp fleet deploy ci-runner-01 --add checkout-guard` starts blocking on that machine as soon as it next pulls. `:observe` is what makes it a shadow rollout: + + ```bash + fp fleet deploy ci-runner-01 --add checkout-guard:observe # records only + fp fleet deploy ci-runner-01 --add checkout-guard # enforces now + ``` + + Observe is not off. The policy is evaluated for real, under the same 10-second timeout and the same error handling as an enforcing one, and every non-allow verdict is recorded — only the enforcement is withheld. That is what makes the measurement worth reading. + + +#### Deltas, replacement, and races + +`--add` and `--remove` read the machine's current set and apply a delta, so nothing you did not name is disturbed. A bare `--add` on a policy the machine already runs keeps its pinned version rather than silently upgrading it; pass `id@version` to move it. + +`--set` replaces everything and is the only way to drop policies you do not name. It cannot be combined with `--add` or `--remove`. Passing none of the three is a usage error, not a no-op. + +```bash +fp fleet deploy ci-runner-01 --add prod-guard@1:observe --remove old-rule +fp fleet deploy ci-runner-01 --set no-force-push --set no-secret-echo +``` + +The write itself is a full replace on every path, because the endpoint takes the whole policy set. There is no server-side lock, so the CLI records the generation it read and refuses if the result is not exactly one higher — that means somebody else deployed in between, and a replace does not merge. Re-read with `fp fleet show`, then deploy again. + +`--create` deploys to a machine id that has not checked in yet, for pre-staging. Without it, an id the server does not know is refused, because a typo would otherwise mint a machine nobody owns carrying policies nobody collects. ### Guardrails @@ -357,6 +406,31 @@ What enforcement actually did. **Session-only**, same reason as above. | `fp guardrails summary` | Coverage, blocked/evaluated totals, a deny sparkline, and the per-policy table. | `--since` (`1h`, `6h`, `24h`, `7d`); `--machine` | | `fp guardrails timeline` | Decisions bucketed over the window, summed across every policy source. | `--since` (`1h`, `6h`, `24h`, `7d`); `--machine` | +A `(no policy)` row in the summary is normal rather than a gap: most evaluations are allows nothing objected to, and the row keeps the denominator on screen. + +### Roll a policy out without breaking anyone + +The three groups above are one sequence. Decide locally, publish, shadow, measure, then enforce. + +```bash +fp policies test ./checkout.policy.mjs --tool Bash --command "git push --force" --expect deny +fp policies publish checkout-guard ./checkout.policy.mjs --description "Block force-push" +fp fleet deploy ci-runner-01 --add checkout-guard:observe +fp guardrails summary --since 24h --machine ci-runner-01 +fp fleet deploy ci-runner-01 --add checkout-guard:enforce +``` + +| Step | Why it is there | +| --- | --- | +| `fp policies test` | No server and no auth. Runs the real file against a context you describe and prints allow, deny, or instruct per registered policy. `--expect` turns that into a CI assertion. | +| `fp policies publish` | Mints a new immutable version; it never edits one in place. | +| `fp fleet deploy --add :observe` | The shadow rollout. **`:observe` is not optional here** — a bare `--add` enforces the moment the machine pulls. | +| `fp guardrails summary` | Separates the unsafe matches from the legitimate work the policy would also have blocked. | +| `fp fleet deploy --add :enforce` | Promotion, once the recorded verdicts say what you expected. | +| `fp fleet rollback ` | The way back if enforcement goes wrong. | + +The single-machine equivalent, with no Cloud involved, is publishing the pack with `failproofai publish --effect observe` and reading the recorded verdicts on the [local dashboard](/reference/local-dashboard). + ## Global flags | Flag | Description | @@ -389,7 +463,9 @@ What enforcement actually did. **Session-only**, same reason as above. | `FP_ANALYTICS_DISABLED` or `DO_NOT_TRACK` | Disable anonymous CLI analytics. | | `NO_COLOR` | Disable colored output. | -Explicit flags override environment variables, which override saved configuration. In API-key mode, select the tenant explicitly with `--org` or `FP_ORG`. +Explicit flags override environment variables, which override saved configuration. + +In API-key mode the organization saved by a human `fp login` is **ignored**, not merely overridden — only an explicit `--org` or `FP_ORG` is sent. Nothing is inherited from a saved login, so pass `--org` whenever the key can act for more than one organization. Omitting it does not fail loudly: an instance-scoped key with no `--org` resolves server-side to the **default** organization and answers with that organization's data, with no error anywhere. Run `fp whoami` first to confirm which tenant a key is actually talking to. The `AGENTEYE_*` spellings of these are **not read by `fp`** and never were — the CLI declares `FP_*` (`fp_cli/app.py`), and an unknown variable is not an error. Setting `AGENTEYE_DASHBOARD_URL` does not retarget the CLI; it is ignored and the command silently runs against the saved dashboard instead. diff --git a/docs/reference/custom-agents.mdx b/docs/reference/custom-agents.mdx index 938a3b5b9..48a4ffe0a 100644 --- a/docs/reference/custom-agents.mdx +++ b/docs/reference/custom-agents.mdx @@ -15,7 +15,7 @@ What every setting, method and field does. If you are instrumenting for the firs -Python 3.10 or newer. No runtime dependencies. +Python 3.10 or newer, and **zero runtime dependencies** — a policy, not an accident. The SDK installs into other people's agent processes, so anything it declared they would inherit. A test scans the core modules and launches a fresh interpreter to prove no framework lands in `sys.modules`. ## Install @@ -29,8 +29,8 @@ The package is installed as `failproofai-sdk` and imported in Python as `failpro - 1. Go to **Admin → Keys** and create a key with `events:add`. - 2. [Connect the Failproof daemon to Cloud](/start/setup#connect-a-machine-to-cloud) on the agent machine. + 1. Go to **Administration → Keys** and create a key with `events:add`. + 2. [Connect the Failproof daemon to Cloud](/start/setup#connect-failproof-ai-cloud) on the agent machine. 3. Run one instrumented session, then find its exact ID under **Observe → Events**. 4. Go to **Observe → Sessions**, select the same environment, and open the reconstructed trace. @@ -38,11 +38,12 @@ The package is installed as `failproofai-sdk` and imported in Python as `failpro ```bash - failproofai config \ - --connect https://app.befailproof.ai \ - --token + export FAILPROOFAI_CLOUD_TOKEN= + failproofai config failproofai config --status ``` + + Prefer the environment variable over `--token`: a command-line argument is readable from `ps` by every user on the box, and it lands in shell history and CI logs. @@ -62,7 +63,9 @@ failproofai_sdk.configure( | --- | --- | | `environment` | The label on every event — `production`, `staging`, `prod-eu`. Defaults to `dev`. | | `flush_interval` | How often the background thread writes to disk, in seconds. Defaults to `0.5`. | -| `base_dir` | Where to write. Defaults to the daemon's spool, which is what you want unless you know otherwise. | +| `base_dir` | Where to write. Defaults to `$FAILPROOFAI_HOME/custom-agents`, otherwise `~/.failproofai/custom-agents` — the spool the daemon watches, which is what you want unless you know otherwise. | + +`configure()` is **keyword-only** — `configure(None, 0.5, "prod")` is a `TypeError`. It also validates every argument before applying any of them, and raises `ValueError` if `flush_interval` is not a finite number greater than zero, so a rejected call leaves the SDK exactly as it was rather than half-moved. Set by environment variable instead: @@ -81,6 +84,39 @@ Set by environment variable instead: Events are queued in memory and written in the background every `flush_interval` seconds, with a final flush at interpreter exit. A process killed outright loses whatever had not been written yet. +## Scopes + +Three context managers, each usable with `with` and `async with`: + +| Scope | Binds | Emits | +| --- | --- | --- | +| `session()` | A session id. Yields it | Nothing — identity only | +| `agent(name)` | An agent id, and a `parent_id` inferred from the enclosing agent | `agent_start` on entry, `agent_end` on exit | +| `tool_call(name)` | Nothing new — inherits the enclosing identity | `tool_use` on entry, `tool_result` on exit | + +```python +with failproofai_sdk.session(): + with failproofai_sdk.agent("planner"): + with failproofai_sdk.tool_call("web_search", input={"q": q}) as t: + t.output = search(q) +``` + +`tool_call()` defaults `tool_call_id` to a fresh `uuid4().hex` and resolves identity once, at entry, so a tool that pushes its own scope inside cannot make the closing `tool_result` land on a different agent. On failure it emits `tool_result(error="TypeName: msg")` and **no `error` event** — a tool failure the loop catches is not a run-level error, and one that propagates is reported exactly once, by the enclosing `agent()`. Cancellation closes the leaf with no error string at all. + +## Adapters + +`instrument()` wires a supported framework to the scopes above. With no argument it auto-detects the frameworks already imported in this process; pass a name to install exactly one. + +```python +failproofai_sdk.instrument() # everything already imported +failproofai_sdk.instrument("crewai") # exactly one +failproofai_sdk.uninstrument("crewai") # restore the original attributes +``` + +The four names are `"langchain"` (which covers LangGraph, since LangGraph runs on langchain-core's callback manager), `"crewai"`, `"llama_index"` and `"pydantic_ai"`. The framework import happens inside the call, which is what keeps `import failproofai_sdk` zero-dependency. + +Each framework has its own page: [LangChain](/start/integrations/langchain), [CrewAI](/start/integrations/crewai), [LlamaIndex](/start/integrations/llamaindex), [Pydantic AI](/start/integrations/pydantic-ai). + ## Identity Every event belongs to a session and an agent. **The scopes fill both in**, so you rarely pass them: @@ -91,7 +127,18 @@ with failproofai_sdk.session(): failproofai_sdk.event.tool_use(tool_name="search", tool_call_id="c1") ``` -Passing `session_id` or `agent_id` explicitly still works and wins. With neither bound nor passed, the call raises `TypeError` rather than emitting an event Cloud would quietly discard. +Passing `session_id` or `agent_id` explicitly still works and wins. + +The two are **not** symmetrical when nothing is bound: + +| Omitted | With nothing bound | +| --- | --- | +| `session_id` | Raises `TypeError`, rather than emitting an event Cloud would quietly discard | +| `agent_id` | Falls back to `main`, so events emitted inside `session()` with no `agent()` around them all land under a single agent named `main` | + +Inventing a session id would scatter one run across as many sessions as it has emit sites, which is why only that one raises. + +`session_id` and `agent_id` are also the only two names checked for emptiness: a non-string raises `TypeError`, and an empty or whitespace-only id raises `ValueError`. The server *accepts* a blank id, so without that check every event sent with one would be grouped under a single blank id and look present while being silently merged. Identity rides on context variables. It follows `asyncio` tasks automatically, but **not** new threads — wrap a worker in `failproofai_sdk.propagate()` or its events land unattached. @@ -99,16 +146,16 @@ Passing `session_id` or `agent_id` explicitly still works and wins. With neither ## Event catalog -Fifteen methods. Most come in **pairs** — you call the opener, then the closer, and the SDK times the gap. +Fifteen methods. Most come in **pairs** — an opener and a closer. -| | Opens | Closes | -| --- | --- | --- | -| **Agents** | `agent_start` | `agent_end` | -| | `agent_pause` | `agent_resume` | -| **Models** | `model_request` | `model_response` | -| **Tools** | `tool_use` | `tool_result` | -| **Hooks** | `hook_triggered` | `hook_completed` | -| **Humans** | `human_wait` | `human_input` | +| | Opens | Closes | Timed by the SDK | +| --- | --- | --- | --- | +| **Agents** | `agent_start` | `agent_end` | No | +| | `agent_pause` | `agent_resume` | Yes | +| **Models** | `model_request` | `model_response` | No — correlated in Cloud | +| **Tools** | `tool_use` | `tool_result` | Yes | +| **Hooks** | `hook_triggered` | `hook_completed` | Yes | +| **Humans** | `human_wait` | `human_input` | Yes | Three stand alone: `error`, `human_pause`, `human_interrupt`. @@ -144,17 +191,20 @@ Every method also takes `session_id` and `agent_id`, which the scopes fill in fo **One rule: give the closing event the same id as its opener.** That is what pairs them, and what lets the SDK time the gap. -| Pair | Matched on | -| --- | --- | -| `tool_use` → `tool_result` | `tool_call_id` | -| `hook_triggered` → `hook_completed` | `hook_id` | -| `agent_pause` → `agent_resume` | `pause_id` | -| `human_wait` → `human_input` | `input_id` | -| `model_request` → `model_response` | `request_id` | +| Pair | Matched on | Tracking key | +| --- | --- | --- | +| `tool_use` → `tool_result` | `tool_call_id` | `tool:{session_id}:{tool_call_id}` | +| `hook_triggered` → `hook_completed` | `hook_id` | `hook:{session_id}:{hook_id}` | +| `agent_pause` → `agent_resume` | `pause_id` | `pause:{session_id}:{pause_id}` | +| `human_wait` → `human_input` | `input_id` | `human:{session_id}:{input_id}` | -**Do not pass `duration_ms` yourself.** The SDK measures it, and passing it raises `ValueError`. +The key is namespaced by session, which is what makes both edge cases below true. -The one exception is `model_response`, where only you know the real provider latency. Pass a whole number of milliseconds — a float raises, because the column is a 32-bit integer and would otherwise land empty. +**Do not pass `duration_ms` yourself** on those four closers. The SDK measures it, and passing it raises `ValueError`. + + + `model_request` and `model_response` are a pair Cloud correlates on `request_id`. The SDK does not track them and measures nothing locally, which is why `model_response` is the one event that takes `duration_ms` from you. Pass a whole number of milliseconds — a float raises, because the column is a 32-bit integer and would otherwise land empty. + @@ -179,6 +229,17 @@ failproofai_sdk.event.tool_use( Prefer JSON types if you want to query them later. Anything else — a UUID, a datetime, a `Decimal`, a set, bytes, a model object — is stored as a string. + + **Eleven names are the exception, and they raise rather than stringify.** Ingest lifts each into a typed column and stores `NULL` for anything else, at `200 OK`, invisibly — so the SDK refuses them at the call site instead. + + | Names | Must be | + | --- | --- | + | `duration_ms`, `input_tokens`, `output_tokens` | An `int` in the unsigned 32-bit range | + | `tool_name`, `tool_call_id`, `hook_name`, `hook_id`, `input_id`, `pause_id`, `error_type`, `model` | A string — `None` raises `ValueError`, any non-string raises `TypeError` | + + This applies whether the value arrives as a named parameter or as one of your own extras. `model_response` validates its `input_tokens` and `output_tokens` on the way in for the same reason — they are the likeliest to be filled straight from a provider's usage object. + + **Prefix your field names.** Extras are applied last, so a field called `model`, `tool_name` or `outcome` silently overwrites the real one. The framework adapters use `fw_`; do the same and nothing can collide. @@ -187,6 +248,16 @@ Prefer JSON types if you want to query them later. Anything else — a UUID, a d These five names are reserved and rejected outright: `timestamp`, `session_id`, `agent_id`, `type`, `environment`. +## Other exports + +| Export | What it is | +| --- | --- | +| `current()` | The identity bound right now, as an `Identity`. `current().session_id is None` means nothing is bound | +| `Identity` | `session_id`, `agent_id`, `parent_id`, `depth`. Never `None` itself — check the fields | +| `propagate(fn)` | Wraps `fn` so it runs with the identity bound at the moment of wrapping. Needed for `Thread`, `pool.submit`, `pool.map`, `run_in_executor` | +| `_writer.flush_now()` | Drain and write buffered entries immediately, for a test or a forced flush before exit | +| `__version__` | The installed SDK version | + ## Deliver and verify diff --git a/docs/reference/evaluator-sdk.mdx b/docs/reference/evaluator-sdk.mdx index 1c3da5d4d..7f3627f0d 100644 --- a/docs/reference/evaluator-sdk.mdx +++ b/docs/reference/evaluator-sdk.mdx @@ -6,14 +6,38 @@ icon: "gauge" An evaluator receives a completed agent session and returns the quality signals you care about: numeric scores, an explanation for each score, and an optional summary. Failproof AI stores these results beside the trace and charts them across agents and environments. +The package is `agenteye-evaluator`, imported as `agenteye_evaluator`. It requires Python 3.10 or newer and depends on `fastapi`, `pydantic>=2`, and `structlog`. + + + **`pip install agenteye-evaluator` from public PyPI is not the install path.** The package is published only as a private release artifact, and the name is unclaimed on public PyPI — an unqualified install could pull a stranger's package into the service that reads your production transcripts. Use the ladder below. + + ## Set up an evaluator - - Install the SDK and the server used to run it. + + Work down this ladder and stop at the first rung that applies. + + Inside the monorepo, where an `evaluator-sdk/` directory exists: + + ```bash + pip install ./evaluator-sdk + ``` + + Otherwise, from the private release. Wheels are attached to GitHub Releases on `agenteye-enterprise/releases`, tagged `evaluator-sdk/v`, and you need `gh auth login` plus access to that repository: + + ```bash + gh release download evaluator-sdk/v \ + --repo agenteye-enterprise/releases --pattern '*.whl' + pip install ./agenteye_evaluator-*.whl + ``` + + If neither works, ask your Failproof AI contact for the wheel rather than improvising an install. + + `uvicorn` is deliberately not a dependency, so install the server separately: ```bash - pip install failproofai-sdk uvicorn + pip install 'uvicorn[standard]' ``` @@ -22,7 +46,7 @@ An evaluator receives a completed agent session and returns the quality signals ```python import os - from failproofai.evaluator import Evaluator, EvalResponse + from agenteye_evaluator import Evaluator, EvalResponse app = Evaluator(token=os.environ.get("EVALUATOR_TOKEN")) @@ -41,6 +65,12 @@ An evaluator receives a completed agent session and returns the quality signals reasoning={"tool_reliability": f"{tool_errors} tool errors"}, ) ``` + + The full constructor is `Evaluator(token: str | None = None, *, title: str = "AgentEye Evaluator")`. The token is compared with `hmac.compare_digest`; `title` is the FastAPI app title and is cosmetic. + + + `token=None` disables authentication entirely. That is fine on a laptop and a hole in production, where the endpoint receives whole session transcripts. + @@ -56,6 +86,8 @@ An evaluator receives a completed agent session and returns the quality signals ```bash curl http://127.0.0.1:8080/health ``` + + Each decorator returns the function unchanged, so `evaluate(req)` stays directly callable. That is what makes unit tests cheap — build an `EvalRequest` and call the handler, no HTTP involved. @@ -91,9 +123,30 @@ An evaluator receives a completed agent session and returns the quality signals For a self-hosted Cloud instance, automatic evaluation is disabled until `EVALUATOR_ENDPOINT` is set on the server process. Restart the server after changing evaluator environment variables. -The service exposes `GET /health`, `GET /config`, `POST /evaluate`, and optionally `GET /evaluate/{job_id}`. Return `JobPending` for asynchronous work and register `@app.job_lookup` so Failproof AI can poll it. +## Decorators and routes + +| Decorator | Route | Required | +| --- | --- | --- | +| `@app.evaluator` | `POST /evaluate` | Yes. | +| `@app.job_lookup` | `GET /evaluate/{job_id}` | Only if you ever return `JobPending`. Without it, polls get a 404. | +| `@app.config` | `GET /config` | No, but required for any session that never emits `agent_end` — see below. | + +Each decorator accepts a sync or an async function, returns it unchanged, and raises `ValueError` if you register it twice. + +| Route | Auth | +| --- | --- | +| `GET /health` | Open even when a token is set. | +| `POST /evaluate` | Bearer. | +| `GET /evaluate/{job_id}` | Bearer. | +| `GET /config` | Bearer. | + +The bearer scheme match is case-insensitive. `GET /config` with no `@app.config` registered still returns `{"default_poll_interval_secs": 10}`, so the SDK always advertises a cadence. + +The SDK caps evaluation request bodies at 25 MiB, checked against `Content-Length` before the body is read. Over the cap is a 413, which is a 4xx and therefore terminal. Unknown request fields are ignored so services remain compatible as the event contract grows. -When a token is configured, all routes except health require the same bearer token that Failproof AI sends as `EVALUATOR_TOKEN`. + + Registering `@app.config` with an `inactivity_timeout_secs` is what enables the fallback scanner. Without it, a session that never emitted `agent_end` — anything abandoned, crashed, or still idling — is never enqueued for evaluation at all. Values of zero or less are dropped. + ## SDK types @@ -105,22 +158,55 @@ When a token is configured, all routes except health require the same bearer tok | `JobPending` | `job_id`, `next_poll_secs` | | `EvaluatorConfig` | `inactivity_timeout_secs`, `default_poll_interval_secs` | -## Decorators and routes +## Request and response fields -| Decorator | Route | Required | +| Field | Type | Notes | | --- | --- | --- | -| `@app.evaluator` | `POST /evaluate` | Yes | -| `@app.job_lookup` | `GET /evaluate/{job_id}` | When returning `JobPending` | -| `@app.config` | `GET /config` | No | +| `EvalRequest.schema_version` | `str` | Currently `"1"`. | +| `session_id`, `agent_id`, `environment` | `str` | Session identity and environment. | +| `started_at` | `datetime` | Timestamp of the first event. | +| `ended_at` | `datetime \| None` | The `agent_end` event's timestamp, not "when the session stopped". Sessions enqueued by the inactivity scanner never had an `agent_end` and arrive `None`. | +| `events` | `list[AgentEvent]` | Full ordered event stream. | +| `AgentEvent.id` | `int` | Backend event row identifier. | +| `AgentEvent.ts` | `datetime` | Event timestamp. | +| `AgentEvent.event_type` | `str` | Event family such as `tool_use`. | +| `AgentEvent.payload` | `dict[str, Any]` | The whole event JSON flattened, so event-specific fields sit at the top level and `payload["type"]` duplicates `event_type`. | +| `EvalResponse.scores` | `dict[str, float] \| None` | Numeric dimensions charted in evaluations. | +| `EvalResponse.reasoning` | `dict[str, str] \| None` | Per-score explanations; keys should mirror `scores`. | +| `EvalResponse.summary` | `str \| None` | Overall evaluation narrative. Truncated at 8192 bytes server-side; `last_error` at 2048. | + +Serialization uses `exclude_none`, so unset fields are omitted rather than sent as `null`. + + + Deriving a duration from `ended_at` crashes on real data. Every session the inactivity scanner enqueues arrives with `ended_at` set to `None`. Check it before subtracting. + + +## Return shapes -The SDK caps evaluation request bodies at 25 MiB. Unknown request fields are ignored so services remain compatible as the event contract grows. +Your handler may return exactly one of three things. Anything else is a `TypeError`, which surfaces as a 500. + +| Return | Wire `status` | Terminal | +| --- | --- | --- | +| `EvalResponse(...)` | `done` | Yes — scores stored. | +| `JobPending(job_id=...)` | `pending` | No — the server polls. | +| A raw `dict` with `status` in `done`, `pending`, or `error` | As given | `error` is terminal. | + +The `error` status has no typed model. To fail terminally you must return a raw dict, and `error` must be a non-empty `str`: + +```python +return {"status": "error", "error": "model service unavailable"} +``` + + + **Raising is not reporting.** An exception becomes a generic 500 whose body is `"evaluator raised an internal error"` — your exception text never reaches the server, and the server treats every 5xx as transient and retries it. Return the `error` dict when you want the failure recorded. + ## Return asynchronous work Use `JobPending` when evaluation cannot finish inside one request. The job ID is opaque to Failproof AI and must remain resolvable by your service until the result is collected or the server timeout expires. ```python -from failproofai.evaluator import EvalRequest, EvalResponse, Evaluator, JobPending +from agenteye_evaluator import EvalRequest, EvalResponse, Evaluator, JobPending app = Evaluator(token="shared-secret") @@ -141,24 +227,7 @@ def lookup(job_id: str): ) ``` -Polling cadence is selected in this order: `JobPending.next_poll_secs`, `EvaluatorConfig.default_poll_interval_secs`, then the server's `EVALUATOR_POLLING_INTERVAL_SECS`. Values are clamped between 1 second and 1 hour. The server's default wall-clock polling cap is one hour. - -## Request and response fields - -| Field | Type | Notes | -| --- | --- | --- | -| `EvalRequest.schema_version` | `str` | Currently `"1"`. | -| `session_id`, `agent_id`, `environment` | `str` | Session identity and environment. | -| `started_at` | `datetime` | Timestamp of the first event. | -| `ended_at` | `datetime \| None` | Present when the session emitted an end event. | -| `events` | `list[AgentEvent]` | Full ordered event stream. | -| `AgentEvent.id` | `int` | Backend event row identifier. | -| `AgentEvent.ts` | `datetime` | Event timestamp. | -| `AgentEvent.event_type` | `str` | Event family such as `tool_use`. | -| `AgentEvent.payload` | `dict[str, Any]` | Complete event payload. | -| `EvalResponse.scores` | `dict[str, float] \| None` | Numeric dimensions charted in evaluations. | -| `EvalResponse.reasoning` | `dict[str, str] \| None` | Per-score explanations; keys should mirror `scores`. | -| `EvalResponse.summary` | `str \| None` | Overall evaluation narrative. | +Polling cadence is selected in this order: `JobPending.next_poll_secs`, `EvaluatorConfig.default_poll_interval_secs`, then the server's `EVALUATOR_POLLING_INTERVAL_SECS`. Values are clamped between 1 second and 1 hour. The server's default wall-clock polling cap is one hour, after which the result is recorded as `timeout`. ## Server operator settings @@ -171,20 +240,25 @@ Automatic evaluation is deployment-wide and remains disabled when `EVALUATOR_END | `EVALUATOR_WORKERS` | `2` | Concurrent dispatcher workers. | | `EVALUATOR_CLAIM_BATCH` | `4` | Sessions claimed per dispatcher pass. | | `EVALUATOR_POLLING_INTERVAL_SECS` | `10` | Fallback async polling cadence. | -| `EVALUATOR_REQUEST_TIMEOUT_MS` | `30000` | Per-request evaluator timeout. | +| `EVALUATOR_REQUEST_TIMEOUT_MS` | `30000` | Per-request evaluator timeout, applied to the POST and to each poll. | | `EVALUATOR_MAX_ATTEMPTS` | `5` | Delivery attempts before terminal failure. | | `EVALUATOR_CONFIG_REFRESH_SECS` | `300` | Refresh cadence for `/config`. | | `EVALUATOR_MAX_POLL_DURATION_SECS` | `3600` | Maximum wall-clock async polling time. | +Two consequences of that table are worth stating outright: + +- **Workers times claim batch is your concurrency.** At the defaults that is 8 concurrent calls against your endpoint, deployment-wide. Size the service for that number, not for one. +- **4xx is terminal and 5xx, 429, or a transport failure is retried** with backoff up to `EVALUATOR_MAX_ATTEMPTS`. A token mismatch is a 401, so it fails immediately rather than retrying — which is what to check first when nothing arrives. + The server can also constrain which organizations use the deployment-global evaluator. Treat endpoint, token, retry, and organization-gate changes as operator configuration and restart or roll the server after changing them. ## Security and operations - Put the evaluator behind HTTPS when traffic crosses a trusted network boundary. -- Configure a non-empty bearer token and keep it identical on both services. -- Do not log the token or full sensitive prompts from request payloads. +- Configure a non-empty bearer token and keep it identical on both services. `token=None` accepts every caller. +- Do not log the token or full sensitive prompts from request payloads. The SDK does not: validation failures return 422 without echoing the payload, 500s never echo exception text, and the token appears in no log field. - Make synchronous handlers idempotent; retries may repeat a request. - Persist asynchronous job state outside process memory in production. - Return stable score keys. Renaming a key creates a new chart series rather than changing the old one. -The SDK emits structured lifecycle logs such as `eval received`, `eval responded`, `job lookup`, `config returned`, `auth rejected`, and handler exceptions. It does not configure logging handlers; use the host application's logging configuration. +The SDK emits structured lifecycle logs such as `eval received`, `eval responded`, `job lookup`, `config returned`, `auth rejected`, and handler exceptions. `/config` responses tag `source="user"` against `source="default"`, so you can tell whether your `@app.config` was picked up. It does not configure logging handlers; use the host application's logging configuration. diff --git a/docs/reference/events-and-configuration.mdx b/docs/reference/events-and-configuration.mdx index a7804d03f..2bf5bc97c 100644 --- a/docs/reference/events-and-configuration.mdx +++ b/docs/reference/events-and-configuration.mdx @@ -4,6 +4,17 @@ description: "Reference the event model, environments, local storage, and config icon: "list-tree" --- +## Two event vocabularies + +Failproof AI carries two different kinds of event, and they never share a name. + +| Vocabulary | Produced by | Names look like | Reference | +| --- | --- | --- | --- | +| Telemetry events | `failproofai-sdk`, from inside your agent | `tool_use`, `tool_result`, `hook_triggered`, `model_response` | [Python SDK](/reference/custom-agents) | +| Hook events | The agent harness, normalized by failproofai | `PreToolUse`, `PostToolUse`, `Stop`, `UserPromptSubmit` | [Custom policies](/reference/policy-sdk#choose-the-event) | + +The hook side normalizes every harness's own event names into 29 canonical types. The telemetry side has 15 methods and its own field contract. `hook_triggered` and `PreToolUse` are not two spellings of one thing. + ## Event families - Agent start, end, pause, and resume @@ -53,32 +64,103 @@ Every event carries a timestamp, session ID, agent ID, event type, and environme ## Reserved SDK fields -Do not use `timestamp`, `session_id`, `agent_id`, `type`, or `environment` as custom Python SDK fields. Paired event durations such as tool result duration are calculated by the SDK and cannot be supplied manually. +Five names are refused outright as custom fields, because the SDK sets them on every event: + +`timestamp`, `session_id`, `agent_id`, `type`, `environment`. + +Eleven more are accepted but type-checked, because the ingest endpoint lifts them out of the payload into typed columns and silently stores `NULL` for anything of the wrong type. The SDK raises instead, at the call site where you can still see what produced the value. + +| Names | Required type | +| --- | --- | +| `duration_ms`, `input_tokens`, `output_tokens` | `int`, within the unsigned 32-bit range. A float or a bool raises. | +| `tool_name`, `tool_call_id`, `hook_name`, `hook_id`, `input_id`, `pause_id`, `error_type`, `model` | `str`. `None` raises, because the row would arrive at 200 OK and be invisible to every filter on that field. | + +### Durations + +The SDK measures the gap between four paired events and refuses a `duration_ms` you pass yourself: `tool_use` → `tool_result`, `hook_triggered` → `hook_completed`, `agent_pause` → `agent_resume`, and `human_wait` → `human_input`. + +`model_response` is the exception. The SDK never times model calls, so you supply `duration_ms` yourself, as a whole number of milliseconds. See [correlation rules](/reference/custom-agents) for how a pair is matched. ## Local boundaries - Failproof AI state lives under `~/.failproofai` unless explicitly configured otherwise. - `failproofai-sdk` always spools under `~/.failproofai/custom-agents`. `FAILPROOFAI_HOME` relocates that home; the `custom-agents` segment is always appended, so the spool cannot be placed outside it. Only the SDK's own `configure(base_dir=...)` writes elsewhere. `AGENTEYE_HOME` is read by the older `agenteye-collector` to choose what to watch, and no longer affects where the SDK writes. -- Environment labels can be set through SDK configuration or `AGENTEYE_ENVIRONMENT`. -- Daemon credentials are stored separately from non-secret settings. +- Credentials live in `~/.failproofai/credentials.json`, written `0600`. They are deliberately not in `config.json`, which is written with a plain write and inherits the umask, so it lands readable by every local user on the box. + +Use stable, low-cardinality environment names. A comma is rejected on both writers: the daemon refuses one in `collector.environment`, and the SDK raises on `configure(environment=...)`. The reason is the same on both sides—the ingest endpoint splits this field on commas and skips the whole line, answering `200 OK` while storing nothing. + +## Machine configuration + +The daemon's non-secret settings are `~/.failproofai/config.json`. `hooks_verbosity`, `redact` and `environment` have no CLI flag — edit the file directly. `sessions`, `hooks` and `machine_id` are written by `failproofai config` when it connects this machine to Cloud. + +| Key | Values | Default | What it controls | +| --- | --- | --- | --- | +| `collector.sessions` | `true` / `false` | `false`, and `true` once you connect to Cloud | Ship agent session transcripts. A transcript carries prompts, file contents, and whatever was pasted into a terminal. | +| `collector.hooks` | `true` / `false` | `true` | Ship hook activity. Carries decisions and tool names, never file contents. | +| `collector.hooks_verbosity` | `all` / `decisions` / `off` | `decisions` | `decisions` keeps every deny and instruct exact and aggregates the roughly 99% that allow. | +| `collector.redact` | `minimal` / `off` | `minimal` | Redaction applied before anything leaves the machine. | +| `collector.environment` | any string without a comma | `local` | The label stamped on every event this machine sends. | +| `collector.machine_id` | any string | an id already on disk, else a fresh random UUID | Which machine the dashboard groups this data under. Written by `--machine-id`. It is never derived from the hostname. | + + + `failproofai backfill --help` names `~/.failproofai/config.toml`. That is a stale path from an earlier home layout; no current build writes it. The file is `config.json`. + + +Connecting to Cloud sends both policy decisions and full session transcripts. + + + `--no-transcripts` does not turn transcripts off on the setup path. `failproofai config --token --no-transcripts` parses the flag and then never reads it, and the wizard connects with sessions on regardless, so transcripts keep shipping while you believe you opted out. -Use stable, low-cardinality environment names. Commas are not supported in daemon environment labels. + To send decisions only, set `collector.sessions` to `false` in `~/.failproofai/config.json` after connecting: + + ```json + { + "collector": { + "sessions": false + } + } + ``` + + +A backfill follows the same collector settings, so it never sends something your configuration says you do not want. Re-sending is safe: redaction is deterministic, so a re-sent event hashes identically to its first send and collapses into the row already there. ## Change an environment label +The SDK and the daemon each stamp their own label, and they are set in different places. + - - Environment labels are assigned by the emitting SDK or Failproof daemon. After changing one, open **Observe → Sessions** and use the environment filter to confirm new sessions carry the new value. Existing sessions retain their original environment. + + Set it in code, or in the environment of the process that emits the events. The default is `dev`. - - - For `failproofai-sdk`, configure the environment in code or with its legacy environment variable. Rerun daemon configuration when changing machine-level settings. + ```python + import failproofai_sdk + + failproofai_sdk.configure(environment="production-us-east") + ``` ```bash export AGENTEYE_ENVIRONMENT=production-us-east - failproofai config + ``` + + `AGENTEYE_ENVIRONMENT` is read only by `failproofai-sdk`. Neither the CLI nor the daemon reads it, so exporting it changes nothing about the machine. + + + Edit `collector.environment` in `~/.failproofai/config.json`. The default is `local` and no CLI flag sets it. + + ```json + { + "collector": { + "environment": "production-us-east" + } + } + ``` + + ```bash failproofai config --status fp sessions --since 1h --env production-us-east ``` + + Environment labels are assigned by the emitting SDK or Failproof daemon, so there is nothing to change here. After changing one at its source, open **Observe → Sessions** and use the environment filter to confirm new sessions carry the new value. Existing sessions retain their original environment. + diff --git a/docs/reference/failproof-cli.mdx b/docs/reference/failproof-cli.mdx index 49242ed3d..e7da246b1 100644 --- a/docs/reference/failproof-cli.mdx +++ b/docs/reference/failproof-cli.mdx @@ -1,149 +1,135 @@ --- title: "Failproof AI CLI" -description: "Install hooks, manage local policies, connect Cloud, and operate the local daemon." +description: "Set up a machine, manage policies, inspect local activity, and connect Cloud." icon: "terminal" --- -Install the local CLI with `npm install -g failproofai`. Run it with no arguments to open the local policy dashboard. +Install with `npm install -g failproofai`. Run `failproofai` with no arguments to open the local dashboard. -The package requires Node.js 20.9 or newer. Bun 1.3 or newer is supported for development and source installs. `failproofai configure` and `failproofai setup` are aliases for `failproofai config`; `failproofai p` is an alias for `failproofai policies`. +## Core commands + +| Command | What it does | +| --- | --- | +| `failproofai config` | Set up agents, the background service, and optional Cloud connection | +| `failproofai config --token ` | Set up and connect without questions | +| `failproofai config --status` | Show connection, service version, and pause state | +| `failproofai policies` | List policies and whether they are on | +| `failproofai policies add ` | Turn on one policy | +| `failproofai policies add /` | Install a policy pack | +| `failproofai policies remove ` | Turn off a policy or remove a pack | +| `failproofai policies show /` | Inspect a pack before installing it | +| `failproofai publish` | Publish your policies as a pack | +| `failproofai audit` | Scan local agent history | +| `failproofai harness` | Manage extra session locations | +| `failproofai flush` | Send queued events now | +| `failproofai backfill` | Re-read older agent history | +| `failproofai update` | Finish an npm upgrade and update the service | +| `failproofai uninstall` | Remove hooks and the service | + +`policy`, `pack`, and `p` are accepted aliases for `policies`, but the documentation uses `policies`. ## Set up a machine ```bash npm install -g failproofai -failproofai config \ - --connect https://app.befailproof.ai \ - --token \ - --machine-label checkout-prod-01 -failproofai policies --install +failproofai config +failproofai policies add FailproofAI/policies failproofai config --status ``` -Run `failproofai` without arguments to open the local policy dashboard. +Setup chooses no policy pack. Before you add one, only `block-failproofai-commands` runs. -| Command | Outcome | -| --- | --- | -| `failproofai config` | Run interactive machine setup | -| `failproofai config --connect --token ` | Connect Cloud ingestion and policy delivery | -| `failproofai config --status` | Show connection, daemon, delivery, and pause state | -| `failproofai policies` | List builtin, custom, convention, pack, and Cloud-managed policies | -| `failproofai policies --install` | Install hooks and enable policies | -| `failproofai policy add ` | Enable one policy — a builtin, or `:` from an installed pack | -| `failproofai policy remove ` | Disable one policy, same naming | -| `failproofai policies --uninstall` | Disable policies or remove harness hooks | -| `failproofai pack list` | List installed policy packs and every policy each one carries | -| `failproofai pack add ` | Install a policy pack from a GitHub release; no tag takes the newest and pins it | -| `failproofai pack add --bundled` | Install the builtin policies as a pack, from this package, with no network | -| `failproofai pack build ` | Build the three release assets for a pack of your own | -| `failproofai pack remove ` | Deactivate an installed pack | -| `failproofai audit` | Scan local agent history and open the local audit view | -| `failproofai audit --schedule [days] --email
` | Schedule recurring local scans and email their findings | -| `failproofai audit --status` | Show the report address, interval, and next scheduled scan | -| `failproofai audit --no-schedule` | Stop recurring scans without deleting audit history | -| `failproofai harness list` | List extra capture paths | -| `failproofai flush --wait` | Deliver the current event spool | -| `failproofai backfill --since 30d` | Re-read previously passed history | -| `failproofai config --pause [duration]` | Pause one local session for 30 minutes by default, up to 8 hours | -| `failproofai config --resume` | Resume one paused local session; add `--all` to clear all pauses | -| `failproofai update` | Finish package migrations and update the daemon | -| `failproofai migrate --dry-run` | Preview or run pending home-layout migrations | -| `failproofai uninstall` | Remove hooks and the daemon before removing the package | -| `failproofai --version` | Print the installed package version | -| `failproofai --help` | Show commands and global usage | - -## Configuration flags +For unattended Cloud setup: -| Flag | Use | -| --- | --- | -| `--connect --token ` | Connect non-interactively | -| `--machine-id ` | Set the stable machine ID | -| `--machine-label ` | Set or change the dashboard label | -| `--no-transcripts` | Send decisions without transcript content | -| `--disconnect` | Stop Cloud policy pulls and event delivery | -| `--status` | Show current machine state | -| `--pause [duration]` | Pause the newest session in the current directory; accepts seconds, minutes, or hours and defaults to 30 minutes | -| `--resume` | End a matching pause early | -| `--session ` | Target an explicit session for pause or resume | -| `--all` | With `--resume`, end every active pause | +```bash +export FAILPROOFAI_CLOUD_TOKEN="" +failproofai config +``` -Local pauses suspend builtin, custom, convention, and pack policies for one session. They always expire and do not disable Cloud-managed policies. `block-failproofai-commands` — which is always on and cannot itself be disabled or paused — prevents an instrumented agent from using this escape hatch itself. +To send decisions without transcripts, connect first and set `collector.sessions` to `false` in `~/.failproofai/config.json`. The current `--no-transcripts` setup flag is parsed but does not apply that setting. -## Policy flags +Use `--machine-label ` after the machine is connected to rename it. `--connect ` is the narrower enrol-only command for a machine already set up. -| Flag | Use | -| --- | --- | -| `--install`, `-i` | Enable policies and install harness hooks | -| `--uninstall`, `-u` | Disable policies or remove hooks | -| `--cli ` | Target one or more supported harnesses | -| `--scope user\|project\|local\|all` | Choose the configuration scope; `all` is for uninstall | -| `--beta` | Include beta policies | -| `--custom`, `-c ` | Validate and load a custom policy file; repeatable | +## Policies and packs -## Delivery and maintenance flags +```bash +failproofai policies +failproofai policies add block-sudo +failproofai policies show owner/repo +failproofai policies add owner/repo --category git,database +failproofai policies remove owner/repo +``` -| Command | Flags | +Anything with a slash is a pack source; anything without one is a policy name. + +Useful pack selection flags: + +| Flag | Use | | --- | --- | -| `backfill` | `--since <30d\|6m\|YYYY-MM-DD>`, `--dry-run` | -| `flush` | `--wait`, `--timeout ` | -| `update` | `--no-daemon` | -| `migrate` | `--dry-run` | -| `uninstall` | `--purge`, `--dry-run`, `--yes` | +| `--policy a,b` | Select named policies | +| `--category x,y` | Select categories | +| `--all` | Select everything | +| `--cli ` | Restrict to named harnesses | -`failproofai update` should be run after `npm install -g failproofai@latest`; it performs home-layout migrations, installs the matching daemon binary, and restarts the service. `--no-daemon` performs only the layout migration. +Use `failproofai policies -i -c ` to load a custom policy from any path. Convention files named `*policies.{js,mjs,ts}` under `.failproofai/policies/` load automatically. -## Harness paths +## Pause enforcement -```text -failproofai harness list [harness] -failproofai harness add-path [label=] -failproofai harness remove-path +```bash +failproofai config --pause 10m +failproofai config --status +failproofai config --resume ``` -Supported harness names are `claude`, `codex`, `copilot`, `cursor`, `opencode`, `pi`, `hermes`, `openclaw`, `factory`, `devin`, `antigravity`, and `goose`. +A pause applies to one local session and always expires. It does not pause Cloud-managed policies or `block-failproofai-commands`. + +## Audit and delivery + +```bash +failproofai audit +failproofai audit --schedule 7 --email team@example.com +failproofai audit --status +failproofai flush --wait +failproofai backfill --since 30d --dry-run +``` -Labels namespace derived agent IDs when two roots contain copies of the same project. Overlapping roots and duplicate labels are rejected to prevent duplicate collection or cursor corruption. Extra-path configuration reloads without a daemon restart. +`backfill` reads `~/.failproofai/config.json`. Drop `--dry-run` to send the history again. -Container environments can replace file-configured extra paths with a comma-separated variable named `FAILPROOFAI__EXTRA_PATHS`, for example: +## Extra session locations ```bash -export FAILPROOFAI_OPENCLAW_EXTRA_PATHS="user1=/srv/openclaw-a,user2=/srv/openclaw-b" +failproofai harness list +failproofai harness add-path claude work=/srv/team/.claude/projects +failproofai harness remove-path claude work ``` -## Environment variables +Labels prevent sessions from two copied homes merging under the same derived agent ID. -Use configuration files for persistent machine behavior. Environment variables are most useful for containers, tests, and one process. +Supported harnesses are `claude`, `codex`, `copilot`, `cursor`, `opencode`, `pi`, `hermes`, `openclaw`, `factory`, `devin`, `antigravity`, and `goose`. Scope support differs; see [Harnesses](/reference/harnesses). + +## Useful environment variables | Variable | Use | | --- | --- | -| `FAILPROOFAI_HOME` | Relocate the complete `~/.failproofai` layout | -| `FAILPROOFAI_LOG_LEVEL` | Set local logging verbosity | -| `FAILPROOFAI_HOOK_LOG_FILE` | Write hook diagnostics to a selected file | -| `FAILPROOFAI_TELEMETRY_DISABLED=1` | Disable anonymous telemetry for this process | -| `FAILPROOFAI_NO_FIRST_RUN=1` | Skip interactive first-run setup | -| `FAILPROOFAI_NO_AUTO_AUDIT=1` | Skip the post-setup local audit | -| `FAILPROOFAI_LLM_BASE_URL` | Override the OpenAI-compatible endpoint used by LLM policies | -| `FAILPROOFAI_LLM_API_KEY` | Supply the API key used by LLM policies | -| `FAILPROOFAI_LLM_MODEL` | Select the model used by LLM policies | -| `FAILPROOFAI_POLICY_LOAD_TIMEOUT_MS` | Bound custom policy module loading | -| `FAILPROOFAI_NO_DOWNLOAD=1` | Refuse to fetch packs and daemon binaries; what is installed keeps enforcing | -| `FAILPROOFAI_PACK_BASE_URL` | Fetch packs from a mirror instead of `github.com` | -| `FAILPROOFAI__EXTRA_PATHS` | Replace configured extra capture paths for one harness | -| `NO_COLOR` | Disable colored terminal output | - -Agent-specific home variables such as `CLAUDE_PROJECTS_PATH`, `CURSOR_HOME`, `HERMES_HOME`, and `OPENCLAW_HOME` override where Failproof AI discovers local sessions for that harness. - -## Pause or remove a machine safely +| `FAILPROOFAI_CLOUD_TOKEN` | Cloud machine key | +| `FAILPROOFAI_CLOUD_URL` | Override the Cloud URL | +| `FAILPROOFAI_HOME` | Move the local state directory | +| `FAILPROOFAI_NO_DOWNLOAD=1` | Refuse pack and service downloads | +| `FAILPROOFAI_DAEMON_BASE_URL` | Use a service-binary mirror | +| `FAILPROOFAI_PACK_BASE_URL` | Use a pack mirror | +| `FAILPROOFAI_NO_FIRST_RUN=1` | Disable first-run prompts | +| `FAILPROOFAI_TELEMETRY_DISABLED=1` | Disable anonymous CLI telemetry | +| `FAILPROOFAI__EXTRA_PATHS` | Replace extra paths for one harness | +| `NO_COLOR` | Disable terminal color | + +## Remove or upgrade ```bash -failproofai config --pause -failproofai config --status -failproofai config --resume +npm install -g failproofai@latest +failproofai update ``` -A local session pause does not disable Cloud-managed policies. Restore Cloud deployments through the Cloud enforcement workflow when the rollout itself is the problem. - -Before removing the npm package, remove installed hooks and the daemon: +Before removing the package: ```bash failproofai uninstall --dry-run @@ -151,8 +137,8 @@ failproofai uninstall --yes npm rm -g failproofai ``` -Run `failproofai --help` for version-specific details. - - Run `failproofai uninstall` before `npm rm -g failproofai`; npm does not remove installed agent hooks or the daemon service. + Run `failproofai uninstall` before removing the npm package. npm does not remove installed hooks or the background service. + +Run `failproofai help ` for the complete options in your installed version. diff --git a/docs/reference/harnesses.mdx b/docs/reference/harnesses.mdx index ceafd3c9c..5e4cb06bf 100644 --- a/docs/reference/harnesses.mdx +++ b/docs/reference/harnesses.mdx @@ -13,6 +13,8 @@ The same policies and the same session history apply whichever one an agent runs An agent that runs in **none** of the twelve is instrumented directly with the [Python SDK](/reference/custom-agents). That is a different contract, and worth stating plainly: the SDK delivers tracing, sessions, evaluations and audits — **it does not enforce policies on its own.** Blocking an unsafe action before it executes needs an enforcement hook at your runtime's tool boundary; [contact us](mailto:support@befailproof.ai) and we will map it. +## Hook scopes + | Harness | Supported hook scopes | | --- | --- | | Claude Code | User, project, local | @@ -20,11 +22,32 @@ An agent that runs in **none** of the twelve is instrumented directly with the [ | Factory Droid, Devin CLI, Antigravity CLI, Goose | User, project | | Hermes, OpenClaw | User | +Claude Code is the only harness with a **local** scope. Hermes and OpenClaw have no project configuration at all — they are user scope only, and the CLI refuses `--scope project` for them. + Each integration normalizes its native hook event names, tool names, and tool-input fields before policies run. A policy can only act on events the harness exposes; test end-of-turn and instruction behavior on the exact harness and version you deploy. +## Where the hook configuration lands + +| Harness | User scope | Project scope | +| --- | --- | --- | +| Claude Code | `~/.claude/settings.json` | `.claude/settings.json` (local: `.claude/settings.local.json`) | +| Codex | `~/.codex/hooks.json` | `.codex/hooks.json` | +| GitHub Copilot CLI | `~/.copilot/hooks/failproofai.json` | `.github/hooks/failproofai.json` | +| Cursor | `~/.cursor/hooks.json` | `.cursor/hooks.json` | +| OpenCode | `~/.config/opencode/opencode.json` | `.opencode/opencode.json` | +| Pi | `~/.pi/agent/settings.json` | `.pi/settings.json` | +| Hermes | `~/.hermes/config.yaml` | — | +| OpenClaw | `~/.openclaw/openclaw.json` | — | +| Factory Droid | `~/.factory/hooks.json` | `.factory/hooks.json` | +| Devin CLI | `~/.config/devin/config.json` | `.devin/config.json` | +| Antigravity CLI | `~/.gemini/config/hooks.json` | `.agents/hooks.json` | +| Goose | `~/.agents/plugins/failproofai/hooks/hooks.json` | `.agents/plugins/failproofai/hooks/hooks.json` | + +OpenCode, Pi and OpenClaw are plugin integrations rather than shell-hook integrations: the file above registers a plugin or extension package, which calls the failproofai binary and translates its verdict. + ## Enforcement capability -“Block” means the current adapter's returned verdict is consumed by the named harness. Post-tool blocking may replace the result shown to the model but cannot undo a tool side effect that already happened. +"Block" means the current adapter's returned verdict is consumed by the named harness. Post-tool blocking may replace the result shown to the model but cannot undo a tool side effect that already happened. | Harness | Verified blocking events | Observe-only or non-blocking caveats | | --- | --- | --- | @@ -32,23 +55,83 @@ Each integration normalizes its native hook event names, tool names, and tool-in | Codex | `PreToolUse`, `PermissionRequest`, `UserPromptSubmit`, `Stop`, `SubagentStop`, `PostToolUse` | Post-tool blocking replaces the result after execution; session-start and compact events are observational in the current adapter. | | GitHub Copilot CLI | `PreToolUse`, `UserPromptSubmit`, `PermissionRequest`, `Stop`, `SubagentStop`, `PostToolUse` | Post-tool blocking replaces the result after execution; session and notification events are observational. | | Cursor | `PreToolUse`, `UserPromptSubmit`, `Stop` | `PostToolUse` and session events are observational. | -| OpenCode | `PreToolUse` | Post-tool and lifecycle events are observational; current stop handling is guidance for a later turn rather than a verified gate. | +| OpenCode | `PreToolUse` | Post-tool and lifecycle events are observational; current stop handling is guidance for a later turn rather than a verified gate. `PermissionRequest` never runs at all. | | Pi | `PreToolUse`, `UserPromptSubmit` | Post-tool and lifecycle events are observational; stop guidance applies to a later turn. | -| Hermes | `PreToolUse` | Post-tool, session, and subagent-stop verdicts are not gates. | +| Hermes | `PreToolUse` | Post-tool, session, and subagent-stop verdicts are not gates. No `Stop` event is installed. | | OpenClaw | `PreToolUse`, `UserPromptSubmit`, `Stop` | Post-tool, session, subagent-stop, and compaction events are observational. | | Factory Droid | `PreToolUse`, `UserPromptSubmit`, `Stop`, `PreCompact` | Post-tool and subagent-stop verdicts are observational. | | Devin CLI | `PreToolUse`, `UserPromptSubmit`, `Stop`, conditional `PermissionRequest` | Permission hooks do not run in every permission mode; post-tool and session events are observational. | | Antigravity CLI | `PreToolUse`, `Stop` | User-prompt and post-tool verdicts are observational; prompt instructions can still be injected. | | Goose | `PreToolUse` | User-prompt, post-tool, and session events are observational. A native blocking stop hook exists upstream but is not installed by the current adapter. | -Capabilities are version-sensitive. Re-test after upgrading an agent CLI, especially when a policy relies on prompt, stop, permission, or post-tool behavior rather than the common pre-tool gate. +An event absent from both columns is **not verified** — treat it as unknown, never as blocking. + +### Conditions on a listed gate + +Several rows above are real gates that are nonetheless bounded or conditional. A policy that depends on one of these needs the condition as much as the row. + +| Harness | Event | Condition | +| --- | --- | --- | +| Claude Code | `Stop`, `SubagentStop` | Capped by `CLAUDE_CODE_STOP_HOOK_BLOCK_CAP`, default 8. Discarded on end-turn paths | +| Cursor | `Stop` | Capped by `loop_limit`, default 5, and consumed only when the turn completes — a user abort or a turn error discards it | +| Cursor | `Stop` | **Cursor Cloud Agent VMs run no `stop` or `subagentStop` hooks at all.** The gate covers local sessions only | +| Codex | `SubagentStop` | Only ThreadSpawn subagents dispatch it; every other subagent source never runs the hook | +| GitHub Copilot CLI | `SubagentStop` | Skipped entirely for `isSidekick` subagents | +| Devin CLI | `PermissionRequest` | Never fires under `--permission-mode dangerous`, and never for auto-approved read-only tools | +| OpenCode | `PermissionRequest` | A dead hook: `permission.ask` is declared and documented upstream but never invoked, so the policy does not even run | +| Hermes | `Stop` | No `Stop` event is installed, by choice. The five `require-*-before-stop` builtins are inapplicable on Hermes | +| Goose | `Stop` | Same outcome: no `Stop` is installed, so the five `require-*-before-stop` builtins are inapplicable | + +### Where `instruct()` degrades + +A `deny` is not the only verdict a policy returns. `instruct()` hands the agent a directive and lets the action proceed — but not every harness has a channel to carry one. Where there is none, failproofai allows the action and writes the instruction to stderr for the operator's logs; the model never sees it. + +| Harness | Events where `instruct()` degrades to a stderr note | +| --- | --- | +| Hermes | Every event | +| Goose | Every event | +| Pi | Every event except `Stop` | +| OpenClaw | Every event except `Stop` | +| Factory Droid | Every event except `Stop` | +| Antigravity CLI | Every event except `Stop` and `UserPromptSubmit` | + +Everywhere else, the instruction is returned through the harness's own additional-context channel. + +### Versions these claims were probed against + +A version is part of the claim, not a footnote. Re-test after upgrading an agent CLI, especially when a policy relies on prompt, stop, permission, or post-tool behavior rather than the common pre-tool gate. + +| Harness | Probed version | +| --- | --- | +| Claude Code | 2.1.220 | +| Codex | `fe01054a`, with `PostToolUse` re-probed live at 0.147.0 | +| GitHub Copilot CLI | 1.0.71, some call sites re-read in 1.0.68 and 1.0.78 | +| Cursor | cursor-agent 2026.07.16-899851b | +| OpenCode | 1.18.9, re-checked at 1.14.33 | +| Pi | 0.80.10 | +| Hermes | hermes-agent `5771a6e` | +| OpenClaw | v2026.7.2 | +| Factory Droid | droid 0.175.1 | +| Devin CLI | 3000.2.17 | +| Antigravity CLI | agy 1.1.8 | +| Goose | 1.43.0 | + + + Several Codex rows cite source paths that were restructured after `fe01054a` and no longer exist at 0.147.0. They are not known-wrong, but they are unverified against any shipping Codex and are due for a re-probe. Only the Codex `PostToolUse` row was re-probed live. + + +## VS Code agent mode + +VS Code's built-in Copilot Chat agent mode is **not a thirteenth integration**. It discovers hook configuration from `.github/hooks/*.json`, `~/.copilot/hooks/*.json` and `~/.claude/settings.json` — exactly the paths the `copilot` and `claude` installs already write — and uses the same Claude-shaped deny contract. Installing either one already enforces inside VS Code agent-mode sessions. + +The feature is a preview and needs an active GitHub Copilot subscription plus agent mode. ## Install capture and policy hooks 1. Open **Administration → Keys** and create a key with `events:add` and `policies:pull`, named for the machine or environment. - 2. On the target machine, connect the local CLI with the displayed key and install the harness hooks. + 2. On the target machine, connect the local CLI with the displayed key. 3. Start a new agent session, then confirm its hook and session events under **Observe → Events**. 4. Open **Observe → policy** for the same time window and confirm a policy decision is attributed to the machine. @@ -65,16 +148,16 @@ Capabilities are version-sensitive. Re-test after upgrading an agent CLI, especi ![The Policy page used to verify policy decisions from a newly connected harness.](/images/dashboard/policy-observe.png) - Install hooks for every detected harness: + `failproofai config` wires hooks into every supported agent CLI it finds, installs the daemon, and connects Cloud when a key is present: ```bash - failproofai config \ - --connect https://app.befailproof.ai \ - --token - failproofai policies --install + failproofai config --token + failproofai policies add FailproofAI/policies ``` - Or target named harnesses and a configuration scope: + Setup chooses no policies, which is why the second command is there. + + To wire one harness by hand — with `--cli` omitted, `--install` detects what is installed and prompts: ```bash failproofai policies --install \ @@ -82,7 +165,7 @@ Capabilities are version-sensitive. Re-test after upgrading an agent CLI, especi --scope user ``` - Project scope keeps hook configuration with a repository. User scope covers work across repositories. Claude Code also supports local scope; support varies by harness and the CLI rejects unsupported combinations. + Project scope keeps hook configuration with a repository. User scope covers work across repositories. Check the scope table above before combining `--cli` and `--scope`; the CLI rejects an unsupported pair. Verify the machine and its events: @@ -94,6 +177,10 @@ Capabilities are version-sensitive. Re-test after upgrading an agent CLI, especi + + In headless `copilot -p` runs started from a fresh directory, the project-scope `.github/hooks/failproofai.json` was **not** loaded. Treat user scope as the reliable enforcement point for Copilot in CI until that is verified otherwise. + + ## Add a non-default session path @@ -113,6 +200,8 @@ Capabilities are version-sensitive. Re-test after upgrading an agent CLI, especi ``` Remove a path with `failproofai harness remove-path claude checkout`. + + The label namespaces derived agent ids. Two locations holding copies of the same project derive the same id from the transcript, so without a label they merge into one agent. Overlapping roots and duplicate labels are refused — see [the CLI reference](/reference/failproof-cli) for why each rejection exists. diff --git a/docs/reference/http-api.mdx b/docs/reference/http-api.mdx index ea88e20eb..fcfa72d2a 100644 --- a/docs/reference/http-api.mdx +++ b/docs/reference/http-api.mdx @@ -4,43 +4,53 @@ description: "Authenticate to the public Failproof AI Cloud `/v1` API and use th icon: "braces" --- -The public API is served under `/v1` on your Failproof AI dashboard origin. +The public API is served under `/v1` on your Failproof AI dashboard origin. On Failproof AI Cloud that origin is `https://app.befailproof.ai`, so the base path is `https://app.befailproof.ai/v1`. On a self-hosted deployment it is your own dashboard host followed by `/v1`; pass that host to `fp` with `--base-url ` or `FP_DASHBOARD_URL`. ## Create a key and make a request - 1. Open **Administration → Keys**, select **Create key**, and choose the narrowest permission preset that covers the integration. + 1. Open **Administration → Keys**, select **new key**, and choose the narrowest permission preset that covers the integration. 2. Add individual grants only when needed, create the key, and copy its one-time secret. 3. Make a test request to `/v1/sessions` and confirm the key remains active in the Keys page. 4. Rotate or disable the key from its action menu when the integration changes ownership. ![The new API key drawer with permission presets and individual grants.](/images/dashboard/key-create.png) - The creation drawer is shown above. The one-time secret appears only after you select **create**; copy it before closing that confirmation. + The creation drawer is shown above. The one-time secret appears only after the key is created; copy it before closing that confirmation. - Create a read key and use it directly with either `fp` or `curl`: + Create a read key and use it directly with either `fp` or `curl`. `fp` reads the key from `--api-key` or from `FP_API_KEY`: ```bash fp keys create reliability-reader \ --permission-set read-only - fp --api-key sessions --since 24h + export FP_API_KEY="" + fp --org reliability-team sessions --since 24h ``` ```bash curl "https://app.befailproof.ai/v1/sessions?limit=20" \ - -H "Authorization: Bearer $FAILPROOFAI_KEY" + -H "Authorization: Bearer $FP_API_KEY" \ + -H "X-AgentEye-Org: reliability-team" ``` -Keys are scoped to an organization and permission set. A request without the endpoint's required permission returns `403` and identifies the missing permission. +Keys are scoped to an organization and to a flat list of permissions. A permission set only seeds that list at creation time — the key stores the expanded grants, so changing the set afterwards does not change the key. A request without the endpoint's required permission returns `403` and identifies the missing permission. ## Organization selection -An organization key acts on its organization automatically. An instance-scoped key can select an organization per request: +An organization key acts on its organization automatically. An instance-scoped key selects an organization per request, and a tenant with more than one organization must always name one in key mode — `fp` never sends a saved organization when it is authenticating with a key. + +| Caller | How you name the organization | +| --- | --- | +| `fp`, per invocation | `--org `, before the subcommand | +| `fp`, from the environment | `FP_ORG` | +| Raw HTTP | the `X-AgentEye-Org: ` request header | + +`X-AgentEye-Org` keeps its original spelling on the wire. It is the header the dashboard reads to resolve the active organization, and renaming it in a client breaks the request. @@ -48,27 +58,67 @@ An organization key acts on its organization automatically. An instance-scoped k - Use `--org` before the command, or send the organization header for an instance-scoped API key. + Use `--org` before the command, or send the organization header for an instance-scoped API key. `fp orgs list` does not discover the slug for you here — it refuses in key mode; read the slug from the dashboard's organization switcher instead. ```bash - fp orgs list fp --org reliability-team sessions --since 24h ``` ```bash curl "https://app.befailproof.ai/v1/usage" \ - -H "Authorization: Bearer $FAILPROOFAI_KEY" \ + -H "Authorization: Bearer $FP_API_KEY" \ -H "X-AgentEye-Org: reliability-team" ``` +## Pagination + +List endpoints are keyset-paginated, and the same contract applies whether you call them through `fp` or through `curl`. + +| Element | Meaning | +| --- | --- | +| `limit` query parameter | Rows requested in this call. `fp` clamps a single request to 200 rows on every endpoint. A raw request is bound by the server's own per-endpoint ceiling instead. | +| `cursor` query parameter | An opaque token that resumes the feed after a previous page. | +| `next_cursor` response field | The token for the next page. `null` means the feed is exhausted. | + +Those ceilings differ by endpoint, and each endpoint page names its own default and cap: + +| Endpoint | Default | Cap | +| --- | --- | --- | +| `/events`, `/events/summary` | 50 | 1000 | +| `/issues`, `/alerts/{id}/issues` | 200 | 1000 | +| `/evaluation-jobs` | 100 | 500 | +| `/audits/findings` | 100 | 500 | +| `/sessions`, `/evaluations` | 50 | 200 | +| `/audits/{id}/runs` | 50 | 200 | + +Read a full feed by re-issuing the same request with `cursor` set to the previous response's `next_cursor` until `next_cursor` comes back `null`. The equivalent `fp` flags — `--all`, `--cursor` and `--page-size` — are documented in the [Cloud CLI reference](/reference/cloud-cli). + +## Endpoint reference and error handling + Use the generated endpoint pages in this section for current paths, parameters, permission requirements, and status codes. The specification is generated from the server route annotations and checked against the `/v1` router. -The current specification has complete route, method, parameter, permission, and status-code coverage. Some response bodies remain intentionally untyped because the server still constructs them as dynamic JSON. Inspect a real response before generating a strongly typed client around an endpoint without a response schema. +The current specification has complete route, method, parameter, permission, and status-code coverage. Request bodies are typed; response bodies are not — the spec carries no response schemas today, because the server still constructs them as dynamic JSON. Inspect a real response before generating a strongly typed client. + +Use `Content-Type: application/json` for JSON writes. Treat `401` as missing or invalid authentication, `403` as a valid identity without the required permission, `404` as a missing or organization-inaccessible resource, `409` as a state conflict, `400` as a rejected parameter — an unknown filter value, a malformed query parameter, a statement that would not run — and `422` as an invalid field or permission value in a request body. Error responses include a human-readable message; permission failures also name the required grant. + +## Surfaces that `/v1` does not carry + +Cloud-managed policies, the fleet and the guardrail decision feed are operator surfaces — reads and writes alike. They are root-only on the server and deliberately absent from `/v1`, which is internet-facing. Publishing a policy version or deploying it to a machine, for example, is not something an API key can do, and reading back what the fleet decided is on the same surface. + +An API key therefore gets a refusal by design on those routes, and `fp` refuses before it opens a connection rather than letting the request 404 with no explanation: + +| Command | Behavior under an API key | +| --- | --- | +| `fp policies` | Refuses with exit `2` and names cloud-managed policies as an operator surface | +| `fp fleet` | Refuses with exit `2` and names the fleet as an operator surface | +| `fp guardrails` | Refuses with exit `2` and names the guardrail feed as an operator surface | +| `fp agent` | Refuses with exit `2`; the assistant is implemented by the dashboard, not the API | +| `fp orgs` | Refuses with exit `2`; org membership belongs to a signed-in user, and a key already acts for one org | -Use `Content-Type: application/json` for JSON writes. Treat `401` as missing or invalid authentication, `403` as a valid identity without the required permission, `404` as a missing or organization-inaccessible resource, `409` as a state conflict, and `422` as an invalid field or permission value. Error responses include a human-readable message; permission failures also name the required grant. +Run these under a signed-in user session (`fp login`) instead, or use the dashboard. The read and administration families — sessions, events, evaluations, audits, issues, alerts, keys, permission sets, users, settings, queries and usage — each have a `/v1` route and work with a key. The one gap is `fp keys update`: it needs `keys:update`, a permission no API key can hold, so a key can mint and disable keys but never edit one's permissions. `fp login` and `fp logout` also refuse in key mode, for a different reason: a key already is the credential, and it is never saved to disk. - Policy enforcement deployment is intentionally managed outside the ordinary public `/v1` surface. Use the supported Cloud deployment workflow. + Policy enforcement deployment is intentionally managed outside the ordinary public `/v1` surface. Use the supported Cloud deployment workflow described in [Deploy policies](/policies/deploy). diff --git a/docs/reference/local-dashboard.mdx b/docs/reference/local-dashboard.mdx index a9c224a98..dae41682c 100644 --- a/docs/reference/local-dashboard.mdx +++ b/docs/reference/local-dashboard.mdx @@ -6,14 +6,16 @@ icon: "monitor-cog" Run `failproofai` without arguments to start the bundled dashboard at `http://localhost:8020`. It reads local agent histories, policy configuration, audit results, and hook activity directly from the machine. -The local dashboard is separate from Failproof AI Cloud. It works without a Cloud account and does not prove that events were delivered to your organization. +On a machine that has not been set up, the bare command runs first-run setup before it opens the dashboard. It does that only on an interactive terminal: piped or in CI, it prints a one-line hint and opens the dashboard, because a wizard nobody can answer must never block the command you typed. Set `FAILPROOFAI_NO_FIRST_RUN=1` to skip the redirect entirely and go straight to the dashboard. + +The local dashboard is separate from Failproof AI Cloud. It works without a Cloud account and does not prove that events were delivered to your organization. On a machine that completed setup, the decisions it shows were made by the `failproofaid` daemon, which is the only evaluator there; on a machine that has not been set up, hooks evaluate in-process. ## Dashboard areas | Area | What you can accomplish | | --- | --- | -| Policies → Activity | Inspect local allow, instruct, and deny decisions; filter by decision, event, CLI, tool, source, policy, and session. | -| Policies → Configure | Enable builtins, edit supported parameters, toggle discovered custom policies, and select target harnesses. | +| Policies → Activity | Inspect local allow, instruct, and deny decisions; filter by decision, event, CLI, source, policy, and session. | +| Policies → Configure | Enable policies from an installed pack, edit supported parameters, toggle discovered custom and convention policies, and select target harnesses. | | Projects | Browse discovered projects across supported agent histories and compare their most recent sessions. | | Project sessions | Open one local transcript, review raw ordered entries and subagents, download it, and correlate policy activity. | | Audit | Review the last offline scan, risky patterns, strengths, affected projects, and suggested builtin policies. | @@ -24,8 +26,8 @@ The local dashboard is separate from Failproof AI Cloud. It works without a Clou 1. Open **Policies → Activity** and set the decision and source filters. - 2. Narrow by event, harness, tool, or policy name. - 3. Expand a row to inspect its reason, matched policies, source, execution mode, and duration. + 2. Narrow by event, harness, policy name, or session. + 3. The row carries the decision, event, harness, tool, policy, permission mode, and duration. Expand it for the full reason, every matched policy, the deciding source, the cloud deployment, and the session's working directory and transcript path. Tool name is shown on the row but is not a filter. 4. Follow the session link to place the decision in transcript context. A denied-looking row can still be observational on a harness/event pair that does not consume blocking verdicts. The detail view calls out verified enforcement capability. @@ -45,19 +47,22 @@ The local dashboard is separate from Failproof AI Cloud. It works without a Clou - 1. Open **Policies → Configure** and choose the harnesses and configuration scope. - 2. Enable a builtin or discovered custom policy. - 3. For a parameterized builtin, open its configuration control and save supported values. + 1. Open **Policies → Configure** and choose the harnesses and configuration scope. The tab is addressable directly at `http://localhost:8020/policies?tab=policies`. + 2. Enable a policy from an installed pack, or a discovered custom or convention policy. + 3. For a parameterized policy, open its configuration control and save supported values. 4. Return to Activity and run matching and non-matching actions. - Convention policies show their project or user source. Explicit custom-path changes may require rerunning CLI configuration so the selected path is recorded. + The list is empty until a pack is installed — this build compiles in no policies of its own, so run `failproofai policies add FailproofAI/policies` first. Convention policies show their project or user source. Explicit custom-path changes may require rerunning CLI configuration so the selected path is recorded. ```bash - failproofai policy add block-sudo --scope project + failproofai policies show FailproofAI/policies + failproofai policies add block-sudo --scope project failproofai policies --install --custom ./security.policies.ts --scope project failproofai policies ``` + + `policies show /` reads a pack's contents before you install it. `--scope` takes `user`, `project`, or `local`; only Claude Code supports `local`, and Hermes and OpenClaw take `user` only. @@ -65,7 +70,7 @@ The local dashboard is separate from Failproof AI Cloud. It works without a Clou The Projects page combines supported local history stores. Select a project to list its sessions, then open a session for the raw log viewer, subagent segments, download action, and session-scoped policy activity. -If a project or session is missing, confirm the harness uses its default history location or register an extra root with `failproofai harness add-path`. +If a project or session is missing, confirm the harness uses its default history location or register an extra root with `failproofai harness add-path [