From 644a4c09ce03e1d8c4d15c3ee4a19aad9daa6995 Mon Sep 17 00:00:00 2001 From: agenteye-skill-sync Date: Thu, 27 Aug 2026 08:13:04 +0000 Subject: [PATCH] Sync agenteye-evaluator skill from agenteye@912c13f MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Automated mirror of evaluator-sdk/skill/. Source of truth: FailproofAI/agenteye — do not hand-edit. --- skills/agenteye-evaluator/SKILL.md | 10 +-- .../references/brainstorm.md | 34 ++++----- .../agenteye-evaluator/references/scaffold.md | 2 +- .../references/session-data.md | 70 +++++++++++-------- 4 files changed, 63 insertions(+), 53 deletions(-) diff --git a/skills/agenteye-evaluator/SKILL.md b/skills/agenteye-evaluator/SKILL.md index 5b84171..eb034c2 100644 --- a/skills/agenteye-evaluator/SKILL.md +++ b/skills/agenteye-evaluator/SKILL.md @@ -9,7 +9,7 @@ description: |- Served by the `agenteye-evaluator` Python SDK, with the `agenteye` CLI supplying real session data to design against. - NOT for reading eval results that already exist or checking whether quality dropped (that's `agenteye-cli` — `agenteye evals`), instrumenting an agent with the AgentEye SDK (that's `agenteye-python-sdk`), or alerting on scores. + NOT for reading eval results that already exist or checking whether quality dropped (that's `fp-cli` — `fp evals`), instrumenting an agent with the AgentEye SDK (that's `agenteye-python-sdk`), or alerting on scores. --- # AgentEye Evaluator @@ -23,7 +23,7 @@ run the service, AgentEye calls it. agent run ends (agent_end event) → AgentEye server POSTs the full transcript to YOUR service → you return {"scores": {"helpfulness": 0.9, ...}} - → scores land in the evaluations table → visible in the dashboard and `agenteye evals` + → scores land in the evaluations table → visible in the dashboard and `fp evals` ``` The SDK part of this is small — a decorator and two models. **The hard part is @@ -316,13 +316,13 @@ Then restart the server. Notes worth knowing before you debug: Scores landing is the only proof. After deploying, finish a session and check: ```bash -agenteye --json evals --session-id # your scores, or status=error/timeout -agenteye --json evals --aggregate --since 24h +fp --json evals --session-id # your scores, or status=error/timeout +fp --json evals --aggregate --since 24h ``` If a session shows `status: error` or `timeout`, the `error` field carries your message (the one from the error dict — exceptions never reach it). Reading -existing scores from here on is the `agenteye-cli` skill's job. +existing scores from here on is the `fp-cli` skill's job. **Debug order when nothing appears:** is `EVALUATOR_ENDPOINT` set on the server → did the session emit `agent_end` (or does `@app.config` return diff --git a/skills/agenteye-evaluator/references/brainstorm.md b/skills/agenteye-evaluator/references/brainstorm.md index d7b1b2e..10513e3 100644 --- a/skills/agenteye-evaluator/references/brainstorm.md +++ b/skills/agenteye-evaluator/references/brainstorm.md @@ -39,30 +39,30 @@ learning what to look *for*. Before you open a single session, quantify the landscape so your corpus is *chosen from evidence*, not guessed. Run, at minimum: -- **Facets — what even exists.** `agenteye --json list agents`, `list envs`, +- **Facets — what even exists.** `fp --json list agents`, `list envs`, `list error_types`, `list score_filters`. This is the vocabulary of their deployment: which agents run, which environments, what errors occur, and which score keys (if any) already exist. -- **Where it hurts, and how often.** `agenteye --json errors --aggregate --since 7d` - for the error hotspots; `agenteye --json sessions --status error,timeout --since 7d +- **Where it hurts, and how often.** `fp --json errors --aggregate --since 7d` + for the error hotspots; `fp --json sessions --status error,timeout --since 7d --all --limit 1000` for the count and identity of failed runs (compare against a healthy window to get a rate, not just a number). -- **What's already scored.** `agenteye --json evals --aggregate --since 7d` → +- **What's already scored.** `fp --json evals --aggregate --since 7d` → `score_stats` tells you which dimensions are tracked today and how they're distributed — so you don't re-propose something they already have, and you can spot a score that's uniformly high (measuring nothing) or already regressing. -- **Anything the flags can't express — drop to SQL.** `agenteye --json query run +- **Anything the flags can't express — drop to SQL.** `fp --json query run --sql "…"` runs over the exposed tables `events`, `evaluations`, `agent_sessions` - (`agenteye --json query schema` prints the columns first — check it before + (`fp --json query schema` prints the columns first — check it before guessing one). This is how you get the shape of the whole dataset: ```bash # event-type histogram across the population - agenteye --json query run --sql \ + fp --json query run --sql \ "SELECT event_type, count() c FROM events GROUP BY event_type ORDER BY c DESC" # the biggest / longest sessions — your outlier candidates - agenteye --json query run --sql \ + fp --json query run --sql \ "SELECT session_id, count() c FROM events GROUP BY session_id ORDER BY c DESC LIMIT 20" ``` @@ -88,7 +88,7 @@ random: with the most errors. Pull mechanics are in [`session-data.md`](session-data.md): the export endpoint gives -a byte-identical fixture, or `agenteye --json events --full --session-id --order +a byte-identical fixture, or `fp --json events --full --session-id --order asc --all --limit 1000`. Skim on the payload-free feed first (its one-line server `summary` per event is ideal for reading a session's *shape*), then `--full` only on the ones you're reading line by line. **Confirm the corpus with the user** before you @@ -133,7 +133,7 @@ is expressible in SQL, compute it per session and compare the cohorts: ```bash # does "how it ended" actually track the user's good/bad call? -agenteye --json query run --sql \ +fp --json query run --sql \ "SELECT session_id, countIf(event_type='agent_end') AS ended, count() AS events FROM events GROUP BY session_id ORDER BY ended ASC, events DESC LIMIT 30" ``` @@ -166,7 +166,7 @@ in their data* — the schema allowing a field doesn't mean their instrumentatio it: ```bash -agenteye --json events --full --session-id --all --limit 1000 \ +fp --json events --full --session-id --all --limit 1000 \ | jq '[.events[] | select(.event_type=="tool_use") | .payload.tool_name] | unique' ``` @@ -255,15 +255,15 @@ quick index. | To learn… | Run | |---|---| -| what exists (agents, envs, errors, score keys) | `agenteye --json list agents` / `list envs` / `list error_types` / `list score_filters` | -| where it hurts, how often | `agenteye --json errors --aggregate --since 7d` · `agenteye --json sessions --status error,timeout --since 7d --all --limit 1000` | -| what's already scored / regressing | `agenteye --json evals --aggregate --since 7d` (→ `score_stats`) · `agenteye --json evals --score :..0.5 --since 7d --all --limit 200` | -| a session's shape, then its content | `agenteye --json events --session-id --order asc --all --limit 1000` → add `--full` | -| anything the flags can't (duration, tokens, loops, histograms) | `agenteye --json query run --sql "…"` over `events`/`evaluations`/`agent_sessions` · `query schema` for columns | +| what exists (agents, envs, errors, score keys) | `fp --json list agents` / `list envs` / `list error_types` / `list score_filters` | +| where it hurts, how often | `fp --json errors --aggregate --since 7d` · `fp --json sessions --status error,timeout --since 7d --all --limit 1000` | +| what's already scored / regressing | `fp --json evals --aggregate --since 7d` (→ `score_stats`) · `fp --json evals --score :..0.5 --since 7d --all --limit 200` | +| a session's shape, then its content | `fp --json events --session-id --order asc --all --limit 1000` → add `--full` | +| anything the flags can't (duration, tokens, loops, histograms) | `fp --json query run --sql "…"` over `events`/`evaluations`/`agent_sessions` · `query schema` for columns | **Run them *properly* — the gotchas that make a command lie:** -- **Globals go before the command:** `agenteye --json events …`, never `agenteye +- **Globals go before the command:** `fp --json events …`, never `fp events --json` (exit 2). - **`--all` is capped by `--limit` (default 50).** A bare `--all` returns 50 rows and looks complete. Always `--all --limit 1000`; bigger needs cursor paging. diff --git a/skills/agenteye-evaluator/references/scaffold.md b/skills/agenteye-evaluator/references/scaffold.md index 55d354a..540a37e 100644 --- a/skills/agenteye-evaluator/references/scaffold.md +++ b/skills/agenteye-evaluator/references/scaffold.md @@ -261,7 +261,7 @@ EVALUATOR_TOKEN=dev-secret # byte-identical to the evaluator's Then finish a session and confirm the scores landed: ```bash -agenteye --json evals --session-id +fp --json evals --session-id ``` If nothing appears, work down: is `EVALUATOR_ENDPOINT` set on the server (unset is diff --git a/skills/agenteye-evaluator/references/session-data.md b/skills/agenteye-evaluator/references/session-data.md index ce51cac..24ba5a7 100644 --- a/skills/agenteye-evaluator/references/session-data.md +++ b/skills/agenteye-evaluator/references/session-data.md @@ -1,9 +1,10 @@ # Getting real session data Two jobs: **find sessions worth looking at**, and **turn one into a fixture** you -can replay through your evaluator. Both go through the `agenteye` CLI, which -needs a logged-in session (`agenteye login`) and a dashboard URL -(`AGENTEYE_DASHBOARD_URL` or `--base-url`). +can replay through your evaluator. Both go through the `fp` CLI, which needs a +logged-in session (`fp login`) and a dashboard URL (`FP_DASHBOARD_URL` or +`--base-url`). `fp login` records both in its config, so the recipes below read +them from there rather than asking you to repeat yourself. ## Contents @@ -26,19 +27,26 @@ The CLI has no `export` command, but the endpoint takes the same `ae_session` cookie your CLI login already stored (it's permission-gated on `events:read`), so `curl` reaches it with no new credential: +One file holds all three things this needs — the dashboard URL, the session +token, and the active org — so read them together: + ```bash -BASE="${AGENTEYE_DASHBOARD_URL:?set your dashboard URL}" -TOKEN=$(python3 -c "import json,os,pathlib; \ - p=pathlib.Path(os.environ.get('AGENTEYE_HOME') or (pathlib.Path.home()/'.agenteye'))/'cli.json'; \ - print(json.load(open(p))['session_token'])") +eval "$(python3 -c "import json,os,pathlib; \ + p=pathlib.Path(os.environ.get('FP_HOME') or (pathlib.Path.home()/'.failproofai'/'fpcli'))/'cli-auth.json'; \ + c=json.load(open(p)); \ + print(f'BASE={c[\"base_url\"]}\nTOKEN={c[\"session_token\"]}\nORG={c.get(\"org\",\"\")}')")" mkdir -p fixtures -curl -sSf -b "ae_session=$TOKEN" "$BASE/api/sessions/run-001/export" -o fixtures/run-001.json +curl -sSf -b "ae_session=$TOKEN" -H "X-AgentEye-Org: $ORG" \ + "$BASE/api/sessions/run-001/export" -o fixtures/run-001.json ``` -Multi-org logins need the active tenant too — add `-H "X-AgentEye-Org: "` -(`agenteye --json orgs current` tells you the slug). If it 401s, the session -expired: `agenteye login`. If it 403s, the login lacks `events:read`. +The config lives at `~/.failproofai/fpcli/cli-auth.json` (mode 0600), or under +`$FP_HOME/fpcli/` if you moved it. `X-AgentEye-Org` keeps its name on purpose — +the wire headers and the `ae_session` cookie are a contract with the server and +did not rename with the CLI. On a single-org login `$ORG` is harmless to send. +If it 401s, the session expired: `fp login`. If it 403s, the login lacks +`events:read`. Then it round-trips straight into the model, which is the whole point: @@ -55,7 +63,7 @@ what production sends. If `curl` isn't available or the endpoint is unreachable, you can approximate the body from the events feed. Be honest with the user that it's an approximation. -**Build it from `events` alone — never from `agenteye sessions`.** The sessions +**Build it from `events` alone — never from `fp sessions`.** The sessions feed looks like the right source and isn't: - It's anchored on `agent_start`, so a session without one returns **zero rows** @@ -70,7 +78,7 @@ The events feed carries `session_id`, `agent_id`, and `environment` on every row so with `--order asc` everything you need is derivable: ```bash -agenteye --json events --full --session-id run-001 --order asc --all --limit 1000 \ +fp --json events --full --session-id run-001 --order asc --all --limit 1000 \ > /tmp/events.json ``` @@ -109,19 +117,19 @@ You want a good one and a bad one. Discover valid filter values before filtering guessing an env or agent id wastes a round trip: ```bash -agenteye --json list agents # valid agent ids -agenteye --json list envs # valid environments -agenteye --json list score_filters # score keys that already exist -agenteye --json list error_types +fp --json list agents # valid agent ids +fp --json list envs # valid environments +fp --json list score_filters # score keys that already exist +fp --json list error_types ``` | goal | command | |---|---| -| failed runs | `agenteye --json sessions --status error,timeout --since 7d --all --limit 1000` | -| where it hurts | `agenteye --json errors --aggregate --since 7d` | -| recent runs | `agenteye --json sessions --since 24h --all --limit 200` | -| already-scored bad runs | `agenteye --json evals --score helpfulness:..0.5 --since 7d --all --limit 200` | -| which score regressed | `agenteye --json evals --aggregate --since 7d` → `score_stats` | +| failed runs | `fp --json sessions --status error,timeout --since 7d --all --limit 1000` | +| where it hurts | `fp --json errors --aggregate --since 7d` | +| recent runs | `fp --json sessions --since 24h --all --limit 200` | +| already-scored bad runs | `fp --json evals --score helpfulness:..0.5 --since 7d --all --limit 200` | +| which score regressed | `fp --json evals --aggregate --since 7d` → `score_stats` | There is **no latency or duration filter** on any command, and `sessions` has no `--score` flag (score-based discovery goes through `evals`, then take @@ -129,17 +137,17 @@ There is **no latency or duration filter** on any command, and `sessions` has no ordering — drop to SQL: ```bash -agenteye --json query run --sql "SELECT session_id, count() c FROM events GROUP BY session_id ORDER BY c DESC LIMIT 10" +fp --json query run --sql "SELECT session_id, count() c FROM events GROUP BY session_id ORDER BY c DESC LIMIT 10" ``` -Exposed tables: `events`, `evaluations`, `agent_sessions`. `agenteye --json query schema` +Exposed tables: `events`, `evaluations`, `agent_sessions`. `fp --json query schema` shows the layout. ## Reading a session ```bash -agenteye --json events --session-id run-001 --order asc --all --limit 1000 # timeline (no payload) -agenteye --json events --full --session-id run-001 --order asc --all --limit 1000 # with payload +fp --json events --session-id run-001 --order asc --all --limit 1000 # timeline (no payload) +fp --json events --full --session-id run-001 --order asc --all --limit 1000 # with payload ``` The default feed is payload-free and carries a server-computed one-line `summary` @@ -178,20 +186,22 @@ assume one exists just because an example scores it. varies; a field the schema allows may never be populated in their deployment: ```bash -agenteye --json events --full --session-id run-001 --all --limit 1000 \ +fp --json events --full --session-id run-001 --all --limit 1000 \ | jq '[.events[] | select(.event_type=="tool_use") | .payload.tool_name] | unique' ``` ## CLI gotchas -- **Globals go before the command**: `agenteye --json events`, never - `agenteye events --json` (exit 2). +- **Globals go before the command**: `fp --json events`, never + `fp events --json` (exit 2). - **`--all` is capped by `--limit`, which defaults to 50.** A bare `--all` returns 50 rows with `next_cursor: null`, looking complete. Pass `--all --limit 1000`. Server-side the limit clamps at 1000 — a bigger session needs cursor paging. - **`--since` is a closed enum**: `all`, `15m`, `1h`, `6h`, `24h`, `7d`. Anything else is a usage error; use `--from`/`--to` with full RFC3339 (`2026-05-01T00:00:00Z`). -- **Exit codes**: 0 ok · 2 usage · 3 unreachable · 4 not signed in (`agenteye login`, + (`fp guardrails` is the exception — it takes the five relative windows only, no + `all` and no `--from`/`--to`.) +- **Exit codes**: 0 ok · 2 usage · 3 unreachable · 4 not signed in (`fp login`, which you can't complete for them — it needs an emailed code) · 5 missing permission · 6 not found. - `sessions`/`events` filters accept repeated flags and CSV; `evals`/`errors`