Multi-agent fraud investigation and orchestration. A supervisor agent routes each flagged case to four independent specialist agents, aggregates their findings into a combined risk and an explicit confidence, and then either resolves the case autonomously or escalates it to a human reviewer when the panel is unsure or its specialists conflict. Every action any agent takes — every tool call, every finding, every verdict, every human decision — is appended to a shared Redis-backed event log that is the system's only source of truth.
Every transaction, device, account, IP, merchant, network link and precedent case in this repository is SYNTHETIC, generated by
eval/case_bank/build_cases.pyfrom a fixed seed. Nothing here is derived from, sampled from, or representative of real financial records, real customers, or any real institution's fraud data. The KYC, device-fingerprint and IP-intelligence tool results are labelled mocks that stand in for vendor integrations described in Known Limitations.
┌──────────────────────────────────────────┐
POST /api/cases ───▶ │ SUPERVISOR AGENT │
│ routes → aggregates → decides │
└───────┬──────────────────────────┬───────┘
│ asyncio.gather (parallel)│
┌───────────────────┼─────────────┬────────────┴──────┐
▼ ▼ ▼ ▼
┌────────────────┐ ┌─────────────────┐ ┌──────────────┐ ┌──────────────┐
│ transaction │ │ identity │ │ network │ │ historical │
│ pattern │ │ signal │ │ analysis │ │ case │
│ velocity, │ │ device f/print, │ │ NetworkX │ │ pgvector │
│ z-score, MCC │ │ geo, KYC │ │ ring score │ │ precedent │
└───────┬────────┘ └────────┬────────┘ └──────┬───────┘ └──────┬───────┘
│ │ │ │
└───────────────────┴────────┬────────┴────────────────┘
▼
┌───────────────────────────────────────────────┐
│ SHARED EVENT LOG (Redis Streams) │
│ append-only · XADD · single source of truth │
│ tool_call │ finding │ verdict │ escalation │
│ │ human_decision │
└────┬─────────────────────┬────────────────┬───┘
│ │ │
▼ ▼ ▼
┌────────────────┐ ┌──────────────────┐ ┌──────────────┐
│ HITL review │ │ WS /ws/cases/… │ │ GET …/trace │
│ queue │ │ live dashboard │ │ audit trail │
└───────┬────────┘ └──────────────────┘ └──────────────┘
│ human approves / declines / asks for detail
└──────────▶ writes `human_decision` back to the log
Nothing reads around the log. SentryMesh.case_status() reconstructs a case's entire state — status,
verdict, findings, errors, human decision — from the log alone, and ReviewQueue.rebuild_from_log()
regenerates the whole review queue from it. Both are covered by tests, which is what turns "the event
log is the source of truth" from a claim into a property.
| Layer | Module | What it does |
|---|---|---|
| 1. Shared event log | sentrymesh/event_log.py |
Append-only Redis stream (XADD), with a derived per-case index written in the same atomic Lua call |
| 2. Specialists | sentrymesh/agents/*_agent.py |
Four independent LLM tool-use loops, one per signal category |
| 3. Supervisor | sentrymesh/agents/supervisor_agent.py |
Fan-out, disagreement measurement, noisy-OR aggregation, threshold decision |
| 4. HITL review | sentrymesh/hitl.py |
Redis list (FIFO) + hash (O(1) lookup) work queue; decisions written back to the log |
| 5. API + dashboard | backend/main.py, frontend/ |
FastAPI + WebSocket trace stream; React/Vite/Tailwind dashboard |
Why a Redis stream and not a list. Server-assigned <ms>-<seq> IDs give a total order that
survives four specialists appending concurrently from one asyncio.gather; XRANGE gives
"everything after the last event the dashboard saw" as a primitive; consumer groups are available
later without a rewrite. The per-case index is a derived cache, appended in the same Lua script as
the XADD so it cannot drift, and rebuildable from the stream alone.
Why every specialist runs on every case. Selective routing would be cheaper, and it was rejected: if a category can be scored without ever being looked at, you cannot distinguish "the network agent found nothing" from "the network agent was never asked", and the eval's per-category false-positive analysis becomes unattributable. Four LLM calls per case is the price of every verdict resting on the same four-signal evidence base.
Why the LLM aggregates but code decides. The supervisor's model call reasons about why
specialists conflict — "a shared-device ring with clean identity signals suggests a mule account
rather than a compromised one" — and returns a combined risk and confidence. Turning those numbers
into an action is decide(), a pure function over (risk, confidence, disagreement, findings, thresholds). An LLM is a good place for the interpretation and a bad place for a threshold that
compliance will one day have to explain in writing. Every threshold lives in sentrymesh/config.py.
The obvious implementation — flag a wide spread between the four risk scores — is wrong here, and measurably so: it escalated 21 of 23 seeded cases. The specialists examine orthogonal categories, so a low score means "my category shows nothing", not "this case is fine". Identity at 80 with network at 10 is a device-based takeover with no ring: a coherent story, not a contradiction.
Three corrections came out of that, all visible in the git history:
- Abstention is not dissent. A specialist reporting confidence below
abstention_confidence(0.40) is saying its category is uninformative. It is excluded from the disagreement statistics and the aggregate, and costs a little panel confidence — where a failed specialist costs a lot, because we cannot know what it would have found. - Significance rests on corroboration, not spread. A case escalates when exactly one specialist
claims elevated risk with nothing supporting it (one signal can be a false positive; two
independent ones rarely are), when precedent contradicts the live panel by ≥45 points, or when a
specialist failed.
pairwise_spreadandrisk_stdevare still computed and shown on the trace as diagnostics — they just no longer gate the decision. - Aggregation is noisy-OR, not a mean. Averaging let three categories that found nothing cancel
one conclusive red flag. Evidence above a neutral floor now combines as
1 - Π(1 - pᵢ).
Requires Python 3.10+, Node 20+, and (optionally) Redis and PostgreSQL+pgvector.
git clone <this repo> && cd sentrymesh
cp .env.example .env # optional; every value has a working default
python -m pip install -r requirements.txt
python eval/case_bank/build_cases.py # regenerate the seeded bank (deterministic)
python -m pytest tests/ -q # 46 tests
python eval/run_harness.py # writes eval/results/harness_report.{json,md}
uvicorn backend.main:app --reload # API on :8000
cd frontend && npm install && npm run dev # dashboard on :5173Or the whole stack, with real Redis and pgvector:
ANTHROPIC_API_KEY=sk-... docker compose up --build
# dashboard http://localhost:5173 · API http://localhost:8000/docsSentryMesh runs on three interchangeable backends. Agent code is identical across all three — the
same tool schemas, the same terminal submit_finding, the same event-log writes.
1. Any OpenAI-compatible provider (free tiers available). "OpenAI-compatible" is a request
format, not the company — none of these is OpenAI and none needs an OpenAI key. One adapter serves
all of them, so switching provider is two lines of .env:
# Groq — free key at console.groq.com
SENTRYMESH_LLM_BASE_URL=https://api.groq.com/openai/v1
SENTRYMESH_LLM_API_KEY=gsk_...
SENTRYMESH_MODEL=llama-3.3-70b-versatile
# Google Gemini — free key at aistudio.google.com
# SENTRYMESH_LLM_BASE_URL=https://generativelanguage.googleapis.com/v1beta/openai
# SENTRYMESH_MODEL=gemini-2.0-flash
# Ollama — fully local, no key, no account
# SENTRYMESH_LLM_BASE_URL=http://localhost:11434/v1
# SENTRYMESH_MODEL=qwen2.5:14bEvery specialist ends its turn by calling submit_finding, so the model must support tool
calling. One that doesn't fails with error_class="agent_protocol_error" — which the harness
reports separately from wrong verdicts, so a bad model choice is visible rather than quietly
degrading the numbers.
On a free tier metered per minute, four specialists starting at once can exhaust a minute's token
budget in a single burst. SENTRYMESH_MAX_CONCURRENT_SPECIALISTS=2 trades wall-clock for staying
under the limit; every specialist still runs on every case, and no finding changes.
2. Anthropic. Set ANTHROPIC_API_KEY. Roughly $1 per full 23-case harness run on
claude-haiku-4-5.
3. Nothing. With no provider configured, SentryMesh runs the offline deterministic reasoner:
it still executes every specialist's real tools against the real case data and still writes every
tool call to the event log, but the judgement comes from each agent's explicit scoring function
instead of a model. Labelled in every event payload, on the dashboard's top bar, and in the harness
report as run_mode: "offline_deterministic" — because numbers produced that way measure the
scoring functions, not agent reasoning, and must never be presented as the latter.
The tests always pin backend 3 regardless of your .env (see tests/conftest.py) — a test suite
that calls a live provider is slow, costs money, and turns assertions about verdicts into assertions
about what a model felt like saying. The eval harness is where a real provider belongs.
The OpenAI-compatible adapter was verified end to end against Groq on real cases. It works: the supervisor fans out, all four specialists run genuine tool-use loops, and the verdicts are right.
SM-001 auto_decline risk 88.5 conf 0.88 (ground truth: fraud)
transaction_pattern risk 85 2 tool calls
identity_signal risk 80 3 tool calls
network_analysis risk 65 2 tool calls
historical_case risk 90 2 tool calls
A full 23-case run does not fit in Groq's free tier, and the harness recorded exactly why. Run
aad848 (openai/gpt-oss-20b, preserved at eval/results/harness_report_live_groq_partial.json):
| Cases that completed cleanly | 3 of 23 (SM-001, SM-003, SM-007) |
llm_rate_limit failures |
78 |
agent_protocol_error failures |
5 |
| Genuinely wrong auto-decisions | 1 |
Groq's free tier caps a model at 100,000 tokens/day. One case costs roughly 14K tokens (5 agents × a multi-turn tool loop, where each turn resends the accumulated tool results), so the whole bank needs ~320K tokens — over three days of free quota for a single model. The run died about seven cases in.
Two things this makes concrete rather than theoretical:
- The error classifier earned its place. It filed 80 infrastructure failures separately from the 1 wrong decision, so the report shows a quota wall as a quota wall. Had those been merged into one "failed" bucket — the thing the harness spec forbids — this run would read as catastrophic reasoning failure instead of an exhausted free tier. The distinction is the difference between "the model is bad" and "buy more tokens."
- Small models really do fumble strict tool schemas.
llama-3.1-8b-instantandqwen3.6-27bboth failed tool-call validation outright onsubmit_findingand could not be used at all.llama-3.3-70b-versatileandopenai/gpt-oss-20bhandled it. (openai/gpt-oss-20bis an open-weights model on Groq's hardware — not OpenAI's API, and no OpenAI key is involved.)
Therefore the canonical results below are the offline deterministic run, which is the only complete 23-case run available. They measure the scoring functions and the orchestration, not LLM reasoning. To produce complete live numbers you need a paid tier, a provider with a larger daily budget, or a run spread across several days.
Run dd7076, 2026-08-08, 23 synthetic cases (14 fraud / 7 legitimate / 2 ambiguous), 259 ms wall
clock, run_mode: offline_deterministic, event log and review queue on real Redis, retrieval on the
local cosine index.
Reproduce with python eval/run_harness.py. Full output: eval/results/harness_report.md.
An escalated case is not a prediction, but it is not free either. Both views are reported.
| View | Scored | Precision | Recall | F1 | False-positive rate | Accuracy | TP/FP/TN/FN |
|---|---|---|---|---|---|---|---|
auto_only — escalations excluded |
9 | 1.000 | 1.000 | 1.000 | 0.0% | 100% | 5/0/4/0 |
escalation_as_flag — escalation counts as a flag |
21 | 0.824 | 1.000 | 0.903 | 42.9% | 85.7% | 14/3/4/0 |
The first row is not the result. The gap between the rows is the result. auto_only says the
system never once committed to a wrong answer. escalation_as_flag says that on 3 of 7 legitimate
cases a customer's payment was still held pending a human. A report that printed only the first row
would be lying by omission, which is why the harness computes both and the dashboard shows both.
| Category | Cases | Scored | P | R | F1 | FPR | Decisions |
|---|---|---|---|---|---|---|---|
card_testing |
4 | 2 | 1.000 | 1.000 | 1.000 | — | 2 decline, 2 escalate |
account_takeover |
5 | 2 | 1.000 | 1.000 | 1.000 | — | 2 decline, 3 escalate |
synthetic_identity |
3 | 0 | n/a | n/a | n/a | n/a | 3 escalate |
money_muling |
4 | 1 | 1.000 | 1.000 | 1.000 | — | 1 decline, 3 escalate |
legitimate |
7 | 4 | n/a | n/a | n/a | 0.0% | 4 approve, 3 escalate |
| Difficulty | Cases | Scored | P | R | F1 | FPR (auto) | FPR (as flag) | Decisions |
|---|---|---|---|---|---|---|---|---|
| easy | 4 | 4 | 1.000 | 1.000 | 1.000 | 0.0% | 0.0% | 3 decline, 1 approve |
| medium | 11 | 4 | 1.000 | 1.000 | 1.000 | 0.0% | 50.0% | 2 decline, 2 approve, 7 escalate |
| hard | 8 | 1 | n/a | n/a | n/a | 0.0% | 50.0% | 1 approve, 7 escalate |
| Auto-resolved | 9 / 23 |
| Escalated to a human | 14 / 23 (61%) |
| Ambiguous cases escalated | 2 / 2 (100%) |
| Decisive cases escalated | 12 / 21 (57%) |
| Escalation lift on ambiguity | 1.75× |
Infrastructure failures, agent-protocol failures and genuinely wrong decisions are counted separately and never summed into one "failed" bucket — an API rate limit and a wrong verdict are different findings about different parts of the system.
| Bucket | Count |
|---|---|
| Infrastructure failures (rate limit, timeout, auth, 5xx) | 0 |
| Agent protocol failures (bad tool call, turn limit, tool error) | 0 |
| Wrong auto-decisions | 0 |
| Decisive cases the system refused to commit to | 12 |
Modelled on the treatment PatchPilot's README gave its own 100% hard-tier result. Seven slices came
back at ≥95% accuracy, and the harness flags every one of them as suspect rather than reporting them
as wins. This section is generated from the run, not written by hand; eval/scrutiny.py produces it
and the dashboard renders it. Everything below is quoted from the report for run dd7076.
1. auto_only P/R/F1 = 1.000 is a selection effect, not a result. The system escalated 61% of
cases. Perfect precision on the 9 it chose to answer is partly the system grading its own homework:
it picked which questions to be marked on. difficulty=hard scored 1 of 8 cases and reported 100%
accuracy on that single case. category=synthetic_identity scored zero of 3 — the whole category
was escalated, so it has no measured accuracy at all. A reader skimming the category table would see
three 1.000s and miss that the hardest category produced no measurement whatsoever.
2. The multi-agent architecture is not earning its cost on most cases. The harness runs an
ablation: it re-runs the supervisor's aggregation and decision functions over a reduced panel
using findings already in the event log — no extra model calls, so it measures the decision logic
exactly as it ran. Result: of 9 auto-decisions, only 3 survive removal of their loudest specialist;
6 collapse to escalate.
| Case | Decision | Remove | Becomes |
|---|---|---|---|
| SM-001 | auto_decline | identity_signal (100) | escalate |
| SM-002 | auto_decline | identity_signal (75) | escalate |
| SM-005 | auto_decline | identity_signal (80) | escalate |
| SM-006 | auto_decline | identity_signal (70) | escalate |
| SM-012 | auto_decline | network_analysis (90) | escalate |
| SM-016 | auto_approve | transaction_pattern (26) | escalate |
A multi-signal share of 33% is the most important number in this report. On two thirds of its
auto-decisions, one specialist plus the threshold logic would have produced the same answer, and the
other three specialists cost four LLM calls for nothing. Four of those six collapse on
identity_signal specifically. That is a concrete, falsifiable weakness in the architecture, and it
would not be visible from any accuracy metric.
3. The retrieval specialist has a contamination advantage it would not have in production.
eval/precedent_cases.json and eval/case_bank/ were written by the same person, and retrieval is
lexical hashed TF-IDF — so precedent summaries share vocabulary with the cases they "predict", and
the top match for nearly every case is its authored twin. 3 auto-decisions (SM-005, SM-006, SM-012)
change when historical_case is removed. Against a real case archive, with real vocabulary drift
and no authored correspondence, those hits would not land. Any per-category strength tracing back to
this specialist should be discounted to roughly nothing.
4. Sample size. 23 cases, 21 with a decisive ground truth, per-category slices of 2–7. Confidence
intervals at that size swallow most of the differences between categories. money_muling reports
1.000 on one scored case. None of these numbers would survive being quoted as a benchmark.
5. The run mode changes what every number means. No ANTHROPIC_API_KEY was set, so these figures
measure the deterministic scoring functions, not agent reasoning. The orchestration, event log,
disagreement detection and thresholds are exercised exactly as they would be with a model — but this
is not evidence about LLM quality and must not be presented as such.
6. What the harness cannot see at all. The case bank was written by the same person who wrote the detectors, so it encodes the same assumptions about what fraud looks like. Nothing here tests the patterns neither the author nor the agents thought of, and that blind spot is where a real fraud system's losses actually come from. Adversarial cases authored by someone else would be the single highest-value addition to this eval.
What genuinely holds up. Zero wrong auto-decisions and zero infrastructure failures, so the
metrics are measured on complete evidence. Both deliberately-ambiguous cases escalated, at 1.75× the
rate of decisive ones — escalation does track ambiguity, though 57% of decisive cases escalate too,
so that lift is weaker evidence of judgement than it first looks. And the difficulty=hard slice's
single auto-decision does survive ablation with all four specialists contributing.
Specific technical gaps, not hedges.
Escalation rate is 61%, which is not deployable. A team reviewing 14 of every 23 flagged payments
by hand would not buy this. The thresholds in config.py could be loosened to push it down, but the
error rate that is currently hidden behind escalation would then become visible — which is the point
of quoting escalation_as_flag alongside auto_only.
Specialist calls are genuinely parallel; the harness is not. The four specialists run under
asyncio.gather, so with the Anthropic backend that is four in-flight HTTP requests, not four
sequential ones. The eval harness deliberately runs cases one at a time so that per-case latency and
the event ordering in each trace stay interpretable; running 23 cases concurrently would be faster
and would make the traces harder to read.
Embeddings: Dense semantic search with TF-IDF fallback. sentrymesh/retrieval.py now includes DenseSemanticIndex powered by sentence-transformers (all-MiniLM-L6-v2), while retaining the deterministic hashed TF-IDF index (LocalCosineIndex) for offline and zero-dependency runs.
pgvector is optional and silently downgrades. Without a reachable DATABASE_URL, retrieval falls
back to an in-process NumPy cosine index over the same corpus. Numerically equivalent at 22
precedents; not equivalent at 22 million. SENTRYMESH_REQUIRE_PGVECTOR=1 makes the downgrade a hard
error, and docker-compose.yml sets it.
The Redis downgrade is a footgun, now defused rather than removed. REDIS_URL defaults to
redis://127.0.0.1:6379/0, not localhost, because on a dual-stack host localhost resolves to
::1 first — and a Redis listening only on IPv4 makes the asyncio client spend its entire connect
timeout on the dead IPv6 attempt. With the original 2-second timeout that read as "Redis is down" and
the event log silently fell back to in-memory while the sync client connected fine, which is how a
harness run ended up reporting event_log: in_memory on a machine with a healthy Redis. The timeout
is now 10s, and EventLog.downgrade_reason is surfaced through /health and the harness report so
the fallback can never again be silent. It is still a fallback: SENTRYMESH_REQUIRE_REDIS=1 is the
only setting that makes it an error, and compose sets it.
The review queue rebuild is fully durable. ReviewQueue.rebuild_from_log() reconstructs the work list
from the event log alone, including specialist findings, risk scores, confidence, and metadata, making recovery
from Redis loss seamless.
WebSocket streaming with instant event notifications. EventLog.follow() combines an asyncio.Event
notifier on append() with cursor-based stream tailing, providing immediate sub-millisecond push without
unnecessary polling latency.
Context & token trimming is implemented. get_transaction_history returns concise pre-aggregated
statistical summaries and capped sample records; intermediate tool payloads exceeding 2,500 characters are
compacted during multi-turn loops, drastically reducing token consumption per investigation.
Token usage & cost tracking. LLMRunResult and Verdict now aggregate input_tokens, output_tokens,
and estimated dollar costs (cost_usd) directly into the shared event stream and evaluation payloads.
Adaptive routing & Tabular ML scoring. Supervisor supports configurable adaptive dispatching
(SENTRYMESH_ADAPTIVE_ROSTER) to skip unneeded specialists on simple cases, and TransactionPatternAgent
includes a composite tabular ML anomaly scoring tool (get_ml_anomaly_score).
KYC, device fingerprinting and IP intelligence are labelled mocks. They read fields out of the synthetic case file. A real integration means a KYC provider (Onfido/Persona-class: document verification, SSN/identity-graph checks, ongoing watchlist screening), a device-intelligence vendor (fingerprint reuse graphs, emulator and automation detection), and an IP-intelligence feed (ASN classification, proxy/hosting/residential-proxy detection). Each is an async call with its own latency, failure mode, and cost per lookup.
Merchant-category risk weights are a constant in the repo. ELEVATED_RISK_CATEGORIES should come
from the card scheme's MCC risk table, refreshed per issuer. As written, they are one person's
intuition about which categories are risky.
No authentication. The dashboard's "human reviewer" is a text field. Reviewer identity is written to the event log unverified, so the audit trail records what was claimed, not who did it. Real HITL needs SSO, per-reviewer authorisation, and a reviewer identity the log can actually trust.
Ambiguous ground truth is scored on escalation only. The 2 ambiguous cases are excluded from the precision/recall matrices, because forcing them in would mean inventing a right answer. They are judged solely on whether the system declined to commit — a weak test that 2 cases cannot carry.
Single run, no variance. Every number here comes from one harness run. With a live provider the specialists are non-deterministic, so a serious evaluation needs n runs with variance reported. The harness supports repeated runs but does not aggregate across them.
| Method | Path | Purpose |
|---|---|---|
POST |
/api/cases |
Submit a case; kicks off the supervisor + specialist run (202) |
GET |
/api/cases |
All cases known to the event log |
GET |
/api/cases/{id} |
Case status + verdict, reconstructed from the log |
GET |
/api/cases/{id}/trace |
Full reconstructed event-log trace |
WS |
/ws/cases/{id} |
Live trace stream (replays history, then tails) |
GET |
/api/review-queue |
Cases pending human review |
POST |
/api/review-queue/{id} |
Human decision: approve / decline / request_detail |
POST |
/api/review-queue/_rebuild |
Rebuild the queue from the event log alone |
GET |
/api/eval/report |
Latest harness_report.json |
GET |
/api/eval/report/{id} |
One seeded case's result + its full trace |
GET |
/api/case-bank |
The seeded synthetic bank (eval pages only) |
GET |
/health |
Status + which backends this process is actually running on |
- Live Investigation — pick a seeded case, watch the WebSocket render each specialist's tool calls and reasoning and the supervisor's aggregation as they happen.
- Review Queue — escalated cases with every specialist's finding summarised, and approve / decline / request-more-detail actions. "Request more detail" genuinely re-runs the named specialist and appends a new finding to the trace.
- Eval Leaderboard — per-category precision/recall tables, confusion matrix, ablation study, and the honesty writeup rendered in the UI rather than buried in a JSON file.
- Case Detail — the full audit trail for any case, first tool call through final human decision.
python -m pytest tests/ -q # 49 passedtest_event_log.py covers append ordering under concurrent writers, per-case trace reconstruction,
cursor reads, and index rebuild-from-stream. test_supervisor.py covers the abstention/dissent
split, each disagreement rule, noisy-OR aggregation properties, and every branch of decide() — with
the LLM mocked, because the harness is the end-to-end test. test_api.py covers status codes, the
full review round trip, the specialist re-run, and queue reconstruction from the log.
Project by Ayush Verma — ayushv3533e@gmail.com