From c1d9ea6b878e388ba1eb8b74a02bb06e5a790dfa Mon Sep 17 00:00:00 2001 From: Cursor Agent Date: Fri, 4 Sep 2026 17:31:20 +0000 Subject: [PATCH 1/2] docs: README/CONTRIBUTING/CLAUDE reflect the harness x model benchmark - README: Harnesses section (axis explanation, table of felix/baseline/ opencode/grok-build/template with install lines, matrix examples, how coding-agent harnesses attach over MCP); results table labelled as the pre-1.2 felix-only spread; plan_coherence caveat and quarantine/metadata notes under Scoring; Development shows both CI configurations; Related Repos lists the harness repos - CONTRIBUTING: smoke test means mock LLM too (old 'real LLM smoke' recipe removed), no-felix test recipe, live coding-agent example, 'Testing a harness plugin' section - CLAUDE.md: Harnesses table; Felix-specific sections (agents, CentralPost, conflict resolution, helix) labelled as such; SSE flow via harness.step; provider/model semantics per harness; conventions and CI job list updated Co-authored-by: Jason --- CLAUDE.md | 13 +++++++++++- CONTRIBUTING.md | 46 ++++++++++++++++++++++++++++++---------- README.md | 56 +++++++++++++++++++++++++++++++++++++++++-------- 3 files changed, 94 insertions(+), 21 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index a7f73f9..fcb3ac3 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -201,7 +201,18 @@ python scripts/run_benchmark.py --harness felix --harness baseline --smoke-test python scripts/run_scenario.py crashlanded --harness felix --harness-opt no_think=true --harness-opt parallel=false ``` -## Agents (map to roles, not colonists) +## Harnesses + +| Harness | Where | What | +|---------|-------|------| +| `felix` | in tree, extra `felix` (`src/rle/harness/felix/`) | MapAnalyst + 6 role agents over CentralPost, merged by ActionResolver; helix phases; the original stack. Everything in the next four sections describes this harness only. | +| `baseline` | in tree | Unmanaged colony — paired control for every run. | +| `opencode` / `grok-build` | own repos (`AppSprout-dev/rle-harness-*`) | Coding agents driven via `HeadlessCliHarness`: one prompt per tick, act through the RLE MCP tools, `end_turn`; the ledger of writes is scored. | +| `template` | own repo | Starting point for new harnesses; RLE CI installs it as the plugin-API contract test. | + +Every harness receives the same neutral brief (`rle.harness.brief`: scenario goals, state snapshot, MAP_SUMMARY, action catalog). Anything beyond that — role splits, bootstrap playbook, helix temperature, tool framing — is the harness's own prompt engineering and is part of what is measured. Options are per plugin (`--harness-opt key=value`, validated by its pydantic schema); Felix's are `parallel`, `no_think`, `helix_preset`, `role_timeout_s`, `exclude_agent`, `provider_kwargs`, `visualize`. + +## Felix harness: agents (map to roles, not colonists) | Agent | Domain | Key Actions | |-------|--------|-------------| diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 7a73416..cc6df23 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -55,36 +55,60 @@ LM Studio settings: Flash Attention ON, Context 10000, GPU Offload max, Keep in ### Quick test (no RimWorld needed) ```bash -# Smoke test — tests the full pipeline with fake game state -python scripts/run_benchmark.py --smoke-test --ticks 3 - -# Smoke test with real LLM (needs LM Studio running) -OPENAI_API_KEY=lm-studio python scripts/run_benchmark.py \ - --smoke-test --provider openai \ - --model unsloth/nvidia-nemotron-3-nano-4b \ - --base-url http://localhost:1234/v1 \ - --no-think --ticks 3 +# Smoke test — full pipeline with fake game state and each harness's mock agent +# (no LLM calls; proves plumbing, not colony management) +python scripts/run_benchmark.py --smoke-test --ticks 3 --harness felix --harness baseline + +# Which harnesses are installed and usable here +python scripts/run_benchmark.py --harness list + +# Run the suite as CI does, with and without the felix extra +uv sync --extra dev --extra felix --extra mcp && pytest +UV_PROJECT_ENVIRONMENT=.venv-nofelix uv sync --extra dev --extra mcp && \ + UV_PROJECT_ENVIRONMENT=.venv-nofelix uv run --no-sync pytest +python scripts/check_harness_boundary.py ``` +`--smoke-test` always uses mock LLMs. To exercise a real model against the fake game state, run a real harness with `--ticks` small against a live RIMAPI instead (below). + ### Live game test ```bash # Start RimWorld with RIMAPI mod, load a colony, then: OPENAI_API_KEY=lm-studio python scripts/run_scenario.py crashlanded_survival \ + --harness felix \ --provider openai \ --model unsloth/nvidia-nemotron-3-nano-4b \ --base-url http://localhost:1234/v1 \ --no-think --visualize --ticks 10 + +# Same scenario, a coding-agent harness (needs the plugin + its binary installed) +uv pip install git+https://github.com/AppSprout-dev/rle-harness-opencode +python scripts/run_scenario.py crashlanded_survival --harness opencode --model openai/gpt-4o --ticks 10 ``` +### Testing a harness plugin + +External plugins depend on RLE core and use `rle.testing`: + +```python +from rle.testing import run_harness_smoke + +async def test_smoke() -> None: + report = await run_harness_smoke("your-harness", ticks=3) + assert report.ok +``` + +`run_harness_smoke` builds `plugin.smoke(...)`, drives it through `RLEGameLoop` against `MockRimAPI`, and returns the tick results plus every POST the harness made. Coding-agent plugins return `rle.testing.scripted_agent.ScriptedMcpHarness` from `smoke()` so the MCP round trip is covered without the binary. See [rle-harness-template](https://github.com/AppSprout-dev/rle-harness-template) for the full layout. + ### Docker benchmark (headless, no display) ```bash # Build image (see docker/README.md for prerequisites) docker compose -f docker/docker-compose.yml up -d -# Run benchmark against container -python scripts/run_benchmark.py --docker --runs 4 --output results/docker/ +# Run a harness x model matrix against the container, 4 paired runs each +python scripts/run_benchmark.py --docker --runs 4 --harness felix --harness baseline --output results/docker/ ``` ### OpenRouter (cloud, no local GPU needed) diff --git a/README.md b/README.md index fb7b44a..06a18d1 100644 --- a/README.md +++ b/README.md @@ -34,6 +34,27 @@ python scripts/run_benchmark.py --harness felix --harness baseline --smoke-test Writing a harness: see [docs/harness-plugins.md](docs/harness-plugins.md). Third-party harnesses (OpenCode, Grok Build, ...) live in their own `AppSprout-dev/rle-harness-*` repos and are installed with `pip`, never committed here. +## Harnesses + +The benchmark has two axes. `--model` picks the LLM; `--harness` picks the decision architecture around it. Every harness gets the same scenarios, saves, neutral scenario brief (goals, state, MAP_SUMMARY, action catalog) and scoring, and is paired against the same unmanaged baseline. Leaderboard rows are `harness/model`. + +| Harness | Where | What it is | Install | +|---------|-------|------------|---------| +| `felix` | this repo (extra `felix`) | MapAnalyst + 6 role agents over Felix SDK CentralPost, merged by ActionResolver — the original RLE stack | `uv sync --extra felix` | +| `baseline` | this repo | Unmanaged colony (RimWorld's own pawn AI) — the paired control | built in | +| `opencode` | [rle-harness-opencode](https://github.com/AppSprout-dev/rle-harness-opencode) | [OpenCode](https://opencode.ai) coding agent, one prompt per tick, acting through the RLE MCP tools | `uv pip install git+https://github.com/AppSprout-dev/rle-harness-opencode` | +| `grok-build` | [rle-harness-grok-build](https://github.com/AppSprout-dev/rle-harness-grok-build) | [Grok Build](https://github.com/xai-org/grok-build) coding agent, headless `grok -p` per tick, acting through the RLE MCP tools | `uv pip install git+https://github.com/AppSprout-dev/rle-harness-grok-build` | +| `template` | [rle-harness-template](https://github.com/AppSprout-dev/rle-harness-template) | Copy-me starting point for your own harness | `uv pip install git+https://github.com/AppSprout-dev/rle-harness-template` | + +```bash +python scripts/run_benchmark.py --harness list # installed plugins + availability +python scripts/run_benchmark.py --harness felix --harness opencode --model openai/gpt-4o --runs 4 +python scripts/run_scenario.py crashlanded --harness grok-build --model grok-4.6 --tick-interval 30 +python scripts/run_scenario.py crashlanded --harness felix --harness-opt no_think=true --harness-opt parallel=false +``` + +Coding-agent harnesses attach to RLE over MCP: RLE hosts a RimAPI tool server in-process (`rle.mcp`, extra `mcp`), the agent calls `get_brief`, then write tools (`work_priority`, `blueprint`, ...), then `end_turn`; the writes that reached the game are what gets scored. `--smoke-test` needs none of the binaries — each plugin ships a scripted stand-in that plays the same round trip. + ## The Felix harness: 7 agents | Agent | Domain | Key Actions | @@ -163,11 +184,15 @@ cd ../rimapi-dashboard && bun run start ### Other commands ```bash -# Full benchmark (mock game state, real LLM) -python scripts/run_benchmark.py --smoke-test --ticks 10 +# Smoke test: mock game state + each harness's mock agent (no LLM, no RimWorld) +python scripts/run_benchmark.py --smoke-test --ticks 10 --harness felix --harness baseline + +# Harness matrix against a live game, 4 paired runs each +python scripts/run_benchmark.py --harness felix --harness opencode --runs 4 --output results/matrix/ -# List scenarios +# List scenarios / harnesses python scripts/run_scenario.py --list +python scripts/run_benchmark.py --harness list # Visualize CSV results python scripts/visualize_results.py results/ --all @@ -178,7 +203,9 @@ python scripts/analyze_spread.py --spread-dir results/spread ## Benchmark Results -**11-model spread** — Crashlanded, 10 ticks, seed 42, 2026-06-11. **N=1, content-first — not statistically valid (no confidence intervals). Winners advance to N=4; N=4 is not published.** Ranked by mean composite across the run. Featured numbers live on the [HF card](https://huggingface.co/datasets/AppSprout/rle-benchmarks) and [rle.appsprout.dev](https://rle.appsprout.dev) (same `site_data.json` payload). +**11-model spread, `felix` harness, scoring 1.1** — Crashlanded, 10 ticks, seed 42, 2026-06-11. **N=1, content-first — not statistically valid (no confidence intervals). Winners advance to N=4; N=4 is not published.** Ranked by mean composite across the run. Featured numbers live on the [HF card](https://huggingface.co/datasets/AppSprout/rle-benchmarks) and [rle.appsprout.dev](https://rle.appsprout.dev) (same `site_data.json` payload). + +These rows predate the harness axis and scoring 1.2: they were all produced by the Felix harness and include the since-removed `coordination` / `communication_efficiency` metrics, so they are not comparable to 1.2 runs. The next published spread will be a harness × model matrix at scoring 1.2. | # | Model | Mean | Final | vs baseline | Cost | |---|-------|------|-------|-------------|------| @@ -225,21 +252,32 @@ Measured against a pinned no-agent baseline (4 seeds, mean time-to-end 8.0 days) Both process metrics (`efficiency`, `plan_coherence`) are computed from the writes that actually reached RIMAPI, so any harness is scored the same way, and both return a neutral 0.5 for ticks with no writes so the unmanaged baseline earns no free points. The pre-1.2 `coordination` / `communication_efficiency` metrics were removed because they were ≈1.0 by construction (issue #51). +`plan_coherence` is measured after whatever coordination a harness does internally, so it is a floor a competent harness clears rather than a way to rank harnesses against each other; the colony outcome metrics (86% of the weight) are what separate them. Harness-internal process data (Felix's CentralPost traffic, resolver conflicts, a coding agent's tool-call count) is recorded in the event log as diagnostics, not scored. + +Every run also records `harness`, `harness_options`, `harness_versions`, per-tick step latency and cost, and a `harness_failed` flag when RIMAPI plumbing errors (null-ref cascades, invalid plant defs) occurred; quarantined runs are excluded from leaderboard means. + ## Development ```bash -pytest # Run all tests -ruff check src/ tests/ scripts/ # Lint -mypy src/ # Type check +uv sync --extra dev --extra felix --extra mcp +pytest # Run all tests (Felix-only modules skip without the extra) +ruff check src/ tests/ scripts/ # Lint +mypy src/ # Type check +python scripts/check_harness_boundary.py # felix confined to harness/felix; no third-party harness code in tree ``` +CI runs the suite twice — with and without the `felix` extra — plus a contract job that installs [rle-harness-template](https://github.com/AppSprout-dev/rle-harness-template) from GitHub and checks it appears in `--harness list` and passes `--smoke-test`. + ## Related Repos | Repo | What | Notes | |------|------|-------| -| [felix-agent-sdk](https://github.com/AppSprout-dev/felix-agent-sdk) | Agent framework (LLMAgent, CentralPost, HelixGeometry, providers) | pip dependency | +| [rle-harness-template](https://github.com/AppSprout-dev/rle-harness-template) | Template for a harness plugin | Start here to add a harness; RLE CI installs it as the plugin-API contract test | +| [rle-harness-opencode](https://github.com/AppSprout-dev/rle-harness-opencode) | OpenCode as a harness | `opencode serve` + HTTP API over the RLE MCP server | +| [rle-harness-grok-build](https://github.com/AppSprout-dev/rle-harness-grok-build) | Grok Build as a harness | headless `grok -p`, session resumed per tick, over the RLE MCP server | +| [felix-agent-sdk](https://github.com/AppSprout-dev/felix-agent-sdk) | Agent framework behind the `felix` harness (LLMAgent, CentralPost, HelixGeometry, providers) | optional extra `felix` | | [RIMAPI](https://github.com/IlyaChichkov/RIMAPI) | C# RimWorld mod (REST API + SSE) | We contribute upstream. [Our fork](https://github.com/AppSprout-dev/RIMAPI) has `rle-testing` branch. | -| [rimapi-dashboard](https://github.com/AppSprout-dev/rimapi-dashboard) | React dashboard with RLE widgets | Runs on :3000, reads tick data from :9000 | +| [rimapi-dashboard](https://github.com/AppSprout-dev/rimapi-dashboard) | React dashboard with RLE widgets | Runs on :3000, reads tick data from :9000 (`latest_tick.json` now carries `harness` + `extras`) | ## License From 80592f987565220f5628fb03bace59ea9da86a15 Mon Sep 17 00:00:00 2001 From: Cursor Agent Date: Fri, 4 Sep 2026 17:31:57 +0000 Subject: [PATCH 2/2] docs(CLAUDE): CentralPost/SSE/helix sections scoped to the Felix harness Co-authored-by: Jason --- CLAUDE.md | 53 +++++++++++++++++++++++++++++------------------------ 1 file changed, 29 insertions(+), 24 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index fcb3ac3..7c033d8 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -226,7 +226,7 @@ Every harness receives the same neutral brief (`rle.harness.brief`: scenario goa ### MapAnalyst + Spatial Awareness -MapAnalyst runs before the other 6 agents each tick. It reads terrain data from RIMAPI (`/api/v1/map/terrain`) and produces a deterministic spatial analysis: +The deterministic spatial analysis is core (`rle.harness.brief.build_map_summary`, from RIMAPI `/api/v1/map/terrain`) and every harness gets it in its brief. In the Felix harness, MapAnalyst runs before the other 6 agents each tick and narrates it: - **MAP_SUMMARY** — compact ~500 token text injected into every agent's context - **SHELTER_SITE** — verified 7x7 rectangle on solid ground near colony center @@ -236,9 +236,9 @@ MapAnalyst runs before the other 6 agents each tick. It reads terrain data from All role agents are told: "MUST use coordinates from MAP_SUMMARY, do NOT invent coordinates." -### Bootstrap Playbook (day < 3) +### Bootstrap Playbook (day < 3) — Felix only -Tick-specific priorities injected into all agents: +Tick-specific priorities injected into all Felix agents (other harnesses get no playbook; that is their harness's problem to solve): - Tick 1: Stockpile + work priorities + growing zone (Plant_Rice) - Tick 2: 5x5 shelter walls + door + 3 beds (WoodLog) - Tick 3: Campfire/stove + research bench + research target @@ -257,34 +257,36 @@ Tick-specific priorities injected into all agents: The 5 advanced saves (first_winter, toxic_fallout, raid_defense, plague_response, ship_launch) are built via `scripts/create_scenario_saves.py` — declarative RIMAPI calls that load the base crashlanded save, spawn items/pawns, trigger incidents, and write each scenario. Requires RimWorld running with a map loaded. Saves land in AppData and are mirrored to `docker/saves/`. Use `--only ` for a single rebuild or `--difficulty-only` for offline byte-patching. -## CentralPost Hub-Spoke Communication +## Felix harness: CentralPost Hub-Spoke Communication -Agents communicate through Felix SDK's CentralPost, not through the orchestrator: +Felix agents communicate through Felix SDK's CentralPost, owned by `FelixHarness` (the loop knows nothing about it): - **Before deliberation**: `process_all_messages()` routes previous tick's messages to agent spoke inbound queues. Agents read via `_get_spoke_context()`. - **MapAnalyst first**: Deliberates, sends TASK_COMPLETE with spatial analysis. Messages routed immediately so role agents see it. - **After deliberation**: Each role agent sends `TASK_COMPLETE` with role, summary, confidence, action types. -- **After scoring**: Hub broadcasts `STATUS_UPDATE` with composite score + all 10 metrics. +- **After scoring** (`on_tick_end`): Hub broadcasts `STATUS_UPDATE` with the composite score + all metrics, and a `STATUS_UPDATE` listing last tick's failed writes ("DO NOT REPEAT"). - **On phase change**: Hub broadcasts `PHASE_ANNOUNCE` when macro_time crosses 0.4 (exploration→analysis) or 0.7 (analysis→synthesis). +Message and conflict counts are emitted on the `CONFLICT` event as diagnostics; since scoring 1.2 they do not feed the composite. + ## SSE Events RimAPISSEClient connects to `/api/v1/events` and buffers real-time game events (raids, deaths, mental breaks). Each tick: 1. GameStateManager drains SSE buffer → `pending_events` -2. Game loop injects events into all agents via `set_pending_events()` -3. Each agent's `filter_game_state()` includes role-relevant events as `"recent_events"` +2. The loop passes them to `harness.step(state, tick, macro_time, events)`; the brief carries the most recent ones as `recent_events` +3. Felix: `FelixHarness` injects them into every agent via `set_pending_events()`, and each agent's `filter_game_state()` includes role-relevant events as `"recent_events"` -## Conflict Resolution (4 rules) +## Felix harness: Conflict Resolution (4 rules) 1. Emergency roles promoted during crises (DefenseCommander during raids, MedicalOfficer during plague) 2. Same-pawn conflicts: lowest action priority number wins 3. Role priority tiebreak (ResourceManager=3, DefenseCommander=3, MedicalOfficer=4, MapAnalyst=10, others=5) 4. Final tiebreak: highest plan confidence score -## Helix Phase Adaptation +## Felix harness: Helix Phase Adaptation -Macro helix: `t = min(1.0, game_day / expected_duration_days)` drives agent behavior: +`macro_time = min(1.0, game_day / expected_duration_days)` is computed by the loop and handed to every harness. Felix maps it onto helix phases that drive agent temperature and prompt directives: - **Exploration** (t < 0.4): High temperature, diverse strategies - **Analysis** (0.4 <= t < 0.7): Medium temp, evaluate trade-offs - **Synthesis** (t >= 0.7): Low temperature, decisive actions @@ -320,28 +322,31 @@ Scenarios can override weights. TimeSeriesRecorder exports per-tick CSV. Each defines victory/failure conditions, scoring weight overrides, and max ticks. -## Provider Configuration +## Provider / Model Configuration -Provider-agnostic via felix-agent-sdk. CLI flags: `--provider`, `--model`, `--base-url`. +`--provider`, `--model`, `--base-url` (and the matching `.env` fields) are plain strings on `RLEConfig`; the selected harness interprets them. Model naming therefore follows the harness: -| Provider | Model | Command | -|----------|-------|---------| -| LM Studio (local) | Nemotron Nano 4B | `--provider openai --model unsloth/nvidia-nemotron-3-nano-4b --base-url http://localhost:1234/v1` | -| OpenRouter (cloud) | Nemotron 30B | `OPENAI_API_KEY= --provider openai --model nvidia/nemotron-3-nano-30b-a3b --base-url https://openrouter.ai/api/v1` | -| Anthropic | Claude | `--provider anthropic --model claude-sonnet-4-5` | -| OpenAI | GPT-4o | `--provider openai --model gpt-4o` | +| Harness | Provider/model meaning | Example | +|---------|------------------------|---------| +| `felix` | Felix SDK provider registry (`anthropic`, `openai`, `local`, `claude-code`) in `rle/harness/felix/provider_factory.py` | `--provider openai --model unsloth/nvidia-nemotron-3-nano-4b --base-url http://localhost:1234/v1` | +| `felix` | | `OPENAI_API_KEY= --provider openai --model nvidia/nemotron-3-nano-30b-a3b --base-url https://openrouter.ai/api/v1` | +| `felix` | | `--provider anthropic --model claude-sonnet-4-5` | +| `opencode` | OpenCode's `provider/model` ids, credentials from `opencode auth` | `--model openai/gpt-4o`, `--model anthropic/claude-sonnet-4-5` | +| `grok-build` | Grok Build's `-m` model id, auth via `XAI_API_KEY` or cached login | `--model grok-4.6` | -Use `--no-think` for thinking models (Qwen3.5, Nemotron) — injects `` assistant prefix to skip reasoning chain. +Felix-only: `--no-think` (or `--harness-opt no_think=true`) injects a `` assistant prefill so thinking models (Qwen3.5, Nemotron) skip the reasoning chain. Leaderboard rows are `harness/model`, so the same model under two harnesses is two rows. ## Conventions - Python 3.14+, `uv` for package management, `hatchling` build backend - Async-first (httpx AsyncClient, async game loop) -- Parallel-first: MapAnalyst runs first (sequential), then 6 role agents deliberate concurrently via `asyncio.to_thread` + `asyncio.gather` (`--sequential` to disable) +- Core is framework-free: `felix_agent_sdk` is imported only under `src/rle/harness/felix/`; everything else runs with the `felix` extra uninstalled (CI `test-no-felix` job + `scripts/check_harness_boundary.py`). Third-party harnesses never live in this tree (ADR-004). +- Harness-agnostic scoring: metrics read the executed write stream, never a harness's internal messaging +- Felix harness: MapAnalyst runs first (sequential), then 6 role agents deliberate concurrently via `asyncio.gather` (`--sequential` / `--harness-opt parallel=false` to disable) - Pydantic v2 models with frozen=True for game state and results -- mypy strict mode — all code must pass `mypy src/` with `strict = true` +- mypy strict mode — all code must pass `mypy src/` with `strict = true`; `py.typed` is shipped so external harness packages type-check against RLE - No scipy/numpy — stdlib only for statistics (random, math). See ADR-003 for rationale -- Felix Agent SDK for providers, agents, helix geometry, CentralPost communication +- Felix Agent SDK (inside the `felix` harness only) for providers, agents, helix geometry, CentralPost communication - JSON repair + parse retry for LLM output resilience (strips think tags, trailing commas, extracts first JSON object) - Real RIMAPI data via state adapters + deterministic terrain analysis - Tests use pytest-asyncio with auto mode @@ -350,7 +355,7 @@ Use `--no-think` for thinking models (Qwen3.5, Nemotron) — injects `` GitHub Actions workflows in `.github/workflows/`: -- **ci.yml** — On every push/PR: ruff lint, mypy strict, pytest, smoke-test +- **ci.yml** — On every push/PR: ruff lint, mypy strict, harness boundary check, pytest (with `felix`+`mcp` extras), `test-no-felix` (core + `mcp` only: suite, `--harness list`, baseline smoke), `smoke-test` (felix + baseline matrix), `external-plugin-contract` (installs `rle-harness-template` from GitHub, asserts it lists and passes smoke) - **benchmark.yml** — Manual dispatch + weekly schedule: Docker benchmark template (requires self-hosted runner with game files) ## Package Structure