M2.1 — Build fair multi-model orchestration (scheduler + pluggable runner + budget)

[GiggleLiu]

Background

New here? Read #9 first — it explains the project and defines every term below.

Issue #2 runs one AI bug-hunting session (one model exploring the library and submitting counterexamples). To build a leaderboard we must run many sessions across many AI models — and the comparison is only fair if every model gets the identical environment, prompt, and budget. This issue builds that orchestrator. It also puts the AI behind a small swappable interface called AgentRunner, so a different agent program (e.g. opencode) can be plugged in later without changing the rest of the system.

Objective

Run N sessions across M models — fairly, in parallel, and resumably after interruption — with the AI behind a swappable AgentRunner.

Interface (Input → Output)

• AgentRunner (the swap point): given (environment, model, per-session budget) → returns (the certificates found, accounting). The default implementation wraps the M1.2 — Build the bug-hunting session: explore → certify → verify, under budget #2 mini-SWE-agent session.
• Scheduler: given (list of models, sessions per model, total budget, per-model budget, parallelism, optional --resume) → writes per-model results files + a checkpoint file + a budget ledger (how much each model spent, how much is left).

Technical recommendations (suggestions)

• Wrap M1.2 — Build the bug-hunting session: explore → certify → verify, under budget #2's logic as MiniSweRunner behind AgentRunner; keep the scheduler and checker free of any mini-SWE-agent-specific imports so another agent can drop in later.
• Save progress to a checkpoint file so --resume can continue where it left off.
• Stop launching new sessions once a budget cap is reached.

Verification (how a reviewer confirms this is done)

Run one acceptance run using a fake AI (FakeRunner — returns a canned certificate, makes no real model calls, so the test is fast and free) over 2 models. It should demonstrate three things:

1. Swappable: the whole pipeline runs end-to-end with the fake AI. (Proves the rest of the system needs zero mini-SWE-agent-specific code.)
2. Resumable: kill the process partway, re-run with --resume → it finishes only the remaining sessions, does not re-charge the finished ones, and produces no duplicates. (Without --resume it would redo everything — that's the contrast.)
3. Budget-fair: with a total budget of $0.20, total spend stays ≤ $0.20, the scheduler stops launching once that's hit (remaining sessions marked "skipped — budget"), and each model is capped equally. (Turn the cap off and it overspends — that's the contrast.)

Dependencies

Depends on #2 (the session) and #3 (the results format).

Out of scope

Plugging in a second real agent like opencode (deferred — we build only the swap point now); the tamper-proof checker and de-duplication (#5).

[Ferrari-72]

Issue #4 Implementation Complete — Fair Multi-Model Orchestration

What Was Built

Commit: 21f7fde

Deliverables

File	Purpose
benchmark/runner.py	AgentRunner ABC (the swap point) + FakeRunner (no API calls) + MiniSweRunner (wraps #2's run_one)
benchmark/scheduler.py	Scheduler: runs N sessions × M models, checkpoint/resume, per-model budget cap, parallelism
benchmark/tests/test_scheduler.py	25 tests covering all three verification scenarios from the issue

Verification (from the issue)

1. Swappable — pipeline runs end-to-end with FakeRunner:

runner = FakeRunner(cost_per_rule=0.01)
s = Scheduler(runner=runner, models=[...], rules=[...], ...)
results = s.run_all()  # no API calls, no pred calls

test_no_mini_swe_import_needed asserts the scheduler source never imports minisweagent or benchmark.run_mini — the rest of the system is fully decoupled.

2. Resumable — kill partway, resume → only remaining sessions run:

• Checkpoint written to JSON after every completed session
• --resume loads checkpoint and skips already-finished rules
• test_resume_skips_finished_rules: pre-seed rule1 for model-a → it is never re-called
• test_no_resume_reruns_everything: contrast — without --resume, rule1 is called again

3. Budget-fair — total spend ≤ cap, excess marked "skipped_budget":

• Optimistic reservation under lock: each worker reserves its limit before running, corrects back on completion → concurrent sessions cannot collectively overspend
• Each model gets an equal share via _per_model_cap(total_budget, n_models)
• test_total_spend_never_exceeds_budget: $0.05 budget, $0.02/rule → _total_spent ≤ 0.05
• test_excess_sessions_marked_skipped_budget: sessions beyond the cap carry result="skipped_budget" and cost=0.0

Test results

benchmark/tests/test_scheduler.py — 25 passed
Full suite: 99 passed, 2 skipped, 0 failed

Design notes

• MiniSweRunner uses a lazy import (from benchmark.run_mini import run_one) so loading scheduler.py never pulls in minisweagent — a second agent (opencode, etc.) can be plugged in with a new AgentRunner subclass and zero scheduler changes.
• Budget enforcement is min(per_rule_budget, remaining_model_cap, remaining_total) computed atomically under threading.Lock, making it safe for parallelism > 1.