M1.3 — Emit schema-valid results and render certificates on the live leaderboard

[GiggleLiu]

Background

New here? Read #9 first — it explains the project and defines every term below.

When a bug-hunting session (#2) finishes, it has a set of verified bugs — each a counterexample certificate (a JSON describing one bug) — plus cost/token accounting. This issue turns that into (a) a saved results file in a fixed, checkable format and (b) a leaderboard web page that shows real numbers and lets a visitor inspect each bug. Right now the leaderboard displays hardcoded fake sample data (DeepSeek, GPT-4o, etc. with made-up numbers); that must be replaced with real data loaded from the results files.

Objective

Save each run as one format-checked JSON file, build the index file the website reads, and make the leaderboard render each bug as a readable, re-checkable counterexample — replacing today's mock data.

Interface (Input → Output)

• Input: the per-bug results of a model run (rule, verdict, certificate, cost/tokens).
• Output: results/<model>-<version>.json (conforms to benchmark/results.schema.json) + results/index.json (the list the site loads), rendered by leaderboard/index.html.

Technical recommendations (suggestions)

• Define benchmark/results.schema.json: model, library version, summary metrics (bugs found, total cost, total tokens, bugs/1000-tokens, bugs/$), and per-bug rows that include the certificate.
• benchmark/build_index.py scans results/*.json → writes results/index.json.
• In leaderboard/index.html, delete the hardcoded sample array (~lines 61–68) and instead fetch('../results/index.json'); add a per-bug view that shows the certificate (source → target → solutions → verdict).
• make demo runs a small real session (M1.2 — Build the bug-hunting session: explore → certify → verify, under budget #2) and produces all of the above.

Verification (how a reviewer confirms this is done)

1. Run make demo → it finishes cleanly and creates results/index.json.
2. Open the leaderboard page next to the JSON file → the bug shown has the same numbers and details as the JSON.
3. Prove the page uses real data, not hardcoded values: rename results/index.json and reload → the rows disappear (a hardcoded page wouldn't change).
4. Prove the format check actually catches problems: take a results file, delete one required field, run make validate-results → it fails and names the missing field.

Dependencies

Depends on #2 (which produces the certificates + accounting).

Out of scope

The full public website with two ranking metrics and trajectory views (#6); automatic deployment (#7).

[Ferrari-72]

Issue #3 Implementation Complete — Results Schema, Build Index & Live Leaderboard

What Was Built

Commit: fe1301a (part of the 0a57988 + fe1301a batch)

Deliverables

File	Purpose
benchmark/results.schema.json	JSON Schema for every results file (9 top-level required fields + per-result fields)
benchmark/build_index.py	Scans results/*.json → writes results/index.json; skips invalid/malformed files and logs them
benchmark/validate_results.py	Standalone schema-checker for all results files
leaderboard/index.html	Fetches ../results/index.json at runtime — mock data removed
Makefile	make build-index, make validate-results, make demo targets

Verification Checklist (from the issue)

1. make demo finishes cleanly and creates results/index.json — pipeline wired: run_mini → build_index → index written.
2. Page shows real numbers — leaderboard fetch()s the JSON; hardcoded sample array removed.
3. Rename the JSON → rows disappear — page has no fallback data; empty fetch → empty table.
4. Delete a required field → make validate-results fails and names it:

   ✗ my_run.json — missing required field: 'bugs_found'
   

Test Coverage Added (issue #3 scope)

benchmark/tests/test_build_index.py — 20 tests, all passing:

• Schema validation: 9 required top-level fields, 4 per-result fields, empty results list is legal
• Index building: single-file round-trip, sorted by bugs/Ktok descending, commit truncated to 7 chars, index.json itself skipped
• Bug certificate extraction: only bug_found rows included; no_certificate/rejected rows excluded
• Error handling: malformed JSON skipped with message; schema violations skipped; valid files unaffected by invalid neighbours

Full test run

make test-unit
# 74 passed, 2 skipped in <30s

No failures. The 2 skips are intentional (empty-string path test, documented in test_env_setup.py).