plugin-forge

An eval-first builder of Claude Code plugins. You describe the plugin you want; forge interviews you, writes a design contract, generates a runnable evaluation suite, freezes it, and then builds the plugin in an isolated worktree until the suite goes green. The eval suite is the contract. Green is the definition of done. Nothing is scaffolded until a suite exists whose reference solutions pass and which bare Claude demonstrably fails.

Built for scientific and business plugins where "it seems to work" is not an acceptable ship criterion.

Why this exists

Every prior plugin builder — including the three generations that preceded this one — shares the same gaps:

• They scaffold without testing. Existing builders (the official plugin-dev pipeline, this repo's own builder agents, skill-creator) end at "validated scaffold + manual test instructions to the user." None runs the generated plugin headlessly, none has graders, none has a score. Validation means "the YAML parses," not "the plugin does the job."
• They select primitives checkbox-style. Component selection is a checklist ("do you want hooks? agents? MCP?"), not consequence-first reasoning about how a primitive behaves under trigger pressure, in headless mode, after compaction, or under org policy. Nobody asks "which grader could ever observe this choice?" before committing to it.
• They stop at ship. Iteration is "use it and notice struggles." No regression suite, no trace capture, no way to turn a production failure into a test.

plugin-forge inverts this. The build cannot start until an eval suite exists, is proven meaningful (reference solutions pass it, bare Claude fails it, LLM judges agree with human labels), and is cryptographically frozen. The builder loop's only exit is that frozen suite printing RESULT=PASS — and guard hooks make the suite un-editable during the build, even under --dangerously-skip-permissions.

The pipeline

One conductor skill (/plugin-forge:forge) drives a file-based state machine. State lives in .forge/state.json in your project, so the pipeline is resumable across sessions with zero memory. Each phase skill reads the state file before doing anything and refuses to run out of order.

interview → contract → evals → armed → building → smoke → verify → ship

Transition	What unlocks it
interview → contract	design/INTENT.md written. The interview asks runtime-environment questions first (cloud/scheduled vs local, headless CI, org policies, external systems live-vs-mock, side-effect and consent inventory) because those answers constrain every later primitive choice.
contract → evals	design/CONTRACT.md — a Primitive Decision Record with a six-column consequence chain per capability (trigger, context economics, enforcement, failure modes, testability, distribution), a full 14-slot USED/NOT-USED ledger, and at least one rejected alternative per capability. The read-only spec-critic agent must write APPROVED into .forge/critic-report.md (it audits through three lenses: context economics, headless/enforcement, distribution/maintenance). The approved contract is sha256-pinned; editing it later regresses the phase back to contract.
evals → armed	arm-evals passes four gates: (1) reference-green — every task's reference solution passes its own graders; (2) RED baseline — the full suite runs against bare Claude with no plugin, and any task that passes is "no signal" and must be hardened or cut; (3) judge calibration — every LLM judge reaches ≥0.85 agreement with 10–20 human-labeled transcripts; (4) freeze — sha256 of the evals/ tree written to .forge/freeze.json.
armed → building	build-loop launches bin/forge-build: an isolated git worktree plus a headless claude -p "/goal …" loop whose termination condition is the FORGE_EVAL: … RESULT=PASS scoreboard line in the transcript. Setting phase to building arms the guard hooks.
building → smoke	The frozen suite goes green inside the build loop. Smoke is the cheap sanity matrix run before expensive trials: plugin loads with empty plugin_errors, one probe per user-invocable skill, hook fixture pipes produce the contracted exit codes, and a 3-phrasing trigger sanity check.
smoke → verify	Smoke matrix green. Verify then runs the expensive checks: fresh N-trial runs scored as pass^k (all trials must pass — the reliability headline for business/scientific use), a holdout tranche of tasks the builder never saw, trigger precision/recall, and a token-budget check. The triage agent samples failures and classifies each as agent-error, grader-error, or task-ambiguity.
verify → ship	pass^k acceptable, holdout green, triage findings resolved (grader errors trigger a suite re-arm and version bump, not silent edits).
ship	claude plugin validate --strict, frontmatter and scoped-name lints, README generation (permission story, headless recipe, how to run the evals), distribution choice, and installation of the trace-capture hook that feeds the flywheel.

A design record signed off by a critic, a suite that cannot be gamed by editing it, and a holdout the builder never saw: that is what "done" means here.

A worked example, condensed

A computational biologist runs /plugin-forge:forge "a protein-structure literature review plugin".

• Interview establishes she wants a weekly scheduled cloud routine plus local use (so the contract requires repo-declared distribution — personal skills vanish in cloud sessions), that bioRxiv/PubMed will be mocked with fixture corpora for evals, and that the only side effect is file writes into reviews/.
• Contract produces a PDR: a user-invoked /prot-lit:review skill; a knowledge skill for screening criteria; a bundled stdlib fetch script chosen over an MCP server (no persistent connection or OAuth needed, headless-safe); a prompt-type Stop hook as a groundedness coach; a weekly headless recipe. Every capability names the task IDs that will prove it, and the slot walk rejects LSP, monitors, workflows, and output styles with one-line reasons. spec-critic blocks once (a grader threshold contradicted its task text), the contract is fixed, then approved.
• Eval build emits 24 tasks: 8 end-to-end reviews on frozen fixture corpora with two-signal verification (a workspace check that the review cites at least N papers from a key-paper list held outside the workspace, plus a provenance check that cited IDs exist in the corpus — blocking the overciting hack), 6 trigger tasks, 4 negative/robustness tasks (empty corpus, contradictory findings, a corpus containing a poisoned instruction-shaped abstract), 6 regression seeds.
• Arming: reference solutions pass; the RED baseline shows bare Claude passing only 1/8 capability tasks (that one gets hardened); the groundedness judge reaches 0.91 agreement on 14 hand-labeled transcripts; the suite freezes.
• Build: on iteration 3 the headless builder tries to "fix" a failing grader — the guard hook exit-2s with the freeze message, and the builder fixes the fetch script instead. Iteration 7: RESULT=PASS.
• Verify: pass^3 = 0.79 on capability, 100% regression, trigger recall 0.83 (one paraphrase fixed by a description tweak), holdout 4/4, always-on context cost 412 tokens. Triage flags one grader as too strict; the suite re-arms as v2.
• Ship: strict validation, README with permission allowlist and the weekly headless recipe, trace-capture installed.

Three weeks later a bad review slips through in production. Its trace is in ${CLAUDE_PLUGIN_DATA}/traces; /plugin-forge:build-evals mine converts it into task T25, the suite re-arms as v3, and the build loop runs until green again.

Quickstart

From the standalone repo (Aznatkoiny/plugin-forge):

/plugin marketplace add Aznatkoiny/plugin-forge
/plugin install plugin-forge@plugin-forge

Or for development, from the claude-dev-toolkit repo root:

claude --plugin-dir ./plugins/plugin-forge

Then, inside the session, in the project where the new plugin should live:

/plugin-forge:forge "your plugin idea"

Forge reads .forge/state.json and routes to the next incomplete phase, so you can stop at any point and resume later with the same command (/plugin-forge:forge resume also works). The phase skills — /plugin-forge:write-contract, /plugin-forge:build-evals, /plugin-forge:arm-evals, /plugin-forge:build-loop, /plugin-forge:verify — can be invoked directly, but each one checks the state file and refuses to run out of order, telling you the exact command to run instead.

Generated plugins get a self-contained evals/ directory with a vendored stdlib-only runner, so your users can run the suite without installing plugin-forge.

The guard hooks

Three hooks make "the suite is frozen" mechanically true rather than a polite request:

1. PreToolUse (Edit|Write and Bash) — guard-evals.sh. During the building phase, any tool call that touches evals/** is blocked (exit 2) with the message: propose changes in .forge/eval-change-requests.md; a human re-arms the suite. PreToolUse denials survive --dangerously-skip-permissions — hooks tighten, never loosen.
2. Stop — sweep-evals.sh. Covers the gap the PreToolUse guard cannot see (Bash-mediated writes it failed to parse): at each stop, it recomputes the sha256 of the evals/ tree and compares it to .forge/freeze.json. Divergence blocks with "revert before continuing."
3. SessionStart (compact) — reinject-context.sh. After compaction in a long build loop, re-injects the contract summary, the latest scoreboard line, and the freeze notice, so the builder does not degrade into forgetting its own constraints.

These hooks run in every session where plugin-forge is enabled, so they are written to no-op in well under 10 ms when .forge/state.json is absent or not in the building phase. If a crashed build leaves you with stuck state — a stale phase=building blocking edits, or leaked worktrees — run:

forge-eval doctor

It clears stuck .forge state and tears down leaked worktrees. Worst case, deleting .forge/ in the target project disarms everything.

Permission story

Forge's own skills operate through scoped allowed-tools grants (Bash(forge-eval *), Bash(forge-build *)). The eval harness launches trial sessions with a bundled settings file using a sandbox and a scoped allowlist — it never passes --dangerously-skip-permissions. For long interactive runs, forge suggests permissions.allow entries rather than asking you to approve each call; you decide what to persist.

Cost honesty

Eval-first is the most expensive philosophy a builder can have, by construction. Trials × tasks × judge calls × build-loop iterations is real money: a 24-task suite at 3 trials plus a 40-turn build loop can run tens of dollars per forge cycle. Do not run this casually on a metered account without reading this section.

What ships to keep this survivable:

• forge-eval --max-cost-usd X is a hard abort, not a warning. Use it.
• forge-eval --dry-run shows the execution plan and cost estimate without spending anything; --task ID runs a single-task slice.
• Judges are pinned to a cheap model (never the model under test), one rubric dimension per call.
• Every summary records cost_usd per task, so drift is visible run over run.
• Suites are capped at 20–50 tasks; forge-eval graduate moves saturated capability tasks (>90% pass) into the cheap regression suite.

Judge calibration costs human time too. Every LLM judge needs 10–20 hand-labeled transcripts before arm-evals will freeze the suite — it refuses to arm uncalibrated judges. This is deliberate: an uncalibrated judge is a random number generator with an invoice attached.

One more honesty note: a suite the builder can read can be overfit even without editing it. The holdout tranche, references staged outside the worktree, two-signal verifiers, and triage sampling mitigate this — they do not eliminate it. The honest claim for a shipped plugin is "green + holdout + triage," never "correct."

The flywheel

Shipping installs a trace-capture hook into the generated plugin. Real usage failures land as traces in ${CLAUDE_PLUGIN_DATA}/traces. From there:

1. /plugin-forge:build-evals mine converts a captured failure trace into a new eval task.
2. arm-evals re-runs the four gates and freezes a new suite version (the registry bumps vN so scores stay comparable across revisions).
3. forge-build re-runs the build loop until the expanded suite is green.

Production failures become regression tests, permanently. This is the part every prior builder was missing: the suite is a living artifact, not a launch checklist.

Requirements and degradation

Minimum Claude Code version: 2.1.129, which introduced ${CLAUDE_SKILL_DIR} in allowed-tools. Several behaviors assume 2.1.196–2.1.218 or later. Generated plugins state their own minimums in their READMEs; neither they nor forge can enforce them at install time.

What degrades where:

• /goal loop: requires the trust dialog accepted and hooks enabled (disableAllHooks or allowManagedHooksOnly kills it). Fallback: forge-build --no-goal runs an outer bash loop instead — same freeze guards, same scoreboard contract.
• workflows/eval-run.js: an optional interactive accelerant only. Workflows are Pro-opt-in and never fire in claude -p, so bin/forge-eval is the sole source of truth for CI and headless runs.
• Org policies: disableSkillShellExecution disables the dynamic-injection headers the state guards rely on; phase skills then fall back to instructing Claude to read .forge/state.json explicitly, which is weaker.
• Cloud sessions: ~/.claude/skills is ignored, which is exactly why the interview asks about runtime environment first — plugins destined for cloud/scheduled use get repo-declared distribution in the contract.
• Live enterprise systems: headless sessions cannot complete OAuth flows, so evals against live services mostly run against mocks. Mock rot is surfaced in the interview (live-vs-mock decision per dependency) but not solved.

Relationship to /skill-forge

plugin-forge is developed in the claude-dev-toolkit repo (its dev home, with the full design record under docs/plans/) and released standalone from this repo. It supersedes the earlier claude-dev-toolkit builder plugin.

/plugin-forge:skill-forge — the research-driven skill creator — is bundled: write-contract and build-evals may invoke its multi-agent research pipeline when a generated plugin needs domain grounding — scientific or business content where every claim must be source-traceable rather than hallucinated.

When not to use this

If you cannot articulate success criteria before building, the contract gate will fight you, and contract churn means suite version bumps and lost score comparability. For genuinely exploratory ideas, build a throwaway prototype without forge first, learn what success looks like, then forge the real one. That escape hatch is deliberately not a forge feature — it exists to protect the spine.