Skip to content

Generalize the catalog contract: BeamMCP.Catalog carries tools, resources and prompts - #21

Merged
HackTuah merged 6 commits into
mainfrom
slice/008-catalog-generalization
Sep 13, 2026
Merged

HackTuah merged 6 commits into
mainfrom
slice/008-catalog-generalization

Conversation

@HackTuah

@HackTuah HackTuah commented Sep 8, 2026

Copy link
Copy Markdown
Member

Replaces BeamMCP.ToolCatalog with BeamMCP.Catalog, whose one callback carries all three
catalog-shaped concepts MCP has:

%{tools: [BeamMCP.ToolSpec.t()], resources: [], prompts: []}

resources and prompts are required and may be empty. Nothing reads them today. They exist
so that serving them later (SCR-283, SCR-284) adds a reader rather than changing this contract a
second time.

This is a breaking change, and it lands at a minor on purpose

Every host implementing BeamMCP.ToolCatalog must change. Under this package's 0.x policy a
break lands at a minor0.4.0 — and the README recommends ~> 0.3.0 (README.md:17),
which excludes 0.4.0. So no existing consumer is carried across this break by mix deps.update;
a host that wants it asks for it. That policy is not a note in a file, it is pinned by
readme_claims_test.exs with a major == 0 guard.

The callback is renamed too, all/0capabilities/0. Keeping the name while changing the
return from a list to a map would compile against every existing host and fail at the first
request with a BadMapError — a silent shape change. Renaming makes the break arrive at compile
time as an unimplemented callback, which is the loudest place it can arrive.

The three decisions

(a) Clean break, no deprecated delegate. Two reasons; the second is load-bearing. The version
policy above already protects consumers. And a delegate would create a second reader:
ToolCatalog.all/0 living beside Catalog.capabilities/0 lets a host implement both and make
them disagree, with nothing to catch it. The compatibility shim would reintroduce the exact defect
the contract exists to prevent, to soften a break the version policy already handles.

(b) The single-lookup guarantee is preserved and structurally stronger. Before, tools/list
called catalog.all() and ToolCatalog.fetch/2 called catalog.all() — two call sites of the
host's function, agreeing by convention. Now there is one reader:

def tools(catalog), do: catalog.capabilities().tools

server.ex:185 advertises through it; Catalog.fetch/2 resolves through it. The host's function
is called from one place in lib/, so the two paths cannot be given different sources without
editing that line. Pinned by effect in catalog_test.exs: the names tools/list advertises
are compared against the names tools/call accepts, for a catalog whose tool exists in no other
catalog in the suite — so neither path can agree by coincidence, which is how slice 002's original
test could pass.

(c) fetch/2's @spec is documented honestly, not caught. It says {:ok, t} | :error and a
malformed host catalog makes it raise. Catching would make a host bug indistinguishable from "no
such tool": a broken catalog would present exactly as a working one that lacks that tool. That is
the advertise-versus-call disagreement this behaviour exists to prevent, reintroduced by the error
handling meant to be defensive. Catalog.validate/1 at Server.new/1 refuses these at startup
instead, so the raise conditions are unreachable for a server that started.

The fetch/2 ledger was wrong, and the correction is worse than the count

PR #10's ledger claims three host shapes raise. Reproduced on this tree
(slices/008-catalog-generalization/logs/probe-fetch-spec.txt) — four raise, and a fifth case
the ledger missed does not raise at all
:

host spec.name is a binary, not an atom       {:RAISED, ArgumentError}
all/0 returns a non-list map                  {:RAISED, BadMapError}
all/0 returns nil                             {:RAISED, Protocol.UndefinedError}
all/0 returns a bare map, not a %ToolSpec{}   {:returned, {:ok, %{name: :echo}}}
module is not loaded / does not exist         {:RAISED, UndefinedFunctionError}
CONTROL: a well-formed catalog                {:returned, {:ok, %BeamMCP.ToolSpec{...}}}

The fourth line is the one that matters. It returns {:ok, %{name: :echo}} successfully,
violating the @spec silently and handing a struct-shaped thing that is not a struct to dispatch.
A raise is a loud wrong answer; this is a quiet one. All five are refused at Server.new/1 now.

Red before green

The init-time refusal was removed from Server.new/1 first
(logs/red-malformed-catalog.txt) — five tests, each Expected exception ArgumentError but nothing was raised:

13 tests, 5 failures
TEST_EXIT=2

Restored (logs/green-malformed-catalog.txt): 13 tests, 0 failures, TEST_EXIT=0.

Mutation — both KILL, zero variance

logs/mutation-catalog.txt, tree 8f47e1f80c4dc356b3d84d72eb1d468cfa461c49, 2 passes each. Two
invocations, because the mutants have different targets:

TARGET=lib/beam_mcp/catalog.ex  Mcat1 | 187 tests,  2 failures TEST_EXIT=2 | 187 tests,  2 failures TEST_EXIT=2
TARGET=lib/beam_mcp/server.ex   Mcat2 | 187 tests, 12 failures TEST_EXIT=2 | 187 tests, 12 failures TEST_EXIT=2

Mcat1 drops :prompts from @required_keys — the smallest possible relaxation, one key out of
three. Killed by the startup refusal and, independently, by the README claim test, because the
README promises that refusal. Mcat2 leaves tools/list reading the catalog and makes
tools/call answer "which tool does this name mean" from the request — slice 002's defect in the
direction it actually occurred. Killed by the single-lookup test and eleven others.

lib/ is byte-identical between the scored tree and the head of this branch
(git diff 8f47e1f HEAD -- lib/ → no output), so the table is a score for the tree that ships.

Mcat2's first version was a compiler kill, and it is recorded as one

It returned {:ok, spec} unconditionally. Elixir narrowed find_tool/2's return to that shape,
declared the caller's :error -> clause unreachable, and --warnings-as-errors failed the build
— the score line came back with no test count at all:

Mcat2 |  TEST_EXIT=1 |  TEST_EXIT=1

CONVENTIONS.md: a compiler kill records a kill that never happened. No test ran. The mutant keeps
an :error branch now so the suite scores it instead of the compiler. Corrected so it could be
scored, not weakened until it killed
— every name a client can actually send is still callable
under it. The failed run is kept in the log above the corrected one.

The call-site list was measured, and the brief's was wrong in both directions

The brief named four readers. The tree has five:

lib/beam_mcp/server.ex:185          tools/list advertises
lib/beam_mcp/server.ex:304          find_tool/2, the tools/call gate
lib/beam_mcp/server.ex:313          new/1's init-time validation
lib/beam_mcp/transport/http.ex:225  init/1's export check
lib/beam_mcp/transport/http.ex:836  the x-mcp-header annotation lookup

server.ex:313 and http.ex:225 were missing from it; server.ex:311 was on it and is not
affected, because it consumes a ToolSpec, not the catalog.

The two init paths check different things on purpose. Server.new/1 calls capabilities/0
host code, at runtime. Transport.HTTP.init/1 does not, because Plug's default init_mode is
:compile: init/1 runs at the host's compile time, where a correct catalog reading config or
ETS would crash. It checks function_exported?/3 only and leaves the shape to new/1.

An amended commit, recorded rather than hidden

4a2d2e9's message read "172 tests before, 185 after". 185 was typed, not measured — the run
printed 187 tests, 0 failures. CONVENTIONS.md forbids exactly that. The commit was amended
before anything was pushed and is now 8f47e1f, and the correction is also written into FINDINGS,
because an amended commit leaves no trace of the original and the defect is the habit, not the
digit.

Also in here: four tracked probes the gate cannot see

tools/probe_ping.exs and the three probes in tools/probes/ each carried three breaks from this
change — @behaviour BeamMCP.ToolCatalog, def all, and catalog: spelled tool_catalog:. The
gate never compiles tools/*.exs, so it stayed green over all of them. Found by grepping the
whole tree rather than lib/, fixed, and proven by running them: mix run tools/probe_ping.exs
serves echo from tools/list, and each network probe prints PROBE_DONE at N=1 with the same
{297, :econnreset, ..., true} slice 006 measured.

That gap is a defect in the verification instrument, not in this slice, and is filed separately.

Gate, on the head of this branch

format pass · compile pass · test pass · credo pass · optional deps pass · docs pass
reuse  pass (314 tracked; 93 in scope, 91 headered + 2 sidecar; excluded 219 archive + 2 licence text)
licence files pass
Gate OK.  GATE_EXIT=0

187 tests, 0 failures.

Records: slices/008-catalog-generalization/. Issue: SCR-294.

…and prompts

BREAKING, to a host contract rather than to the wire. Taken now, before the
resources and prompts slices exist, so it is taken once.

(a) MIGRATION PATH: a clean break, no deprecated delegate.

    The decisive argument is not taste, it is the version policy already in
    place. Breaks land at the minor while this package is 0.x, and the README
    recommends `~> 0.3.0`, which EXCLUDES 0.4.0 -- measured in slice 003's
    version table. No consumer is carried across by a routine mix deps.update;
    that tight pin exists for exactly this.

    And a delegate would cost the thing the contract is for: keeping
    ToolCatalog as a shim means two entry points into "which tools exist",
    which is the second reader that decision (b) forbids.

(b) THE SINGLE-LOOKUP GUARANTEE: preserved, and strengthened.

    tool_catalog.ex promised it in prose and left two call sites --
    server.ex:185 called catalog.all() directly, Catalog.fetch/2 called it
    again. Now both go through Catalog.tools/1, one function, so advertise and
    call cannot drift apart without deleting it. Slice 002 fixed a real defect
    of exactly this shape: tools/list honoured an injected catalog and
    tools/call ignored it.

    Pinned by effect, not by prose: catalog_test.exs compares what tools/list
    advertises against what tools/call accepts, for a catalog whose tool exists
    in no other catalog in the suite -- so neither path can pass by the
    coincidence that made slice 002's original test green.

(c) THE fetch/2 @SPEC: documented honestly, not caught.

    Catching would turn a host bug into :error, which is indistinguishable from
    "no such tool" -- the advertise-versus-call confusion this behaviour exists
    to prevent, reintroduced by the error handling meant to be defensive. The
    raise conditions are listed in the @doc and one is pinned by a test.

    THE COUNT IN PR #10's LEDGER IS WRONG AND IS CORRECTED. It claims three
    shapes raise. Measured on this tree
    (slices/008-catalog-generalization/logs/probe-fetch-spec.txt):

      host spec.name is a binary            ArgumentError
      capabilities/0 returns a non-list map BadMapError            <- not the
      capabilities/0 returns nil            Protocol.UndefinedError   one error
      module not loaded                     UndefinedFunctionError

    Four, not three: "returns a non-list" is two different raises depending on
    the shape. And a FIFTH case the ledger missed entirely, which is worse than
    a raise because nothing fails: a bare map entry returns {:ok, %{name: :echo}}
    SUCCESSFULLY, violating the @SPEC silently. That one is closed by init-time
    validation rather than by documentation.

CALLBACK RENAMED, not just re-typed. Keeping all/0 while changing its return
from a list to a map compiles against every existing host and fails at the
first request with a BadMapError. capabilities/0 makes the break arrive at
compile time as an unimplemented callback. The option is :catalog, not
:tool_catalog, for the same reason: a catalog carrying resources and prompts is
not a tool catalog.

A malformed catalog is refused at Server.new/1 -- runtime, so calling the host
is safe. Transport.HTTP.init/1 checks only the export, deliberately: under
Plug's default init_mode it is the host's COMPILE time, and a correct catalog
reading config would fail there.

The call-site list I was given missed several: server.ex's alias, moduledoc,
typespec and new/1 key; stdio.ex's two moduledoc mentions; http.ex's alias,
moduledoc, opts lookup and the ArgumentError text naming all/0. server.ex:311
(dispatch) is NOT affected -- it consumes a ToolSpec, which is unchanged.

    187 tests, 0 failures -- quoted from the run, and corrected: the first
    version of this message typed 185, which no command had printed.
    Gate OK.  GATE_EXIT=0

Signed-off-by: Ayla Croft <aylacroft@proton.me>
…e reader

Mcat1 drops :prompts from @required_keys -- one key out of three, no message
change, no behaviour change for a correct host. If the smallest relaxation of
the check survives, the check is decoration. It dies twice over: at the startup
refusal, and independently at the README claim test, because the README promises
that refusal.

Mcat2 leaves tools/list reading the catalog and makes tools/call answer "which
tool does this name mean" from the request instead -- slice 002's defect in the
direction it actually occurred, advertise and call on different sources.

Mcat2's first version returned {:ok, spec} unconditionally. Elixir narrowed
find_tool/2's return to that shape, called the caller's `:error ->` clause
unreachable, and --warnings-as-errors failed the build: the score line came back
with no test count at all. That is a compiler kill, which records a kill that
never happened. It keeps an :error branch now so the suite scores it rather than
the compiler -- corrected so it could be scored, not weakened until it killed.
Every name a client can actually send is still callable under it.

Both targets differ from the harness default, so each is scored by its own
invocation with TARGET set; the mutant headers say which.

Signed-off-by: Ayla Croft <aylacroft@proton.me>
… scores

PLAN and FINDINGS in slices/008-catalog-generalization/, plus the scored
mutation archive.

The PLAN specifies no review rounds, and says so rather than leaving the absence
to be inferred: the brief was a single pass, nobody is scoring this slice, and a
round boundary with no reviewer behind it is a heading. The mutants and the gate
stand in.

FINDINGS records four things that were measured rather than assumed:

  - PR #10's ledger says three host shapes make fetch/2 raise. Four do, and a
    fifth case the ledger missed is worse than any of them: a bare map entry
    RETURNS {:ok, %{name: :echo}} successfully, violating the @SPEC silently.
    All five are refused at Server.new/1 now.
  - The brief's call-site list was wrong in both directions -- two readers
    missing, one listed that is not affected because it consumes a ToolSpec.
  - Mcat2's first version was a compiler kill, left in the log above the
    corrected run.
  - 4a2d2e9's message typed "185 after" where the run printed 187. The commit
    was amended before anything was pushed; recorded here too, because an
    amended commit leaves no trace and the defect is the habit, not the digit.

Also stated plainly: the Linear issue was filed at record time, not before the
code, which is a departure from slice 007 and costs something real -- nothing
external witnessed the acceptance criteria before they were met.

Signed-off-by: Ayla Croft <aylacroft@proton.me>
Found after the records were written, by grepping for ToolCatalog across the
whole tree rather than across lib/. Four tracked scripts still declared
@behaviour BeamMCP.ToolCatalog, defined all/0, and passed tool_catalog: -- three
separate breaks each, none of which the gate can see, because it never compiles
tools/*.exs.

They are not archives. tools/probe_ping.exs and the three network probes are
instruments a later slice re-runs, and an instrument that no longer runs is the
same defect as an archive nobody can refetch.

Proven by running them, not by reading them:

  mix run tools/probe_ping.exs                        -> tools/list serves echo, exit 0
  N=1 MIX_ENV=test mix run tools/probes/write_shape.exs     -> PROBE_DONE, exit 0
  N=1 MIX_ENV=test mix run tools/probes/drain_mechanism.exs -> PROBE_DONE, exit 0
  N=1 MIX_ENV=test mix run tools/probes/loss_site.exs       -> PROBE_DONE, exit 0

All three network probes still report {297, :econnreset, ..., true}, the shape
slice 006 measured, so the contract change moved nothing they observe.

Signed-off-by: Ayla Croft <aylacroft@proton.me>
Criterion 1 was written as "nothing in lib/ reads tool_catalog.ex" and was true
while four tracked instruments outside lib/ were broken by the same change. The
finding is the scope of the grep, not the four files.

Signed-off-by: Ayla Croft <aylacroft@proton.me>
The publication step added in 009 refuses new tracked paths under
slices/; these six were written before the records moved out of the
tree. Copied to the internal repository and verified identical by
diff -r before untracking, then removed. The allowlist is unchanged:
it grandfathers published history, and this branch is not merged, so
there is no published history here to grandfather.

Signed-off-by: Ayla Croft <aylacroft@proton.me>
@HackTuah
HackTuah force-pushed the slice/008-catalog-generalization branch from f19492a to d0cbb8b Compare September 13, 2026 16:42
@HackTuah
HackTuah merged commit eea0d0d into main Sep 13, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant