More validators share the #78 substring-without-word-boundary defect (agnews/query4, bookreview/query1, crmarenapro/query1)

[marc-shade]

#78 (query_crmarenapro/query9, CLOSED COMPLETED) fixed one validator whose fast-path was a plain substring test. The same defect is present in at least three more validators on main @ 9ed8bdd, so the fix did not generalize. Each accepts a content-free answer that merely contains the ground-truth token anywhere in the text.

1. query_bookreview/query1 — the worst case

gt = "2020"
if gt in llm_output:                 # substring, no boundary, no field check
    return True, "Ground truth found in LLM output."

Any output containing the four characters 2020 passes: "2020s", "$2020.00", "article id 2020", or an in-progress status line. Observed: a run that answered "Pass 1 is running (27/112 batches). While it completes, here's what's established so far..." — i.e. it never produced an answer — was scored correct because the narration mentioned 2020.

2. query_agnews/query4

gt = "Africa"
if gt.lower() in llm_output.lower():
    return True, "Ground truth found in LLM output."

"The largest region is Asia, not Africa" passes. Any answer that reasons about the regions (all five region names typically appear) passes regardless of the conclusion.

3. query_crmarenapro/query1

expected = "Authority"
if expected.lower() in llm_output_clean.lower():
    return True, f"Found expected BANT factor: {expected}"

An answer of "Budget, Authority, Need, Timeline" (listing all BANT factors, i.e. declining to choose) passes. Same shape as #78's query9.

Reproduction

import importlib.util
def v(path, text):
    s = importlib.util.spec_from_file_location("v", path); m = importlib.util.module_from_spec(s); s.loader.exec_module(m)
    r = m.validate(text); return r[0] if isinstance(r, tuple) else r

assert v("query_bookreview/query1/validate.py", "Pass 1 is running... established in 2020s") is True
assert v("query_agnews/query4/validate.py", "Asia, not Africa") is True
assert v("query_crmarenapro/query1/validate.py", "Budget, Authority, Need, Timeline") is True

Why it matters

These pass-throughs inflate Pass@1 for every entry on the leaderboard, not any one agent: an agent that hedges, lists all options, or emits a status line gets credit. A content-free probe (empty / "I don't know" / a shotgun of plausible tokens) is a cheap screen — a validator that passes any of those cannot distinguish a correct answer from a non-answer.

Suggested fix

Same remedy as #78, applied as a sweep rather than per-query: require a word-boundary / exact-field match (e.g. re.search(rf"\b{re.escape(gt)}\b", out)), and for the single-value questions reject an answer that also contains the other admissible values. A repo-wide screen that runs each validate.py against a fixed set of content-free probes and flags any that pass would catch the rest.

Environment

• Repo main @ 9ed8bdd
• Found while scoring a full 54-query × 5-run submission; the three above were the only validators (of 54) that passed a content-free probe.