[Leaderboard] Scout (OceanBase Lab) - GLM-5.2 - 90.62% Pass@1

[WeiJiuQi]

Scout (OceanBase Lab) — Leaderboard Submission

• Agent name: Scout
• Team: OceanBase Lab
• Backbone LLM: GLM-5.2 (thinking disabled)
• Hints: Yes (db_description_withhint.txt)
• Tuned prompts: Yes
• Coverage: 54 queries across 12 datasets; five retained outcomes per query, 270 answer/trace pairs
• Pass@1: 90.62%

Architecture

Scout combines a tool-using main agent with on-demand planner subagents. It profiles database schemas, explores data, and uses structured workflows for cross-database joins, transformations, aggregation and ranking. A deterministic workflow compiler and data kernel execute these plans, while model-backed tools support text extraction and classification. SQL/database-query tools and a Python execution tool provide additional computation paths. Intermediate results remain available within the query for subsequent reasoning and bounded plan repair.

Results

Dataset	Passing trials	Total trials	Local Pass@1
bookreview	15	15	1.0000
googlelocal	20	20	1.0000
music_brainz_20k	15	15	1.0000
PATENTS	15	15	1.0000
PANCANCER_ATLAS	15	15	1.0000
stockindex	15	15	1.0000
stockmarket	24	25	0.9600
crmarenapro	59	65	0.9077
DEPS_DEV_V1	9	10	0.9000
yelp	30	35	0.8571
GITHUB_REPOS	16	20	0.8000
agnews	9	20	0.4500

Notes

• Submission file: leaderboard_submissions/oceanbase_lab_scout.json
• Execution traces: scout_trace.tar.xz

[Ruiying-Ma]

Hi @WeiJiuQi, thanks for the submission and the complete traces.

The bundle's README describes the runs as "selective replacements and reindexing; not a fresh unselected frozen evaluation", and shows 19 trials across 13 queries renumbered from later runs (e.g. DEPS_DEV_V1 query1 run_1 and run_2 come from original run_3 and run_4, with the originals gone).

Were failed runs dropped and replaced? The leaderboard requires five unselected trials per query, so if so we would need a fresh five-run evaluation without cherry picking.

[WeiJiuQi]

Hi @Ruiying-Ma, thank you for your prompt response!

Yes, some trials were removed and replaced. We would like to clarify what triggered those replacements and how we handled them.

We initially launched five evaluation runs covering all 270 queries in a single batch for submission. However, after the runs completed, we found that some queries had crashed due to LLM API concurrency limits and intermittent API instability.

To avoid the cost of repeating the entire evaluation, we removed the affected dataset-level trials, renumbered the subsequent trials to close the gaps, and ran additional trials to restore the intended trial count. This explains why some trials in the submission were renumbered from later runs, as you noted.

Importantly, we did not replace only the individual queries that crashed. Instead, we discarded the entire trial for the dataset containing those queries and ran additional full dataset-level trials. The replacement criterion was an execution crash, not answer correctness. Trials were not replaced merely because they produced incorrect answers, and the submitted traces still include incorrect answers. Our intention was to recover from execution failures, not to select better-performing results.

This is also why we explicitly described the submission in the README as “selective replacements and reindexing; not a fresh unselected frozen evaluation.” We wanted to disclose exactly what had happened rather than present these results as an untouched evaluation.

We hope this explanation helps clarify why the replacements were made and that they were solely intended to recover from API-related crashes, rather than to select results based on answer correctness.

We would really appreciate your guidance on whether this handling is acceptable for the leaderboard. If you believe a fresh five-run evaluation is still necessary, we would of course be happy to rerun the benchmark accordingly. Thanks!