Skip to content

docs: refresh bilingual readmes and agent benchmark figures - #480

Open
LivXue wants to merge 10 commits into
mainfrom
docs/raven_agents_performance
Open

LivXue wants to merge 10 commits into
mainfrom
docs/raven_agents_performance

Conversation

@LivXue

@LivXue LivXue commented Sep 17, 2026

Copy link
Copy Markdown
Member

Summary

Present Raven's orchestration capabilities and four specialized agents with benchmark figures, and keep the English and Chinese READMEs aligned.

  • Add the Multi-Agent Orchestration Benchmark at the end of the introduction. Compare Hermes, Claude Code, and Raven with shared 0-1 scales for Node F1, Edge F1, Partial Order Accuracy, and Exact Match Rate.
  • Move Raven Agents before Benchmarks and add concise Research, Code, Design, and Oncall descriptions with their figures.
  • Replace the third-party agent table with a centered card image and explain how Raven connects to and orchestrates these agents. Host image files as GitHub attachments.
  • Standardize primary heading markers and align section order, captions, model settings paths, and the EverMind ecosystem catalog across both languages.

Type

  • Fix
  • Feature
  • Docs
  • CI / tooling
  • Refactor
  • Other

Verification

Commands run after rebasing onto the latest main:

uv run --no-sync pytest tests/test_readme_scope_canon.py tests/test_living_docs.py -q -o addopts=''
# 4 passed
uv run --no-sync pre-commit run --files README.md README.zh-CN.md
# All applicable checks passed
make check-large-files
# Passed
make check-source-language
# Passed
env PYTHONPATH=. uv run --no-sync python scripts/check_commit_messages.py origin/main..HEAD
# Passed
git diff --check
# Passed

Compared all 27 headings, 12 code blocks, image URLs and sizing, commands, paths, numeric values, and architecture graph structure across both languages. Verified all 24 orchestration scores against the supplied results and confirmed that the uploaded chart matches the local PNG byte for byte.

  • Relevant tests pass locally
  • Relevant lint / type checks pass locally
  • User-facing docs or screenshots are updated when needed

Risk

This changes documentation structure, wording, and linked figures. Runtime behavior is unchanged. Figures depend on GitHub attachment availability. Roll back by reverting the changes to README.md and README.zh-CN.md.

  • Security impact considered
  • Backward compatibility considered
  • Rollback path is clear for risky changes

Related Issues

#462, #465, #474, #477, #479

@gloryfromca gloryfromca left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No blockers; this can merge as far as I am concerned.

Reviewed the full github/main...HEAD diff at c2b7ed0f48cd, including the follow-up that expands the two single-column charts. I covered the repository rules in AGENTS.md, canonical architecture terminology from CONTEXT-MAP.md/CONTEXT.md, bilingual structure and content parity, GitHub rendering, all newly added external and attachment URLs, surrounding README/UI/agent references, commit history, backward compatibility, and test integrity. This is documentation-only, changes no runtime callers or architecture, and does not weaken or remove tests.

Validation: uv run pytest tests/test_readme_scope_canon.py tests/test_living_docs.py tests/test_docker_runtime.py -q -o addopts='' passed (13 tests); the large-file and source-language scripts passed; git diff --check github/main...HEAD passed. GitHub's renderer preserved the matched image/heading structure in both READMEs and the new full-width attributes.

LivXue and others added 8 commits September 17, 2026 15:42
Reference the transparent GitHub-hosted PNG in both READMEs and preserve
all thirteen agent names as alt text.

Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Codex <noreply@openai.com>
Reference the new GitHub attachment in both READMEs while keeping the
centered layout and 70 percent display width.

Co-authored-by: Codex <noreply@openai.com>
Introduce the four specialized agents and their benchmark figures, move
the agent overview before benchmarks, standardize section markers, and
describe third-party agent orchestration.

Align the Chinese ecosystem catalog, model settings paths, and captions
with the English README. Keep image URLs, layouts, commands, and
configuration guidance consistent across both languages.

Co-authored-by: Codex <noreply@openai.com>
Place the linked benchmark chart at the end of the Raven introduction in
both READMEs. Show Hermes, Claude Code, and Raven on shared linear scales,
with Node F1, Edge F1, Partial Order Accuracy, and Exact Match Rate.

Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Codex <noreply@openai.com>
Stack the six paired figures in both READMEs and refresh their attachment
links with layouts designed for full-width display. Preserve centered
captions, linked full-size images, and all 89 benchmark values.

Co-authored-by: Codex <noreply@openai.com>
@LivXue
LivXue force-pushed the docs/raven_agents_performance branch from c2b7ed0 to 9242447 Compare September 17, 2026 15:47

@gloryfromca gloryfromca left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: restore the Quick Start contract and replace the unreadable AI4S chart.

I reviewed the new chart-layout commit and the full rebased github/main...HEAD diff. I covered the current AGENTS.md and CONTEXT terminology rules, the rebase and commit history, GitHub-facing document structure and assets, the installer test that consumes the README, backward compatibility, and test integrity. Runtime architecture and callers are unchanged, and this PR does not modify or weaken tests.

Validation: test_readme_quickstart_matches_the_installer fails locally with StopIteration, matching the current CI shard failure; the other focused README/living-doc/Docker tests pass (13 tests). The large-file and source-language checks and git diff --check pass.

Comment thread README.md
Comment thread README.md
The check located the Quick Start section by a literal rocket prefix, so
restyling the README headings to a new prefix broke it with a bare
StopIteration that named neither the file nor what it was looking for.

Match the heading text instead. The English and Chinese READMEs carry
different heading words, so each file now supplies its own, and the
anchor survives a future prefix change. A count assertion replaces
next(), which would have silently taken the first of two same-named
sections, and the failure message now names the file and the heading.

Reading both files as UTF-8 keeps the Chinese README readable on hosts
whose default encoding is not UTF-8.

Co-authored-by: Claude (claude-opus-5[1m]) <noreply@anthropic.com>

@gloryfromca gloryfromca left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No blockers; this can merge as far as I am concerned.

Reviewed the a0cedc452 test-only delta and the full current PR diff. The semantic, localized Quick Start lookup fixes the prior CI failure while preserving the three installer-content assertions and adding exact-one-section validation. The author's native-resolution evidence, independently reproduced from identical asset bytes, refuted my AI4S finding; I withdrew it and resolved both threads I opened.

Covered the current test caller and contract, AGENTS.md language rule, commit history, backward compatibility, test integrity, and the unchanged runtime/architecture surface. Validation: the full onboarding test file passed (332 tests); focused README/living-doc/Docker tests passed (13 tests); large-file, source-language, and git diff --check checks passed.

Both figures drew one horizontal bar per entry, so their height grew with
the number of entries: eleven rows and thirteen rows made them the two
tallest images in the READMEs. Vertical columns decouple height from the
entry count, which costs label width instead and is what the pre-wrapped
method and model names in the plotting data were already written for.

DataAgentBench drops from 1320 to 876 px tall and PresentBench from 1420
to 944 px, both at the same 2000 px width, so each reads about a third
shorter at the full line width the READMEs render them at.

Every plotted value is unchanged: the renderer asserts its output equals
the values files the previous figures published, and the uploaded images
were downloaded back and compared byte for byte against the local
renders.

Co-authored-by: Claude (claude-opus-5[1m]) <noreply@anthropic.com>

@gloryfromca gloryfromca left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No blockers; this can merge as far as I am concerned.

Reviewed the new chart-rotation delta and the full current PR comparison. Both replacement URLs resolve to valid 2000 px-wide PNGs; native-resolution comparison confirms every prior method, model, category, and value is preserved, while labels, legends, axes, and values remain unclipped and non-overlapping. The English and Chinese READMEs still use identical assets and links.

Covered AGENTS.md/CLAUDE.md rules, the diff, README rendering consumers and installer-test caller, commit history, backward compatibility, test integrity, and the unchanged runtime/architecture surface. Validation at this head: 345 README/onboarding/living-doc/Docker tests passed; source-language, large-file, and whitespace checks passed; all completed GitHub checks pass (one independent unit shard was still running when reviewed). Patch-risk result: merge, low impact and regression likelihood, strong changed-path protection, easy revert, high confidence.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants