diff --git a/src/plugins/claude/build.sh b/src/plugins/claude/build.sh index 9eaf32c..a767672 100755 --- a/src/plugins/claude/build.sh +++ b/src/plugins/claude/build.sh @@ -4,7 +4,6 @@ # # The Claude Code marketplace consumes a repo whose ROOT is the marketplace: # .claude-plugin/marketplace.json marketplace manifest -# plugins/braintrust/ skills plugin (MCP + skills) # plugins/trace-claude-code/ tracing plugin (thin daemon hook shim) # # Everything is plain shell + config — no compiled artifacts — so the whole diff --git a/src/plugins/claude/content/.claude-plugin/marketplace.json b/src/plugins/claude/content/.claude-plugin/marketplace.json index 818727d..ef0902b 100644 --- a/src/plugins/claude/content/.claude-plugin/marketplace.json +++ b/src/plugins/claude/content/.claude-plugin/marketplace.json @@ -2,18 +2,12 @@ "$schema": "https://anthropic.com/claude-code/marketplace.schema.json", "name": "braintrust-claude-plugin", "version": "1.3.1", - "description": "Braintrust plugins for LLM evaluation, logging, and observability", + "description": "Braintrust session tracing for Claude Code", "owner": { "name": "Braintrust", "email": "support@braintrust.dev" }, "plugins": [ - { - "name": "braintrust", - "description": "Enables AI agents to use Braintrust for LLM evaluation, logging, and observability. Provides correct API usage, working examples, and helper scripts.", - "source": "./plugins/braintrust", - "category": "development" - }, { "name": "trace-claude-code", "description": "Automatically trace Claude Code conversations to Braintrust. Captures user messages, assistant responses, and tool calls for observability.", diff --git a/src/plugins/claude/content/AGENTS.md b/src/plugins/claude/content/AGENTS.md index 900a7b1..b631f33 100644 --- a/src/plugins/claude/content/AGENTS.md +++ b/src/plugins/claude/content/AGENTS.md @@ -2,7 +2,7 @@ ## About this repository -This is the **Braintrust Claude Code plugin marketplace** - a repository that distributes Claude Code plugins for Braintrust integration. +This is the **Braintrust Claude Code plugin marketplace** - a repository that distributes Claude Code tracing for Braintrust. ### Structure @@ -11,23 +11,20 @@ claude-plugin/ ├── .claude-plugin/ │ └── marketplace.json # Marketplace catalog (lists available plugins) ├── plugins/ -│ ├── braintrust/ # Plugin: Braintrust evaluation & logging │ └── trace-claude-code/ # Plugin: Session tracing to Braintrust -└── evals/ # Evaluation suite for testing the plugins +└── evals/ # Evaluation suite for Braintrust MCP behavior ``` ### Plugins | Plugin | Description | |--------|-------------| -| `braintrust` | Enables AI agents to use Braintrust for LLM evaluation, logging, and observability. Includes MCP server config and the `troubleshoot-braintrust-mcp` skill. | | `trace-claude-code` | Forwards Claude Code lifecycle hooks to `bt trace hook --source claude-code`; the shared daemon builds and delivers traces. | ### Terminology - **Marketplace**: A repository with a `marketplace.json` that catalogs multiple plugins for distribution - **Plugin**: An installable unit with its own `.claude-plugin/plugin.json` manifest -- **Skill**: A capability within a plugin (e.g., `troubleshoot-braintrust-mcp` is a skill in the `braintrust` plugin) ## Style conventions diff --git a/src/plugins/claude/content/CONTRIBUTING.md b/src/plugins/claude/content/CONTRIBUTING.md index 63b40fe..b798a7e 100644 --- a/src/plugins/claude/content/CONTRIBUTING.md +++ b/src/plugins/claude/content/CONTRIBUTING.md @@ -1,4 +1,4 @@ -# Developing the Claude plugins +# Developing the Claude plugin This distribution tree is generated from the `braintrustdata/braintrust-coding-agent-plugins` monorepo. Make source changes @@ -9,7 +9,6 @@ there rather than editing the generated distribution repository. Load a plugin directly from the assembled tree: ```bash -claude --plugin-dir /path/to/repo/src/plugins/claude/content/plugins/braintrust claude --plugin-dir /path/to/repo/src/plugins/claude/content/plugins/trace-claude-code ``` @@ -29,7 +28,7 @@ cargo test --manifest-path bt-daemon/Cargo.toml --all-features --locked ## Evaluation suite -The `evals/` directory tests the Braintrust skills plugin: +The `evals/` directory tests Braintrust MCP behavior directly: ```bash cd evals @@ -38,6 +37,6 @@ uv run braintrust eval . ## Releases -Versions are stored in each plugin's `.claude-plugin/plugin.json`. Use the +The version is stored in the plugin's `.claude-plugin/plugin.json`. Use the monorepo's release workflow to bump versions, publish the generated distribution tree, tag the release, and create release notes. diff --git a/src/plugins/claude/content/README.md b/src/plugins/claude/content/README.md index 3b8aa1d..97f0b41 100644 --- a/src/plugins/claude/content/README.md +++ b/src/plugins/claude/content/README.md @@ -5,7 +5,7 @@ > Don't edit files here — make changes and file issues in that repository, and they > will be rebuilt into this one. -A Claude Code plugin marketplace for [Braintrust](https://braintrust.dev) integration - LLM evaluation, logging, observability, and session tracing. +A Claude Code plugin marketplace for tracing Claude Code sessions to [Braintrust](https://braintrust.dev). ## Prerequisites @@ -14,14 +14,13 @@ A Claude Code plugin marketplace for [Braintrust](https://braintrust.dev) integr ## Supported Claude surfaces -These marketplace plugins support Claude Code CLI and Claude Code mode in the -desktop app. They do not currently support the Cowork tab, which runs tools and +This marketplace plugin supports Claude Code CLI and Claude Code mode in the +desktop app. It does not currently support the Cowork tab, which runs tools and hooks inside a separate VM without the host's `bt` installation, Braintrust configuration, or environment variables. In Cowork, use the Braintrust connector provided through Claude for MCP access. -The connector is separate from the `braintrust` marketplace plugin. Automatic -Cowork session tracing is not currently supported. +Automatic Cowork session tracing is not currently supported. ## Installation @@ -31,22 +30,7 @@ Add the marketplace: claude plugin marketplace add braintrustdata/braintrust-claude-plugin ``` -Then install the plugins you need: - -## Plugins - -### braintrust - -Enables AI agents to use Braintrust for LLM evaluation, logging, and observability. - -- Query Braintrust projects, experiments, datasets, and logs -- Instrument your code with the Braintrust SDK and write evals - -```bash -claude plugin install braintrust@braintrust-claude-plugin -``` - -### trace-claude-code +Then enable tracing: Automatically traces Claude Code conversations to Braintrust through the shared Braintrust daemon. The plugin contains only a fail-open hook forwarder; `bt` @@ -63,6 +47,9 @@ Every registered lifecycle event is forwarded synchronously to `bt trace hook --source claude-code`, preserving per-session ordering. Hook failures never fail a Claude Code turn. +This marketplace does not install or configure the Braintrust MCP server. Use +your agent's native connector or MCP configuration when you want MCP access. + #### Additional root metadata For a persistent route, pass a JSON object to `bt trace enable claude diff --git a/src/plugins/claude/content/plugins/braintrust/.claude-plugin/plugin.json b/src/plugins/claude/content/plugins/braintrust/.claude-plugin/plugin.json deleted file mode 100644 index 2b6f680..0000000 --- a/src/plugins/claude/content/plugins/braintrust/.claude-plugin/plugin.json +++ /dev/null @@ -1,21 +0,0 @@ -{ - "$schema": "https://json.schemastore.org/claude-code-plugin-manifest.json", - "name": "braintrust", - "displayName": "Braintrust", - "description": "Enables AI agents to use Braintrust for LLM evaluation, logging, and observability. Provides correct API usage, working examples, and helper scripts for common operations.", - "version": "2.0.1", - "author": { - "name": "Braintrust", - "email": "support@braintrust.dev", - "url": "https://braintrust.dev" - }, - "homepage": "https://braintrust.dev", - "repository": "https://github.com/braintrustdata/braintrust-claude-plugin", - "license": "MIT", - "keywords": [ - "braintrust", - "evals", - "llm-evaluation", - "observability" - ] -} diff --git a/src/plugins/claude/content/plugins/braintrust/.mcp.json b/src/plugins/claude/content/plugins/braintrust/.mcp.json deleted file mode 100644 index b5d76ad..0000000 --- a/src/plugins/claude/content/plugins/braintrust/.mcp.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "mcpServers": { - "braintrust": { - "type": "http", - "url": "${BRAINTRUST_API_URL:-https://api.braintrust.dev}/mcp", - "headers": { - "Authorization": "Bearer ${BRAINTRUST_API_KEY}" - } - } - } -} diff --git a/src/plugins/claude/content/plugins/braintrust/skills/troubleshoot-braintrust-mcp/SKILL.md b/src/plugins/claude/content/plugins/braintrust/skills/troubleshoot-braintrust-mcp/SKILL.md deleted file mode 100644 index e4e582a..0000000 --- a/src/plugins/claude/content/plugins/braintrust/skills/troubleshoot-braintrust-mcp/SKILL.md +++ /dev/null @@ -1,57 +0,0 @@ ---- -name: troubleshoot-braintrust-mcp -description: | - This plugin auto-configures a "braintrust" MCP server. If you can't see it or reach it, activate this skill -version: 1.0.0 ---- - -This Claude plugin automatically sets up a Braintrust MCP connection. The connection reads the `BRAINTRUST_API_KEY` environment variable to establish the MCP connection. - -## Check the Claude surface first - -This plugin MCP configuration supports Claude Code CLI and Claude Code mode in -the desktop app. It does not configure MCP inside the Cowork tab because Cowork -runs in a separate VM without the host environment variables or network -context expected by `.mcp.json`. - -If the user is in Cowork, stop these plugin troubleshooting steps and direct -them to add or use the Braintrust connector through Claude. Do not ask them to -copy `BRAINTRUST_API_KEY` into the Cowork VM. - -## Troubleshooting steps - -### 1. Verify the environment variable is set - -Run `echo $BRAINTRUST_API_KEY` to check if the variable is exported - -API keys can be created at https://www.braintrust.dev/app/settings?subroute=api-keys - -### 2. Verify the API key is valid - -Test the key by calling the Braintrust API: - -```bash -curl -s https://api.braintrust.dev/api/self/me -H "Authorization: Bearer $BRAINTRUST_API_KEY" -``` - -- If valid: returns JSON with user info (id, email, organizations, etc.) -- If invalid: returns an authentication error - -NOTE: Even if you can curl the api via http, continue to attempt MCP setup. Http is just a troubleshooting tool, not a replacement for MCP - -### 3. Check if the MCP server is reachable - -If the key is valid but connection still fails, check if the MCP server is up: - -```bash -curl -s -o /dev/null -w "%{http_code}" https://api.braintrust.dev/mcp -``` - -- Any HTTP response (even 401 or 405) means the server is reachable -- Connection timeout or "connection refused" means the server may be down - -### 4. Contact support - -If nothing else works, encourage the user to reach out: -- Discord: https://discord.com/invite/6G8s47F44X -- Email: support@braintrust.dev diff --git a/src/plugins/claude/validate.sh b/src/plugins/claude/validate.sh index 0c6274e..33b2d0a 100755 --- a/src/plugins/claude/validate.sh +++ b/src/plugins/claude/validate.sh @@ -3,7 +3,7 @@ # validate.sh — Sanity-check a built Claude Code dist tree in $1 before publishing. # # Fails (non-zero) on the mistakes that would ship a broken marketplace: -# - missing marketplace manifest / plugin manifests / hooks / skill +# - missing marketplace manifest / tracing plugin manifest / hooks # - malformed JSON in any manifest # - marketplace entries whose `source` path does not exist in the tree # @@ -30,9 +30,6 @@ check_json "$MARKETPLACE" # Required files for each shipped plugin. required=( - "plugins/braintrust/.claude-plugin/plugin.json" - "plugins/braintrust/.mcp.json" - "plugins/braintrust/skills/troubleshoot-braintrust-mcp/SKILL.md" "plugins/trace-claude-code/.claude-plugin/plugin.json" "plugins/trace-claude-code/hooks/hooks.json" "plugins/trace-claude-code/hooks/forward.sh" @@ -42,6 +39,25 @@ for rel in "${required[@]}"; do case "$rel" in *.json) check_json "$TARGET_DIR/$rel";; esac done +python3 - "$MARKETPLACE" "$TARGET_DIR/plugins" <<'PY' \ + || fail "Claude marketplace must contain only the tracing plugin" +import json +import sys +from pathlib import Path + +with open(sys.argv[1]) as f: + plugins = json.load(f)["plugins"] + +assert [plugin["name"] for plugin in plugins] == ["trace-claude-code"] +assert sorted(path.name for path in Path(sys.argv[2]).iterdir() if path.is_dir()) == [ + "trace-claude-code" +] +PY + +if find "$TARGET_DIR" -name '.mcp.json' -print -quit | grep -q .; then + fail "Claude dist still contains an MCP proxy configuration" +fi + python3 - "$TARGET_DIR/plugins/trace-claude-code/hooks/hooks.json" <<'PY' \ || fail "Claude hooks do not all use the blocking daemon forwarder" import json diff --git a/src/plugins/codex/build.sh b/src/plugins/codex/build.sh index c85ca7b..750b121 100755 --- a/src/plugins/codex/build.sh +++ b/src/plugins/codex/build.sh @@ -4,7 +4,6 @@ # # The Codex marketplace consumes a repo whose ROOT is the marketplace: # .agents/plugins/marketplace.json marketplace manifest -# plugins/braintrust-codex-plugin/ skills plugin (MCP + skills) # plugins/trace-codex/ tracing plugin (thin daemon hook shims) # # The tracing plugin contains no tracing runtime or platform-specific binary; diff --git a/src/plugins/codex/content/.agents/plugins/marketplace.json b/src/plugins/codex/content/.agents/plugins/marketplace.json index 01fd1da..6ae6fec 100644 --- a/src/plugins/codex/content/.agents/plugins/marketplace.json +++ b/src/plugins/codex/content/.agents/plugins/marketplace.json @@ -1,21 +1,9 @@ { "name": "braintrust-codex-plugins", "interface": { - "displayName": "Braintrust Codex Plugins" + "displayName": "Braintrust Codex Plugin" }, "plugins": [ - { - "name": "braintrust", - "source": { - "source": "local", - "path": "./plugins/braintrust-codex-plugin" - }, - "policy": { - "installation": "AVAILABLE", - "authentication": "ON_INSTALL" - }, - "category": "Developer tools" - }, { "name": "trace-codex", "source": { diff --git a/src/plugins/codex/content/AGENTS.md b/src/plugins/codex/content/AGENTS.md index 2e4bf58..bd55923 100644 --- a/src/plugins/codex/content/AGENTS.md +++ b/src/plugins/codex/content/AGENTS.md @@ -4,18 +4,11 @@ Guidelines for AI agents working in this repo. ## Repo purpose -This repo is a monorepo of two independent [Codex marketplace plugins](https://developers.openai.com/codex/plugins): +This repo distributes one [Codex marketplace plugin](https://developers.openai.com/codex/plugins): -- `plugins/braintrust-codex-plugin/` — packages the [Braintrust MCP server](https://www.braintrust.dev/docs/integrations/developer-tools/mcp) plus a routing skill. -- `plugins/trace-codex/` — an opt-in plugin that traces Codex sessions to Braintrust (session, turn, and tool spans) via Codex lifecycle hooks. Do **not** merge tracing behavior into the MCP/skills plugin; they are separate, independently installable plugins. +- `plugins/trace-codex/` — an opt-in plugin that traces Codex sessions to Braintrust (session, turn, and tool spans) via Codex lifecycle hooks. -Both plugins are listed as separate entries in `.agents/plugins/marketplace.json` (the repo marketplace). - -Key files for the MCP/skills plugin: - -- `plugins/braintrust-codex-plugin/.codex-plugin/plugin.json` — plugin manifest (version, UI metadata, default prompts) -- `plugins/braintrust-codex-plugin/.mcp.json` — MCP server definition -- `plugins/braintrust-codex-plugin/skills/braintrust/` — agent skills exposed through the plugin +The plugin is listed in `.agents/plugins/marketplace.json` (the repo marketplace). Key files for the tracing plugin (see [`plugins/trace-codex/AGENTS.md`](plugins/trace-codex/AGENTS.md) for its architecture): @@ -26,10 +19,8 @@ Key files for the tracing plugin (see [`plugins/trace-codex/AGENTS.md`](plugins/ ## Making changes -- **Skills**: There is only one simple skill in this repo which handles routing and tool definitions, it should not be modified significantly. -- **MCP config**: edit `plugins/braintrust-codex-plugin/.mcp.json` to change the MCP server command or environment variables. -- **Plugin metadata**: edit the relevant `.codex-plugin/plugin.json` for display name, description, brand color, default prompts, etc. -- **Marketplace**: edit `.agents/plugins/marketplace.json` to change plugin entries, categories, or install policies. +- **Plugin metadata**: edit `plugins/trace-codex/.codex-plugin/plugin.json`. +- **Marketplace**: edit `.agents/plugins/marketplace.json` to change the plugin entry, category, or install policy. ## Releasing a new version diff --git a/src/plugins/codex/content/README.md b/src/plugins/codex/content/README.md index 9e18fc7..3e87ba8 100644 --- a/src/plugins/codex/content/README.md +++ b/src/plugins/codex/content/README.md @@ -1,11 +1,11 @@ -# Braintrust Codex Plugins +# Braintrust Codex Plugin > **This repository is generated.** It is built from > [braintrustdata/braintrust-coding-agent-plugins](https://github.com/braintrustdata/braintrust-coding-agent-plugins). > Don't edit files here — make changes and file issues in that repository, and they > will be rebuilt into this one. -Braintrust [Codex plugins](https://developers.openai.com/codex/plugins) — skills and daemon-backed session tracing. +Daemon-backed Braintrust session tracing for [Codex](https://developers.openai.com/codex/plugins). ## Quickstart @@ -13,10 +13,7 @@ Add this repo as a Codex plugin marketplace: ```bash codex plugin marketplace add braintrustdata/braintrust-codex-plugin -# OPTIONAL: TRACE CODEX PLUGIN codex plugin add trace-codex@braintrust-codex-plugins -# OPTIONAL: SKILLS PLUGIN -codex plugin add braintrust@braintrust-codex-plugins ``` The recommended tracing setup is: @@ -33,6 +30,5 @@ daemon. Restart Codex after setup. See the plugin's [README](/plugins/trace-codex/README.md) for details. -## skills plugin - -see the plugin's [README](/plugins/braintrust-codex-plugin/README.md) for details +This marketplace does not install or configure the Braintrust MCP server. Use +Codex's native connector or MCP configuration when you want MCP access. diff --git a/src/plugins/codex/content/plugins/braintrust-codex-plugin/.codex-plugin/plugin.json b/src/plugins/codex/content/plugins/braintrust-codex-plugin/.codex-plugin/plugin.json deleted file mode 100644 index d3c43dd..0000000 --- a/src/plugins/codex/content/plugins/braintrust-codex-plugin/.codex-plugin/plugin.json +++ /dev/null @@ -1,46 +0,0 @@ -{ - "name": "braintrust", - "version": "1.0.1", - "description": "Access Braintrust docs, logs, experiments, datasets, prompts, schemas, SQL queries, and permalinks through the official Braintrust MCP server.", - "author": { - "name": "Braintrust", - "url": "https://www.braintrust.dev/" - }, - "homepage": "https://www.braintrust.dev/docs/integrations/developer-tools/mcp", - "repository": "https://github.com/braintrustdata/braintrust", - "keywords": [ - "braintrust", - "mcp", - "evals", - "observability", - "tracing", - "experiments", - "datasets", - "prompts", - "logs" - ], - "skills": "./skills/", - "mcpServers": "./.mcp.json", - "interface": { - "displayName": "Braintrust", - "shortDescription": "Query Braintrust evals, logs, docs, and traces", - "longDescription": "Use the official Braintrust MCP server from Codex to search Braintrust documentation, resolve object names and URLs, inspect schemas, query experiments and logs with SQL, summarize experiment results, and generate shareable permalinks.", - "developerName": "Braintrust", - "category": "Coding", - "capabilities": [ - "Interactive" - ], - "websiteURL": "https://www.braintrust.dev/", - "privacyPolicyURL": "https://www.braintrust.dev/legal/privacy-policy", - "termsOfServiceURL": "https://www.braintrust.dev/legal/terms-of-service", - "defaultPrompt": [ - "Summarize my latest experiment results and compare them against the baseline.", - "Find recent production log traces with errors or low scores and give me shareable links to the worst ones.", - "How do I add Braintrust tracing to my app and run my first eval? Walk me through the SDK quickstart." - ], - "brandColor": "#2C1FEB", - "composerIcon": "./assets/favicon.svg", - "logo": "./assets/logo.png", - "screenshots": [] - } -} diff --git a/src/plugins/codex/content/plugins/braintrust-codex-plugin/.mcp.json b/src/plugins/codex/content/plugins/braintrust-codex-plugin/.mcp.json deleted file mode 100644 index 670ab52..0000000 --- a/src/plugins/codex/content/plugins/braintrust-codex-plugin/.mcp.json +++ /dev/null @@ -1,9 +0,0 @@ -{ - "mcpServers": { - "braintrust": { - "type": "http", - "url": "https://api.braintrust.dev/mcp", - "note": "Official Braintrust MCP server. Uses OAuth on first connection. Braintrust also supports API-key bearer auth; Codex TOML can configure bearer_token_env_var = \"BRAINTRUST_API_KEY\". For EU data plane, use https://api-eu.braintrust.dev/mcp." - } - } -} \ No newline at end of file diff --git a/src/plugins/codex/content/plugins/braintrust-codex-plugin/README.md b/src/plugins/codex/content/plugins/braintrust-codex-plugin/README.md deleted file mode 100644 index 8715c95..0000000 --- a/src/plugins/codex/content/plugins/braintrust-codex-plugin/README.md +++ /dev/null @@ -1,12 +0,0 @@ -# Braintrust Codex Plugin - -This plugin adds various Braintrust Skills and an mcp connection to codex. - -## Quickstart - -```bash -codex plugin marketplace add braintrustdata/braintrust-codex-plugin -codex plugin add braintrust@braintrust-codex-plugins -``` - -Braintrust plugins will now appear in `/skills` and `/mcp` diff --git a/src/plugins/codex/content/plugins/braintrust-codex-plugin/assets/favicon.svg b/src/plugins/codex/content/plugins/braintrust-codex-plugin/assets/favicon.svg deleted file mode 100644 index c496637..0000000 --- a/src/plugins/codex/content/plugins/braintrust-codex-plugin/assets/favicon.svg +++ /dev/null @@ -1,3 +0,0 @@ - - - diff --git a/src/plugins/codex/content/plugins/braintrust-codex-plugin/assets/logo.png b/src/plugins/codex/content/plugins/braintrust-codex-plugin/assets/logo.png deleted file mode 100644 index 93d89a7..0000000 Binary files a/src/plugins/codex/content/plugins/braintrust-codex-plugin/assets/logo.png and /dev/null differ diff --git a/src/plugins/codex/content/plugins/braintrust-codex-plugin/skills/braintrust/SKILL.md b/src/plugins/codex/content/plugins/braintrust-codex-plugin/skills/braintrust/SKILL.md deleted file mode 100644 index 2bf7041..0000000 --- a/src/plugins/codex/content/plugins/braintrust-codex-plugin/skills/braintrust/SKILL.md +++ /dev/null @@ -1,72 +0,0 @@ ---- -name: braintrust -description: Use the official Braintrust MCP server to search Braintrust docs, inspect projects, query logs and experiments, summarize eval results, and generate permalinks. ---- - -# Braintrust - -Use this skill when the user asks to work with Braintrust data, documentation, experiments, datasets, prompts, logs, traces, evals, or SDK setup. - -## Primary MCP server - -Use the `braintrust` MCP server. - -Default US data plane URL: - -`https://api.braintrust.dev/mcp` - -EU data plane URL: - -`https://api-eu.braintrust.dev/mcp` - -Self-hosted URL: - -Use the organization's configured Braintrust API URL with `/mcp`. - -## Authentication - -Prefer the plugin-provided OAuth flow for the remote MCP server. - -Braintrust also supports API-key bearer authentication. In Codex TOML, the verified configuration is: - -```toml -[mcp_servers.braintrust] -url = "https://api.braintrust.dev/mcp" -bearer_token_env_var = "BRAINTRUST_API_KEY" -``` - -Never print or expose the API key. - -## Tool guide - -Use `search_docs` for Braintrust documentation questions. - -Use `docs://sdk-install` before `search_docs` when the user asks to install Braintrust, set up tracing, add observability, or configure an eval. - -Use `resolve_object` to convert Braintrust names, IDs, and URLs into object metadata. - -Use `list_recent_objects` to discover accessible projects, experiments, datasets, prompts, functions, and other recent objects. - -Use `infer_schema` before writing SQL when the available fields are unclear. - -Use `sql_query` to query experiments, datasets, project logs, traces, and summaries. - -Use `summarize_experiment` for aggregate experiment metrics and baseline comparisons. - -Use `generate_permalink` when the user needs a shareable Braintrust link. - -## Workflow guidance - -For production debugging, first resolve the relevant project or object, infer schema if needed, query logs with `sql_query`, then generate permalinks for important traces. - -For experiment analysis, list or resolve the experiment, summarize it, compare against a baseline when available, then use SQL for row-level examples. - -For documentation and setup, read `docs://sdk-install` for SDK installation tasks and use `search_docs` for follow-up details. - -## CLI versus MCP - -Use MCP when the user wants conversational exploration of Braintrust data or docs. - -Prefer the `bt` CLI for repeatable workflows, CI gates, running evals, scripted operations, and setup flows that should be deterministic. - -The Braintrust MCP is currently best treated as read-oriented. Do not invent write capabilities. diff --git a/src/plugins/codex/validate.sh b/src/plugins/codex/validate.sh index d60309e..019ca1a 100755 --- a/src/plugins/codex/validate.sh +++ b/src/plugins/codex/validate.sh @@ -30,9 +30,6 @@ check_json "$MARKETPLACE" # Required files for each shipped plugin. required=( - "plugins/braintrust-codex-plugin/.codex-plugin/plugin.json" - "plugins/braintrust-codex-plugin/.mcp.json" - "plugins/braintrust-codex-plugin/skills/braintrust/SKILL.md" "plugins/trace-codex/.codex-plugin/plugin.json" "plugins/trace-codex/hooks/hooks.json" "plugins/trace-codex/bin/codex-hook.sh" @@ -42,6 +39,23 @@ for rel in "${required[@]}"; do case "$rel" in *.json) check_json "$TARGET_DIR/$rel";; esac done +python3 - "$MARKETPLACE" "$TARGET_DIR/plugins" <<'PY' \ + || fail "Codex marketplace must contain only the tracing plugin" +import json +import sys +from pathlib import Path + +with open(sys.argv[1]) as f: + plugins = json.load(f)["plugins"] + +assert [plugin["name"] for plugin in plugins] == ["trace-codex"] +assert sorted(path.name for path in Path(sys.argv[2]).iterdir() if path.is_dir()) == ["trace-codex"] +PY + +if find "$TARGET_DIR" -name '.mcp.json' -print -quit | grep -q .; then + fail "Codex dist still contains an MCP proxy configuration" +fi + python3 - "$TARGET_DIR/plugins/trace-codex/hooks/hooks.json" <<'PY' \ || fail "Codex hooks do not all use the daemon forwarders" import json