Skip to content

feat(harbor): upload standard verifier output - #716

Open
David Leen (dleen) wants to merge 1 commit into
braintrustdata:mainfrom
dleen:codex/harbor-verifier-evidence
Open

feat(harbor): upload standard verifier output#716
David Leen (dleen) wants to merge 1 commit into
braintrustdata:mainfrom
dleen:codex/harbor-verifier-evidence

Conversation

@dleen

Copy link
Copy Markdown

Summary

  • ingest Harbor standard verifier output (test-stdout.txt, test-stderr.txt, and ctrf.json)
  • show a bounded summary on the verification span and each score, with one shared verifier-output.json attachment
  • support multi-step trials and scoreless verifier failures while respecting attachments=none
  • apply the integration standard structured-key normalization and configured redact_patterns before upload
  • enforce safe bounded reads, reject symlinks and non-regular files, and share the trial attachment budget

Complexity

  • verifier output collector: 14 to 9
  • bounded file reader: 13 to 10
  • _sync_final_result: 56 to 48, matching upstream

Testing

  • nox -s test_harbor(latest): 34 passed with Harbor 0.22.0
  • nox -s test_harbor(0.16.0): 34 passed
  • focused pre-commit hooks: passed
  • pylint --errors-only on changed Python files: passed
  • strict live backfill confirmed verifier_output_summary and verifier-output.json on the verification span

@dleen
David Leen (dleen) force-pushed the codex/harbor-verifier-evidence branch from 02ccbe3 to 9e75b9f Compare August 28, 2026 01:45

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 02ccbe3675

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "Codex (@codex) review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "Codex (@codex) address that feedback".

execution_input["extra_instructions"] = extra_instructions
selected_artifacts, artifact_attachment_warnings = _artifact_attachments(result, self.config)
metadata["harbor"]["warnings"].extend(artifact_attachment_warnings)
attachment_bytes = _attachments_size(selected_artifacts)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Include trajectory images in the attachment budget

When trajectory_mode="atif", import_trajectory() can add image Attachments from atif.py to the agent spans, but this counter starts with only selected_artifacts and is not updated after the trajectory import. For example, with both size limits set to 5 MB, a 4 MB trajectory image followed by a 2 MB verifier attachment is accepted even though the trial uploads 6 MB, so the newly shared max_total_attachment_bytes limit is not actually enforced. Include ATIF attachment sizes in the remaining budget before collecting verifier and reward-detail evidence.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant