docs(eval): define terminal observation boundary - #3
Merged
Conversation
Why: - Assistant turn completion could be mistaken for a terminal task outcome, causing premature or missing observations across resumable tasks. What: - Define nonterminal waiting and recoverable Ward or Seal stop states. - Narrow completed, failed, and Host-supplied abandoned outcomes. - Require one observation only after the eventual terminal outcome. - Document the Ward state-root preflight and aborted-run restart boundary. Validation: - gofmt -l . - go vet ./... - go test ./... - go test -race ./... - go build ./cmd/evalctl - Ward exact-root permission smoke and multi-turn Host E2E
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Owner
Author
|
@codex review |
|
Codex Review: Didn't find any major issues. Bravo. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
목적
assistant 턴 종료를 Task terminal로 오인하면, 사용자 입력이나 설정을 기다렸다가 재개되는 Task에서 Eval 관찰이 너무 일찍 기록되거나 누락될 수 있습니다. 이 PR은 CLI·JSON 스키마·Journal 형식·20행 제한을 바꾸지 않고 Native Agent와 Host가 적용할 terminal 판정 경계를 문서화합니다.
변경 내용
completed,failed,abandoned의 terminal 의미를 좁히고, 사용자 침묵으로abandoned를 추론하지 않도록 명시했습니다.evalctl observe를 호출하지 않고, 같은 Task가 재개되면 최종 terminal에서 한 번만 호출하도록 명시했습니다.wardprofile의 Eval 전용 writable root preflight와 실패한 실험 run의 중단·격리·새 activation 경계를 문서화했습니다.설계 판단
abandoned의 Agent 기록과 terminal 이후 reopen correction은 지원하지 않습니다.검증 방법
검증 결과
gofmt -l .결과 없음go vet ./...통과go test ./...통과go test -race ./...통과go build ./cmd/evalctl통과wardprofile과 Eval exact writable root 적용 확인status=recorded,slot=1,durability=confirmed, 디렉터리0700, 파일0600확인completed1회, 재개 경로 없는failed1회, 사용자 무응답 시abandoned미생성을 확인영향 범위
README.md의 Host experiment 운영 계약wardprofile preflight와 실험 run 재시작 절차~/.codex/config.toml과~/.codex/AGENTS.md는 Host별 설정이므로 이 PR에 포함하지 않음리스크
wardprofile을 선택하는 데 의존합니다.abandoned는 Journal에 Agent observation으로 남지 않습니다.롤백 방법
리뷰어가 중점적으로 봐야 할 부분
failed와 Host-suppliedabandoned의 권한 경계가 명확한지Agent Note
evalctl observe를 한 번만 호출합니다.abandoned, run 중단·격리를 관리해야 합니다.AGENTS.md로딩과 managed permission profile 동작에 의존합니다.