feat(eval): add bounded host experiment recorder - #2
Merged
Conversation
Why: - The bounded 20-task experiment does not require v1 compatibility, report scaffolding, or module-specific schemas. - Keeping those contracts would preserve unsupported product surface and inflate the recorder implementation. What: - Remove the v0.1 schema, fixtures, verifier, report template, distributed protocol documents, and toolkit manifest. - Leave only repository metadata before adding the clean-slate recorder. Validation: - git diff --cached --check - No runtime tests apply to this deletion-only commit
Why: - Eval needs a small host-owned instrument before product surfaces or analysis are justified. - The experiment must preserve primary task outcomes when input or private storage is unavailable. What: - Add the observe-only evalctl contract and a twenty-slot typed journal. - Add strict duplicate-key and integer decoding with one stored-row schema. - Add rooted private storage, exclusive locking, atomic replacement, and commit-aware durability reporting for Darwin, Linux, and Windows. Validation: - go test ./... - go test -race ./... - go vet ./... - GOOS=darwin/linux/windows cross-build and vet - Manual observe smoke with contiguous slots
Why: - The bounded experiment is useful only if malformed input and storage failures cannot corrupt or expand the journal. What: - Cover strict decoding, field relationships, canonical rows, and the twenty-slot limit. - Exercise concurrent writers, invalid state, permissions, symlinks, hard links, lock timeout, and rooted path replacement. - Inject partial writes and pre/post-replace failures to distinguish skipped writes from recorded durability uncertainty. - Add Windows owner, protected DACL, reparse, hard-link, and LockFileEx coverage. Validation: - go test ./... - go test -race ./... - go vet ./... - Darwin, Linux, and Windows test binaries compiled
Why: - The experiment recorder needs remote proof on every supported operating system before the host policy is enabled. What: - Run formatting, vet, tests, and evalctl builds on Linux, macOS, and Windows. - Run the race detector on Linux. - Reject reintroduction of removed Plugin, compatibility, analyzer, validator-command, and local-path surfaces. Validation: - Workflow YAML parsed locally - Surface commands passed locally - go test, race, and vet passed locally
Why: - The recorder needs an explicit experimental population and human decision boundary without restoring product documentation layers. What: - Document the observe-only contract, private journal, and failure semantics. - Define eligible host tasks, silent terminal recording, denominator ownership, and the twentieth-task stop. - Record promotion, privacy review, and cohort-size boundaries. Validation: - Markdown and links reviewed locally - go test ./... passed - git diff --check passed
7 tasks
Why: - Windows checkout converted Go sources before the shared gofmt gate, so the Windows job failed before tests ran. What: - Mark Go sources as LF in repository attributes across all runners. Validation: - gofmt -l . produced no output - git check-attr reports eol=lf for Unix and Windows Go files
Why: - Elevated Windows runners can use a token default owner that differs from the current-user SID. - The DACL accessor boolean reports whether the ACL was defaulted, not whether it is present. What: - Set the current user explicitly as owner and sole full-control trustee on new private objects. - Check SE_DACL_PRESENT through descriptor control bits before inspecting state-root access. Validation: - Windows amd64 store and CLI test binaries compiled - Windows vet passed - Local go test and race test passed - GitHub Windows execution pending on the updated head
Why: - GitHub Windows temp directories inherit broader ACLs than an accepted private state root. - Windows may block a state-root rename while os.Root handles are open instead of allowing the Unix path-move scenario. What: - Run CLI tests against recorder-created private state roots. - Keep the Unix anchored-rename test and add a Windows assertion for either a blocked move or an anchored commit. Validation: - Local unit and race tests passed - Windows amd64 store and CLI test binaries compiled - Windows vet passed - GitHub Windows execution pending on the updated head
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6618f0391d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Why: - An absolute-path miss during a state-root move could classify an anchored Journal as empty and risk replacing prior rows. What: - Treat the Journal as absent only when rooted and absolute entries are both absent. - Reject one-sided absence and file-identity mismatch as unsafe. Validation: - go test -count=1 ./internal/store - go test -count=1 ./... - go test -race -count=1 ./...
Why: - The rooted-read regression requires a deterministic state-root move that Unix permits while os.Root handles remain open. What: - Seed a Journal, move its state root, and assert the anchored read fails closed without changing prior bytes. Validation: - go test -count=1 ./internal/store - go test -race -count=1 ./... - git diff --check
Why: - Native Windows normally provides USERPROFILE rather than HOME, so the default recorder path could be skipped as unsafe. What: - Resolve explicit and XDG roots before a platform-aware user-home fallback. - Inject os.UserHomeDir through the internal CLI runtime for deterministic tests. Validation: - go test -count=1 ./internal/state ./internal/cli - go vet ./... - GOOS=windows GOARCH=amd64 go vet ./... - Darwin, Linux, and Windows evalctl cross-builds
Why: - Root precedence and the Windows USERPROFILE fallback need regression coverage independent of the host environment. What: - Cover explicit, XDG, resolver-error, and invalid-home cases with injected dependencies. - Add a Windows integration test that records without --state-root when only USERPROFILE is set. Validation: - go test -count=1 ./internal/state ./internal/cli - go test -race -count=1 ./internal/state ./internal/cli - Windows state, CLI, and Store test binaries cross-compiled - Windows execution pending GitHub Actions
Why: - The default Journal documentation described only the Unix HOME fallback. What: - Distinguish the shared XDG path from Darwin/Linux HOME and Windows USERPROFILE fallbacks. Validation: - Markdown source reviewed - git diff --check
Why: - Upgraded checkouts may still contain previously ignored private observations that git add -A must never stage. What: - Keep only the two retired checkout-local private paths ignored. - Add a CI canary that verifies they remain untracked, ignored, and absent from dry-run staging. Validation: - git check-ignore passed for both root-only paths - git ls-files reported no tracked legacy private data - workflow static check passed when actionlint was available - git diff --check
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
목적
Eval을 아직 Toolkit 제품으로 승격하지 않고, 다음 20개 eligible root Codex Task에서 기록 지속성과 실사용 부담만 확인하는 bounded Host experiment로 교체합니다. 기존 Native Agent MVP는 호환 계층, 분석 명령, Plugin 자동 선택, 모듈별 확장 계약을 한 번에 제품화해 실제 데이터가 없는 상태에서 책임 범위를 키웠습니다. 이 PR은 실험 실행에 필요한 recorder와 private-store 안전 경계만 남깁니다.
변경 내용
evalctl --version과 항상 best-effort인evalctl observe [--state-root ABS]로 제한했습니다.eval-experiment/v1typed Journal을 추가했습니다.os.Root로 commit 경로를 고정하고 kernel lock, private ownership/mode, symlink/reparse-point, hard-link, temp sync, atomic replace, directory sync 경계를 Darwin/Linux/Windows에 구현했습니다.설계 판단
used:true + version:null을 허용해 usage cohort에는 포함하고 exact-version 판단 자료로는 쓰지 않음기존 PR #1 리뷰 대응
scripts/verify.sh를 제거했고 CI는 저장소 자체 Go 경계만 사용합니다.recorded+durability:"unconfirmed"로 반환합니다.int64로 검증하며 fractional/exponent 표현을 거부합니다.toolkit-module.json을 제거했습니다.검증 방법
검증 결과
go test -count=1 ./...통과go test -race -count=1 ./...통과go vet ./...통과gofmt및git diff --check통과278d407의 GitHub Actions run 32979861433에서 Linux/macOS/Windows format·vet·test·build, Linux race, clean-slate surface 전부 통과codex exec --ephemeral --json에서 completed/failed/abandoned 각각 정확히 1행 기록 확인영향 범위
리스크
durability:"unconfirmed"로 반환합니다.롤백 방법
evalctlprovisioning을 중단하면 즉시 정지합니다.리뷰어가 중점적으로 봐야 할 부분
skipped가 아니라 committed/unconfirmed로 분리되는지os.Root기반 rename과 lock 아래 Journal 전체 검증이 경로 교체·동시 writer를 막는지Agent Note