fix(tests): unwrap the CLI envelope in two E2E test classes - #635
Merged
Conversation
`_json_ok` returns the whole `{"status": ..., "data": {...}}` envelope, but
two classes read it as if it were the payload.
TestE2ENotificationSubscriptions (introduced in ae840ba, #615) indexed
`data["subscriptions"]` / `["errors"]` / `["project_wide_excluded"]`
directly, so all 7 tests raised `KeyError: 'subscriptions'` on every run.
The KeyError fired before the data-dependent `pytest.skip` paths could be
evaluated, so the class was dead rather than honestly skipped. Unwrap once
in a `_payload()` helper instead of per call site -- the envelope/payload
confusion is what shipped the bug, and a `["data"]` on each call invites it
again.
`test_data_app_secrets_round_trip` reached for `set_result["raw_output"]`,
a key that exists nowhere in the codebase. That killed three assertions
that plaintext secrets must never appear in CLI output -- they could only
ever raise KeyError, never check anything. Keep the raw CliRunner result so
the assertions can inspect the command's actual stdout.
Verified live against project 5946: the notification class now reports
5 passed, 3 skipped, with the skips being exactly the three data-dependent
tests the class docstring describes. These E2E tests need E2E_API_TOKEN and
do not run in CI, which is why the breakage shipped unnoticed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
_json_ok(tests/test_e2e.py) returns the whole{"status": ..., "data": {...}}CLI envelope. Two test classes read it as if it were the payload.1.
TestE2ENotificationSubscriptions— all 7 tests broken since #615Introduced by ae840ba (#615). The tests index
data["subscriptions"],data["errors"]anddata["project_wide_excluded"]directly, so every test raisedKeyError: 'subscriptions'on every run.The command itself is fine — verified live,
kbagent --json notification listreturns a correct envelope withdata.subscriptionspresent.Worse than a plain failure: the class is deliberately designed so its data-dependent assertions
pytest.skipon a project with no subscriptions (see the class docstring on vacuous passes). TheKeyErrorfired before those skip paths could be evaluated, so the class was dead rather than honestly skipped.Fixed by unwrapping once in a
_payload()helper rather than adding["data"]to each of the 10 call sites — the envelope/payload confusion is exactly what shipped the bug, and a per-call-site["data"]invites the same slip again. This matches the file's existing convention (_run_ok+ explicit["data"]).2.
test_data_app_secrets_round_trip— three dead security assertionsFound by scanning the rest of the file for the same mistake (AST scan over every
_json_ok-derived variable, not an eyeball pass).The test reached for
set_result["raw_output"]/list_result["raw_output"]/get_result["raw_output"]— a key that exists nowhere in the codebase (grep -rn raw_output src/→ 0 hits). Those three assertions state that a plaintext secret must never appear in CLI output. They could only ever raiseKeyError, never check anything.The intent was clearly the command's raw stdout, which
_json_okdiscards. Fixed by keeping the rawCliRunnerresult so the assertions inspect.output.Verification
Before:
7 failed, 1 passedAfter:
5 passed, 3 skippedThe 3 skips are exactly the data-dependent ones the class docstring describes (project 5946 has zero notification subscriptions), confirming the skip paths now actually trigger instead of being masked by the
KeyError.An AST scan of the whole file reports 0 remaining envelope/payload confusions.
Also clean:
ruff check,ruff format --check,ty check, and collection of all 115 tests in the file.Not verified live:
test_data_app_secrets_round_tripskips in my environment — besides the token it needsE2E_DATA_APP_GIT_REPO_PUBLIC, which is not set. Static check only:set_data_app_secretsreturns key names insecrets_set, never values, so the revived assertions should pass — but they still need a run with that variable set.Notes
E2E_API_TOKEN/E2E_URLand do not run in CI, which is why the breakage shipped unnoticed.