Skip to content

fix: verify launchd server startup after install - #63

Open
git-ksk wants to merge 1 commit into
CursorTouch:mainfrom
git-ksk:fix/launchd-startup-acceptance
Open

fix: verify launchd server startup after install#63
git-ksk wants to merge 1 commit into
CursorTouch:mainfrom
git-ksk:fix/launchd-startup-acceptance

Conversation

@git-ksk

@git-ksk git-ksk commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Verify a freshly bootstrapped launch agent by waiting for the configured TCP endpoint to accept connections instead of treating launchctl bootstrap success as server startup success.
  • Use a bounded 10-second readiness window, returning immediately once the endpoint is healthy.
  • Reject delayed exits, non-zero exit codes, and terminating signals instead of optimistically accepting a merely-running process.
  • Retry transient launchctl print failures while polling and only parse top-level launchd fields.
  • Clean up a newly installed broken KeepAlive agent after verified startup failure.
  • Preserve the previous plist/loaded state when a --force reinstall fails, including rollback after startup verification failure.
  • During --force, use a short bounded retry when determining whether the old job is loaded so one transient launchctl print failure is not misclassified as unloaded.
  • For same-port force reinstalls, briefly wait for the old listener to release the socket after bootout.

Root cause

install previously checked only the return code from launchctl bootstrap. Bootstrap confirms that launchd accepted the job definition; it does not guarantee that the server process stayed alive or reached its listen endpoint. A short process-state-only check also cannot reliably distinguish a healthy server from a process that is still importing and will fail shortly afterward.

Validation

  • focused launchd tests: 14 passed
  • full suite: 347 passed
  • polling/deadline/sleep paths are covered with a fake clock rather than only zero-timeout single-iteration tests
  • git diff --check: clean
  • no added line exceeds the repository's 100-character limit
  • Ruff baseline is not worsened
  • real launchd acceptance:
    • healthy direct executable: ready in about 2.7-3.4s
    • delayed exit: rejected in about 4.3-4.6s
    • SIGTERM: rejected correctly
    • explicit port conflict: rejected immediately
    • force rollback: previous plist restored byte-for-byte and previously loaded service restarted
    • same-port force reinstall: succeeds after old socket release
  • production com.macos-mcp.server was left unchanged; final check showed it running with the same plist SHA256 and 127.0.0.1:8100 accepting connections
  • throwaway acceptance labels/files were cleaned up

Residual cold uvx note

An isolated cold uvx bootstrap exceeded the 10-second readiness window because dependency download dominated startup; prewarming itself took about 31s. This PR intentionally does not change TCC, so isolated uvx healthy readiness timing was not measured after that prewarm. The installed direct executable reached readiness in about 2.7-3.4s.

Non-changes

@Jeomon

Jeomon commented Aug 29, 2026

Copy link
Copy Markdown
Member

I checked this out and ran the verification function against real launchd jobs on macOS 26.5.2. The immediate-exit detection does work, but I don't think the 1s budget catches the failure mode this is aimed at.

Timings from this repo's venv:

  • import macos_mcp.__main__ (warm): ~2.1s
  • healthy server accepting connections: 3.41s
  • server failing on a port conflict (rc=3): 3.76s
  • verification budget in this PR: 1.0s

At one second a healthy server and a doomed one look identical, because both are a live process still importing, so launchctl print reports state = running with last exit code = (never exited).

I bootstrapped a throwaway agent whose program stays alive 3.8s and then exits 3, to mimic the port-conflict timing, and called _wait_for_launch_agent_start directly:

started=True  detail='process is running; endpoint is still starting'  (after 1.04s)

So install still prints a success line for a server that never comes up. It also means the accepting connections branch is unreachable in practice, since the endpoint needs 3.4s to answer.

The instant-exit case is fine. launchd registers it at about 0.32s:

t=0.17s | state = running         | last exit code = (never exited)
t=0.32s | state = spawn scheduled | last exit code = 78: EX_CONFIG

and the 78: EX_CONFIG and (never exited) formats the parser relies on are correct on macOS 26. But install calls _resolve_program() first, so exec-level failures are the least likely mode. The realistic ones (an explicit --port that is occupied, an import error, a permission crash) all land after 2s.

On the tests: every case passes timeout=0, so the loop body runs exactly once. The deadline, the sleep(0.1) polling and the saw_running accumulation across iterations are never exercised, which is why the timing mismatch doesn't surface. The 7 tests pass, they just can't fail on this.

Two other things I noticed:

  1. A verified failure leaves a crash-looping agent installed. The exception is raised after the plist is written and bootstrapped, and the plist sets KeepAlive: True, so launchd keeps respawning the broken job on the throttle interval and it comes back at every login. Worth doing a bootout and removing the plist on that path, or at least pointing at macos-mcp uninstall next to the log path.

  2. The first non-zero launchctl print fails the install outright with no retry, inside a loop that exists to poll. Print succeeded at t=0.03s in every run here so this isn't something I hit, but it turns a launchd hiccup into a failed install.

Minor: _launchctl_field matches on line.strip(), and real output contains nested state = active lines (two of them under Finder). First-match ordering keeps it correct today, but anchoring on the top-level indentation would make that guarantee explicit.

Suggestion: keep the exit-code and signal rejection as they are, and make the endpoint probe the actual success signal with a bound nearer 8-10s. The healthy path returns as soon as the port answers (3.4s here), so it isn't a fixed wait, and the failure path then produces a real error instead of an optimistic success line. A test that steps a fake clock through the loop with timeout > 0 would cover the part that is currently untested.

@git-ksk
git-ksk force-pushed the fix/launchd-startup-acceptance branch from 4d7b253 to 7cd8d3a Compare August 29, 2026 12:57
@git-ksk

git-ksk commented Aug 29, 2026

Copy link
Copy Markdown
Contributor Author

@Jeomon Thank you for checking this against real launchd jobs and for the detailed timings. Your point about a 1s process-state check being unable to distinguish a healthy server from a doomed one was exactly right.

I updated the PR so startup success now requires the configured endpoint to accept connections within a bounded 10s window, returning as soon as it is ready. The polling path is exercised with a fake clock, including deadline/sleep behavior rather than only a zero-timeout single iteration.

I also incorporated the other failure-path points you called out: verified failures clean up the broken KeepAlive agent, transient launchctl print failures are retried during startup polling, launchd field parsing is restricted to top-level fields, and failed --force reinstalls roll back the previous plist/loaded state. As a final hardening step, the initial --force check for whether the old job is loaded now uses a short bounded retry as well, so one transient print failure is not treated as unloaded.

Real launchd acceptance now has the healthy direct executable ready in about 2.7-3.4s; a delayed exit is rejected around 4.3-4.6s; SIGTERM and an explicit port conflict are rejected correctly; rollback restores the prior plist byte-for-byte and restarts the previously loaded service; and a same-port force reinstall succeeds after allowing the old socket to release.

One residual note: an isolated cold uvx bootstrap exceeded 10s because dependency download dominated startup, with prewarming itself around 31s. This PR intentionally leaves TCC untouched, so I did not measure isolated uvx healthy readiness after that prewarm. The installed direct executable is the path measured above.

Thanks again for the concrete reproduction and review; it materially improved the startup acceptance behavior without broadening the PR into TCC/signing or unrelated launchd work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants