Availability fixes - #60
Merged
Merged
Conversation
A connected but unresponsive client could stall the initial window size and playback sends inside the job select loop. That blocked the stop arm, so job stop cancelled the token to no effect and the job stayed running rack-wide. The mux now accepts a batch of initial messages delivered from the client's own writer task, and the select loop no longer awaits any client socket. Cancellation also interrupts a writer task mid send instead of waiting for the next message. Streamed output sends get the same treatment via a 60 second timeout. A consumer that makes no TCP progress for that long is treated as dead and the job is cancelled, matching the existing send error policy. The close handshake is bounded too, so the streaming task cannot outlive the job while holding the socket. Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com>
The proxy preferred its home sled but fell back to an arbitrary sticky default when home was missing from the sled map. If the home sush server is unreachable, the sled is effectively dead, and there is a second switch. Untargeted requests now go to home or fail with 503, and the sticky default is gone. A proxy with no configured home, which happens only in development, still picks any known sled. Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com>
Reads reset the streaming drain timeout so a slow consumer can finish the tail, but that let an escaped descendant that keeps writing run the job forever. Process death now also arms an absolute one minute limit, after which the job ends with a warning and its recorded exit status. Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com>
A job whose target matches no sled enqueues rack-wide but runs nowhere, and both server and client waits are ten silent minutes. Job start now checks explicit cubby and baseboard targets against the rack inventory before the signing ceremony and asks for confirmation on a miss, since a sled cut off from gossip may still be directly reachable. Offline signing, JSON output, and an unavailable inventory skip the prompt. A watch that has seen no status from any sled after fifteen polls says so once instead of spinning silently. Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com>
Clippy warnings that only fail in CI make just lint an unreliable gate. Match the CI flags. Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com>
A client that stopped reading could park its writer task in an untimed close, and a job that ended naturally never cancelled its writers at all, leaking a task and a socket per stalled client. Closing is now bounded by a timeout and dropping the mux cancels every writer, so a wedged client costs at most a few seconds of cleanup. Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com>
A streaming job with no consumer fills the pipe and blocks with the process alive, so neither the drain timers nor the linger ever arm, and the job stays Running forever. Kill the job if no consumer attaches within the linger period. The linger still governs after EOF, so posthumous attachment is unaffected. Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com>
Job stop sent SIGKILL immediately and gave up after one attempt, even if the kill failed and left the process alive with the stop arm permanently disabled. Send SIGTERM first so the job can clean up, upgrade to SIGKILL after a grace period, and retry a failed kill a few times before giving up. Resolves the 2-stage stop TODO. Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com>
Watch polls were untargeted, so the proxy sent them to its home sled: settlement waited on gossip instead of the executing sled, and alternating requests between sleds could bounce 401s while a login gossips. The job_status endpoint now accepts the routing hint, and both watch loops pass it when the target names exactly one baseboard. Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com>
A playback error left the shared output file cursor mid-file, so the next recorded write would overwrite the tail of the record and silently diverge from the audit hash. Re-seek to the end on error, and cancel the job if even that fails. Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com>
Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com>
The fetch logged in via the target sled but sent the status request unrouted, recreating the login bounce that watch polls just fixed. Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com>
A stopped job that ignored SIGTERM but closed its pipes broke out of the loop before the grace period expired, and the reap then waited forever. The reap select now escalates too. Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com>
A proxy answering 502 or 503 was invisible in the switch zone logs. Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com>
The Ctrl-C stop from a routed watch went unrouted to the home sled, costing a second login. The via hint rides JobStopParams because an endpoint gets one query struct; sleds ignore it as usual. Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Various robustness fixes on both server and client side.