fix(sandbox): stop the dev-server probe leaking a keep-alive connection every tick - #6830
fix(sandbox): stop the dev-server probe leaking a keep-alive connection every tick#6830pedrofrxncx wants to merge 1 commit into
Conversation
…on every tick head() builds a fresh http.Transport on every probe tick (every 1s while booting, every 30s once online) and never closes it or drains its idle connection pool. With keep-alive left on, each successful HEAD leaves an idle persistConn + readLoop goroutine open indefinitely, since the Transport itself is discarded but the goroutine keeps it (and the socket) alive. Over a long-running sandbox this accumulates goroutines and file descriptors without bound. Set DisableKeepAlives on the per-tick Transport so the connection closes right after each HEAD instead of idling forever.
There was a problem hiding this comment.
2 issues found across 2 files
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="packages/sandbox/daemon-go/internal/probe/probe_test.go">
<violation number="1" location="packages/sandbox/daemon-go/internal/probe/probe_test.go:22">
P3: On IPv6-only hosts, this port extraction fails because `httptest.NewServer` can return a URL such as `http://[::1]:<port>`. Parse `srv.URL` with `net/url` and `net.SplitHostPort` so the regression test works on every loopback address supported by `DialLoopback`.</violation>
<violation number="2" location="packages/sandbox/daemon-go/internal/probe/probe_test.go:27">
P3: This asserts on the global `runtime.NumGoroutine()`, so unrelated goroutines (parallel tests, race-detector/GC wakeups, runtime background activity) can transiently push the count above `before+5`, making the test flaky even when nothing leaks. The threshold and 2s wait reduce but do not remove that risk. Tighten it by asserting the specific readLoop goroutine count (e.g. report via the leak that this guards), or widen the tolerance and note the flakiness.</violation>
</file>
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
| t.Fatalf("parse port from %q: %v", srv.URL, err) | ||
| } | ||
|
|
||
| before := runtime.NumGoroutine() |
There was a problem hiding this comment.
P3: This asserts on the global runtime.NumGoroutine(), so unrelated goroutines (parallel tests, race-detector/GC wakeups, runtime background activity) can transiently push the count above before+5, making the test flaky even when nothing leaks. The threshold and 2s wait reduce but do not remove that risk. Tighten it by asserting the specific readLoop goroutine count (e.g. report via the leak that this guards), or widen the tolerance and note the flakiness.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/sandbox/daemon-go/internal/probe/probe_test.go, line 27:
<comment>This asserts on the global `runtime.NumGoroutine()`, so unrelated goroutines (parallel tests, race-detector/GC wakeups, runtime background activity) can transiently push the count above `before+5`, making the test flaky even when nothing leaks. The threshold and 2s wait reduce but do not remove that risk. Tighten it by asserting the specific readLoop goroutine count (e.g. report via the leak that this guards), or widen the tolerance and note the flakiness.</comment>
<file context>
@@ -0,0 +1,49 @@
+ t.Fatalf("parse port from %q: %v", srv.URL, err)
+ }
+
+ before := runtime.NumGoroutine()
+
+ for i := 0; i < 20; i++ {
</file context>
| })) | ||
| defer srv.Close() | ||
|
|
||
| port, err := strconv.Atoi(srv.URL[len("http://127.0.0.1:"):]) |
There was a problem hiding this comment.
P3: On IPv6-only hosts, this port extraction fails because httptest.NewServer can return a URL such as http://[::1]:<port>. Parse srv.URL with net/url and net.SplitHostPort so the regression test works on every loopback address supported by DialLoopback.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/sandbox/daemon-go/internal/probe/probe_test.go, line 22:
<comment>On IPv6-only hosts, this port extraction fails because `httptest.NewServer` can return a URL such as `http://[::1]:<port>`. Parse `srv.URL` with `net/url` and `net.SplitHostPort` so the regression test works on every loopback address supported by `DialLoopback`.</comment>
<file context>
@@ -0,0 +1,49 @@
+ }))
+ defer srv.Close()
+
+ port, err := strconv.Atoi(srv.URL[len("http://127.0.0.1:"):])
+ if err != nil {
+ t.Fatalf("parse port from %q: %v", srv.URL, err)
</file context>
Source: bug found while auditing
packages/sandbox/daemon-gofor resource leaks (my W3 focus area this tick).Payoff: the daemon's dev-server prober (
internal/probe/probe.go) polls the sandboxed app every 1s while it's booting/offline and every 30s once online, for the entire life of the sandbox process. Eachhead()call builds a brand-newhttp.Transportfor a single HEAD request but never setsDisableKeepAlivesor closes it. If the target server keeps the connection alive (the common case), the resulting idlepersistConnand itsreadLoopgoroutine stay open indefinitely, holding the discardedTransport(and a socket) alive with it — since nothing ever tells that connection to close. Over a long-running sandbox this leaks one goroutine + one file descriptor per successful probe tick, unbounded, which is exactly the kind of drip that eventually causes FD exhaustion or OOM in a daemon meant to stay up for the whole session (this daemon's own health probe kills the pod on a single miss, so a slow FD leak surfacing as a missed probe is a real production risk, not theoretical).Failure scenario: long-running sandbox with a dev server that supports keep-alive (nearly all HTTP servers do by default) → each probe tick after the first leaves one more idle connection + goroutine open → goroutine/FD count grows without bound for the sandbox's lifetime.
Fix: set
DisableKeepAlives: trueon the per-tickTransport— correct here since the transport is discarded after a single request anyway, so there's no pooling benefit to lose, only the connection closes promptly instead of idling forever.Regression test:
internal/probe/probe_test.go—TestHeadDoesNotLeakKeepAliveGoroutinespins up anhttptest.Server, callshead()20 times, and asserts the goroutine count doesn't grow. Verified it fails without the fix (goroutine count went from 3 → 63 after 20 calls) and passes with it.Reviewer command:
cd packages/sandbox/daemon-go && go test ./internal/probe/... -run TestHeadDoesNotLeakKeepAliveGoroutine -vLocally verified:
gofmt -l,go vet ./internal/probe/...,go build ./..., and the targeted test above all pass. Full CI validates the rest.Summary by cubic
Stops the dev-server probe from leaking an idle keep-alive connection and its goroutine on every tick, which could exhaust file descriptors or memory in long-running sandboxes.
Bug Fixes
DisableKeepAlives: trueon the per-tickTransportso the connection closes after each HEAD request.TestHeadDoesNotLeakKeepAliveGoroutineto guard against regression.Written for commit 3b7b801. Summary will update on new commits.