Skip to content

fix(sandbox): stop the dev-server probe leaking a keep-alive connection every tick - #6830

Open
pedrofrxncx wants to merge 1 commit into
mainfrom
fix/probe-keepalive-goroutine-leak-w3
Open

fix(sandbox): stop the dev-server probe leaking a keep-alive connection every tick#6830
pedrofrxncx wants to merge 1 commit into
mainfrom
fix/probe-keepalive-goroutine-leak-w3

Conversation

@pedrofrxncx

@pedrofrxncx pedrofrxncx commented Sep 1, 2026

Copy link
Copy Markdown
Collaborator

Source: bug found while auditing packages/sandbox/daemon-go for resource leaks (my W3 focus area this tick).

Payoff: the daemon's dev-server prober (internal/probe/probe.go) polls the sandboxed app every 1s while it's booting/offline and every 30s once online, for the entire life of the sandbox process. Each head() call builds a brand-new http.Transport for a single HEAD request but never sets DisableKeepAlives or closes it. If the target server keeps the connection alive (the common case), the resulting idle persistConn and its readLoop goroutine stay open indefinitely, holding the discarded Transport (and a socket) alive with it — since nothing ever tells that connection to close. Over a long-running sandbox this leaks one goroutine + one file descriptor per successful probe tick, unbounded, which is exactly the kind of drip that eventually causes FD exhaustion or OOM in a daemon meant to stay up for the whole session (this daemon's own health probe kills the pod on a single miss, so a slow FD leak surfacing as a missed probe is a real production risk, not theoretical).

Failure scenario: long-running sandbox with a dev server that supports keep-alive (nearly all HTTP servers do by default) → each probe tick after the first leaves one more idle connection + goroutine open → goroutine/FD count grows without bound for the sandbox's lifetime.

Fix: set DisableKeepAlives: true on the per-tick Transport — correct here since the transport is discarded after a single request anyway, so there's no pooling benefit to lose, only the connection closes promptly instead of idling forever.

Regression test: internal/probe/probe_test.goTestHeadDoesNotLeakKeepAliveGoroutine spins up an httptest.Server, calls head() 20 times, and asserts the goroutine count doesn't grow. Verified it fails without the fix (goroutine count went from 3 → 63 after 20 calls) and passes with it.

Reviewer command: cd packages/sandbox/daemon-go && go test ./internal/probe/... -run TestHeadDoesNotLeakKeepAliveGoroutine -v

Locally verified: gofmt -l, go vet ./internal/probe/..., go build ./..., and the targeted test above all pass. Full CI validates the rest.


Summary by cubic

Stops the dev-server probe from leaking an idle keep-alive connection and its goroutine on every tick, which could exhaust file descriptors or memory in long-running sandboxes.

Bug Fixes

  • Set DisableKeepAlives: true on the per-tick Transport so the connection closes after each HEAD request.
  • Added TestHeadDoesNotLeakKeepAliveGoroutine to guard against regression.

Written for commit 3b7b801. Summary will update on new commits.

Review in cubic

…on every tick

head() builds a fresh http.Transport on every probe tick (every 1s while
booting, every 30s once online) and never closes it or drains its idle
connection pool. With keep-alive left on, each successful HEAD leaves an
idle persistConn + readLoop goroutine open indefinitely, since the
Transport itself is discarded but the goroutine keeps it (and the socket)
alive. Over a long-running sandbox this accumulates goroutines and file
descriptors without bound.

Set DisableKeepAlives on the per-tick Transport so the connection closes
right after each HEAD instead of idling forever.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2 issues found across 2 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/sandbox/daemon-go/internal/probe/probe_test.go">

<violation number="1" location="packages/sandbox/daemon-go/internal/probe/probe_test.go:22">
P3: On IPv6-only hosts, this port extraction fails because `httptest.NewServer` can return a URL such as `http://[::1]:<port>`. Parse `srv.URL` with `net/url` and `net.SplitHostPort` so the regression test works on every loopback address supported by `DialLoopback`.</violation>

<violation number="2" location="packages/sandbox/daemon-go/internal/probe/probe_test.go:27">
P3: This asserts on the global `runtime.NumGoroutine()`, so unrelated goroutines (parallel tests, race-detector/GC wakeups, runtime background activity) can transiently push the count above `before+5`, making the test flaky even when nothing leaks. The threshold and 2s wait reduce but do not remove that risk. Tighten it by asserting the specific readLoop goroutine count (e.g. report via the leak that this guards), or widen the tolerance and note the flakiness.</violation>
</file>

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

t.Fatalf("parse port from %q: %v", srv.URL, err)
}

before := runtime.NumGoroutine()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: This asserts on the global runtime.NumGoroutine(), so unrelated goroutines (parallel tests, race-detector/GC wakeups, runtime background activity) can transiently push the count above before+5, making the test flaky even when nothing leaks. The threshold and 2s wait reduce but do not remove that risk. Tighten it by asserting the specific readLoop goroutine count (e.g. report via the leak that this guards), or widen the tolerance and note the flakiness.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/sandbox/daemon-go/internal/probe/probe_test.go, line 27:

<comment>This asserts on the global `runtime.NumGoroutine()`, so unrelated goroutines (parallel tests, race-detector/GC wakeups, runtime background activity) can transiently push the count above `before+5`, making the test flaky even when nothing leaks. The threshold and 2s wait reduce but do not remove that risk. Tighten it by asserting the specific readLoop goroutine count (e.g. report via the leak that this guards), or widen the tolerance and note the flakiness.</comment>

<file context>
@@ -0,0 +1,49 @@
+		t.Fatalf("parse port from %q: %v", srv.URL, err)
+	}
+
+	before := runtime.NumGoroutine()
+
+	for i := 0; i < 20; i++ {
</file context>

}))
defer srv.Close()

port, err := strconv.Atoi(srv.URL[len("http://127.0.0.1:"):])

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: On IPv6-only hosts, this port extraction fails because httptest.NewServer can return a URL such as http://[::1]:<port>. Parse srv.URL with net/url and net.SplitHostPort so the regression test works on every loopback address supported by DialLoopback.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/sandbox/daemon-go/internal/probe/probe_test.go, line 22:

<comment>On IPv6-only hosts, this port extraction fails because `httptest.NewServer` can return a URL such as `http://[::1]:<port>`. Parse `srv.URL` with `net/url` and `net.SplitHostPort` so the regression test works on every loopback address supported by `DialLoopback`.</comment>

<file context>
@@ -0,0 +1,49 @@
+	}))
+	defer srv.Close()
+
+	port, err := strconv.Atoi(srv.URL[len("http://127.0.0.1:"):])
+	if err != nil {
+		t.Fatalf("parse port from %q: %v", srv.URL, err)
</file context>

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant