fix(e2e): stabilize listen-port races and reload polls - #725
Merged
Conversation
CI flakes on macOS were "rtp2httpd did not start on port 49xxx" because find_free_port() used bind(0), which returns ports from the kernel ephemeral pool. Parallel xdist workers then reused those ports as client source ports before rtp2httpd could bind. Take listen ports from a per-worker range below 32768, probe the IPv4 wildcard, and fail-fast in R2HProcess.start() with process logs when startup dies or times out. Co-authored-by: Stackie Jia <jsq2627@gmail.com>
SIGHUP restarts workers and briefly drops the listen socket. Polling http_get() could raise ConnectionRefusedError on FreeBSD instead of returning a non-200 status, aborting the retry loop. Co-authored-by: Stackie Jia <jsq2627@gmail.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| sockets.append(sock) | ||
| if family == socket.AF_INET6: | ||
| sock.setsockopt(socket.IPPROTO_IPV6, socket.IPV6_V6ONLY, 1) | ||
| sock.bind((bind_host, port)) |
| def _can_bind_udp(port: int) -> bool: | ||
| try: | ||
| with socket.socket(socket.AF_INET, socket.SOCK_DGRAM) as sock: | ||
| sock.bind(("", port)) |
| def test_find_free_port_is_bindable_on_wildcard(): | ||
| port = find_free_port() | ||
| with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock: | ||
| sock.bind(("0.0.0.0", port)) |
| port = find_free_udp_port() | ||
| assert start <= port < end | ||
| with socket.socket(socket.AF_INET, socket.SOCK_DGRAM) as sock: | ||
| sock.bind(("", port)) |
Contributor
Documentation previewThe documentation preview has been deployed for this pull request. |
stackia
marked this pull request as ready for review
August 22, 2026 11:27
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
FreeBSD CI has no /bin/false, so the new helper test aborted in Popen before it could assert the exit-code path. Use a chmod +x /bin/sh script instead, and surface spawn OSError from R2HProcess.start(). Co-authored-by: Stackie Jia <jsq2627@gmail.com>
| assert odd == even + 1 | ||
| assert start <= even < odd < end | ||
| with socket.socket(socket.AF_INET, socket.SOCK_DGRAM) as s1, socket.socket(socket.AF_INET, socket.SOCK_DGRAM) as s2: | ||
| s1.bind(("", even)) |
| assert start <= even < odd < end | ||
| with socket.socket(socket.AF_INET, socket.SOCK_DGRAM) as s1, socket.socket(socket.AF_INET, socket.SOCK_DGRAM) as s2: | ||
| s1.bind(("", even)) | ||
| s2.bind(("", odd)) |
Co-authored-by: Stackie Jia <jsq2627@gmail.com>
Co-authored-by: Stackie Jia <jsq2627@gmail.com>
Co-authored-by: Stackie Jia <jsq2627@gmail.com>
Co-authored-by: Stackie Jia <jsq2627@gmail.com>
Co-authored-by: Stackie Jia <jsq2627@gmail.com>
Co-authored-by: Stackie Jia <jsq2627@gmail.com>
Co-authored-by: Stackie Jia <jsq2627@gmail.com>
Co-authored-by: Stackie Jia <jsq2627@gmail.com>
Co-authored-by: Stackie Jia <jsq2627@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Recent E2E CI failures (especially macOS) were not product regressions. The dominant flake was:
Failed ports (
49220,49234,49334,50597,51059) all sit in the macOS ephemeral range (49152-65535).find_free_port()usedbind(0)and immediately closed the socket. Under pytest-xdist, another worker's clientconnect()could take that port as a source port before rtp2httpd bound0.0.0.0/::.Changes
14000-32399), with a short reuse cooldown.0.0.0.0(not just127.0.0.1) so we do not hand out a port that rtp2httpd cannot bind on the wildcard.R2HProcess.start()always captures logs, fail-fasts if the process exits, and includes the log + exit code in the error.test_pages.pytreatConnectionRefusedErroras "not ready yet"./bin/shscript (/bin/falseis missing on FreeBSD).Stability gate
10/10 consecutive
E2E Testsworkflow runs passed (ubuntu + macos + freebsd on each):The preceding run on
bf78c19failed only because FreeBSD has no/bin/false; that helper test was fixed before this 10-run streak.Test plan
uv run ruff check e2e./scripts/run-e2e.sh --co./scripts/run-e2e.sh -p 1 test_port_helpers.py./scripts/run-e2e.sh— 576 passed locally