Skip to content

Fix yt-dlp stream stalls and unreaped processes - #15

Merged
Isma-L154 merged 1 commit into
mainfrom
fix/stream-subprocess-lifecycle
Aug 21, 2026
Merged

Fix yt-dlp stream stalls and unreaped processes#15
Isma-L154 merged 1 commit into
mainfrom
fix/stream-subprocess-lifecycle

Conversation

@Isma-L154

Copy link
Copy Markdown
Owner

Closes #6

(Replaces #14, which GitHub auto-closed when its base branch was deleted on merge of #13. Same branch, same commit, now rebased onto main.)

The bugs

1. Playback could stall silently

spawn_stream() gave the child stderr=subprocess.PIPE, and nothing read that pipe while the track played — classify_stream_error() only drained it after playback ended.

The kernel pipe buffer is ~64 KB. A chatty yt-dlp (fragment retries, HTTP warnings, throttling notices) fills it, blocks on write, and stops producing audio. FFmpeg starves, playback stops, and nothing is reported to the user. -q --no-warnings reduces the volume of stderr but does not bound it.

2. The child was never fully released

kill_stream() sent SIGKILL and waited, but left proc.stdout / proc.stderr for the GC to close and swallowed TimeoutExpired.

This one was observed in production, not inferred. The live host had a yt-dlp zombie parented to the bot that had been sitting there for nearly 20 days:

PID    PPID   STAT  ELAPSED        COMMAND
46490  43631  Z     19-19:07:53    yt-dlp

The fix

Replaces the loose (Popen, kill_stream, classify_stream_error) trio with an AudioStream class that owns the child end to end.

Concern Before After
stderr pipe nobody reads → blocks at 64 KB temporary file → never blocks
Error text read after kill(), whole buffer tail only (16 KB), captured during close()
stdout pipe, closed by the GC eventually pipe (kept — it is what bounds memory), closed explicitly
Reaping kill + wait(2), TimeoutExpired swallowed kill + wait(5), a survivor is logged
Re-entrancy close() is idempotent

stdout deliberately stays a pipe: that backpressure is what keeps memory bounded on a 1 GB host, since yt-dlp blocks as soon as FFmpeg stops reading rather than buffering a whole track in RAM. Only stderr moves to a file.

close() blocks briefly (waiting on the child, reading the file), so utils/player.py calls it through run_in_executor in both the playback loop and destroy() — never straight from the event loop.

Verification

Real end-to-end streaming

Ran the real spawn_stream against live sources and confirmed audio flows and the process is reaped:

Source Time to first byte Bytes Reaped Pipes closed
SoundCloud (scsearch1:lofi hip hop) 2.86s 262,144 yes yes
YouTube 2.42s 262,144 yes yes

YouTube initially returned HTTP 403 on the local box — that turned out to be a five-month-old yt-dlp (2026.3.3), not this change. Updating to 2026.08.19 fixed it, which is direct evidence for #11.

The regression test that matters

test_a_process_flooding_stderr_still_delivers_audio spawns a child that writes 1 MB to stderr (~16x the pipe buffer) and then writes to stdout. Under the old piped-stderr code the child deadlocks and no audio ever arrives; it now streams normally.

test_close_reaps_the_process_leaving_no_zombie asserts poll() returns a code after close() — the exact property the 19-day zombie violated.

Suite

120 passed in ~0.8s, three consecutive runs, no flakes. Tests use proc.wait() rather than fixed sleeps so they stay deterministic on slow CI.

Notes for review

  • tests/test_media_helpers.py shrinks: its mock-based kill_stream / classify_stream_error tests are replaced by tests/test_audio_stream.py, which exercises the same behaviour against real subprocesses. Mocks cannot reproduce a full pipe buffer, which is the whole point.
  • .gitignore gains --Frag* / *.part / *.ytdl: streaming to stdout can drop fragment artifacts in the working directory, and one nearly got committed while verifying this.
  • classify_error() is cached by close(), so the player can ask why a stream failed after teardown without touching the disk again.

Replace the loose (Popen, kill_stream, classify_stream_error) trio with an
AudioStream class that owns the child process end to end.

Two problems it fixes:

stderr was a pipe nobody read while a track played. classify_stream_error only
drained it after playback ended, so a chatty yt-dlp (fragment retries, HTTP
warnings, throttling notices) would fill the ~64 KB kernel buffer, block on
write, and stop producing audio. FFmpeg then starved and playback stalled with
nothing reported to the user. stderr now goes to a temporary file, which never
blocks. classify_error() reads only the tail, since the real error is written
last and can sit behind megabytes of progress noise.

The child was killed but not fully released. Pipes were left for the GC to
close and TimeoutExpired was swallowed. close() now kills, reaps, captures the
error text while the file is still open, and closes both handles - and is
idempotent. The live host had accumulated a yt-dlp zombie that survived 19 days
on exactly this path.

close() blocks briefly, so the player calls it through an executor rather than
from the event loop, in both the playback loop and destroy().

Verified end to end against real yt-dlp: SoundCloud 2.86s to first byte,
YouTube 2.42s, 256 KB streamed each, process reaped, pipes closed. The new
flood test reproduces the stall directly - 1 MB of stderr, 16x the pipe buffer,
which deadlocks under the old code and now delivers audio normally.

Closes #6
@Isma-L154
Isma-L154 merged commit 10714d2 into main Aug 21, 2026
3 checks passed
@Isma-L154
Isma-L154 deleted the fix/stream-subprocess-lifecycle branch August 21, 2026 01:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

yt-dlp stream subprocess: unread stderr can stall playback, and processes are not fully reaped

1 participant