Skip to content

Make deployment reproducible and self-updating - #18

Merged
Isma-L154 merged 2 commits into
mainfrom
fix/reproducible-deployment
Aug 21, 2026
Merged

Make deployment reproducible and self-updating#18
Isma-L154 merged 2 commits into
mainfrom
fix/reproducible-deployment

Conversation

@Isma-L154

Copy link
Copy Markdown
Owner

Closes #11
Closes #12

Three related problems with how this bot gets onto a server, all verified against the live EC2 host.

1. Dependencies floated

Every requirement used >=, so two deploys a week apart installed different versions. The live host and the dev box had genuinely drifted from the same file:

Package Dev box Live host
lyricsgenius 3.10.0 3.12.2
aiohttp 3.13.3 3.14.3

Everything is now pinned — except yt-dlp, which must float. YouTube changes its player and extractors constantly, and a five-month-old build (2026.3.3) returned HTTP 403 on every YouTube URL until it was updated. The comment in requirements.txt explains this so nobody "fixes" it later by pinning it.

2. yt-dlp had no way to stay current

setup.sh now installs loopify-ytdlp-update.timer, running daily with a randomised delay and Persistent=true so it catches up after downtime rather than silently skipping.

The updater runs as root so it can restart the service, but drops to the app user via runuser for the pip install — otherwise the venv ends up owned by root.

Both paths tested on the live host:

Case Expected Result
Already current log and exit, no restart PID unchanged (119766 → 119766)
Stale (2026.7.4) upgrade and restart yt-dlp 2026.07.04 -> 2026.08.19; restarting loopify-bot, PID 119766 → 120033
venv ownership after upgrade stays ubuntu ubuntu:ubuntu .venv/bin/yt-dlp

A failed upgrade leaves the working version installed — a newer dependency is never worth trading a running bot for.

3. The documented update path did not work

deploy/README.md said to run git pull, but the host was deployed by rsync and has no .git:

fatal: not a git repository (or any of the parent directories): .git

This is not a documentation nit — it is why fixes merged to main never reached the server. The yt-dlp zombie in #6 survived 19 days there after proc.wait() had already landed in the repo.

Now: provisioning clones, deploy/update.sh handles updates, and the bot logs what it is running at startup.

Running commit 61919d6 — yt-dlp 2026.08.19, FFmpeg 6.1.1-3ubuntu5, Python 3.12.3

That line survives the service's ProtectHome=read-only / ProtectSystem=strict sandbox, which was the risk worth checking.

The CRLF landmine this uncovered

The shell scripts were stored in the repository with CRLF line endings. Harmless while deployment was rsync from Windows — but the moment the docs say git clone, a Linux checkout gets CRLF, bash reads the shebang as /usr/bin/env bash\r, and the script will not run. Changing the docs would have introduced the break.

.gitattributes pins *.sh / *.yml to LF and the tracked files are renormalised. Verified on a real Linux checkout of this branch:

deploy/launch_ec2.sh         0 CR      bash -n: OK
deploy/setup.sh              0 CR      bash -n: OK
deploy/update-ytdlp.sh       0 CR      bash -n: OK
deploy/update.sh             0 CR      bash -n: OK

A gap found by testing, then fixed

The first version of update.sh failed with git's own "There is no tracking information" wall of text on a host converted from a file copy — the exact situation the conversion instructions produce. The second commit detects a missing upstream and says what to run, and the recipe in both the script and the README now sets it.

Both paths re-verified on the host:

!! Branch 'master' has no upstream, so there is nothing to pull from.
   ...
     git branch --set-upstream-to=origin/main master
==> Fetching...
==> Already up to date.
==> requirements.txt unchanged; skipping dependency install.
==> Restarting loopify-bot...
==> Running commit 61919d6.

End state on the live host

The EC2 was converted to a git checkout as part of this (secrets backed up first; .env and cookies.txt are gitignored and came through untouched at chmod 600). It now runs this branch:

Before After
Deployed code rsync, unknown commit git checkout, 61919d6
yt-dlp 2026.7.4 (6 weeks stale) 2026.08.19, auto-updating
Zombie processes 1, aged 19d 19h 0
Bot RSS 96 MB 49 MB

That zombie count is the #6 fix confirmed in production, not just in tests.

Verification

  • 201 passed locally (21 new, covering runtime version reporting).
  • shellcheck -S style deploy/*.sh — clean.
  • The version lookup degrades to "unknown" rather than failing the boot on a host without git, without FFmpeg, or with a hung subprocess. Tested for FileNotFoundError, PermissionError and TimeoutExpired, and asserted that _first_line passes a timeout at all.
  • A test asserts the startup line cannot leak DISCORD_TOKEN.

Note for a follow-up

journalctl shows systemd rejecting one line of the live .env:

Ignoring invalid environment assignment '<a URL>': /home/ubuntu/LoopifyBot/.env

Harmless — it is ignored and the bot reads config through python-dotenv anyway — but the file is parsed by two things with different rules, and they can disagree on quoting. Filed separately rather than expanding this PR.

Three related problems with how this bot gets onto a server.

Dependencies floated. Every requirement used >=, so two deploys a week apart
installed different versions. The live host and the dev box had drifted to
lyricsgenius 3.12.2 vs 3.10.0 and aiohttp 3.14.3 vs 3.13.3 from the same file.
Everything is now pinned except yt-dlp, which must float: YouTube changes its
player constantly and a five-month-old build returned HTTP 403 on every
YouTube URL until it was updated. A systemd timer now keeps it current daily,
restarting the bot only when the version actually changed and leaving the
working version installed if the upgrade fails.

The documented update path did not work. deploy/README told you to run
git pull, but the host was deployed by rsync and has no .git, so fixes merged
to main never reached it - which is why a yt-dlp zombie survived 19 days there
after the fix had already landed. Provisioning now clones, update.sh handles
updates and refuses to run on a non-git checkout with instructions to convert
it, and the bot logs its commit, yt-dlp, FFmpeg and Python versions at startup
so journalctl can answer "what is actually running" directly.

The shell scripts had CRLF line endings in the repository. That was harmless
while deployment was rsync from Windows, but the moment the docs say
"git clone" a Linux checkout gets CRLF and bash rejects the shebang. A
.gitattributes pins *.sh and *.yml to LF, and the tracked files are
renormalised.

Verified with shellcheck at style level: clean.

Closes #11
Closes #12
A host converted from a file copy has a git checkout but no tracking branch, so
git pull stopped with its own "There is no tracking information" wall of text -
in exactly the situation the conversion instructions create.

Detect it up front and say what to run. The conversion recipe in both the script
and deploy/README now sets the upstream, so following it produces a host that
update.sh can actually update.
@Isma-L154
Isma-L154 merged commit 8248ce1 into main Aug 21, 2026
3 checks passed
@Isma-L154
Isma-L154 deleted the fix/reproducible-deployment branch August 21, 2026 02:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant