Make deployment reproducible and self-updating - #18
Merged
Conversation
Three related problems with how this bot gets onto a server. Dependencies floated. Every requirement used >=, so two deploys a week apart installed different versions. The live host and the dev box had drifted to lyricsgenius 3.12.2 vs 3.10.0 and aiohttp 3.14.3 vs 3.13.3 from the same file. Everything is now pinned except yt-dlp, which must float: YouTube changes its player constantly and a five-month-old build returned HTTP 403 on every YouTube URL until it was updated. A systemd timer now keeps it current daily, restarting the bot only when the version actually changed and leaving the working version installed if the upgrade fails. The documented update path did not work. deploy/README told you to run git pull, but the host was deployed by rsync and has no .git, so fixes merged to main never reached it - which is why a yt-dlp zombie survived 19 days there after the fix had already landed. Provisioning now clones, update.sh handles updates and refuses to run on a non-git checkout with instructions to convert it, and the bot logs its commit, yt-dlp, FFmpeg and Python versions at startup so journalctl can answer "what is actually running" directly. The shell scripts had CRLF line endings in the repository. That was harmless while deployment was rsync from Windows, but the moment the docs say "git clone" a Linux checkout gets CRLF and bash rejects the shebang. A .gitattributes pins *.sh and *.yml to LF, and the tracked files are renormalised. Verified with shellcheck at style level: clean. Closes #11 Closes #12
A host converted from a file copy has a git checkout but no tracking branch, so git pull stopped with its own "There is no tracking information" wall of text - in exactly the situation the conversion instructions create. Detect it up front and say what to run. The conversion recipe in both the script and deploy/README now sets the upstream, so following it produces a host that update.sh can actually update.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #11
Closes #12
Three related problems with how this bot gets onto a server, all verified against the live EC2 host.
1. Dependencies floated
Every requirement used
>=, so two deploys a week apart installed different versions. The live host and the dev box had genuinely drifted from the same file:lyricsgeniusaiohttpEverything is now pinned — except
yt-dlp, which must float. YouTube changes its player and extractors constantly, and a five-month-old build (2026.3.3) returnedHTTP 403on every YouTube URL until it was updated. The comment inrequirements.txtexplains this so nobody "fixes" it later by pinning it.2. yt-dlp had no way to stay current
setup.shnow installsloopify-ytdlp-update.timer, running daily with a randomised delay andPersistent=trueso it catches up after downtime rather than silently skipping.The updater runs as root so it can restart the service, but drops to the app user via
runuserfor thepip install— otherwise the venv ends up owned by root.Both paths tested on the live host:
2026.7.4)yt-dlp 2026.07.04 -> 2026.08.19; restarting loopify-bot, PID 119766 → 120033ubuntuubuntu:ubuntu .venv/bin/yt-dlpA failed upgrade leaves the working version installed — a newer dependency is never worth trading a running bot for.
3. The documented update path did not work
deploy/README.mdsaid to rungit pull, but the host was deployed by rsync and has no.git:This is not a documentation nit — it is why fixes merged to
mainnever reached the server. Theyt-dlpzombie in #6 survived 19 days there afterproc.wait()had already landed in the repo.Now: provisioning clones,
deploy/update.shhandles updates, and the bot logs what it is running at startup.That line survives the service's
ProtectHome=read-only/ProtectSystem=strictsandbox, which was the risk worth checking.The CRLF landmine this uncovered
The shell scripts were stored in the repository with CRLF line endings. Harmless while deployment was rsync from Windows — but the moment the docs say
git clone, a Linux checkout gets CRLF,bashreads the shebang as/usr/bin/env bash\r, and the script will not run. Changing the docs would have introduced the break..gitattributespins*.sh/*.ymlto LF and the tracked files are renormalised. Verified on a real Linux checkout of this branch:A gap found by testing, then fixed
The first version of
update.shfailed with git's own "There is no tracking information" wall of text on a host converted from a file copy — the exact situation the conversion instructions produce. The second commit detects a missing upstream and says what to run, and the recipe in both the script and the README now sets it.Both paths re-verified on the host:
End state on the live host
The EC2 was converted to a git checkout as part of this (secrets backed up first;
.envandcookies.txtare gitignored and came through untouched atchmod 600). It now runs this branch:61919d6That zombie count is the #6 fix confirmed in production, not just in tests.
Verification
201 passedlocally (21 new, covering runtime version reporting).shellcheck -S style deploy/*.sh— clean."unknown"rather than failing the boot on a host without git, without FFmpeg, or with a hung subprocess. Tested forFileNotFoundError,PermissionErrorandTimeoutExpired, and asserted that_first_linepasses a timeout at all.DISCORD_TOKEN.Note for a follow-up
journalctlshows systemd rejecting one line of the live.env:Harmless — it is ignored and the bot reads config through
python-dotenvanyway — but the file is parsed by two things with different rules, and they can disagree on quoting. Filed separately rather than expanding this PR.