Skip to content

Unattended save backup via sysmodule, TLS verification, and server hardening - #7

Open
dmuiX wants to merge 8 commits into
prodeveloper0:mainfrom
dmuiX:main
Open

Unattended save backup via sysmodule, TLS verification, and server hardening#7
dmuiX wants to merge 8 commits into
prodeveloper0:mainfrom
dmuiX:main

Conversation

@dmuiX

@dmuiX dmuiX commented Aug 2, 2026

Copy link
Copy Markdown

Adds unattended save backup, hardens the server for real deployment, and
brings the docs in line with what the code actually does.

Everything here has been run on hardware — firmware 22.5.0 with Atmosphere
1.11.2, backing up three accounts against a server behind Traefik.

Background service (the main addition)

An NRO only runs while it is open, so an app can never back up a save that
changed while you were playing. This adds a sysmodule that starts with the
console and stays resident.

It wakes every five minutes and asks one question: has any save data changed
since the last upload? If not it does nothing at all — not even opening a
socket — which is the normal case and costs a few file reads. When a game ends
it checks immediately instead of waiting out the interval. While a game runs it
stays off the save files entirely, because the game holds them open.

Backing up at boot, which the first version did, sounds reasonable and works
badly: a Switch is closed, not switched off, so reboots are weeks apart and the
module would almost never run. What triggers a backup is change, not the clock.

  • Install and remove it from inside the app; the module is embedded in the
    NRO's romfs, so there is no copying files into hex-named folders by hand.
  • Service log in the app reads the module's log back — a sysmodule has no
    screen, and the alternative was pulling the SD card.
  • Sync logic moved out of MainScreen into sync.{hpp,cpp} so the app and the
    service run the exact same code path and cannot drift apart.

It can take the console down, and it is built to survive that

It has killed hid and am during development, each time via 2001-0132
(kernel LimitReached) — an error that names the victim, not the cause. So the
module does not rely on being correct:

  • It parks its own boot2.flag before anything risky and puts it back only
    after a clean round. If it dies, the flag stays parked and the next boot
    comes up without it. A boot loop is structurally impossible rather than
    merely unlikely.
  • It distinguishes "we crashed" (crash_reports, our program id in the
    filename) from "a system process died" (fatal_reports) — the two real
    incidents landed in different directories.
  • A crash during a backup no longer disables it for good. Nobody watches a
    console for notifications, so a module that switched itself off would stay
    off. It backs off in steps instead — 5 min, 20, 80, capped at two hours — and
    retries on its own. Only a crash during startup, where retrying means a boot
    loop, still stands down.

The README documents the memory situation in detail, including the measurements
that refused to support a threshold: the run that killed the console had more
free pool at its low point than five runs that survived. It says so rather than
quoting a number that would look authoritative and be wrong.

It also documents a second limit that is not the pool. Adding one more resident
sysmodule alongside this one made every homebrew launch panic the console
immediately after boot — Atmosphere panic occurred!, program id
0100000000000034. That id is fatal, Atmosphère's own crash reporter, which
could not get a session for lbl to turn the backlight on and aborted; the
report decodes to 2021-0003, sm::ResultOutOfSessions. It lands in
atmosphere/fatal_errors/, while crash_reports/ and fatal_reports/ stay
empty, so the obvious places suggest nothing happened. Removing the extra
sysmodule fixed it. The README writes down how to read that report, and is
explicit that this is one A/B on one console: it shows one module too many
was enough, not that any particular module is at fault.

Client

  • Verify the server certificate by default. The credentials sit in the URL and
    libcurl offers them preemptively, so without verification anyone answering
    the handshake gets the password. Private CAs go in sdmc:/uNSS/cacert.pem;
    remote.insecureSkipVerify is the last resort.
  • Automatic push on launch, throttled by change rather than by a timer.
  • Skip titles whose saves have not changed, decided before archiving.
  • Postpone automatic backups while a game is running (pm:dmnt); the manual
    Push button stays unrestricted.
  • Each account gets only its own saves. The console returns every save on the
    system when asked for a list, so all accounts were previously trying to open
    each other's — and "this account never played that game" was being reported
    as a failure, which kept a healthy round from ever recording success.

Server

  • Tests for the two places where a mistake is expensive and silent: the plain
    text wire format the C++ client parses by hand, and the PC/D
    revision transaction. Both new tests fail against the previous queries.
  • Fix revision selection when timestamps collide. created_at has one-second
    resolution, and ORDER BY created_at DESC then returned the oldest row, so
    a restore could hand back a previous save. Ties now break on rowid, and the
    list and per-title lookup use the same rule so they cannot disagree.
  • Add the missing uvloop dependency — main.py requested it, so the
    container crashed on startup.
  • Run the container as a non-root user, publish images to ghcr.io for amd64 and
    arm64, and declare the port in the image.
  • A Traefik variant with basicAuth, since the client cannot follow redirects
    and no login flow can ever complete. The stack takes the password in plain
    text and hashes it itself with bcrypt cost 12, so only one representation of
    the secret has to be maintained.

Notes

  • Server responses stay plain text; the wire format is unchanged and now has
    tests pinning it.
  • Shared client sources are symlinked into the sysmodule rather than copied, so
    both binaries always build from the same code.
  • build-all.sh builds the module first and refreshes the copy inside the
    NRO — the NRO rule depends on the elf and nacp, never on romfs contents, so
    fixing the module and running make in client/ otherwise produced an app
    that still installed the previous one.

🤖 Generated with Claude Code

dmuiX added 2 commits August 2, 2026 18:50
- feat: Skip titles whose save data has not changed since the last upload
- feat: Give each account only the saves that belong to it

The push / pull routines were lambdas inside MainScreen, tied to the GUI
only by their log callback. Moving them into sync.{hpp,cpp} lets a headless
caller reuse the exact same code path, so an app and a background service
cannot drift apart. No behavioural change on its own.

Change detection compares the newest mtime inside each save against
saves/.syncstate and filters during the probe, so unchanged titles are not
even archived. countChangedTitles() answers the same question without
touching the network, which lets a caller decide whether there is anything
to do before opening a socket. collectTargetTitles() is shared so that
"nothing changed" and "nothing to upload" cannot disagree - they were
separate copies of the same filter before.

The console returns every save on the system when asked for a list, without
separating them by user, and that was never filtered. So every account
received every title and spent the round trying to open saves belonging to
someone else. Mounting a save that does not exist also returned the same
value as failing to mount one that does, so a healthy round always ended
"with errors" - and a round that ends with errors never records the time it
succeeded, which meant the next round packed and uploaded everything again
from scratch.

archiveAllSaveData() also could not report failures: it returned OK once it
had walked the list, and per-title errors only reached a callback that
always answered true. A server that was down still produced a return value
of 0, which both callers read as success. Failures are counted and returned
negatively; SAVEDATA_* codes are positive, so the two cannot be confused.
The credentials sit in the URL and libcurl offers them as an Authorization
header on the first request. Without verification, anyone who answers the
handshake receives the password in the clear. bcrypt protects the hash at
rest on the server; it does nothing for this leg.

Let's Encrypt validates against the console's built-in list - ISRG Root X1
has been there since 10.1.0 - so the common case needs nothing extra. For a
private CA there is sdmc:/uNSS/cacert.pem: this curl uses the libnx SSL
backend, which passes CAINFO to sslContextImportServerPki, so a single file
can make the console trust a root the firmware never shipped. That is the
first thing to try; turning verification off is the last resort, because
its failure mode looks like "cannot connect" rather than "certificate
refused".

Failures now carry the underlying result alongside the -1. That -1 covered
"server said 400" and "never connected" with one value, which is what made
a DNS problem take a day to find.
@dmuiX

dmuiX commented Aug 2, 2026

Copy link
Copy Markdown
Author

Sorry first pr version were a bit too many commits. Now its densed down to the important ones.

Is working fine on my switch as of now. Have been testing it intensively. And had some problems as you definetly must slim down the installed sysmodule otherwise it will not run. But I suppose this is standing inside the README

@dmuiX
dmuiX force-pushed the main branch 2 times, most recently from a38b27c to d83c68a Compare August 2, 2026 18:03
dmuiX added 6 commits August 2, 2026 20:12
An NRO only runs while it is open - start a game and it is gone - so
unattended backups need a sysmodule. This one boots with the console and
stays resident, waking every five minutes to ask whether any save data
changed since the last upload. If nothing changed it does nothing at all,
not even opening a socket, which is the normal case and costs a few file
reads. When a game ends it checks straight away rather than waiting out the
interval, since the moment after someone saves and quits is the best time
to copy a save. While a game runs it stays off the save files entirely.

Backing up at boot instead, which the first version did, sounds reasonable
and works badly: a Switch is closed, not switched off, so reboots are weeks
apart and the module would finish before the first game even started.
Config is re-read every round, so editing config.ini takes effect without a
reboot.

Shared sources are symlinked from client/source rather than copied, so both
binaries always build from the same code. build-all.sh builds the module
first and refreshes the copy the app embeds; it also drops the NRO, because
the NRO rule depends on the elf and the nacp but never on romfs contents -
without that, fixing the module and running make produced an app that still
installed the previous one.

Notes from bringing this up on hardware (22.5.0 / Atmosphere 1.11.2):

- The program id must sit in the custom range. 0x0100000000554E53 was never
  loaded by boot2 at all; 0x4200000000554E53 works.
- Services must be opened in __appInit before smExit(). But opening
  network, ns and account there runs alongside HID, which then failed to
  get resources and died with 2001-0132, boot-looping the console. They
  move to initLateServices() after a 30 second grace period, and the
  session is kept rather than released - sfdnsres has no initialize
  function and opens itself per request, so closing the session disabled
  getaddrinfo while the log still said "network ready".
- INNER_HEAP_SIZE is 1 MiB, measured rather than guessed. Sysmodules share
  one small pool and a static heap leaves it the moment the module loads.
  6 MiB starved hid, 2 MiB killed am. 256 KB is too tight in the other
  direction: a round peaking at 189 KB with a roomy heap peaked at 245 KB
  with a tight one, and compression, title lookup and upload failed at once.
- NsApplicationControlData is 0x24000 bytes and the main thread stack is
  16 KB, so asking for a title name overflowed it in the function prologue.

Since the module can take the console down, it does not rely on being
correct. It parks its own boot2.flag before anything risky and restores it
only after a clean round, so a crash costs one error screen rather than a
boot loop. Crashes are attributed by program id - ours land in
crash_reports, a dying system process in fatal_reports, and the two real
incidents landed in different directories. A crash during a backup pauses
in growing steps (5, 20, 80 min, capped at two hours) and then retries on
its own, because nobody watches a console for notifications and a module
that switched itself off would stay off. Only a crash during startup, where
retrying means a boot loop, still stands down.
- feat: Show the service log inside the app
- feat: Automatic backup on launch, throttled by change rather than time

Installing a sysmodule by hand means copying two files into a hex-named
folder under atmosphere/contents - easy to get wrong. The module is
embedded in the NRO's romfs and the app writes it out itself: "Install
background service" copies exefs.nsp, creates the boot2 flag and records
the bundled version; "Remove background service" deletes them again. An
already installed but outdated module is updated silently on launch, while
the first install stays a deliberate button press, since it starts a
process at every boot. When the module has stood itself down after a crash,
the app offers to re-enable it rather than leaving it stuck.

The sysmodule has no screen, so finding out what it had been doing meant
pulling the SD card or fetching the log over FTP - which is exactly what
once delayed a diagnosis by a day, with the module failing every upload for
hours and the only witness a file nobody could see. "Service log" follows
the end of that file while new lines arrive and lets go as soon as you
scroll up, because a view that jumps around cannot be read. Lines
mentioning a failure are red. Only the last 500 lines are held in memory.

With sync.autoPushOnLaunch the client starts pushing as soon as it opens.
autoPushIntervalHours is a brake rather than a trigger and defaults to 0:
what decides whether anything is uploaded is whether the save data changed,
so backups happen several times a day while playing, or not for weeks. A
fixed interval could only ever be wrong in one of the two directions. The
timestamp is written only after a successful run, so a failed backup is
retried on the next launch instead of counting as done.

A running game keeps its save open, so an automatic push waits for it to
end (pm:dmnt); the manual Push button stays unrestricted. pm:dmnt is opened
before smExit, because the lazy open inside isGameRunning() failed
afterwards and the fail-open path then reported "no game running" - which
would have let a backup archive a save file the game still held open.

remote.insecureSkipVerify is read here and passed to the HTTP layer.
- chore: Add tests for the revision transaction and the wire format

created_at is CURRENT_TIMESTAMP and only has second resolution, so two
uploads of the same title within one second get identical timestamps.
ORDER BY created_at DESC then has no defined order, and SQLite was measured
returning the oldest row - so restoring could hand back a previous save,
and the save list could advertise a revision that is not the current one.

Both queries were affected. GROUP BY title_id HAVING MAX(created_at)
reduces each title to one row but never says which row, and returned the
oldest for the same reason. Both now select the newest row explicitly and
break ties on rowid, which grows with insertion order. Using one rule in
both places matters: a disagreement between the list and the per-title
lookup would mean the client downloads a different revision than the one it
chose from the list.

The tests cover this and the rest of the revision transaction - a pending
revision must not hide the last completed one, a completed revision must
not be writable again - plus the plain text wire format the C++ client
parses by hand, where an empty list has to be an empty body rather than
"[]". Both new revision tests fail against the previous queries, which is
the point of them. Each test gets a fresh server in a temporary directory,
so they neither touch nor need a real database.
- fix: Run the server as a non-root user
- feat: Declare the server port in the image
- chore: exec uvicorn so signals reach the server

main.py requests loop="uvloop" but the package was not listed in
requirements.txt, so the Docker container crashed on startup.

run-linux.sh now execs uvicorn, so signals reach the server rather than the
wrapping shell - killing that shell used to leave uvicorn orphaned.

Switching the container off root is not just a USER line: the database path
is relative, so metadata.sqlite lives in /app, and SQLite writes its
journal next to the database. Without write access to the directory - not
merely the file - every write fails with "unable to open database file"
even when the bind mount is correct. So /app is chowned at build time and
the image drops to that account. UID and GID are build args (default
1000:100) and can also be overridden per deployment with the compose user:
key; the mounted database and savedata directory have to belong to
whichever is used.

EXPOSE opens nothing on the host; it records the port in the image
metadata, where reverse proxies read it. Traefik picks it up on its own as
long as exactly one port is declared, so a deployment no longer needs a
port label to keep in sync. Without either, Traefik builds no router and
answers 404, which reads like a routing problem rather than a missing port.
Builds server/Dockerfile on pushes to main and on version tags, for amd64
and arm64. Uses the automatically provided GITHUB_TOKEN, so no secret has
to be configured. Images pushed to ghcr.io are private until their
visibility is changed by hand.

The owner name is lowercased before it reaches a tag - ghcr.io rejects
upper case, and account names can contain it.

There is deliberately no paths filter. It would apply to the tag push as
well: there is only one push block, and GitHub evaluates the filter for
every ref in it. Tagging a release on a commit that happens not to touch
server/ - a docs or client commit, which is the common case - would then
silently produce no image and no version tags at all. The cost of leaving
it out is a rebuild on commits that change nothing in the image, which
takes well under a minute; a missing release does not announce itself.
- docs: Add compose file for running the server behind Traefik

Covers the new [sync] keys, what the automatic push does and the two rules
that bound it, and a section on the background service: how to install it
from the app, what it reads, where it logs, and why restoring is
deliberately not part of it.

The memory section deliberately states no threshold. Ten module starts were
logged on hardware: the run that took the console down had 4780 KB free in
the system pool at its low point, while five runs that survived a full
round reached 3448-3960 KB. Free memory does not separate them, so no
figure can honestly be drawn from the data, and the text says that rather
than quoting one that would look authoritative and be wrong. What the
crashes had in common was fixed in code, not in free space. The advice that
remains is qualitative - fewer resident sysmodules is more headroom - plus
how to read the module's own pool samples to compare a console against
itself before and after a change.

compose.traefik.yml is for setups that already terminate TLS in a proxy: it
joins an external network and publishes no port of its own. Access control
is basicAuth rather than a login portal because the client cannot follow
redirects, so anything redirect-based can never complete, while libcurl
does send credentials taken straight from the URL.

Getting the hash to Traefik is the awkward part, because a container label
cannot be filled from a secret store - Compose resolves ${...} while
parsing, long before a store injects anything, and Traefik discards a
router whose middleware is invalid, so the service answers 404 rather than
prompting. A helper container bridges that: it reads the credentials from
its environment, hashes the password with bcrypt cost 12 and writes the
middleware into Traefik's dynamic configuration, which Traefik reads on its
own. It stays resident with a healthcheck that tests the file, so the
server can depend on service_healthy rather than on it having run once, and
it refuses to produce a middleware that lets everyone in.

Also records which characters the password may contain, since it travels
inside a URL in config.ini where @ splits the host off and # truncates the
rest, and the ini parser strips quotes from both ends of a value.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant