Handle stale terraform locks after nixpkgs bumps

README.md

@@ -149,6 +149,28 @@ VM with the pairing enabled, let the reconciler apply, then assert the runtime
 state by querying the live service API — not the Terraform state. Eval-only or
 build-only success is not evidence the reconciliation works.
 
+## Troubleshooting
+
+**`declarative-<svc>.service` fails with `locked provider … does not match
+configured version constraint`.** The reconciler's `.terraform.lock.hcl` is host
+state under the service's state directory and survives redeployments, while the
+generated `required_providers` constraint is pinned to whatever provider version
+nixpkgs packages — so bumping the `nixpkgs` input moves the constraint out from
+under the lock. The reconciler runs `tofu init -upgrade` to re-lock by itself
+(still offline: the plugin dir baked in by `opentofu.withPlugins` is the only
+available source). On a host still stuck from an older revision, drop the lock
+and re-run:
+
+```sh
+rm /var/lib/<svc>/declarative-terraform/.terraform.lock.hcl
+systemctl restart declarative-<svc>.service
+```
+
+The `terraform.tfstate` next to it is untouched, so the next apply reconciles
+against the existing state rather than recreating resources. For a
+`DynamicUser=` service such as Keycloak the real directory lives under
+`/var/lib/private/`, reachable as root through the `/var/lib/<svc>` symlink.
+
 ## Design decisions
 
 - **Executor:** OpenTofu (MPL 2.0); `terraform` (BSL 1.1, unfree) is not used.

modules/lib/default.nix

@@ -563,7 +563,13 @@ rec {
         for id in ${lib.escapeShellArgs (lib.attrNames allCredentials)}; do
           export "TF_VAR_$id=$(cat "$CREDENTIALS_DIRECTORY/$id")"
         done
-        tofu init -no-color
+        # -upgrade: the provider version is pinned to whatever nixpkgs
+        # packages, so a nixpkgs bump moves the required_providers
+        # constraint while .terraform.lock.hcl in the state dir still pins
+        # the old version -- a plain init would fail. re-selecting stays
+        # offline: the plugin dir baked in by withPlugins is the only
+        # available source.
+        tofu init -upgrade -no-color
         tofu apply -auto-approve -input=false -no-color
       '';
     };

services/forgejo/checks.nix

@@ -10,7 +10,9 @@
 #     a repository by its numeric id — so a successful apply proves reference
 #     resolution and apply ordering. A user with a `passwordFile` also proves
 #     per-secret credential indirection — the value is loaded from a host file
-#     and kept out of the generated `.tf.json`. Requires KVM (a NixOS VM test).
+#     and kept out of the generated `.tf.json`. A stale
+#     .terraform.lock.hcl is then planted to prove the reconciler re-locks
+#     offline after a provider version bump. Requires KVM (a NixOS VM test).
 { pkgs, self }:
 {
   forgejo = pkgs.testers.runNixOSTest {
@@ -145,6 +147,21 @@
       assert "0 added, 0 changed, 0 destroyed" in apply_lines[-1], \
           f"reapply was not a no-op: {apply_lines[-1]}"
 
+      # A provider version bump moves the generated required_providers
+      # constraint while .terraform.lock.hcl -- host state under the service's
+      # state dir -- still pins the old version. Simulate that skew by
+      # rewriting the recorded version: a plain `tofu init` aborts with
+      # "locked provider ... does not match configured version constraint",
+      # so this is what the reconciler's `-upgrade` buys. The VM has no
+      # network, so re-locking here also proves the re-selection stays offline
+      # (the plugin dir baked in by withPlugins is the only source).
+      lock = "/var/lib/forgejo/declarative-terraform/.terraform.lock.hcl"
+      machine.succeed(f"sed -i 's/version *=.*/version = \"0.0.1\"/' {lock}")
+      machine.succeed(f"grep -q 0.0.1 {lock}")
+      machine.succeed("systemctl restart declarative-forgejo.service")
+      relocked = machine.succeed(f"cat {lock}")
+      assert "0.0.1" not in relocked, f"stale lock was not re-locked: {relocked}"
+
       # Adding an admin-scoped resource (a user needs write:admin + read:user)
       # mus work because the scopen is the maximal "all" token
       machine.succeed("/run/current-system/specialisation/widenScope/bin/switch-to-configuration test")