Skip to content

fix(release): stale-lock takeover ignores the PID it writes and the lock is never refreshed — a >30min release gets its lock stolen #792

Description

@BryanFRD

ReleaseLock documents its takeover rule as (src/monorepo/run/lock.rs:35):

Stale locks (older than STALE_LOCK_TTL with the PID no longer alive) are taken over with a warning.

take_over_if_stale only checks mtime:

let modified = metadata.modified().ok().and_then(|t| t.elapsed().ok()).unwrap_or(Duration::ZERO);
if modified < STALE_LOCK_TTL { return Ok(false); }
let _ = std::fs::remove_file(path);
Ok(true)

The PID is written into the lockfile for exactly this check and then never read. Two consequences:

  1. A live release past 30 minutes loses its lock. The lockfile is written once at acquire and never touched again, so its mtime is the start time, not a liveness signal. A 200-package monorepo doing cargo publish with verification, or a release waiting on the private-registry index-lag retry from Release: cargo publish fails on private-registry index lag (E1018) instead of retrying #621, clears 30 minutes easily. A concurrent run then deletes the lockfile and proceeds — producing precisely the half-pushed tag sets and non-FF rejects the lock exists to prevent, with no error anywhere.

  2. A crashed run under 30 minutes still blocks. The common failure (CI job killed at 10 minutes) leaves a lock that is refused until the TTL elapses, even though the PID is demonstrably gone. --force-unlock works, but that is a manual step for the most common recovery case.

The hostname line has the same fate — written, never read — so cross-machine locks cannot be distinguished either.

Fix

  • Read the lockfile. If the recorded hostname matches this host and the PID is not alive, take over immediately regardless of TTL (kill(pid, 0) on Unix, OpenProcess on Windows).
  • If the hostname differs, fall back to the TTL rule — a remote PID says nothing useful.
  • Refresh the lockfile mtime periodically for the duration of the run (a touch at each phase boundary is enough; there is already a phase-level checkpoint from feat(release): crash-resume checkpoint for interrupted releases #549 to hang it off) so the TTL measures staleness rather than duration.
  • Raise STALE_LOCK_TTL once liveness is the primary signal — the TTL becomes the fallback for a dead remote host, where hours is a more honest number than 30 minutes.

Tests

  • Write a lockfile with this host and a PID that is definitely dead; assert acquire takes over without waiting for the TTL.
  • Write a lockfile with a live PID and an mtime older than the TTL; assert acquire refuses.
  • Write a lockfile with a foreign hostname and an old mtime; assert the TTL path still takes over.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Medium prioritybugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions