Make the monitor the only process that writes the database

[rhaegar325]

Fix #627

Base this PR on update_monitor, not main. It sits on top of that branch.

The problem

Gadi mounts every Lustre filesystem with localflock:

/scratch/tm70   lustre  rw,nosuid,nodev,localflock,lazystatfs
/g/data/tm70    lustre  rw,nosuid,nodev,localflock,lazystatfs

localflock makes file locks node-local. SQLite's whole concurrency model
rests on those locks, so the ~100 worker jobs of a batch — one per compute
node — can each believe they hold an exclusive lock on cmor_tasks.db and
write at the same time.

Two runs show the two ways this ends:

piControl_parallel_thorough_cc_check   integrity_check: 2nd reference to page 14
                                       100 rows, only 86 still readable
                                       monitor died, 99 outcomes lost

1pctCO2-01                             integrity_check: ok
                                       14 of 100 start_time values silently gone

The second is the nastier one. SQLite writes whole pages, not rows, and 99
rows share one 4096-byte page. Two nodes read that page, each edit their own
row, and the later write-back discards the earlier one. Nobody gets an error.

Retrying cannot help. synchronous=FULL cannot help. The fault is locking, not
durability.

The fix

Stop writing the database from more than one node.

worker (×100 nodes) ──► logs/<var>/status.json   one file, one writer
                                 │
                          monitor reads
                                 │
                        monitor ──► cmor_tasks.db   sole writer
                                          │
                          dashboard / moppy-tui / batch-report   read-only

Each worker writes only its own file, so there is nothing to contend over.
Writes go through a temp file plus os.replace, which is atomic on Lustre, so
the monitor never reads a half-written document.

The database keeps exactly the same schema and contents, so the dashboard,
moppy-tui and moppy-batch-report need no changes.

What changed

task_status.py (new)	TaskStatusFile plus the readers the monitor uses. Method names mirror TaskTracker, so the worker diff stays one-for-one.
cmor_python_script.j2	Seven writes and one read move to the status file. The worker no longer imports TaskTracker at all.
tracking.py	apply_worker_status() writes a whole row in one statement, keeping the worker's own timestamps. synchronous=OFF → NORMAL, affordable again now there is one writer.
batch_cmoriser.py	The monitor ingests status files each poll, clears a stale file before resubmitting, and --append-variable becomes a request file instead of a database row.

qstat stays. Across all past batches, 80 of 455 failures were only ever
knowable from PBS: the worker was SIGKILLed and never wrote anything. Its
status file is stuck at running and only the exit status can settle it.

One compatibility note

A moppy-cmorise older than this monitor writes its append request to the old
monitor_requests table. That happened during testing and the request was
silently dropped. The monitor now drains that table as well as the file queue,
so a mismatched CLI still works. The shim is commented and can go once no old
CLI is in use.

[codecov]

Codecov Report

❌ Patch coverage is 95.18072% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 78.5%. Comparing base (9938f8a) to head (d4e9b28).

Files with missing lines	Patch %	Lines
src/access_moppy/task_status.py	94.2%	5 Missing ⚠️
src/access_moppy/batch_cmoriser.py	95.8%	3 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##            main    #680     +/-   ##
=======================================
+ Coverage   78.2%   78.5%   +0.3%     
=======================================
  Files         40      41      +1     
  Lines       8827    8948    +121     
  Branches    1665    1671      +6     
=======================================
+ Hits        6901    7023    +122     
  Misses      1591    1591             
+ Partials     335     334      -1     

Flag	Coverage Δ
unit	78.5% <95.2%> (+0.3%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.