fix: preserve original source lvolID through failback cutover by swap… - #1276
Open
geoffrey1330 wants to merge 107 commits into
Open
fix: preserve original source lvolID through failback cutover by swap…#1276geoffrey1330 wants to merge 107 commits into
geoffrey1330 wants to merge 107 commits into
Conversation
…turns a dict rpc_client.subsystem_get returns ONE subsystem dict (single_or_none), not a list. Indexing it with [0] raised KeyError(0), the helper's best-effort except swallowed it as a warning, and the eviction silently never executed — run 20260824_104449 failed with the same 40x add_ns -32602 while the fix was nominally in place. The test fake modelled the wrong shape too (a list), which is exactly how the bug got past the suite; it now models reality. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ver adds nvmf_subsystem_remove_ns acknowledges before it completes (the same async false-success that dropped a shared subsystem in the PVC-expand incident, whose fix polls for confirmation). The fail-back eviction removed the stale namespace and add_ns raced the removal and lost on all 8 retries -- run 20260824_110959: 40 evictions logged, 40 add_ns -32602 right behind them. The eviction now polls the subsystem until the namespace is actually gone (bounded, 20s) before returning, matches the stale entry by uuid as well as nsid (the preserved identity collides on both axes), and the test fake now models the acknowledged-but-lingering removal so this race stays pinned. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ot just the primary Run 20260824_113711 proved the eviction itself now works: the primary's add_ns returned result:1 for the first time. But the preserved-NQN subsystem exists on EVERY node of the recovered HA set, each still holding the original volume's namespace at the preserved nsid. The HA peer's add_ns failed with the same -32602 the eviction was written for, add_lvol_on_node's peer failure rolled the whole cutover back, and all 5 fail-back cutovers died on max retry (0/5). _create_target_lvol_clone now calls _evict_stale_namespace for each online HA peer right before that peer's add_lvol_on_node, exactly as it already did for the primary. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…attempt; add soak cases 7-9 Case 6 (run 20260824_144226) exposed the stall: a node outage mid-create left a REP_* landing volume whose id was never stored on the task, and every retry of the transfer died on "LVol name must be unique" (~31s loop for the rest of the run), stalling three volumes' chains and the case behind the 180s lag gate. The runner now probes for a record already wearing the derived name before creating: adopt it when online, wait when in_deletion, force-delete when half-created. New soak cases per the extended test plan: - case 7: 20 namespaced volumes on 2 shared subsystems (10 ns each), randomly assigned across 2 clients, replication + fail-over + fail-back with NQN/nsid identity asserted per namespace. The deployer gains an add_client mode to grow an existing lab to 2 clients. - case 8: sequential-pressure catch-up: 64k/QD64/4-job fills of 50G per volume, repeated; peak backlog and catch-up time are recorded and the backlog must drain under the lag gate every cycle. - case 9: chaos: random SPDK-container kills on BOTH clusters' nodes while replication runs (seeded, logged with the pipeline phases active at each kill), then full catch-up + fail-over integrity verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…sn shutdown's stderr Run 20260824_153107: case 6's sn shutdown was silently REFUSED because the source cluster was still ACTIVE - REBALANCING from the previous case's restore (open device_migration + balancing_on_restart tasks); the refusal went to stderr, which the driver's exec channel drops, so the test stared at an online node for the full 900s budget. Cases 5 and 6 now wait for the victim's cluster to settle before shutting the node down, and the shutdown command merges stderr so a refusal is visible. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… not nqn alone Soak case 7 (run 20260824_174611) caught this on its first run: namespaced volumes SHARE a subsystem, so replicate_lvol_on_target_cluster's existing-copy guard -- an nqn-only match -- fired for every namespace after the first. Namespace 1's fail-over copy already carried the shared nqn, so namespaces 2..N returned ITS target lvol id and were never failed over at all: 9 of 10 volumes silently absent after a DR fail-over, with every call reporting success. The guard now compares nqn AND ns_id, which is the identity a fail-over copy actually preserves. The existing idempotency test passed only because its fake left the already-failed-over copy at the model-default nsid while the source carries nsid 7; a real copy preserves the source's nsid. Fixed, and a companion test pins that a SIBLING namespace on the same nqn is not mistaken for this volume. Also scale the leftover-volume drain budget with the number of victims: case 7 leaves 20 namespaced volumes plus their REP_* landing copies, and 37 of them did not drain inside the flat 300s, failing case 8 in its prologue. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…oss chaos kills CHAOS_EVENTS and friends can now be set per run (stage_and_run_repl_cases case9 CHAOS_EVENTS=100) instead of editing the driver on the box. Case 9 also records whether fio survived each kill: that is the promotion-window signal the spdk ANA-transition fix targets, so a chaos soak now reports it directly instead of only asserting recovery. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…face fail-over errors - SPDK_IMAGE -> main-2a03661a-amd64 (sha256:961410ae...), the first ultra build containing the promotion-window ANA-transition fix (spdk R26.3 554c80f11). Verified it was built FROM spdk-core:R26.3-latest whose manifest was created at 18:41:57, before this ultra build started at 18:42:52 -- the floating-tag manifest race makes that check necessary. - CLIENT_COUNT 2, so a namespaced (case 7) run does not need add_client. - do_failover now captures the controller's error and log output. A bare 'returned no connection strings' cost two lab runs to diagnose; the reason was being written to a stderr nobody read. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two unrelated, pre-existing issues currently failing every PR's "Python checks" job (found while chasing an unrelated PR's CI failure — these files aren't touched by that PR at all, so the merge-commit CI check was just surfacing main's own already-broken state): - tests/unit/test_ucs_debounce_bound.py: unused `MagicMock` import (F401). - simplyblock_core/test/test_replication_chain_completeness.py: unused `import copy` (F401), and a lambda that sequenced `added.append(...)` with a return value via a `(side_effect, value)[-1]` tuple-index trick -- mypy's func-returns-value check flags embedding append()'s None result in the tuple literal. Replaced the lambda with a small nested function that does the append as its own statement and returns the real value normally; same behavior, no trick, no warning. No functional change -- verified via pytest (22 passed) and unittest (7 passed) on the two affected test modules.
Soak case 7 (run 20260824_215758) got 14 of 20 namespaces failed over and then died in add_ns on the 15th. The replication destination is chosen per volume by capacity (_get_next_3_nodes), with nothing tying volumes that SHARE a subsystem to one target node. Because a fail-over copy preserves the volume's NQN and nsid, scattering siblings splits a single shared subsystem across unrelated target primaries: each one advertises the same NQN carrying only its own subset of namespaces, and a sibling whose nsid is already taken there cannot be added. A volume that shares its NQN with an already-replicating sibling now inherits that sibling's replication node. Also raise the soak's fail-over diagnostic capture to DEBUG: the SPDK response behind 'Failed to add bdev to subsystem' is logged there, and the controller only re-reports its own generic message. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… a kill Run 20260824_224909 showed the soak damaging the cluster instead of testing it: the recovery gate accepted status == online while health_check was still False and the source cluster sat in SUSPENDED/IN_ACTIVATION, so kills kept landing on a half-recovered 2-node cluster. It now requires every node online AND healthy and every cluster active/degraded before the next event. fio also died at event 3 (the promotion-window EIO) and stayed dead, so the remaining 97 events would have run against an idle client. The case now reconnects and restarts the workload after an outage, and still reports every death. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Gracefully shutting one node of a 2-node cluster made the SURVIVING node
abort its own SPDK application:
JC detected a network outage nd=1 njms=2
JC aborts the node due to network outage
spdk_abort_node: Forcing application shutdown via abort. (core dumped)
The journal component requires jc_ha_nmin_jms() reachable journals --
2 normally, 1 when the dual-node flag is set. A 2-node cluster losing its
peer is left with 1 of 2, so the survivor fail-stopped. Every client path
vanished at once, fio took hard EIO and XFS shut down (soak case 6), and
the cluster went SUSPENDED. The spdk fork implements the tolerance and
exposes the jc_set_dual_node RPC, documenting that in a dual-node
configuration a single connected JM is enough -- but nothing in the
control plane ever called it, so no 2-node cluster has ever had it on.
apply_jc_dual_node() now sets the flag across the cluster whenever a node
is added (so growing 2 -> 3 also CLEARS it) and whenever a node is
brought back up (a restarted node returns with the JC default). The flag
tracks MEMBERSHIP, not the online count: a 3-node cluster with one node
down must keep requiring two journals.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…554c80f11) Iteration 12 of the 2026-08-24 run failed fio verify on four blocks, and the received magics were random (c32e, b24e, e6b4, c8b7) rather than fio's constant 0xacca. Stale-but-valid data would still carry acca and fail as an offset or crc mismatch, so those buffers never held an fio header at all -- the map named a location and the bytes handed back were not that location's content. The placement history exonerates the map at every observable point: no steady-state re-home across the nine dump points of iterations 11-12, every group back on its pre-outage home, and the three roles identical at range level (12,810 groups) at post_incident. ultra b44de698 fits that shape: a reactively promoted distrib used to signal JC leadership immediately while, under write protection, the parity desynchronisation check was still outstanding, so reads served in that window came off desynchronised parity. The signal now waits for the check to finish. spdk moves a311a6852 -> 554c80f11, which keeps the upstream retry-state fix (#3686) as an ancestor and adds the promotion-window ANA transition fix. The gate now pins the ultra commit as well, so a stale image cannot quietly re-test the build that already failed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… a retained count With the JC dual-node fix in place case 6 finally kept the client alive through a primary outage (fio_alive=True, replicated 5 -> 13 during the outage) and then failed on its own assertion: during=13, after=10. replicated_count counts RETAINED replicated snapshots, and retention keeps only the newest generations, so it is bounded and falls after a burst. It cannot express 'replication resumed'. Cases 5 and 6 now require a point-in-time created AFTER recovery to reach the target, which is what resumption actually means, using the harness primitive that already exists for it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replication kept a flat count of internal snapshots, so history never went
back further than a couple of cadence ticks and a fail-over could only ever
land on 'a minute ago' -- useless against a logical corruption, which the
newest copy has faithfully replicated.
A replication policy can now carry a retention SCHEDULE:
--retention-schedule '15m:2h,1h:11h,1d:7d'
'one snapshot every 15 minutes for the last 2 hours, then hourly for 11
hours, then daily for 7 days'. Snapshots past the total span are pruned.
Selection is a pure function of (times, schedule, now) in the new
snapshot_retention module, so it is unit-tested without a cluster: a
minute-cadence stream over 8 days collapses to ~25 retained snapshots that
still reach back 6+ days.
The schedule never overrides MIN_KEEP_REPLICATED: the newest pair is always
retained, because deleting a snapshot swap-merges its segments into the
successor chained to it. An unparseable schedule is rejected when the policy
is created, and if one ever reaches the runner it is logged and treated as
'no schedule' rather than crashing replication or silently dropping history.
Fail-over gains a generation selector: generation 0 is the newest replicated
point-in-time (previous behaviour), higher values walk back through the
retained history. Asking for more generations than exist is an explicit
error rather than a silent fall back to the newest.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… a supported configuration The product minimum is 3 nodes. The 2-node src/fresh clusters this lab used are exactly what produced the 2026-08-24/25 failure chain: with one node down the survivor holds 1 of 2 journal members and the JC aborts it (whole-cluster outage on a single node stop), and the restart rebalance has no third failure domain to place into, so device_migration loops on 'no allowed placement' forever and pins the cluster in REBALANCING, blocking sn shutdown and every node-down test behind it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A policy created with --consistency-group makes all attached volumes ONE crash-consistent unit: 1) Same-LVS invariant: the group pins to the first member's node/LVS. Attaching the policy to a volume elsewhere FAILS before any state is written; creating a volume under the policy forces placement onto the pinned node (an explicit conflicting --host is an error). 2) Group snapshots: an auto-managed ConsistencyGroup record lives and dies with the policy. One SPDK call (bdev_lvol_snapshot_group, spdk branch consistency-groups) freezes IO on every member blob, snapshots them one after the other, and unfreezes; SPDK unfreezes-first-then-GCs on mid-sequence failure. The controller mirrors the single-snapshot path per member around that call: replica registration, records, chain links, events, replication tasks. All-or-nothing: any registration or record failure rolls the whole generation back, and the group_seq counter moves only after full success. 3) The cadence (snapshot_monitor) snapshots CG policies as a group - one generation per tick with group-wide back-pressure - and its members leave the per-volume loop. 4) Membership epochs: a late joiner is active from the FIRST group snapshot after its attach (joined_seq = last_group_seq + 1); a detached member's epoch closes at the current generation. Failing over to generation N warns which current members that point-in-time does NOT contain and which contained volumes are no longer members - in the controller result, the CLI failover table, and the API response (200 + warnings body instead of the empty 204). Group provenance (group_id, group_seq) is stamped on member snapshots and travels onto the replicated target copies, which is what the fail-over generation selector actually returns. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… its namespace bdev_lvol_clone_register acknowledges before the bdev is examinable -- the third member of the acknowledge-before-complete family, after remove_ns (PVC-expand, eb127ee) and the case-3 eviction. The HA-peer leg of a namespaced fail-over issues nvmf_subsystem_add_ns immediately after the register and lost the race every time: -32602 with the peer's subsystem EMPTY, while the bdev existed moments later (run 20260825_122423, LVS_13/LVOL_121, uuid f01ea33c). The peer failure then rolled back the whole fail-over of that namespace. The stack build now polls get_bdevs (bounded, 20s) after clone_register and only then proceeds to the namespace add. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…inery
A finished transfer retires its REP_ landing volume by emptying the
record's bdev_stack (the blob lives on as the converted, chained
snapshot) and removing the record. Routing that retirement through
delete_lvol flipped the record to in_deletion first, so any interruption
before remove() stranded a record the monitor can never finish: with an
empty stack there is nothing to issue, the delete-status poll answers 4
('no async delete request exists') forever -- 856x in 30 minutes in run
20260825_125156 -- and every cleanup that waits for volumes to drain
times out behind it (case 7: 33 stuck, case 9: 2 stuck, both dead in
their prologues).
Two-sided fix:
- the retirement path tears down the nvmf plumbing DIRECTLY per node
(delete_lvol_from_node(force=True); the empty stack means no blob
work) and then removes the record, never entering in_deletion;
- the monitor retires any in_deletion record with an empty bdev_stack
record-only. Deliberately NO fallback delete of top_bdev: that bdev IS
the converted snapshot, deleting it would destroy replicated data.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
app_LVolMonitor received only the SHARED files, so a fix living in lvol_monitor.py itself (the empty-stack record retirement) verified green while the running monitor never contained it -- the import probe checks the shared modules, not the service's own file. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
case 10 -- online migration under sustained heavy IO: 3 volumes per source node at 8k randwrite QD64 x 4 jobs, replication must reach and hold a bounded lag against that load, the final online migration must complete for every volume with fio uninterrupted, and the IO freeze during each cutover is MEASURED as the client actually experiences it (100ms heartbeat probe per volume; the largest gap inside the commit window is the freeze; 30s sanity bound). case 11 -- retention ladder + generation fail-overs: policy with --retention-schedule 5m:15m,7m:30m,10m:1h, ~2h of history under a 30s-fsynced record stream, then (a)/(b) kept-and-pruned verified INDEPENDENTLY of the retention code (gap/count/horizon analysis per tier), and (c) three rounds of fail-over to a randomly selected older generation with exact-data validation (every record up to the generation's snapshot present, none after), each followed by a full fail-back cutover. case 12 -- consistency groups (gated: requires the consistency-groups build; refuses with a clear message otherwise): CG policy, 3 members pinned to one LVS, strictly ordered fsynced writes A->B->C, group generations verified complete (every group_seq covers all members), retention per schedule, then fail-over to the latest and to a random earlier generation verifying BOTH generation correlation and crash consistency: seq(A) >= seq(B) >= seq(C), max skew 1. Policies with extra flags get distinct names so a schedule/CG policy never silently reuses a plain one another case left behind. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ts NQN; pin the master-line image _remove_lvol_subsys_from_node deleted the subsystem whenever it observed it empty. For per-volume subsystems that is correct cleanup; for NAMESPACED volumes the shared subsystem is legitimately empty in the window between one member's teardown and the next member's add -- and a stuck in_deletion member's retry loop observes that window sooner or later. Run 20260825_224221: 8 of 20 namespaced fail-overs landed, then a looping rollback record deleted the shared subsystem on the HA peer, and every following member's nvmf_subsystem_add_ns died -32602 against a missing subsystem. Delete-on-empty now checks for other live claimants of the NQN on the node first; the LAST member out still removes it. Also pin the deployer to sha256:a3854cd4 (main-d91ff03a-amd64), the first ultra build FROM spdk-core:master-latest. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… the outage
Two defects found by the 2026-08-25 soak, both in the repair path.
Duplicate paths. repair_multipath_controller() is fanned out over a thread
pool with no serialization, so two workers read the same missing={ip} and both
attach it; SPDK admits both because its -EEXIST guard runs before the async
probe and compares only the active path, and the target issues two cntlids. The
result on 2026-08-25 was remote_jm_1e7ff71e carrying (96.179, 97.9, 97.9). It
was never repaired because _collect_attached_ips() returns a SET, so the
duplicate read as 2-of-2 and the control plane called the node healthy while
the soak's own verifier counted 3-of-2 for 900 s and gave up.
So: a per-(node, controller) lock, non-blocking because the loser of the race
has nothing to add; duplicate_attached_paths() over a path LIST; a prune that
detaches the duplicated address and lets the existing missing-path loop
re-attach it exactly once (SPDK's bdev_nvme_delete removes every controller
matching a trid, so one copy cannot be singled out); and a refusal to prune
when the address is the only one attached, which would take the bdev down
instead of repairing it. health_controller now reports a duplicate as
UNHEALTHY rather than passing it silently.
Dial holds outliving their cause. dial_backoff exists so a refusing address
cannot burn a healthy node's app thread on connect polling, and that is still
wanted. But the ceiling was 300 s and a hold clears only on a success that
allowed() refuses to let anybody attempt, so it had to time out. Every
all-nodes NIC flap in run 20260825_155730 therefore stalled path healing on a
plateau of exactly 15 missing paths -- all of them one held address -- for
250-306 s, i.e. the ceiling, while the address had been reachable within 30 s.
All three data corruptions in that run happened inside those windows.
Ceiling drops to 60 s: a probe once a minute does not burn app-thread time, so
the breaker keeps its purpose while bounding how long a returning path stays
unrepaired. New clear() drops a hold on evidence of reachability, wired to the
one piece of evidence available locally -- an address with a live enabled path
on this very controller. Deliberately NOT driven by the peer's DB status: a
node whose record says ONLINE while its SPDK is dead is the case this module
was written for.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…longer restart wait The harness ran for 1h56m through a data corruption and reported PASS on every checkpoint. fio prints verify failures WITHOUT the "fio: " prefix -- verify: bad magic header a8a4, wanted acca at file .../vol2/soak_mp_2.1.0 -- and FIO_HARD_ERROR_MARKERS carried "fio: verify" and "verify failed", neither of which matches that. So vol6 (18:53) and vol2 (19:43) corrupted silently while their fio kept running, and the fault surfaced only at 20:49 when vol4's fio *process* died and the rc-file branch caught it. A data verification failure is the most important thing this harness can find; it must never again depend on fio also crashing. Markers now cover the unprefixed forms, checked against the three real failure lines from that run. verify_dump=1 so a mismatch writes the received and expected 4 KiB buffers. Every corruption so far has died with the returned bytes unidentified -- we could not distinguish stale data from parity noise from a neighbouring block, and the volumes live on instance store, so they vanish when the fleet stops. That is how the 08-24 evidence was lost, and it nearly repeated on 08-25. RESTART_TIMEOUT passthrough in the launcher: the 900 s default aborted run 20260825_085018 at iteration 4, where a JC abort plus a stranded controller reset meant both nodes needed ~36 min to return -- and they did return healthy, with no fio error. Until that recovery time is fixed, a longer wait measures the product rather than the harness's patience. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two defects in the same rollback, both silent: 1. A peer failure removed the copy from the PRIMARY only. With three target nodes, a failure on the tertiary left the SECONDARY holding the namespace. The next sibling's primary then auto-assigned an nsid the peer had already given to another volume, and its replica add was rejected -- "wanted nsid=2 ... holds=[(2, <other volume>)]" (case 7, run 20260826_223631). Divergence between the paths of one shared subsystem is exactly what the primary-assigns/replicas-reuse rule exists to prevent, and the rollback was creating it. 2. It passed the LVol and StorageNode RECORDS to delete_lvol_from_node(lvol_id, node_id), whose "except KeyError: return True" swallowed the type mismatch -- so the rollback reported success while deleting nothing at all. The fail-back clone had the same call. The fail-over clone now tracks every node it placed the copy on and removes it from all of them (sync on the replicas, which is what a non-leader accepts), and both sites pass ids. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Soak case 7 failed six times on one shape: 20 volumes across 2 shared subsystems, fail-over, and an add_ns rejected on an HA peer -- wanted nsid=1 max_namespaces=10 holds=[(1,531e4060) ... (7,ebdd3e0d)] The peer held the whole group while the copy's primary held none of it, so the primary auto-assigned 1 and the peer already had 1. Two defects: 1. The sibling-affinity pick added earlier ran against the ADVISORY nqn from _resolve_lvol_subsystem. claim_lvol_ns_slot's transaction recounts with concurrent creates visible and may join the lvol to a DIFFERENT subsystem, leaving it in a group whose other members replicate elsewhere. The pick is now re-derived after the claim (_realign_replication_node_after_claim). 2. Auto-assignment on the primary is only safe when the primary sees the whole subsystem. _claim_target_nsid now claims from the union of what every node of the target HA set actually holds and hands the same number to the primary and every replica, so a split group cannot collide even if one is reintroduced by a node that was down at placement time. It returns 0 -- auto-assign, unchanged behaviour -- when the subsystem exists nowhere yet or any node is unreadable, since claiming against a partial view is the bug being fixed. The nsid is NOT carried over from the source cluster: it is local to a subsystem's HA set and clients resolve their paths through connect_lvol. test_failover_target asserted the source's number; it now asserts the claim, and its add_lvol_on_node fake -- which never assigned an nsid at all -- was made faithful to the real primary add. The nsid-consistency fake likewise answered "7" whatever was requested, hiding a claim/persist mismatch. 13 new unit tests for the claim and the realignment; 1895 pass.
Run 20260826_233417: the nsid claim worked -- all ten copies got distinct nsids 1..10 -- yet case 7 still failed, because the ten copies of one subsystem landed on THREE target primaries: a3220281: 1,2,3,6 50037447: 1,2,4,5,7,8,9,10 5198fb03: 3..10 No node advertised the whole subsystem. A shared NQN whose paths expose different namespace sets is incoherent to the client kernel, which showed 8 of 10 namespaces on one client and 0 on the other. The sibling rule was only in add_lvol_ha. `volume add --replication-policy` does not go through it: replication_start attaches the policy and picked the target node purely by capacity. It now consults _sibling_replication_node first, and keeps the subsystem whole even when that node is the snapshot origin it would rather avoid -- placement is an optimisation, a coherent subsystem is not. _create_target_lvol_clone re-checks at creation time as the last line of defence: the target node was chosen when the policy was attached, so it can be stale by the time the copy is built. If the subsystem already lives somewhere else in that cluster the copy follows it, and if that home node is offline the copy fails loudly rather than silently splitting the subsystem. Two test fakes were missing state their real counterparts have (StorageNode .cluster_id, DBController.get_lvols) and were completed rather than worked around. 1900 pass.
Run 20260826_235940 failed over all 20 volumes correctly -- both HA pairs exposed identical namespace sets, clients saw 10/10 -- then wedged on fail-back with all 20 cutover tasks suspended on: Subsystem ...e3f4f160 is full on the target HA set (10 namespaces); no nsid to claim for 9b78b3f7 After a fail-over the ORIGINAL volume stays on its cluster still holding its namespace. The fail-back returns into that same subsystem, and a shared subsystem is sized to its group (max_namespaces=10 for ten volumes), so there is no slot for the returning volume. Dedicated subsystems hid this for three years of cases 3/5/8: sized 1 they were equally full, but each fail-back minted a fresh per-volume NQN and never came back to an occupied one. The original is now deleted before the returning volume is built, which is what frees the slot. Its SNAPSHOTS are deliberately left alone: the newest one still present on that cluster is the common base the fail-back clones and transfers a delta against -- that is what makes a fail-back an online migration rather than a full copy. delete_lvol only removes a parent snapshot that was already soft-deleted, so the base survives. The volume being superseded is resolved through the LVolReplication record (source_lvol of the record naming this volume as target), not by matching the uuid at the tail of the NQN: with a shared subsystem all ten volumes carry the SAME NQN, so that match is ambiguous. A first fail-over has no such record and retires nothing. Three more test fakes were missing get_lvol_replication_objects, which the real DBController has; completed rather than worked around. 1905 pass.
R26.3 did not fall behind on the shared-subsystem work -- it grew its OWN answers to the same problems, so neither side could simply win. Ten conflicts, all in lvol_controller.py, each resolved to keep both: * add_lvol_on_node: theirs contributes the explicit primary_nsid argument, the ns_uuid, and the cross-LVS case A/B split (subsystem absent on this node vs nsid slot occupied); ours contributes the "say WHY" diagnostic and is_primary on every _fail_after_bdev. requested_nsid is now `primary_nsid or (lvol.ns_id or None)`: both callers dictate it, the explicit argument first, then the control plane's claim. The hard-fail theirs gated on `primary_nsid is not None` now keys on requested_nsid, so a claimed nsid gets the same protection -- re-claiming into another subsystem loses the shared NQN either way. * replication_start: both grew a sibling-affinity rule. Kept ours (the shared _sibling_replication_node helper, paired with the post-claim realignment, and the excluded-nodes case) plus the check theirs had and ours lacked: the sibling's node must be ONLINE, else fall through to a capacity pick and let _create_target_lvol_clone refuse to split the subsystem. * _create_target_lvol_clone: kept theirs' subsys_port fix, namespace-pointer resolution and per-peer cntlids, and ours' per-peer eviction plus the rollback across every node that already carries the copy. * _evict_stale_namespace: took ours. Theirs still indexes subsystem_get()[0], but that returns one dict -- the KeyError(0) was swallowed by the best-effort except and the eviction silently never ran. Two semantic conflicts the merge created, both resolved rather than papered over: 1. Migration and fail-over want OPPOSITE things from the nsid. During an online migration the client is attached to source and target at once under one NQN, so the positions must match; on a fail-over it reconnects and resolves paths through connect_lvol, so insisting on the source's number only collides with the sibling holding that slot. The claim is now gated on `not for_migration`. 2. That makes nqn+nsid an unreliable way to find "this volume's copy on the target": siblings share the NQN and the copy's nsid may differ by design. The guard resolves through the LVolReplication record instead, keeping nqn+nsid as the fallback for when no record exists yet. Five test fakes were missing state their real counterparts have (StorageNode.get_lvol_subsys_port / rpc_client, LVol.namespace / max_namespace_per_subsys, add_lvol_on_node's new kwargs); completed rather than worked around. 2016 pass.
repl_soak.py echoes every command it runs, and the checkout step's URL carries a personal access token (https://x-access-token:<token>@github.com/...). It was printed in full into the run log and into every lab log copied off the mgmt node -- logs that get pasted into tickets and chat and then sit on the instance for its lifetime. Redacting in log() rather than at the call site: a future command that happens to carry a credential is redacted by default instead of depending on whoever adds it to remember. The failure path prints the command and the captured streams too, so those go through it as well. The token that was already exposed needs rotating; this only stops the leak.
Case 10 (online migration under heavy IO, run 20260827_110415) measured the client-observed freeze at avg 40.4s, max 71.9s, with fio logging 8 errors. The freeze is bdev_lvol_transfer_final_step, which copies everything written since the cutover clone's base snapshot -- so the freeze is as long as the write window in front of it. Four things made that window enormous. 1. SHRINK_ROUNDS was a fixed 2: a count, not a convergence criterion. Under load it simply stopped while the delta was still large. Rounds now repeat until one transfers within REPL_CUTOVER_CONVERGE_TARGET_SEC (2s), bounded by REPL_CUTOVER_MAX_SHRINK_ROUNDS -- a volume written faster than it replicates freezes anyway, and says so, rather than looping forever. 2. Each round returned to the task scheduler, so TASK_EXEC_INTERVAL_SEC (10s) of fresh writes joined every round -- a floor no number of rounds could beat. The loop now polls at 200ms and takes the next snapshot immediately, before any yielding decision, because that IS the mechanism: a round must carry only what was written while the previous one transferred. The inline window scales with the last round (the runner is single-threaded, so a flat budget would stall other volumes' cutovers): short rounds stay inline, slow ones yield, and yielding is free there because the freeze is far away. 3. The operator preconnect gate sat BETWEEN the base snapshot and the freeze, so every second of it was a second the frozen step had to copy. With no operator the 120s fallback fired 34 times in that run. It is now opt-in via REPL_CUTOVER_PROCEED_REQUIRED (default off). Deployments whose operator posts cutover-proceed set it and keep that cost until the clone's base can be advanced after the signal -- noted, not yet built. 4. Nothing stopped the other volumes on the same lvstore from replicating through a cutover, stretching every round. Two priorities now share an lvstore: a volume in final cutover owns it for the convergence rounds AND the freeze, and consistency groups outrank loose volumes -- a group's members transfer in parallel with each other while everything else waits, which serializes groups against each other instead of interleaving them. A group cuts over as a group: members are exempt from a sibling's claim. 2037 pass.
…can hold Two harness defects from run 20260827_110415. Case 11 died on "mount: /dev/nvme0n1 already mounted or mount point busy" and never tested retention at all -- it tested case 10's debris. That case's cutover froze client IO for 25-72s, its 13 unmounts hit their 15s timeout, and the `umount -l` fallback only DETACHES the path: the mount stays live until the hung IO drains, so the device was still mounted when the next case connected. Every command in the cleanup path was best-effort, so nothing noticed. Cleanup now verifies its own result against /proc/mounts (the only source that does not believe a lazy unmount's exit code) and escalates: kill the holders with fuser -km first -- umount -l on a mount someone still has open never completes -- then unmount, disconnect, and re-read, up to three rounds. A case refuses to start on a client that will not come clean, so contamination is reported where it happens instead of failing the next case on something unrelated, and cleanup_client warns loudly when it leaves debris. Mounting also checks the device is not already mounted, which turns a bare rc=32 into the actual reason. Case 13 was killed after 50 minutes without running a single chaos round: its 20 policies x 2 volumes inherited the default 1-minute cadence, which asks for 40 transfers a minute. The lag settled in a stable 313-383s band with outstanding pinned at 35-40 against a 180s gate -- not diverging, but never converging, so it would have burned the full 2h timeout. These cases now use a 10-minute cadence (CHAOS_PHASE_INTERVAL_MIN) with the gate derived from it, and print both so a stuck setup is diagnosable from the log alone. 2037 pass.
Run 20260827_172734 deployed the convergence loop and the freeze did not move:
9-55s server-side, 20-77s client-observed. The loop was running and logging --
and every line read
cutover convergence: lvol=... round 1 transferred in -0.00s
Negative zero, twelve times out of twelve. replication_commit creates the
cutover task with shrink_round=1 and shrink_snap_id but never stamped
shrink_started_at, and the loop defaulted the missing value to now:
elapsed = time.time() - params.get("shrink_started_at", time.time())
so round 1 always measured as instant, always compared under the 2s target,
and always declared convergence. Exactly one round ran and the freeze then
copied everything written since that round's snapshot was TAKEN -- including
the whole time it spent replicating, which is minutes under load. That is the
entire freeze.
The task now carries the stamp, and an UNMEASURED round is treated as not
converged rather than as instant: a missing measurement must never read as
"the delta is small".
The other three fixes did work in that run -- the per-LVS exclusion logged 320
holds ("lvol b12f0312 is in final cutover on lvstore ..."), and no cutover
waited on the operator gate.
Every earlier test supplied shrink_started_at by hand, which is exactly why
they passed while the real creation path was broken; the new ones drive the
controller's own param block and the unmeasured-round path. 2039 pass.
Run 20260827_185009 stalled with the convergence finally working. All 20
volumes entered their cutover together and every one wrote cutover_lvs,
because the claim was a marker with no mutual exclusion. The replication side
then picked the arbitrary first claim as the owner and held everyone else --
including the other nine cutovers' own shrink snapshots. Per lvstore, one
volume progressed and nine sat at "round 1: waiting to replicate":
ACTIVE LVS CLAIMS: 20 (LVS_10 x10, LVS_13 x10)
currently HELD transfers: 53, snapshot tasks stuck in `new`: 75
and in the run before it, queued cutovers ran their shrink deadline down until
17 of them died of "max retry reached (8/8)".
The exclusion is what was asked for; the missing half is that a cutover which
cannot have the lvstore must WAIT rather than start. It now checks for an
existing owner before claiming: the earliest active claim wins (deterministic,
so two racing tasks agree instead of each seeing the other), and a loser
suspends with its shrink phase untouched, no retry burned, and its deadline
pushed out -- queueing is not a failure and must not time out. A consistency
group is exempt in the direction that matters: a sibling of the owner's group
joins it rather than queueing, because a group cuts over together.
2046 pass, including a behavioural test that fails if a queued task starts its
shrink phase, burns a retry, or lets its deadline run down.
Run 20260827_194551: all 20 fail-back cutovers ended as "max retry reached (8/8)" -- 160 failed attempts that produced not one log line, so three investigations of that run could not name the failing branch. _finalize's failure path only wrote function_result, and the max-retry branch then overwrote it, destroying the last trace of the cause. Each failed attempt is now logged with the lvol and the attempt number, the reason is kept in last_error where the max-retry branch cannot clobber it, and giving up reports "max retry reached (8) after: <cause>" instead of the symptom alone. The fast-retry branch (target node not online) says which node and what state, since that one can burn all eight attempts in 80 seconds. No behaviour change beyond observability -- the run has to be repeated to get the cause, and this is what makes the repeat worth anything.
…hrink-round delays
…d PYTHONHASHSEED-dependent collisions
… stop operator seeing perpetual failed_over
Seven case-7 investigations could not answer one question: of the 588s a
convergence round takes, how much is DATA TRANSFER and how much is
orchestration? Round duration was the only number available, and it spans the
snapshot, the landing-volume create, the hub attach, the transfer, the detach,
add_clone and convert on two nodes, DB writes, and up to
TASK_EXEC_INTERVAL_SEC of task-runner latency per state change. Every
hardware-level theory tested against that number -- qpair fair share, poll
groups, 2 MiB granularity, chain depth -- came out an order of magnitude off,
because the number is not a throughput.
New simplyblock_core/xfer_timing.py emits one parseable line per phase, each
carrying its own epoch stamp (container clocks are skewed from the host's, so
`docker service logs -t` ordering cannot be trusted across services):
XFER-TIMING t=... phase=transfer_complete lvol=1c8874f3 snap=a0f48bf5 \
round=2 ms=1843.2 bytes=33554432 mbps=18.2 ok=1
Instrumented, in pipeline order: take_shrink_snapshot, landing_volume_create,
hub_attach, transfer_submit, transfer_running (with offset -- the only direct
read on throughput), transfer_complete (with bytes), hub_detach,
chain_add_clone and chain_convert per node, replicate_finish, round_total,
round_gap_to_next_snapshot, and task_pass per runner pass so scheduler latency
is visible. The freeze is broken down separately: fence_source,
final_step_transfer, final_peer_add_clone, enable_target_paths, and
freeze_total measured fence -> paths-live, which is the window that has to fit
inside the client's 8s fast_io_fail_tmo.
Harness: fio's aggregate bandwidth has been written to /tmp/fio_repl.log all
along under --status-interval=15 and never read -- which is why three analyses
GUESSED the client write rate and were 5x low. fio_bandwidth() now reports it
at phase boundaries, and collect_xfer_timing() pulls the CP lines off the
services, including when the fail-back FAILS, which is exactly when the
breakdown is needed.
scripts/xfer_timing_report.py turns a collected dump into the breakdown, with
an UNACCOUNTED line: if round_total dwarfs the sum of its measured parts, the
missing time is somewhere nobody is looking, and that gap is the finding.
Validated against a synthetic dump rather than discovered on the lab.
Pure instrumentation, no behaviour change. Two test fakes were missing methods
their real counterparts have (LVol.get_id) and were completed. 2048 pass.
Soak run 20260827_224741: two volumes on a 1-minute cadence for 124 minutes under `--retention-schedule 5m:15m,7m:30m,10m:1h` ended with 2 and 3 internal snapshots, at CONSECUTIVE cadence ticks 63-64s apart. Nothing in the log said which retention path had run, so the investigation could not tell an over-pruning bug from a dead cadence. The ladder itself is provably correct -- the new test feeds it exactly the history that run should have produced (124 x 1-minute snapshots) and it retains 14 points with gaps 300/300/360/420/420/420/180/300/600... So retention is NOT over-pruning. What the test also shows is that an ABSENT schedule keeps exactly the 2 newest -- consecutive cadence ticks -- which is precisely the shape the soak produced. That makes "the schedule never reached _prune_internal_snapshots" the leading explanation, even though the CLI -> add_policy -> policy.retention_schedule -> get_replication_policy_by_id chain all inspect correctly (the lookup even normalises the cluster/uuid form). So _prune_internal_snapshots now logs, on every prune, whether it is applying a schedule (and which) or falling back to FLAT keep-newest, plus the snapshot count and keep floor. The next case-11 run answers the question from its own log instead of needing another two-hour repro. The test's spread assertion allows smaller gaps at TIER BOUNDARIES, where the last bucket of one tier legitimately sits close to the first of the next.
The instrumented run finally located the 588s. Run 20260828_115307, case 7
fail-back, 1087 timing events:
polls per in-flight transfer : p50 = 1 max = 2
gap between polls : p50 = 81.2s p90 = 105.0s
transfer states ever seen : {'Done': 86} -- never once "In progress"
duration vs bytes : pearson r = -0.532
zero-byte transfers : p50 34.1s, as costly as data-carrying ones
total moved : 0.18 GiB across 4781s of "transfer" time
Every poll found the transfer already finished, and transfers that moved NOTHING
cost the same tens of seconds as those that moved data. So this was never data
movement: transfers complete fast and then sit COMPLETED BUT UNNOTICED until the
runner's next pass revisits that task. A convergence round cannot take its next
snapshot until the previous one is marked replicated, so ~50s of observation
latency per round, over a dozen rounds, IS the 588s.
Every hardware theory we tested -- qpair fair share, the single nvmf poll group,
2 MiB granularity, chain depth, COW, network saturation -- was aimed at making
data move faster, and none of them could have changed this number.
Two sources of latency, both removed:
1. Submit and the Done check happened on DIFFERENT passes. The submitting pass
now waits for the transfer, polling every 100ms, and runs the finish (chain,
convert -- what sets target_replicated_snap_uuid) in the same pass. The wait
is bounded so one slow transfer cannot hold the single-threaded runner: a
volume whose cutover owns its lvstore gets a generous budget (nothing else on
that lvstore can run anyway), everything else gets 5s and falls back to the
old pass-based path.
2. The cutover runner's loop slept 3s after EVERY task that returned False --
the normal result for a queued or mid-round task. With 20 volumes that was
20*3 + 10 = ~70s per pass, matching the measured 75s per-volume spacing. The
per-task backoff is gone and the pass interval is now adaptive: 200ms while
any cutover is mid-round, the normal interval when the cluster is idle. The
owner lookup also takes the prefetched task list -- it re-read get_job_tasks
per task, which is O(N^2) DB reads and unaffordable at a 200ms cadence.
Also: the snapshot monitor now logs a deferred cadence tick with the snapshot it
is waiting on. Case 11 ended with 2 internal snapshots after 124 minutes at a
1-minute cadence and nothing said why.
Worst-case path is now 100ms to notice completion plus 200ms for the converging
loop to see the marker. 2063 tests pass, including ones that pin both intervals
below a second.
…ting and including self in owner sort
…light, sync loop, and shrink deadline reset on retry
…g replication-features branch
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
…ping new clone UUID in FN_REPLICATION_FINAL