Fault ribbons from the fault's own surface, and the fault-patch multigrid (#629)

[lmoresi]

Fault-resolving meshes built from the fault's own surface, the solver machinery that makes them cheap, and the measured record behind both. This is the #629 campaign branch; the issue thread carries every measurement in sequence.

What this delivers

Mesh side — the fault is the input. The user supplies their own discretisation of the fault (a structured grid in 3-D, polylines in 2-D; curved is fine) and one call builds the resolved band by offsetting that discretisation — no remeshing — embeds it, labels and splits the mid-surface, and returns levels that nest by construction:

• place_fault_ribbon(base, sheet_grid, width) — 3-D: extruded prism band (always two layers across any width), split mid-surface as a boundary ready for add_fault_bc, nested 2:1 bridge level.
• place_fault_ribbon_2d(base, traces, width) — 2-D networks: several strands placed sequentially, stop-short (the junction ruling: the intact gap is the linkage), cut and split in one add_fault network call.
• place_thin_volume(..., mesher="ladder") — the underlying band builder. Straight 2-D traces keep the gmsh transfinite path; curved 2-D and all 3-D bands are pure numpy extrusions with mitred offsets and loud refusals (over-tight bends, inverted extrusions, protruding bands).
• mesh.cells_labelled(name, value) — empty-safe cell masks from placement labels.

Three contracts, each learned from a measured failure: the supplied geometry is honoured exactly (the tip margin is extrapolated surround along end tangents — shrinking the request biased 3-D peak slip by 14%); any painted rheology stops at the fault footprint, never the margin (whole-band paint made the fault mechanically longer than requested); and every coarser level subsamples one parametrisation (independently recomputed normals or reach vectors break vertex nesting — measured in both dimensions).

Solver side — the fault-patch multigrid. The design note (docs/developer/design/fault-patch-multigrid-2026-08.md) is the specification; in brief: ordinary Galerkin multigrid handles everything smooth, and small strong ASM subdomains keyed on the fault physics restore the multigrid contract where coarse spaces cannot see.

• set_custom_fmg(..., fac_zone=...) keys the finest level's strong patch on painted cells (volumetric fault representations only — a split-node fault needs no zone patch, by measurement); the structural (cut/split) rows are unioned in automatically, and the old solver._fac_zone_cells spelling raises rather than being silently ignored.
• Transfer hygiene: structural-zero weights dropped before the Galerkin product (the chain returns to native density), and native refine() pairs get the exact any-degree nested prolongation.
• The outer Krylov is flexible via the managed-option mechanism (this merge reconciles the branch's Stokes outer Krylov is PETSc-default gmres around fgmres inner solves — the flexible-outer rule applied to the sub-blocks but the outer was never set #624 fix with development's Stokes outer Krylov defaults to gmres while the fieldsplit sub-blocks are Krylov solves — must be fgmres #576 fix of the same defect).
• utilities/dm_labels.py fixes the The general 3-D outcrop cap: a zone may meet any boundary #589 empty-stratum getIndices() segfault at source; the dead is None guards in nvb, reconnect and fault_split (petsc4py returns a non-None NULL-handle wrapper) are routed through it. This piece is independent and cherry-pickable if the rest is held.

Measured record (rig scale; the #629 thread has full context)

• 2-D composed benchmark (enshrined as test_1022_composed_ribbon_fmg.py): FMG 8–9 iterations against GAMG 82 at native operator density; slip and leak identical; the zone patch takes a 1e4 contrast from 34 iterations to 2.
• 3-D curved fault: FMG 6–8 against GAMG 56–70, physics identical; [L0, L1] + finest reaches GAMG wall-clock parity with 8x iteration headroom; the AL penalty at gamma = 1 puts FMG ahead outright.
• Split vs TI on identical geometry: element budgets within 0.2%; the split representation is w-independent to 0.1%; TI converges onto it as w shrinks, with the bias largest on the strand with the smallest length/w.

Tests

New or extended: test_1022 (composed benchmark, tier B, negative control verified), test_0855 (3-D ladder geometry: exact volume, curvature refusal, nesting; the curved 2-D ladder; the two-strand 2-D wrapper), test_1017_nested_native_and_fac, test_0860–0862 (outcrop, daylighting, conforming sheet), parallel ptest_0854/0855. Touched suites and level_1 and tier_a pass locally after the development merge.

Known-remaining, deliberately out of scope

• The UW_FAC_* / UW_CUSTOM_MG_* / UW_MG_SMOOTH_ITS environment knobs are still TODO(MEASURE) A/B affordances; retiring them awaits the production-proportion measurements.
• The FAC patch split and the parallel split of fault networks are serial; the parallel design rules are recorded in the design note (seam consistency is the one item with valence).
• visualisation/glyphs.py duplicates PR Visualisation: principal-stress glyphs and stress trajectories #601's file byte-for-byte so the fault figures work from this branch; whichever PR merges second dedupes cleanly.

Underworld development team with AI support from Claude Code

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[lmoresi]

Adversarial review (self-review per project practice; findings verified against the diff, most significant first).

1. The honoured-paint rule is documentation, not API. The campaign's measured lesson — rheology painted on the whole band makes the fault mechanically longer than requested (tip lobes ~2 elements past the mapped tips) — is enforced nowhere: both wrappers return band labels only, and every consumer (our own rig scripts included) reconstructs the fault footprint by hand with geometry-specific code. The mistake we made twice remains one careless mask away. Proposed fix, cheap and general: the wrappers already know each trace's extended parametrisation, so a nearest-sample test (is the centroid's nearest rung a user point or an extension point?) yields a per-strand footprint mask to return in info. We will push this.

2. place_fault_ribbon_2d accepts duplicate trace labels. Nothing rejects [("Main", A), ("Main", B)]; the failure surfaces downstream in the add_fault network call with an ambiguous-boundary shape rather than at the API boundary. One-line check. We will push this.

3. The ladder domain gate is bounding-box only. A structured band inside the domain's bbox but outside a non-convex domain passes _ladder_domain_gate_2d/the 3-D check and fails later in the carve with a less specific refusal. Loud either way, so correctness holds, but the error names the wrong cause. Documented limitation; a point-in-polygon gate is a follow-up, not this PR.

4. visualisation/glyphs.py duplicates PR #601 byte-for-byte (declared in the description). Reviewers should prefer merging #601 first; whichever lands second dedupes with no conflict, but two open PRs carrying the same file is a state to close quickly.

5. Scope notes verified rather than assumed: the UW_FAC_*/UW_CUSTOM_MG_* knobs remain TODO(MEASURE) as declared; the split path at np>1 goes through the gathered-serial topology (guards present at the four entry points), so the serial-only claim is about the FAC patch split and network seams, as the design note states.

Tests after the development merge: touched suites 66 passed; level_1 and tier_a 1065 passed / 31 skipped / 1 known xfail.

Underworld development team with AI support from Claude Code

[lmoresi]

Review findings 1 and 2 are fixed on the branch: the fault-footprint masks are returned by both wrappers (finding 1 — the honoured-paint rule is now API, with tests asserting the footprint is a strict subset of the band per strand), and duplicate trace labels are refused at the API boundary (finding 2). Finding 3 (bbox-only domain gate) remains a documented limitation with a loud downstream refusal; findings 4 and 5 are state, not defects.

Underworld development team with AI support from Claude Code

[lmoresi]

CI triage and fixes (f31b9eb), one cause at a time:

• test_1021 (2 failures — genuine, reproduced locally): the test asserted the pre-Geometric aggregation for placed-hierarchy FMG: coarse levels as decimated point clouds, not band-only meshes #629 blanket rule that the rotated coarse solve is always svd. This branch gates it — svd only when a VERIFIED rigid-rotation null mode exists, since blanket svd was measured as most of ~0.8 s per V-cycle (Custom-P V-cycles across a cut level cost more than they save (7x slower than GAMG at 11x fewer iterations) #622) — and the bundle fixtures pin the inner annulus boundary, so no rotation mode survives and redundant/LU is correct there. The assertions now state the gated semantics, and the svd arm keeps fix(rotated-bc): Schur preconditioning parity + 3D rotation nullspace — Zhong #248 blow-out (44 its -> 1) #306's regression protection via a new test: free-slip on BOTH boundaries, where the rotation mode is genuine, the recorded mode count must be 1 and the coarse solve must be svd. All 19 tests in the file pass locally.
• test_0861 (2 failures — CI-only, pass locally): the setback placements' wall margin held for our local gmsh but not CI's — the carve cavity extends ~(clearance+1)·h_local, and at the worst local h a different gmsh deals this box, the 0.6 default clearance overruns the 0.25 setback. The tests now use clearance=0.3 (the measured production thin-shell choice), which keeps the cavity inside the margin at worst-case h.
• test_0054 (1 failure — pre-existing CI flake): the hang-watchdog family is flaky on CI independent of this branch — development's own latest completed CI failed the sibling test_0053, and nothing on this branch is reachable at the import path the watchdog arms (the new glyphs module is not imported by visualisation/__init__). No change; rerunning.

Underworld development team with AI support from Claude Code

[lmoresi]

test_0054 root-caused and fixed (4b5ad9c). Native thread samples of two live hangs establish the mechanism: faulthandler.dump_traceback_later(repeat=True) walks live frames from its C thread without synchronisation, and against a still-importing interpreter that walk loops forever (the C thread pinned in dump_traceback for an entire sample window, the main thread starved mid-import) or reads garbage and dies at SIGSEGV — the CI -11. The environment-armed watchdog was arming during import underworld3.mpi, inside the very import it then dumped over. The trigger is conditional, which is why it read as flaky: it needs a slow import (cold bytecode caches — CI always, or locally straight after a build) and piped child output (exactly how the test runs); this branch's larger import graph pushed the import past the 1.0 s trigger where development stays under it.

Four hardenings, the last structural: the reporter thread pre-loads everything it touches and formats stacks without source-text reads (lookup_lines=False — file, line and function carry the hang report); the reporter re-arms only its own timer, since re-issuing dump_traceback_later cond-waits on the C thread mid-dump; and the environment-armed watchdog now arms at the END of import underworld3, trading self-coverage of the import graph for safety everywhere the tool is for. The previously first-run-reproducing context went from 12/15 frozen to clean under the test's own scenario, and the test passes 6/6 locally.

One latent issue noted for the record, pre-existing and untouched: a natural process exit while a repeat dump is in flight can hang finalisation (the atexit disarm waits on the dump). The test SIGKILLs its child and never sees it; an env-armed production run that exits during a dump tick could. Worth its own small issue.

CI re-running now.

Underworld development team with AI support from Claude Code