Document the long-HDF5-path segfault and the Gadi MatMult placement findings (#645, #635)

docs/advanced/performance.md

@@ -22,6 +22,71 @@ Tools for identifying bottlenecks:
 - Batch operations for better performance
 - Parallel scaling
 
+## Choosing MPI ranks for a MatMult-bound Stokes solve (#635)
+
+A large nonlinear Stokes solve is usually **memory-bandwidth bound, not
+compute bound**. In a controlled Gadi campaign on a 2-D power-law Barr–Houseman
+fault benchmark (P2/P1, cell size 0.01, `n=3`, viscosity contrast 1e5, tolerance
+1e-8), roughly **97% of solve time was inside PETSc `MatMult`**, and every
+configuration ran the identical numerical workload — four SNES solves, 24 KSP
+solves, the same mesh SHA-256.
+
+That is the useful diagnostic: when the solver path is fixed and the runtime still
+moves, you are looking at throughput, not convergence.
+
+| queue / launch | ranks | Stokes solve (s) | peak memory | `MatMult` (Mflop/s) |
+|---|---:|---:|---:|---:|
+| `normal`, default placement | 8 | 4932 | 8.99 GB | 6707 |
+| `normal`, explicit core binding | 8 | 5751 | 9.13 GB | 5738 |
+| `normal`, binding + no-ML attempt | 8 | 5708 | 8.97 GB | 5782 |
+| `normal`, binding + no-ML attempt | 4 | 8964 | 7.31 GB | 3842 |
+| `normal`, binding + no-ML attempt | 12 | **3288** | 10.1 GB | **9995** |
+| `normalsr` (Sapphire Rapids) | 8 | 3992 | 9.25 GB | 8308 |
+
+What this supports:
+
+- **Runtime tracks `MatMult` throughput**, not nonlinear behaviour. Twelve ranks gave
+  the best wall time in this sample (56:02); four ranks cut peak memory to 7.3 GB but
+  took 2:30:48.
+- **Fewer ranks buy memory, not speed.** If a job is near a memory ceiling, dropping
+  ranks is a legitimate trade — but expect the wall time to move roughly inversely.
+- **Newer nodes help.** The `normalsr` 8-rank run beat the `normal` 8-rank run, though
+  it was still slower than 12 ranks on `normal`.
+
+```{warning}
+**Explicit core binding did not help here**, and this campaign cannot say whether it
+ever does. `--map-by core --bind-to core` was *slower* than default placement at 8
+ranks (5751 s vs 4932 s), but the configurations ran concurrently on different nodes,
+and a repeat of the same default-placement case varied by 8.6% on its own (4540 s vs
+4932 s). Node-to-node variation is the same size as the effect. A controlled same-node
+comparison, or a socket-distributed mapping, is needed before drawing any affinity
+conclusion — do not read this table as a recommendation against binding.
+```
+
+**Harmless Open MPI noise.** Messages of the form
+
+```text
+[LOG_CAT_ML] component basesmuma is not available but requested in hierarchy
+[LOG_CAT_ML] ml_discover_hierarchy exited with error
+```
+
+are **nonfatal**. Setting `OMPI_MCA_coll=^ml` neither suppressed them nor changed the
+timings materially.
+
+**Serialise your BLAS.** Every run above set one thread per rank, which is what you
+want when MPI already owns the cores:
+
+```bash
+export OPENBLAS_NUM_THREADS=1
+export OMP_NUM_THREADS=1
+export MKL_NUM_THREADS=1
+export NUMEXPR_NUM_THREADS=1
+```
+
+**Still open.** A recommended Gadi rank placement for bandwidth-bound solves, whether
+UW3's examples should ship explicit mapping/binding options, and a small repeatable
+PETSc timing benchmark to accompany the setup notes — see #635.
+
 ## Related Documentation
 
 - [Developer: Performance Guidelines](../developer/guidelines/performance-optimization.md)

docs/advanced/troubleshooting.md

@@ -50,3 +50,42 @@ everywhere.
 
 See the `Stokes.penalty` property docstring for the full description, and the
 `CONSTRAINED_FREESLIP_MULTIPLIER` design note for the derivation.
+
+## A long output path can segfault parallel HDF5 output (#645)
+
+**Symptom.** A native segmentation fault inside `mesh.write()` or
+`mesh.write_timestep()` on an HPC filesystem, with no Python traceback. The mesh,
+labels, fields, solve, MPI size and output calls are all valid, and the same script
+succeeds when only the output directory is renamed.
+
+**Cause.** The failure follows the *length of the full generated filename*, not the
+validity of the path. UW3 appends its own suffixes to whatever you pass — a mesh write
+becomes `output.mesh.00000.h5`, and a P2 velocity variable becomes
+`output.mesh.U.00000.h5` — so a descriptive case directory can push the complete path
+past what the native PETSc/HDF5/MPI-I/O stack tolerates.
+
+```{warning}
+This happens **well below** the advertised limits. In the reported case every path
+component was under `NAME_MAX` (255), the filesystem allowed `PATH_MAX=4096`, and PETSc
+was configured with `PETSC_MAX_PATH_LEN=4096`. A 286-character filename still crashed.
+```
+
+**Reported thresholds.** On Gadi (PETSc 3.25.4, HDF5 1.12.2p, Open MPI 4.1.7, MPI-enabled
+h5py 3.16.0, GPFS) a staged test failed at a **259-character** filename, and 252
+characters is the shortest length observed to fail on that stack. The exact first unsafe
+length was not established, so treat these as observations on one stack rather than a
+portable threshold. A macOS stack (PETSc 3.25.0, HDF5 1.14.6, Open MPI 5.0.10) did not
+reproduce the original failure, though the full threshold matrix was not repeated there.
+
+**What to do.**
+
+1. Keep the output root and the generated case identifier **compact**, especially on HPC.
+2. Put fixed configuration in HDF5 metadata or a metrics file rather than encoding every
+   parameter in the directory name. A directory named for the two or three parameters
+   that actually vary across a campaign is enough to tell runs apart.
+3. If output segfaults natively, **print the full generated filename first** — including
+   the UW3 suffixes — before investigating the mesh, the labels or the solver.
+
+Shortening the longest filename from 286 to 189 characters made an otherwise unchanged
+eight-rank benchmark pass end to end, and a subsequent 192-rank run at 1/64 resolution
+completed every mesh, Stokes, HDF5, XDMF and postprocessing stage.