Forward the location policy on the empty-rank path (#611)

[lmoresi]

Fixes #611. global_evaluate deadlocked whenever a query set left some rank with no points — 900 s with no progress on a 300-point query at np=4, while passing at np=2.

What it actually was

Native stacks during the hang name both halves:

3 ranks   DMSwarmMigrate -> DMSwarmDataExCreate -> MPI_Comm_dup
1 rank    DMLocatePoints -> DMGetBoundingBox    -> MPI_Allreduce

A mismatched collective — every rank inside MPI, none in locator code. So neither slowness (ruled out: 300 points, 900 s) nor a location failure.

The cell-location policy (mesh._hint_is_authoritative) decides whether the barycentric hint may bypass PETSc's DMLocatePoints, which is collective on the mesh DM communicator. The policy is a mesh capability, and instrumenting confirmed it agrees on every rank (auth=True on all four).

It was discarded in transit. create_structure had no hint array to pass when a rank held zero points, and passed hintAuthoritative = 0 hardcoded beside the NULL:

ierr = DMInterpolationSetUp_UW(self._ipInfo, dm, 0, 1, NULL, 0)

so that rank alone entered the collective. Forwarding the caller's policy is the fix; petsc_tools.c additionally accepts a NULL hint when N == 0, since bypassing is trivially correct with nothing to locate.

Why np=2 hid it

The test biases every point into x > 0.5. At np=4 one rank owns solely x < 0.5 and receives 0 of 300 points — and that is precisely the rank the watchdog roll call singled out. At np=2 the coarser partition leaves both ranks straddling the split, so neither is empty.

Two wrong turns, reverted rather than shipped

Recorded because both look plausible:

• Pre-touching mesh.dm.getBoundingBox() — useless. PETSc does not cache it, so the later call inside the locator reduces again.
• Making the DMInterpolation cache decision unanimous — also useless. Tracing showed all four ranks MISS the cache, so it was never the divergence. This one I would have shipped on reasoning alone; the trace is what stopped it.

A docstring worth correcting separately

_location_capability says it is "deliberately NOT reduced across ranks: the evaluator runs on COMM_SELF [...] petsc_interpolate is reached only by ranks holding points". Both halves are false — DMLocatePoints(dm, ...) uses the mesh DM's communicator, and a trace shows every rank entering petsc_interpolate, including one with zero points. The conclusion (don't reduce the policy) is right, but for the wrong reason — and the wrong reason is what made the hardcoded 0 look harmless. Not changed here to keep the diff to the defect.

Tests

New test_1076_global_evaluate_empty_rank.py: a query biased so some rank is empty, plus an unbiased control that passed before the fix — a failure there means the ordinary path broke rather than the empty-rank path being repaired.

• new regression: green at np=1, 2, 4
• test_0760 (previously hung at np=4): green at np=1, 2, 4
• test_0750, test_0755, test_1064 at np=2: green
• level_1 and tier_a serial: 1074 passed, 0 failed

Underworld development team with AI support from Claude Code

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.