Repository navigation
cell_size() diverges on a rank that owns no cells (np=8, tiny mesh) — and that mesh already hangs in construction on development #698
Description
Found while adversarially reviewing #697; not caused by it, filed so it is not lost.
What happens
A 14-cell UnstructuredSimplexBox at np=8 gives cells per rank [2,2,2,2,0,2,2,2]. Calling mesh.cell_size() then diverges — per-rank stacks from the #678 supervisor:
ranks 0,1,2,3,5,6,7 inside _assemble_cell_size (discretisation_mesh.py:3296)
rank 4 already past it, in the next statement
Rank 4 is the starved one. _assemble_cell_size short-circuits on radii.size == 0 and returns before touching var.data, which its peers do evaluate.
Why this is not #697
_assemble_cell_sizeis documented as deliberately rank-local, and its empty-partition early return dates from Nitsche BC: scale penalty by a local per-cell mesh size #275 (a7f0b11f) — long before Make mesh.cell_size() local and partition independent #692 or Take the cell radius from PETSc, and fix the radius accessors with it (#694) #697.- The same mesh at np=8 hangs on
developmentearlier still, during mesh construction, beforecell_size()is reached. So the configuration is pathological independently, and a clean before/after attribution is not available at that size.
At realistic mesh sizes np=8 is green: both partition-independence guards pass at np=2/4/8, and test_1069 passes at np=2 and np=8 on the default local_h=True path.
What needs deciding
Two separate things, and they may have different owners:
- Whether a rank owning zero cells is a supported state for
cell_size(), or whether the early return needs to be paired with whatever its peers do. The rank-local claim in the docstring is the thing to check, not assume. - Why a 14-cell mesh at np=8 hangs during construction at all. That is the same family as Fix cell-family detection on empty MPI ranks #691 (
isSimplexrank-local on an empty rank) and Passive swarm save/advection hangs or crashes on empty MPI ranks #679/fix: swarm empty-rank save deadlock (MPIO dtype) and estimate_dt reshape crash #680 (empty-rank swarm save), all found in the last week.
Reproduce with the supervisor so a hang reports instead of sitting:
pixi run -e amr-dev python scripts/mpi_supervisor.py --silence 60 -- \
mpirun -n 8 python -u probe.pyUnderworld development team with AI support from Claude Code
Activity
Allocate the cell-size variable before testing for an empty partition (…
Remove the early return from _assemble_cell_size entirely: BOTH steps…
lmoresi commentedon Sep 7, 2026
Fixed and demonstrated in #697 (
97cade70).There were two collectives behind that early return, not one. My first attempt hoisted only
var.dataand the job still hung, one step later, with the populated ranks inpack_raw_data_to_petsc— assigning intovar.datafires the array's write-back callback, which is collective as well.The early return is gone entirely. A starved rank walks the same path and writes a zero-length slice:
The reproducer is a region submesh, which is the realistic case and the one the maintainer pointed at — a submesh keeps only the cells carrying its label, so a partition whose share of the parent lies outside that region owns none. A box split at z=0.85 gives an outer slab with cells per rank
[12, 11, 0, 19, 0, 0, 0, 0]at np=8: five starved ranks out of eight. It hung before and returns now.Correcting the original report: I used an over-decomposed 14-cell full mesh. That was a poor choice — degenerate, and it hits an unrelated intermittent hang in mesh construction that masked the real defect. A full mesh having a zero-cell rank is an artefact of over-decomposition; a submesh having one is normal.
Guarded by
tests/parallel/test_0779_submesh_cell_size_starved_rank.py, which asserts every rank returns (a regression hangs rather than fails) and separately that the field is still filled where cells exist, so the fix cannot be satisfied by never filling it.Underworld development team with AI support from Claude Code