diff --git a/.claude/skills/faults-in-convection/SKILL.md b/.claude/skills/faults-in-convection/SKILL.md new file mode 120000 index 000000000..a5f701634 --- /dev/null +++ b/.claude/skills/faults-in-convection/SKILL.md @@ -0,0 +1 @@ +../../../docs/developer/guides/faults-in-convection.md \ No newline at end of file diff --git a/docs/developer/design/FAULT_MULTIGRID_HIERARCHIES.md b/docs/developer/design/FAULT_MULTIGRID_HIERARCHIES.md index dd36ac4c6..21b68ea46 100644 --- a/docs/developer/design/FAULT_MULTIGRID_HIERARCHIES.md +++ b/docs/developer/design/FAULT_MULTIGRID_HIERARCHIES.md @@ -154,6 +154,10 @@ A, the existing path, had three defects at np > 1: cell-located transfer, which asks the other ranks for the nodes it cannot locate locally, and unreached unknowns are dropped rather than repaired: A converges at np = 2 (15 iterations against 13 serial), C in 6 at np = 2 and 7 at np = 4 (k = 2). + (Superseded in October 2026: the add-on is now partitioned once before the cut, + so the split levels are co-partitioned with each other and nothing is asked of + other ranks on those pairs; see + [parallel-addon-distribution-2026-10](parallel-addon-distribution-2026-10.md).) The pairing rides every redistribution as a pair of DM labels. The migration star forest PETSc returns for an initial distribution does not index the stratified point @@ -173,6 +177,27 @@ every problem, for no gain. B2 is worse than A, and badly so for the viscous fault. A smeared band on a coarse level is a continuous shear that the split fine operator rejects, so the coarse space has to contain the discontinuity itself. +## C on the ladder, in a convecting lid (2026-10-14) + +A fault placed as a ladder (`place_fault_ribbon_2d`, `cut=False` spines) on a copy of the +base's finest level has no add-on levels, and its spines are not replayed onto the +levels, so the tail it can be given is A. In a yielding, convecting box (4 x 1, base +cellSize 0.125 with two refinements, rung spacing S = 0.004, frictionless split spine +with a weak bridge to the surface) the welded tail capped the velocity sub-solve at 200 +once the lid mobilised. C was built by hand: the same ladder at 4S on the base's middle +level and at 2S on its finest, each split along its own rungs, as the mid levels, with +the coarsest base level welded. One solve from rest on a mobilising state, 4 ranks: + +| Tail | Levels (cells) | Newton increments | Velocity its, total | Newton phase, per increment | Caps | +|---|---|---|---|---|---| +| A, welded | 642, 2568, 10272, 11604 | 39 | 1849 | 71-200 | 2 | +| C, every ladder on the finest base level | 642, 2568, 10272, 10573, 10966, 11612 | 42 | 636 | 26-29 | 0 | +| C, lean (ladders on the coarser base levels) | 642, 2924, 10950, 11612 | 42 | 695 | 29-38 | 0 | + +The lean tail keeps C's counts at A's cost per cycle. A constant band width on every level +and a band two of each level's own rungs wide gave the same counts. The recipe and the +driver rules that go with it are in `guides/faults-in-convection.md`. + ## State (library, 2026-10-13) In the library: the add-on levels of one `adapt` call, each split by `add_fault`; the @@ -181,5 +206,7 @@ cell-located transfer (serial and parallel) and the per-level frames in layer with every level split, extruded in parallel; 3-D sheet faults on add-on meshes. Not yet: fault networks through the levels (a network is replayed on every level, but a one-cell junction ligament closes at the coarser spacing and the replay then stops -there, leaving that level and those beneath welded), faults near a boundary, -co-locating the levels with the finest as an optimisation. +there, leaving that level and those beneath welded), faults near a boundary, and the +ribbon replay that +would give a placed ladder its split levels automatically (a driver builds them by hand: +`guides/faults-in-convection.md`). diff --git a/docs/developer/design/fault-parallel-placement-2026-09.md b/docs/developer/design/fault-parallel-placement-2026-09.md index e0ab9a10d..1e55eea31 100644 --- a/docs/developer/design/fault-parallel-placement-2026-09.md +++ b/docs/developer/design/fault-parallel-placement-2026-09.md @@ -8,6 +8,17 @@ subsystem description is `../subsystems/conforming-surfaces-and-fault-zones.md` and the multigrid side is `fault-patch-multigrid-2026-08.md`. Issues: #670 (the gather), #671 (the tail), PRs #669 and #672. +```{note} +For the cut, the 3-D sheet placement and the split reached through the Mesh API, the +per-region gather and the per-split moves described here are superseded by one +partition of the add-on hierarchy before any surgery +([parallel-addon-distribution-2026-10](parallel-addon-distribution-2026-10.md)), and +so is the band routes' gather (`FaultNetwork.build`, `place_fault_ribbon_2d` in +gather mode). The gather remains the mechanism of a direct `place_sheet` or +`place_thin_volume` call; the seam-ligament and seam-conform modes below are +unchanged. +``` + ## The frame: adapt-on-top The base mesh is the persistent, distributed object. It holds the bulk diff --git a/docs/developer/design/parallel-addon-distribution-2026-10.md b/docs/developer/design/parallel-addon-distribution-2026-10.md new file mode 100644 index 000000000..d08b75f93 --- /dev/null +++ b/docs/developer/design/parallel-addon-distribution-2026-10.md @@ -0,0 +1,228 @@ +# Distributing the add-on once (October 2026) + +Design note, 2026-10-17. The decision on how the add-on — the finest mesh built on a +static base hierarchy by `adapt`, a cut, a placement or a split, together with its +add-on levels — is partitioned in parallel, and the measurements behind it. It +follows the September ruling in +[fault-parallel-placement-2026-09](fault-parallel-placement-2026-09.md) (the base is +never rebalanced; load fixes go on the child; every transfer to the child must be +partition-agnostic) and the hierarchy of +[FAULT_MULTIGRID_HIERARCHIES](FAULT_MULTIGRID_HIERARCHIES.md). Implementation: +`utilities/addon_partition.py`; tests `tests/parallel/test_1096_*`. The probes +named below are kept with the campaign, outside this repository. + +## The problem + +The base hierarchy is partitioned once and refined in place, so its levels are +co-partitioned, and `adapt` (the in-place `uwnvb` transform) keeps the add-on levels +co-partitioned with it. Every divergence comes from fault surgery, and each surgery +moves cells on its own terms: + +- `split_fault` / `add_fault` move each fault's cell star to the rank already holding + most of it (`_redistribute_fault_interior`), on the finest and again on each + replayed add-on level, each level to *its own* majority rank; a second pass + (`colocate_split_level`) then moves every level's star to the finest's owner. +- `place_sheet` / `place_thin_volume` gather the surgery region to one rank + (`_gather_region(s)`); the 3-D sheet's mask does not cover the cavity's growth + rounds, so on a partitioned adapt child the carve touched a shared point and the + placement refused where the same placement passes serially (`TODO(BUG)` at the mask). +- `adopt_hierarchy` moves *copies of the base's static levels* to follow a placed + mesh (`_colocate_level`). +- The 2-D `add_fault` cut constructs its child without the parent's tail, so solvers + on it fell to GAMG silently (`TODO(BUG)` at the constructor). + +The transfers then pair levels that are not co-partitioned by point location: the +cell-located builder all-gathers the fine nodes it cannot place locally as queries, +and the node-cloud builders needed two fixes on 2026-10-17 (f21f3d85) to stop +producing wrong rows on such a pair. Pair diagnostics, checkpoints and the placement +each carry their own partition handling. + +## Measured (hyperion, f21f3d85, under a 16-rank study) + +Refined patch box, split (`BoxInternalPatch(refinement=2)` then `split_fault`, +19072 cells, P2 velocity, tolerance 1e-4; `probe_scaling.py`): + +| np | velocity iterations | finest P2 nodes per rank (max / mean) | +|---|---|---| +| 1 | 4 | 28274 | +| 2 | 5 | 17118 / 12501 (1.16) | +| 4 | 5 | 9695 / 7789 / 7599 / 5529 (1.27) | +| 8 | 6 | 7500 / 3555 / 4113 / 3461 / 3399 / 3837 / 3762 / 2147 (2.12) | + +The iteration count is now near serial; the owner rank carries twice the mean at +eight ranks, and 3026 of its 7500 nodes lie outside its local coarse hull, so the +finest transfer on that rank is built from remote answers. + +Adapt-on-top fault, 2-D, two add-on levels, np = 4 +(`parallel_load.py 2`): finest cells per rank +240 / 292 / 243 / 695 (max / mean 1.89); the base-to-first-add-on transfer misses +265 of 2528 owned nodes (10 %) and asks the other ranks for them; the add-on pair +misses 14 of 2992 after the levels are co-located. Six velocity iterations. + +A per-cell field rides a shell redistribution exactly through +`DMPlex.distributeField` on the migration star forest `distribute()` returns +(`probe_migration_sf.py`, np = 2: zero error). The earlier reading that the migration +forest "maps points to the wrong depth" came from reading remote point numbers +against the local chart; the forest is a point map, and `distributeField` is the API +for riding it. + +## Measured after (same fixtures, the partition in place) + +Refined patch box, split (`probe_load_3d.py`): + +| np | finest P2 nodes per rank (max / mean) | before | velocity iterations | wall | +|---|---|---|---|---| +| 2 | 13683 / 16090 (1.08) | 1.16 | 5 | 65 s (72) | +| 4 | 8170 / 8470 / 7766 / 7697 (1.06) | 1.27 | 6 | 34 s (38) | +| 8 | 5899 / 4062 / 4007 / 4165 / 3772 / 3642 / 3949 / 3907 (1.41) | 2.12 | 6 | 25 s (30) | + +At eight ranks the owner holds exactly the fault region (3524 of 19072 cells) and +nothing else, after 13 balance sweeps; the region is the indivisible unit. The +2-D adapt-on-top fault, +np 2: finest cells per rank 742 / 618 (max / mean 1.09, from 1.38); every add-on pair +co-partitioned, every recorded parent contains its cell; the base-to-add-on pair +fetched, no query. At np 4 and 8 on that 1360-cell fixture the fault region itself +(544 cells with its growth layers, lifted to units) exceeds the mean load, so the +owner keeps it whole and the balance stops there: a region is the indivisible unit, +as the September note's granularity remark says, and the test bounds the load by +the region's weight. The 3-D sheet placement on a partitioned adapt child, refused +before ("the gather mask under-reached" at np 2 on the disc fixture of test_0867), +places with nothing gathered and splits with nothing moved (test_1097). + +## The decision + +**One assignment for the whole add-on, computed on the finest level before any +surgery, applied to every add-on level through the parent-cell relation, once.** The +base levels never move. Concretely: + +1. **Units.** The unit of assignment is a cell of the coarsest add-on level together + with all its descendants on the finer add-on levels (the finest mesh itself when + there is no add-on level below it). Every add-on cell has one parent on the level + below, found by centroid containment while the levels are co-partitioned (exact + for a bisection hierarchy). Every cell of a unit goes to the same rank, so every + add-on pair stays co-partitioned exactly, and the base-to-add-on pair is + co-partitioned except where a unit moved. +2. **Fault regions.** The cells the surgery will touch — the star of the vertices + within reach of the fault geometry, plus growth layers so every point the carve + or the split touches is unshared (one for a cut, two for a 3-D carve) — are + marked on every add-on level, lifted to units and glued: a region is one + indivisible block, sent to the rank already holding most of it, or left where it + is when it is already interior. Every interior surface the levels already carry + is glued the same way, so a second partition (a second sheet, a fault after a + material interface) keeps the first whole. Regions whose shells touch merge (the + existing `_gather_regions` rule). The split's pairs and the fault cell fans are + therefore rank-local by construction, and never because a later pass moved them. +3. **Balance.** By diffusion over the seams: in each sweep any rank heavier than a + face-neighbouring rank by more than the tolerance offers it units that belong to + no region, up to half the difference, the cells farthest from the faults first, + and a receiver's intake is capped by the largest half-gap offered to it. Sweeps + stop when every rank is within the tolerance of the mean or nothing can move. The + load fix is on the child; the base's partition is read, never written. +4. **Provenance rides.** Each level carries, per cell, its parent cell on the level + below and its own pre-move identity, as cell-wise fields moved by + `distributeField`; two star forests then give every cell its parent's current + address. The add-on pairs are co-partitioned and keep the existing cell-located + transfer, which now finds every node locally (its query exchange still runs as a + collective, with nothing to send). The base-to-add-on pair is built by + `provenance_transfer_parallel`: the coarse cell's Lagrange basis evaluated at the + fine node (the dual-basis evaluation of `fault_mg`), with the parent's node + columns and coordinates fetched through one star forest. Its contract is a + collective refusal if any fine node lies in no cell of the level the provenance + is recorded on. +5. **Surgery afterwards.** Cut, placement and split run on the partitioned levels + and move nothing; the split's seam test becomes a collective refusal. A level + created by surgery is related to the level it re-represents by locating its + cell centroids there (a local test, since the region is rank-interior), and that + level's record gives the parent on the base. + +What this retires: `colocate_split_level` and the per-level fault-interior moves of +the cut, placement and split pipelines. Still in place, outside this change: +`_colocate_level` in `custom_mg.adopt_hierarchy`, which the placed-band route +(`FaultNetwork.build` with the thin volume, `place_surface`'s ladder) uses because +`place_thin_volume` still gathers its own regions; the per-mesh fault-interior move +of `extrude_sheet`; and the node-cloud builders' cross-partition path, reachable for +a pair with an empty column on a hierarchy registered by hand. + +## The band routes and the direct placement (18 October) + +`FaultNetwork.build` (the 3-D band) and `place_fault_ribbon_2d` in gather mode now +partition before placing, with `mark_band`: a superset of the thin-volume +placement's own mask, at the placement's reach plus one band width from the band's +path (the skin lies within a mitred half width of it), with two growth layers. The +placement then moves nothing, `adopt_hierarchy` keeps the base's own levels under +the partitioned host instead of co-locating copies of them, and the pair the +partition separated is read from the recorded provenance (`test_1099`). The ribbon's +ligament and conform modes mesh across the seams in place and are unchanged. + +A direct `place_sheet` call, without the Mesh API's partition, now gathers with two +growth layers, the rule the partition uses for sheets. With one, the carve's growth +rounds reached a shared point on a partitioned adapt child and the placement +refused (`test_1097::test_direct_place_sheet_on_a_partitioned_adapt_child`, shown +failing before). + +## Cost: the per-point loops compiled (18 October) + +The partition, the region growth shared with the placement gather, the marking +rules, the redistribution's per-cell fields, and the mesh's own point-location +index and per-cell node table walked the mesh one petsc4py call per point. +`utilities/_plex_topology.pyx` builds the tables they need in one C loop per +stratum: closures by depth, supports, a per-cell section and its offsets, and each +cell's dof rows. The loops above them are now array operations. The node-cloud +transfer builders set one matrix row per Python call; they now assemble once +(`custom_mg._assemble_node_transfer`). Every converted function was checked against +its previous version, entry for entry, serially and on four ranks +(`check_topology_equivalence.py`, +`check_transfer_assembly.py`); `tests/test_1098` pins the tables to DMPlex's own +point-by-point answers. + +| partition_addon, 4 ranks | cells | before | after | Python calls | +|---|---|---|---|---| +| 3-D adapt child, one sheet | 87525 | 6.87 s | 1.35 s | 1.46 M to 48 k | +| 2-D adapt child, k = 2 | 94910 | 2.94 s | 0.64 s | | + +What remains of the 3-D time is mostly PETSc building interpolation matrices for the +mesh's coordinate tables in the Mesh constructor, in C. A fault split of 19,072 +cells takes 3.6 s, of which the splitter's own per-point Python is about 0.1 s and +the four Mesh constructions most of the rest; the splitter's whole-mesh rebuild +(one cone per cell, one callback per point) grows linearly and is the next loop to +compile if splits at a million cells per rank are needed. + +## What was weighed and rejected + +- **Partition the base around the faults at construction.** Perfect co-partition and + balance, nothing ever moves. Rejected: it is a fix on the base, a moving fault + invalidates it every step, and the base is built before the model knows its faults. +- **Keep the base's partition and split through the seams** (the 2-D `conform` mode + one dimension up). Nothing moves and balance is the partitioner's. Rejected for + the split: the rank-interior fan is what the contact solve's pair block and the + fault-normal accumulation are built on, the 3-D cut through a seam needs the global + patch and a geometric side rule at every shared rim vertex, and the along-strike + seam — the generic case on a base graded toward the fault — is refused today. + The `conform` mode stays as it is for the 2-D band. +- **Leave the gathers and keep repairing the transfers.** The iteration counts are + already near serial after f21f3d85, but every consumer still handles the + partition separately, the owner rank carries twice the mean, and the next + surgery (the 3-D placement on an adapt child) refused for a reason that the one + partition step settles. + +## Stages and tests + +Every stage ships its regression test first, shown failing (Charter §8), under +`tests/parallel/` at 2, 4 and 8 ranks, and the serial reduction. + +1. The 2-D `add_fault` cut child carries its parent's tail (the `TODO(BUG)`): a + refined box, `add_fault`, the velocity preconditioner is the custom FMG and not + GAMG, serial and np = 2. +2. `addon_partition`: after `add_fault` / `split_fault` on an adapt child and on the + refined patch box, every add-on pair reports zero misses, every fault star is + rank-interior with no pair dropped, the finest cells per rank are within the + balance tolerance of the mean, and the velocity iteration count is within a few + of serial. The finest transfer reproduces the coordinate field (test_1095 stands). +3. The provenance transfer replaces location for every pair of such a hierarchy: + the base boundary's moved units are fetched, the transfer stats show no query. +4. The 3-D sheet placement on a partitioned adapt child (the refused fixture) places + and splits at np = 2, 4, 8 with the same cell and pair counts as serial up to the + fill's noise, and the gather reports nothing moved. + +Out of scope here, recorded for later: `extrude_sheet` still builds on one rank; +checkpoint reload of a partitioned add-on; the 3-D split through a seam. diff --git a/docs/developer/guides/faults-in-convection.md b/docs/developer/guides/faults-in-convection.md new file mode 100644 index 000000000..00d86dc0e --- /dev/null +++ b/docs/developer/guides/faults-in-convection.md @@ -0,0 +1,123 @@ +--- +name: faults-in-convection +description: Recipe for a split-node fault in a TIME-DEPENDENT convection model (yielding lid, fault placed as the structured ladder by place_fault_ribbon_2d on a copy of the base's finest level, Stokes on the faulted mesh, temperature on the unfaulted base). Covers the multigrid hierarchy that keeps the faulted velocity solve bounded (the lean split-ladder chain, "C on the ladder"), the warm-start rules of the rotated nonlinear loop, rebuilding the faulted mesh for a moving trench, and the traps measured on the 4 x 1 box. Reach for THIS before writing or tuning a fault-in-convection driver; for the fault mechanics on an adapted mesh see `adapt-on-top-faults`, for the yield-law solver see `nonlinear-solver`. +families: [Stokes, AdvDiffusion] +kind: recipe +--- + +# Faults in convection + +A split-node fault in a convecting, yielding lid. The pattern measured on a 4 x 1 box +(2026-10, branch `feature/box-faults`), with the fault-multigrid library of +`feature/fault-split-multigrid`: + +- the base mesh carries the temperature and its transport and is never cut; +- the faulted mesh is a copy of the base's finest level with the fault placed as the + structured ladder, the spine split and in frictionless contact from three rung + spacings below the surface, the last rungs to the surface a weak isotropic bridge; +- Stokes is solved on the faulted mesh, and the velocity is evaluated back onto the + base for the transport step. + +## The multigrid hierarchy: split the coarse levels + +`set_custom_fmg(stokes, base_levels, field_id=0)` with the base's own levels gives the +WELDED hierarchy (A in `design/FAULT_MULTIGRID_HIERARCHIES.md`): only the finest level +carries the fault. Its coarse spaces cannot represent the slip, the weak bridge or the +ladder cells, and the velocity sub-solve caps at 200 iterations once Newton starts on a +mobilising lid. A placed ladder is not replayed onto the levels by `add_fault`, so the +split levels have to be built by hand as the mid levels: the same ladder at twice the +rung spacing placed on the finest base level (beside the faulted mesh's own ladder), +and at four times the rung spacing on the base level one step below, each split along +its own rungs. The coarser base levels stay welded. + +```python +from underworld3.utilities.custom_mg import set_custom_fmg +from underworld3.utilities.place_surface import place_fault_ribbon_2d + +base_levels = base._coarse_level_meshes()[:-1] + [base] # coarsest .. finest + +def placed_ladder(host, coarsening): + # the spine and trace sampled at coarsening x the rung spacing S; the finest + # sample count is a multiple of 2**chain, so every coarser sampling is a subset + spine, trace = fault_trace(coarsening) + width = 2 * S * coarsening # two of the level's own rungs + mesh, info = place_fault_ribbon_2d(host, [("Fault", trace)], width, + spines=[("Fault", spine)], margin_rings=1, + clearance=0.7, split=True, mesher="ladder") + return mesh, info + +chain = 2 +mids = [placed_ladder(base_levels[len(base_levels) - k], 2 ** k)[0] + for k in range(chain, 0, -1)] +faulted, info = placed_ladder(base, 1) +stokes = uw.systems.Stokes(faulted, ...) +set_custom_fmg(stokes, base_levels[:len(base_levels) - chain] + mids, field_id=0) +``` + +Measured on one hard state (a mobilising lid, slip ~270, one solve from rest, 4 ranks; +velocity iterations as the rotated loop reports them per Newton increment): + +| Hierarchy | Levels (cells) | Newton increments | Velocity its, total | Velocity its per increment, Newton phase | Caps (200) | +|---|---|---|---|---|---| +| A, welded | 642, 2568, 10272, 11604 | 39 | 1849 | 71-200 | 2 | +| ladder chain all on the finest | 642, 2568, 10272, 10573, 10966, 11612 | 42 | 636 | 26-29 | 0 | +| **lean chain** (above) | 642, 2924, 10950, 11612 | 42 | 695 | 29-38 | 0 | + +A cycle of the chain placed entirely on the finest level visits three levels of ~11k +cells, about twice the work of a lean or welded cycle, so the lean chain is the cheapest +per solve. In the time-stepping run every velocity sub-solve capped under A once the lid +mobilised; under the lean chain none did, at 4-5 Newton increments per step and the same +slip. A band of constant width (two finest rungs) on every level gave the same counts as +the widened bands. The band patch smoother (`fac_zone=band`) did not help the welded +hierarchy: 114-150 velocity iterations per increment against 91-162 without. + +## The rotated nonlinear loop + +`add_fault_bc` sends the solve through the rotated strong-constraint loop +(`rotated_bc.solve_rotated_freeslip`). Walls with plain essential BCs compose with it; +block-constraint multipliers do not. A closed box needs +`petsc_use_pressure_nullspace = True`. + +- `solve(picard=N)` with the default (consistent Newton) tangent takes at least N + Picard iterations on a blended Jacobian, then Newton; + `stokes.consistent_jacobian = "continuation"` switches from Picard to Newton by + itself. +- The first solve on a faulted mesh starts from rest. A warm start from a velocity + computed without the fault stalls the Picard line search (measured 8e-2 of the + reference residual after 2 iterations); from rest the solve converges in 18-34 Picard + and 5-8 Newton iterations on the states measured. +- The loop switches from Picard to Newton when the residual has fallen by + `newton_switch_rtol` relative to the same reference as its convergence test (the + larger of this solve's starting residual and the rest-state residual), so a warm + start near convergence reaches Newton without first shrinking its own residual a + hundredfold under Picard. (Before 2026-10-23 the switch was measured from the warm + start's own residual, warm steps stayed in Picard to the end, and the advice was + `newton_switch_rtol = 1.0`; that is no longer needed.) + +## A moving trench + +A fault fixed in space loses its slab once the convergence zone migrates; the slab +stays on the fault when the trench moves with the overriding plate (the median surface +velocity over a window beyond the fault, not the sliver next to it). The faulted mesh is +then rebuilt as the trench moves: + +- carry the solution across a rebuild by evaluating the old velocity and pressure at + each new node shifted back by the trench displacement, so the old slip lands on the + new fault; +- start the rebuilt solve with Picard to a residual reduction of 1e-2 of its own start. + Newton from the transferred field stalls (measured 40 iterations at 0.1 of the + reference residual), and so does Newton after one order of Picard reduction; +- rebuild every few rungs (four, half a base cell here), not every rung: a rebuild step + costs about twice an ordinary one. + +## Cost and diagnostics + +- Through an overturn the nonlinear solve, not the multigrid, sets the cost: 15-25 + Newton increments per step on the moving-trench run while the velocity sub-solve + stayed bounded. +- Compare solver cost by iteration counts (Newton increments, velocity cycles), not + seconds: on a laptop the wall time moves with power state and load. +- A refined mesh read back from its checkpoint has its cells joined to the wrong + vertices (open roadmap item). Renderers rebuild the mesh from the run's parameters + and remap the fields by coordinate; draw P1 fields on the mesh's own cells, not on a + Delaunay of their DOFs. diff --git a/docs/developer/index.md b/docs/developer/index.md index 162a2dfd5..d260bb4b5 100644 --- a/docs/developer/index.md +++ b/docs/developer/index.md @@ -40,6 +40,7 @@ same topic are reference or historical material subordinate to the governing doc | Thin sheet (plane stress, sphere, faults) | [subsystems/thin-sheet.md](subsystems/thin-sheet.md) | | Fault multigrid (split-level hierarchy, parallel pairing) | [subsystems/fault-split-multigrid.md](subsystems/fault-split-multigrid.md) | | Fault networks in parallel (placement, split, tail) | [design/fault-parallel-placement-2026-09.md](design/fault-parallel-placement-2026-09.md) (subsystem reference: [conforming-surfaces-and-fault-zones.md](subsystems/conforming-surfaces-and-fault-zones.md)) | +| The add-on's partition in parallel (one partition for the add-on hierarchy) | [design/parallel-addon-distribution-2026-10.md](design/parallel-addon-distribution-2026-10.md) | | Analytic & benchmark solutions | [subsystems/analytic-solutions.md](subsystems/analytic-solutions.md) | | Units | [design/UNITS_SIMPLIFIED_DESIGN_2025-11.md](design/UNITS_SIMPLIFIED_DESIGN_2025-11.md) | | Testing tiers | [TESTING-RELIABILITY-SYSTEM.md](TESTING-RELIABILITY-SYSTEM.md) | @@ -65,6 +66,7 @@ pages, never copies. A change to a family is reviewed against its guides. | [Plasticity solvers](guides/plasticity-solvers.md) | viscoplastic and VEP Stokes | | [Adaptive meshing](guides/adaptive-meshing.md) | moving-mesh convection | | [Adapt-on-top faults](guides/adapt-on-top-faults.md) | fault models on adapted meshes | +| [Faults in convection](guides/faults-in-convection.md) | split-node faults in time-dependent Stokes | | [Free-surface convection](guides/free-surface-convection.md) | free-surface Stokes | | [Visualisation](guides/uw-visualisation.md), [CeTZ figures](guides/cetz-figures.md) | figures | | [Line diagrams](guides/uw-line-diagrams.md) | drawn figures | diff --git a/docs/developer/subsystems/conforming-surfaces-and-fault-zones.md b/docs/developer/subsystems/conforming-surfaces-and-fault-zones.md index 8bc12808c..b4f1c7299 100644 --- a/docs/developer/subsystems/conforming-surfaces-and-fault-zones.md +++ b/docs/developer/subsystems/conforming-surfaces-and-fault-zones.md @@ -282,6 +282,15 @@ a labelled edge, so a placed surface comes through any later repair with the facets it was placed with. The fill itself rarely needs repair (see above), but the composition remains sound. +```{note} +Since October 2026 the Mesh API (`add_conforming_sheet`, `add_fault`) and the band +routes (`FaultNetwork.build`, `place_fault_ribbon_2d` in gather mode) partition the +add-on for the surface before placing it, so the gather below finds the region +rank-interior and moves nothing (`info["n_gathered"]` is then zero). The gather is +what a direct call to `place_sheet` or `place_thin_volume` still does. See +[parallel-addon-distribution-2026-10](../design/parallel-addon-distribution-2026-10.md). +``` + The 3-D forms are **parallel** by gather-first: the surface's region is redistributed onto one rank, the serial carve-and-fill runs there, and every rank rebuilds its chart collectively (uninterpolate + `DMPlexInterpolate`). diff --git a/docs/developer/subsystems/fault-split-multigrid.md b/docs/developer/subsystems/fault-split-multigrid.md index 6a0207c38..698a84f0b 100644 --- a/docs/developer/subsystems/fault-split-multigrid.md +++ b/docs/developer/subsystems/fault-split-multigrid.md @@ -101,22 +101,37 @@ zero). Two further rules came out of the same case: ## Parallel -Every split level is redistributed so its fault star is rank-interior (the split's own -rule), so the pair blocks are rank-local on every level. The transfer locates each -owned fine node in the *local* coarse mesh (owned plus ghost cells); the nodes whose -coarse cell lives elsewhere are all-gathered as query points, answered by the rank -holding the cell with global column indices, and the asker keeps the best answer. The -levels need not be co-partitioned. +The add-on is partitioned **once, before any surgery**, as one hierarchy +(`utilities/addon_partition.partition_addon`, design note +[parallel-addon-distribution-2026-10](../design/parallel-addon-distribution-2026-10.md)). +`add_fault`, `add_conforming_sheet`, `split_fault` and `split_faults` call it in +parallel: the fault regions — each fault's cell star plus growth layers, marked on +every add-on level because a coarser level's star reaches further — are glued and +sent to the rank already holding most of them, the load is balanced by diffusion +over the seams (free cells move from a heavier rank to a lighter neighbour, the +cells farthest from the faults first), and the one assignment is applied to every +add-on level through the parent-cell relation. The background levels never move. +The cut, the placement and the split then run on rank-interior regions and move +nothing; the replay of the faults on the add-on levels (`split_tail_levels`) arrives +on levels already partitioned with the finest. + +Every add-on pair is therefore co-partitioned exactly and the cell-located transfer +finds every node in its local coarse cells (`level_transfer_stats` records `misses` +zero; the query exchange still runs, with nothing to send). The one pair the +partition separates — the base's finest static level against the first add-on +level — carries the exact relation: each level records, per cell, where its parent +on the level below lives (provenance riding the move through `distributeField`, the +new addresses resolved through two star forests), and `provenance_transfer_parallel` +fetches each fine cell's parent through one star forest (its record carries +`fetched_remote`). Nothing on that pair is located across partitions; a fine node in +no recorded cell is a collective refusal. A coincident pairing rides a redistribution as a pair of DM labels -(`fault_split.stamp_point_pairs` / `collect_point_pairs`): one value per pair on its -Minus and Plus points. The migration star forest PETSc returns for an *initial* -distribution does not index the stratified numbering (measured: three quarters of the -points mapped to the wrong depth), so the forest is not used. `extrude_sheet` builds the -layer on one rank, distributes it with the labels on, redistributes the fault stars -rank-interior and reads the pairing back; a balanced partition readily puts the whole -Minus side on one rank and the whole Plus side on another, since the slit disconnects -them. +(`fault_split.stamp_point_pairs` / `collect_point_pairs`), and a per-cell field as +a cell-wise section through `distributeField` on the migration star forest +(`redistribute_by_assignment(fields=...)`). `extrude_sheet` builds the layer on +one rank and distributes it with the labels on; its fault stars are then made +rank-interior by the older per-fault move, which remains for that path. ## Measured (box in simple shear, straight fault, 2026-10-13) @@ -181,9 +196,4 @@ welded tails converge with the serial slip; the thick twin extruded with its pai welded. A network whose junction ligament is one fine cell wide closes at the next coarser spacing (the two labels meet), so such a network gets split levels only where the ligament survives — on `test_0866`'s kissing Y with one add-on level, none. -- In parallel each split level's fault star is moved onto the rank holding the - finest level's star (`fault_mg.colocate_split_level`), so a transfer between - levels locates most fault-band nodes locally (without it, 48 % of the finest - nodes were located remotely at np = 2, 58 % at np = 8); the remote queries - that remain are the band's rim. Not measured at scale. - The one-rank build of `extrude_sheet` bounds the layer size by one rank's memory. diff --git a/docs/developer/subsystems/model-orchestration.md b/docs/developer/subsystems/model-orchestration.md index 44a20368a..71fc161ef 100644 --- a/docs/developer/subsystems/model-orchestration.md +++ b/docs/developer/subsystems/model-orchestration.md @@ -12,7 +12,7 @@ The Model object serves as a central orchestrator for Underworld3 simulations, p The Model object addresses three fundamental needs in geodynamic simulation workflows: -1. **Component Lifecycle Management**: Meshes, swarms, and variables form complex dependency networks. The Model acts as a central authority that owns these relationships, breaking circular references that would otherwise complicate memory management and serialization. +1. **Component Lifecycle Management**: Meshes, swarms, and variables form complex dependency networks. The Model indexes them, weakly, so they can be found, listed and serialised; it does not own them (see [Weak References and Ownership](#weak-references-and-ownership)). 2. **Mesh Swapping Support**: Simulations often require adaptive remeshing or mesh refinement. By mediating the relationship between swarms and meshes, the Model enables mesh replacement with automatic notification to all dependent objects. @@ -55,11 +55,12 @@ class Model: self.name = name or f"Model_{id(self)}" self.state = ModelState.CONFIGURED - # Component tracking - self._mesh = None + # Component tracking: an index, never an owner (all weak) + self._meshes = weakref.WeakValueDictionary() # id -> mesh + self._primary_mesh_ref = None # weakref to the primary mesh self._swarms = weakref.WeakValueDictionary() - self._variables = {} # name -> variable mapping - self._solvers = {} # name -> solver mapping + self._variables = weakref.WeakValueDictionary() # name -> variable + self._solvers = weakref.WeakValueDictionary() # name -> solver # Simple configuration self.materials = {} # material_name -> {property: value/expression} @@ -69,9 +70,42 @@ class Model: self._version = 0 # Incremented when model structure changes ``` -### Weak References - -Swarms use `WeakValueDictionary` to prevent circular references. When a swarm is garbage collected, it automatically disappears from the model's registry without requiring explicit cleanup. +### Weak References and Ownership + +The model indexes meshes, variables, swarms, solvers and the step transcript's +part objects by weak reference: it finds them, it does not keep them. A +strong registry kept every mesh a process ever built, with its DM hierarchy, +variables and solvers (a faulted-mesh rebuild loop grew by a full mesh per +rebuild until it ran out of memory). Ownership is: + +- a mesh owns its variables (`mesh.vars`), so a variable created and not kept + lives as long as its mesh; `mesh.vars` holds the base variable, and the base + holds the user-facing `MeshVariable` wrapper, so the object the model indexes + lives as long as the mesh too; +- a variable holds its mesh, so code holding only a variable keeps the mesh; +- a solver holds its mesh, and a swarm holds its mesh; +- coordinate symbols (`mesh.X`) and variable symbols hold the mesh and the + variable weakly, so sympy's caches of those symbols pin nothing; +- the primary mesh is a weak reference: once it is freed, `model.mesh` is + `None` and the next mesh built becomes primary. + +A dropped mesh is freed when nothing outside it refers to it. A mesh and its +variables form a reference cycle, so the cyclic collector frees them, and it +runs at a different moment on each rank. Two rules follow: + +- a traversal that makes a collective call per registered mesh (the snapshot + writers) takes its list from `Model._live_meshes()`, which collects first, + so every rank visits the same meshes; +- nothing in a mesh's cycle destroys a parallel PETSc object from `__del__`. + petsc4py defers the destroy of a collected object to a list that + `PETSc.garbage_cleanup` empties, and building a mesh (collective over its + DM's communicator) calls it. A temporary discarded at a collective point + destroys its vectors there (the variable statistics do). `Swarm.__del__` + destroys its DMSwarm: a swarm is in no cycle, so it is freed by reference + count where the program drops it, the same point on every rank. + +A driver's own references are its own: a driver that keeps the previous mesh +past a rebuild (for a warm-start transfer, say) keeps that mesh alive. ### Materials as Dictionaries diff --git a/docs/developer/subsystems/stress-transport.md b/docs/developer/subsystems/stress-transport.md index a1def6f95..d5230f4cd 100644 --- a/docs/developer/subsystems/stress-transport.md +++ b/docs/developer/subsystems/stress-transport.md @@ -8,7 +8,7 @@ what limits it, and how to keep a run inside those limits. ```python stokes = uw.systems.Stokes(mesh, velocityField=v, pressureField=p) -stokes.stress_transport = "integration_point" # or "semi_lagrangian" (the default), "forward", "eulerian" +stokes.stress_transport = "integration_point" # or "semi_lagrangian" (the default), "forward", "eulerian", "lagrangian", "none" stokes.constitutive_model = uw.constitutive_models.ViscoElasticPlasticFlowModel( stokes.Unknowns, order=1, integrator="bdf", objective_rate="upper_convected") stokes.constitutive_model.Parameters.shear_viscosity_0 = eta_p @@ -17,7 +17,7 @@ stokes.constitutive_model.Parameters.solvent_viscosity = eta_s # Oldroyd-B; o stokes.constitutive_model.Parameters.dt_elastic = dt ``` -## The five histories +## The six histories | `stress_transport` | storage | carried by | stable at | fails by | |---|---|---|---|---| @@ -26,6 +26,7 @@ stokes.constitutive_model.Parameters.dt_elastic = dt | `forward` | discontinuous P1 per cell, fitted from the arrivals | fixed launch set of interior points (the integration points), one forward trajectory a step; the flux is read back at the launch points through a continuous P1 projection; an inflow cell's uncovered share is filled with the inflow value | the cylinder walls at dt 0.04; below Courant one with `flux_smoothing` at c = 0.023 (Waters-King 1/16, dt 0.0125: 0.9543 at t 1 and 0.5185 at t 6.5, against nodal 0.9622 and 0.5171) | the same cell-scale mode as the integration-point history without that smoothing (diverges at t 2.4 there); first order only; does not cross a periodic seam or follow a moving mesh | | `lagrangian` (particles) | a swarm the solver owns and advects, one value per particle, read through a discontinuous cells proxy | the material points themselves: the constitutive flux is evaluated at the particles each step and never projected back to the mesh; a particle that entered through an inflow takes the inflow value | any Courant number; no numerical diffusion of the history | the cost and bookkeeping of a swarm, and a proxy that needs its cells kept populated (population control refills them); the conformation check does not read a per-point tensor from it | | `eulerian` (SUPG grid) | continuous P1 | assembled transport equation with streamline upwinding | with DEVSS | without DEVSS the velocity block loses its preconditioner as the stress grows | +| `none` (grid, untransported) | continuous P1 | nothing: the history stays at its node | any step whose displacement is far below a cell (a tidal or seismic cycle: the box fault model moves 1e-3 of a cell a step) | it is not a transport at all, so it is wrong as soon as material moves by a cell; a traced-back history on a split-node fault reads the other side of the slit (measured: a quarter of the slip rate on the second step), which this flavour avoids by not tracing | The nodal and integration-point flavours store the stress after every solve by the same global L2 projection onto the continuous space. The integration-point diff --git a/docs/reviews/2026-10/parallel-addon-distribution-adversarial-review.md b/docs/reviews/2026-10/parallel-addon-distribution-adversarial-review.md new file mode 100644 index 000000000..8cccc4156 --- /dev/null +++ b/docs/reviews/2026-10/parallel-addon-distribution-adversarial-review.md @@ -0,0 +1,112 @@ +# The add-on partitioned once: adversarial review record + +Branch `feature/parallel-addon-distribution`, from `feature/tidal-shell` at `f21f3d85`. +One round of three independent reviewers (parallel semantics and numerics; the wiring +into the cut, placement and split pipelines; tests, style, documentation and +contracts), read-only against the branch diff, 2026-10-17. Design note: +[parallel-addon-distribution-2026-10](../../developer/design/parallel-addon-distribution-2026-10.md). + +## What the branch does + +| Change | Why | +|---|---| +| `utilities/addon_partition.partition_addon`: one assignment for the add-on hierarchy, computed before any surgery and applied to every add-on level through the parent-cell relation; fault regions glued and sent to their majority rank, the load balanced by diffusion over the seams; provenance riding the move | every surgery moved cells on its own terms, so no two levels shared a partition and every consumer carried its own cross-partition handling | +| `add_fault`, `add_conforming_sheet`, `split_fault`, `split_faults` partition first; the level replay arrives partitioned | the split moved each level's star to its own majority rank; the 3-D placement refused on a partitioned adapt child ("the gather mask under-reached") | +| `fault_mg.provenance_transfer_parallel`: the pair the partition moved apart is built from the recorded parent, fetched through one star forest | the cell-located transfer all-gathered the misses as queries | +| the 2-D `add_fault` cut child takes the base's coarse levels | every solver on it fell back to GAMG | +| `colocate_split_level`, `fault_star_owner`, `_fault_star_cells` removed; `_redistribute_fault_interior` kept for `extrude_sheet` only | superseded | + +## Findings and their state + +Blocking: + +- **A second partition on a partitioned lineage was refused** (wiring review). The + "partition once" guard read only its argument; a placed or cut child of a + partitioned host fell through to a co-partition check that fails by design, so the + fault-network route (`add_conforming_sheet` per sheet), sheet-then-fault and two + `split_fault` calls all broke in parallel. Fixed: the first add-on level's parent + on the base is read from the recorded provenance when the pair is no longer + co-partitioned (`_base_parent_addresses`); the guard is gone. Tests: + `test_1096::test_a_second_partition_keeps_the_first_fault_whole`, + `test_1097::test_two_sheets_placed_in_turn_then_split_together`. +- **A split fault was not glued by a re-partition** (semantics review). Its facets + are single-sided after the split, so a support-based interior test marked nothing, + and the balance could peel the first fault's star across a seam. Fixed: glued + surfaces are named (`embedded_surface_names`: every pairing's label and sides, every + placed sheet), not detected; this also stops a domain-spanning internal boundary + being glued (should-fix in the same review). + +Should-fix, fixed: + +- Empty region ids gave a NaN fault centre and disordered the balance; regions with + no cell anywhere are dropped. +- The provenance branch skipped the finest level's FAC patch; it now sits inside the + builder branches. +- The provenance lookup located fine centroids in an ancestor's possibly stale + geometry; the relation is recorded when the surgery makes the mesh + (`inherit_provenance` in the cut, placement and split), and a mesh carrying its own + record needs no location at transfer time. +- `locate_in_cells_exhaustive` scanned every cell per miss; it widens to the nearest + 128 and 1024 cells first and scans only what still misses, and with the record + taken at surgery time it runs once per surgery, not per transfer. The two + inside-tolerances now agree. +- `mark_polylines` read a rank-local cell size at seam vertices; reconciled over the + star forest. +- The balance could overshoot by a whole unit and trade it back; a unit heavier than + twice the remaining gap is skipped. +- A level assigned wholly to one rank would be repartitioned by the Mesh constructor + under the provenance; refused collectively. +- Two non-collective raises inside collective sequences made collective. +- Serial and parallel tails differed for a sheet fault on a refined base with no + registered tail; the 3-D path takes the native coarse levels as the 2-D one does. +- Stale docstrings (`split_faults`, `split_fault`, `stamp_point_pairs`, + `_fault_labels_touch_seam`, `Mesh.add_fault`), the dead `groups` / `pairs` + parameters of `_redistribute_fault_interior`, the uncommented swallow in + `_face_other_rank`, the clearance default duplicated from `place_sheet`. +- Tests: the provenance pair's record (`fetched_remote`) and the per-level lineage + asserted; `test_1097` compares facets and pairs with its own serial run and the + tail by identity; the serial tail test by object identity; the fixtures shared in + `tests/parallel/fault_fixtures.py` (and `test_1089` moved onto them). +- Documentation: the design note's unit definition, its retirement list (honest about + `_colocate_level` in `adopt_hierarchy` and the extrude move), the query exchange + that still runs empty; supersession notes in the September placement note, + `FAULT_MULTIGRID_HIERARCHIES.md` and the conforming-surfaces subsystem page. +- Probes: the broken and unrelated ones deleted, a README dating the rest. + +Left open at the review, and their state on 18 October: + +- `place_thin_volume` behind `FaultNetwork.build` and the 2-D ribbon gathered its + own regions and `adopt_hierarchy` co-located the tail. **Done**: both routes + partition first with `mark_band`; `adopt_hierarchy` does not co-locate behind a + partitioned host (`test_1099`). +- A direct `place_sheet` call kept the under-reaching mask. **Done**: two growth + layers (`test_1097::test_direct_place_sheet_on_a_partitioned_adapt_child`, shown + failing first). +- Python loops per cell, face and vertex. **Done** for the partition, the region + growth, the marks, the redistribution's fields, the mesh's point-location index + and node table, and the node-cloud transfer assembly (`_plex_topology.pyx`; + equivalence checked entry for entry; `test_1098`). The 3-D partition at 87,525 + cells on four ranks went from 6.87 s to 1.35 s. The splitter's whole-mesh rebuild + is still per point (0.1 s of a 3.6 s split at 19,072 cells). +- When nothing needs to move, `partition_addon` stamps its record on the input mesh, + so a later placement on that same base that does not partition skips the tail's + co-location; its transfers then take the cross-partition path, correct and slower. +- The balance's seam adjacency is the original partition's; received units are not + passed on. It converges on the fixtures; not a bound. +- `_face_other_rank` and `place_surface._propagate_vertex` return early on a + `getGraph` exception before a star-forest collective (pre-existing pattern). +- `DMPlex.getPeriodicity()` segfaults on a non-periodic mesh in this petsc4py; the + periodic test reads the cell-coordinate handle instead. +- Only np = 2 runs in CI; the 4- and 8-rank runs here are by hand. + +## Attacks that found nothing + +- Collective ordering in `partition_addon`, the balance sweeps, `_region_cells`, + `_merge_regions`, the provenance transfer (local work only inside the guards). +- `_resolve_parents`: the root count is each rank's own pre-move cell count; one leaf + per root, so REPLACE is the reduction. +- The base is never redistributed in place; the cached wraps keep their identity, so + `provenance_source` matches by identity, newest record first. +- The l2g packing through doubles is exact; constrained columns survive and drop. +- Serial behaviour of every path is unchanged except the two tail fixes. +- CI reach: every new test file is in a `scripts/test.sh` glob; markers complete. diff --git a/docs/reviews/README.md b/docs/reviews/README.md index 456d3e38e..7344f0638 100644 --- a/docs/reviews/README.md +++ b/docs/reviews/README.md @@ -265,6 +265,7 @@ docs/reviews/ | Nonlinear defaults and η_reg ([2026-10](2026-10/nonlinear-defaults-794-adversarial-review.md), PR #794) | Solvers/Constitutive | adversarial, 2 rounds | Passing | 2026-10-06 | Fixes on branch | | JIT setup cost ([2026-10](2026-10/jit-setup-cost-823-adversarial-review.md), #823) | JIT/Solvers | adversarial, 3 rounds | Passing | 2026-10-07 | Fixes on branch | | Restore exactness and step ratio ([2026-10](2026-10/restore-exact-adversarial-review.md)) | Checkpoint/Time stepping | adversarial, 1 round (3 reviewers) | Passing | 2026-10-12 | Fixes on branch | +| Parallel add-on distribution ([2026-10](2026-10/parallel-addon-distribution-adversarial-review.md)) | Faults/Multigrid/Parallel | adversarial, 1 round (3 reviewers) | Passing | 2026-10-17 | Fixes on branch | | Units System (2026-02) | Units/Scaling | 60+ | Passing | 2026-02-01 | Approved (2026-03-10) | | Data Access (2026-02) | Array/Math Interface | 75+ | Passing | 2026-02-01 | Approved (2026-03-10) | | Reduction Operations | Mesh/Swarm Arrays | 5 | ✅ 100% | 2025-10-25 | ✅ Approved | diff --git a/setup.py b/setup.py index 924d753b3..acc1ea96f 100644 --- a/setup.py +++ b/setup.py @@ -273,6 +273,17 @@ def configure(): extra_compile_args=extra_compile_args, **conf, ), + # Bulk DMPlex topology tables (closures by depth, supports, per-cell + # sections) in one C loop per stratum, for the fault and add-on machinery. + Extension( + "underworld3.utilities._plex_topology", + sources=[ + "src/underworld3/utilities/_plex_topology.pyx", + ], + extra_compile_args=extra_compile_args, + define_macros=[("NPY_NO_DEPRECATED_API", "NPY_1_7_API_VERSION")], + **conf, + ), Extension( "underworld3.function._dmfield_wrapper", sources=[ diff --git a/src/underworld3/adjoint.py b/src/underworld3/adjoint.py index c87026df9..ce8bf74d5 100644 --- a/src/underworld3/adjoint.py +++ b/src/underworld3/adjoint.py @@ -623,7 +623,7 @@ def gradient(self, misfit, parameters: Iterable = (), fields: Iterable = ()): # ------------------------------------------------------------------ def _mesh(self): - return next(iter(self.model._variables.values())).mesh + return next(iter(list(self.model._variables.values()))).mesh def _tokens(self): """``{token: variable}`` — how each variable PRINTS inside a residual. @@ -633,7 +633,7 @@ def _tokens(self): token is taken from the symbol's own text (:func:`_token_of`). Matching on the name found the user's fields and silently missed every history, which cut the chain at the first step.""" - return {_token_of(var): var for var in self.model._variables.values() + return {_token_of(var): var for var in list(self.model._variables.values()) if hasattr(var, "sym")} def _reads(self, solver, unknown): diff --git a/src/underworld3/checkpoint/disk_snapshot.py b/src/underworld3/checkpoint/disk_snapshot.py index 72b44011c..043a9376e 100644 --- a/src/underworld3/checkpoint/disk_snapshot.py +++ b/src/underworld3/checkpoint/disk_snapshot.py @@ -73,7 +73,7 @@ ) -def _collect_metadata(model) -> dict: +def _collect_metadata(model, meshes) -> dict: """Build the metadata dict that gets written into ``/metadata`` attrs. Stable, h5-friendly types only: strings, ints, floats, lists of @@ -85,7 +85,6 @@ def _collect_metadata(model) -> dict: ) # Mesh-derived info (gracefully absent if no mesh registered). - meshes = list(model._meshes.values()) first_mesh = meshes[0] if meshes else None mesh_names = [m.name for m in meshes] if first_mesh is not None: @@ -175,6 +174,13 @@ def write_snapshot_skeleton(model, path: str) -> str: respectively. Writing is rank-0-only at this phase since no collective PETSc operations are involved yet. """ + return _write_skeleton(model, path, model._live_meshes()) + + +def _write_skeleton(model, path, meshes): + """:func:`write_snapshot_skeleton` for the meshes ``meshes``, listed by the + caller on every rank (the collection behind the list is never made on + rank 0 alone).""" import h5py with uw.selective_ranks(0) as should_execute: @@ -182,7 +188,7 @@ def write_snapshot_skeleton(model, path: str) -> str: uw.mpi.barrier() return path - metadata = _collect_metadata(model) + metadata = _collect_metadata(model, meshes) with h5py.File(path, "w") as f: md_group = f.create_group(_GROUP_METADATA) @@ -297,8 +303,12 @@ def write_snapshot(model, path: str) -> str: """ import h5py + # Every rank lists the meshes after one collection, so every rank makes + # the same collective writes below (see Model._live_meshes). + meshes = model._live_meshes() + # Phase-1 layer: metadata + skeleton groups. - write_snapshot_skeleton(model, path) + _write_skeleton(model, path, meshes) bulk_dir = _bulk_dir_for(path) # rank-0 creates the bulk directory; collective ops below need it @@ -314,7 +324,7 @@ def write_snapshot(model, path: str) -> str: # bulk directory. write_checkpoint is collective (PETSc HDF5 # viewer), so all ranks must participate. mesh_records: list[dict] = [] - for mesh_index, mesh in enumerate(list(model._meshes.values())): + for mesh_index, mesh in enumerate(meshes): # Loaded meshes commonly use their complete source path as ``name``. # Embedding that path in every PETSc-HDF5 bulk filename can exceed # practical MPI-I/O pathname limits even when the wrapper path itself @@ -397,7 +407,7 @@ def write_snapshot(model, path: str) -> str: # Phase 3c: solver step state. solver_group = f.require_group(_GROUP_SOLVER_STATES) - for solver in _solvers_of(model): + for solver in _solvers_of(meshes): if solver.name not in solver_group: _write_state_bearer_to_group( solver_group.create_group(solver.name), solver) @@ -499,7 +509,8 @@ def read_snapshot(model, path: str) -> None: ) # Build {original_name -> registered Mesh} for lookup - meshes_by_name = {m.name: m for m in model._meshes.values()} + meshes = model._live_meshes() + meshes_by_name = {m.name: m for m in meshes} with h5py.File(path, "r") as f: meshes_group = f[_GROUP_MESHES] @@ -555,7 +566,7 @@ def read_snapshot(model, path: str) -> None: # (built since, or a file written before solver states were kept) goes # back to the state of one that has never solved, as in a new run. stored = f[_GROUP_SOLVER_STATES] if _GROUP_SOLVER_STATES in f else {} - for solver in _solvers_of(model): + for solver in _solvers_of(meshes): if solver.name in stored: _read_state_bearer_into(stored[solver.name], solver) else: diff --git a/src/underworld3/checkpoint/snapshot.py b/src/underworld3/checkpoint/snapshot.py index 817169ec2..1d730e3d2 100644 --- a/src/underworld3/checkpoint/snapshot.py +++ b/src/underworld3/checkpoint/snapshot.py @@ -203,21 +203,22 @@ def snapshot(model, *, path: Optional[str] = None) -> Snapshot: ) snap = Snapshot(backend=InMemoryBackend()) - for mesh in list(model._meshes.values()): + meshes = model._live_meshes() + for mesh in meshes: _capture_mesh(snap, mesh) for swarm in list(model._swarms.values()): _capture_swarm(snap, swarm) for obj in list(model._state_bearers): _capture_state_bearer(snap, obj) _capture_expressions(snap) - for solver in _solvers_of(model): + for solver in _solvers_of(meshes): snap.solver_states[solver.name] = copy.deepcopy(solver.state) return snap -def _solvers_of(model): - """Every solver built on the model's meshes.""" - return [solver for mesh in list(model._meshes.values()) +def _solvers_of(meshes): + """Every solver built on ``meshes`` (a list from ``Model._live_meshes``).""" + return [solver for mesh in meshes for solver in mesh._equation_systems_register] @@ -395,8 +396,9 @@ def restore(model, snap: Snapshot) -> None: f"current {SNAPSHOT_SCHEMA_VERSION}; on-disk migration is v1.1" ) - meshes_by_name = {m.name: m for m in model._meshes.values()} - swarms_by_name = {_swarm_stable_name(s): s for s in model._swarms.values()} + meshes = model._live_meshes() + meshes_by_name = {m.name: m for m in meshes} + swarms_by_name = {_swarm_stable_name(s): s for s in list(model._swarms.values())} for mesh_name in snap.mesh_names: mesh = meshes_by_name.get(mesh_name) @@ -447,7 +449,7 @@ def restore(model, snap: Snapshot) -> None: # with it the preconditioner is rebuilt on the next solve. A solver built # since the capture (some are built on first use) had not solved at the # captured point, so it goes back to the state of one that never has. - for solver in _solvers_of(model): + for solver in _solvers_of(meshes): captured = snap.solver_states.get(solver.name) solver.state = (copy.deepcopy(captured) if captured is not None else type(solver.state)()) diff --git a/src/underworld3/coordinates.py b/src/underworld3/coordinates.py index 47bb74984..a6b414db4 100644 --- a/src/underworld3/coordinates.py +++ b/src/underworld3/coordinates.py @@ -29,6 +29,7 @@ from enum import Enum import tempfile +import weakref import numpy as np from petsc4py import PETSc @@ -112,8 +113,11 @@ def __new__(cls, index, system, pretty_str=None, latex_str=None, mesh=None, axis return obj def __init__(self, index, system, pretty_str=None, latex_str=None, mesh=None, axis_index=None): - # Store UW3-specific attributes - self._mesh = mesh + # Store UW3-specific attributes. The mesh owns its coordinates, so the + # coordinate holds it weakly: a coordinate symbol outlives its mesh in + # sympy's function caches, and a strong link there pinned the mesh, + # its DM hierarchy and its solvers for the rest of the process. + self._mesh_ref = weakref.ref(mesh) if mesh is not None else None self._axis_index = axis_index if axis_index is not None else index self._coord_name = pretty_str or f"x_{index}" @@ -206,14 +210,14 @@ def data(self): numpy.ndarray Coordinate values for this axis at all mesh nodes """ - if self._mesh is None: + if self.mesh is None: raise ValueError("UWCoordinate not attached to a mesh") - return self._mesh.X.coords[:, self._axis_index] + return self.mesh.X.coords[:, self._axis_index] @property def mesh(self): - """Parent mesh.""" - return self._mesh + """Parent mesh (``None`` once the mesh has been freed).""" + return self._mesh_ref() if self._mesh_ref is not None else None @property def axis(self): @@ -520,6 +524,11 @@ class GeographicCoordinateAccessor: approximately 10-11 arcminutes, which matters for regional models. """ + @property + def mesh(self): + """The mesh, read through the coordinate system (held weakly there).""" + return self.cs.mesh + def __init__(self, coordinate_system): """ Initialize geographic coordinate accessor. @@ -530,7 +539,6 @@ def __init__(self, coordinate_system): The coordinate system object with GEOGRAPHIC type """ self.cs = coordinate_system - self.mesh = coordinate_system.mesh # Cache for coordinate arrays self._lon_cache = None @@ -975,6 +983,11 @@ class SphericalCoordinateAccessor: use a GEOGRAPHIC mesh with ``mesh.X.geo`` instead. """ + @property + def mesh(self): + """The mesh, read through the coordinate system (held weakly there).""" + return self.cs.mesh + def __init__(self, coordinate_system): """ Initialize spherical/polar coordinate accessor. @@ -985,7 +998,6 @@ def __init__(self, coordinate_system): The coordinate system object with SPHERICAL or CYLINDRICAL2D type """ self.cs = coordinate_system - self.mesh = coordinate_system.mesh # 2 for polar (cdim=2 mesh), 3 for spherical (cdim=3 mesh — # either a 3-D volume mesh or a 2-manifold in 3-space). # Always branch on cdim, not dim: spherical coords are a @@ -1370,6 +1382,11 @@ class CoordinateSystem: """ + @property + def mesh(self): + """The mesh this system belongs to (``None`` once it has been freed).""" + return self._mesh_ref() + def __init__( self, mesh, @@ -1383,7 +1400,9 @@ def __init__( "This object is not meant to be reconstructed by SymPy." ) - self.mesh = mesh + # Held weakly: the mesh owns its coordinate system, and the system's + # symbols live on in sympy's caches after the mesh is dropped. + self._mesh_ref = weakref.ref(mesh) self.coordinate_type = system # are the mesh coordinates XYZ or have they been replaced by diff --git a/src/underworld3/discretisation/discretisation_mesh.py b/src/underworld3/discretisation/discretisation_mesh.py index 94f95499b..199c6a119 100644 --- a/src/underworld3/discretisation/discretisation_mesh.py +++ b/src/underworld3/discretisation/discretisation_mesh.py @@ -437,6 +437,15 @@ def __init__( self.instance = Mesh.mesh_instances Mesh.mesh_instances += 1 + # Building a mesh is collective over its DM's communicator, so it is + # where the PETSc objects of meshes dropped since the last one are + # destroyed in parallel. A file is read on the communicator the + # reader below takes (``comm=``, else COMM_WORLD). + from underworld3.mpi import _free_petsc_garbage + _free_petsc_garbage(plex_or_meshfile.getComm() + if isinstance(plex_or_meshfile, PETSc.DM) + else kwargs.get("comm", PETSc.COMM_WORLD)) + # Coordinate units come from the model (not a user parameter): # the model owns the unit system so all meshes and variables agree. import underworld3 as uw @@ -606,8 +615,10 @@ def __init__( populated_families = [family for family in cell_families if family is not None] self.isSimplex = all(populated_families) if populated_families else simplex - # Using WeakValueDictionary to prevent circular references - self._vars = weakref.WeakValueDictionary() + # The mesh owns its variables (a variable created and not kept lives as + # long as its mesh). Each variable holds its mesh back; the cycle is + # collected with the mesh once nothing else holds either. + self._vars = {} self._block_vars = {} # a list of equation systems that will @@ -652,8 +663,11 @@ def __init__( self.qdegree = qdegree # Populate the element information for this mesh. This is intended to be - # human readable because the mesh is quite simple: either quads / tris in 2D - # tetrahedra / hexahedra in 3D + # human readable because the mesh is quite simple: lines in 1D (a surface + # submesh of a 2-D domain), quads / tris in 2D, tetrahedra / hexahedra + # in 3D. ``entities[k]`` counts a cell's points of codimension k + # (entities[1] its faces, entities[dim] its vertices); face_entities[dim] + # counts a face's vertices. from dataclasses import dataclass @@ -663,7 +677,12 @@ class ElementInfo: entities: tuple face_entities: tuple - if self.isSimplex: + if self.dim == 1: + # A line cell, simplex or not: two faces, which are its two + # vertices. Without this a 1-D mesh fell through to the + # tetrahedron entry and read 4 faces and 4 vertices per cell. + self._element = ElementInfo("line", (1, 2), (0, 1)) + elif self.isSimplex: if self.dim == 2: self._element = ElementInfo("triangle", (1, 3, 3), (0, 1, 2)) else: @@ -5685,35 +5704,28 @@ def _cell_node_indices(self, degree, continuous): if key in self._cell_node_array: return self._cell_node_array[key] + from underworld3.utilities._plex_topology import closure_dof_rows + dmnew = self._basis_coordinate_dm(degree, continuous) - section = dmnew.getLocalSection() cStart, cEnd = self.dm.getHeightStratum(0) - cdim = self.cdim - - rows = [] - for cell in range(cStart, cEnd): - cell_rows = [] - for point in dmnew.getTransitiveClosure(cell)[0]: - ndof = section.getDof(point) // cdim - if ndof: - offset = section.getOffset(point) // cdim - cell_rows.extend(range(offset, offset + ndof)) - rows.append(cell_rows) + rows = closure_dof_rows(dmnew, dmnew.getLocalSection(), + numpy.arange(cStart, cEnd), self.cdim) dmnew.destroy() expected = comb(degree + self.dim, self.dim) - if not self.isSimplex or any(len(r) != expected for r in rows): + counts = (rows >= 0).sum(axis=1) + if not self.isSimplex or (counts != expected).any(): # A tensor-product Q_k cell carries (k+1)^dim nodes, so the monomial # basis of total degree <= k would not be square against them and # the dual-basis construction does not apply. Say so here rather # than return a ragged array the caller has to second-guess. - got = sorted({len(r) for r in rows}) + got = sorted(set(counts.tolist())) raise NotImplementedError( f"_cell_node_indices needs a simplex mesh: expected " f"{expected} nodes per cell for degree {degree} in {self.dim}D, " f"got {got}") - self._cell_node_array[key] = numpy.asarray(rows, dtype=numpy.int64) + self._cell_node_array[key] = rows.reshape(cEnd - cStart, expected) return self._cell_node_array[key] def _get_coords_for_basis(self, degree, continuous): @@ -5762,13 +5774,10 @@ def _coord_rows_for_points(self, nav_dm, points): so the rows returned here are bit-identical to the previous affine indexing (verified) and serial navigation is unchanged. """ - coord_section = nav_dm.getCoordinateSection() - cdim = self.cdim - return numpy.fromiter( - (coord_section.getOffset(p) // cdim for p in points), - dtype=numpy.int64, - count=len(points), - ) + from underworld3.utilities._plex_topology import section_offsets + + return section_offsets(nav_dm.getCoordinateSection(), + numpy.asarray(points, dtype=numpy.int64)) // self.cdim def _navigation_coords_from_dm(self, nav_dm): """Section-consistent LOCAL coordinate rows for a navigation DM. @@ -5806,84 +5815,45 @@ def _build_kd_tree_index(self): nav_coords = self._nav_coords cStart, cEnd = nav_dm.getHeightStratum(0) - fStart, fEnd = nav_dm.getHeightStratum(1) - pStart, pEnd = nav_dm.getDepthStratum(0) - cell_num_faces = self.element.entities[1] - cell_num_points = self.element.entities[self.dim] - face_num_points = self.element.face_entities[self.dim] - control_points_list = [] - control_points_cell_list = [] - centroids_list = [] + from underworld3.utilities._plex_topology import closure_points + + # Each cell's vertices in closure order, read as one table. The count + # is the closure's own: the element table's vertex count does not + # describe a 1-D surface mesh (it gave 4 for a line cell, and the + # earlier per-cell slice then took the edge point itself as a vertex). + cells = numpy.arange(cStart, cEnd) + verts = closure_points(nav_dm, cells, 0) + if (verts < 0).any(): + raise RuntimeError( + "_build_kd_tree_index: the cells do not all have the same " + "number of vertices") + cell_num_points = verts.shape[1] + n_cells = cEnd - cStart + # Use raw internal array for KD-tree construction (avoid unit-aware wrapping) + P = nav_coords[self._coord_rows_for_points(nav_dm, verts.ravel())].reshape( + n_cells, cell_num_points, self.cdim) + centroids = P.mean(axis=1) # Largest distance from a cell centroid to one of that cell's own # vertices, maximised over local cells. A convex cell is the convex # hull of its vertices, so every point of it lies within this distance # of its centroid — which makes it the rejection radius the locator # needs (see _get_closest_local_cells_internal). - cell_reach = 0.0 - - for cell, cell_id in enumerate(range(cStart, cEnd)): - - cell_faces = nav_dm.getCone(cell_id) - points = nav_dm.getTransitiveClosure(cell_id)[0][-cell_num_points:] - # Use raw internal array for KD-tree construction (avoid unit-aware wrapping) - cell_point_coords = nav_coords[self._coord_rows_for_points(nav_dm, points)] - cell_centroid = cell_point_coords.mean(axis=0) - centroids_list.append(cell_centroid) - cell_reach = max(cell_reach, float(numpy.linalg.norm( - cell_point_coords - cell_centroid, axis=1).max())) - - # for face in range(cell_num_faces): - - # points = self.dm.getTransitiveClosure(cell_faces[face])[0][ - # -face_num_points: - # ] - # point_coords = self.data[points - pStart] - - # face_centroid = point_coords.mean(axis=0) - # cell_centroid = cell_point_coords.mean(axis=0) - - # # 2D case - # if self.dim == 2: - # vector = point_coords[1] - point_coords[0] - # normal = numpy.array((-vector[1], vector[0])) - - # # 3D simplex case (probably also OK for hexes) - # else: - # normal = numpy.cross( - # (point_coords[1] - point_coords[0]), - # (point_coords[2] - point_coords[0]), - # ) - - # inward_outward = numpy.sign(normal.dot(face_centroid - cell_centroid)) - # normal *= inward_outward / numpy.sqrt(normal.dot(normal)) - - # inside_control_point = -1e-3 * normal + face_centroid - - # control_points_list.append(inside_control_point) - # control_points_cell_list.append(cell_id) - # control_points_list.append(cell_centroid) - # control_points_cell_list.append(cell_id) - - # Add points near the cell vertices - - for i in range(cell_point_coords.shape[0]): - control_points_list.append(0.99 * cell_point_coords[i] + 0.01 * cell_centroid) - control_points_cell_list.append(cell_id) - - # Add centroid - control_points_list.append(cell_centroid) - control_points_cell_list.append(cell_id) + cell_reach = (float(numpy.linalg.norm(P - centroids[:, None, :], + axis=2).max()) + if n_cells else 0.0) + # Control points: each cell's vertices pulled 1% toward its centroid, + # then the centroid itself, cell by cell. + control = numpy.concatenate( + [0.99 * P + 0.01 * centroids[:, None, :], centroids[:, None, :]], axis=1) # A rank can own zero cells (small mesh / imbalanced partition): - # shape the point arrays explicitly so an empty list becomes a - # well-formed (0, cdim) array rather than a 1-D numpy.array([]), - # which crashed the KDTree construction (issue #399). - self._indexCoords = numpy.array( - control_points_list, dtype=numpy.float64).reshape(-1, self.cdim) + # shape the point arrays explicitly so an empty table is a + # well-formed (0, cdim) array, which the KDTree accepts (issue #399). + self._indexCoords = numpy.ascontiguousarray( + control, dtype=numpy.float64).reshape(-1, self.cdim) self._index = uw.kdtree.KDTree(self._indexCoords) - # self._index.build_index() - self._indexMap = numpy.array(control_points_cell_list, dtype=numpy.int64) + self._indexMap = numpy.repeat(cells, cell_num_points + 1).astype(numpy.int64) # Cell-centroid kdtree, built from the nav-DM cells in the # same enumeration order as _indexMap, so the indices it @@ -5891,8 +5861,8 @@ def _build_kd_tree_index(self): # We keep _nav_centroids separate from _centroids (which is # the main-DM cell centroids set in __init__) so the FE-side # ``_centroids`` semantics are unchanged on manifold meshes. - self._nav_centroids = numpy.array( - centroids_list, dtype=numpy.float64).reshape(-1, self.cdim) + self._nav_centroids = numpy.ascontiguousarray( + centroids, dtype=numpy.float64).reshape(-1, self.cdim) self._centroid_index = uw.kdtree.KDTree(self._nav_centroids) # Rejection radius for the lost-point walk. Rebuilt with the kd-tree @@ -7960,6 +7930,10 @@ def _adopt_cut_child(self, cut_dm, boundaries, info, mg_coarsening_ratio, child.regions = self.regions child._parent_mesh_version = self._mesh_version child._surface_info = info + if getattr(self, "_embedded_surfaces", None): + child._embedded_surfaces = set(self._embedded_surfaces) + from underworld3.utilities.addon_partition import inherit_provenance + inherit_provenance(child, self) # Mesh-owned custom-P geometric-MG tail. Adding a surface refines this # mesh, so this mesh plus everything below it is a valid coarse tail and @@ -8288,13 +8262,30 @@ def add_conforming_sheet(self, points, triangles, name, *, "add_conforming_sheet is 3-D; in 2-D use " "add_conforming_surface.") - boundaries = self._boundaries_with(name) + # In parallel the add-on is partitioned for the sheet first — the + # placement's own mask with two growth layers, on this level and on + # every add-on level below it — so the carve and the fill run on a + # rank-interior region and the placement gathers nothing; a split of + # the sheet later moves nothing either. + host = self + if self.dm.getComm().getSize() > 1: + from underworld3.utilities.addon_partition import (mark_sheets, + partition_addon) + host = partition_addon( + self, mark_sheets([(points, triangles)], clearance), layers=2, + verbose=verbose) + boundaries = host._boundaries_with(name) cut_dm, info = place_sheet( - self.dm, points, triangles, label=name, + host.dm, points, triangles, label=name, label_value=boundaries[name].value, clearance=clearance, verbose=verbose, setback=setback, size=size) - return self._adopt_cut_child(cut_dm, boundaries, info, - mg_coarsening_ratio, verbose) + child = host._adopt_cut_child(cut_dm, boundaries, info, + mg_coarsening_ratio, verbose) + # a placed sheet is a fault surface to be: every later partition + # keeps it whole (addon_partition.embedded_surface_names) + child._embedded_surfaces = ( + set(getattr(host, "_embedded_surfaces", ()) or ()) | {name}) + return child def extrude(self, thickness, layers=2, base="Base", surface="Surface", qdegree=None, verbose=False): @@ -8332,11 +8323,10 @@ def add_fault(self, faults, verbose=False, cut=True, exclude=None, crossings are represented as OFFSET segments (a one-to-two-cell ligament). The result inherits a MESH-OWNED geometric-MG tail (the parent's coarse levels, the cut mesh finest — a cut is the same - grid re-represented); a parent without one yields a standalone - mesh — no tail, since - the coarse levels do not carry the fault (see - :meth:`add_conforming_surface`); solvers take their - algebraic-multigrid defaults. + grid re-represented): the parent's registered tail, or the coarse + levels of a parent built with ``refinement >= 1``. In parallel the + add-on is partitioned once for the faults before the cut + (``utilities.addon_partition``), so nothing moves afterwards. ``cut=False`` labels the mesh edges already on each polyline (the embedded spines of a placed band); ``across_seams=True`` then @@ -8371,40 +8361,27 @@ def add_fault(self, faults, verbose=False, cut=True, exclude=None, # The split mesh INHERITS a mesh-owned geometric-MG tail: a cut # re-represents the same grid with the surface conformed (finer only # by the duplicated vertices), so the parent's coarse levels serve - # with the cut mesh as the finest level. The static background - # levels serve unchanged; the add-on levels are split too (above). - # Without this every solver on a split mesh fell to GAMG unless it - # called set_custom_fmg by hand. The FAC zone is NOT inherited: a + # with the cut mesh as the finest level (fault_split.add_fault hands + # them to the cut child — partitioned with it in parallel). The static + # background levels serve unchanged; the add-on levels are split too, + # each on its own partition, which the add-on partition made the + # finest's: nothing is moved here. The FAC zone is NOT inherited: a # split fault needs no patch (the keying ruling). - own_tail = getattr(self, "_custom_mg_coarse_meshes", None) - if (own_tail is not None - and getattr(child, "_custom_mg_coarse_meshes", None) is None): - tail = list(own_tail) - # a placed band's spines (cut=False) have no counterpart on the - # levels, which were never placed: only a CUT fault is replayed - if split_levels and cut: - from underworld3.utilities.fault_mg import ( - addon_level_indices, split_tail_levels) - idx = addon_level_indices(tail) - if idx: - split = split_tail_levels([tail[k] for k in idx], - child._fault_specs, - verbose=verbose) - if self.dm.getComm().getSize() > 1: - # each level's star onto the rank holding the finest's, - # so the transfers locate the band rank-locally - from underworld3.utilities.fault_mg import ( - colocate_split_level, fault_star_owner) - target = fault_star_owner(child) - split = [colocate_split_level(m, target) - if getattr(m, "_fault_point_pairs", None) else m - for m in split] - for k, m in zip(idx, split): - tail[k] = m + tail = getattr(child, "_custom_mg_coarse_meshes", None) + # a placed band's spines (cut=False) have no counterpart on the + # levels, which were never placed: only a CUT fault is replayed + if tail is not None and split_levels and cut: + from underworld3.utilities.fault_mg import ( + addon_level_indices, split_tail_levels) + tail = list(tail) + idx = addon_level_indices(tail) + if idx: + split = split_tail_levels([tail[k] for k in idx], + child._fault_specs, + verbose=verbose) + for k, m in zip(idx, split): + tail[k] = m child._custom_mg_coarse_meshes = tail - child._custom_mg_builder = getattr(self, "_custom_mg_builder", - "barycentric") - child._custom_mg_fac_zone = None return child diff --git a/src/underworld3/discretisation/discretisation_mesh_variables.py b/src/underworld3/discretisation/discretisation_mesh_variables.py index 1b8622567..b0626cf1c 100644 --- a/src/underworld3/discretisation/discretisation_mesh_variables.py +++ b/src/underworld3/discretisation/discretisation_mesh_variables.py @@ -223,6 +223,9 @@ def __init__( units = params["units"] units_backend = params["units_backend"] remesh_policy = params.get("remesh_policy", remesh_policy) + # consumed: kept, its strong "mesh" entry pinned the mesh for as + # long as the variable lived (the variable holds its mesh weakly) + del self._init_params else: # Direct initialization (should not happen with __new__ pattern, but for safety) pass @@ -294,6 +297,10 @@ def __init__( self.vtype = vtype self._mesh_ref = weakref.ref(mesh) + # A variable keeps its mesh alive: a helper may return only the + # variable. The variable's symbols hold the variable weakly, so caches + # of those symbols do not pin it, or the mesh. + self._mesh_owner = mesh self.shape = num_components self.degree = degree self.continuous = continuous @@ -1223,6 +1230,13 @@ def _petsc_payload_name(data_file, data_name): ``None`` when the file was written without ``petsc_reload=True``. """ + # TODO(BUG): the writer puts the native payload under + # /restart/petsc/topologies (Mesh._write_petsc_reload_file, since + # 36c01399) while this looks for a top-level "topologies" group, so a + # payload written with petsc_reload=True is never found and a split + # mesh's read_timestep refuses (tests/test_0864_split_checkpoint_ + # roundtrip.py::test_roundtrip_keeps_the_two_sides_distinct fails on + # feature/tidal-shell too). Found 2026-10-18; not fixed on this branch. import h5py found = None @@ -2453,11 +2467,12 @@ def _replace_from_adapted_mesh(self, temp_var, adapted_mesh): self._data_cache = None self._array_cache = None - def __del__(self): - if self._lvec: - self._lvec.destroy() - if self._gvec: - self._gvec.destroy() + # No __del__ destroying the vectors: a variable is in a reference cycle + # with its mesh, so it is freed by the cyclic collector, at a different + # moment on each rank. An explicit destroy of the parallel vector there + # would be a collective call made by one rank. petsc4py frees both vectors + # when the variable goes, deferring the parallel one to the collective + # cleanup that the next mesh construction runs (mpi._free_petsc_garbage). @property def mesh(self): @@ -3799,6 +3814,19 @@ def _scalar_stats(self): "rms": vrms, } + def _discard_temporary(self, temporary): + """Take a statistics temporary off the mesh and destroy its vectors. + + Every rank reaches this at the same point, the end of a collective + statistics call, so the parallel vector is destroyed collectively here. + Left to the collector, it waits on petsc4py's deferred list until the + next mesh is built, one vector per call on a fixed mesh.""" + if temporary.name in self.mesh.vars: + del self.mesh.vars[temporary.name] + for vector in (temporary._lvec, temporary._gvec): + if vector is not None: + vector.destroy() + def _vector_stats(self): """Statistics for vector variables using magnitude.""" import numpy as np @@ -3845,9 +3873,7 @@ def _vector_stats(self): return mag_stats finally: - # Cleanup temporary variable - if magnitude_var.name in self.mesh.vars: - del self.mesh.vars[magnitude_var.name] + self._discard_temporary(magnitude_var) def _tensor_stats(self): """Statistics for tensor variables using Frobenius norm.""" @@ -3905,9 +3931,7 @@ def _tensor_stats(self): return frob_stats finally: - # Cleanup temporary variable - if frobenius_var.name in self.mesh.vars: - del self.mesh.vars[frobenius_var.name] + self._discard_temporary(frobenius_var) @property def coords(self) -> numpy.ndarray: diff --git a/src/underworld3/discretisation/enhanced_variables.py b/src/underworld3/discretisation/enhanced_variables.py index 311ef6e63..3b34164bd 100644 --- a/src/underworld3/discretisation/enhanced_variables.py +++ b/src/underworld3/discretisation/enhanced_variables.py @@ -145,6 +145,11 @@ def __init__( units_backend=units_backend, **kwargs, ) + # The mesh owns its variables through ``mesh.vars``, which holds the + # base; the base holds this wrapper back, so the variable users see + # lives as long as its mesh. The model indexes the wrapper weakly, and + # units, scaling and the adjoint find variables there. + self._base_var._wrapper = self # STRICT UNITS MODE CHECK # Enforce units-scales contract: variables with units require reference quantities @@ -238,7 +243,7 @@ def _setup_persistence_features(self): # Store qualified name for later reference (for persistent variables) model = uw.get_default_model() self._qualified_name = None - for name, var in model._variables.items(): + for name, var in list(model._variables.items()): if var is self: self._qualified_name = name break diff --git a/src/underworld3/meshing/extrude.py b/src/underworld3/meshing/extrude.py index 218931438..73f2ac7ff 100644 --- a/src/underworld3/meshing/extrude.py +++ b/src/underworld3/meshing/extrude.py @@ -73,7 +73,7 @@ def _gather_sheet(dm, pairs): """Everything onto rank 0: ``(dm, pairs)`` in the gathered numbering.""" from underworld3.utilities.fault_split import redistribute_by_assignment cS, cE = dm.getHeightStratum(0) - gathered, pairs, _dropped = redistribute_by_assignment( + gathered, pairs, _dropped, _fields = redistribute_by_assignment( dm, np.zeros(cE - cS, dtype=np.int32), pairs=pairs) return gathered, pairs or {} diff --git a/src/underworld3/meshing/fault_network.py b/src/underworld3/meshing/fault_network.py index 57137750f..bbed338a7 100644 --- a/src/underworld3/meshing/fault_network.py +++ b/src/underworld3/meshing/fault_network.py @@ -569,8 +569,18 @@ def _build_3d_band(self, h_far=None, qdegree=2, realisation="split", "embedded mid-surface. Keep the network interior, " "or place the zone with place_thin_volume directly " "(no embed).") + # In parallel the add-on is partitioned for the band first, so the + # placement gathers nothing, the split moves nothing, and the tail + # is not co-located behind it (utilities.addon_partition). + host = base + if base.dm.getComm().getSize() > 1: + from underworld3.utilities.addon_partition import (mark_band, + partition_addon) + host = partition_addon( + base, mark_band(expanded, self.width, carve_clearance), + layers=2) dm, info = place_thin_volume( - base.dm, expanded, self.width, label="Band", label_value=71, + host.dm, expanded, self.width, label="Band", label_value=71, clearance=carve_clearance, size=h, mesher="network", embed=[np.asarray(P, dtype=float) for _n, P in self.prepared]) values = {} @@ -588,6 +598,8 @@ def _build_3d_band(self, h_far=None, qdegree=2, realisation="split", dm, simplex=True, qdegree=qdegree, coordinate_system_type=base.CoordinateSystem.coordinate_type, boundaries=Enum("boundaries", members), verbose=False) + from underworld3.utilities.addon_partition import inherit_provenance + inherit_provenance(mesh, host) band = mesh.cells_labelled("Band", 71) if realisation == "split": # the pieces the placement kept apart stay apart in the split's @@ -622,7 +634,7 @@ def _build_3d_band(self, h_far=None, qdegree=2, realisation="split", # tail — every solver fell back to GAMG. The keying ruling # stands: the split needs no FAC patch; the weak plane's patch # is its band. - adopt_hierarchy(mesh, base, + adopt_hierarchy(mesh, host, fac_zone=None if realisation == "split" else band) # honoured footprints: band cells within the USER patch's own # in-plane outline and half a width of its plane — planar patches diff --git a/src/underworld3/model.py b/src/underworld3/model.py index 06603c751..fa05f690d 100644 --- a/src/underworld3/model.py +++ b/src/underworld3/model.py @@ -17,6 +17,7 @@ - Materials are just dictionaries of expressions/values """ +import gc import weakref import json from typing import Optional, Dict, Any, List @@ -416,8 +417,15 @@ class Model(PintNativeModelMixin, BaseModel): ) # Declare private attributes for Pydantic v2 - _meshes: Any = PrivateAttr(default_factory=dict) + # Weak, like the swarms below: a strong registry kept every mesh ever + # built, with its DM hierarchy, its variables' vectors and its solvers' + # matrices. A driver that rebuilds a faulted mesh at each fault move grew + # by ~0.4 GB per rank per rebuild until it was killed (kaiju, 2026-10). + _meshes: Any = PrivateAttr(default_factory=weakref.WeakValueDictionary) _primary_mesh_id: Optional[int] = PrivateAttr(default=None) + # The primary mesh itself, held weakly: an id alone outlives its mesh and + # can be reused by an unrelated later mesh (differently on each rank). + _primary_mesh_ref: Any = PrivateAttr(default=None) # WeakValueDictionary so dropping the user's last reference to a swarm # actually lets it be garbage-collected. Without this, transient swarms # used inside functions (global expression evaluation, checkpoint reads, @@ -429,8 +437,11 @@ class Model(PintNativeModelMixin, BaseModel): # traversal (summaries, snapshots) should iterate a copy such as # ``list(self._swarms.items())``. _swarms: Any = PrivateAttr(default_factory=weakref.WeakValueDictionary) - _variables: Dict[str, Any] = PrivateAttr(default_factory=dict) - _solvers: Dict[str, Any] = PrivateAttr(default_factory=dict) + # Weak: the model indexes variables and solvers, it does not own them (a + # mesh owns its variables; a variable or solver holds its mesh), so a + # strong entry here pinned the mesh. + _variables: Any = PrivateAttr(default_factory=weakref.WeakValueDictionary) + _solvers: Any = PrivateAttr(default_factory=weakref.WeakValueDictionary) # State-bearing objects (DDt instances, parameter-history holders, # any helper that exposes a .state dataclass per the Snapshottable @@ -479,7 +490,9 @@ class Model(PintNativeModelMixin, BaseModel): _parts: Dict[str, Any] = PrivateAttr(default_factory=dict) # part -> the live object behind it (solver, history manager), so a # pass over the transcript can call the operator a record names. - _part_objects: Dict[str, Any] = PrivateAttr(default_factory=dict) + # Weak: a record names its part's object while it lives; a strong entry + # kept every solver that ever solved in a step, and its mesh. + _part_objects: Any = PrivateAttr(default_factory=weakref.WeakValueDictionary) # Set while rewind() is doing its own restore, so load_state does not log a # second, less informative note for the same backtrack. _restoring: Any = PrivateAttr(default=False) @@ -528,10 +541,9 @@ def __init__(self, name: Optional[str] = None, **kwargs): @property def mesh(self): - """The primary mesh for this model""" - if self._primary_mesh_id is None: - return None - return self._meshes.get(self._primary_mesh_id) + """The primary mesh for this model (``None`` once it has been freed).""" + ref = self._primary_mesh_ref + return ref() if ref is not None else None def _register_mesh(self, mesh): """ @@ -546,9 +558,10 @@ def _register_mesh(self, mesh): mesh_id = id(mesh) self._meshes[mesh_id] = mesh - # Set as primary mesh if no primary mesh exists - if self._primary_mesh_id is None: + # Set as primary mesh if there is none, or it has been freed + if self.mesh is None: self._primary_mesh_id = mesh_id + self._primary_mesh_ref = weakref.ref(mesh) self.version += 1 @@ -560,6 +573,19 @@ def get_mesh(self, mesh_id: int): """Get a mesh by ID from the model registry""" return self._meshes.get(mesh_id) + def _live_meshes(self): + """The registered meshes, after a collection pass. + + A dropped mesh with variables is a reference cycle (the mesh owns its + variables, each variable holds its mesh), so it stays in the weak + registry until the cyclic collector runs, and that runs at a different + moment on each rank. A traversal that makes a collective call per mesh + (a snapshot's checkpoint writes) takes its list from here, so every + rank visits the same meshes. + """ + gc.collect() + return list(self._meshes.values()) + def set_primary_mesh(self, mesh): """ Set a mesh as the primary mesh for this model. @@ -571,12 +597,11 @@ def set_primary_mesh(self, mesh): """ mesh_id = id(mesh) if mesh_id not in self._meshes: - # Register the mesh first self._register_mesh(mesh) - else: - # Just update the primary mesh pointer - self._primary_mesh_id = mesh_id - self.version += 1 + # always this mesh: registering alone keeps an existing live primary + self._primary_mesh_id = mesh_id + self._primary_mesh_ref = weakref.ref(mesh) + self.version += 1 def _register_swarm(self, swarm): """ @@ -609,7 +634,7 @@ def _unregister_swarm(self, swarm): except ImportError: return names_to_drop = [ - name for name, var in self._variables.items() + name for name, var in list(self._variables.items()) if isinstance(var, SwarmVariable) and getattr(var, "_swarm_ref", None) is not None and var._swarm_ref() is swarm ] @@ -865,7 +890,7 @@ def get_qualified_name(self, variable): str or None : Qualified name if found, None otherwise """ # Search through all registered names - for name, registered_var in self._variables.items(): + for name, registered_var in list(self._variables.items()): if registered_var is variable: return name return None @@ -2388,7 +2413,7 @@ def validate_reference_quantities(self, raise_on_error=False): results = {"valid": True, "warnings": [], "errors": []} # Check all registered variables - for var_name, var in self._variables.items(): + for var_name, var in list(self._variables.items()): # Skip variables without units if not hasattr(var, "units") or var.units is None: continue @@ -5733,14 +5758,14 @@ def describe(self, depth=2): facts["variables"] = len(self._variables) facts["swarms"] = len(self._swarms) facts["solvers"] = len(self._solvers) + sum( - 1 for obj in getattr(self, "_part_objects", {}).values() - if not any(obj is s for s in self._solvers.values())) + 1 for obj in list(getattr(self, "_part_objects", {}).values()) + if not any(obj is s for s in list(self._solvers.values()))) if self.materials: facts["materials"] = list(self.materials.keys()) children = [] if depth > 0: held = list(self._meshes.values()) + list(self._swarms.values()) + list(self._solvers.values()) - for obj in getattr(self, "_part_objects", {}).values(): + for obj in list(getattr(self, "_part_objects", {}).values()): if not any(obj is h for h in held): held.append(obj) for obj in held: diff --git a/src/underworld3/mpi.py b/src/underworld3/mpi.py index f9d12b140..97761d3d2 100644 --- a/src/underworld3/mpi.py +++ b/src/underworld3/mpi.py @@ -76,6 +76,26 @@ def barrier(): comm.Barrier() +def _free_petsc_garbage(comm=None): + """Destroy the PETSc objects whose Python wrappers have been collected. + + COLLECTIVE over ``comm`` (``COMM_WORLD`` when ``None``): every rank of + it must call it at the same point; a mesh on ``COMM_SELF`` passes that. In parallel + petsc4py does not destroy an object when Python collects its wrapper -- + ranks collect in different orders and the destroy is collective -- but + puts it on a deferred list that only ``PETSc.garbage_cleanup`` empties. + Without a call, every DM, matrix and vector of a dropped mesh lives for + the rest of the process (measured: 52.7 MB per faulted-mesh rebuild at + np = 2 with every Python object freed). The local collection first puts + anything just dropped on the list. + """ + import gc + from petsc4py import PETSc + + gc.collect() + PETSc.garbage_cleanup(comm) + + def _selected_ranks(rank_selector, total_size): """The ranks of a ``total_size`` communicator that ``rank_selector`` selects. diff --git a/src/underworld3/swarm.py b/src/underworld3/swarm.py index 478c14807..6d0a47877 100644 --- a/src/underworld3/swarm.py +++ b/src/underworld3/swarm.py @@ -1366,10 +1366,17 @@ def _create_proxy_variable(self): units=self._units, ) # The remesh helper calls this on REINIT vars after an - # adapt. Bound here so the closure captures ``self`` (the - # SwarmVariable) rather than the proxy MeshVariable. - self._meshVar._remesh_reinit_callback = ( - lambda _self=self: setattr(_self, "_proxy_stale", True)) + # adapt. It holds the SwarmVariable weakly: the proxy's mesh + # owns the proxy, and a strong capture made that mesh keep the + # variable, its swarm's next mesh, and that mesh's solvers. + variable_ref = weakref.ref(self) + + def _mark_proxy_stale(): + variable = variable_ref() + if variable is not None: + variable._proxy_stale = True + + self._meshVar._remesh_reinit_callback = _mark_proxy_stale def _update(self): """ @@ -3351,8 +3358,6 @@ def __init__(self, mesh, recycle_rate=0, verbose=False, clip_to_mesh=True): self.verbose = verbose self._clip_to_mesh = clip_to_mesh - # Store reference to model instead of direct mesh reference - # This enables dynamic mesh handover while maintaining access to mesh services import underworld3 as uw model = uw.get_default_model() @@ -3360,8 +3365,11 @@ def __init__(self, mesh, recycle_rate=0, verbose=False, clip_to_mesh=True): # Register mesh with model if not already present model._register_mesh(mesh) - # Store reference to this swarm's specific mesh for proxy operations - self._mesh_id = id(mesh) + # The swarm holds its mesh. Found by id in the model's weak registry, + # a mesh nobody else held was freed under the swarm at the next + # collection, and an unrelated later mesh at a reused address could + # answer for it. + self._mesh = mesh self._model_ref = weakref.ref(model) self.dim = mesh.dim @@ -3776,10 +3784,7 @@ def _route_by_nearest_centroid(self): @property def mesh(self): """The mesh this swarm operates on""" - model = self._model_ref() - if model is None: - raise RuntimeError("Model has been garbage collected") - return model.get_mesh(self._mesh_id) + return self._mesh @mesh.setter def mesh(self, new_mesh): @@ -3803,7 +3808,7 @@ def mesh(self, new_mesh): # Register new mesh with model model._register_mesh(new_mesh) - if id(new_mesh) == self._mesh_id: + if new_mesh is self._mesh: # Check if swarm is already compatible with target mesh if self.dim == new_mesh.dim and self.cdim == new_mesh.cdim: # Dimensions match, check if proxy variables need updating @@ -3843,7 +3848,7 @@ def mesh(self, new_mesh): model._update_mesh_for_swarm(self, new_mesh) # Update swarm's mesh reference - self._mesh_id = id(new_mesh) + self._mesh = new_mesh # Update swarm's cached dimensions self.dim = new_mesh.dim diff --git a/src/underworld3/systems/solvers.py b/src/underworld3/systems/solvers.py index fcd7fe7fb..738c5a570 100644 --- a/src/underworld3/systems/solvers.py +++ b/src/underworld3/systems/solvers.py @@ -551,7 +551,12 @@ def stress_transport(self) -> str: swarm of material points the solver creates and advects, reading the constitutive flux at the particles each step and never projecting it back to the mesh: no numerical diffusion of the history, at the cost of - the swarm (see :class:`~underworld3.systems.ddt.Lagrangian`). The default + the swarm (see :class:`~underworld3.systems.ddt.Lagrangian`). ``"none"`` + keeps the stress history on the grid and never transports it: the + choice when the displacement per step is far below a cell (a tidal or + seismic cycle), where every transport is a numerical cost with no + physics in it, and where a traced-back query across a split-node fault + can read the other side of the slit. The default is ``"semi_lagrangian"``. Set it before the constitutive model is assigned: assigning the model creates the history, and the choice cannot change after that. @@ -562,10 +567,10 @@ def stress_transport(self) -> str: def stress_transport(self, value): value = str(value) if value not in ("semi_lagrangian", "integration_point", "forward", - "lagrangian", "eulerian"): + "lagrangian", "eulerian", "none"): raise ValueError( "stress_transport must be 'semi_lagrangian', 'integration_point', " - f"'forward', 'lagrangian' or 'eulerian', not {value!r}.") + f"'forward', 'lagrangian', 'eulerian' or 'none', not {value!r}.") if self.Unknowns.DFDt is not None: raise RuntimeError( "the stress history already exists: set stress_transport before the " @@ -747,17 +752,18 @@ def _create_stress_history_ddt(self, order=2): proxy_location="cells", **lag_common, ) - elif self.stress_transport == "eulerian": + elif self.stress_transport in ("eulerian", "none"): if ddt_kwargs: raise NotImplementedError( f"{type(cm).__name__} asks its stress history for " f"{sorted(ddt_kwargs)}, which only the semi-Lagrangian flavour " "provides; use stress_transport='semi_lagrangian' for it.") + # "none": the same grid store, with the transport switched off self.Unknowns.DFDt = uw.systems.ddt.EulerianSUPG( self.mesh, sympy.Matrix.zeros(self.mesh.cdim, self.mesh.cdim), self.u.sym, - transport_on_update=True, + transport_on_update=(self.stress_transport == "eulerian"), **common, ) else: @@ -2797,8 +2803,11 @@ def _check_velocity_preconditioner(self): if getattr(self, "_pc_user_override", False): return # the user owns this block's pc_type # The rotated free-slip path builds its OWN fieldsplit KSP and leaves - # this one un-set-up: there is nothing here to inspect. - if getattr(self, "_rotated_freeslip_bcs", None): + # this one un-set-up: there is nothing here to inspect. A fault contact + # (split-node pairs) takes the same rotated loop with no free-slip wall + # registered, so it is skipped by its own marker. + if (getattr(self, "_rotated_freeslip_bcs", None) + or getattr(self, "_fault_contact_faults", None)): return # A split that is not set up has no blocks -- after a rebuild, or a warm # solve that converged before any linear solve -- and asking for them diff --git a/src/underworld3/systems/thin_sheet.py b/src/underworld3/systems/thin_sheet.py index 81cf5e973..a68d7c726 100644 --- a/src/underworld3/systems/thin_sheet.py +++ b/src/underworld3/systems/thin_sheet.py @@ -448,6 +448,33 @@ def basal_drag(self, value): self._needs_function_rewire = True self._basal_drag.sym = value + @property + def normal_velocity(self): + r"""Prescribed velocity of the sheet along its own normal, :math:`w` + (a number or an expression of position; time enters through a + constant the caller updates). On a manifold it is the datum of the + tangency constraint, :math:`\mathbf{u}\cdot\hat{\mathbf{n}} = w` at + every node, so the embedded velocity carries the normal motion and the + in-plane strain rate picks up its curvature term + :math:`w\,\nabla_s\hat{\mathbf{n}}` (:math:`(w/R)\,\mathbf{P}` on a + sphere): the breathing of a tidally flexed shell, say. ``None`` keeps + the sheet in its tangent plane. Re-evaluated at every solve.""" + from underworld3.utilities.rotated_bc import SHEET_TANGENCY + return self._rotated_freeslip_datum.get(SHEET_TANGENCY) + + @normal_velocity.setter + def normal_velocity(self, w): + from underworld3.utilities.rotated_bc import SHEET_TANGENCY + if self.mesh.dim == self.mesh.cdim: + raise NotImplementedError( + "a flat sheet has no normal direction to move along; the normal " + "velocity is a manifold sheet's datum (ThinSheet on a SphericalManifold).") + if w is None: + self._rotated_freeslip_datum.pop(SHEET_TANGENCY, None) + else: + self._rotated_freeslip_datum[SHEET_TANGENCY] = ( + w if isinstance(w, (int, float)) else sympy.sympify(w)) + def add_basal_drag(self, conds, boundary=None, reference_velocity=None): r"""Couple the sheet to its base by a drag :math:`-c\,(\mathbf{u} - \mathbf{u}_\mathrm{ref})` per unit area. diff --git a/src/underworld3/units.py b/src/underworld3/units.py index 089fab46b..185b7fb03 100644 --- a/src/underworld3/units.py +++ b/src/underworld3/units.py @@ -372,7 +372,7 @@ def _extract_units_from_sympy_expression(expr): continue # Find which variable this symbol belongs to - for var_name, variable in model._variables.items(): + for var_name, variable in list(model._variables.items()): if hasattr(variable, "sym") and hasattr(variable, "has_units"): # Check if this symbol matches any component of the variable if hasattr(variable, "num_components"): diff --git a/src/underworld3/utilities/_plex_topology.pyx b/src/underworld3/utilities/_plex_topology.pyx new file mode 100644 index 000000000..8971ba941 --- /dev/null +++ b/src/underworld3/utilities/_plex_topology.pyx @@ -0,0 +1,184 @@ +# cython: language_level=3, boundscheck=False, wraparound=False +"""Bulk DMPlex topology tables, built in C. + +The fault and add-on machinery asks the same questions of every cell, face or +vertex of a mesh — which vertices close this cell, which cells support this +face — and asked them one point at a time through petsc4py, a Python call per +point. These functions answer them for a whole stratum in one C loop and return +numpy tables, so the logic above them can be written as array operations. + +All of them are rank-local: they read this rank's chart and call no collective. +""" +import numpy as np +cimport numpy as cnp + +from petsc4py.PETSc cimport DM, PetscDM, Section, PetscSection + +cnp.import_array() + +cdef extern from "petsc.h": + ctypedef int PetscInt + ctypedef int PetscErrorCode + ctypedef enum PetscBool: + PETSC_FALSE + PETSC_TRUE + PetscErrorCode DMPlexGetChart(PetscDM, PetscInt *, PetscInt *) + PetscErrorCode DMPlexGetDepthStratum(PetscDM, PetscInt, PetscInt *, PetscInt *) + PetscErrorCode DMPlexGetTransitiveClosure(PetscDM, PetscInt, PetscBool, + PetscInt *, PetscInt **) + PetscErrorCode DMPlexRestoreTransitiveClosure(PetscDM, PetscInt, PetscBool, + PetscInt *, PetscInt **) + PetscErrorCode DMPlexGetSupportSize(PetscDM, PetscInt, PetscInt *) + PetscErrorCode DMPlexGetSupport(PetscDM, PetscInt, const PetscInt **) + PetscErrorCode PetscSectionSetChart(PetscSection, PetscInt, PetscInt) + PetscErrorCode PetscSectionSetDof(PetscSection, PetscInt, PetscInt) + PetscErrorCode PetscSectionSetUp(PetscSection) + PetscErrorCode PetscSectionGetOffset(PetscSection, PetscInt, PetscInt *) + + +cdef inline void _check(PetscErrorCode ierr, str what) except *: + if ierr != 0: + raise RuntimeError(f"PETSc error {ierr} in {what}") + + +def closure_points(DM dm, points, int depth): + """The points of depth ``depth`` in the transitive closure of each of + ``points``, in closure order: an ``(n, k)`` int64 table, ``k`` the largest + count found, short rows padded with -1. ``depth`` 0 gives a cell's or a + face's vertices, 1 a cell's edges.""" + cdef cnp.int64_t[::1] pts = np.ascontiguousarray(points, dtype=np.int64) + cdef Py_ssize_t n = pts.shape[0] + cdef PetscInt dS, dE, size = 0, i, p + cdef PetscInt *closure = NULL + cdef Py_ssize_t r, j, cnt, kmax = 0 + _check(DMPlexGetDepthStratum(dm.dm, depth, &dS, &dE), "DMPlexGetDepthStratum") + # first pass: the row width + for r in range(n): + p = pts[r] + _check(DMPlexGetTransitiveClosure(dm.dm, p, PETSC_TRUE, &size, &closure), + "DMPlexGetTransitiveClosure") + cnt = 0 + for i in range(size): + if dS <= closure[2 * i] < dE: + cnt += 1 + _check(DMPlexRestoreTransitiveClosure(dm.dm, p, PETSC_TRUE, &size, &closure), + "DMPlexRestoreTransitiveClosure") + if cnt > kmax: + kmax = cnt + out = np.full((n, kmax), -1, dtype=np.int64) + cdef cnp.int64_t[:, ::1] o = out + for r in range(n): + p = pts[r] + _check(DMPlexGetTransitiveClosure(dm.dm, p, PETSC_TRUE, &size, &closure), + "DMPlexGetTransitiveClosure") + j = 0 + for i in range(size): + if dS <= closure[2 * i] < dE: + o[r, j] = closure[2 * i] + j += 1 + _check(DMPlexRestoreTransitiveClosure(dm.dm, p, PETSC_TRUE, &size, &closure), + "DMPlexRestoreTransitiveClosure") + return out + + +def supports(DM dm, points): + """The support of each of ``points``: an ``(n, k)`` int64 table, ``k`` the + largest support found (two for the faces of a manifold mesh), short rows + padded with -1.""" + cdef cnp.int64_t[::1] pts = np.ascontiguousarray(points, dtype=np.int64) + cdef Py_ssize_t n = pts.shape[0] + cdef PetscInt size, i, p + cdef const PetscInt *sup = NULL + cdef Py_ssize_t r, kmax = 0 + for r in range(n): + _check(DMPlexGetSupportSize(dm.dm, pts[r], &size), + "DMPlexGetSupportSize") + if size > kmax: + kmax = size + out = np.full((n, kmax), -1, dtype=np.int64) + cdef cnp.int64_t[:, ::1] o = out + for r in range(n): + p = pts[r] + _check(DMPlexGetSupportSize(dm.dm, p, &size), "DMPlexGetSupportSize") + _check(DMPlexGetSupport(dm.dm, p, &sup), "DMPlexGetSupport") + for i in range(size): + o[r, i] = sup[i] + return out + + +def fill_section(Section section, DM dm, points, int ndof): + """Set ``section`` to the chart of ``dm`` with ``ndof`` dofs on each of + ``points`` and none elsewhere, and set it up.""" + cdef cnp.int64_t[::1] pts = np.ascontiguousarray(points, dtype=np.int64) + cdef PetscInt pStart, pEnd + cdef Py_ssize_t r + _check(DMPlexGetChart(dm.dm, &pStart, &pEnd), "DMPlexGetChart") + _check(PetscSectionSetChart(section.sec, pStart, pEnd), "PetscSectionSetChart") + for r in range(pts.shape[0]): + _check(PetscSectionSetDof(section.sec, pts[r], ndof), + "PetscSectionSetDof") + _check(PetscSectionSetUp(section.sec), "PetscSectionSetUp") + + +def section_offsets(Section section, points): + """The offset of each of ``points`` in ``section``: an int64 array.""" + cdef cnp.int64_t[::1] pts = np.ascontiguousarray(points, dtype=np.int64) + out = np.empty(pts.shape[0], dtype=np.int64) + cdef cnp.int64_t[::1] o = out + cdef PetscInt off + cdef Py_ssize_t r + for r in range(pts.shape[0]): + _check(PetscSectionGetOffset(section.sec, pts[r], &off), + "PetscSectionGetOffset") + o[r] = off + return out + + +cdef extern from "petsc.h": + PetscErrorCode PetscSectionGetDof(PetscSection, PetscInt, PetscInt *) + + +def closure_dof_rows(DM dm, Section section, points, int block): + """The rows of each point's closure dofs in a local vector laid out by + ``section``, in closure order, counted in blocks of ``block`` (the + coordinate dimension for a coordinate field, so a row is one node): an + ``(n, k)`` int64 table, ``k`` the largest count, short rows padded with -1. + A point with ``d`` dofs at offset ``o`` contributes rows ``o / block`` to + ``(o + d) / block - 1``.""" + cdef cnp.int64_t[::1] pts = np.ascontiguousarray(points, dtype=np.int64) + cdef Py_ssize_t n = pts.shape[0] + cdef PetscInt size = 0, i, p, q, dof, off + cdef PetscInt *closure = NULL + cdef Py_ssize_t r, j, cnt, kmax = 0, m + for r in range(n): + p = pts[r] + _check(DMPlexGetTransitiveClosure(dm.dm, p, PETSC_TRUE, &size, &closure), + "DMPlexGetTransitiveClosure") + cnt = 0 + for i in range(size): + _check(PetscSectionGetDof(section.sec, closure[2 * i], &dof), + "PetscSectionGetDof") + cnt += dof // block + _check(DMPlexRestoreTransitiveClosure(dm.dm, p, PETSC_TRUE, &size, &closure), + "DMPlexRestoreTransitiveClosure") + if cnt > kmax: + kmax = cnt + out = np.full((n, kmax), -1, dtype=np.int64) + cdef cnp.int64_t[:, ::1] o = out + for r in range(n): + p = pts[r] + _check(DMPlexGetTransitiveClosure(dm.dm, p, PETSC_TRUE, &size, &closure), + "DMPlexGetTransitiveClosure") + j = 0 + for i in range(size): + q = closure[2 * i] + _check(PetscSectionGetDof(section.sec, q, &dof), "PetscSectionGetDof") + if dof: + _check(PetscSectionGetOffset(section.sec, q, &off), + "PetscSectionGetOffset") + for m in range(dof // block): + o[r, j] = off // block + m + j += 1 + _check(DMPlexRestoreTransitiveClosure(dm.dm, p, PETSC_TRUE, &size, &closure), + "DMPlexRestoreTransitiveClosure") + return out diff --git a/src/underworld3/utilities/addon_partition.py b/src/underworld3/utilities/addon_partition.py new file mode 100644 index 000000000..da7e0a0d0 --- /dev/null +++ b/src/underworld3/utilities/addon_partition.py @@ -0,0 +1,803 @@ +r"""One partition for the add-on hierarchy. + +The add-on is the finest mesh built on a static base hierarchy — an ``adapt`` +child, the mesh a fault is about to be cut into or placed into — together with +the add-on levels refined from the base toward the fault. ``adapt`` leaves the +add-on co-partitioned with the base; fault surgery needs the fault's cell star +rank-interior (the split's pairs and the placement's cavity are rank-local by +construction) and so used to move cells on its own terms, once per surgery +and once per level, after which no two levels shared a partition and every +transfer, gather and diagnostic carried its own cross-partition handling. + +:func:`partition_addon` moves the add-on **once, before any surgery**: one +assignment computed on the finest level and applied to every add-on level +through the parent-cell relation, so the add-on levels stay co-partitioned +with each other exactly, the fault regions are rank-interior, and the load is +balanced by diffusion over the seams: any rank heavier than a neighbour passes +it cells that belong to no fault region, the owner's farthest first. The base +levels never move (load fixes go on the child). Every +level carries, per cell, the cell it came from and its parent on the level +below, so the one pair that is no longer co-partitioned — the base's finest +level against the first add-on level — is an exact relation to fetch, not a +search. Design note: +``docs/developer/design/parallel-addon-distribution-2026-10.md``. + +The unit of assignment is a cell of the coarsest add-on level with every +descendant it has on the finer levels (the finest mesh itself when there is +no add-on level below it). All the cells of a unit go to one rank. +""" +import numpy as np +from mpi4py import MPI +from petsc4py import PETSc + +import underworld3 as uw +# A fine cell's centroid lies inside its parent for nested refinement; the +# tolerance is the cell locator's own, so a point it accepts is accepted here. +from underworld3.utilities.fault_mg import _INSIDE_TOL + +__all__ = ["partition_addon", "mark_labels", "mark_polylines", "mark_sheets", + "mark_band"] + + + +def _allreduce(comm, values, op=MPI.SUM): + """Reduce a float array over the communicator (buffer-based; the + pickling form applies the operator to Python objects).""" + out = np.array(values, dtype=float) + comm.Allreduce(MPI.IN_PLACE, out, op=op) + return out + + +# --------------------------------------------------------------------------- # +# Marks: which vertices the fault touches +# --------------------------------------------------------------------------- # +def _mark_facet_vertices(dm, facets, value, ids): + """Raise ``ids`` (chart-length) to at least ``value`` on every vertex of + the given facets (points of the height-1 stratum; others are skipped).""" + from underworld3.utilities._plex_topology import closure_points + fS, fE = dm.getHeightStratum(1) + pStart, _pEnd = dm.getChart() + facets = np.asarray(facets, dtype=np.int64) + facets = facets[(facets >= fS) & (facets < fE)] + if facets.size: + v = closure_points(dm, facets, 0) + v = v[v >= 0] - pStart + np.maximum.at(ids, v, value) + + +def mark_labels(labels, groups=None): + """The marking rule for labelled fault facets: region ``k`` on every vertex + of a facet of group ``k`` (one group per fault when ``groups`` is None). + + ``labels`` is a sequence of ``(name, value)`` pairs; ``groups`` a list of + lists of those pairs, the faults that must share a rank. Returns the + function of a level's DM that :func:`partition_addon` takes as + ``region_marks``.""" + if groups is None: + groups = [[lab] for lab in labels] + + def marks(dm): + pStart, pEnd = dm.getChart() + ids = np.zeros(pEnd - pStart, dtype=np.int32) + for g, group in enumerate(groups, start=1): + for name, value in group: + if (dm.hasLabel(name) + and dm.getLabel(name).getStratumSize(int(value)) > 0): + _mark_facet_vertices( + dm, dm.getLabel(name).getStratumIS(int(value)).getIndices(), + g, ids) + return ids + return marks + + +def _segment_distance(X, poly): + """Distance from each point of ``X`` (N, 2) to the polyline ``poly`` (M, 2).""" + best = np.full(X.shape[0], np.inf) + for a, b in zip(poly[:-1], poly[1:]): + ab = b - a + t = np.clip(((X - a) @ ab) / max(float(ab @ ab), 1e-300), 0.0, 1.0) + d = np.linalg.norm(X - (a + t[:, None] * ab), axis=1) + best = np.minimum(best, d) + return best + + +def mark_polylines(polylines): + """The marking rule for 2-D line cuts: region ``k`` on every vertex within + one local cell diameter of polyline ``k`` — the vertices the cut may pull + onto the trace and the corners of every cell it crosses, at the level's + own resolution.""" + polylines = [np.asarray(poly, dtype=float)[:, :2] for poly in polylines] + + def marks(dm): + from underworld3.utilities._plex_topology import closure_points + from underworld3.utilities.edge_split import cell_diameters + from underworld3.utilities.place_surface import _propagate_vertex + vS, vE = dm.getDepthStratum(0) + cS, cE = dm.getHeightStratum(0) + pStart, pEnd = dm.getChart() + X = dm.getCoordinatesLocal().array.reshape( + -1, dm.getCoordinateDim())[: vE - vS] + h_chart = np.zeros(pEnd - pStart) + if cE > cS: + h_cell = np.asarray(cell_diameters(dm)) + CV = closure_points(dm, np.arange(cS, cE), 0) - pStart + np.maximum.at(h_chart, CV, np.broadcast_to(h_cell[:, None], CV.shape)) + # a seam vertex sees only its local cells: the largest incident cell + # over every rank, so the reach is a function of the mesh alone + h_chart = _propagate_vertex(dm, h_chart, MPI.MAX, np.maximum) + h_vertex = h_chart[vS - pStart: vE - pStart] + ids = np.zeros(pEnd - pStart, dtype=np.int32) + for k, poly in enumerate(polylines, start=1): + near = np.flatnonzero(_segment_distance(X[:, :2], poly) <= h_vertex) + ids[near + vS - pStart] = np.maximum(ids[near + vS - pStart], k) + return ids + return marks + + +def mark_sheets(sheets, clearance): + """The marking rule for 3-D sheet placements: the placement's own mask — + region ``k`` on every vertex within ``(clearance + 1)`` local cell + diameters of sheet ``k`` (``(points, triangles)`` pairs) — so the carve, + its growth rounds and the split all fall inside the region once the + growth layers are added.""" + sheets = [(np.asarray(pts, dtype=float), np.asarray(tri, dtype=np.int64)) + for pts, tri in sheets] + + def marks(dm): + from underworld3.utilities.place_surface import ( + _coords, _sheet_distance_within, _tet_vertices, _vertex_h_3d) + vS, vE = dm.getDepthStratum(0) + pStart, pEnd = dm.getChart() + X = _coords(dm)[: vE - vS] + cells = _tet_vertices(dm) + h_vertex, _h_cell = _vertex_h_3d(dm, cells, len(X)) + reach = (float(clearance) + 1.0) * h_vertex + ids = np.zeros(pEnd - pStart, dtype=np.int32) + for k, (points, triangles) in enumerate(sheets, start=1): + d = _sheet_distance_within(X, points, triangles, reach) + near = np.flatnonzero(d < reach) + ids[near + vS - pStart] = np.maximum(ids[near + vS - pStart], k) + return ids + return marks + + +def mark_band(paths, width, clearance): + """The marking rule for a placed fault band (``place_thin_volume``): a + superset of the placement's own mask, so its gather finds the region + rank-interior. The placement marks every vertex within + ``max(clearance h, 0.6 width) + h`` of the band's skin (``h`` the + smallest incident cell diameter); the skin lies within a mitred half + width of the band's path, so a vertex within that reach plus one band + width of the path is marked here. ``paths`` are the paths the band is + built along: ``(n, 2)`` polylines in 2-D, planar convex ``(n, 3)`` + polygons in 3-D (the patches).""" + paths = [np.asarray(P, dtype=float) for P in paths] + + def marks(dm): + from underworld3.utilities.place_surface import ( + _coords, _sheet_distance_within, _tet_vertices, _vertex_h_3d) + vS, vE = dm.getDepthStratum(0) + pStart, pEnd = dm.getChart() + X = _coords(dm)[: vE - vS] + h_vertex, _h_cell = _vertex_h_3d(dm, _tet_vertices(dm) if dm.getDimension() == 3 + else _cell_vertices(dm), len(X)) + reach = (np.maximum(float(clearance) * h_vertex, 0.6 * float(width)) + + h_vertex + float(width)) + ids = np.zeros(pEnd - pStart, dtype=np.int32) + for k, P in enumerate(paths, start=1): + if dm.getDimension() == 2: + d = _segment_distance(X[:, :2], P[:, :2]) + else: + fan = np.array([(0, i, i + 1) for i in range(1, len(P) - 1)], + dtype=np.int64) + d = _sheet_distance_within(X, P, fan, reach) + near = np.flatnonzero(d < reach) + ids[near + vS - pStart] = np.maximum(ids[near + vS - pStart], k) + return ids + return marks + + +def _cell_vertices(dm): + """(n_cells, dim + 1) local vertex indices of every simplex.""" + from underworld3.utilities._plex_topology import closure_points + vS, _vE = dm.getDepthStratum(0) + cS, cE = dm.getHeightStratum(0) + return (closure_points(dm, np.arange(cS, cE), 0) - vS).reshape( + cE - cS, dm.getDimension() + 1) + +def embedded_surface_names(mesh): + """The fault surfaces ``mesh`` already carries, as boundary names: every + split fault (its own label and its ``Plus`` / ``Minus`` sides) and every + sheet placed into it or an ancestor (``_embedded_surfaces``, recorded by + the placement). These are glued by every later partition, so a second + fault or sheet never breaks up the first. A material interface or a + domain-spanning internal boundary is not among them.""" + names = set(getattr(mesh, "_embedded_surfaces", ()) or ()) + for name in (getattr(mesh, "_fault_point_pairs", {}) or {}): + names.update((name, f"{name}Plus", f"{name}Minus")) + return names + + +def _embedded_surface_marks(mesh, names, first_id): + """Chart-length ids gluing the named surfaces: ``first_id + j`` on every + vertex of a facet of the ``j``-th name (sorted). ``names`` is the same on + every rank, so the ids are.""" + dm = mesh.dm + pStart, pEnd = dm.getChart() + ids = np.zeros(pEnd - pStart, dtype=np.int32) + present = {b.name: int(b.value) for b in mesh.boundaries} + for j, name in enumerate(names): + if name not in present or not dm.hasLabel(name): + continue + lab = dm.getLabel(name) + if lab.getStratumSize(present[name]) > 0: + _mark_facet_vertices(dm, lab.getStratumIS(present[name]).getIndices(), + first_id + j, ids) + return ids + + +def inherit_provenance(child, host): + """Record on ``child`` — a mesh made from ``host`` by a cut, a placement or + a split — the parent of each of its cells on the level ``host`` records + them against, while the two geometries still agree. The surgery was + rank-local (the partition made the region interior), so every child + centroid lies in a local host cell. Recording it here, rather than + locating later, keeps the relation exact after the child's nodes move. + A no-op when ``host`` carries no record (serial, or never partitioned). + Collective.""" + from underworld3.utilities.fault_mg import locate_in_cells_exhaustive + parent = getattr(host, "_addon_parent", None) + if parent is None or getattr(host, "_addon_below", None) is None: + return + comm = child.dm.getComm().tompi4py() + fcent, _v, cvf = _cell_centroids(child) + _c, vxy_h, cv_h = _cell_centroids(host) + cell = np.zeros(cvf.shape[0], dtype=np.int64) + n_bad = 0 + if cvf.shape[0]: + if cv_h.shape[0] == 0: + n_bad = cvf.shape[0] + else: + cell, lam = locate_in_cells_exhaustive(fcent, vxy_h, cv_h) + n_bad = int((lam.min(axis=1) < -_INSIDE_TOL).sum()) + n_bad = int(comm.allreduce(n_bad)) + if n_bad: + raise RuntimeError( + f"addon_partition: {n_bad} cell(s) of a surgery's result lie in no " + f"local cell of the mesh it was made from; the surgery was not " + f"rank-local.") + child._addon_parent = np.asarray(parent, dtype=np.int64)[cell] + child._addon_below = host._addon_below + + +# --------------------------------------------------------------------------- # +# The parent relation and the units +# --------------------------------------------------------------------------- # +def _cell_centroids(mesh): + from underworld3.utilities.fault_mg import _cell_vertex_table + vxy, cv = _cell_vertex_table(mesh) + return vxy[cv].mean(axis=1), vxy, cv + + +def _parent_cells_local(coarse, fine): + """``parent[k]``: the local cell of ``coarse`` holding the centroid of local + cell ``k`` of ``fine``. Exact for nested refinement and for a grid that + re-represents the coarse one; raises if any centroid falls outside every + local coarse cell, which means the two levels are not co-partitioned — + the add-on has already been moved and must not be partitioned twice.""" + from underworld3.utilities.fault_mg import locate_in_cells_exhaustive + comm = fine.dm.getComm().tompi4py() + fcent, _vf, cvf = _cell_centroids(fine) + _cc, vxy_c, cv_c = _cell_centroids(coarse) + n_f, n_c = cvf.shape[0], cv_c.shape[0] + cell = np.full(n_f, -1, dtype=np.int64) + if n_f and n_c: + cell, lam = locate_in_cells_exhaustive(fcent, vxy_c, cv_c) + cell = np.where(lam.min(axis=1) < -_INSIDE_TOL, -1, cell).astype(np.int64) + n_bad = int(comm.allreduce(int((cell < 0).sum()) + (n_f > 0 and n_c == 0))) + if n_bad: + raise RuntimeError( + f"addon_partition: {n_bad} fine cell(s) have no local parent cell " + f"on the level below; the add-on levels are not co-partitioned " + f"(an add-on is partitioned once, as adapt made it).") + return cell + + +def _base_parent_addresses(base, lev, comm): + """``(n_cells, 2)``: rank and local cell of each cell's parent on ``base``. + + Located in this rank's base cells while the two are co-partitioned (as + ``adapt`` leaves them). Once a partition has moved the add-on, the pair is + not co-partitioned any more and the relation is READ, not searched: the + mesh that recorded it (``fault_mg.provenance_source`` — ``lev`` itself, or + the level a cut or a placement re-represented) is located locally, the + surgery having been rank-local, and its recorded parent is taken. That is + what lets a second sheet, or a fault after a material interface, be + partitioned on top of a first.""" + from underworld3.utilities.fault_mg import (locate_in_cells_exhaustive, + provenance_source) + pre = provenance_source(base, lev) + if pre is None: + local = _parent_cells_local(base, lev) + return np.column_stack([np.full(local.shape[0], comm.rank, + dtype=np.int64), local]) + if pre is lev: + return np.asarray(lev._addon_parent, dtype=np.int64).copy() + fcent, _v, cvf = _cell_centroids(lev) + _c, vxy_p, cv_p = _cell_centroids(pre) + cell = np.zeros(cvf.shape[0], dtype=np.int64) + n_bad = 0 + if cvf.shape[0]: + if cv_p.shape[0] == 0: + n_bad = cvf.shape[0] + else: + cell, lam = locate_in_cells_exhaustive(fcent, vxy_p, cv_p) + n_bad = int((lam.min(axis=1) < -_INSIDE_TOL).sum()) + n_bad = int(comm.allreduce(n_bad)) + if n_bad: + raise RuntimeError( + f"addon_partition: {n_bad} cell(s) lie in no local cell of the " + f"level their provenance is recorded on; the surgery since the " + f"last partition was not rank-local.") + return np.asarray(pre._addon_parent, dtype=np.int64)[cell] + + +def _levels_of(mesh): + """``(base, static, addon)``: the static level the first add-on level's + transfer reads from (or None), the static wraps that form the tail under + the add-on, and the add-on levels coarsest first, excluding ``mesh``.""" + from underworld3.utilities.fault_mg import addon_level_indices + tail = getattr(mesh, "_custom_mg_coarse_meshes", None) + if tail is not None: + idx = set(addon_level_indices(tail)) + static = [m for k, m in enumerate(tail) if k not in idx] + addon = [m for k, m in enumerate(tail) if k in idx] + return (static[-1] if static else None), static, addon + # a mesh built with refinement >= 1: the split's rule makes the coarse + # levels below the finest the tail, the finest wrap being this very grid + wraps = (list(mesh._coarse_level_meshes())[:-1] + if len(getattr(mesh, "dm_hierarchy", []) or []) > 1 else []) + return (wraps[-1] if wraps else None), wraps, [] + + +# --------------------------------------------------------------------------- # +# Seam neighbours +# --------------------------------------------------------------------------- # +def _face_other_rank(dm): + """Per local face, the rank holding the cell across the seam, or -1 for a + face interior to this rank's cells. A shared face has one root and one + leaf: the leaf side reads the root's rank off the forest and the root + side gets the leaf's rank by one reduction.""" + fS, fE = dm.getHeightStratum(1) + pStart, pEnd = dm.getChart() + other = np.full(pEnd - pStart, -1, dtype=np.int64) + comm = dm.getComm().tompi4py() + if comm.size == 1: + return other[fS - pStart: fE - pStart] + sf = dm.getPointSF() + try: + nroots, ilocal, iremote = sf.getGraph() + except (ValueError, TypeError): + # Sanctioned: petsc4py raises on a forest with no graph set, which is + # a rank that shares no point; every face is then interior. + return other[fS - pStart: fE - pStart] + leaf = np.full(pEnd - pStart, -1, dtype=np.int64) + if ilocal is not None and len(ilocal): + leaf[np.asarray(ilocal)] = comm.rank + other[np.asarray(ilocal)] = np.asarray(iremote)[:, 0] + root = np.full(pEnd - pStart, -1, dtype=np.int64) + sf.reduceBegin(MPI.INT64_T, leaf, root, MPI.MAX) + sf.reduceEnd(MPI.INT64_T, leaf, root, MPI.MAX) + has_leaf = root >= 0 + other[has_leaf] = root[has_leaf] + return other[fS - pStart: fE - pStart] + + +def _unit_adjacency(dm, unit_of_cell, face_other): + """The face adjacency of the units: ``(ua, ub)`` pairs of local units that + share a face, and ``(u, rank)`` pairs of local units with a face on the + seam to ``rank``. Computed once; the assignment changes, the mesh does + not.""" + from underworld3.utilities._plex_topology import supports + cS, cE = dm.getHeightStratum(0) + fS, fE = dm.getHeightStratum(1) + sup = supports(dm, np.arange(fS, fE)) + if sup.shape[1] < 2: + sup = np.column_stack([sup, np.full((sup.shape[0], 2 - sup.shape[1]), -1)]) + both = (sup[:, 0] >= 0) & (sup[:, 1] >= 0) + ua = unit_of_cell[sup[both, 0] - cS] + ub = unit_of_cell[sup[both, 1] - cS] + inner = np.column_stack([ua, ub])[ua != ub].astype(np.int64) + one = (sup[:, 0] >= 0) & (sup[:, 1] < 0) & (face_other >= 0) + seam = np.column_stack([unit_of_cell[sup[one, 0] - cS], + face_other[one]]).astype(np.int64) + return inner.reshape(-1, 2), seam.reshape(-1, 2) + + +def _units_touching(inner, seam, assign_unit, n_ranks): + """``(unit, rank)`` pairs: every local unit with the ranks it touches + across a face, under the current assignment, each pair once. A pair is + encoded as ``unit * n_ranks + rank`` so the deduplication is a 1-D sort.""" + a, b = inner[:, 0], inner[:, 1] + keys = np.unique(np.concatenate([ + a * n_ranks + assign_unit[b], + b * n_ranks + assign_unit[a], + seam[:, 0] * n_ranks + seam[:, 1]])) + return np.column_stack([keys // n_ranks, keys % n_ranks]) + + +# --------------------------------------------------------------------------- # +# Balance by diffusion +# --------------------------------------------------------------------------- # +def _peel_balance(dm, unit_of_cell, assign_unit, weight, unit_region, + unit_centroid, region_centres, tolerance, comm, + max_sweeps=100): + """Balance the load by diffusion over the seams: in each sweep a rank + offers free units (no fault region) across each face it shares with a + lighter neighbour, up to half the difference in load, the cells farthest + from the faults first; a receiver's intake is scaled so it never exceeds + the largest half-gap offered to it. A rank at the mean passes cells on, + which is what carries load around a fault region that walls its owner + off from the lightest rank. Stops when every rank is within + ``tolerance`` of the mean, when nothing can move, or at ``max_sweeps``. + Collective in every sweep. Returns the assignment and the sweep count.""" + size = comm.size + face_other = _face_other_rank(dm) + inner, seam = _unit_adjacency(dm, unit_of_cell, face_other) + far = (np.full(assign_unit.shape[0], np.inf) if region_centres.shape[0] == 0 + else np.min(np.linalg.norm( + unit_centroid[:, None, :] - region_centres[None, :, :], axis=2), + axis=1)) + free = unit_region == 0 + n_sweeps = 0 + for _sweep in range(max_sweeps): + load = _allreduce(comm, np.bincount(assign_unit, weights=weight, + minlength=size)) + mean = load.sum() / size + if load.max() <= mean * (1.0 + tolerance): + break + n_sweeps += 1 + touching = _units_touching(inner, seam, assign_unit, size) + mine = free & (assign_unit == comm.rank) + touching = touching[mine[touching[:, 0]]] + proposals = {} # receiver -> list of (unit, weight) + gap = np.zeros(size) # my half-gap to each receiver + lighter = np.flatnonzero(load < load[comm.rank] - tolerance * mean) + taken = np.zeros(assign_unit.shape[0], dtype=bool) + for s in lighter[np.argsort(load[lighter], kind="stable")]: + s = int(s) + if s == comm.rank: + continue + cand = touching[touching[:, 1] == s, 0] + cand = cand[~taken[cand]] + if cand.size == 0: + continue + cand = cand[np.lexsort((cand, -far[cand]))] + room = 0.5 * (load[comm.rank] - load[s]) + gap[s] = room + chosen = [] + for u in cand: + # a unit heavier than twice the room left would overshoot the + # gap and could be traded back next sweep + if room < 0.5 * weight[u]: + continue + chosen.append((int(u), float(weight[u]))) + room -= weight[u] + taken[u] = True + proposals[s] = chosen + offered = np.zeros(size) + for s, chosen in proposals.items(): + offered[s] = sum(w for _u, w in chosen) + offered_all = _allreduce(comm, offered) + gap_all = _allreduce(comm, gap, op=MPI.MAX) + moved = 0 + for s, chosen in proposals.items(): + scale = (min(1.0, gap_all[s] / offered_all[s]) + if offered_all[s] > 0 else 0.0) + budget = scale * offered[s] + for u, w in chosen: + if budget <= 0: + break + assign_unit[u] = s + budget -= w + moved += 1 + if comm.allreduce(moved) == 0: + break + return assign_unit, n_sweeps + + +# --------------------------------------------------------------------------- # +# Provenance through the move +# --------------------------------------------------------------------------- # +def _resolve_parents(below, lev): + """Give level ``lev`` its parents' CURRENT addresses. + + Every cell of ``lev`` knows its parent on ``below`` as it was before the + move (``_addon_parent_old``: rank, local cell); every cell of ``below`` + knows where it was (``_addon_self_old``). A directory on the old owners + (one reduction) maps old to new, and one broadcast hands each fine cell + its parent's new address. Stored as ``lev._addon_parent`` (n_cells, 2): + rank and local cell on ``below`` as it stands now.""" + comm = lev.dm.getComm() + m4 = comm.tompi4py() + parent_old = lev._addon_parent_old + self_old = getattr(below, "_addon_self_old", None) + if self_old is None: + # the level below did not move: old addresses are current + lev._addon_parent = parent_old.copy() + return + n_old = int(below._addon_n_cells_old) + cS, cE = below.dm.getHeightStratum(0) + unit = MPI.INT64_T.Create_contiguous(2) + unit.Commit() + # directory: old cell -> (new rank, new local) + sf_dir = PETSc.SF().create(comm=comm) + sf_dir.setGraph(n_old, None, np.ascontiguousarray(self_old, dtype=np.int32)) + sf_dir.setUp() + now = np.column_stack([np.full(cE - cS, m4.rank), np.arange(cE - cS)] + ).astype(np.int64) + directory = np.full((n_old, 2), -1, dtype=np.int64) + # exactly one leaf per root, so REPLACE is the reduction (the predefined + # reductions do not take a derived datatype) + sf_dir.reduceBegin(unit, now, directory, MPI.REPLACE) + sf_dir.reduceEnd(unit, now, directory, MPI.REPLACE) + sf_dir.destroy() + # fetch: each fine cell reads its parent's entry + sf_get = PETSc.SF().create(comm=comm) + sf_get.setGraph(n_old, None, + np.ascontiguousarray(parent_old, dtype=np.int32)) + sf_get.setUp() + parent_new = np.full(parent_old.shape, -1, dtype=np.int64) + sf_get.bcastBegin(unit, directory, parent_new, MPI.REPLACE) + sf_get.bcastEnd(unit, directory, parent_new, MPI.REPLACE) + sf_get.destroy() + unit.Free() + if comm.tompi4py().allreduce(int((parent_new < 0).sum())): + raise RuntimeError( + "addon_partition internal: a cell's parent was not found after " + "the move.") + lev._addon_parent = parent_new + + +def _moved_wrap(level, dm, fields, pairs): + """A new Mesh over the redistributed ``dm`` of ``level``, carrying the + provenance fields and whatever fault geometry the level had.""" + moved = uw.discretisation.Mesh( + dm, simplex=level.dm.isSimplex(), qdegree=level.qdegree, + coordinate_system_type=level.CoordinateSystem.coordinate_type, + boundaries=level.boundaries, verbose=False) + moved._mg_static = False + for attr in ("_fault_traces", "_fault_specs", "_fault_surfaces", + "_fault_seam_normals", "_embedded_surfaces"): + if getattr(level, attr, None) is not None: + setattr(moved, attr, getattr(level, attr)) + if pairs is not None: + moved._fault_point_pairs = pairs + parent_rank, parent_local, self_rank, self_local = fields + moved._addon_parent_old = np.column_stack([parent_rank, parent_local]) + moved._addon_self_old = np.column_stack([self_rank, self_local]) + cS, cE = level.dm.getHeightStratum(0) + moved._addon_n_cells_old = cE - cS + return moved + + +# --------------------------------------------------------------------------- # +# The entry point +# --------------------------------------------------------------------------- # +def partition_addon(mesh, region_marks, layers=1, tolerance=0.1, + verbose=False): + """Partition ``mesh`` and its add-on levels for the faults ``region_marks`` + marks. + + Parameters + ---------- + mesh : Mesh + The finest add-on level, as ``adapt`` made it (co-partitioned with + its levels), before any cut, placement or split. + region_marks : callable + ``region_marks(dm)`` returns chart-length region ids on a level's DM: + ``0`` for an unmarked point, ``k >= 1`` the fault region a vertex + belongs to (:func:`mark_polylines`, :func:`mark_sheets`, + :func:`mark_labels`). It is evaluated on EVERY add-on level, because + the same fault's cell star reaches further on a coarser level and the + split replayed there needs its own star rank-interior. Every fault + surface the levels already carry (:func:`embedded_surface_names`) is + glued as well, so an add-on can be partitioned again for a further + fault without breaking up the first. Regions whose stars touch merge. + layers : int + Growth layers beyond the marked vertices' cell star: one makes every + point of a star cell unshared (the split's seam rule); the 3-D carve's + growth rounds need two. + tolerance : float + Balance target: every rank's share of finest cells within this + fraction of the mean, if the seams allow it. + + Returns + ------- + Mesh + A new finest mesh whose add-on tail is moved the same way, with the + provenance of every level recorded (``_addon_parent``); or ``mesh`` + itself when every region is already rank-interior and the load is + within tolerance. Serial: ``mesh``. + """ + from underworld3.utilities.fault_split import redistribute_by_assignment + from underworld3.utilities.place_surface import _merge_regions, _region_cells + + comm = mesh.dm.getComm().tompi4py() + if comm.size == 1: + return mesh + base, static, addon = _levels_of(mesh) + levels = addon + [mesh] + # the add-on levels are co-partitioned with each other (adapt made them + # so, or an earlier partition did): their parents are local. The first + # add-on level's parent on the base is local too unless an earlier + # partition moved it, in which case the recorded address is used + parents = [None] + for l in range(1, len(levels)): + parents.append(_parent_cells_local(levels[l - 1], levels[l])) + base_parent = (None if base is None + else _base_parent_addresses(base, levels[0], comm)) + cS0, cE0 = levels[0].dm.getHeightStratum(0) + n_units = cE0 - cS0 + unit_of = [np.arange(n_units, dtype=np.int64)] + for l in range(1, len(levels)): + unit_of.append(unit_of[l - 1][parents[l]]) + weight = np.bincount(unit_of[-1], minlength=n_units).astype(float) + + # fault regions on every level, lifted to units: the faults asked for, + # and every interior surface the levels already carry (placed or split + # by an earlier call), so a second partition keeps those whole + marks = [np.asarray(region_marks(lev.dm), dtype=np.int32) for lev in levels] + n_asked = int(comm.allreduce(max((int(m.max()) for m in marks if m.size), + default=0), op=MPI.MAX)) + glued = sorted(set().union(*comm.allgather( + set().union(*[embedded_surface_names(lev) for lev in levels])))) + for l, lev in enumerate(levels): + marks[l] = np.maximum(marks[l], + _embedded_surface_marks(lev, glued, n_asked + 1)) + unit_region = np.zeros(n_units, dtype=np.int32) + pairs = set() + for l, lev in enumerate(levels): + cell_region, _canon, _regions = _region_cells( + lev.dm, marks[l], layers=layers) + sel = np.flatnonzero(cell_region) + u, k = unit_of[l][sel], cell_region[sel] + # a unit takes the largest id among its cells' and its own; the ids + # that meet in one unit touch + best = unit_region.copy() + np.maximum.at(best, u, k) + held = np.concatenate([k, unit_region[u]]) + top = np.concatenate([best[u], best[u]]) + meet = (held > 0) & (held != top) + if meet.any(): + for a, b in np.unique(np.column_stack([held[meet], top[meet]]), + axis=0).tolist(): + pairs.add((int(a), int(b))) + unit_region = best + n_ids = int(comm.allreduce(int(unit_region.max()) if n_units else 0, + op=MPI.MAX)) + canon = _merge_regions(pairs, n_ids, comm) + lookup = np.zeros(n_ids + 1, dtype=np.int32) + for key, value in canon.items(): + lookup[key] = value + unit_region = lookup[unit_region] + regions = sorted(set(canon.values())) + local = np.array([float(weight[unit_region == k].sum()) for k in regions]) + counts = np.asarray(comm.allgather(local)).reshape(comm.size, len(regions)) + # an id no level marked anywhere (a fault outside the mesh, a glued + # surface absent from these levels) is not a region + present = counts.sum(axis=0) > 0 + regions = [k for k, keep in zip(regions, present) if keep] + counts = counts[:, present] + assign = np.full(n_units, comm.rank, dtype=np.int32) + owner = {} + for j, k in enumerate(regions): + col = counts[:, j] + owner[k] = int(np.argmax(col)) + if np.count_nonzero(col) > 1: + assign[unit_region == k] = owner[k] + + # the fault owners shed far cells to their underloaded neighbours + unit_centroid, _v, _cv = _cell_centroids(levels[0]) + centres = [] + for j, k in enumerate(regions): + sel = unit_region == k + s = (unit_centroid[sel] * weight[sel, None]).sum(axis=0) + tot = _allreduce(comm, np.concatenate([s, [weight[sel].sum()]])) + centres.append(tot[:-1] / tot[-1]) + centres = np.asarray(centres).reshape(len(regions), -1) + load_before = _allreduce(comm, np.bincount(assign, weights=weight, + minlength=comm.size)) + assign, n_sweeps = _peel_balance(levels[0].dm, unit_of[0], assign, weight, + unit_region, unit_centroid, centres, + tolerance, comm) + load_after = _allreduce(comm, np.bincount(assign, weights=weight, + minlength=comm.size)) + n_moved_units = int(comm.allreduce(int((assign != comm.rank).sum()))) + record = { + "regions": len(regions), "owners": owner, + "region_weight": {k: float(counts[:, j].sum()) + for j, k in enumerate(regions)}, + "units_moved": n_moved_units, "sweeps": n_sweeps, + "load_before": load_before.tolist(), "load_after": load_after.tolist(), + } + if n_moved_units == 0: + # already as the partition would leave it: the mesh stands, with the + # record of the decision + mesh._addon_partition = record + if verbose: + uw.pprint(f"[addon_partition] {len(regions)} fault region(s) " + f"already rank-interior, load within tolerance " + f"(max/mean {load_after.max() / load_after.mean():.2f}); " + f"nothing moves") + return mesh + + # Every unit has a cell on every level, so a rank that receives no unit + # holds nothing on any level. A DM with every point on one rank reads as + # not distributed, and the Mesh constructor would partition it afresh, + # which the provenance cannot ride: refuse instead (collectively). + owners_used = int(np.count_nonzero(_allreduce( + comm, np.bincount(assign, minlength=comm.size)))) + if owners_used < 2: + raise RuntimeError( + "addon_partition: the whole add-on would sit on one rank (its " + "fault regions cover it); run on fewer ranks or refine the base.") + + # apply the one assignment to every level, provenance riding along + moved = [] + for l, lev in enumerate(levels): + cS, cE = lev.dm.getHeightStratum(0) + n = cE - cS + a_level = assign[unit_of[l]] + if l > 0: + parent_rank = np.full(n, comm.rank, dtype=np.int64) + parent_local = parents[l] + elif base_parent is not None: + parent_rank, parent_local = base_parent[:, 0], base_parent[:, 1] + else: + parent_rank = np.full(n, -1, dtype=np.int64) + parent_local = np.full(n, -1, dtype=np.int64) + self_rank = np.full(n, comm.rank, dtype=np.int64) + self_local = np.arange(n, dtype=np.int64) + fields = [parent_rank, parent_local, self_rank, self_local] + old_pairs = getattr(lev, "_fault_point_pairs", None) + dm, new_pairs, dropped, new_fields = redistribute_by_assignment( + lev.dm, a_level, pairs=old_pairs, fields=fields) + if dropped: + raise RuntimeError( + f"addon_partition: {dropped} fault pair(s) split across ranks " + f"by the move on level {l}") + # every unit has a cell on every level, so a moved unit moves every + # level and the shell partition never hands the input back here + moved.append(_moved_wrap(lev, dm, new_fields, new_pairs)) + for l, lev in enumerate(moved): + below = base if l == 0 else moved[l - 1] + lev._addon_below = below + if below is None: + lev._addon_parent = None + else: + _resolve_parents(below, lev) + + out = moved[-1] + out.parent = mesh + out._relationship_kind = "refinement" + out._refine_dofs_coincide = False + out.regions = mesh.regions + out._parent_mesh_version = mesh._mesh_version + mesh._registered_children.add(out) + if static or moved[:-1]: + # the tail: the static wraps as they stand (never moved) and the + # add-on levels moved with the finest. A refined base's hierarchy is + # made explicit here, since the moved wrap no longer carries it. + out._custom_mg_coarse_meshes = list(static) + moved[:-1] + out._custom_mg_builder = getattr(mesh, "_custom_mg_builder", "barycentric") + out._custom_mg_fac_zone = getattr(mesh, "_custom_mg_fac_zone", None) + out._addon_partition = record + if verbose: + uw.pprint(f"[addon_partition] {len(regions)} fault region(s) to ranks " + f"{sorted(owner.values())}; {n_moved_units} unit(s) moved " + f"over {len(levels)} level(s); finest cells per rank " + f"max/mean {load_before.max() / load_before.mean():.2f} -> " + f"{load_after.max() / load_after.mean():.2f} after " + f"{n_sweeps} balance sweep(s)") + return out diff --git a/src/underworld3/utilities/custom_mg.py b/src/underworld3/utilities/custom_mg.py index 79928544a..53c77d1bf 100644 --- a/src/underworld3/utilities/custom_mg.py +++ b/src/underworld3/utilities/custom_mg.py @@ -56,7 +56,13 @@ # --------------------------------------------------------------------------- # # Prolongation builders (return scipy CSR matrices) # --------------------------------------------------------------------------- # -def barycentric_prolongation(coarse_coords, fine_coords): +#: What the last barycentric build saw: fine nodes outside the coarse hull +#: (snapped to the nearest coarse node — a wrong row, not an empty one) and the +#: orphan columns it repaired. Read by the parallel builders right after a call. +last_builder_stats = {"outside_hull": 0, "orphans_repaired": 0} + + +def barycentric_prolongation(coarse_coords, fine_coords, repair_orphans=True): """Prolongation by linear interpolation over a Delaunay triangulation of the coarse DOF cloud. Partition of unity (row sums = 1) and linear fields are reproduced exactly; fine points outside the coarse hull fall back to the @@ -99,6 +105,8 @@ def barycentric_prolongation(coarse_coords, fine_coords): tri = Delaunay(coarse_coords) simp = tri.find_simplex(fine_coords) tree = cKDTree(coarse_coords) + last_builder_stats["outside_hull"] = int((simp < 0).sum()) + last_builder_stats["orphans_repaired"] = 0 dim = coarse_coords.shape[1] rows, cols, vals = [], [], [] for i in range(fine_coords.shape[0]): @@ -134,7 +142,17 @@ def barycentric_prolongation(coarse_coords, fine_coords): # The alternative — switching to the global-support RBF builder — removes # the singularity but returns a DENSE transfer (nnz/row == n_coarse), which # makes PᵀAP dense and is unusable at production sizes. - orphans = np.nonzero(np.asarray((P != 0).sum(axis=0)).ravel() == 0)[0] + # In PARALLEL the repair is wrong by construction and is switched off by + # the parallel builders: a rank sees only its own fine nodes, so a coarse + # column reached from another rank (a ghost at a partition boundary, or + # most of an all-gathered coarse cloud) looks orphaned here and the repair + # overwrote a local fine row with a weight-one injection onto a coarse node + # far away (measured: thousands of rows at distance 0.6-0.9 on a box of + # length 3, and a velocity multigrid that needed 55 iterations on 8 ranks + # against 4 in serial). Genuine orphans are dropped at install instead. + orphans = (np.nonzero(np.asarray((P != 0).sum(axis=0)).ravel() == 0)[0] + if repair_orphans else np.zeros(0, dtype=np.int64)) + last_builder_stats["orphans_repaired"] = int(orphans.size) if orphans.size: k = int(min(orphans.size + 8, fine_coords.shape[0])) _, cand = cKDTree(fine_coords).query(coarse_coords[orphans], k=k) @@ -809,7 +827,11 @@ def _fac_patch_split_parallel(P, coords_c, coords_f, lay_c, lay_f, nc, comm, ref = np.asarray([c for c in ref_cnodes if c >= 0], dtype=np.int64) if ref.size: csel[ref] = True - halo_rows = (~row_is_patch) & (bg_cnode >= 0) & csel[np.clip(bg_cnode, 0, None)] + # Index csel only where a row has a background coarse node: a rank can + # hold fine rows and no coarse node at all, and then csel is empty. + halo_rows = np.zeros(n, dtype=bool) + with_bg = (~row_is_patch) & (bg_cnode >= 0) + halo_rows[with_bg] = csel[bg_cnode[with_bg]] node_in_sub = node_is_patch.copy() if n: node_in_sub[node_of_row[halo_rows]] = True @@ -864,6 +886,65 @@ def _coarse_dof_layout(solver, coarse_mesh, field_id=None): return _level_dof_layout(cdm, field_id) +def _node_transfer_no_repair(builder, coarse_coords, fine_coords): + """The node-level transfer for a parallel build: the barycentric builder + with its serial orphan repair switched off (see the note in + :func:`barycentric_prolongation`); any other builder as it is. + + A rank can hold none of a small level. With no fine node it has no rows. + With too few coarse nodes to span a simplex it cannot place its fine nodes + locally: they count as outside the local hull, which is what sends the + "auto" build across partitions, where the coarse cloud is the whole level.""" + import scipy.sparse as sp + n_fine, n_coarse = fine_coords.shape[0], coarse_coords.shape[0] + if n_fine == 0 or n_coarse <= coarse_coords.shape[1]: + last_builder_stats["outside_hull"] = n_fine + last_builder_stats["orphans_repaired"] = 0 + return sp.csr_matrix((n_fine, n_coarse)) + if builder is barycentric_prolongation: + return builder(coarse_coords, fine_coords, repair_orphans=False) + return builder(coarse_coords, fine_coords) + + +def _assemble_node_transfer(Pn, coarse_cols, fine_layout, coarse_layout, ncomp, + comm): + """A node-level transfer ``Pn`` (scipy CSR, fine local nodes by coarse + nodes) as the reduced->reduced MPIAIJ prolongation, assembled in one call. + + Component ``c`` of fine node ``i`` is row ``fine_layout.l2g[i*ncomp + c]`` + and takes, for each weight ``w`` on coarse node ``j``, the column + ``coarse_cols[j, c]``. Only this rank's OWNED rows are set (ghost nodes + are another rank's); constrained coarse DOFs (column ``-1``) drop out, so + the map is reduced->reduced.""" + import scipy.sparse as sp + l2g_f = np.asarray(fine_layout.l2g, dtype=np.int64) + fstart, fend = fine_layout.rstart, fine_layout.rend + nloc_f = fend - fstart + nloc_c = coarse_layout.rend - coarse_layout.rstart + Pn = Pn.tocsr() + node = np.repeat(np.arange(Pn.shape[0]), np.diff(Pn.indptr)) + rows, cols, vals = [], [], [] + for c in range(ncomp): + grow = l2g_f[node * ncomp + c] + gcol = coarse_cols[Pn.indices, c] + keep = (grow >= fstart) & (grow < fend) & (gcol >= 0) + rows.append(grow[keep] - fstart) + cols.append(gcol[keep]) + vals.append(Pn.data[keep]) + rows, cols, vals = (np.concatenate(rows), np.concatenate(cols), + np.concatenate(vals)) + n_cols = int(comm.tompi4py().allreduce(nloc_c)) + loc = sp.csr_matrix((vals, (rows, cols)), shape=(nloc_f, n_cols)) + loc.sum_duplicates() + loc.sort_indices() + P = PETSc.Mat().createAIJ( + size=((nloc_f, None), (nloc_c, None)), + csr=(loc.indptr.astype(PETSc.IntType), loc.indices.astype(PETSc.IntType), + loc.data), comm=comm) + P.assemble() + return P + + def _build_parallel_transfer(coarse_coords, fine_coords, coarse_layout, fine_layout, ncomp, builder, comm): """One reduced->reduced prolongation as an MPIAIJ matrix. @@ -874,33 +955,11 @@ def _build_parallel_transfer(coarse_coords, fine_coords, coarse_layout, coarse ``l2g`` to global columns (off-rank columns are fine for MPIAIJ). Constrained coarse DOFs (``l2g == -1``) drop out -> reduced->reduced.""" l2g_c = coarse_layout.l2g - l2g_f, fstart, fend = fine_layout.l2g, fine_layout.rstart, fine_layout.rend Pn = _drop_structural_zeros( - builder(coarse_coords, fine_coords)) # (n_f_nodes, n_c_nodes), local - nloc_f = fend - fstart - nloc_c = coarse_layout.rend - coarse_layout.rstart - - P = PETSc.Mat().create(comm=comm) - P.setSizes(((nloc_f, None), (nloc_c, None))) - P.setType("aij") - P.setUp() - for i in range(Pn.shape[0]): # fine local node - js = Pn.indices[Pn.indptr[i]:Pn.indptr[i + 1]] - ws = Pn.data[Pn.indptr[i]:Pn.indptr[i + 1]] - for c in range(ncomp): - grow = int(l2g_f[i * ncomp + c]) - if grow < fstart or grow >= fend: # set OWNED rows only - continue - gcols, vals = [], [] - for jj, w in zip(js.tolist(), ws.tolist()): - gcol = int(l2g_c[jj * ncomp + c]) - if gcol >= 0: # skip constrained coarse DOFs - gcols.append(gcol) - vals.append(w) - if gcols: - P.setValues([grow], gcols, vals, addv=PETSc.InsertMode.INSERT_VALUES) - P.assemble() - return P + _node_transfer_no_repair(builder, coarse_coords, fine_coords)) # (n_f_nodes, n_c_nodes), local + return _assemble_node_transfer( + Pn, np.asarray(l2g_c, dtype=np.int64).reshape(-1, ncomp), fine_layout, + coarse_layout, ncomp, comm) def _gather_coarse_cloud(coarse_coords, coarse_layout, ncomp, comm): @@ -936,35 +995,13 @@ def _build_crosspart_transfer(coarse_coords, fine_coords, coarse_layout, small — that is what makes a coarse MG level coarse), so point location spans partitions. Columns are the coarse GLOBAL reduced indices (off-rank columns are fine for MPIAIJ); constrained coarse DOFs (col < 0) drop out.""" - l2g_f, fstart, fend = fine_layout.l2g, fine_layout.rstart, fine_layout.rend coords_u, cols_u = _gather_coarse_cloud(coarse_coords, coarse_layout, ncomp, comm) - Pn = _drop_structural_zeros(builder(coords_u, fine_coords)) # (n_f_local, Nu) - nloc_f = fend - fstart - nloc_c = coarse_layout.rend - coarse_layout.rstart - - P = PETSc.Mat().create(comm=comm) - P.setSizes(((nloc_f, None), (nloc_c, None))) - P.setType("aij") - P.setUp() - for i in range(Pn.shape[0]): # fine local node - js = Pn.indices[Pn.indptr[i]:Pn.indptr[i + 1]] - ws = Pn.data[Pn.indptr[i]:Pn.indptr[i + 1]] - for c in range(ncomp): - grow = int(l2g_f[i * ncomp + c]) - if grow < fstart or grow >= fend: # set OWNED rows only - continue - gcols, vals = [], [] - for jj, w in zip(js.tolist(), ws.tolist()): - gcol = int(cols_u[jj, c]) - if gcol >= 0: # skip constrained coarse DOFs - gcols.append(gcol) - vals.append(w) - if gcols: - P.setValues([grow], gcols, vals, addv=PETSc.InsertMode.INSERT_VALUES) - P.assemble() - return P + Pn = _drop_structural_zeros( + _node_transfer_no_repair(builder, coords_u, fine_coords)) # (n_f_local, Nu) + return _assemble_node_transfer(Pn, np.asarray(cols_u, dtype=np.int64), + fine_layout, coarse_layout, ncomp, comm) def _count_zero_columns_parallel(P, comm): @@ -1451,10 +1488,16 @@ def build(self, solver): rmap, nfull = (_reduced_map(solver.dm, self.field_id) if finest else _coarse_reduced_map(solver, mesh, self.field_id)) maps.append(rmap) - nc = nfull // c.shape[0] - if nfull % c.shape[0] != 0: + # The component count is the field's: take it over the whole level, + # since a rank can hold no node of a small one. + nodes, dofs = c.shape[0], nfull + if parallel: + level_comm = solver.dm.comm.tompi4py() + nodes, dofs = level_comm.allreduce(nodes), level_comm.allreduce(dofs) + nc = dofs // nodes + if dofs % nodes != 0: raise RuntimeError( - f"level {k}: full DOFs {nfull} not divisible by nodes {c.shape[0]}") + f"level {k}: full DOFs {dofs} not divisible by nodes {nodes}") coords.append(c) ncomp.append(nc) @@ -1480,6 +1523,24 @@ def build(self, solver): comm = solver.dm.comm from underworld3.utilities import fault_mg for l in range(1, nlev): + # A pair the add-on partition recorded (the base's finest static + # level against the first add-on level, which the partition moved + # apart, and any unsplit pair of moved add-on levels) carries the + # exact parent relation: each fine cell's parent is fetched from + # its rank through a star forest, and nothing is located across + # partitions. Split coarse levels are never recorded against, so + # their pairs keep the side-aware cell-located transfer. + pre = (fault_mg.provenance_source(self.level_meshes[l - 1], + self.level_meshes[l]) + if parallel else None) + + def provenance_transfer(): + P = fault_mg.provenance_transfer_parallel( + self.level_meshes[l - 1], self.level_meshes[l], pre, degree, + continuous, coords[l - 1], coords[l], maps[l - 1], maps[l], + nc, comm) + self.level_transfer_stats[l] = fault_mg.last_transfer_stats + return P # Split-level multigrid: every pair touching a split level takes # the cell-located transfer. A split COARSE level defeats the # node-cloud builders (they triangulate the coarse nodes, and @@ -1495,7 +1556,9 @@ def build(self, solver): if self.has_split_levels and ( fault_mg.is_split_level(self.level_meshes[l - 1]) or fault_mg.is_split_level(self.level_meshes[l])): - if parallel: + if pre is not None: + P = provenance_transfer() + elif parallel: P = fault_mg.cell_located_transfer_parallel( self.level_meshes[l - 1], self.level_meshes[l], degree, continuous, coords[l - 1], coords[l], maps[l - 1], @@ -1513,7 +1576,9 @@ def build(self, solver): if parallel: args = (coords[l - 1], coords[l], maps[l - 1], maps[l], nc, self.builder, comm) - if self.cross_partition is True: + if pre is not None: + P = provenance_transfer() + elif self.cross_partition is True: P = _build_crosspart_transfer(*args) else: P = _build_parallel_transfer(*args) @@ -1528,10 +1593,19 @@ def build(self, solver): # nonsense (KSP reason -11 at the first iteration). The # few genuine orphans are the repair's job below. n_zero = _count_zero_columns_parallel(P, comm) - if self.cross_partition == "auto" and n_zero > 0: + # A fine node outside its rank's LOCAL coarse hull is snapped + # to the nearest local coarse node: a WRONG row, which no + # zero-column count can see. It happens whenever the fine + # partition is not the coarse one — a split finest whose + # fault star was moved to one rank (measured: 3026 of 7500 + # nodes on that rank, 2192 wrong rows). Such a level is + # rebuilt across partitions whatever its column count. + n_out = comm.tompi4py().allreduce( + int(last_builder_stats.get("outside_hull", 0))) + if self.cross_partition == "auto" and (n_zero > 0 or n_out > 0): P_x = _build_crosspart_transfer(*args) n_x = _count_zero_columns_parallel(P_x, comm) - if n_x < n_zero: + if n_out > 0 or n_x < n_zero: P.destroy() P = P_x else: @@ -2217,7 +2291,11 @@ def _colocate_level(coarse_mesh, fine_mesh): if comm.size == 1: return coarse_mesh cS, cE = dm.getHeightStratum(0) - cen_c = np.array([dm.computeCellGeometryFVM(c)[1] for c in range(cS, cE)]) + # Shaped from the coordinate dimension, not from the cells found: a coarse + # level is small, and on a large run most ranks own none of it. + cdim = dm.getCoordinateDim() + cen_c = np.array([dm.computeCellGeometryFVM(c)[1] for c in range(cS, cE)], + dtype=float).reshape(-1, cdim) fdm = fine_mesh.dm fS, fE = fdm.getHeightStratum(0) # OWNED fine cells only: a ghost cell belongs to another rank @@ -2228,8 +2306,8 @@ def _colocate_level(coarse_mesh, fine_mesh): except (ValueError, TypeError): ghost = set() owned_f = [c for c in range(fS, fE) if c not in ghost] - cen_f = (np.array([fdm.computeCellGeometryFVM(c)[1] for c in owned_f]) - if owned_f else np.zeros((0, cen_c.shape[1]))) + cen_f = np.array([fdm.computeCellGeometryFVM(c)[1] for c in owned_f], + dtype=float).reshape(-1, cdim) # every rank offers its nearest owned fine cell to EVERY coarse centroid # in the mesh (all-gathered: the coarse levels are small) cen_all = np.vstack(comm.allgather(cen_c)) @@ -2280,11 +2358,16 @@ def adopt_hierarchy(mesh, base_mesh, fac_zone=None, builder=None): own = getattr(base_mesh, "_custom_mg_coarse_meshes", None) tail = (list(own) + [base_mesh] if own is not None else list(base_mesh._coarse_level_meshes())) - # In parallel the placed mesh is gathered onto its surgery rank while - # the tail is load-balanced: co-locate every level with the finest so - # the transfers pair rank-locally (the coarse levels are small; the - # fine mesh and its FAC patch never move). - if mesh.dm.getComm().getSize() > 1: + # A base the add-on partition produced (addon_partition.partition_addon, + # called by the placement routes before they place) is aligned with its + # tail already: the placement moved nothing, the pair it separated is + # read from the recorded provenance, and nothing is co-located. + # Otherwise the placed mesh was gathered onto its surgery rank while the + # tail is load-balanced: co-locate every level with the finest so the + # transfers pair rank-locally (the coarse levels are small; the fine + # mesh and its FAC patch never move). + if (mesh.dm.getComm().getSize() > 1 + and getattr(base_mesh, "_addon_partition", None) is None): tail = [_colocate_level(level, mesh) for level in tail] mesh._custom_mg_coarse_meshes = tail mesh._custom_mg_builder = (builder if builder is not None diff --git a/src/underworld3/utilities/edge_split.py b/src/underworld3/utilities/edge_split.py index 2d1073d21..f75eadd39 100644 --- a/src/underworld3/utilities/edge_split.py +++ b/src/underworld3/utilities/edge_split.py @@ -62,32 +62,32 @@ def _register_transform(): from underworld3.utilities import _nvb_transform # noqa: F401 (registers on import) +def _edge_ends(dm): + """The two vertex points of every edge, ``(n_edges, 2)``, in cone order, + read as one table.""" + from underworld3.utilities._plex_topology import closure_points + eS, eE = dm.getDepthStratum(1) + return closure_points(dm, np.arange(eS, eE), 0).reshape(eE - eS, 2) + + def _edge_lengths(dm): """Length of every edge, indexed by ``edge_point - edge_start``.""" cdim = dm.getCoordinateDim() vS, _vE = dm.getDepthStratum(0) eS, eE = dm.getDepthStratum(1) X = np.asarray(dm.getCoordinatesLocal().array).reshape(-1, cdim) - ends = np.array([dm.getCone(e) for e in range(eS, eE)], dtype=np.int64) - vS + ends = _edge_ends(dm) - vS d = X[ends[:, 0]] - X[ends[:, 1]] return np.sqrt(np.einsum("ij,ij->i", d, d)) def _cell_edges(dm): - """Edge points of each cell, as a list indexed by ``cell - cell_start``. - - In 2-D a cell's cone is already its edges; in 3-D the cone holds faces, so - the edges come from the transitive closure filtered to the edge stratum. - """ - eS, eE = dm.getDepthStratum(1) + """Edge points of each cell, as a list indexed by ``cell - cell_start``: + the edge stratum of the cell's closure, in closure order (in 2-D that is + the cone), read as one table.""" + from underworld3.utilities._plex_topology import closure_points cS, cE = dm.getHeightStratum(0) - if dm.getDimension() == 2: - return [np.asarray(dm.getCone(c), dtype=np.int64) for c in range(cS, cE)] - out = [] - for c in range(cS, cE): - closure = dm.getTransitiveClosure(c)[0] - out.append(np.array([p for p in closure if eS <= p < eE], dtype=np.int64)) - return out + return list(closure_points(dm, np.arange(cS, cE), 1)) def cell_diameters(dm, return_tables=False): @@ -177,7 +177,7 @@ def _edge_strength(dm): vS, _vE = dm.getDepthStratum(0) eS, eE = dm.getDepthStratum(1) X = np.asarray(dm.getCoordinatesLocal().array).reshape(-1, cdim) - ends = np.array([dm.getCone(e) for e in range(eS, eE)], dtype=np.int64) - vS + ends = _edge_ends(dm) - vS d = X[ends[:, 0]] - X[ends[:, 1]] length = np.sqrt(np.einsum("ij,ij->i", d, d)) mid = 0.5 * (X[ends[:, 0]] + X[ends[:, 1]]) diff --git a/src/underworld3/utilities/fault_contact.py b/src/underworld3/utilities/fault_contact.py index e1239a8c6..014893c34 100644 --- a/src/underworld3/utilities/fault_contact.py +++ b/src/underworld3/utilities/fault_contact.py @@ -358,10 +358,16 @@ def ViscousFaultLaw(eta_f): return SymbolicFaultLaw(float(eta_f) * slip_rate) -def CoulombFaultLaw(mu, sigma_n, V0): +def CoulombFaultLaw(mu, sigma_n, V0, background=0.0): r"""Regularised Coulomb friction, :math:`\tau = \mu\sigma_n\,\tfrac{2}{\pi}\arctan(V/V_0)`. + ``background`` adds a constant effective normal stress (compression + positive) to the reaction-fed one: the overburden less the pore pressure + of a fault that stays clamped while a tide modulates its strength + (Berne et al. 2024 on the tiger stripes: an overburden of about 50 kPa + against tidal normal tractions of about 25 kPa). + Smooth, bounded by the strength :math:`\mu\sigma_n`, monotone. Below the regularisation velocity :math:`V_0` the fault sticks (residual creep :math:`\sim V_0`); above it the shear traction saturates and the @@ -372,16 +378,16 @@ def CoulombFaultLaw(mu, sigma_n, V0): """ if sigma_n == "reaction": # the strength clamp is the law's own: bare friction loses all - # strength the moment the normal stress turns tensile - strength = float(mu) * sympy.Max(normal_stress, 0) + # strength the moment the (effective) normal stress turns tensile + strength = float(mu) * sympy.Max(normal_stress + float(background), 0) else: - strength = float(mu) * float(sigma_n) + strength = float(mu) * (float(sigma_n) + float(background)) return SymbolicFaultLaw( strength * (2 / sympy.pi) * sympy.atan(slip_rate / float(V0))) def add_coulomb_fault_bc(solver, conds, boundary, sigma_n=None, V0=1.0e-5, - normal=None): + normal=None, background=0.0): """Regularised Coulomb friction on the split fault ``boundary`` (value-first: ``conds`` is the friction coefficient mu). @@ -401,8 +407,10 @@ def add_coulomb_fault_bc(solver, conds, boundary, sigma_n=None, V0=1.0e-5, if mu <= 0.0 or float(V0) <= 0.0 or ( sigma_n != "reaction" and float(sigma_n) <= 0.0): raise ValueError("mu, sigma_n and V0 must all be positive.") + if float(background) < 0.0: + raise ValueError("background (the constant effective normal stress, compression positive) must be >= 0.") add_frictionless_fault_bc(solver, boundary, normal=normal) - _register_law(solver, boundary, CoulombFaultLaw(mu, sigma_n, V0)) + _register_law(solver, boundary, CoulombFaultLaw(mu, sigma_n, V0, background)) def RateStateFaultLaw(a, b, V0, f0, Dc, sigma_n="reaction"): @@ -767,6 +775,91 @@ def _tangent_frame(nrm): for i, j in ((0, 1), (1, 2), (2, 0))]) +def _linear_trace_fit(facets, facet_verts, n_rows): + r"""Operators that recover a traction on a P2-triangle fault trace as a + linear field on the facet vertices. + + For a traction :math:`\sigma = \sum_v \sigma_v \lambda_v` linear on a + facet of area :math:`A`, the consistent P2 load is exact and closed-form: + + .. math:: + + \int \phi_{v_i} \sigma = \tfrac{A}{30}\sigma_i + - \tfrac{A}{60}(\sigma_j + \sigma_k), \qquad + \int \phi_{e_{ij}} \sigma = \tfrac{A}{15}(2\sigma_i + 2\sigma_j + + \sigma_k). + + Assembled over the facets this gives ``G``, pair-node load from vertex + values: about three times as many rows as columns, so the vertex values + are a least-squares fit. Rim vertices are ordinary columns — they have + no pair row of their own, but the midpoints beside them pin them. + + Returns ``(G, ends)``: ``G`` a rank-local AIJ Mat ``(n_rows, n_vertices)`` + and ``ends`` an ``(n_rows, 2)`` array of the vertex columns whose mean + is each pair node's value (a vertex node lists its own column twice). + """ + column = {} + entries = {} + ends = np.full((n_rows, 2), -1, dtype=np.int64) + for (idx, area, _law_id), verts in zip(facets, facet_verts): + cols = [column.setdefault(v, len(column)) for v in verts] + for i in range(3): + k = idx[i] + if k < 0: + continue + ends[k] = (cols[i], cols[i]) + for j in range(3): + w = area / 30.0 if i == j else -area / 60.0 + entries[(k, cols[j])] = entries.get((k, cols[j]), 0.0) + w + for e, (i, j) in enumerate(((0, 1), (1, 2), (2, 0))): + k = idx[3 + e] + if k < 0: + continue + ends[k] = (cols[i], cols[j]) + for m in range(3): + w = (2.0 if m in (i, j) else 1.0) * area / 15.0 + entries[(k, cols[m])] = entries.get((k, cols[m]), 0.0) + w + + G = PETSc.Mat().createAIJ(size=(n_rows, max(len(column), 1)), + comm=PETSc.COMM_SELF) + G.setPreallocationNNZ(9) + for (k, c), w in entries.items(): + G.setValue(int(k), int(c), w, addv=True) + G.assemble() + return G, ends + + +def _fitted_normal_traction(fit, load): + """Pair-node traction from the nodal load through the linear trace fit. + + Solved by LSQR on this rank: the fault is rank-interior after the + redistribution every parallel split performs. A failed fit is reported + to every rank, since the callers run collectively. + """ + G, ends = fit + if not len(ends): + _collective_raise(None) + return np.zeros(0) + n_rows, n_cols = G.getSize() + rhs = PETSc.Vec().createSeq(n_rows) + rhs.setArray(np.asarray(load, dtype=float)) + vertex_values = PETSc.Vec().createSeq(n_cols) + ksp = PETSc.KSP().create(comm=PETSc.COMM_SELF) + ksp.setOperators(G) + ksp.setType("lsqr") + ksp.getPC().setType("none") + ksp.setTolerances(rtol=1e-12, atol=0.0, max_it=20 * n_cols) + ksp.solve(rhs, vertex_values) + reason = ksp.getConvergedReason() + values = vertex_values.getArray().copy() + for obj in (ksp, rhs, vertex_values): + obj.destroy() + _collective_raise(None if reason > 0 else + f"the linear trace fit of the fault normal traction " + f"did not converge (KSP reason {reason}).") + return 0.5 * (values[ends[:, 0]] + values[ends[:, 1]]) + + class _InterfaceAssembler: r"""Residual and consistent tangent of the interface laws, per iterate. @@ -801,6 +894,9 @@ def __init__(self, solver, include=()): rows, lo_p, lo_m, tans, nrms, laws_of, facets = \ [], [], [], [], [], [], [] + # 3-D only: each facet's vertex points (rim vertices included), + # for the linear fit that recovers the normal traction + facet_verts = [] index_of = {} fault_of = [] current = [None] # the fault being walked @@ -890,6 +986,7 @@ def node_index(q): (verts[i], verts[j]))]) for i, j in ((0, 1), (1, 2), (2, 0))) facets.append((idx, area, law_id)) + facet_verts.append(verts) _collective_raise(problem) # petsc4py refuses to narrow index arrays — Vec/Mat setValues need @@ -910,27 +1007,38 @@ def node_index(q): self._dim = dim self._points = {q: k for q, k in index_of.items()} - # Positive trace mass per pair node, for de-smearing the constraint - # reaction into a pointwise normal traction. On the P2 LINE the - # lumped (row-sum) mass is positive (L/6, 2L/3, L/6) and is used - # directly. On the P2 TRIANGLE the lumped VERTEX rows vanish (the - # known trap: row sums are 0, 0, 0, A/3, A/3, A/3), so the triangle - # uses the P1 SUB-LUMPING instead — each straight P2 triangle is - # four P1 sub-triangles of area A/4, whose lumped masses give every - # node a positive weight (A/12 per vertex, A/4 per midpoint). + # Recovering a pointwise normal traction from the constraint + # reaction. On the P2 LINE the lumped (row-sum) trace mass is + # positive (L/6, 2L/3, L/6) and dividing the nodal load by it is + # exact for a uniform traction. On the P2 TRIANGLE the consistent + # load of ANY traction vanishes at the vertex nodes (row sums 0, 0, + # 0, A/3, A/3, A/3), so no nodal division can be pointwise right — + # the former P1 sub-lumping read 0 at every vertex and 4/3 of the + # traction at every midpoint. A consistent-mass solve does not + # rescue it either: the unsplit rim nodes carry no load, and leaving + # them out polluted the whole patch (measured error ~50%). The + # triangle instead fits the traction as a LINEAR field on the facet + # vertices, rim vertices included, whose consistent load matches the + # nodal load in least squares — exact for uniform and linear + # traction (see _linear_trace_fit). # The star-forest completion below is defensive: after the # redistribution that precedes every parallel split the fault is # rank-interior and the exchange is a no-op on the fault rows. + self._traction_fit = (_linear_trace_fit(facets, facet_verts, len(rows)) + if solver.mesh.dim == 3 else None) mass = np.zeros(len(rows)) - for idx, measure, _law_id in facets: - if dim == 2: - weights = (measure / 6.0, 2.0 * measure / 3.0, - measure / 6.0) - else: - weights = (measure / 12.0,) * 3 + (measure / 4.0,) * 3 - for k, w in zip(idx, weights): - if k >= 0: - mass[k] += w + # the branch follows the facet TOPOLOGY (mesh.dim), as the quadrature + # above does, not the component count: a fault on a surface mesh is an + # edge carried by a 3-vector, and the triangle weights gave its mid-edge + # nodes an eighth of their mass (measured: the breathing sphere's fault + # read 5 to 8 times its membrane resultant). The triangle (mesh.dim 3) + # takes no lumped mass at all: its recovery is the linear fit above. + if solver.mesh.dim == 2: + for idx, measure, _law_id in facets: + weights = (measure / 6.0, 2.0 * measure / 3.0, measure / 6.0) + for k, w in zip(idx, weights): + if k >= 0: + mass[k] += w if mpi.size > 1: pStart, pEnd = dm.getChart() chart_mass = np.zeros(pEnd - pStart, dtype=np.float64) @@ -1034,20 +1142,24 @@ def nodal_normal_traction(self, solver, reaction): reaction (negative in compression): the jump-normal constraint's nodal load l_i = (n.r+ - n.r-)/2, de-smeared by the lumped trace mass. The reaction must be the PURE bulk residual — the loop - stashes it before the interface force is added.""" + stashes it before the interface force is added. On a thin sheet the + residual carries the thickness, so this is the membrane RESULTANT + (L sigma_nn): the interface laws' effective normal stress and any + background they add are in the same units.""" dm = solver.dm dim = self._nrm.shape[1] lvec = dm.getLocalVec() dm.globalToLocal(reaction, lvec) a = np.asarray(lvec.getArray()) - sig = np.zeros(len(self._rows)) + load = np.zeros(len(self._rows)) for k in range(len(self._rows)): rp = a[self._lo_p[k]:self._lo_p[k] + dim] rm = a[self._lo_m[k]:self._lo_m[k] + dim] - load = 0.5 * float(self._nrm[k] @ (rp - rm)) - sig[k] = load / max(self._mass[k], 1e-300) + load[k] = 0.5 * float(self._nrm[k] @ (rp - rm)) dm.restoreLocalVec(lvec) - return sig + if self._traction_fit is not None: + return _fitted_normal_traction(self._traction_fit, load) + return load / np.maximum(self._mass, 1e-300) def update_normal_stress(self, solver, reaction): """Picard-lag the SIGNED effective normal stress into the laws: diff --git a/src/underworld3/utilities/fault_mg.py b/src/underworld3/utilities/fault_mg.py index a9d0a0abd..a2f1b7008 100644 --- a/src/underworld3/utilities/fault_mg.py +++ b/src/underworld3/utilities/fault_mg.py @@ -66,9 +66,26 @@ def is_split_level(mesh): def _cell_vertex_table(mesh): """Vertex coordinates ``(n_v, cdim)`` and the ``(n_cells, dim + 1)`` table of - vertex rows per cell — the P1 node table, which the mesh caches.""" - vxy = np.asarray(mesh._get_coords_for_basis(1, True)) - cv = np.asarray(mesh._cell_node_indices(1, True)) + vertex rows per cell. + + Read straight from the DM: the vertex coordinates through the coordinate + section and each cell's vertices from its closure, with no interpolation + built. A periodic mesh carries its coordinates cell by cell, so there the + P1 node table of the mesh is used, which handles them.""" + from underworld3.utilities._plex_topology import closure_points, section_offsets + dm = mesh.dm + cell_coordinates = dm.getCellCoordinatesLocal() + if cell_coordinates is not None and cell_coordinates.handle: + return (np.asarray(mesh._get_coords_for_basis(1, True)), + np.asarray(mesh._cell_node_indices(1, True))) + cdim = dm.getCoordinateDim() + vS, vE = dm.getDepthStratum(0) + cS, cE = dm.getHeightStratum(0) + X = np.asarray(dm.getCoordinatesLocal().array).reshape(-1, cdim) + rows = section_offsets(dm.getCoordinateSection(), np.arange(vS, vE)) // cdim + vxy = X[rows] + cv = (closure_points(dm, np.arange(cS, cE), 0) - vS).reshape( + cE - cS, dm.getDimension() + 1) return vxy, cv @@ -359,6 +376,164 @@ def cell_located_transfer_parallel(coarse, fine, degree, continuous, Call = np.concatenate(hit_cols) if len(hit_cols) > 1 else hit_cols[0] # --- assemble owned rows ---------------------------------------------- # + return _assemble_owned_rows(nodes, Wall, Call, l2g_f, fstart, fend, cstart, + cend, ncomp, nb, comm) + + +def locate_in_cells_exhaustive(points, vxy, cv): + """:func:`locate_in_cells`, widening the search for every point still outside + its candidate. On a graded mesh a large cell's centroid can be farther from a + point inside it than many small neighbours' centroids are, so the nearest + sixteen can all miss: the misses are retried against the nearest 128 and + 1024, and what still misses is tested against every cell, one point at a + time (memory one cell table, not points times cells). Returns + ``(cell, lam)``; a point inside no cell keeps its best candidate with + ``lam.min() < -_INSIDE_TOL``.""" + cell, lam = locate_in_cells(points, vxy, cv) + if cv.shape[0] == 0: + return cell, lam + for k in (128, 1024): + miss = np.flatnonzero(lam.min(axis=1) < -_INSIDE_TOL) + if miss.size == 0: + return cell, lam + c2, l2 = locate_in_cells(points[miss], vxy, cv, k=k) + better = l2.min(axis=1) > lam[miss].min(axis=1) + cell[miss[better]] = c2[better] + lam[miss[better]] = l2[better] + V = vxy[cv] + for k in np.flatnonzero(lam.min(axis=1) < -_INSIDE_TOL): + lam_all = _barycentric(np.repeat(points[k][None, :], cv.shape[0], axis=0), V) + best = int(np.argmax(lam_all.min(axis=1))) + if lam_all[best].min() > lam[k].min(): + cell[k], lam[k] = best, lam_all[best] + return cell, lam + + +# --------------------------------------------------------------------------- # +# The provenance transfer: the one pair the add-on partition moved +# --------------------------------------------------------------------------- # +def provenance_source(coarse, fine): + """The mesh that records, per cell, where its parent on ``coarse`` lives — + ``fine`` itself or an ancestor of it (the add-on level as the add-on + partition left it, before the cut and the split re-represented it; see + ``addon_partition``). ``None`` when the pair carries no such record.""" + m = fine + while m is not None: + if (getattr(m, "_addon_below", None) is coarse + and getattr(m, "_addon_parent", None) is not None): + return m + m = getattr(m, "parent", None) + return None + + +def provenance_transfer_parallel(coarse, fine, pre, degree, continuous, + coarse_coords, fine_coords, coarse_layout, + fine_layout, ncomp, comm): + """The cell-located transfer for a pair the add-on partition moved apart, + built from the recorded parent relation instead of a search. + + ``pre`` carries the record (``provenance_source``): every cell of ``pre`` + knows the rank and local index of its parent on ``coarse``. It is ``fine`` + itself whenever the record was inherited when ``fine`` was made (the cut, + the placement and the split call ``addon_partition.inherit_provenance``), + and then a node's own cell is the recorded cell. Otherwise the fine cell's + centroid is located in ``pre`` (rank-local, since the surgery was, and a + centroid is never on the fault). The recorded parent is then fetched — its node columns and vertex and node + coordinates — through one star forest. Co-resident parents are a local + copy; the moved units' parents come from their ranks. Nothing is + all-gathered. ``coarse_layout`` / ``fine_layout`` are + :class:`custom_mg.LevelLayout`. + """ + m4 = comm.tompi4py() + cn = np.asarray(coarse._cell_node_indices(degree, continuous)) + fn = np.asarray(fine._cell_node_indices(degree, continuous)) + Xc = np.asarray(coarse_coords) + Xf = np.asarray(fine_coords) + vxy_c, cv_c = _cell_vertex_table(coarse) + vxy_f, cv_f = _cell_vertex_table(fine) + vxy_p, cv_p = _cell_vertex_table(pre) + l2g_c = np.asarray(coarse_layout.l2g, dtype=np.int64).reshape(-1, ncomp) + l2g_f = np.asarray(fine_layout.l2g, dtype=np.int64).reshape(-1, ncomp) + fstart, fend = fine_layout.rstart, fine_layout.rend + cstart, cend = coarse_layout.rstart, coarse_layout.rend + n_f = Xf.shape[0] + nb = cn.shape[1] + nv = cv_c.shape[1] + dim = Xc.shape[1] + + owner = _fine_cell_of_node(fn, n_f) + if m4.allreduce(int((owner < 0).sum())): + raise RuntimeError("fault_mg: a fine node belongs to no cell") + mine = np.flatnonzero(((l2g_f >= fstart) & (l2g_f < fend)).any(axis=1)) + fcent = vxy_f[cv_f].mean(axis=1) + if pre is fine: + # the record is on this very mesh: a node's cell IS the recorded cell + pcell = owner[mine] + plam = np.zeros((mine.size, nv)) + elif mine.size: + pcell, plam = locate_in_cells_exhaustive(fcent[owner[mine]], vxy_p, cv_p) + else: + pcell = np.zeros(0, dtype=np.int64) + plam = np.zeros((0, nv)) + n_out = int(m4.allreduce(int((plam.min(axis=1) < -_INSIDE_TOL).sum()) + if mine.size else 0)) + if n_out: + raise RuntimeError( + f"fault_mg: {n_out} fine node(s) lie in no cell of the level as the " + f"add-on partition left it; the surgery was not rank-local.") + parent = np.asarray(pre._addon_parent, dtype=np.int32)[pcell] # (nm, 2) + + # one fetch per distinct parent: node columns, node coordinates and + # vertex coordinates, packed as doubles (indices exact below 2**53) + uniq, inv = np.unique(parent, axis=0, return_inverse=True) + inv = inv.ravel() + width = nb * ncomp + nb * dim + nv * dim + nc_loc = cv_c.shape[0] + root = np.empty((nc_loc, width)) + # explicit widths: a rank can own no cell of the coarse level, and + # reshape(0, -1) cannot infer one + root[:, : nb * ncomp] = l2g_c[cn].reshape(nc_loc, nb * ncomp) + root[:, nb * ncomp: nb * ncomp + nb * dim] = Xc[cn].reshape(nc_loc, nb * dim) + root[:, nb * ncomp + nb * dim:] = vxy_c[cv_c].reshape(nc_loc, nv * dim) + sf = PETSc.SF().create(comm=comm) + sf.setGraph(nc_loc, None, np.ascontiguousarray(uniq, dtype=np.int32)) + sf.setUp() + got = np.zeros((uniq.shape[0], width)) + unit = MPI.DOUBLE.Create_contiguous(width) + unit.Commit() + sf.bcastBegin(unit, root, got, MPI.REPLACE) + sf.bcastEnd(unit, root, got, MPI.REPLACE) + unit.Free() + sf.destroy() + cols_u = np.rint(got[:, : nb * ncomp]).astype(np.int64).reshape(-1, nb, ncomp) + Xn_u = got[:, nb * ncomp: nb * ncomp + nb * dim].reshape(-1, dim) + V_u = got[:, nb * ncomp + nb * dim:].reshape(-1, dim) + # the fetched parents as a virtual coarse mesh: cell k has nodes + # k*nb .. and vertices k*nv .. + nu = uniq.shape[0] + cn_u = np.arange(nu * nb).reshape(nu, nb) + cv_u = np.arange(nu * nv).reshape(nu, nv) + lam = _barycentric(Xf[mine], V_u[cv_u[inv]]) if mine.size else np.zeros((0, nv)) + W = (_node_weights(Xf[mine], inv, lam, Xn_u, cn_u, V_u, cv_u, degree) + if mine.size else np.zeros((0, nb))) + gcols = cols_u[inv] # (nm, nb, ncomp) + + global last_transfer_stats + last_transfer_stats = { + "owned_nodes": int(m4.allreduce(int(mine.size))), + "fetched_remote": int(m4.allreduce(int((uniq[:, 0] != m4.rank).sum()))), + "local_coarse_cells": m4.allgather(int(nc_loc)), + "local_fine_cells": m4.allgather(int(cv_f.shape[0])), + } + return _assemble_owned_rows(mine, W, gcols, l2g_f, fstart, fend, cstart, + cend, ncomp, nb, comm) + + +def _assemble_owned_rows(nodes, Wall, Call, l2g_f, fstart, fend, cstart, cend, + ncomp, nb, comm): + """The owned rows of a reduced->reduced MPIAIJ transfer from per-node + weights ``Wall (n, nb)`` on global coarse columns ``Call (n, nb, ncomp)``.""" + m4 = comm.tompi4py() nloc_f = fend - fstart nloc_c = cend - cstart rows_l, cols_l, vals_l = [], [], [] @@ -433,7 +608,10 @@ def split_tail_levels(levels, fault_specs, verbose=False): level = levels[k] problem = None try: - split = add_fault(level, entries, verbose=verbose, **kwargs) + # the level arrives partitioned with the finest (the add-on + # partition ran once, for every level): no second partition + split = add_fault(level, entries, verbose=verbose, partition=False, + **kwargs) except (ValueError, RuntimeError, NotImplementedError) as exc: problem = str(exc) # the split's refusals are collective; a failure on one rank after its @@ -454,89 +632,6 @@ def split_tail_levels(levels, fault_specs, verbose=False): return out -def _fault_star_cells(mesh): - """The local cells of every split fault's star on ``mesh``: cells incident to a - vertex of a Plus or Minus facet, plus one growth layer (the same two-layer - rule as the split's own redistribution, so a moved star stays - rank-interior). Returns local cell indices (``cS``-based).""" - dm = mesh.dm - cS, cE = dm.getHeightStratum(0) - fS, fE = dm.getHeightStratum(1) - vS, vE = dm.getDepthStratum(0) - names = [b.name for b in mesh.boundaries] - verts = set() - for name in getattr(mesh, "_fault_point_pairs", {}): - for side in ("Plus", "Minus"): - lab_name = f"{name}{side}" - if lab_name not in names: - continue - lab = dm.getLabel(lab_name) - value = mesh.boundaries[lab_name].value - if lab and lab.getStratumSize(value) > 0: - for f in lab.getStratumIS(value).getIndices(): - if fS <= int(f) < fE: - verts.update(int(q) for q in dm.getTransitiveClosure(int(f))[0] - if vS <= int(q) < vE) - - def star(vs): - cells = set() - for v in vs: - cells.update(int(q) for q in dm.getTransitiveClosure(v, useCone=False)[0] - if cS <= int(q) < cE) - return cells - - cells = star(verts) - ring = set() - for c in cells: - ring.update(int(q) for q in dm.getTransitiveClosure(c)[0] if vS <= int(q) < vE) - cells |= star(ring) - return np.array(sorted(c - cS for c in cells), dtype=np.int64) - - -def fault_star_owner(mesh): - """The rank holding most of the split faults' star on ``mesh`` (collective).""" - comm = mesh.dm.getComm().tompi4py() - counts = np.asarray(comm.allgather(int(_fault_star_cells(mesh).size))) - return int(np.argmax(counts)) - - -def colocate_split_level(level, target): - """Move a split level's fault star (two layers, as the split gathers it) onto - rank ``target`` — the rank holding the FINEST level's star — carrying the - pairing through as labels. The split gathers each level's star onto the - rank already holding most of it, which differs from level to level, and a - transfer between stars on different ranks locates most of the fault-band - nodes remotely (measured: 48 % of the finest nodes at np = 2, 58 % at - np = 8). With the stars on one rank the misses are the band's rim only. - Returns ``level`` itself when nothing moves. Collective.""" - from underworld3.utilities.fault_split import redistribute_by_assignment - - comm = level.dm.getComm().tompi4py() - if comm.size == 1: - return level - cS, cE = level.dm.getHeightStratum(0) - assign = np.full(cE - cS, comm.rank, dtype=np.int32) - assign[_fault_star_cells(level)] = target - dm, pairs, dropped = redistribute_by_assignment( - level.dm, assign, pairs=dict(level._fault_point_pairs)) - if dm is level.dm: - return level - if dropped: - raise RuntimeError( - f"fault_mg: {dropped} pair(s) lost co-locating a level's fault star") - moved = uw.discretisation.Mesh( - dm, simplex=level.dm.isSimplex(), qdegree=level.qdegree, - coordinate_system_type=level.CoordinateSystem.coordinate_type, - boundaries=level.boundaries, verbose=False) - moved._fault_point_pairs = pairs - for attr in ("_fault_traces", "_fault_specs", "_fault_surfaces", "_fault_seam_normals"): - if getattr(level, attr, None) is not None: - setattr(moved, attr, getattr(level, attr)) - moved._custom_mg_coarse_meshes = None - moved._mg_static = False - return moved - - # --------------------------------------------------------------------------- # # Per-level pair rotation # --------------------------------------------------------------------------- # diff --git a/src/underworld3/utilities/fault_split.py b/src/underworld3/utilities/fault_split.py index 736d9ba74..764ea4f90 100644 --- a/src/underworld3/utilities/fault_split.py +++ b/src/underworld3/utilities/fault_split.py @@ -44,11 +44,14 @@ partition seam — the replicas are then rank-local and the star-forest carries over by renumbering alone, exactly as the deletion pass argues for its own seam freeze. A fault that DOES touch the seam is not the user's problem: -:func:`split_fault` (and :func:`add_fault`) REDISTRIBUTE first — the fault's -cell star plus one growth layer moves to the rank that already owns most of -it (:func:`_redistribute_fault_interior`), the chain or patch becomes -rank-interior, and the split runs with serial topology. Only a direct call -to the low-level splitters on a seam-touching fault is refused. The 2-D +:func:`split_fault` (and :func:`add_fault`) PARTITION the add-on first — one +assignment for the finest level and every add-on level below it, each fault's +cell star plus a growth layer glued and sent to the rank that already owns +most of it, the load balanced +(:func:`~underworld3.utilities.addon_partition.partition_addon`) — so the +chain or patch is rank-interior and the split runs with serial topology, +moving nothing itself. Only a direct call to the low-level splitters on a +seam-touching fault is refused. The 2-D split can instead run THROUGH the seam (``across_seams=True``, the path of a band meshed through it by ``place_thin_volume(seams="conform")``): the chain is assembled globally from the star-forest identities of its @@ -1598,7 +1601,7 @@ def _label_embedded_edges(dm, segments, values, exclude=None, verbose=False, def add_fault(mesh, faults, verbose=False, cut=True, exclude=None, - blind=1, across_seams=False, **place_kwargs): + blind=1, across_seams=False, partition=True, **place_kwargs): """Cut AND split one or more faults into a mesh; return the split Mesh. The one-call form of the split-node pipeline: for each fault, the tips @@ -1648,6 +1651,12 @@ def add_fault(mesh, faults, verbose=False, cut=True, exclude=None, split through it — a chain end on the seam is a crossing, not a clipped end, so it is neither blinded nor pinned; nothing is redistributed. See :func:`split_fault`. + partition : bool, optional + In parallel, partition the add-on hierarchy for these faults BEFORE + the cut (``addon_partition.partition_addon``: every fault's cell star + rank-interior on every add-on level, the load balanced, nothing moved + by the cut or the split afterwards). ``False`` for a level that is + already partitioned — the replay of the faults on the add-on levels. **place_kwargs 3-D only: options of :func:`place_surface.place_sheet` (``clearance``, ``size``, ``setback``). A bisection-refined mesh needs ``clearance`` @@ -1673,7 +1682,8 @@ def add_fault(mesh, faults, verbose=False, cut=True, exclude=None, faults = [faults] if mesh.dim == 3: - return _add_fault_sheets(mesh, faults, verbose=verbose, **place_kwargs) + return _add_fault_sheets(mesh, faults, verbose=verbose, + partition=partition, **place_kwargs) if place_kwargs: raise TypeError(f"add_fault: {sorted(place_kwargs)} apply to 3-D sheets only") @@ -1709,6 +1719,18 @@ def add_fault(mesh, faults, verbose=False, cut=True, exclude=None, candidate += 1 members[name] = values[name] = candidate + # The add-on is partitioned ONCE for the whole network, before the cut: + # each fault's cell star rank-interior on this level and on every add-on + # level below it (the replay needs its own star), the load balanced. The + # cut and the splits then move nothing. A level of the replay arrives + # partitioned (partition=False); the band meshed through the seams + # (across_seams) is split in place by design. + if uw.mpi.size > 1 and cut and partition and not across_seams: + from underworld3.utilities.addon_partition import (mark_polylines, + partition_addon) + mesh = partition_addon(mesh, mark_polylines([poly for _n, poly in segments]), + layers=1, verbose=verbose) + dm = mesh.dm if cut: for name, poly in segments: @@ -1727,17 +1749,11 @@ def add_fault(mesh, faults, verbose=False, cut=True, exclude=None, verbose=verbose, blind=blind, across_seams=across_seams) - # In parallel the balanced partition's cuts are attracted to the refined - # fault band, so a cut chain generally touches the seam. Redistribute - # ONCE, keyed on EVERY fault's facets together, BEFORE any split: each - # fault's cell star (plus one growth layer) moves to the rank that - # already owns most of the union, the chains become rank-interior, and - # the splits below run with serial topology. Doing it here rather than - # per split also keeps a network's prior pairings valid — a pairing - # cannot yet migrate through a redistribution, so split_fault refuses - # exactly the per-split case this pre-pass avoids. Label-only faults - # never touch the seam (the placement clipped the band there); one - # that does is a placement defect and is refused, collectively. + # The split needs every chain rank-interior. The add-on partition above + # made it so (and made it so on every add-on level), so a chain touching + # the seam here is a defect of the marking, not a case to redistribute. + # Label-only faults never touch the seam (the placement clipped the band + # there); one that does is a placement defect. Both refusals collective. if uw.mpi.size > 1 and dm.getDimension() == 2 and not across_seams: labels = [(name, values[name]) for name, _poly in segments] if _fault_labels_touch_seam(dm, labels): @@ -1746,7 +1762,10 @@ def add_fault(mesh, faults, verbose=False, cut=True, exclude=None, "add_fault(cut=False): a fault edge touches the " "partition seam. The embedded spines are rank-interior " "only when the band was placed with seams='ligament'.") - dm = _redistribute_fault_interior(dm, labels, verbose=verbose) + raise RuntimeError( + "add_fault: a cut chain touches the partition seam after the " + "add-on partition; the marking under-reached " + "(addon_partition.mark_polylines).") cut = Mesh(dm, simplex=mesh.dm.isSimplex(), coordinate_system_type=mesh.CoordinateSystem.coordinate_type, @@ -1758,6 +1777,27 @@ def add_fault(mesh, faults, verbose=False, cut=True, exclude=None, cut.regions = mesh.regions cut._parent_mesh_version = mesh._mesh_version mesh._registered_children.add(cut) + from underworld3.utilities.addon_partition import inherit_provenance + inherit_provenance(cut, mesh) + # A cut re-represents the parent's finest grid with the fault conformed, + # so the coarse levels of a base built with refinement >= 1 serve the cut + # mesh as its tail, the cut mesh being the finest level (the rule + # split_fault and Mesh._adopt_cut_child apply). A parent that owns an + # add-on tail hands it over in Mesh.add_fault, where the add-on levels are + # split too. Without this every solver on an add_fault child of a refined + # base fell back to algebraic multigrid (measured 2026-10-17: 2-D box, + # refinement=2, velocity KSP at its 200-iteration cap). + own_tail = getattr(mesh, "_custom_mg_coarse_meshes", None) + if own_tail is not None: + cut._custom_mg_coarse_meshes = list(own_tail) + cut._custom_mg_builder = getattr(mesh, "_custom_mg_builder", + "barycentric") + cut._custom_mg_fac_zone = None + elif len(getattr(mesh, "dm_hierarchy", []) or []) > 1: + cut._custom_mg_coarse_meshes = list(mesh._coarse_level_meshes()[:-1]) + cut._custom_mg_builder = getattr(mesh, "_custom_mg_builder", + "barycentric") + cut._custom_mg_fac_zone = None out = cut for name, poly in segments: @@ -1781,15 +1821,17 @@ def add_fault(mesh, faults, verbose=False, cut=True, exclude=None, return out -def _add_fault_sheets(mesh, faults, verbose=False, **place_kwargs): +def _add_fault_sheets(mesh, faults, verbose=False, partition=True, + **place_kwargs): """The 3-D form of :func:`add_fault`: each fault is a triangulated sheet (a ``FaultSurface`` or a ``(name, points, triangles)`` triple), placed into the mesh with :func:`place_surface.place_sheet` so its triangles become - interior faces, then every sheet is split (:func:`split_faults`, which - redistributes the network's cell stars rank-interior first). Like the 2-D - cut, the placed mesh carries NO multigrid tail of its own: the caller - (``Mesh.add_fault``) hands the parent's tail down, with its add-on levels - split the same way.""" + interior faces, then every sheet is split (:func:`split_faults`). In + parallel the add-on is partitioned for the sheets first (the placement's + own mask with two growth layers, on every add-on level), so the placement + gathers nothing and the split moves nothing. The placed mesh takes the + parent's multigrid tail; the caller (``Mesh.add_fault``) splits its add-on + levels the same way.""" from underworld3.discretisation import Mesh from underworld3.utilities.place_surface import place_sheet @@ -1806,6 +1848,16 @@ def _add_fault_sheets(mesh, faults, verbose=False, **place_kwargs): sheets.append((name, np.asarray(points, dtype=float), np.asarray(triangles, dtype=np.int64))) + if uw.mpi.size > 1 and partition: + from underworld3.utilities.addon_partition import (mark_sheets, + partition_addon) + import inspect + clearance = place_kwargs.get( + "clearance", + inspect.signature(place_sheet).parameters["clearance"].default) + mesh = partition_addon( + mesh, mark_sheets([(p, t) for _n, p, t in sheets], clearance), + layers=2, verbose=verbose) cut = mesh for name, points, triangles in sheets: boundaries = cut._boundaries_with(name) @@ -1821,6 +1873,21 @@ def _add_fault_sheets(mesh, faults, verbose=False, **place_kwargs): child.regions = cut.regions child._parent_mesh_version = cut._mesh_version child._surface_info = info + child._embedded_surfaces = ( + set(getattr(cut, "_embedded_surfaces", ()) or ()) | {name}) + from underworld3.utilities.addon_partition import inherit_provenance + inherit_provenance(child, cut) + own_tail = getattr(cut, "_custom_mg_coarse_meshes", None) + if own_tail is None and len(getattr(cut, "dm_hierarchy", []) or []) > 1: + # a mesh built with refinement >= 1: its coarse levels are the + # tail, the placed mesh re-representing its finest grid (the 2-D + # add_fault rule), in serial as in parallel + own_tail = list(cut._coarse_level_meshes()[:-1]) + if own_tail is not None: + child._custom_mg_coarse_meshes = list(own_tail) + child._custom_mg_builder = getattr(cut, "_custom_mg_builder", + "barycentric") + child._custom_mg_fac_zone = None cut._registered_children.add(child) cut = child out = split_faults(cut, [name for name, _p, _t in sheets], verbose=verbose) @@ -1866,12 +1933,14 @@ def _fault_labels_touch_seam(dm, labels): ``labels`` is a sequence of ``(name, value)`` pairs. The test matches the 2-D split's refusal exactly: a fault facet that is itself shared, or has a shared cone vertex, would put a chain vertex on the seam — - the configuration :func:`_redistribute_fault_interior` exists to - dissolve. Every rank returns the same verdict (allreduce), so the - caller's redistribute-or-not decision is collective by construction. + the configuration the add-on partition + (:func:`~underworld3.utilities.addon_partition.partition_addon`) + dissolves. Every rank returns the same verdict (allreduce), so the + caller's partition-or-not decision is collective by construction. """ if uw.mpi.size == 1: return False + from underworld3.utilities._plex_topology import closure_points shared = _shared_points(dm) pStart, _pEnd = dm.getChart() fS, fE = dm.getHeightStratum(1) @@ -1882,20 +1951,19 @@ def _fault_labels_touch_seam(dm, labels): if not (dm.hasLabel(name) and dm.getLabel(name).getStratumSize(int(value)) > 0): continue - for f in dm.getLabel(name).getStratumIS(int(value)).getIndices(): - f = int(f) - if not (fS <= f < fE): - continue - if shared[f - pStart] or any( - shared[int(q) - pStart] - for q in dm.getTransitiveClosure(f)[0]): - touch = True - break + f = np.asarray(dm.getLabel(name).getStratumIS(int(value)).getIndices(), + dtype=np.int64) + f = f[(f >= fS) & (f < fE)] + if not f.size: + continue + # the facet itself and every point below it in its closure + below = [closure_points(dm, f, d) for d in range(dm.getDimension() - 1)] + pts = np.concatenate([f] + [b[b >= 0] for b in below]) + touch = bool(shared[pts - pStart].any()) return bool(uw.mpi.comm.allreduce(touch, op=MPI.LOR)) -def _redistribute_fault_interior(dm, labels, verbose=False, groups=None, - pairs=None): +def _redistribute_fault_interior(dm, labels, verbose=False): """Redistribute a cut mesh so each fault's cell star is rank-interior. The default partition's balance cuts are ATTRACTED to the locally @@ -1911,65 +1979,35 @@ def _redistribute_fault_interior(dm, labels, verbose=False, groups=None, construction. The coincident pair blocks of the contact solve are rank-local for the same reason. ``labels`` is a sequence of ``(name, value)`` pairs — a NETWORK is redistributed in one move, - keyed on the union of its faults, which is what lets every split - that follows run without migrating any prior pairing. Returns a - NEW dm; the input is untouched. - - ``groups`` partitions ``labels`` into the faults that must share a - rank (a junction-connected cluster); each group is one region of - :func:`place_surface._gather_regions`, moved to its own rank — or not - moved at all when its star is already interior to one — so two - faults a domain apart are never gathered together (#670). Without - ``groups`` the whole network is one region, as before. - ``_fault_point_pairs``) the pairing is carried through the move and - ``(dm, pairs, n_dropped)`` comes back — see - :func:`redistribute_by_assignment`. + keyed on the union of its faults. Returns a NEW dm; the input is + untouched. + + The add-on pipelines (cut, placement, split) partition the whole add-on + hierarchy instead (``addon_partition.partition_addon``); this per-mesh + move remains for the extruded layer, which is built on one rank and + distributed afresh (``meshing.extrude.extrude_sheet``). """ from underworld3.utilities.place_surface import _gather_regions comm = dm.getComm().tompi4py() if comm.size == 1: - return (dm, pairs, 0) if pairs is not None else dm - if groups is None: - groups = [list(labels)] - - fS, fE = dm.getHeightStratum(1) - vS, vE = dm.getDepthStratum(0) - pStart, pEnd = dm.getChart() - ids = np.zeros(pEnd - pStart, dtype=np.int32) - for g, group in enumerate(groups, start=1): - for name, value in group: - if not (dm.hasLabel(name) - and dm.getLabel(name).getStratumSize(int(value)) > 0): - continue - for f in dm.getLabel(name).getStratumIS(int(value)).getIndices(): - if fS <= int(f) < fE: - for q in dm.getTransitiveClosure(int(f))[0]: - if vS <= int(q) < vE: - i = int(q) - pStart - ids[i] = max(ids[i], g) + return dm + from underworld3.utilities.addon_partition import mark_labels + ids = (mark_labels(list(labels), [list(labels)])(dm) > 0).astype(np.int32) if comm.allreduce(int(ids.max()) if ids.size else 0, op=MPI.MAX) == 0: names = sorted(name for name, _value in labels) raise RuntimeError( f"fault_split: no facets labelled {names} found on any rank.") - # the pairing rides the move as labels (stamped on the input, removed - # again below, so the input stays untouched) - names = stamp_point_pairs(dm, pairs) if pairs is not None else [] work, n_region, n_moved, owner, _canon = _gather_regions(dm, ids, layers=1) if work is dm: work = dm.clone() # the contract: a new dm, untouched input - new_pairs, dropped = (collect_point_pairs(work, names) - if pairs is not None else (None, 0)) - for lab in [l for n in names for l in _pair_label_names(n)]: - if dm.hasLabel(lab): - dm.removeLabel(lab) if verbose: names = sorted(name for name, _value in labels) uw.pprint(f"[fault_split] {names}: {len(owner)} region(s) of " f"{n_region} cells, {n_moved} moved (owners " f"{sorted(owner.values())}); local cells now " f"{work.getHeightStratum(0)[1]}") - return (work, new_pairs, dropped) if pairs is not None else work + return work def _pair_label_names(name): @@ -1979,10 +2017,10 @@ def _pair_label_names(name): def stamp_point_pairs(dm, pairs_by_fault): """Write a coincident-point pairing into two labels per fault (one value per pair, on its Minus point and on its Plus point) so a redistribution - carries it: labels ride every point migration, where the migration star - forest PETSc returns for an INITIAL distribution does not index the - stratified numbering (measured: three quarters of the points mapped to the - wrong depth). The ids are globally unique (each rank's pairs offset by the + carries it: labels ride every point migration, with no reading of the + migration star forest's point map (per-cell values ride the same move + through ``distributeField`` in :func:`redistribute_by_assignment`). The + ids are globally unique (each rank's pairs offset by the ranks before it). Returns the fault names, for :func:`collect_point_pairs`. Collective.""" comm = dm.getComm().tompi4py() @@ -2042,52 +2080,78 @@ def points_by_value(lab_name): return out, n_dropped -def redistribute_by_assignment(dm, assign, pairs=None): +def redistribute_by_assignment(dm, assign, pairs=None, fields=None): """Redistribute ``dm`` so local cell ``k`` goes to rank ``assign[k]``. - Returns ``(new_dm, new_pairs, n_dropped)``: the pairing ``pairs`` (a - mesh's ``_fault_point_pairs``, or None) carried through the move with - :func:`stamp_point_pairs` / :func:`collect_point_pairs`. Nothing moves when - every cell is already where it is asked to be (the input comes back). The - shell-partitioner idiom the fault-interior move and the level co-location - both use.""" + Returns ``(new_dm, new_pairs, n_dropped, new_fields)``: the pairing + ``pairs`` (a mesh's ``_fault_point_pairs``, or None) carried through the + move with :func:`stamp_point_pairs` / :func:`collect_point_pairs`, and the + per-cell integer ``fields`` (a list of arrays over the local cells, or + None) carried through ``distributeField`` on the migration star forest, + each returned over the new local cells. Nothing moves when every cell is + already where it is asked to be (the input comes back). The + shell-partitioner idiom of the fault-interior move and the add-on + partition.""" comm = dm.getComm().tompi4py() assign = np.asarray(assign, dtype=np.int32) if comm.size == 1 or not comm.allreduce(int((assign != comm.rank).sum())): - return dm, pairs, 0 + return dm, pairs, 0, fields work = dm.clone() names = stamp_point_pairs(work, pairs) if pairs else [] + cS, cE = work.getHeightStratum(0) + sec = vec = None + if fields: + # one cell-wise section, the fields interleaved; integers travel as + # doubles, exact below 2**53 + from underworld3.utilities._plex_topology import (fill_section, + section_offsets) + nf = len(fields) + cells = np.arange(cS, cE) + sec = PETSc.Section().create(comm=work.getComm()) + fill_section(sec, work, cells, nf) + vec = PETSc.Vec().createSeq(sec.getStorageSize(), comm=PETSc.COMM_SELF) + arr = vec.getArray() + offs = section_offsets(sec, cells) + for j, f in enumerate(fields): + arr[offs + j] = np.asarray(f, dtype=float) + vec.setArray(arr) part = work.getPartitioner() part.setType(PETSc.Partitioner.Type.SHELL) order = np.argsort(assign, kind="stable").astype(np.int32) sizes = np.bincount(assign, minlength=comm.size).astype(np.int32) part.setShellPartition(comm.size, sizes=sizes, points=order) - work.distribute() + sf = work.distribute() + new_fields = None + if fields: + newsec, newvec = work.distributeField(sf, sec, vec) + ncS, ncE = work.getHeightStratum(0) + narr = newvec.getArray() + offs = section_offsets(newsec, np.arange(ncS, ncE)) + new_fields = [np.rint(narr[offs + j]).astype(np.int64) if offs.size + else np.zeros(0, dtype=np.int64) for j in range(nf)] new_pairs, dropped = (collect_point_pairs(work, names) if pairs else (None, 0)) for lab in [l for n in names for l in _pair_label_names(n)]: if dm.hasLabel(lab): dm.removeLabel(lab) - return work, new_pairs, dropped + return work, new_pairs, dropped, new_fields def split_faults(mesh, names, verbose=False, groups=None, across_seams=False): """Split a NETWORK of already-labelled faults, any dimension, any np. - The parallel obstruction to sequential :func:`split_fault` calls is - the redistribution each one performs: a prior fault's pairing holds - point ids of the current distribution and cannot yet migrate. So the - network redistributes ONCE, keyed on the union of every fault's - facets (each star moves to the rank already owning most of it), and - every split that follows runs with serial topology — the same - pre-pass the 2-D ``add_fault`` performs for freshly cut chains, made + In parallel the add-on is partitioned ONCE for the whole network + (``addon_partition.partition_addon``: every fault's star rank-interior + on every add-on level, the load balanced), and every split that + follows runs with serial topology and moves nothing — the same + partition the 2-D ``add_fault`` performs before its cut, made available for faults that already carry labels (the 3-D embedded patches, a reloaded mesh). ``groups``, a list of lists of names, says which faults must share a - rank (a junction-connected cluster); each group is redistributed to - its own rank, and one whose star is already interior to a rank is - not moved (#670). Faults not named in any group form one more. + rank (a junction-connected cluster); each group is one region of the + partition, sent to the rank already holding most of it. Faults not + named in any group form one more. ``across_seams`` splits THROUGH the seams instead (2-D, the band meshed through them): nothing is redistributed, see @@ -2107,20 +2171,13 @@ def split_faults(mesh, names, verbose=False, groups=None, if rest: label_groups.append([(n, value[n]) for n in rest]) if _fault_labels_touch_seam(mesh.dm, labels): - dm = _redistribute_fault_interior(mesh.dm, labels, - verbose=verbose, - groups=label_groups) - out = Mesh(dm, simplex=mesh.dm.isSimplex(), - coordinate_system_type=( - mesh.CoordinateSystem.coordinate_type), - qdegree=mesh.qdegree, boundaries=mesh.boundaries, - verbose=False) - out.parent = mesh - out._relationship_kind = "refinement" - out._refine_dofs_coincide = False - out.regions = mesh.regions - out._parent_mesh_version = mesh._mesh_version - mesh._registered_children.add(out) + # the add-on partition: every star rank-interior on every level, + # the pairing of any prior fault carried through the move + from underworld3.utilities.addon_partition import (mark_labels, + partition_addon) + out = partition_addon(mesh, mark_labels(labels, label_groups), + layers=1, verbose=verbose) + mesh = out for n in names: out = split_fault(out, n, verbose=verbose, across_seams=across_seams) # The network's child INHERITS a mesh-owned geometric-MG tail — the @@ -2131,6 +2188,8 @@ def split_faults(mesh, names, verbose=False, groups=None, # NOT inherited (a split fault needs no patch — the keying ruling); # callers that key a zone re-adopt on the child explicitly. own_tail = getattr(mesh, "_custom_mg_coarse_meshes", None) + if own_tail is None and len(getattr(mesh, "dm_hierarchy", []) or []) > 1: + own_tail = mesh._coarse_level_meshes()[:-1] # as in split_fault if (own_tail is not None and getattr(out, "_custom_mg_coarse_meshes", None) is None): out._custom_mg_coarse_meshes = list(own_tail) @@ -2201,11 +2260,11 @@ def split_fault(mesh, name, orientation=None, verbose=False, fault moves, re-cut the base mesh at the new position and split again — the same non-cumulative pattern as :meth:`Mesh.adapt`. - The split mesh carries NO geometric-multigrid tail. The coarse levels do - not carry the fault, so an essential condition on it would constrain zero - coarse DOFs and leave the custom-P coarse operator singular (see the - warning on :meth:`Mesh.add_conforming_surface`); solvers on this mesh get - their algebraic-multigrid defaults instead. + The split mesh takes the multigrid tail of ``mesh``: its own registered + levels, or the coarse levels of a mesh built with ``refinement >= 1``. + ``Mesh.add_fault`` splits the add-on levels of that tail as well; a tail + left welded under a split finest level is solved through the rotated + fault frame (see ``utilities.fault_mg``). Parameters ---------- @@ -2244,28 +2303,18 @@ def split_fault(mesh, name, orientation=None, verbose=False, if uw.mpi.size > 1 and not across: labels = [(name, int(mesh.boundaries[name].value))] # The split requires the fault's cell star rank-interior, and - # balance cuts are attracted to the refined fault region — gather - # the (thin) star onto one rank first. 3-D always redistributes - # (the patch star essentially always straddles the seam on a - # graded mesh); 2-D redistributes only when the chain actually - # touches the seam, which is what lets add_fault's one-shot - # union redistribution make every subsequent per-fault split a - # no-move here. Both branches of the decision are collective. + # balance cuts are attracted to the refined fault region. A mesh + # that add_fault or add_conforming_sheet partitioned is already so + # and nothing moves here; otherwise the add-on is partitioned for + # this fault. The decision is collective. if _fault_labels_touch_seam(source_dm, labels): - if getattr(mesh, "_fault_point_pairs", {}): - # a prior fault's pairing holds point ids of the CURRENT - # distribution; redistribution renumbers every point, so - # carrying it through needs pairing migration along the - # migration SF — not built yet. Refuse loudly (and - # identically on every rank: the pairing dict and the - # seam verdict agree across ranks by construction). - raise NotImplementedError( - "fault_split: multi-fault networks in parallel need " - "pairing migration through the redistribution — pass " - "the whole network to ONE add_fault call (2-D), or " - "split the network in serial.") - source_dm = _redistribute_fault_interior( - source_dm, labels, verbose=verbose) + # the add-on partition: the star rank-interior on every level, + # a prior fault's pairing carried through the move as labels + from underworld3.utilities.addon_partition import (mark_labels, + partition_addon) + mesh = partition_addon(mesh, mark_labels(labels), layers=1, + verbose=verbose) + source_dm = mesh.dm splitter = (split_along_label if source_dm.getDimension() == 2 else split_along_label_3d) value = int(mesh.boundaries[name].value) @@ -2417,8 +2466,32 @@ def carry_keys(d, what): child._parent_mesh_version = mesh._mesh_version child._fault_point_pairs = dict(prior) child._fault_point_pairs[name] = pairs + if getattr(mesh, "_embedded_surfaces", None): + child._embedded_surfaces = set(mesh._embedded_surfaces) + from underworld3.utilities.addon_partition import inherit_provenance + inherit_provenance(child, mesh) if prior_normals or seam_normals: child._fault_seam_normals = dict(prior_normals) child._fault_seam_normals[name] = seam_normals + # The split re-represents the same grid (only the fault vertices are + # duplicated), so a mesh-owned geometric-MG tail serves it unchanged — + # the rule add_fault and split_faults already applied to their results. + # Here too, or the direct 3-D pipeline (adapt -> add_conforming_sheet -> + # split_fault) silently solved on GAMG. The FAC zone is not inherited: + # a split fault needs no patch. + own_tail = getattr(mesh, "_custom_mg_coarse_meshes", None) + if own_tail is None and len(getattr(mesh, "dm_hierarchy", []) or []) > 1: + # A mesh built with refinement >= 1 (a refined box, a refined box with + # a conforming patch) carries its coarse levels natively; the cut child + # takes them through _coarse_level_meshes, and so does the split. The + # finest level is this mesh itself, which the split re-represents, so + # it is not a level of its own (a refined cuboid needs no second mesh + # to get geometric multigrid — ruling 2026-10-17). + own_tail = mesh._coarse_level_meshes()[:-1] + if own_tail is not None: + child._custom_mg_coarse_meshes = list(own_tail) + child._custom_mg_builder = getattr(mesh, "_custom_mg_builder", + "barycentric") + child._custom_mg_fac_zone = None mesh._registered_children.add(child) return child diff --git a/src/underworld3/utilities/line_cut.py b/src/underworld3/utilities/line_cut.py index 515baf5ee..fff7e0288 100644 --- a/src/underworld3/utilities/line_cut.py +++ b/src/underworld3/utilities/line_cut.py @@ -108,7 +108,9 @@ def _edge_vertices(dm): """(n_edges, 2) local vertex indices of every edge, in cone order.""" vS, _vE = dm.getDepthStratum(0) eS, eE = dm.getDepthStratum(1) - return np.array([dm.getCone(e) for e in range(eS, eE)], dtype=np.int64) - vS + # (0, 2) on a rank that holds no edge, so the edge arithmetic still applies + return np.array([dm.getCone(e) for e in range(eS, eE)], + dtype=np.int64).reshape(-1, 2) - vS def _coords(dm): diff --git a/src/underworld3/utilities/nondimensional.py b/src/underworld3/utilities/nondimensional.py index 0976b6c09..27a71c8c5 100644 --- a/src/underworld3/utilities/nondimensional.py +++ b/src/underworld3/utilities/nondimensional.py @@ -31,7 +31,7 @@ def derive_scaling_coefficients(model): fundamental_scales = model._fundamental_scales # Process all registered variables - for name, var in model._variables.items(): + for name, var in list(model._variables.items()): if hasattr(var, "dimensionality") and var.dimensionality: # Try to derive scaling coefficient from dimensionality scale = _derive_scale_for_variable(var, fundamental_scales, model._reference_quantities) @@ -464,7 +464,7 @@ def apply_nondimensional_scaling(expr, model): # Find all function symbols (variables) for func_symbol in expr.atoms(sympy.Function): # Try to find corresponding variable in model - for var_name, var in model._variables.items(): + for var_name, var in list(model._variables.items()): if hasattr(var, "sym"): # Check if this function matches the variable's symbolic form if str(func_symbol.func) == str(var.sym.func): diff --git a/src/underworld3/utilities/place_surface.py b/src/underworld3/utilities/place_surface.py index becd3a07e..fd567acc7 100644 --- a/src/underworld3/utilities/place_surface.py +++ b/src/underworld3/utilities/place_surface.py @@ -67,6 +67,30 @@ refused with the reason; interior surfaces, the fault case, run at any rank count. +TODO(BUG): the sheet's cavity shell fails to close at fine resolution in an +orientation- and grading-dependent way, and raising ``clearance`` does not +help. Measured 2026-09-17 (3-D disc, base 0.2, h_near 0.0125, three adapt +levels): with the refinement band reaching 0.4 from the fault, a 60-orientation +sweep placed 60/60; with the band tightened to 0.1, 20 of 24 shards died on +their first sample with "the sheet's cavity shell did not close in 20 growth +rounds; raise `clearance`". On one fixed orientation, band 0.1 SUCCEEDS at the +default clearance 0.6 (and at 0.9 and 1.2) while band 0.2 FAILS at 0.6 — so it +is not monotonic in either tightness or clearance, and the growth-round cap is +being hit for reasons that depend on the particular adapted mesh. A wide band +is the working configuration at this resolution. + +TODO(BUG): the gather is never undone, so placement leaves the mesh badly +IMBALANCED and every solve on it is effectively serial. Measured 2026-09-17 +(3-D sheet on a box, base 0.2 / h_near 0.05 / one adapt level, np=4): cells +per rank are [5177, 6137, 5191, 5254] (max/mean 1.13) after ``adapt``, and +[1399, 17348, 1132, 1284] (max/mean 3.28) after ``add_conforming_sheet`` — +82% of the mesh on the surgery rank. ``split_fault`` leaves that untouched +(its own redistribute is bounded by the fault star, as documented), so the +cost is here and not in the contact stream. Consequence: an MPI fault solve +is SLOWER than serial — a 48k-cell 3-D sample solves in 202 s at np=1 and +281 s at np=2. A rebalancing redistribute after the collective rebuild is +what is missing. + Both dimensions, one fill. The 2-D curve (:func:`place_along_lines`) and the 3-D sheet (:func:`place_sheet`) both delegate their cavity fill to gmsh — constrained triangulation with Steiner insertion is exactly what a mesh @@ -1306,11 +1330,10 @@ def place_along_lines(dm, lines, label=CUT_LABEL, label_value=1, def _tet_vertices(dm): """(n_cells, 4) local vertex indices of every tetrahedron.""" - vS, vE = dm.getDepthStratum(0) + from underworld3.utilities._plex_topology import closure_points + vS, _vE = dm.getDepthStratum(0) cS, cE = dm.getHeightStratum(0) - return np.array([[int(p) - vS for p in dm.getTransitiveClosure(c)[0] - if vS <= p < vE] for c in range(cS, cE)], - dtype=np.int64).reshape(cE - cS, 4) + return (closure_points(dm, np.arange(cS, cE), 0) - vS).reshape(cE - cS, 4) def _face_vertex_triple(dm, f, vS): @@ -1570,9 +1593,10 @@ def _vertex_h_3d(dm, cells, n_vertices): vS, _vE = dm.getDepthStratum(0) pStart, pEnd = dm.getChart() work = np.full(pEnd - pStart, np.inf) - for c, tet in enumerate(cells): - idx = tet + vS - pStart - work[idx] = np.minimum(work[idx], h_cell[c]) + if len(cells): + cells = np.asarray(cells, dtype=np.int64) + np.minimum.at(work, cells + vS - pStart, + np.broadcast_to(h_cell[:, None], cells.shape)) work = _propagate_vertex(dm, work, MPI.MIN, np.minimum) return work[vS - pStart: vS - pStart + n_vertices], h_cell @@ -1588,13 +1612,15 @@ def _true_wall_vertex_mask(dm, n_vertices): """ vS, _vE = dm.getDepthStratum(0) pStart, pEnd = dm.getChart() + from underworld3.utilities._plex_topology import closure_points, supports shared = _shared_point_flags(dm) mark = np.zeros(pEnd - pStart, dtype=np.int32) - for f in range(*dm.getHeightStratum(1)): - if len(dm.getSupport(f)) == 1 and not shared[f - pStart]: - for q in dm.getTransitiveClosure(f)[0]: - if dm.getPointDepth(int(q)) == 0: - mark[int(q) - pStart] = 1 + fS, fE = dm.getHeightStratum(1) + faces = np.arange(fS, fE) + n_support = (supports(dm, faces) >= 0).sum(axis=1) + wall = faces[(n_support == 1) & (shared[faces - pStart] == 0)] + if wall.size: + mark[closure_points(dm, wall, 0).ravel() - pStart] = 1 mark = _propagate_vertex(dm, mark, MPI.MAX, np.maximum) return mark[vS - pStart: vS - pStart + n_vertices] == 1 @@ -1632,8 +1658,9 @@ def find(a): return (comp + 1).astype(np.int32) -def _gather_region(dm, vertex_mark_chart, verbose=False): - """Redistribute so every marked vertex's cell star (+1 layer) is one rank's. +def _gather_region(dm, vertex_mark_chart, verbose=False, layers=1): + """Redistribute so every marked vertex's cell star (+ ``layers``) is one + rank's. The one-region form of :func:`_gather_regions` (a 0/1 mask is one region). Returns ``(new_dm, n_moved)``: the global count of cells @@ -1642,9 +1669,94 @@ def _gather_region(dm, vertex_mark_chart, verbose=False): it. The input is untouched. """ ids = (np.asarray(vertex_mark_chart) != 0).astype(np.int32) - work, n_region, _n_moved, _owner, _canon = _gather_regions( - dm, ids, verbose=verbose, layers=1) - return work, n_region + work, _n_region, n_moved, _owner, _canon = _gather_regions( + dm, ids, verbose=verbose, layers=layers) + return work, n_moved + + +def _merge_regions(pairs, n_ids, comm): + """Union-find over the gathered set of touching region ``pairs`` — + identical on every rank. Returns ``canon`` mapping every region id in + ``1..n_ids`` to its merged id. Collective.""" + parent = list(range(n_ids + 1)) + + def find(a): + while parent[a] != a: + parent[a] = parent[parent[a]] + a = parent[a] + return a + + for a, b in sorted(set().union(*comm.allgather(pairs))): + ra, rb = find(a), find(b) + if ra != rb: + parent[max(ra, rb)] = min(ra, rb) + return {k: find(k) for k in range(1, n_ids + 1)} + + +def _region_cells(dm, vertex_region_chart, layers=1): + """The cells of each marked region: the marked vertices' cell star grown by + ``layers``, with touching regions merged (the marking rule of + :func:`_gather_regions`, which applies it; :func:`addon_partition.partition_addon` + lifts it to its units). Returns ``(cell_region, canon, regions)``: the + merged region id per local cell (``0`` outside every region), the id map + and the sorted list of merged ids. Collective; the dm is read, not changed.""" + comm = dm.getComm().tompi4py() + ids_in = np.asarray(vertex_region_chart, dtype=np.int32) + n_ids = int(comm.allreduce(int(ids_in.max()) if ids_in.size else 0, + op=MPI.MAX)) + cS, cE = dm.getHeightStratum(0) + pStart, pEnd = dm.getChart() + pairs = set() # (a, b): regions that touch, to be merged + + def add_pairs(a, b): + if a.size: + for x, y in np.unique(np.column_stack([a, b]), axis=0).tolist(): + pairs.add((int(x), int(y))) + + def reconcile(chart): + # every rank agrees on every point it can see; where a rank's own + # id loses to a neighbour's under MAX the two regions touch + before = chart.copy() + after = _propagate_vertex(dm, chart, MPI.MAX, np.maximum) + touch = (before > 0) & (after != before) + add_pairs(before[touch], after[touch]) + return after + + from underworld3.utilities._plex_topology import closure_points + # each cell's vertices, as chart indices + CV = closure_points(dm, np.arange(cS, cE), 0) - pStart + cell_region = np.zeros(cE - cS, dtype=np.int32) + + def note_touching(ids, best): + # every nonzero id that differs from its row's largest touches it + rows, cols = np.nonzero((ids > 0) & (ids != best[:, None])) + add_pairs(ids[rows, cols], best[rows]) + + def claim_cells(chart): + # a cell takes the largest id among its own and its vertices'; the + # ids that meet in one cell touch + ids = np.column_stack([cell_region, chart[CV]]) + best = ids.max(axis=1) + note_touching(ids, best) + cell_region[:] = best + + claim_cells(reconcile(ids_in.copy())) + # Growth layers: a point is unshared exactly when all its incident + # cells are co-resident, so each layer makes the previous cells' + # closures unshared. + for _grow in range(int(layers)): + layer = np.zeros(pEnd - pStart, dtype=np.int32) + held = np.broadcast_to(cell_region[:, None], CV.shape) + np.maximum.at(layer, CV, held) + # a vertex in cells of two ids: each cell's id touches the vertex's + note_touching(held.reshape(-1, 1), layer[CV].reshape(-1)) + claim_cells(reconcile(layer)) + + canon = _merge_regions(pairs, n_ids, comm) + lookup = np.zeros(n_ids + 1, dtype=np.int32) + for k, v in canon.items(): + lookup[k] = v + return lookup[cell_region], canon, sorted(set(canon.values())) def _gather_regions(dm, vertex_region_chart, verbose=False, layers=1): @@ -1691,69 +1803,7 @@ def _gather_regions(dm, vertex_region_chart, verbose=False, layers=1): work = dm.clone() cS, cE = work.getHeightStratum(0) - vS, vE = work.getDepthStratum(0) - pStart, pEnd = work.getChart() - pairs = set() # (a, b): regions that touch, to be merged - - def reconcile(chart): - # every rank agrees on every point it can see; where a rank's own - # id loses to a neighbour's under MAX the two regions touch - before = chart.copy() - after = _propagate_vertex(work, chart, MPI.MAX, np.maximum) - touch = (before > 0) & (after != before) - for a, b in zip(before[touch], after[touch]): - pairs.add((int(a), int(b))) - return after - - cell_region = np.zeros(cE - cS, dtype=np.int32) - - def claim_cells(chart): - for v in range(vS, vE): - k = int(chart[v - pStart]) - if k == 0: - continue - for q in work.getTransitiveClosure(v, useCone=False)[0]: - if cS <= int(q) < cE: - c = int(q) - cS - if cell_region[c] and cell_region[c] != k: - pairs.add((int(cell_region[c]), k)) - cell_region[c] = max(cell_region[c], k) - - claim_cells(reconcile(ids_in.copy())) - # Growth layers: a point is unshared exactly when all its incident - # cells are co-resident, so each layer makes the previous cells' - # closures unshared. - for _grow in range(int(layers)): - layer = np.zeros(pEnd - pStart, dtype=np.int32) - for c in np.flatnonzero(cell_region): - k = int(cell_region[c]) - for q in work.getTransitiveClosure(int(c) + cS)[0]: - if vS <= int(q) < vE: - i = int(q) - pStart - if layer[i] and layer[i] != k: - pairs.add((int(layer[i]), k)) - layer[i] = max(layer[i], k) - claim_cells(reconcile(layer)) - - # merge touching regions: union-find on the gathered pair set, the same - # on every rank - parent = list(range(n_ids + 1)) - - def find(a): - while parent[a] != a: - parent[a] = parent[parent[a]] - a = parent[a] - return a - - for a, b in sorted(set().union(*comm.allgather(pairs))): - ra, rb = find(a), find(b) - if ra != rb: - parent[max(ra, rb)] = min(ra, rb) - canon = {k: find(k) for k in range(1, n_ids + 1)} - for c in np.flatnonzero(cell_region): - cell_region[c] = canon[int(cell_region[c])] - - regions = sorted(set(canon.values())) + cell_region, canon, regions = _region_cells(work, ids_in, layers=layers) local = np.array([int((cell_region == k).sum()) for k in regions], dtype=np.int64) counts = np.asarray(comm.allgather(local)) # (size, n_regions) @@ -1795,7 +1845,7 @@ def find(a): def _carve_cavity_3d(dm, X, cells, sheet_pts, sheet_tris, clearance, held_cells, h_vertex, on_wall, shared_chart, - open_deletable=None, open_near=None): + open_deletable=None, open_near=None, extra_victims=None): """Victims, dropped tets and the closed cavity shell around the sheet. The same two-part rule as 2-D — vertices within the clearance go, and any @@ -1829,6 +1879,9 @@ def _carve_cavity_3d(dm, X, cells, sheet_pts, sheet_tris, clearance, held_vertex[cells[c]] = True victim = ((d_sheet < clearance * h_vertex) & (~on_wall | on_open) & ~held_vertex) + if extra_victims is not None: + # vertices a previous fill of this cavity left flat cells on + victim |= extra_victims & ~on_wall & ~held_vertex drop = victim[cells].any(axis=1) cen = X[cells].mean(axis=1) @@ -2468,6 +2521,15 @@ def _closed_shell_3d(dm, X, cells, drop, victim, held_cells, shared_chart, original mesh did not). Growing the drop at every non-manifold edge merges the wedges; dropping more cells only enlarges the fill. + An edge-manifold shell can still be one of two shapes gmsh fills wrongly, + both measured on newest-vertex-bisection children at the default + clearance: a KEPT cell wholly enclosed by the cavity, whose faces close a + separate shell component that gmsh fills too (the fill overlaps the kept + cell and the domain volume grows by exactly its volume); and a PINCHED + vertex, where the shell faces around the vertex form more than one fan + (gmsh returns zero-volume tets there). Both are grown out the same way: + the enclosed cells join the drop, and so does every pinched vertex's star. + ``open_vertex`` (a vertex mask) lets the cavity OPEN onto the boundary near an outcrop — the bowl. A dropped cell's wall face whose vertices all carry the mask becomes a CAP face (returned separately; the caller @@ -2529,16 +2591,22 @@ def _closed_shell_3d(dm, X, cells, drop, victim, held_cells, shared_chart, for e in ((a, b), (a, c), (b, c)): edge_count[e] += 1 bad = [e for e, k in edge_count.items() if k != 2] - if not bad: - break - for a, b in bad: - grow = (cells == a).any(axis=1) & (cells == b).any(axis=1) - if (grow & held).any(): - raise RuntimeError( - f"closing the {noun}'s cavity shell needs a cell held " - "for a surface already embedded; move the object away " - "or raise `clearance`.") - drop |= grow + grow = np.zeros(len(cells), dtype=bool) + if bad: + for a, b in bad: + grow |= (cells == a).any(axis=1) & (cells == b).any(axis=1) + else: + grow |= _enclosed_kept_cells(face_support, drop) + for v in _pinched_shell_vertices(shell + cap_faces): + grow |= (cells == v).any(axis=1) + if not grow.any(): + break + if (grow & held).any(): + raise RuntimeError( + f"closing the {noun}'s cavity shell needs a cell held " + "for a surface already embedded; move the object away " + "or raise `clearance`.") + drop |= grow else: raise RuntimeError( f"the {noun}'s cavity shell did not close in 20 growth rounds; " @@ -2550,6 +2618,72 @@ def _closed_shell_3d(dm, X, cells, drop, victim, held_cells, shared_chart, return shell, cap_faces, drop +def _enclosed_kept_cells(face_support, drop): + """Kept cells that cannot reach an open face without crossing the cavity. + + An open face has one cell on this rank: the domain wall, or a partition + seam on a distributed mesh. Kept cells are flooded from those faces + through faces shared with other kept cells; whatever the flood misses is + wholly enclosed by dropped cells. Returns a cell mask. + """ + neighbours = [[] for _ in range(len(drop))] + reached = np.zeros(len(drop), dtype=bool) + stack = [] + for support in face_support.values(): + if len(support) == 1: + c = support[0] + if not drop[c] and not reached[c]: + reached[c] = True + stack.append(c) + elif not drop[support[0]] and not drop[support[1]]: + neighbours[support[0]].append(support[1]) + neighbours[support[1]].append(support[0]) + while stack: + c = stack.pop() + for j in neighbours[c]: + if not reached[j]: + reached[j] = True + stack.append(j) + return ~drop & ~reached + + +def _pinched_shell_vertices(faces): + """Shell vertices whose incident faces form more than one fan. + + ``faces`` is a list of ``(face, vertices)``. Around a manifold vertex the + incident faces are connected through the edges they share at that + vertex; a vertex where two parts of the shell merely touch has two or + more such fans. + """ + from collections import defaultdict + + around = defaultdict(list) + for k, (_f, verts) in enumerate(faces): + for v in verts: + around[v].append(k) + pinched = [] + for v, incident in around.items(): + through = defaultdict(list) + for k in incident: + for w in faces[k][1]: + if w != v: + through[w].append(k) + fan = {incident[0]} + frontier = [incident[0]] + while frontier: + k = frontier.pop() + for w in faces[k][1]: + if w == v: + continue + for j in through[w]: + if j not in fan: + fan.add(j) + frontier.append(j) + if len(fan) != len(incident): + pinched.append(v) + return pinched + + def _gmsh_fill_3d(shell_xyz, shell_tris, sheet_pts, sheet_tris, h, cap=None): """Tetrahedralise inside the shell with the sheet embedded, via gmsh. @@ -2661,6 +2795,40 @@ def cap_tag_of(k): gmsh.finalize() +# Refills place_sheet will try before refusing. One is measured to clear +# every flat cell seen; the rest are margin, each deleting a few more shell +# vertices. +_SHEET_FILL_ATTEMPTS = 5 + + +def _flat_fill_shell_nodes(fill_pts, fill_tets, n_shell, h, rel_tol=1e-8): + """Shell nodes of the fill's flat tets, as fill-local indices. + + A tet is flat when its volume is below ``rel_tol * h**3``: the genuine + cells of a fill sit near ``1e-3 h**3`` at worst, the flat ones near + ``1e-17 h**3``, so the threshold separates them by orders of magnitude on + either side. Nodes ``< n_shell`` are shell vertices, which the carve can + delete; a flat tet with no shell vertex is built from sheet points alone + and no refill of the cavity can remove it, so that refuses. + """ + tets = np.asarray(fill_tets) + P = fill_pts[tets] + volume = np.abs(np.einsum("ij,ij->i", + np.cross(P[:, 1] - P[:, 0], P[:, 2] - P[:, 0]), + P[:, 3] - P[:, 0])) / 6.0 + flat = tets[volume < rel_tol * h**3] + if not len(flat): + return np.zeros(0, dtype=np.int64) + nodes = np.unique(flat.ravel()) + nodes = nodes[nodes < n_shell] + if not len(nodes): + raise RuntimeError( + "the cavity fill made a flat cell from sheet points alone; the " + "sheet is finer than the mesh can take — re-triangulate it with " + "`size=`.") + return nodes + + def _attach_uninterp_vertex_sf(new, dm, v_old_to_compact, nc_new, nroots, extra_leaves=()): """Give the uninterpolated plex its vertex star-forest, BEFORE interpolate. @@ -3139,6 +3307,16 @@ def place_sheet(dm, points, triangles, label=CUT_LABEL, label_value=1, # so a wider mask here only moves cells the surgery never touches (#670: # a +2 margin gathered 91% of a fixture whose cavity was 6%). The carve # asserts nothing shared afterwards, so an under-reach is loud. + # The Mesh API (add_conforming_sheet, add_fault) partitions the add-on + # for the sheet BEFORE calling here, with this mask plus two growth + # layers on every add-on level (addon_partition.mark_sheets), so the + # gather below finds the region rank-interior and moves nothing. A dm + # handed in directly is gathered with the same two layers: the carve's + # growth rounds (_carve_cavity_3d's enclosed-cell and pinched-vertex + # rounds) take whole vertex stars at the shell, one more diameter out, + # and with one layer the grown cavity touched a shared point (measured + # 2026-10-17 on a partitioned adapt child at 2 and 8 ranks; + # tests/parallel/test_1097). mark = np.zeros(pEnd - pStart, dtype=np.int32) mark[np.flatnonzero(d_sheet < (clearance + 1.0) * h_vertex) + vS - pStart] = 1 @@ -3147,7 +3325,7 @@ def place_sheet(dm, points, triangles, label=CUT_LABEL, label_value=1, [_owned_cell_volume(dm)], dtype=float) comm.Allreduce(MPI.IN_PLACE, volume_before, op=MPI.SUM) - dm_work, moved = _gather_region(dm, mark, verbose=verbose) + dm_work, moved = _gather_region(dm, mark, verbose=verbose, layers=2) if moved: vS, vE = dm_work.getDepthStratum(0) pStart, pEnd = dm_work.getChart() @@ -3196,95 +3374,122 @@ def place_sheet(dm, points, triangles, label=CUT_LABEL, label_value=1, deletable, near, _regions = _outcrop_frame_3d( X, on_wall, dom_verts, dom_tris, dom_region, sheet_pts, chain_edges) - victims, drop_ids, shell, cap_faces = _carve_cavity_3d( - dm_work, X, cells, sheet_pts, sheet_tris, clearance, - held_cells, h_vertex, on_wall, shared, - open_deletable=deletable, open_near=near) - # The gather's contract, asserted: nothing the surgery touches is - # shared. A violation is a marking defect and must be loud. - touched = set() - for c in drop_ids: - for q in dm_work.getTransitiveClosure(int(c) + cS)[0]: - touched.add(int(q)) - if any(shared[q - pStart] for q in touched): - raise RuntimeError( - "place_sheet internal: the gathered region touches a " - "shared point; the gather mask under-reached.") - - shell_vert_ids = sorted( - {v for _f, verts in shell for v in verts} - | {v for _f, verts in cap_faces for v in verts}) - local = {v: i for i, v in enumerate(shell_vert_ids)} - shell_xyz = X[shell_vert_ids] - shell_tris = np.array([[local[v] for v in verts] - for _f, verts in shell], dtype=np.int64) - - cap_payload = None - removed_wall = [] - if cap_faces: - # Labels each removed wall face carried, read PER FACE - # before the rebuild forgets them — the bowl can span - # several walls (a curved boundary's facets), so each new - # wall triangle takes the labels of the old face it lies - # on, not a set common to the whole bowl (the volume - # path's rule). - names = [dm_work.getLabelName(i) - for i in range(dm_work.getNumLabels()) - if dm_work.getLabelName(i) - not in reconnect._TOPOLOGY_LABELS] - for f, verts in cap_faces: - pairs_f = [] - for name in names: - val = dm_work.getLabel(name).getValue(int(f)) - if val >= 0: - pairs_f.append((name, int(val))) - removed_wall.append((X[np.asarray(verts)].copy(), - pairs_f)) - - cap_tris_mesh = np.array([verts for _f, verts in cap_faces], - dtype=np.int64) - d_cap, at_cap = _nearest_facet( - X[cap_tris_mesh].mean(axis=1), dom_verts, dom_tris) - if (d_cap > 1e-9).any(): + extra_victims = np.zeros(len(X), dtype=bool) + for _attempt in range(_SHEET_FILL_ATTEMPTS): + victims, drop_ids, shell, cap_faces = _carve_cavity_3d( + dm_work, X, cells, sheet_pts, sheet_tris, clearance, + held_cells, h_vertex, on_wall, shared, + open_deletable=deletable, open_near=near, + extra_victims=extra_victims) + # The gather's contract, asserted: nothing the surgery + # touches is shared. A violation is a marking defect and + # must be loud. + touched = set() + for c in drop_ids: + for q in dm_work.getTransitiveClosure(int(c) + cS)[0]: + touched.add(int(q)) + if any(shared[q - pStart] for q in touched): raise RuntimeError( - "an outcrop bowl wall face lies off the gathered " - "boundary complex; the wall mask and the complex " - "disagree") - alive = np.ones(len(X), dtype=bool) - alive[np.asarray(victims, dtype=np.int64)] = False - cap_nodes, chain_nodes, cap_extra, cap_tris = \ - _outcrop_collar_3d( - X, alive, cap_tris_mesh, dom_region[at_cap], - dom_planes, sheet_pts, - np.zeros((0, 3), dtype=np.int64), - np.zeros(0, dtype=np.int64), None, chain=chain) - cap_payload = { - "rim_shell_local": [local[v] for v in cap_nodes], - "chain_sheet_local": list(chain_nodes), - "tris": cap_tris, - "extra_xyz": cap_extra, - } - - fill = _gmsh_fill_3d(shell_xyz, shell_tris, sheet_pts, - sheet_tris, h, cap=cap_payload) - (fill_pts, fill_tets, sheet_out, moved_nodes, n_shell, - cap_out) = fill - if moved_nodes: - raise RuntimeError( - f"the fill moved {moved_nodes} constrained node(s); the " - "cavity cannot be sewn back. A defect, not a tolerance.") - if sheet_out is None or len(sheet_out) != len(sheet_tris): - raise RuntimeError( - "the fill remeshed the sheet " - f"({0 if sheet_out is None else len(sheet_out)} " - f"triangles for {len(sheet_tris)} given).") - if cap_payload is not None and ( - cap_out is None - or len(cap_out) != len(cap_payload["tris"])): + "place_sheet internal: the gathered region touches a " + "shared point; the gather mask under-reached.") + + shell_vert_ids = sorted( + {v for _f, verts in shell for v in verts} + | {v for _f, verts in cap_faces for v in verts}) + local = {v: i for i, v in enumerate(shell_vert_ids)} + shell_xyz = X[shell_vert_ids] + shell_tris = np.array([[local[v] for v in verts] + for _f, verts in shell], + dtype=np.int64) + + cap_payload = None + removed_wall = [] + if cap_faces: + # Labels each removed wall face carried, read PER FACE + # before the rebuild forgets them — the bowl can span + # several walls (a curved boundary's facets), so each + # new wall triangle takes the labels of the old face it + # lies on, not a set common to the whole bowl (the + # volume path's rule). + names = [dm_work.getLabelName(i) + for i in range(dm_work.getNumLabels()) + if dm_work.getLabelName(i) + not in reconnect._TOPOLOGY_LABELS] + for f, verts in cap_faces: + pairs_f = [] + for name in names: + val = dm_work.getLabel(name).getValue(int(f)) + if val >= 0: + pairs_f.append((name, int(val))) + removed_wall.append((X[np.asarray(verts)].copy(), + pairs_f)) + + cap_tris_mesh = np.array( + [verts for _f, verts in cap_faces], dtype=np.int64) + d_cap, at_cap = _nearest_facet( + X[cap_tris_mesh].mean(axis=1), dom_verts, dom_tris) + if (d_cap > 1e-9).any(): + raise RuntimeError( + "an outcrop bowl wall face lies off the gathered " + "boundary complex; the wall mask and the complex " + "disagree") + alive = np.ones(len(X), dtype=bool) + alive[np.asarray(victims, dtype=np.int64)] = False + cap_nodes, chain_nodes, cap_extra, cap_tris = \ + _outcrop_collar_3d( + X, alive, cap_tris_mesh, dom_region[at_cap], + dom_planes, sheet_pts, + np.zeros((0, 3), dtype=np.int64), + np.zeros(0, dtype=np.int64), None, chain=chain) + cap_payload = { + "rim_shell_local": [local[v] for v in cap_nodes], + "chain_sheet_local": list(chain_nodes), + "tris": cap_tris, + "extra_xyz": cap_extra, + } + + fill = _gmsh_fill_3d(shell_xyz, shell_tris, sheet_pts, + sheet_tris, h, cap=cap_payload) + (fill_pts, fill_tets, sheet_out, moved_nodes, n_shell, + cap_out) = fill + if moved_nodes: + raise RuntimeError( + f"the fill moved {moved_nodes} constrained node(s); " + "the cavity cannot be sewn back. A defect, not a " + "tolerance.") + if sheet_out is None or len(sheet_out) != len(sheet_tris): + raise RuntimeError( + "the fill remeshed the sheet " + f"({0 if sheet_out is None else len(sheet_out)} " + f"triangles for {len(sheet_tris)} given).") + if cap_payload is not None and ( + cap_out is None + or len(cap_out) != len(cap_payload["tris"])): + raise RuntimeError( + "the fill remeshed the outcrop cap " + f"({0 if cap_out is None else len(cap_out)} " + f"triangles for {len(cap_payload['tris'])} given).") + + # A fill can span nearly coplanar shell nodes with a flat + # cell (measured ~1e-20 on bisection children, from gmsh's + # Delaunay and HXT algorithms alike) that none of the gates + # sees. Deleting those shell vertices and refilling the + # larger cavity removes it, measured in one round. + flat_nodes = _flat_fill_shell_nodes(fill_pts, fill_tets, + n_shell, h) + if not len(flat_nodes): + break + grow = np.asarray(shell_vert_ids, dtype=np.int64)[flat_nodes] + if on_wall[grow].any(): + raise RuntimeError( + "the cavity fill left a flat cell against the domain " + "wall; move the sheet further from the wall or raise " + "`clearance`.") + extra_victims[grow] = True + else: raise RuntimeError( - "the fill remeshed the outcrop cap " - f"({0 if cap_out is None else len(cap_out)} triangles " - f"for {len(cap_payload['tris'])} given).") + f"the cavity fill still held flat cells after " + f"{_SHEET_FILL_ATTEMPTS} refills; raise `clearance`.") # Exception, not just RuntimeError/ValueError: a raw gmsh error # (e.g. a PLC intersection) is a plain Exception, and an # uncaught raise on the surgery rank is a HANG for its peers — @@ -8408,6 +8613,13 @@ def place_thin_volume(dm, patches, width, label=ZONE_LABEL, label_value=1, "extra_xyz": cap_extra, } + # TODO(BUG): no flat-cell gate on this fill. place_sheet's fill + # was measured leaving ~1e-20 cells across nearly coplanar shell + # nodes on bisection children (both gmsh Delaunay and HXT); this + # annulus fill shares the shell and gmsh, so it is likely exposed + # too. place_sheet deletes those shell vertices and refills + # (_flat_fill_shell_nodes); the same loop belongs here. + # Found 2026-09-15 building the tumbling-fault Mohr sweep. fill = _gmsh_fill_annulus_3d(shell_xyz, shell_tris, skin_xyz_m, skin_tris_fill_m, size_out=h, size_in=size, cap=cap_payload) @@ -9091,7 +9303,6 @@ def place_fault_ribbon_2d(base_mesh, traces, width, *, margin_rings=2, f"{labels}") if spines is None: spines = [(label, np.asarray(P, dtype=float)) for label, P in traces] - dm = base_mesh.dm spacing_all, rungs_all, extended = [], [], [] for label, P in spines: P = np.asarray(P, dtype=float) @@ -9104,6 +9315,18 @@ def place_fault_ribbon_2d(base_mesh, traces, width, *, margin_rings=2, float(np.linalg.norm(np.diff(P, axis=0), axis=1).mean())) rungs_all.append(len(P)) + # In parallel, gather mode: the add-on is partitioned for the bands + # first, so each placement's gather moves nothing and the tail is not + # co-located behind it (utilities.addon_partition). The ligament and + # conform modes mesh across the seams in place, by design. + host = base_mesh + if base_mesh.dm.getComm().getSize() > 1 and seams == "gather": + from underworld3.utilities.addon_partition import (mark_band, + partition_addon) + host = partition_addon(base_mesh, mark_band(extended, width, clearance), + layers=2, verbose=verbose) + dm = host.dm + if mesher == "network": # ONE placement call for the whole network: the ribbons are fused # in CAD, so touching strands and shared bands are ordinary cells @@ -9135,6 +9358,8 @@ def place_fault_ribbon_2d(base_mesh, traces, width, *, margin_rings=2, dm, simplex=True, qdegree=base_mesh.qdegree, coordinate_system_type=base_mesh.CoordinateSystem.coordinate_type, boundaries=base_mesh.boundaries, verbose=False) + from underworld3.utilities.addon_partition import inherit_provenance + inherit_provenance(mesh, host) # The placed mesh OWNS the base's multigrid tail (the coarse levels do # not need the fault): every solver on it drives FMG automatically, # and the band label is its FAC patch key for volumetric rheologies. @@ -9142,7 +9367,7 @@ def place_fault_ribbon_2d(base_mesh, traces, width, *, margin_rings=2, band_all = np.zeros(int(mesh.dm.getHeightStratum(0)[1]), dtype=bool) for k in range(len(spines)): band_all |= mesh.cells_labelled(band_label, band_value + k) - adopt_hierarchy(mesh, base_mesh, fac_zone=band_all) + adopt_hierarchy(mesh, host, fac_zone=band_all) if split: # ONE network call — cut all, then split all (chained add_fault # calls do not compose: each split re-derives the pairing records diff --git a/src/underworld3/utilities/rotated_bc.py b/src/underworld3/utilities/rotated_bc.py index 0a8213c2b..071606852 100644 --- a/src/underworld3/utilities/rotated_bc.py +++ b/src/underworld3/utilities/rotated_bc.py @@ -578,6 +578,23 @@ def build_rotation(solver, boundaries, datum_specs=None): if fault_names: from underworld3.utilities import fault_contact s2 = 1.0 / np.sqrt(2.0) + # A sheet moving along its normal (the datum of SHEET_TANGENCY): a pair + # node takes the tangency through its pair block, so the MEAN row along + # the surface normal carries the datum there, (v+ + v-)/sqrt2 . e = sqrt2 w + # with e the frame's surface-normal vector (sign against the analytic + # normal); the jump row stays at 0. Without this the fault nodes stood + # still while the sheet breathed (measured: a datum error of 0.018 against + # an amplitude of 0.1, and spurious stress along the trace). + sheet_dspec = None if datum_specs is None else datum_specs.get(SHEET_TANGENCY) + sheet_normal_fn = None + if manifold and sheet_dspec is not None: + for spec in boundaries: + name_s, normal_s = _boundary_spec(spec) + if name_s == SHEET_TANGENCY: + sheet_normal_fn, sheet_normal_const = _normal_source(solver, normal_s, SHEET_TANGENCY) + pcsec = dm.getCoordinateSection() + pcvec = np.asarray(dm.getCoordinatesLocal().array).reshape(-1, dim) + pv0, pv1 = dm.getDepthStratum(0) # Rank-local failures in this block (a pairing/label mismatch or a # vanishing analytic normal inside _fault_pair_nodes, a fault node # shared with a wall boundary, a pair straddling ranks) must not @@ -637,6 +654,16 @@ def build_rotation(solver, boundaries, datum_specs=None): # vector) are pinned; the slip row stays free normal_rows.append(gm[2]) normal_rows.append(gp[2]) + if sheet_dspec is not None: + xq = _point_coord(dm, dim, pcvec, pcsec, pv0, pv1, q_plus) + if sheet_normal_fn is not None: + ne = np.asarray(sheet_normal_fn(*xq), dtype=float).ravel() + else: + ne = np.asarray(sheet_normal_const, dtype=float).ravel() + ne = ne / (np.linalg.norm(ne) + 1e-30) + sgn_e = 1.0 if float(np.dot(np.asarray(frame[2], dtype=float).ravel(), ne)) >= 0 else -1.0 + # row gp[2] = s2 (v+ + v-) . e = s2 * 2 w * sgn_e + datum_nodes.append((gp[2], q_plus, sheet_dspec, 2.0 * s2 * sgn_e)) except (RuntimeError, ValueError) as err: problem = str(err) fault_contact._collective_raise(problem) @@ -995,7 +1022,8 @@ def _backtracking_line_search(u, d, rnorm, rotated_residual, Q, Qt, normal_rows, and replaced by the accepted iterate. Returns ``(u, improved)``; ``improved`` is False when no step reduced the residual below ``rnorm`` — the iteration has stalled (typically because it is already at the solution) and the caller should - stop rather than accept a non-decreasing step.""" + stop rather than accept a non-decreasing step. Also returns the accepted step + length ``alpha`` (0.0 on a stall), which the loop records per increment.""" alpha = 1.0 for _ls in range(max_halvings): utry = u.copy() @@ -1006,10 +1034,10 @@ def _backtracking_line_search(u, d, rnorm, rotated_residual, Q, Qt, normal_rows, Ftry.destroy() if fnorm < rnorm: u.destroy() - return utry, True + return utry, True, alpha utry.destroy() alpha *= 0.5 - return u, False + return u, False, 0.0 def solve_rotated_freeslip(solver, boundaries, remove_rotation_gauge=True, @@ -1103,6 +1131,8 @@ def solve_rotated_freeslip(solver, boundaries, remove_rotation_gauge=True, * ``"rnorm"``, ``"rnorm0"`` — final and initial rotated residual norms ‖F̂‖ (feed the solve report); * ``"continuation_switched"`` — whether the Picard→Newton tangent switch fired; + * ``"continuation_switch_iteration"`` — the iteration it fired at (``None`` + if it did not); * ``"rotation_reused"`` — the GEOMETRY tier (Q/Qt/constrained rows/ prolongation) came from the cross-solve cache; * ``"workspace_reused"`` — the iteration-0 operator fast path fired @@ -1340,6 +1370,10 @@ def solve_rotated_freeslip(solver, boundaries, remove_rotation_gauge=True, Mp = ctx["Mp"] if ctx is not None else None diag_scale = cache["diag_scale"] if cache is not None else None lin_its = [] + # per-iteration records for campaign diagnostics: |F̂| at each iteration start, + # and the step accepted for each increment (0.0: not applied) + rnorm_history = [] + line_search_alpha = [] # velocity / pressure sub-KSP LAST-APPLICATION counts, one per Newton increment # (iterative path only — the direct-LU path has no sub-KSPs and records None) vel_its_last = [] @@ -1361,6 +1395,40 @@ def solve_rotated_freeslip(solver, boundaries, remove_rotation_gauge=True, # pressure rows) — kept observable so an INCOMPATIBLE datum cannot hide # behind the projection (see the guard after the Newton loop). pnull_gauge = [0.0] + # The rigid-rotation gauge likewise: when rotation modes are a genuine null + # space of the operator (a drag-free sheet on a sphere, a free-slip shell), + # the inner KSP cannot reduce the residual's component along them, so the + # outer norm is measured in that quotient space too. A viscous forcing has no + # such component by construction; a viscoelastic stress history on the + # faceted surface does (measured: the Newton loop floored at rel 1e-4 on a + # tidally flexed Maxwell sheet, with the linear solves converged). + rot_modes = [] # orthonormal, velocity-only, rotated frame + rnull_gauge = [0.0] + + def _collect_rotation_modes(nullspace): + for m in rot_modes: + m.destroy() + rot_modes.clear() + if nullspace is None: + return + for v in nullspace.getVecs(): + if has_pressure: + sp = v.getSubVector(pres_is) + is_pressure = sp.norm() > 0.0 + v.restoreSubVector(pres_is, sp) + if is_pressure: + continue + m = v.copy() + for q in rot_modes: # Gram-Schmidt + m.axpy(-m.dot(q), q) + nm = m.norm() + if nm > 0.0: + m.scale(1.0 / nm) + rot_modes.append(m) + else: + m.destroy() + + _collect_rotation_modes(nsp) def rotated_residual(uvec, keep_cartesian=False): snes.computeFunction(uvec, Fc) @@ -1395,6 +1463,13 @@ def rotated_residual(uvec, keep_cartesian=False): sp.shift(-mean) # project out the pressure-gauge mode Fh.restoreSubVector(pres_is, sp) pnull_gauge[0] = abs(mean) * (max(n_p, 1) ** 0.5) + if rot_modes: + removed = 0.0 + for m in rot_modes: # project out the rotation-gauge modes + c = Fh.dot(m) + Fh.axpy(-c, m) + removed += c * c + rnull_gauge[0] = removed ** 0.5 return Fh # Convergence reference: max(initial residual, REST-STATE residual ‖F̂(0)‖). @@ -1427,10 +1502,15 @@ def rotated_residual(uvec, keep_cartesian=False): # rotated solve as unconverged — including ones that converged at rel 1e-8. exit_reason = _SNES_DIVERGED_MAX_IT phase = "picard" if continuation else "newton" + switch_iteration = None for iters in range(max_it): Fhat = rotated_residual(u, keep_cartesian=True) rnorm = Fhat.norm() - if r0 is None: + rnorm_history.append(float(rnorm)) + if r0 is None or (lift_datum and iters == 1 and r0 == 0.0): + # A cold start whose only forcing is the prescribed datum (a sheet + # breathing with the tide, say) has a zero residual at u = 0, before + # the lift; the reference scale is then the lifted iterate's. r0 = rnorm ref = max(r0, fnorm_rest) if fnorm_rest is not None else r0 if verbose: @@ -1438,7 +1518,9 @@ def rotated_residual(uvec, keep_cartesian=False): f"rel={rnorm/(ref+1e-300):.3e} [{phase}]") # residual convergence (relative to the reference scale, plus an absolute # floor so an already-converged warm start does not chase machine noise). - if rnorm <= rtol * ref + atol: + if rnorm <= rtol * ref + atol and not (lift_datum and iters == 0): + # (the lift of a cold-start datum rides the first increment, so a + # zero residual at u = 0 is not convergence until it has been applied) converged = True exit_reason = _SNES_CONVERGED_FNORM_RELATIVE Fhat.destroy() @@ -1446,12 +1528,16 @@ def rotated_residual(uvec, keep_cartesian=False): # Continuation: switch the frozen (Picard, α=0) tangent to the consistent # (Newton, α=1) tangent once the residual has dropped into Newton's basin (the # loose newton_switch_rtol) and at least `picard` Picard iterations have run. - if continuation and phase == "picard" and rnorm <= switch_rtol * r0 and iters >= picard: + # Measured against the convergence reference, not r0: a warm start is already + # in the basin, and against its own residual it had to shrink that a + # hundredfold under Picard first (12 of 13 iterations on the power-law box). + if continuation and phase == "picard" and rnorm <= switch_rtol * ref and iters >= picard: solver._set_newton_alpha(1.0) phase = "newton" + switch_iteration = iters if verbose: mpi.pprint(f"[rotated_bc] continuation: Picard→Newton at iter {iters} " - f"(rel |F̂| {rnorm/(r0+1e-300):.2e})") + f"(rel |F̂| {rnorm/(ref+1e-300):.2e})") # Iteration 0 may ride the cross-solve fast path: the cached Ahat still # holds the rotated, constraint-eliminated operator and the cached # KSP/PC is set up on it. Every LATER iteration always reassembles — @@ -1494,6 +1580,7 @@ def rotated_residual(uvec, keep_cartesian=False): if ctx is None: Mp = _pressure_mass_schur_pmat(solver) if has_pressure else None nsp = _rotated_nullspace(solver, Q, normal_rows) # J assembled above + _collect_rotation_modes(nsp) elif assemble and Mp is not None: Jp.createSubMatrix(pres_is, pres_is, submat=Mp) # viscosity may be u-dependent if diag_scale is None: @@ -1578,9 +1665,13 @@ def rotated_residual(uvec, keep_cartesian=False): _impose_normal_constraint(u, Q, Qt, normal_rows, datum_map) step_converged = False improved = True + line_search_alpha.append(1.0) elif not step_converged: - u, improved = _backtracking_line_search( + u, improved, alpha = _backtracking_line_search( u, d, rnorm, rotated_residual, Q, Qt, normal_rows, datum_map) + line_search_alpha.append(alpha) + else: + line_search_alpha.append(0.0) # step-norm exit: the increment is not applied # every per-iteration temporary dies HERE, on all exit paths dhat.destroy() d.destroy() @@ -1624,6 +1715,15 @@ def rotated_residual(uvec, keep_cartesian=False): # incompressible domain) parks an O(forcing) constant there — and the loop # above would report convergence while mass conservation is violated. # Surface the component: readable state always, loud when it dominates. + solver._rotated_rotation_gauge_residual = rnull_gauge[0] if rot_modes else 0.0 + if rot_modes and converged and rnull_gauge[0] > 10.0 * max(rnorm, atol): + mpi.pprint(f"[rotated_bc] WARNING: converged in the rotation-gauge QUOTIENT " + f"space, but the projected-out rigid-rotation component " + f"({rnull_gauge[0]:.2e}) dominates the residual ({rnorm:.2e}): the " + f"forcing has a net torque, which free-slip walls cannot balance.") + for m in rot_modes: + m.destroy() + rot_modes.clear() if use_pnull: solver._rotated_pressure_gauge_residual = pnull_gauge[0] if converged and pnull_gauge[0] > 10.0 * max(rnorm, atol): @@ -1684,6 +1784,10 @@ def rotated_residual(uvec, keep_cartesian=False): "ksp_reason": last_reason, "nonlinear_iterations": newton_its, "converged": converged, "ksp_its": lin_its, "rnorm": rnorm, "rnorm0": r0, + # the rest-state residual ‖F̂(0)‖ (warm starts only; None cold) and the + # reference the convergence and switch tests compare against + "fnorm_rest": fnorm_rest, "ref": ref, + "rnorm_history": rnorm_history, "line_search_alpha": line_search_alpha, "vel_its_last": vel_its_last, "pres_its_last": pres_its_last, # keyed on use_lu, NOT on `ctx is None`: the iterative path also exits # with ctx unset when a warm start is already converged at increment 0. @@ -1691,6 +1795,7 @@ def rotated_residual(uvec, keep_cartesian=False): "schur_pre": "none" if use_lu else (ctx or {}).get("schur_pre"), "velocity_pc_type": None if use_lu else (ctx or {}).get("velocity_pc_type"), "continuation_switched": continuation and phase == "newton", + "continuation_switch_iteration": switch_iteration, "rotation_reused": rotation_reused, "workspace_reused": workspace_reused} @@ -1900,6 +2005,18 @@ def _build_rotated_custom_Pl(solver, Q, normal_rows): resolved = custom_mg.build_transfers(solver, field_id=0) h, Ps = resolved if resolved is not None else (None, None) if h is None or Ps is None: + # Loud, not silent: a GAMG velocity block on a fault or free-slip mesh is + # a symptom of a mesh built without its multigrid tail (a gmsh mesh with a + # conforming patch, a split that dropped the tail), not a configuration + # anyone chooses — it crawls against the sub-solve cap on 3-D fault + # meshes. Ruling 2026-10-17: flag the fallback every time it happens. + if not getattr(solver, "_rotated_gamg_fallback_warned", False): + mpi.pprint("[rotated_bc] WARNING: no multigrid hierarchy on this mesh " + "(no adapt tail, no set_custom_fmg registration); the " + "velocity block falls back to GAMG. Build the mesh through " + "adapt() so the tail rides through, or register a coarse " + "tail with set_custom_fmg.") + solver._rotated_gamg_fallback_warned = True return None walls = bool(getattr(solver, "_rotated_freeslip_bcs", None)) if getattr(h, "has_split_levels", False) or \ diff --git a/tests/parallel/fault_fixtures.py b/tests/parallel/fault_fixtures.py new file mode 100644 index 000000000..a16e625d6 --- /dev/null +++ b/tests/parallel/fault_fixtures.py @@ -0,0 +1,99 @@ +"""Fixtures the parallel fault tests share. + +The 2-D adapt-on-top fault (a refined unit box, an add-on of ``k`` levels toward +a straight trace, the trace cut and split) with its simple-shear solve, and the +3-D disc sheet on a bisection child. Charter section 5: one copy, imported by +every test that needs it. +""" +import numpy as np + +import underworld3 as uw + +TIP_A, TIP_B = np.array([0.30, 0.45]), np.array([0.70, 0.55]) +TRACE = np.vstack([TIP_A, TIP_B]) +H_BASE = 0.05 + + +def base_2d(): + return uw.meshing.UnstructuredSimplexBox( + minCoords=(0.0, 0.0), maxCoords=(1.0, 1.0), cellSize=0.1, refinement=1, + qdegree=2) + + +def adapted_2d(k, base=None): + """The add-on mesh of ``k`` levels toward the trace.""" + base = base_2d() if base is None else base + pts = np.column_stack([np.linspace(TIP_A[0], TIP_B[0], 9), + np.linspace(TIP_A[1], TIP_B[1], 9), np.zeros(9)]) + guide = uw.meshing.Surface(f"guide_{k}", base, pts) + guide.discretize() + return base.adapt(guide.refinement_metric_function( + h_near=H_BASE / 2**k, h_far=H_BASE, width=0.08, profile="linear"), + max_levels=k) + + +def faulted_2d(k, split_levels=True): + return adapted_2d(k).add_fault([("Flt", TRACE)], split_levels=split_levels) + + +def shear_fault_solve(mesh, law=0): + """Simple shear on the walls, a fault condition ``law`` on ``Flt``; returns + ``(stokes, info, peak slip)`` with the slip reduced over the ranks.""" + x, y = mesh.X + stokes = uw.systems.Stokes(mesh) + stokes.constitutive_model = uw.constitutive_models.ViscousFlowModel + stokes.constitutive_model.Parameters.shear_viscosity_0 = 1 + stokes.tolerance = 1.0e-6 + stokes.petsc_use_pressure_nullspace = True + for side in ("Top", "Bottom", "Left", "Right"): + stokes.add_dirichlet_bc((y - 0.5, 0.0), side) + stokes.add_fault_bc(law, "Flt") + stokes.solve() + info = stokes._rotated_freeslip_info + from underworld3.utilities.fault_contact import fault_slip + _s, V, _leak = fault_slip(stokes, "Flt", info) + local = float(np.abs(V).max()) if len(V) else 0.0 + return stokes, info, float(uw.mpi.comm.allreduce(local, op=max)) + + +# --- the 3-D disc sheet on a bisection child (tests/test_0867 carries the serial +# copy; the two directories are separate import roots) ---------------------- # +CENTRE = np.array([0.5, 0.5, 0.5]) +RADIUS = 0.2 +H_NEAR = 0.05 +TILTED = np.array([0.3, -0.5, 0.8]) / np.linalg.norm([0.3, -0.5, 0.8]) + + +def disc_sheet(normal=TILTED, n_rim=32, centre=CENTRE, radius=RADIUS): + """A planar disc as a centre fan.""" + helper = np.eye(3)[np.argmin(np.abs(normal))] + e1 = np.cross(normal, helper) + e1 /= np.linalg.norm(e1) + e2 = np.cross(normal, e1) + a = np.linspace(0.0, 2.0 * np.pi, n_rim, endpoint=False) + rim = centre + radius * (np.outer(np.cos(a), e1) + np.outer(np.sin(a), e2)) + pts = np.vstack([centre, rim]) + tris = np.array([(0, 1 + i, 1 + (i + 1) % n_rim) for i in range(n_rim)], + dtype=np.int64) + return pts, tris + + +def graded_disc_metric(normal=TILTED, discs=((CENTRE, RADIUS),)): + """``M = 1/h^2`` from the exact distance to the nearest disc, + ``h_near -> 2 h_near``.""" + def metric(X): + d = np.full(len(X), np.inf) + for centre, radius in discs: + rel = np.asarray(X)[:, :3] - centre + zn = rel @ normal + r = np.linalg.norm(rel - zn[:, None] * normal, axis=1) + d = np.minimum(d, np.sqrt(np.maximum(r - radius, 0.0) ** 2 + zn ** 2)) + h = np.clip(H_NEAR + d * H_NEAR / 0.4, H_NEAR, 2.0 * H_NEAR) + return 1.0 / h**2 + return metric + + +def base_3d(): + return uw.meshing.UnstructuredSimplexBox( + minCoords=(0.0, 0.0, 0.0), maxCoords=(1.0, 1.0, 1.0), + cellSize=0.2, regular=False, qdegree=2, refinement=1) diff --git a/tests/parallel/test_0078_rebuild_memory_parallel.py b/tests/parallel/test_0078_rebuild_memory_parallel.py new file mode 100644 index 000000000..c666659b3 --- /dev/null +++ b/tests/parallel/test_0078_rebuild_memory_parallel.py @@ -0,0 +1,81 @@ +"""Rebuilding a faulted mesh does not grow memory, in parallel. + +In parallel petsc4py does not destroy a PETSc object when Python collects +its wrapper: the ranks may collect in different orders and the destroy is +collective, so the object goes on a deferred list that only a collective +``PETSc.garbage_cleanup()`` empties. Nothing called it, so every mesh a +driver dropped kept its DM hierarchy, matrices and vectors for the rest of +the process even after the Python objects were freed: measured on the +moving-fault fixture at np = 2, 52.7 MB per rebuild with every Python +object freed (2217 DMs alive in PETSc's table after five rebuilds, 98 in +serial). A new mesh now empties the list (``Mesh.__init__`` is collective). + +Run with at least two ranks. +""" +import gc +import sys + +import numpy as np +import pytest +from mpi4py import MPI + +import underworld3 as uw +from underworld3.utilities import memprobe + +# The resident set measures the leak only where freed memory leaves it. On +# macOS it does not: with the fix, PETSc's object table is the same after 4 +# and after 8 rebuilds (284 DMs, 221 Mats, 298 Vecs on rank 0 at np 2) while +# the resident set still rises 14-26 MB per rebuild (2026-10-28). +pytestmark = [pytest.mark.mpi(min_size=2), pytest.mark.level_2, pytest.mark.tier_b, + pytest.mark.timeout(900), + pytest.mark.skipif(sys.platform != "linux", + reason="the resident set tracks freed memory on Linux only")] + +WARMUP, MEASURED = 2, 5 +BOUND_MB = 10.0 # per rebuild; the leak was ~50 at this size + + +def _resident_mb(): + return memprobe.snapshot()["rss_mb"] + + +def _rebuild_and_drop(base, k, centre): + half = np.array([0.20, 0.05]) + tip_a, tip_b = np.array([centre, 0.5]) - half, np.array([centre, 0.5]) + half + samples = np.column_stack([np.linspace(tip_a[0], tip_b[0], 9), + np.linspace(tip_a[1], tip_b[1], 9), np.zeros(9)]) + guide = uw.meshing.Surface(f"guide{k}", base, samples) + guide.discretize() + metric = guide.refinement_metric_function( + h_near=0.0125, h_far=0.05, width=0.08, profile="linear") + child = base.adapt(metric, max_levels=2) + mesh = child.add_fault([("Flt", np.vstack([tip_a, tip_b]))]) + _x, y = mesh.X + stokes = uw.systems.Stokes(mesh) + stokes.constitutive_model = uw.constitutive_models.ViscousFlowModel + stokes.constitutive_model.Parameters.shear_viscosity_0 = 1.0 + stokes.petsc_use_pressure_nullspace = True + for wall in ("Top", "Bottom", "Left", "Right"): + stokes.add_dirichlet_bc((y - 0.5, 0.0), wall) + stokes.add_fault_bc(0, "Flt") + stokes.solve() + del stokes, mesh, child, guide, metric, y + gc.collect() + + +def test_rebuilding_a_faulted_mesh_does_not_grow_memory(): + base = uw.meshing.UnstructuredSimplexBox( + minCoords=(0.0, 0.0), maxCoords=(1.0, 1.0), cellSize=0.1, + refinement=1, qdegree=2) + centres = 0.35 + 0.3 * np.arange(WARMUP + MEASURED + 1) / (WARMUP + MEASURED) + for k in range(WARMUP): + _rebuild_and_drop(base, k, centres[k]) + start = _resident_mb() + for k in range(WARMUP, WARMUP + MEASURED): + _rebuild_and_drop(base, k, centres[k]) + # one more mesh: its construction frees what the last rebuild dropped + uw.meshing.UnstructuredSimplexBox(minCoords=(0.0, 0.0), maxCoords=(1.0, 1.0), + cellSize=0.5) + growth = (_resident_mb() - start) / MEASURED + worst = MPI.COMM_WORLD.allreduce(growth, op=MPI.MAX) + assert worst < BOUND_MB, f"{worst:.1f} MB per rebuild on the worst rank" diff --git a/tests/parallel/test_0079_snapshot_after_a_dropped_mesh_parallel.py b/tests/parallel/test_0079_snapshot_after_a_dropped_mesh_parallel.py new file mode 100644 index 000000000..23d9ad210 --- /dev/null +++ b/tests/parallel/test_0079_snapshot_after_a_dropped_mesh_parallel.py @@ -0,0 +1,68 @@ +"""A disk snapshot after a dropped mesh writes the same meshes on every rank. + +A dropped mesh with a variable is a reference cycle (the mesh owns its +variables, each variable holds its mesh), so it stays in the model's weak +registry until the cyclic collector runs, and the collector runs at a +different moment on each rank. ``write_snapshot`` makes a collective +checkpoint write per registered mesh: a rank that had collected the dropped +mesh and a rank that had not entered different numbers of collective writes +and hung (review 3 of the faulted-rebuild leak fix, 2026-10-27). The +snapshot now collects first on every rank. Here one rank collects and the +other is held off. + +Run with at least two ranks. +""" +import gc + +import pytest +from mpi4py import MPI + +import underworld3 as uw + +# The thread method: a rank blocked in MPI does not see the signal one. +pytestmark = [pytest.mark.mpi(min_size=2), pytest.mark.level_1, pytest.mark.tier_b, + pytest.mark.timeout(120, method="thread")] + + +def _kept_mesh_and_a_dropped_one_one_rank_collected(): + """A kept mesh with a variable, and a dropped one that rank 0 alone has + collected; the collector is held off until the caller enables it.""" + uw.reset_default_model() + kept = uw.meshing.UnstructuredSimplexBox( + minCoords=(0.0, 0.0), maxCoords=(1.0, 1.0), cellSize=0.25) + uw.discretisation.MeshVariable("T", kept, 1, degree=1) + + def make(): + mesh = uw.meshing.UnstructuredSimplexBox( + minCoords=(0.0, 0.0), maxCoords=(1.0, 1.0), cellSize=0.25) + uw.discretisation.MeshVariable("T", mesh, 1, degree=1) + + gc.disable() + make() + with uw.selective_ranks(0) as collects: + if collects: + gc.collect() + return kept + + +def test_a_disk_snapshot_after_a_dropped_mesh_is_rank_uniform(tmp_path): + path = MPI.COMM_WORLD.bcast(str(tmp_path / "state.h5"), root=0) + try: + kept = _kept_mesh_and_a_dropped_one_one_rank_collected() + uw.get_default_model().save_state(file=path) + finally: + gc.enable() + metadata = uw.checkpoint.read_snapshot_metadata(path) + assert metadata["mesh_names"] == [kept.name] + + +def test_an_in_memory_snapshot_after_a_dropped_mesh_is_rank_uniform(): + """The in-memory capture syncs each mesh's vectors (local to global, a + collective) as it goes, so it has the same hazard.""" + try: + kept = _kept_mesh_and_a_dropped_one_one_rank_collected() + state = uw.get_default_model().save_state() + finally: + gc.enable() + names = MPI.COMM_WORLD.allgather(state.mesh_names) + assert all(n == [kept.name] for n in names), names diff --git a/tests/parallel/test_1089_fault_split_multigrid_parallel.py b/tests/parallel/test_1089_fault_split_multigrid_parallel.py index 6bc6cb665..2fd5489c7 100644 --- a/tests/parallel/test_1089_fault_split_multigrid_parallel.py +++ b/tests/parallel/test_1089_fault_split_multigrid_parallel.py @@ -21,44 +21,21 @@ import sympy import underworld3 as uw +from fault_fixtures import faulted_2d, shear_fault_solve pytestmark = [pytest.mark.mpi(min_size=2), pytest.mark.level_2, pytest.mark.tier_b, pytest.mark.timeout(1800)] -TIP_A, TIP_B = np.array([0.30, 0.45]), np.array([0.70, 0.55]) SERIAL_PEAK_SLIP = 0.159581 # k = 2, frictionless, solve tolerance 1e-6 def _faulted_box(k, split_levels=True): - base = uw.meshing.UnstructuredSimplexBox( - minCoords=(0.0, 0.0), maxCoords=(1.0, 1.0), cellSize=0.1, refinement=1, - qdegree=2) - pts = np.column_stack([np.linspace(TIP_A[0], TIP_B[0], 9), - np.linspace(TIP_A[1], TIP_B[1], 9), np.zeros(9)]) - guide = uw.meshing.Surface("guide", base, pts) - guide.discretize() - child = base.adapt(guide.refinement_metric_function( - h_near=0.05 / 2**k, h_far=0.05, width=0.08, profile="linear"), max_levels=k) - return child.add_fault([("Flt", np.vstack([TIP_A, TIP_B]))], - split_levels=split_levels) + return faulted_2d(k, split_levels=split_levels) def _solve(mesh): - x, y = mesh.X - stokes = uw.systems.Stokes(mesh) - stokes.constitutive_model = uw.constitutive_models.ViscousFlowModel - stokes.constitutive_model.Parameters.shear_viscosity_0 = 1 - stokes.tolerance = 1.0e-6 - stokes.petsc_use_pressure_nullspace = True - for side in ("Top", "Bottom", "Left", "Right"): - stokes.add_dirichlet_bc((y - 0.5, 0.0), side) - stokes.add_fault_bc(0, "Flt") - stokes.solve() - info = stokes._rotated_freeslip_info - from underworld3.utilities.fault_contact import fault_slip - _s, V, _leak = fault_slip(stokes, "Flt", info) - local = float(np.abs(V).max()) if len(V) else 0.0 - return info, float(uw.mpi.comm.allreduce(local, op=max)) + _stokes, info, slip = shear_fault_solve(mesh) + return info, slip def test_split_levels_converge_like_serial(): diff --git a/tests/parallel/test_1095_custom_mg_split_finest_parallel.py b/tests/parallel/test_1095_custom_mg_split_finest_parallel.py new file mode 100644 index 000000000..0dc498522 --- /dev/null +++ b/tests/parallel/test_1095_custom_mg_split_finest_parallel.py @@ -0,0 +1,104 @@ +"""Custom-P multigrid transfers into a SPLIT finest level, in parallel. + +A refined patch box (a cuboid with a conforming fault patch, refined twice so +its coarse levels are the multigrid tail) is split along the fault. The split +moves the fault's cell star onto one rank, so the finest partition is no +longer the coarse partition: on that rank thousands of fine nodes lie outside +the LOCAL coarse hull. Two defects followed (found 2026-10-17 on the box fault +model, where the velocity multigrid needed 4 iterations in serial and 22, 37, +55 on 2, 4, 8 ranks): + +* the co-partitioned builder snapped those nodes to the nearest local coarse + node (a wrong row no zero-column count can see), and the automatic rule only + rebuilt a level across partitions on empty COLUMNS; +* the cross-partition builder ran the barycentric builder's serial ORPHAN + REPAIR on each rank's slice of the all-gathered coarse cloud, where most + columns look empty, and overwrote thousands of fine rows with weight-one + injections onto coarse nodes far away (measured: 20272 of 27785 x-rows wrong + on 8 ranks, errors up to 2.9 on a box of length 3). + +The checks: the installed finest transfer reproduces the coordinate field in +the reduced numbering (exact for a Lagrange interpolation), and the velocity +solve takes no more iterations than a few above serial. +""" +import numpy as np +import pytest +import sympy + +import underworld3 as uw + +pytestmark = [pytest.mark.mpi(min_size=2), pytest.mark.level_2, pytest.mark.tier_b, + pytest.mark.timeout(1800)] + +SERIAL_VEL_ITS = 4 # measured, tolerance 1e-4, this geometry + + +def _faulted_box(): + from underworld3.utilities.fault_split import split_fault + Lx, Ly, H = 3.0, 2.0, 1.0 + h, hf, f = 0.2, 0.1, 4 + yf, xa, xb, zb = 1.0, 1.0, 2.0, 0.5 + ztop = H - 2 * hf + patch = np.array([[xa, yf, zb], [xb, yf, zb], [xb, yf, ztop], [xa, yf, ztop]]) + fine = uw.meshing.BoxInternalPatch(cellSize=f * h, minCoords=(0, 0, 0), maxCoords=(Lx, Ly, H), + patch_points=patch, patch_name="Fault", + patch_cellSize=f * hf, grading_distance=8 * hf, + refinement=2, qdegree=3) + return split_fault(fine, "Fault") + + +def _solve(mesh): + v = uw.discretisation.MeshVariable("v", mesh, 3, degree=2) + p = uw.discretisation.MeshVariable("p", mesh, 1, degree=1) + stokes = uw.systems.Stokes(mesh, velocityField=v, pressureField=p) + stokes.constitutive_model = uw.constitutive_models.ViscousFlowModel + stokes.constitutive_model.Parameters.shear_viscosity_0 = 1.0 + stokes.bodyforce = sympy.Matrix([0, 0, 0]) + stokes.tolerance = 1e-4 + stokes.add_dirichlet_bc((0.0, 0.0, 0.0), "Left") + stokes.add_natural_bc(sympy.Matrix([[-0.01, -0.01, 0.0]]), "Right") + stokes.add_dirichlet_bc((None, None, 0.0), "Bottom") + stokes.add_fault_bc(0.0, "Fault") + stokes.solve(zero_init_guess=True) + return stokes + + +def _x_reproduction(stokes, coarse_mesh, P): + """Max |P x_c - x_f| over the x-component rows, in the reduced numbering.""" + from mpi4py import MPI + from underworld3.utilities.custom_mg import _coarse_dof_layout, _level_dof_layout + comm = MPI.COMM_WORLD + Xc = np.asarray(coarse_mesh._get_coords_for_basis(2, True)) + Xf = np.asarray(stokes.mesh._get_coords_for_basis(2, True)) + lay_c = _coarse_dof_layout(stokes, coarse_mesh, 0) + lay_f = _level_dof_layout(stokes.dm, 0) + xc, xf = P.createVecRight(), P.createVecLeft() + for i in range(Xc.shape[0]): + g = int(lay_c.l2g[3 * i]) + if lay_c.rstart <= g < lay_c.rend: + xc.setValue(g, float(Xc[i, 0])) + for i in range(Xf.shape[0]): + g = int(lay_f.l2g[3 * i]) + if lay_f.rstart <= g < lay_f.rend: + xf.setValue(g, float(Xf[i, 0])) + xc.assemble() + xf.assemble() + y = P.createVecLeft() + P.mult(xc, y) + d = np.abs(y.getArray() - xf.getArray()) + mask = np.abs(xf.getArray()) > 0 # the x-rows set above (x = 0 is the fixed face) + return comm.allreduce(float(d[mask].max()) if mask.any() else 0.0, op=MPI.MAX) + + +def test_split_finest_transfer_reproduces_coordinates_and_converges(): + mesh = _faulted_box() + assert len(mesh._custom_mg_coarse_meshes) == 2 + stokes = _solve(mesh) + info = stokes._rotated_freeslip_info + assert info["velocity_pc"] == "custom-FMG" and info["converged"], info + hierarchy = stokes._custom_mg["hierarchy"] + raw = hierarchy.transfers + coarse = hierarchy.level_meshes[-2] + err = _x_reproduction(stokes, coarse, raw[-1]) + assert err < 1e-8, f"finest transfer does not reproduce x: max error {err:.3e}" + assert info["vel_its_last"][-1] <= SERIAL_VEL_ITS + 4, info["vel_its_last"] diff --git a/tests/parallel/test_1096_addon_partition_parallel.py b/tests/parallel/test_1096_addon_partition_parallel.py new file mode 100644 index 000000000..cf9041120 --- /dev/null +++ b/tests/parallel/test_1096_addon_partition_parallel.py @@ -0,0 +1,190 @@ +"""The add-on is partitioned once, before any surgery, as one hierarchy. + +``partition_addon`` computes one assignment on the finest add-on level — the +fault regions glued and sent to their majority rank, the load balanced by +diffusion over the seams — and applies it to every add-on level through the +parent-cell relation, with provenance riding the move. The contracts asserted +here are the ones the fault machinery and the multigrid transfers build on +(design note ``docs/developer/design/parallel-addon-distribution-2026-10.md``): + +* every fault region (the trace's cell star plus a growth layer) lies on one + rank, on EVERY add-on level, so the split's pairs and fans are rank-local by + construction wherever the faults are replayed; +* every add-on pair is co-partitioned exactly: each fine cell's parent cell on + the level below is local; +* every cell's recorded parent, fetched by its current address, contains the + cell's centroid (the relation is exact, on the base boundary too); +* the finest cells per rank are within the balance tolerance of the mean, or + as close as the fault region allows: the region is one indivisible unit. + +Then ``add_fault`` on the partitioned add-on: the cut, the split and the level +replay move nothing (each level's lineage runs back to the partitioned tail), +the split-level transfers report no cross-rank miss, the base boundary is +fetched by provenance, and the solve converges in about the serial count. + +Measured 2026-10-17 on this fixture (k = 2): velocity iterations 6 at np 2 and +4 (serial 6; test_1087); finest cells per rank 742 / 618 at np 2. +""" +import numpy as np +import pytest +from mpi4py import MPI +from petsc4py import PETSc + +import underworld3 as uw +from fault_fixtures import TRACE, adapted_2d, shear_fault_solve +from underworld3.utilities import addon_partition as ap +from underworld3.utilities.fault_mg import (_barycentric, _cell_vertex_table, + addon_level_indices) +from underworld3.utilities.place_surface import _region_cells + +pytestmark = [pytest.mark.mpi(min_size=2), pytest.mark.level_2, pytest.mark.tier_b, + pytest.mark.timeout(900)] + +comm = MPI.COMM_WORLD +K = 2 +TOLERANCE = 0.1 + + +def _cells(mesh): + cS, cE = mesh.dm.getHeightStratum(0) + return cE - cS + + +def _levels(mesh): + """``(static, addon_levels_including_mesh)`` of a mesh-owned tail.""" + tail = mesh._custom_mg_coarse_meshes + idx = addon_level_indices(tail) + static = [tail[i] for i in range(len(tail)) if i not in idx] + return static, [tail[i] for i in idx] + [mesh] + + +def _parents_contain_centroids(below, lev): + """Fetch each cell's parent (by its CURRENT address) and test containment. + Returns the global count of cells NOT inside their parent.""" + par = lev._addon_parent + vxy_b, cv_b = _cell_vertex_table(below) + nb = cv_b.shape[0] + verts = np.ascontiguousarray(vxy_b[cv_b].reshape(nb, -1)) + sf = PETSc.SF().create(comm=PETSc.COMM_WORLD) + sf.setGraph(nb, None, np.ascontiguousarray(par, dtype=np.int32)) + sf.setUp() + got = np.zeros((par.shape[0], verts.shape[1])) + unit = MPI.DOUBLE.Create_contiguous(verts.shape[1]) + unit.Commit() + sf.bcastBegin(unit, verts, got, MPI.REPLACE) + sf.bcastEnd(unit, verts, got, MPI.REPLACE) + unit.Free() + sf.destroy() + fcent, _v, _c = ap._cell_centroids(lev) + lam = _barycentric(fcent, got.reshape(par.shape[0], cv_b.shape[1], -1)) + return comm.allreduce(int((lam.min(axis=1) < -1e-9).sum())) + + +def test_partition_invariants_on_the_adapt_child(): + child = adapted_2d(K) + marks = ap.mark_polylines([TRACE]) + out = ap.partition_addon(child, marks, layers=1, tolerance=TOLERANCE) + record = out._addon_partition + assert record["regions"] == 1 + assert record["units_moved"] > 0 and out is not child + + # the fault region is rank-interior on the new finest AND on every add-on + # level (the replay splits each level on its own star) + static, levels = _levels(out) + for lev in levels: + cell_region, _canon, regions = _region_cells(lev.dm, marks(lev.dm), layers=1) + counts = np.asarray(comm.allgather( + [int((cell_region == r).sum()) for r in regions])) + assert all(np.count_nonzero(counts[:, j]) == 1 for j in range(len(regions))) + + # add-on pairs are co-partitioned and the recorded parents are exact: an + # independent fetch of each parent by its current address contains the + # cell's centroid, on the add-on pairs and on the base boundary + assert len(levels) == len(_levels(child)[1]) + for l in range(1, len(levels)): + assert _parents_contain_centroids(levels[l - 1], levels[l]) == 0 + assert comm.allreduce(int((levels[l]._addon_parent[:, 0] != comm.rank).sum())) == 0 + assert _parents_contain_centroids(static[-1], levels[0]) == 0 + # a cell of the coarsest add-on level has a remote parent exactly when + # its unit moved + remote = comm.allreduce(int((levels[0]._addon_parent[:, 0] != comm.rank).sum())) + assert remote == record["units_moved"] + + # the finest cells are shared out: within tolerance of the mean, or down + # to the one indivisible fault region. The slack beyond the region is the + # diffusion's own stop rule: a rank stops offering when every lighter + # neighbour is within tolerance of it, so a load up to tolerance x mean + # above the region can remain on the owner. + load = np.asarray(comm.allgather(_cells(out)), dtype=float) + mean = load.mean() + region = max(record["region_weight"].values()) + assert load.max() <= max(mean * (1.0 + TOLERANCE), region) + TOLERANCE * mean, \ + (load, record) + # the total is conserved on every level + for a, b in zip(_levels(child)[1], levels): + assert comm.allreduce(_cells(a)) == comm.allreduce(_cells(b)) + + +def test_a_second_partition_keeps_the_first_fault_whole(): + """A split add-on partitioned again for a second fault: the first fault's + pairs survive the move whole (an interior labelled surface is glued like + a marked one), both faults' stars are rank-interior, and the base + boundary's relation is read from the first partition's record rather + than searched (the pair is no longer co-partitioned).""" + from underworld3.utilities.fault_split import _fault_labels_touch_seam + # a third of the box from the first trace: the two regions' shells (the + # stars plus a layer at the base spacing, about 0.15 each) do not touch, + # so they stay two regions + second = np.array([[0.30, 0.12], [0.70, 0.12]]) + split = adapted_2d(K).add_fault([("Flt", TRACE)]) + n_pairs = comm.allreduce(len(split._fault_point_pairs["Flt"])) + out = ap.partition_addon(split, ap.mark_polylines([second]), layers=1, + tolerance=TOLERANCE) + assert out._addon_partition["regions"] == 2 # the new and the glued + static, levels = _levels(out) + assert comm.allreduce(len(out._fault_point_pairs["Flt"])) == n_pairs + for lev in levels: + labels = [(f"Flt{side}", int(lev.boundaries[f"Flt{side}"].value)) + for side in ("Plus", "Minus")] + assert not _fault_labels_touch_seam(lev.dm, labels) + cell_region, _c, regions = _region_cells( + lev.dm, ap.mark_polylines([second])(lev.dm), layers=1) + counts = np.asarray(comm.allgather( + [int((cell_region == r).sum()) for r in regions])) + assert all(np.count_nonzero(counts[:, j]) == 1 for j in range(len(regions))) + assert _parents_contain_centroids(static[-1], levels[0]) == 0 + + +def test_add_fault_moves_nothing_after_the_partition_and_converges(): + from underworld3.utilities.fault_split import _fault_labels_touch_seam + child = adapted_2d(K) + out = child.add_fault([("Flt", TRACE)]) + # split <- cut <- partitioned <- child, for the finest and for every + # replayed add-on level: nothing was moved after the partition + partitioned = out.parent.parent + assert partitioned.parent is child + assert partitioned._addon_partition["units_moved"] > 0 + static, levels = _levels(out) + p_static, p_levels = _levels(partitioned) + assert [s is t for s, t in zip(static, p_static)] == [True] * len(static) + for lev, pre in zip(levels[:-1], p_levels[:-1]): + assert lev.parent.parent is pre + for lev in levels: + assert getattr(lev, "_fault_point_pairs", None), "a level was not split" + labels = [(f"Flt{side}", int(lev.boundaries[f"Flt{side}"].value)) + for side in ("Plus", "Minus")] + assert not _fault_labels_touch_seam(lev.dm, labels) + stokes, info, _slip = shear_fault_solve(out) + assert info["velocity_pc"] == "custom-FMG" and info["converged"], info + assert info["vel_its_last"][-1] <= 9, info["vel_its_last"] + stats = stokes._custom_mg["hierarchy"].level_transfer_stats + # the base boundary (static finest -> first add-on level) is the pair + # the partition moved apart: fetched by provenance, remote parents + # exactly where units moved + boundary = stats[len(static)] + assert "fetched_remote" in boundary, boundary + assert boundary["fetched_remote"] > 0 + # every add-on pair is co-partitioned: the cell-located transfer finds + # every node locally (the old co-location left 14 of 2992 remote) + for st in stats[len(static) + 1:]: + assert st is not None and st["misses"] == 0 and "fetched_remote" not in st, st diff --git a/tests/parallel/test_1097_place_sheet_partitioned_addon_parallel.py b/tests/parallel/test_1097_place_sheet_partitioned_addon_parallel.py new file mode 100644 index 000000000..b45eb2982 --- /dev/null +++ b/tests/parallel/test_1097_place_sheet_partitioned_addon_parallel.py @@ -0,0 +1,162 @@ +"""A 3-D sheet placed and split on a partitioned adapt child, in parallel. + +The fault workflow is base -> ``adapt`` toward the sheet -> ``add_conforming_sheet`` +-> ``split_fault``. Before the add-on partition the placement gathered the +sheet's region itself, with a mask that did not cover the cavity's growth rounds, +so on a partitioned adapt child the carve touched a shared point and the surgery +rank refused ("the gather mask under-reached") where the same placement passes +serially (measured 2026-10-17 on the box fault model at 8 ranks, and on this +fixture at 2 ranks). The split then moved the fault's star again. + +Now ``add_conforming_sheet`` partitions the add-on for the sheet first — the +placement's own mask with two growth layers, on every add-on level — so the +placement gathers nothing, the split moves nothing, the placed mesh keeps the +domain volume and one orientation (as the serial test of the same fixture, +``tests/test_0867``, asserts), the facet and pair counts are the serial ones +(this module's ``__main__`` run at np = 1 on the same host), and the fault +solve runs on the geometric tail with no transfer miss. +""" +import numpy as np +import pytest +import sympy + +import underworld3 as uw +from fault_fixtures import H_NEAR, base_3d, disc_sheet, graded_disc_metric +from serial_reference import emit, serial_reference +from underworld3.utilities.place_surface import _cell_volumes_signed6 + +pytestmark = [pytest.mark.mpi(min_size=2), pytest.mark.level_2, pytest.mark.tier_b, + pytest.mark.timeout(1500)] + + +def _placed_and_split(): + from underworld3.utilities.fault_split import split_fault + base = base_3d() + child = base.adapt(graded_disc_metric(), max_levels=1) + pts, tris = disc_sheet() + placed = child.add_conforming_sheet(pts, tris, "Fault", size=H_NEAR) + return base, placed, split_fault(placed, "Fault") + + +def _counts(placed, split): + """The quantities compared with the serial run: sheet facets, pairs, cells.""" + comm = uw.mpi.comm + return [float(placed._surface_info["n_surface_facets"]), + float(comm.allreduce(len(split._fault_point_pairs["Fault"]))), + float(comm.allreduce(split.dm.getHeightStratum(0)[1] + - split.dm.getHeightStratum(0)[0]))] + + +def test_sheet_placement_gathers_nothing_and_the_split_moves_nothing(): + from underworld3.utilities.fault_split import _fault_labels_touch_seam + comm = uw.mpi.comm + base, placed, split = _placed_and_split() + # the placement ran on a rank-interior region of the partitioned add-on + assert placed._surface_info["n_gathered"] == 0 + assert placed.parent._addon_partition["units_moved"] > 0 + v6 = _cell_volumes_signed6(placed.dm) + volume = comm.allreduce(float(np.abs(v6).sum()) / 6.0) + assert abs(volume - 1.0) < 1e-9 + sign_ok = bool((np.sign(v6) == np.sign(v6[0])).all()) if len(v6) else True + assert comm.allreduce(sign_ok, op=min) + # the split found its star rank-interior and moved nothing + assert split.parent is placed + labels = [(f"Fault{side}", int(split.boundaries[f"Fault{side}"].value)) + for side in ("Plus", "Minus")] + assert not _fault_labels_touch_seam(split.dm, labels) + # the serial run of this fixture: the same sheet facets and pairs; the + # cell count within the fill's partition noise (gmsh refills the cavity + # from a slightly different ring) + reference = serial_reference(__file__, "counts") + facets, pairs, cells = _counts(placed, split) + ref_facets, ref_pairs, ref_cells = reference["values"] + assert facets == ref_facets and pairs == ref_pairs, (facets, pairs, reference) + assert abs(cells - ref_cells) <= 0.02 * ref_cells, (cells, reference) + # the tail rode through: this fixture has no intermediate add-on level, so + # it is exactly the base's static wraps, the adapt child being the first + # level above them (the same objects, never moved) + tail = split._custom_mg_coarse_meshes + wraps = base._coarse_level_meshes() + assert tail is not None and len(tail) == len(wraps) + assert all(t is w for t, w in zip(tail, wraps)) + + +def test_fault_solve_on_the_placed_split_runs_on_the_geometric_tail(): + _base, _placed, split = _placed_and_split() + v = uw.discretisation.MeshVariable("v", split, 3, degree=2) + p = uw.discretisation.MeshVariable("p", split, 1, degree=1) + stokes = uw.systems.Stokes(split, velocityField=v, pressureField=p) + stokes.constitutive_model = uw.constitutive_models.ViscousFlowModel + stokes.constitutive_model.Parameters.shear_viscosity_0 = 1.0 + stokes.bodyforce = sympy.Matrix([0, 0, 0]) + stokes.tolerance = 1e-4 + stokes.add_dirichlet_bc((0.0, 0.0, 0.0), "Bottom") + stokes.add_dirichlet_bc((1.0, 0.0, 0.0), "Top") + stokes.add_fault_bc(0.0, "Fault") + stokes.solve(zero_init_guess=True) + info = stokes._rotated_freeslip_info + assert info["velocity_pc"] == "custom-FMG" and info["converged"], info + stats = stokes._custom_mg["hierarchy"].level_transfer_stats + # the finest pair is the moved one: fetched by provenance + assert stats[-1] is not None and stats[-1]["fetched_remote"] > 0, stats + + +def test_two_sheets_placed_in_turn_then_split_together(): + """The fault-network route places sheets one at a time and splits them + together. The second placement partitions an add-on the first already + partitioned and placed into: the first sheet stays rank-interior (its + facets are glued), nothing is gathered, and the network splits with + every pair whole.""" + from underworld3.utilities.fault_split import (_fault_labels_touch_seam, + split_faults) + from fault_fixtures import CENTRE, TILTED + comm = uw.mpi.comm + c1, c2 = CENTRE - 0.17 * TILTED, CENTRE + 0.17 * TILTED + base = base_3d() + child = base.adapt(graded_disc_metric(discs=((c1, 0.15), (c2, 0.15))), + max_levels=1) + p1, t1 = disc_sheet(centre=c1, radius=0.15) + p2, t2 = disc_sheet(centre=c2, radius=0.15) + one = child.add_conforming_sheet(p1, t1, "FltA", size=H_NEAR) + two = one.add_conforming_sheet(p2, t2, "FltB", size=H_NEAR) + assert one._surface_info["n_gathered"] == 0 + assert two._surface_info["n_gathered"] == 0 + split = split_faults(two, ["FltA", "FltB"]) + for name in ("FltA", "FltB"): + assert comm.allreduce(len(split._fault_point_pairs[name])) > 0 + labels = [(f"{name}{side}", int(split.boundaries[f"{name}{side}"].value)) + for side in ("Plus", "Minus")] + assert not _fault_labels_touch_seam(split.dm, labels) + v6 = _cell_volumes_signed6(split.dm) + assert abs(comm.allreduce(float(np.abs(v6).sum()) / 6.0) - 1.0) < 1e-9 + + +def _fingerprint(mesh): + """Cell count and volume, from the cells themselves (the shared + fingerprint integrates a field, and this mesh carries none).""" + v6 = _cell_volumes_signed6(mesh.dm) + comm = uw.mpi.comm + return [float(comm.allreduce(len(v6))), + comm.allreduce(float(np.abs(v6).sum()) / 6.0)] + + +def test_direct_place_sheet_on_a_partitioned_adapt_child(): + """``place_sheet`` called on a DM directly, without the Mesh API's + partition: its own gather must cover the cavity's growth rounds. Before + the fix the mask grew one layer, the grown cavity touched a shared point + and the surgery rank refused ("the gather mask under-reached") on this + fixture at 2 ranks, where the same placement passes serially.""" + from underworld3.utilities.place_surface import place_sheet + comm = uw.mpi.comm + child = base_3d().adapt(graded_disc_metric(), max_levels=1) + pts, tris = disc_sheet() + dm, info = place_sheet(child.dm, pts, tris, label="Fault", label_value=40, + size=H_NEAR) + assert info["n_surface_facets"] > 0 + v6 = _cell_volumes_signed6(dm) + assert abs(comm.allreduce(float(np.abs(v6).sum()) / 6.0) - 1.0) < 1e-9 + + +if __name__ == "__main__": + _base, _placed, _split = _placed_and_split() + emit(_counts(_placed, _split), _fingerprint(_split)) diff --git a/tests/parallel/test_1099_band_placement_partitioned_parallel.py b/tests/parallel/test_1099_band_placement_partitioned_parallel.py new file mode 100644 index 000000000..021a8e665 --- /dev/null +++ b/tests/parallel/test_1099_band_placement_partitioned_parallel.py @@ -0,0 +1,78 @@ +"""The placed fault band on a partitioned add-on, in parallel. + +The band route (``FaultNetwork.build`` in 3-D, ``place_fault_ribbon_2d`` in +gather mode) placed its band with ``place_thin_volume``, which gathered the +band's region onto a surgery rank; ``adopt_hierarchy`` then moved copies of +the base's coarse levels to follow it (``_colocate_level``), so every +transfer of the tail paired a level with one built for it. Both routes now +partition the add-on for the band first (``addon_partition.mark_band``, a +superset of the placement's own mask): the placement moves nothing, the tail +is the base's own levels with the partitioned host above them, and the pair +the partition separated is read from the recorded provenance. + +Asserted: no cell moved in the placement (the record the placement keeps); +the tail's static levels are the base's own cached wraps, by identity, and +the level above them is the partitioned host; the split pairs are whole; the +solve runs on the geometric tail, the separated pair built by provenance. +""" +import numpy as np +import pytest + +import underworld3 as uw + +pytestmark = [pytest.mark.mpi(min_size=2), pytest.mark.level_2, pytest.mark.tier_b, + pytest.mark.timeout(1800)] + +H = 0.08 +WIDTH = 0.04 +P_A = np.array([[0.30, 0.50, 0.30], [0.70, 0.50, 0.30], + [0.70, 0.50, 0.70], [0.30, 0.50, 0.70]]) + + +def _network(): + fs = uw.meshing.FaultSurface("Main", P_A) + fs.triangulate() + net = uw.meshing.FaultNetwork([fs]) + net.prepare(h=H, ligament=1.0, verbose=False) + net.build(width=WIDTH, realisation="split", h_far=0.24, margin_rings=0.5) + return net + + +def test_band_placement_moves_nothing_and_keeps_the_base_levels(): + from underworld3.utilities.fault_split import _fault_labels_touch_seam + net = _network() + mesh = net.mesh + comm = uw.mpi.comm + assert net.info["n_moved"] == 0, net.info + tail = mesh._custom_mg_coarse_meshes + host = tail[-1] + assert host._addon_partition is not None + wraps = host.parent._coarse_level_meshes() + assert len(tail) == len(wraps) + assert all(t is w for t, w in zip(tail[:-1], wraps[:-1])) + assert comm.allreduce(len(mesh._fault_point_pairs["Main"])) > 0 + labels = [(f"Main{side}", int(mesh.boundaries[f"Main{side}"].value)) + for side in ("Plus", "Minus")] + assert not _fault_labels_touch_seam(mesh.dm, labels) + + +def test_band_solve_runs_on_the_tail_with_the_provenance_pair(): + net = _network() + mesh = net.mesh + x, y, z = mesh.X + v = uw.discretisation.MeshVariable("vB", mesh, 3, degree=2) + p = uw.discretisation.MeshVariable("pB", mesh, 1, degree=0, continuous=False) + stokes = uw.systems.Stokes(mesh, velocityField=v, pressureField=p) + stokes.constitutive_model = uw.constitutive_models.ViscousFlowModel + stokes.constitutive_model.Parameters.shear_viscosity_0 = 1.0 + stokes.bodyforce = [0.0, 0.0, 0.0] + for wall in ("Bottom", "Top", "Left", "Right", "Front", "Back"): + stokes.add_dirichlet_bc((y - 0.5, 0.0, 0.0), wall) + net.apply(stokes) + stokes.petsc_use_pressure_nullspace = True + stokes.tolerance = 1e-4 + info = net.solve(stokes) + assert info.get("converged") and info.get("velocity_pc") == "custom-FMG", info + stats = stokes._custom_mg["hierarchy"].level_transfer_stats + n_static = len(mesh._custom_mg_coarse_meshes) - 1 + assert stats[n_static] is not None and "fetched_remote" in stats[n_static], stats diff --git a/tests/parallel/test_1103_custom_mg_empty_rank_parallel.py b/tests/parallel/test_1103_custom_mg_empty_rank_parallel.py new file mode 100644 index 000000000..5b131dbbc --- /dev/null +++ b/tests/parallel/test_1103_custom_mg_empty_rank_parallel.py @@ -0,0 +1,100 @@ +"""Custom-P multigrid on more ranks than its coarsest level has cells. + +A coarse level is small by construction, so on a large run most ranks own none +of its cells, and the fine mesh of a placed fault is gathered onto one rank, so +most ranks own none of the finest cells either. Found on kaiju at np 104 (a +124-cell coarsest level, the faulted annulus): ``_colocate_level`` shaped its +centroid arrays from the cells it found, an empty rank raised IndexError, and +the ranks that had cells waited in the next all-gather for ever. + +Run with at least four ranks; the 1 x 1 box at cell size 1 has four cells. +""" +import numpy as np +import pytest + +import underworld3 as uw +from underworld3.utilities import custom_mg + +pytestmark = [pytest.mark.mpi(min_size=4), pytest.mark.level_2, pytest.mark.tier_b, + pytest.mark.timeout(300)] + + +def _box(cellSize, refinement=None): + return uw.meshing.UnstructuredSimplexBox( + minCoords=(0.0, 0.0), maxCoords=(1.0, 1.0), cellSize=cellSize, + regular=True, refinement=refinement, qdegree=2) + + +def _cells(mesh): + start, end = mesh.dm.getHeightStratum(0) + return end - start + + +def _wrap(dm, like): + return uw.discretisation.Mesh( + dm.clone(), simplex=True, + coordinate_system_type=like.CoordinateSystem.coordinate_type, + qdegree=2, boundaries=like.boundaries) + + +def _poisson(mesh): + p = uw.systems.Poisson(mesh) + p.constitutive_model = uw.constitutive_models.DiffusionModel + p.constitutive_model.Parameters.diffusivity = 1 + p.f = 0.0 + p.add_dirichlet_bc(0.0, "Bottom") + p.add_dirichlet_bc(1.0, "Top") + p.petsc_options["ksp_rtol"] = 1e-8 + p.petsc_options["ksp_type"] = "cg" + return p + + +def test_colocating_a_level_with_empty_ranks(): + """Every rank returns, and the coarse level keeps all its cells.""" + base = _box(1.0, refinement=3) + coarsest = base._coarse_level_meshes()[0] + total = uw.mpi.comm.allreduce(_cells(coarsest)) + empty = uw.mpi.comm.allreduce(int(_cells(coarsest) == 0 and _cells(base) == 0)) + assert total < uw.mpi.size and empty > 0, (total, empty) # the case under test + + moved = custom_mg._colocate_level(coarsest, base) + assert uw.mpi.comm.allreduce(_cells(moved)) == total + + +def test_custom_fmg_with_a_coarsest_level_smaller_than_the_ranks(): + """The custom-P solve installs its multigrid and agrees with GAMG.""" + m0 = _box(1.0) + dm1 = m0.dm.refine() + dm2 = dm1.refine() + dm3 = dm2.refine() + assert uw.mpi.comm.allreduce(_cells(m0)) < uw.mpi.size # the case under test + + reference = _poisson(_wrap(dm3, m0)) + reference.preconditioner = "gamg" + reference.solve() + + solver = _poisson(_wrap(dm3, m0)) + custom_mg.set_custom_fmg(solver, [_wrap(m0.dm, m0), _wrap(dm1, m0), _wrap(dm2, m0)], + builder="barycentric") + solver.solve() + + assert solver.snes.getKSP().getPC().getType() == "mg" + assert solver.snes.getConvergedReason() > 0 + diff = np.asarray(solver.Unknowns.u.data) - np.asarray(reference.Unknowns.u.data) + num = uw.mpi.comm.allreduce(float(np.sum(diff ** 2))) + den = uw.mpi.comm.allreduce(float(np.sum(np.asarray(reference.Unknowns.u.data) ** 2))) + assert (num / den) ** 0.5 < 1.0e-6 + + + +def test_cutting_a_line_with_empty_ranks(): + """The fault line cut runs on every rank and splits the edges it crosses.""" + from underworld3.utilities.line_cut import cut_along_lines + mesh = _box(1.0) + empty = uw.mpi.comm.allreduce(int(_cells(mesh) == 0)) + assert empty > 0, empty # the case under test + line = np.array([[-0.1, 0.55], [1.1, 0.55]]) # wall to wall + cut, _info = cut_along_lines(mesh.dm, [line]) + start, end = cut.getHeightStratum(0) + assert uw.mpi.comm.allreduce(end - start) > uw.mpi.comm.allreduce(_cells(mesh)) + diff --git a/tests/test_0077_model_registry_does_not_pin.py b/tests/test_0077_model_registry_does_not_pin.py new file mode 100644 index 000000000..01ce18a72 --- /dev/null +++ b/tests/test_0077_model_registry_does_not_pin.py @@ -0,0 +1,210 @@ +"""The default model's registries do not keep a dropped mesh alive. + +Every mesh, mesh variable and solver registers with the default model. The +registries held them by strong reference, so a mesh the caller had dropped +-- with its DM hierarchy, its variables' vectors and its solver's matrices -- +lived for the rest of the process. A driver that rebuilds a faulted mesh +each time a fault moves therefore kept every mesh it ever built: measured on +a box whose straight fault moves at each rebuild, 9 meshes, 21 DMs, +12 matrices, one KSP and one SNES more after each rebuild, 53 MB per rebuild +at k = 3 in serial, and about 0.4 GB per rank per rebuild in the kaiju +production runs (an out-of-memory kill after 24 rebuilds). The swarms and +state-bearers were already held weakly; the meshes, variables and solvers +now are too. +""" +import gc +import weakref + +import numpy as np +import pytest + +import underworld3 as uw + +# level 2: two of the tests solve; tier B: proven on their bugs, not yet hardened +pytestmark = [pytest.mark.level_2, pytest.mark.tier_b] + + +@pytest.fixture(autouse=True) +def _fresh_model(): + uw.reset_default_model() + yield + uw.reset_default_model() + + +def _dropped(make): + """Weak references to what ``make`` returns, after it is dropped.""" + refs = {name: weakref.ref(obj) for name, obj in make().items()} + gc.collect() + return {name: ref() is None for name, ref in refs.items()} + + +def test_a_dropped_mesh_with_a_variable_and_a_solver_is_freed(): + def make(): + mesh = uw.meshing.UnstructuredSimplexBox( + minCoords=(0.0, 0.0), maxCoords=(1.0, 1.0), cellSize=0.25) + variable = uw.discretisation.MeshVariable("T", mesh, 1, degree=1) + poisson = uw.systems.Poisson(mesh, u_Field=variable) + return {"mesh": mesh, "variable": variable, "solver": poisson} + + freed = _dropped(make) + assert all(freed.values()), freed + + +def test_a_dropped_faulted_rebuild_is_freed(): + """The production pattern: adapt, add_fault, solve, drop.""" + base = uw.meshing.UnstructuredSimplexBox( + minCoords=(0.0, 0.0), maxCoords=(1.0, 1.0), cellSize=0.1, + refinement=1, qdegree=2) + + def make(): + tip_a, tip_b = np.array([0.30, 0.45]), np.array([0.70, 0.55]) + samples = np.column_stack([np.linspace(tip_a[0], tip_b[0], 9), + np.linspace(tip_a[1], tip_b[1], 9), + np.zeros(9)]) + guide = uw.meshing.Surface("guide", base, samples) + guide.discretize() + metric = guide.refinement_metric_function( + h_near=0.0125, h_far=0.05, width=0.08, profile="linear") + child = base.adapt(metric, max_levels=2) + mesh = child.add_fault([("Flt", np.vstack([tip_a, tip_b]))]) + _x, y = mesh.X + stokes = uw.systems.Stokes(mesh) + stokes.constitutive_model = uw.constitutive_models.ViscousFlowModel + stokes.constitutive_model.Parameters.shear_viscosity_0 = 1.0 + stokes.petsc_use_pressure_nullspace = True + for wall in ("Top", "Bottom", "Left", "Right"): + stokes.add_dirichlet_bc((y - 0.5, 0.0), wall) + stokes.add_fault_bc(0, "Flt") + stokes.solve() + return {"finest": mesh, "child": child, "solver": stokes} + + freed = _dropped(make) + assert all(freed.values()), freed + + +def test_a_variable_keeps_its_mesh(): + """A helper may return only the variable: its mesh stays usable.""" + def make(): + mesh = uw.meshing.UnstructuredSimplexBox( + minCoords=(0.0, 0.0), maxCoords=(1.0, 1.0), cellSize=0.25) + return uw.discretisation.MeshVariable("T", mesh, 1, degree=1) + + variable = make() + gc.collect() + variable.array[...] = 1.0 + assert variable.mesh is not None + assert float(variable.array.max()) == 1.0 + + +def test_a_solver_that_solved_in_a_step_is_freed(monkeypatch, tmp_path): + """The step transcript names each part's object; it must not keep it.""" + monkeypatch.setenv("UW_TRANSCRIPT", "on") + monkeypatch.chdir(tmp_path) + uw.reset_default_model() + + def make(): + mesh = uw.meshing.UnstructuredSimplexBox( + minCoords=(0.0, 0.0), maxCoords=(1.0, 1.0), cellSize=0.25) + poisson = uw.systems.Poisson(mesh) + poisson.constitutive_model = uw.constitutive_models.DiffusionModel + poisson.constitutive_model.Parameters.diffusivity = 1.0 + poisson.f = 1.0 + poisson.add_dirichlet_bc(0.0, "Bottom") + with uw.get_default_model().step(0.1): + poisson.solve() + return {"mesh": mesh, "solver": poisson} + + freed = _dropped(make) + assert all(freed.values()), freed + + +def test_the_primary_mesh_is_released_and_replaced(): + """A freed primary mesh is not reported, and the next mesh takes over.""" + def make(): + mesh = uw.meshing.UnstructuredSimplexBox( + minCoords=(0.0, 0.0), maxCoords=(1.0, 1.0), cellSize=0.25) + return {"mesh": mesh} + + freed = _dropped(make) + assert freed["mesh"] + assert uw.get_default_model().mesh is None + replacement = uw.meshing.UnstructuredSimplexBox( + minCoords=(0.0, 0.0), maxCoords=(1.0, 1.0), cellSize=0.25) + assert uw.get_default_model().mesh is replacement + + +def test_a_snapshot_does_not_visit_a_dropped_mesh(): + """A dropped mesh with a variable is a reference cycle (the mesh owns its + variables, each variable holds its mesh), so it stays in the weak registry + until the cyclic collector runs, and that runs at a different moment on + each rank. A snapshot makes a collective write per registered mesh, so it + collects first: every rank then visits the same meshes. Here the collector + is held off, as it may be on one rank and not another.""" + kept = uw.meshing.UnstructuredSimplexBox( + minCoords=(0.0, 0.0), maxCoords=(1.0, 1.0), cellSize=0.25) + + def make(): + mesh = uw.meshing.UnstructuredSimplexBox( + minCoords=(0.0, 0.0), maxCoords=(1.0, 1.0), cellSize=0.25) + uw.discretisation.MeshVariable("T", mesh, 1, degree=1) + + gc.disable() + try: + make() + state = uw.get_default_model().save_state() + finally: + gc.enable() + assert state.mesh_names == [kept.name] + + +def test_a_swarm_keeps_its_mesh(): + """A swarm on a mesh nobody else holds keeps that mesh; it found it by id + in the weak registry, so the mesh was freed under it.""" + def make(): + mesh = uw.meshing.UnstructuredSimplexBox( + minCoords=(0.0, 0.0), maxCoords=(1.0, 1.0), cellSize=0.25) + return uw.swarm.Swarm(mesh) + + swarm = make() + gc.collect() + assert swarm.mesh is not None + swarm.populate(fill_param=1) + assert swarm.mesh.dim == 2 + + +def test_a_variable_the_caller_did_not_keep_stays_registered(): + """The mesh owns the user-facing variable, not only its base: the model + indexes the user-facing one, and units, scaling and the adjoint find + variables there.""" + mesh = uw.meshing.UnstructuredSimplexBox( + minCoords=(0.0, 0.0), maxCoords=(1.0, 1.0), cellSize=0.25) + + def make(): + uw.discretisation.MeshVariable("T", mesh, 1, degree=1) + + make() + gc.collect() + variable = uw.get_default_model().get_variable("T") + assert variable is not None + assert variable._base_var is mesh.vars[variable.clean_name] + + +def test_a_proxy_on_a_kept_mesh_does_not_keep_the_swarm_variable(): + """A proxied swarm variable puts its proxy on the swarm's mesh, which owns + the proxy. The proxy's remesh callback held the variable strongly, so a + swarm moved off a long-lived mesh left that mesh keeping the variable and, + through it, the swarm's new mesh.""" + origin = uw.meshing.UnstructuredSimplexBox( + minCoords=(0.0, 0.0), maxCoords=(1.0, 1.0), cellSize=0.25) + + def make(): + moved_to = uw.meshing.UnstructuredSimplexBox( + minCoords=(0.0, 0.0), maxCoords=(1.0, 1.0), cellSize=0.2) + swarm = uw.swarm.Swarm(origin) + variable = uw.swarm.SwarmVariable("S", swarm, 1, proxy_degree=1) + swarm.mesh = moved_to + return {"moved_to": moved_to, "variable": variable, "swarm": swarm} + + freed = _dropped(make) + assert origin.dim == 2 + assert all(freed.values()), freed diff --git a/tests/test_0772_surface_extract.py b/tests/test_0772_surface_extract.py index f180c13fc..b0ea1da73 100644 --- a/tests/test_0772_surface_extract.py +++ b/tests/test_0772_surface_extract.py @@ -95,17 +95,11 @@ def test_extract_surface_roundtrip_bit_exact(builder): # --------------------------------------------------------------------------- # on-surface symbolic evaluate # --------------------------------------------------------------------------- -@pytest.mark.parametrize("builder", [ - _shell, - pytest.param(_annulus, marks=pytest.mark.xfail( - reason="Point-location evaluate on a 1-manifold surface (dim=1, " - "cdim=2) is not yet supported: the in-cell test's FE element-entity " - "handling for edge cells is incomplete. Full dim=1 manifold support " - "(point-location + FE solve) is a tracked follow-up; the free-surface " - "smoother uses restrict/prolongate + surface-graph adjacency, which " - "do not need 1D point-location.", - strict=False)), -]) +# The 1-manifold case (an annulus boundary, dim=1, cdim=2) was an expected +# failure while the mesh's element table had no line entry: point location +# read 4 vertices per line cell. With the entry it evaluates exactly +# (2026-10-18, serial and 2 and 3 ranks; tests/test_1100). +@pytest.mark.parametrize("builder", [_shell, _annulus]) def test_extract_surface_evaluate(builder): m = builder() surf = m.extract_surface("Upper") diff --git a/tests/test_0848_fault_split_3d.py b/tests/test_0848_fault_split_3d.py index fa3c98d43..9e293860f 100644 --- a/tests/test_0848_fault_split_3d.py +++ b/tests/test_0848_fault_split_3d.py @@ -268,6 +268,42 @@ def test_coulomb_slide_stick_3d(split_box): assert results[0.4] < 5e-3 # stuck at ~V0 creep +@pytest.mark.level_2 +@pytest.mark.tier_b +@pytest.mark.parametrize("state", ["uniform", "linear"]) +def test_normal_traction_is_pointwise_3d(split_box, state): + """The recovered normal traction matches an exact Stokes state at EVERY + pair node, vertices and edge midpoints alike. + + On the P2 triangle the consistent load vanishes at the vertex nodes, so + dividing the reaction by a lumped nodal area read 0 at vertices and 4/3 + of the traction at midpoints — right as an integral, wrong pointwise, + and hidden by a median over the patch. The x-normal patch sees + uniform: v = (-(x-c), (y-c), 0) sigma_nn = -2 + linear: v = (-(x-c)(1+(y-c)), (y-c)+(y-c)^2/2, 0) sigma_nn = -2 - 3(y-c) + both exact Stokes solutions with a zero-mean pressure. + """ + from underworld3.utilities import fault_contact + + x, y, z = split_box.X + if state == "uniform": + drive = (-(x - 0.5), (y - 0.5), 0.0) + exact = lambda P: np.full(len(P), -2.0) + tolerance = 1e-3 + else: + drive = (-(x - 0.5) * (1 + (y - 0.5)), (y - 0.5) + (y - 0.5) ** 2 / 2, 0.0) + exact = lambda P: -2.0 - 3.0 * (P[:, 1] - 0.5) + tolerance = 0.06 + + stokes = _stokes_on(split_box, drive, f"nt_{state}") + stokes.petsc_use_pressure_nullspace = True + stokes.add_fault_bc(500.0, boundary="FltA") + fault_contact.solve_with_fault(stokes) + crds, sig = fault_contact.fault_normal_traction( + stokes, "FltA", stokes._rotated_freeslip_info) + assert np.abs(sig - exact(crds)).max() < tolerance + + @pytest.mark.level_1 @pytest.mark.tier_b def test_daylighting_patch_refused(): @@ -314,3 +350,85 @@ def test_fault_surface_route(): fs2.triangulate() with pytest.raises(NotImplementedError, match="planar"): fs2.rim_polygon() + + +@pytest.mark.level_2 +@pytest.mark.tier_b +def test_split_sheet_keeps_the_multigrid_tail(): + """Splitting a sheet placed on an adapt child keeps geometric multigrid. + + The adapt-on-top 3-D fault pipeline is ``adapt`` -> ``add_conforming_sheet`` + -> ``split_fault``. The split re-represents the same grid (only the fault + vertices are duplicated), so the coarse tail the placed mesh owns serves + the split mesh unchanged. Without it the fault solve silently fell back + to GAMG. + """ + from underworld3.utilities import fault_contact + + centre, radius, h = np.array([0.5, 0.5, 0.5]), 0.3, 0.1 + normal = np.array([0.3, 0.5, 0.81]) + normal /= np.linalg.norm(normal) + + def metric(pts): + rel = pts - centre + dn = rel @ normal + r = np.linalg.norm(rel - dn[:, None] * normal, axis=1) + d = np.hypot(dn, np.maximum(r - radius, 0.0)) + return 1.0 / np.minimum(h + d, 0.2) ** 2 + + base = uw.meshing.UnstructuredSimplexBox( + minCoords=(0.0,) * 3, maxCoords=(1.0,) * 3, cellSize=0.2, + regular=False, qdegree=2, refinement=1) + child = base.adapt(metric, max_levels=1) + e1 = np.cross(normal, [1.0, 0.0, 0.0]) + e1 /= np.linalg.norm(e1) + e2 = np.cross(normal, e1) + n_rim = int(np.ceil(2.0 * np.pi * radius / h)) + a = np.linspace(0.0, 2.0 * np.pi, n_rim, endpoint=False) + pts = np.vstack([centre, centre + radius * (np.outer(np.cos(a), e1) + + np.outer(np.sin(a), e2))]) + tris = np.array([(0, 1 + i, 1 + (i + 1) % n_rim) for i in range(n_rim)]) + placed = child.add_conforming_sheet(pts, tris, "Fault", size=h) + split = split_fault(placed, "Fault") + + assert split._custom_mg_coarse_meshes is not None, ( + "split_fault dropped the placed mesh's multigrid tail") + assert split._custom_mg_coarse_meshes == placed._custom_mg_coarse_meshes + assert split._custom_mg_builder == placed._custom_mg_builder + + v = uw.discretisation.MeshVariable("v_tail", split, 3, degree=2) + p = uw.discretisation.MeshVariable("p_tail", split, 1, degree=0, + continuous=False) + stokes = uw.systems.Stokes(split, velocityField=v, pressureField=p) + stokes.constitutive_model = uw.constitutive_models.ViscousFlowModel + stokes.constitutive_model.Parameters.shear_viscosity_0 = 1.0 + stokes.bodyforce = [0.0, 0.0, 0.0] + drive = (-(split.X[0] - 0.5), split.X[1] - 0.5, 0.0) + for wall in ("Bottom", "Top", "Right", "Left", "Front", "Back"): + stokes.add_dirichlet_bc(drive, wall) + stokes.petsc_use_pressure_nullspace = True + stokes.tolerance = 1e-6 + stokes.add_fault_bc(200.0, boundary="Fault") + fault_contact.solve_with_fault(stokes) + info = stokes._rotated_freeslip_info + assert info["velocity_pc"] == "custom-FMG", info["velocity_pc"] + assert info["converged"] + + +@pytest.mark.level_2 +@pytest.mark.tier_b +def test_split_patch_box_keeps_the_refinement_hierarchy(): + """A refined box with a conforming patch is its own multigrid hierarchy: + the split takes the native coarse levels as its tail (the finest level is + the split mesh itself), so a refined cuboid needs no registered coarse + meshes for geometric multigrid.""" + mesh = uw.meshing.BoxInternalPatch(cellSize=0.4, patch_points=PATCH, + patch_name="FltA", refinement=1) + assert len(mesh.dm_hierarchy) == 2 + split = split_fault(mesh, "FltA") + tail = split._custom_mg_coarse_meshes + assert tail is not None and len(tail) == 1, tail + # the one coarse level is the unrefined patch box, half the finest's cells + n_coarse = tail[0].dm.getHeightStratum(0)[1] - tail[0].dm.getHeightStratum(0)[0] + n_fine = split.dm.getHeightStratum(0)[1] - split.dm.getHeightStratum(0)[0] + assert 6 * n_coarse <= n_fine <= 10 * n_coarse, (n_coarse, n_fine) diff --git a/tests/test_0867_place_sheet_adapted_3d.py b/tests/test_0867_place_sheet_adapted_3d.py new file mode 100644 index 000000000..2fbb0671b --- /dev/null +++ b/tests/test_0867_place_sheet_adapted_3d.py @@ -0,0 +1,88 @@ +"""Placing a sheet into an ``adapt()`` child in 3-D. + +The fault workflow is base mesh -> ``adapt`` toward the fault -> +``add_conforming_sheet`` -> ``split_fault``. On a newest-vertex-bisection +child the cavity the carve clears, and the fill gmsh returns for it, can be +wrong in three ways that the earlier gates did not see, all at the default +clearance and none on the gmsh base: + +* a KEPT cell (or cluster) wholly enclosed by dropped cells — its faces + form a separate closed component of the shell, gmsh fills it as part of + the cavity, and the fill overlaps the kept cell (measured: the domain + volume grew by exactly that cell's volume, refused by the volume gate); +* a PINCHED vertex — every shell edge has two faces, but the faces around + the vertex form two or more separate fans — where gmsh returns + zero-volume tets (refused by PETSc's geometry check, ``|J| = 0``); +* a FLAT tet spanning four nearly coplanar shell nodes — volume ~1e-20, + which no gate refused, from either of gmsh's Delaunay or HXT algorithms + depending on the orientation. +""" +import numpy as np +import pytest + +import underworld3 as uw +from underworld3.utilities.place_surface import _cell_volumes_signed6 + +pytestmark = [pytest.mark.level_2, pytest.mark.tier_b] + +CENTRE = np.array([0.5, 0.5, 0.5]) +RADIUS = 0.2 +H_NEAR = 0.05 +TILTED = np.array([0.3, -0.5, 0.8]) / np.linalg.norm([0.3, -0.5, 0.8]) +DIAGONAL = np.array([1.0, 1.0, 1.0]) / np.sqrt(3.0) + + +def _disc_sheet(normal, n_rim=32): + """A planar disc as a centre fan; ``size=`` re-triangulates it.""" + helper = np.eye(3)[np.argmin(np.abs(normal))] + e1 = np.cross(normal, helper) + e1 /= np.linalg.norm(e1) + e2 = np.cross(normal, e1) + a = np.linspace(0.0, 2.0 * np.pi, n_rim, endpoint=False) + rim = CENTRE + RADIUS * (np.outer(np.cos(a), e1) + np.outer(np.sin(a), e2)) + pts = np.vstack([CENTRE, rim]) + tris = np.array([(0, 1 + i, 1 + (i + 1) % n_rim) for i in range(n_rim)], + dtype=np.int64) + return pts, tris + + +def _graded_metric(normal): + """M = 1/h^2 from the exact distance to the disc, h_near -> 2 h_near.""" + def metric(X): + rel = np.asarray(X)[:, :3] - CENTRE + zn = rel @ normal + r = np.linalg.norm(rel - zn[:, None] * normal, axis=1) + d = np.sqrt(np.maximum(r - RADIUS, 0.0) ** 2 + zn ** 2) + h = np.clip(H_NEAR + d * H_NEAR / 0.4, H_NEAR, 2.0 * H_NEAR) + return 1.0 / h**2 + return metric + + +def _uniform_metric(normal): + return lambda X: np.full(len(X), 1.0 / H_NEAR**2) + + +@pytest.fixture(scope="module") +def base(): + return uw.meshing.UnstructuredSimplexBox( + minCoords=(0.0, 0.0, 0.0), maxCoords=(1.0, 1.0, 1.0), + cellSize=0.2, regular=False, qdegree=2, refinement=1) + + +@pytest.mark.parametrize( + "metric, normal", + [(_graded_metric, TILTED), (_uniform_metric, TILTED), + (_graded_metric, DIAGONAL)], + ids=["graded-enclosed-cell", "uniform-pinch-and-flat", "graded-flat"]) +def test_sheet_places_into_an_nvb_child(base, metric, normal): + """The placed mesh keeps the domain volume, one orientation, and no + degenerate cell, and every sheet triangle is a labelled face.""" + child = base.adapt(metric(normal), max_levels=1) + pts, tris = _disc_sheet(normal) + placed = child.add_conforming_sheet(pts, tris, "Fault", size=H_NEAR) + + v6 = _cell_volumes_signed6(placed.dm) + assert abs(np.abs(v6).sum() / 6.0 - 1.0) < 1e-9 + assert (np.sign(v6) == np.sign(v6[0])).all() + assert np.abs(v6).min() / 6.0 > 1e-12 + assert placed._surface_info["n_surface_facets"] > 0 diff --git a/tests/test_1018_rotated_freeslip.py b/tests/test_1018_rotated_freeslip.py index 6a7f56c30..214d3de3b 100644 --- a/tests/test_1018_rotated_freeslip.py +++ b/tests/test_1018_rotated_freeslip.py @@ -11,7 +11,9 @@ from underworld3 import analytic as A from underworld3.utilities import custom_mg -pytestmark = [pytest.mark.level_1, pytest.mark.tier_a] +# level on the module, tier on each test: the two newest are tier B (proven, +# not yet hardened) +pytestmark = [pytest.mark.level_1] _EPS = np.finfo(float).eps @@ -42,6 +44,7 @@ def _solcx_essential(mesh, sol): return v +@pytest.mark.tier_a def test_rotated_freeslip_box_reproduces_essential(): """Box: rotated free-slip on all 4 axis-aligned walls == native essential free-slip.""" mesh = uw.meshing.StructuredQuadBox( @@ -69,6 +72,7 @@ def test_rotated_freeslip_box_reproduces_essential(): assert sol.velocity_error(v) < 1e-3 +@pytest.mark.tier_a def test_rotated_linear_workspace_reuses_unchanged_operator(): """Repeated linear solves reuse the rotated workspace across solves. @@ -329,6 +333,7 @@ def _rampable_rotated_stokes(mesh, k_expr, tag, forcing=None): return s, v +@pytest.mark.tier_a def test_rotated_workspace_constant_ramp_invalidates(): """THE blind-spot regression (PR #418 review, unresolved finding): a rampable UWexpression constant changes value with NO state-counter bump @@ -382,6 +387,7 @@ def test_rotated_workspace_constant_ramp_invalidates(): assert half < 1e-6, f"ramped solve is not the halved velocity ({half:.2e})" +@pytest.mark.tier_a def test_rotated_workspace_deform_invalidates(): """mesh.deform between solves: the geometry changed, so the whole workspace must be rebuilt and the answer must match a fresh solver on the deformed @@ -558,6 +564,7 @@ def test_rotated_workspace_deform_invalidates(): f"survived the deform.") +@pytest.mark.tier_a @pytest.mark.level_2 def test_rotated_freeslip_spherical_shell_3d(): """3D spherical shell, free-slip inner+outer (the Zhong #248 configuration): @@ -689,6 +696,7 @@ def fit(mask): ) +@pytest.mark.tier_a def test_p2_triangle_mass_has_zero_vertex_row_sums(): """WHY mass='auto' is P1-projected on a 3D P2 trace, and not the consistent solve. @@ -714,6 +722,7 @@ def test_p2_triangle_mass_has_zero_vertex_row_sums(): assert np.all(row_sums[3:] > 0.0), "P2 triangle midpoint rows should be positive" +@pytest.mark.tier_a @pytest.mark.level_2 def test_rotated_freeslip_spherical3d_reaction_topography(): """3D reaction loads must be divided by boundary mass to recover pointwise stress. @@ -749,6 +758,7 @@ def test_rotated_freeslip_spherical3d_reaction_topography(): assert np.isclose(cmb_midpoints, 0.77060, rtol=0.05) +@pytest.mark.tier_a def test_rotated_freeslip_annulus_zero_leakage(): """Annulus: per-node radial free-slip on both arcs → machine-zero v_r leakage, with the analytic radial normal.""" @@ -785,6 +795,7 @@ def test_rotated_freeslip_annulus_zero_leakage(): assert rotfrac < 1e-8, f"rotation gauge {rotfrac:.2e} not removed" +@pytest.mark.tier_a def test_rotated_freeslip_geometric_fmg_velocity_block(): """The rotated free-slip velocity block is driven by GEOMETRIC FMG on a custom prolongation (set_custom_fmg) — no direct solve — and still reproduces the @@ -838,6 +849,7 @@ def test_rotated_freeslip_geometric_fmg_velocity_block(): f"(inexact Schur application — is the velocity sub-KSP `preonly` again?)") +@pytest.mark.tier_a def test_rotated_freeslip_boundary_normal_traction_solcx(): """sigma_nn recovered by boundary_normal_traction on the top boundary reproduces the exact SolCx analytic sigma_yy (mean-removed) to a few percent. Exercises the @@ -876,6 +888,7 @@ def test_rotated_freeslip_boundary_normal_traction_solcx(): assert relL2 < 0.10, f"sigma_nn relL2 vs analytic sigma_yy {relL2:.3f} too large" +@pytest.mark.tier_a def test_rotated_freeslip_sigma_nn_lumped_no_overshoot(): """The default (lumped) sigma_nn de-smear is MONOTONE at the SolCx viscosity jump: its total variation matches the analytic (no Gibbs overshoot), whereas the @@ -962,6 +975,7 @@ def plaw_box_ref(): return mesh, np.copy(vE.data), its +@pytest.mark.tier_a def test_rotated_freeslip_nonlinear_matches_essential(plaw_box_ref): """Frozen (Picard) tangent through the rotated path: genuinely iterates and converges to the native essential nonlinear free-slip answer (both impose v_n=0 — @@ -989,6 +1003,7 @@ def test_rotated_freeslip_nonlinear_matches_essential(plaw_box_ref): assert leak < 1e-10, f"{lab} wall-normal velocity {leak:.2e} not machine-zero" +@pytest.mark.tier_a def test_rotated_freeslip_newton_tangent(plaw_box_ref): """consistent_jacobian=True is genuine Newton through the rotated path: it converges in about the same small iteration count as the native essential Newton @@ -1007,6 +1022,7 @@ def test_rotated_freeslip_newton_tangent(plaw_box_ref): assert np.linalg.norm(vR.data - vE) / np.linalg.norm(vE) < 1e-6 +@pytest.mark.tier_a def test_rotated_freeslip_continuation_tangent(plaw_box_ref): """consistent_jacobian='continuation' works through the rotated path: a staged Picard→Newton solve (α=0 to the loose newton_switch_rtol, then α=1) that switches @@ -1031,6 +1047,72 @@ def test_rotated_freeslip_continuation_tangent(plaw_box_ref): "picard did not extend the continuation Picard phase") +@pytest.mark.tier_b +def test_rotated_continuation_warm_start_switches_to_newton_at_once(plaw_box_ref): + """A warm re-solve under consistent_jacobian='continuation' is already in + Newton's basin, and must switch at once. The switch is measured against the + same reference scale as convergence (the larger of the start's residual and + the rest-state residual), not against the warm start's own residual: against + that, a good start must still shrink its residual a hundredfold under Picard + before Newton is allowed. Measured 2026-10-23 on this box, the body force + changed by 1% from a converged state: the switch at iteration 12 and 13 + iterations in all, against 17 for the cold solve.""" + mesh, _, _ = plaw_box_ref + s, _, _ = _powerlaw_stokes(mesh, "wsR", cj="continuation") + for wall in ("Top", "Bottom", "Left", "Right"): + s.add_rotated_freeslip_bc(0, wall) + s.solve() + assert s._rotated_freeslip_info["continuation_switched"] + + x, _y = mesh.X + s.bodyforce = sympy.Matrix([[0.0, -2.02 * sympy.cos(sympy.pi * x)]]) + s.solve(zero_init_guess=False) + info = s._rotated_freeslip_info + assert info["converged"], info + assert info["continuation_switch_iteration"] in (0, 1), info["continuation_switch_iteration"] + assert info["nonlinear_iterations"] <= 4, info["nonlinear_iterations"] + + +@pytest.mark.tier_b +def test_rotated_solve_records_its_diagnostics(plaw_box_ref): + """The rotated solve reports what it measured, so a campaign can record it per + step without scraping verbose logs: the rest-state residual ‖F̂(0)‖ and the + reference the convergence and switch tests use, the residual at every iteration + start, and the accepted line-search step of every increment (0.0 when the + increment was not applied).""" + mesh, _, _ = plaw_box_ref + s, _, _ = _powerlaw_stokes(mesh, "dgR", cj="continuation") + for wall in ("Top", "Bottom", "Left", "Right"): + s.add_rotated_freeslip_bc(0, wall) + + def check(info): + its = info["nonlinear_iterations"] + hist, alpha = info["rnorm_history"], info["line_search_alpha"] + assert len(hist) in (its, its + 1), (len(hist), its) + assert hist[0] == info["rnorm0"] and hist[-1] == info["rnorm"], (hist[0], hist[-1]) + assert len(alpha) == its, (len(alpha), its) + assert all(a == 0.0 or 0.0 < a <= 1.0 for a in alpha), alpha + return its, alpha + + s.solve() + cold = s._rotated_freeslip_info + assert cold["converged"], cold + check(cold) + assert cold["fnorm_rest"] is None # a cold start's r0 IS the rest scale + assert cold["ref"] == cold["rnorm0"] + + x, _y = mesh.X + s.bodyforce = sympy.Matrix([[0.0, -2.02 * sympy.cos(sympy.pi * x)]]) + s.solve(zero_init_guess=False) + warm = s._rotated_freeslip_info + assert warm["converged"], warm + _its, alpha = check(warm) + assert warm["fnorm_rest"] > 0.0 + assert warm["ref"] == max(warm["rnorm0"], warm["fnorm_rest"]) + assert alpha[-1] == 1.0, alpha # a full step is accepted at the solution + + +@pytest.mark.tier_a def test_rotated_freeslip_picard_under_newton_stages_picard_then_newton(plaw_box_ref): """picard>0 under the consistent-Newton tangent (the default since 2026-10-05) stages at least N Picard iterations, then Newton, on the blended Jacobian solve() compiles @@ -1050,6 +1132,7 @@ def test_rotated_freeslip_picard_under_newton_stages_picard_then_newton(plaw_box assert np.linalg.norm(vR.data - vE) / np.linalg.norm(vE) < 1e-6 +@pytest.mark.tier_a def test_rotated_freeslip_nonlinear_warm_start(plaw_box_ref): """Warm-start (zero_init_guess=False) through the nonlinear rotated path: a 2-step 'time loop' re-solving from the previous converged state stays correct and @@ -1065,6 +1148,7 @@ def test_rotated_freeslip_nonlinear_warm_start(plaw_box_ref): assert s._rotated_freeslip_info["nonlinear_iterations"] <= 3 +@pytest.mark.tier_a def test_rotated_freeslip_nonlinear_geometric_fmg(): """The NONLINEAR rotated free-slip velocity block is driven by geometric FMG on the custom prolongation (set_custom_fmg) — the rotated prolongation is built once @@ -1091,6 +1175,7 @@ def test_rotated_freeslip_nonlinear_geometric_fmg(): assert rel < 5e-3, f"nonlinear FMG rotated free-slip differs from essential by {rel:.2e}" +@pytest.mark.tier_a def test_rotated_freeslip_nonlinear_annulus_zero_leakage(): """Genuinely-rotated frame under nonlinear iteration: a power-law annulus with per-node radial free-slip on both arcs genuinely iterates, gives machine-zero @@ -1132,6 +1217,7 @@ def test_rotated_freeslip_nonlinear_annulus_zero_leakage(): assert leak < 1e-10, f"{lab} radial leakage {leak:.2e} not machine-zero" +@pytest.mark.tier_a def test_rotated_freeslip_dynamic_topography_field(): """dynamic_topography writes h = -(sigma_nn - mean)/(rho g) onto a scalar surface MeshVariable (the free-surface hand-off): the field at the top vertices reproduces @@ -1203,6 +1289,7 @@ def _annulus_datum_solve(mode, tag): return mesh, v +@pytest.mark.tier_a def test_rotated_freeslip_datum_zero_matches_freeslip(): """datum={'Upper': 0} (the prescribed-datum path with a zero datum) reproduces plain rotated free-slip to iterative round-off — the held / free-slip case is untouched. @@ -1214,6 +1301,7 @@ def test_rotated_freeslip_datum_zero_matches_freeslip(): assert d < 1e-10, f"zero datum perturbs the free-slip solution by {d:.2e} (> round-off)" +@pytest.mark.tier_a def test_rotated_freeslip_prescribed_normal_datum(): """A prescribed non-zero wall-normal datum u.n = cos(theta) is imposed to the SAME (machine) precision as u.n = 0 at the constrained surface nodes.""" @@ -1230,6 +1318,7 @@ def test_rotated_freeslip_prescribed_normal_datum(): assert vn.max() > 0.9 and vn.min() < -0.9, "prescribed normal velocity magnitude wrong" +@pytest.mark.tier_a def test_rotated_freeslip_nonlinear_prescribed_normal_datum(): """The prescribed wall-normal datum u.n = cos(theta) through the NONLINEAR rotated path (power-law annulus): the loop genuinely iterates and every iterate is kept @@ -1273,6 +1362,7 @@ def test_rotated_freeslip_nonlinear_prescribed_normal_datum(): assert vn.max() > 0.9 and vn.min() < -0.9, "prescribed normal velocity magnitude wrong" +@pytest.mark.tier_a def test_rotated_solve_fields_carry_inhomogeneous_dirichlet_walls(): """The copy-back gap: essential DOFs are absent from the global vector, so the rotated path's field scatter left them at ZERO wherever the Dirichlet @@ -1314,6 +1404,7 @@ def test_rotated_solve_fields_carry_inhomogeneous_dirichlet_walls(): assert np.abs(vd[mask, 1]).max() < 1e-10 +@pytest.mark.tier_a @pytest.mark.parametrize("use_lu", [False, True]) def test_the_rotated_solve_reports_its_own_convergence(use_lu): """The converged reason must describe the ROTATED solve, not a linear step. diff --git a/tests/test_1058_dropped_meshvariable_aux_layout.py b/tests/test_1058_dropped_meshvariable_aux_layout.py index cdc9d3e6e..09e6e0fdd 100644 --- a/tests/test_1058_dropped_meshvariable_aux_layout.py +++ b/tests/test_1058_dropped_meshvariable_aux_layout.py @@ -1,8 +1,7 @@ """A dropped MeshVariable must not corrupt the auxiliary data of later solves. -`mesh.vars` holds variables weakly, but a DMPlex cannot shed a field: a -variable that is dropped and garbage-collected leaves its PETSc field in -the DM. Two places used to assume the registry and the DM field list line +A DMPlex cannot shed a field: a variable taken off `mesh.vars` and +garbage-collected leaves its PETSc field in the DM. Two places used to assume the registry and the DM field list line up by position: - `Mesh.update_lvec` zipped `mesh.vars.values()` against the DM's field @@ -13,13 +12,14 @@ Measured before the fix: a cell-size (P0) field landing in a P2 slot as garbage, NaN residuals (`DIVERGED_FUNCTION_NANORINF`) in one run and a -subtly wrong answer in the next, depending on when the collector ran. The -default Model holds the only strong reference to a variable (the mesh -outlives the model it was created under), so `uw.reset_default_model()`, -which the test suite runs between tests, releases every variable a script -no longer names; the variable-statistics -helpers also delete temporaries from the registry on purpose. The orphan is -an ordinary state, not a misuse. +subtly wrong answer in the next, depending on when the collector ran. + +A mesh owns its variables (since 2026-10-27; before, the default Model held +the only strong reference and a model reset released them), so a variable +is orphaned when it is taken off the mesh: the variable-statistics helpers +delete their temporaries from `mesh.vars` on purpose, and that leaves the +field. The orphan is an ordinary state, not a misuse; ``_orphan`` makes one +the same way. Run: pixi run python -m pytest tests/test_1058_dropped_meshvariable_aux_layout.py -v """ @@ -39,6 +39,14 @@ def _mesh(): minCoords=(0.0, 0.0), maxCoords=(1.0, 1.0), cellSize=0.25, qdegree=3) +def _orphan(mesh, name, components, degree): + """A variable taken off the mesh and collected: its DM field stays.""" + variable = uw.discretisation.MeshVariable(name, mesh, components, degree=degree) + del mesh.vars[variable.clean_name] + del variable + gc.collect() + + def _poisson_with_field_coefficient(mesh, tag): """A Poisson solve whose answer depends on an auxiliary field (the diffusivity is a MeshVariable), so mis-packed aux data changes it.""" @@ -60,13 +68,7 @@ def test_dropped_variable_leaves_an_orphaned_field(): """The premise: dropping a variable does not shrink the DM.""" mesh = _mesh() n_fields = mesh.dm.getNumFields() - # The mesh keeps the model it was created under alive; a variable - # registers with the CURRENT default model, so a reset before and - # after creating it is what releases it (the suite's per-test reset). - uw.reset_default_model() - uw.discretisation.MeshVariable("temporary", mesh, 2, degree=2) - uw.reset_default_model() - gc.collect() + _orphan(mesh, "temporary", 2, 2) assert "temporary" not in mesh.vars assert mesh.dm.getNumFields() == n_fields + 1 @@ -75,11 +77,8 @@ def test_solve_after_a_dropped_variable_matches_a_clean_mesh(): reference, _k, _u = _poisson_with_field_coefficient(_mesh(), "ref") mesh = _mesh() - uw.reset_default_model() - uw.discretisation.MeshVariable("dropped_vector", mesh, 2, degree=2) - uw.discretisation.MeshVariable("dropped_scalar", mesh, 1, degree=1) - uw.reset_default_model() - gc.collect() + _orphan(mesh, "dropped_vector", 2, 2) + _orphan(mesh, "dropped_scalar", 1, 1) assert mesh.dm.getNumFields() > len(mesh.vars) answer, _k, _u = _poisson_with_field_coefficient(mesh, "orphan") @@ -88,10 +87,7 @@ def test_solve_after_a_dropped_variable_matches_a_clean_mesh(): def test_packed_aux_vector_lands_in_the_named_fields(): mesh = _mesh() - uw.reset_default_model() - uw.discretisation.MeshVariable("dropped", mesh, 2, degree=1) - uw.reset_default_model() - gc.collect() + _orphan(mesh, "dropped", 2, 1) assert "dropped" not in mesh.vars x, y = mesh.X a = uw.discretisation.MeshVariable("a_live", mesh, 1, degree=1) diff --git a/tests/test_1092_thin_sheet_prescribed_normal.py b/tests/test_1092_thin_sheet_prescribed_normal.py new file mode 100644 index 000000000..7814a2c8e --- /dev/null +++ b/tests/test_1092_thin_sheet_prescribed_normal.py @@ -0,0 +1,158 @@ +"""A sheet on the sphere moving along its own normal: the prescribed normal +velocity is the datum of the tangency constraint, and the in-plane strain rate +carries its curvature term. Closed forms on the unit sphere for a uniform +membrane with no basal drag (the tidally flexed shell over an ocean): + +- breathing, w = W: no tangential motion, strain rate W P, sheet stress + sigma = 2 eta (eps + tr eps P) = 6 eta W P; +- a degree-l spheroidal pattern, w = W Y_l: tangential velocity + v = 3 W / (2 l (l + 1) - 1) grad_s Y_l (l = 2: 3W/11), from div_s sigma = 0 + with div_s Hess_s Y = grad_s(Delta_s Y) + grad_s Y on the unit sphere. +""" +import numpy as np +import pytest +import sympy +import underworld3 as uw + +pytestmark = [pytest.mark.level_2, pytest.mark.tier_b] + + +def _sphere(cell_size, degree=2): + return uw.meshing.SphericalManifold(radius=1.0, cellSize=cell_size, degree=1, + qdegree=2 * degree + 1) + + +def _sheet(sphere, eta=1.0): + sheet = uw.systems.ThinSheet(sphere) + sheet.constitutive_model = uw.constitutive_models.ViscousFlowModel + sheet.constitutive_model.Parameters.shear_viscosity_0 = eta + sheet.tolerance = 1.0e-9 + return sheet + + +def _split(sheet): + """Velocity at the nodes split into its normal and tangential parts.""" + c = np.asarray(sheet.u.coords) + u = np.asarray(sheet.u.array[:, 0, :]) + rhat = c / np.linalg.norm(c, axis=1)[:, None] + un = (u * rhat).sum(axis=1) + return c, rhat, un, u - un[:, None] * rhat + + +def test_breathing_sphere(): + """w = W everywhere: the velocity is W r^ at every node to solver + precision, no tangential flow, and the sheet stress is 6 eta W P.""" + W, eta = 0.3, 2.0 + sphere = _sphere(0.25) + sheet = _sheet(sphere, eta) + sheet.normal_velocity = W + sheet.solve() + assert sheet._rotated_freeslip_info["converged"] + c, rhat, un, ut = _split(sheet) + assert np.abs(un - W).max() < 1.0e-9 + # the curvature term is discretised on flat facets through the nodal normal + # field, so a small tangential flow of order h^2 remains (0.25% at h = 0.25) + assert np.abs(ut).max() < 1.0e-2 * W + # the stress: evaluate the sheet's in-plane stress at a few cell interiors + pts = np.asarray(sphere.X.coords)[::97] + pts = pts / np.linalg.norm(pts, axis=1)[:, None] + sig = np.asarray(uw.function.evaluate(sheet.stress, pts)).reshape(len(pts), 3, 3) + P = np.eye(3)[None] - pts[:, :, None] * pts[:, None, :] + assert np.abs(sig - 6 * eta * W * P).max() < 0.1 * 6 * eta * W + + +def test_degree_two_membrane(): + """w = W Y20, Y20 = (3 z^2 - 1)/2: the tangential velocity is + (3W/11) grad_s Y20 = (9W/11) z (z^ - z r^), second order in h.""" + W = 0.5 + errs = [] + for h in (0.3, 0.15): + sphere = _sphere(h) + x, y, z = sphere.X + sheet = _sheet(sphere) + sheet.normal_velocity = W * (3 * z ** 2 - 1) / 2 + sheet.solve() + assert sheet._rotated_freeslip_info["converged"] + c, rhat, un, ut = _split(sheet) + zz = c[:, 2] + assert np.abs(un - W * (3 * zz ** 2 - 1) / 2).max() < 1.0e-8 + exact = (9 * W / 11) * zz[:, None] * (np.array([0.0, 0.0, 1.0])[None] - zz[:, None] * rhat) + errs.append(np.abs(ut - exact).max() / np.abs(exact).max()) + # measured: 7.2% (h 0.3), 3.4% (0.2), 2.5% (0.15) in the max norm; the + # curvature term is first order on flat facets, so the order is between 1 and 2 + assert errs[0] < 0.10 + assert errs[1] < 0.04 + assert errs[1] < 0.6 * errs[0] + + +def test_fault_reaction_is_the_membrane_resultant(): + """A split fault in the breathing sheet carries the membrane stress across + it: the no-opening reaction, de-smeared by the trace mass, is the resultant + L * 6 eta W on every pair (the trace mass once took the triangle's weights + on a surface mesh and read 5 to 8 times that).""" + from underworld3.utilities.fault_contact import fault_normal_traction + W, eta, L = 0.1, 1.0, 0.08 + sphere = _sphere(0.15) + phi = np.linspace(-0.3, 0.3, 9) + split = sphere.add_fault(("Flt", np.column_stack([np.cos(phi), np.sin(phi), np.zeros_like(phi)]))) + sheet = _sheet(split, eta) + sheet.thickness = L + sheet.add_fault_bc(0, "Flt") + sheet.normal_velocity = W + sheet.solve() + _c, sig = fault_normal_traction(sheet, "Flt", sheet._rotated_freeslip_info) + expected = L * 6 * eta * W + assert len(sig) > 5 + assert np.abs(sig - expected).max() < 0.02 * expected + + +@pytest.mark.level_3 +def test_maxwell_transfer_function_under_a_tide(): + """A Maxwell membrane under the degree-2 eccentricity tide: the stress trace + on the cos pattern has amplitude (24/11) mu A * wt / sqrt(1 + wt^2) and lags + the displacement by atan(1/wt), wt = 2 pi De (the velocity is instantaneous: + 3/11 of the pattern's gradient whatever the rheology). Measured at 128 steps + per period: amplitude within 1.5%, lag 32 deg against 28 at De 0.3; at the + 64 steps per period used here the lag is about 4 deg larger still (first + order in the step).""" + De, A, mu = 0.3, 0.01, 1.0 + steps, periods = 64, 3 + sphere = _sphere(0.3) + x, y, z = sphere.X + sheet = uw.systems.ThinSheet(sphere) + sheet.constitutive_model = uw.constitutive_models.ViscoElasticPlasticFlowModel( + sheet.Unknowns, order=1, integrator="etd", objective_rate="none") + sheet.constitutive_model.Parameters.shear_modulus = mu + sheet.constitutive_model.Parameters.shear_viscosity_0 = De + sheet.thickness = 0.08 + sheet.tolerance = 1.0e-8 + pat_c = -sympy.Rational(3, 2) * (3 * z ** 2 - 1) / 2 + sympy.Rational(9, 4) * (x ** 2 - y ** 2) + pat_s = 6 * x * y + # projection of the stress trace on the cos pattern, area-weighted on a lat-lon grid + lon = np.linspace(-np.pi, np.pi, 73)[:-1]; lat = np.linspace(-np.pi / 2, np.pi / 2, 37)[1:-1] + LON, LAT = np.meshgrid(lon, lat) + g = np.column_stack([(np.cos(LAT) * np.cos(LON)).ravel(), (np.cos(LAT) * np.sin(LON)).ravel(), np.sin(LAT).ravel()]) + wts = np.cos(LAT).ravel() + pc = np.asarray(sympy.lambdify((x, y, z), pat_c, "numpy")(*g.T)).ravel() + rhat = g / np.linalg.norm(g, axis=1)[:, None] + P = np.eye(3)[None] - rhat[:, :, None] * rhat[:, None, :] + dt = 1.0 / steps + t, tc, ts = 0.0, [], [] + for k in range(steps * periods): + t += dt + c, s = float(np.cos(2 * np.pi * t)), float(np.sin(2 * np.pi * t)) + sheet.normal_velocity = A * 2 * np.pi * (-pat_c * s + pat_s * c) + sheet.solve(timestep=dt, zero_init_guess=(k == 0)) + assert sheet._rotated_freeslip_info["converged"] + if t > periods - 1 - 1e-9: + S = np.asarray(uw.function.evaluate(sheet.stress, g * (1 - 0.3 * 0.3 ** 2))).reshape(-1, 3, 3) + tr = np.einsum("nij,nij->n", P, S) + tc.append((wts * tr * pc).sum() / (wts * pc ** 2).sum()); ts.append(t) + ts, tc = np.array(ts), np.array(tc) + M = np.column_stack([np.cos(2 * np.pi * ts), np.sin(2 * np.pi * ts)]) + a, b = np.linalg.lstsq(M, tc, rcond=None)[0] + amp, lag = np.hypot(a, b), -np.degrees(np.arctan2(b, a)) + wt = 2 * np.pi * De + amp_th, lag_th = (24 / 11) * mu * A * wt / np.sqrt(1 + wt ** 2), np.degrees(np.arctan(1 / wt)) + assert abs(amp - amp_th) < 0.05 * amp_th, (amp, amp_th) + assert lag_th - 2 < lag < lag_th + 12, (lag, lag_th) diff --git a/tests/test_1094_maxwell_step_load_history_flavours.py b/tests/test_1094_maxwell_step_load_history_flavours.py new file mode 100644 index 000000000..636df2e79 --- /dev/null +++ b/tests/test_1094_maxwell_step_load_history_flavours.py @@ -0,0 +1,67 @@ +"""A Maxwell block under a step traction: the creep benchmark, per history flavour. + +Plane strain, incompressible, a bar fixed at x = 0 and pulled by a constant +normal traction sigma at x = L with the long sides free: the deviatoric law +gives epsilon_xx = sigma / (4 mu) at once (the elastic jump) and a steady +creep rate d(epsilon_xx)/dt = sigma / (4 eta) thereafter, so the end velocity +settles to sigma L / (4 eta). ETD-1 holds the strain rate constant over a +step and integrates the stress ODE exactly for it, so with no history the +first step's velocity is sigma L / (4 eta (1 - e^-x)), x = dt mu / eta: the +jump spread over the step (0.946 of jump-over-dt plus creep at x = 1/8). +Every flavour of the stress history must reproduce this +(the history is the stress itself; with a velocity this small the transport +between steps is negligible and the flavours must agree with each other). + +Written when the semi-Lagrangian and Eulerian flavours disagreed by a quarter +on the second step of the box fault model (2026-10-17). +""" +import numpy as np +import pytest +import sympy + +import underworld3 as uw + +pytestmark = [pytest.mark.level_2, pytest.mark.tier_b] + +L, SIGMA, MU, ETA, DT = 3.0, 0.01, 1.0, 1.0, 0.125 + + +def _end_velocities(flavour, nsteps=4, tag="a"): + mesh = uw.meshing.UnstructuredSimplexBox(minCoords=(0.0, 0.0), maxCoords=(L, 1.0), + cellSize=0.25, regular=False, qdegree=3) + v = uw.discretisation.MeshVariable(f"v_{tag}", mesh, 2, degree=2) + p = uw.discretisation.MeshVariable(f"p_{tag}", mesh, 1, degree=1) + stokes = uw.systems.Stokes(mesh, velocityField=v, pressureField=p) + stokes.stress_transport = flavour + stokes.constitutive_model = uw.constitutive_models.ViscoElasticPlasticFlowModel( + stokes.Unknowns, order=1, integrator="etd", objective_rate="none") + stokes.constitutive_model.Parameters.shear_modulus = MU + stokes.constitutive_model.Parameters.shear_viscosity_0 = ETA + stokes.bodyforce = sympy.Matrix([0.0, 0.0]) + stokes.tolerance = 1e-8 + stokes.add_dirichlet_bc((0.0, 0.0), "Left") + stokes.add_natural_bc(sympy.Matrix([[-SIGMA, 0.0]]), "Right") # a positive value pushes in + out = [] + pts = np.array([[L - 0.01, 0.5]]) + for k in range(nsteps): + stokes.solve(timestep=DT, zero_init_guess=(k == 0)) + out.append(float(np.asarray(uw.function.evaluate(v.sym, pts)).reshape(-1, 2)[0, 0])) + return np.array(out) + + +@pytest.mark.parametrize("flavour", ["semi_lagrangian", "eulerian", "integration_point", "none"]) +def test_creep_rate_and_elastic_jump(flavour): + u = _end_velocities(flavour, tag=flavour[:4]) + creep = SIGMA * L / (4 * ETA) + x = DT * MU / ETA + first = creep / (1.0 - np.exp(-x)) # ETD-1's exact first step from a zero history + assert abs(u[0] - first) < 0.08 * first, (flavour, u, first) + # later steps: the creep rate, within the mesh's few per cent + assert np.all(np.abs(u[1:] - creep) < 0.08 * creep), (flavour, u, creep) + + +def test_flavours_agree(): + us = {f: _end_velocities(f, nsteps=3, tag="g" + f[:3]) for f in ("semi_lagrangian", "eulerian", "none")} + a = us["semi_lagrangian"] + for f in ("eulerian", "none"): + assert np.allclose(a, us[f], rtol=0.02), us diff --git a/tests/test_1096_addon_cut_child_tail.py b/tests/test_1096_addon_cut_child_tail.py new file mode 100644 index 000000000..d4c8e424d --- /dev/null +++ b/tests/test_1096_addon_cut_child_tail.py @@ -0,0 +1,58 @@ +"""The 2-D ``add_fault`` child of a refined base carries the base's coarse levels. + +A mesh built with ``refinement >= 1`` is its own geometric-multigrid hierarchy: +its coarse levels are the tail, the finest level is the mesh itself. A cut +re-represents that finest grid with the fault conformed, so the cut-and-split +child takes the coarse levels as its tail with itself as the finest level — +the rule ``split_fault`` and ``Mesh._adopt_cut_child`` already apply. The 2-D +cut path constructed its child bare, so every solver on an ``add_fault`` child +of a refined box fell back to algebraic multigrid silently (measured +2026-10-17: refinement=2, tail 0, velocity KSP at its 200-iteration cap). + +The contract asserted: the tail has exactly one level per base coarsening and +the velocity preconditioner of a fault solve on the child is the custom FMG. +""" +import numpy as np +import pytest + +import underworld3 as uw + +pytestmark = [pytest.mark.level_2, pytest.mark.tier_b] + +TRACE = np.array([[0.30, 0.45], [0.70, 0.55]]) + + +def _refined_box(refinement): + return uw.meshing.UnstructuredSimplexBox( + minCoords=(0.0, 0.0), maxCoords=(1.0, 1.0), cellSize=0.1, + refinement=refinement, qdegree=2) + + +@pytest.mark.parametrize("refinement", [1, 2]) +def test_cut_child_takes_the_base_coarse_levels(refinement): + base = _refined_box(refinement) + child = base.add_fault([("Flt", TRACE)]) + tail = child._custom_mg_coarse_meshes + # the levels are the base's own static wraps, coarsest first, the cut + # mesh re-representing the finest one + wraps = base._coarse_level_meshes() + assert tail is not None and len(tail) == refinement, tail + assert all(t is w for t, w in zip(tail, wraps[:-1])) + assert all(getattr(m, "_mg_static", False) for m in tail) + + +def test_fault_solve_on_the_cut_child_runs_on_the_geometric_tail(): + base = _refined_box(1) + child = base.add_fault([("Flt", TRACE)]) + x, y = child.X + stokes = uw.systems.Stokes(child) + stokes.constitutive_model = uw.constitutive_models.ViscousFlowModel + stokes.constitutive_model.Parameters.shear_viscosity_0 = 1 + stokes.tolerance = 1.0e-6 + stokes.petsc_use_pressure_nullspace = True + for side in ("Top", "Bottom", "Left", "Right"): + stokes.add_dirichlet_bc((y - 0.5, 0.0), side) + stokes.add_fault_bc(0, "Flt") + stokes.solve() + info = stokes._rotated_freeslip_info + assert info["velocity_pc"] == "custom-FMG" and info["converged"], info diff --git a/tests/test_1098_plex_topology_tables.py b/tests/test_1098_plex_topology_tables.py new file mode 100644 index 000000000..bc72d5184 --- /dev/null +++ b/tests/test_1098_plex_topology_tables.py @@ -0,0 +1,84 @@ +"""The compiled DMPlex topology tables answer exactly what DMPlex answers. + +``utilities/_plex_topology`` builds, in one C loop per stratum, the tables the +fault, placement and add-on code used to assemble one petsc4py call at a time: +each point's closure in one depth, each point's support, a per-cell section and +its offsets, each cell's dof rows. The baseline is DMPlex itself, queried point +by point: the tables must agree entry for entry, in closure order, on 2-D and +3-D simplex meshes, on whatever partition the run has (the module is rank-local; +run under ``mpirun`` it is checked on every rank's chart). +""" +import numpy as np +import pytest +from petsc4py import PETSc + +import underworld3 as uw +from underworld3.utilities import _plex_topology as pt + +pytestmark = [pytest.mark.level_1, pytest.mark.tier_b] # proven, not yet hardened + + +@pytest.fixture(scope="module", params=[2, 3], ids=["2d", "3d"]) +def mesh(request): + if request.param == 2: + return uw.meshing.UnstructuredSimplexBox( + minCoords=(0.0, 0.0), maxCoords=(1.0, 1.0), cellSize=0.1, qdegree=2) + return uw.meshing.UnstructuredSimplexBox( + minCoords=(0.0, 0.0, 0.0), maxCoords=(1.0, 1.0, 1.0), cellSize=0.3, + regular=False, qdegree=2) + + +def _closure_in_depth(dm, p, depth): + dS, dE = dm.getDepthStratum(depth) + return [int(q) for q in dm.getTransitiveClosure(p)[0] if dS <= q < dE] + + +def test_closure_points_match_the_plex(mesh): + dm = mesh.dm + cS, cE = dm.getHeightStratum(0) + fS, fE = dm.getHeightStratum(1) + for points in (np.arange(cS, cE), np.arange(fS, fE)): + for depth in range(dm.getDimension()): + table = pt.closure_points(dm, points, depth) + for row, p in zip(table, points): + ref = _closure_in_depth(dm, int(p), depth) + assert row[row >= 0].tolist() == ref + assert (row[len(ref):] == -1).all() + + +def test_supports_match_the_plex(mesh): + dm = mesh.dm + for stratum in (dm.getHeightStratum(1), dm.getDepthStratum(0)): + points = np.arange(*stratum) + table = pt.supports(dm, points) + for row, p in zip(table, points): + assert row[row >= 0].tolist() == [int(q) for q in dm.getSupport(int(p))] + + +def test_cell_section_and_offsets_match_the_plex(mesh): + dm = mesh.dm + cS, cE = dm.getHeightStratum(0) + cells = np.arange(cS, cE) + section = PETSc.Section().create(comm=dm.getComm()) + pt.fill_section(section, dm, cells, 3) + assert section.getChart() == dm.getChart() + assert section.getStorageSize() == 3 * len(cells) + offsets = pt.section_offsets(section, cells) + assert offsets.tolist() == [section.getOffset(int(c)) for c in cells] + assert sorted(offsets.tolist()) == list(range(0, 3 * len(cells), 3)) + + +def test_closure_dof_rows_match_the_plex(mesh): + dm = mesh.dm.getCoordinateDM() + section = mesh.dm.getCoordinateSection() + cdim = mesh.dm.getCoordinateDim() + cS, cE = mesh.dm.getHeightStratum(0) + table = pt.closure_dof_rows(dm, section, np.arange(cS, cE), cdim) + for row, c in zip(table, range(cS, cE)): + ref = [] + for q in dm.getTransitiveClosure(c)[0]: + n = section.getDof(int(q)) // cdim + if n: + o = section.getOffset(int(q)) // cdim + ref.extend(range(o, o + n)) + assert row[row >= 0].tolist() == ref diff --git a/tests/test_1100_element_table_matches_closures.py b/tests/test_1100_element_table_matches_closures.py new file mode 100644 index 000000000..d3d287360 --- /dev/null +++ b/tests/test_1100_element_table_matches_closures.py @@ -0,0 +1,62 @@ +"""The mesh's element table describes the cells the mesh actually has. + +``mesh.element`` says how many faces a cell has (``entities[1]``), how many +vertices (``entities[dim]``) and how many vertices a face has +(``face_entities[dim]``); the face-control-point builders, the geometry audit +and the visualisation slice cell closures by these counts. The table had no +1-D entry, so a 1-D surface mesh (an annulus boundary, ``extract_surface``) +fell through to the tetrahedron entry and read 4 faces and 4 vertices per line +cell; slicing the last 4 points of a line's 3-point closure took the line +itself as a vertex. + +The baseline is DMPlex: every cell's cone size, the vertices in its closure, +and the vertices of each face's closure, on a 1-D surface, 2-D triangles and +quadrilaterals, 3-D tetrahedra and a 2-D surface in 3-D. +""" +import numpy as np +import pytest + +import underworld3 as uw +from underworld3.utilities._plex_topology import closure_points + +pytestmark = [pytest.mark.level_1, pytest.mark.tier_b] # proven, not yet hardened + + +def _annulus_surface(): + annulus = uw.meshing.Annulus(radiusOuter=1.0, radiusInner=0.5, cellSize=0.2) + return annulus.extract_surface("Upper") + + +def _shell_surface(): + shell = uw.meshing.SphericalShell(radiusOuter=1.0, radiusInner=0.5, + cellSize=0.3) + return shell.extract_surface("Upper") + + +MESHES = { + "line (annulus surface)": _annulus_surface, + "triangle": lambda: uw.meshing.UnstructuredSimplexBox( + minCoords=(0.0, 0.0), maxCoords=(1.0, 1.0), cellSize=0.2), + "quadrilateral": lambda: uw.meshing.StructuredQuadBox(elementRes=(4, 3)), + "tetrahedron": lambda: uw.meshing.UnstructuredSimplexBox( + minCoords=(0.0, 0.0, 0.0), maxCoords=(1.0, 1.0, 1.0), cellSize=0.4), + "triangle in 3-D (shell surface)": _shell_surface, +} + + +@pytest.mark.parametrize("name", list(MESHES)) +def test_element_table_matches_the_cells(name): + mesh = MESHES[name]() + dm, el = mesh.dm, mesh.element + cS, cE = dm.getHeightStratum(0) + fS, fE = dm.getHeightStratum(1) + assert cE > cS + cells = np.arange(cS, cE) + cone_sizes = {dm.getConeSize(int(c)) for c in cells} + assert cone_sizes == {el.entities[1]}, (name, cone_sizes, el) + verts = closure_points(dm, cells, 0) + assert verts.shape[1] == el.entities[mesh.dim] and (verts >= 0).all(), \ + (name, verts.shape, el) + face_verts = closure_points(dm, np.arange(fS, fE), 0) + assert face_verts.shape[1] == el.face_entities[mesh.dim] \ + and (face_verts >= 0).all(), (name, face_verts.shape, el)