Repository navigation
Checkpoint round-trip corrupts fault-adjacent DOFs on split meshes #640
Description
On a split (add_fault) mesh, write_timestep -> fresh Mesh(h5) -> read_timestep corrupts velocity DOFs at the cut: the coincident duplicated fault nodes are mixed on read-back, so part of the slip discontinuity is smeared into the first element ring.
Evidence (2-D S-fault rig, 4-strand network, rotated free-slip contact, coarse): raw in-cell fault-frame stress probed at ±0.006 from the main trace, same solve, A/B:
| probe | |tau_nt| mean | |tau_nn| mean |
|---|---|---|
| A — in-memory after solve | 0.12 | 0.86 |
| B — after checkpoint round-trip | 27.1 | 14.1 |
A is correct physics (shear traction released on the frictionless contact; normal deviatoric stress below the far-field value of 2). B is ~200x garbage at the slip/h scale (jump 0.4 over one 0.01 element), and decays over ~2 rings. Velocity probed at ±0.006 also shows a ~13% jump deficit after reload — consistent with cut-node values partially swapped between sides, most likely a coordinate-matched read-back that is ambiguous at coincident duplicates.
Consequences: any post-hoc stress/strain-rate recovery from split-mesh checkpoints is unusable near faults (we caught it rendering a stress field that showed a bright fault line the in-memory solve does not have). Far-field quantities and on-trace slip profiles sampled at band-scale offsets (±0.03) are only mildly biased.
Reproduction: ~/+Simulations/s_fault_rig/diagnostics/rt_probe.py (self-contained A/B; solve converges, prints the table above).
Likely fix territory: read_timestep DOF matching for meshes with duplicated (coincident) vertices — match by plex index/section, not by coordinate. Should land after #638 merges to avoid churning that PR.
Underworld development team with AI support from Claude Code
Activity
Do not read a pasted sample as a closing keyword
Merge pull request #714 from underworldcode/bugfix/checkpoint-cut-dofs
lmoresi commentedon Oct 6, 2026
Fixed by #714, merged to
development(7c9cbbfd). Split-mesh checkpoints are read through the PETSc section rather than the coordinate KD-tree, so fault-adjacent DOFs survive the round trip. Covered bytests/test_0864_split_checkpoint_roundtrip.pyandtests/parallel/ptest_0864_split_checkpoint_parallel.py.lmoresi commentedon Oct 12, 2026
Reopened and labelled
fixed-in-development, under the policy now in CLAUDE.md:The fix stands and the evidence above is unchanged — this is a bookkeeping correction, not a reversal. Closing it on merge was wrong because
developmenthas not been released, so anyone running the release still has the defect; the label says where the fix is and the release sweep closes it.Read split-mesh checkpoints through the section, not the coordinates (u…
lmoresi commentedon Oct 13, 2026
This regressed on
developmentand the label was wrong. PR #847 restores it.Probed rather than taken on the merge reference, which is what found it:
tests/test_0864_split_checkpoint_roundtrip.py42f93470— the merge before #756f414da63— after #756#756 moved the reload payload from the file root into
/restart/petsc. The writer still emits the per-field section and its global vector — both are on disk, one group deeper — but the reader's existence check stayed at the root, so a file written withpetsc_reload=Truewas reported as having no native payload. The #640 guard then refused the good path along with the bad, advising a re-write with the flag the caller had already used. Split-mesh checkpoints could not be read back at all.So the fix from #714 was still present but disarmed. The
fixed-in-developmentlabel was accurate when applied and stopped being true when #756 merged, which no probe would have caught except one run againstdevelopmentafter the fact.Staying open and
fixed-in-developmentis right again once #847 merges. It should not be closed by the release sweep until #847 is in, or the release closes a defect that reproduces.Underworld development team with AI support from Claude Code
Merge pull request #847 from underworldcode/bugfix/split-checkpoint-p…
Find the split-checkpoint payload wherever the writer put it (underwo…