Skip to content

Find the split-checkpoint payload wherever the writer put it (#640 regressed by #756) - #847

Merged
lmoresi merged 1 commit into
developmentfrom
bugfix/split-checkpoint-payload-640
Oct 13, 2026
Merged

lmoresi merged 1 commit into
developmentfrom
bugfix/split-checkpoint-payload-640

Conversation

@lmoresi

@lmoresi lmoresi commented Oct 13, 2026

Copy link
Copy Markdown
Member

development cannot read back a split-mesh checkpoint. #640's fix is in the tree but disarmed, and #640 is labelled fixed-in-development, so #805 would have closed it.

The regression

commit tests/test_0864_split_checkpoint_roundtrip.py
42f93470 — the merge before #756 5 passed
f414da63 — after #756 1 failed

Both built from clean and run locally. The test is level_1/tier_a and reachable from scripts/test.sh via tests/test_08*py, so CI covers it.

#756 (e4d79d71, "Share compact DG1 geometry across XDMF fields") moved the reload payload from the file root into /restart/petsc:

before   topologies/uw_mesh/dms/vRT/section
         topologies/uw_mesh/dms/vRT/vecs/vRT

after    restart/petsc/topologies/uw_mesh/dms/vRT/section
         restart/petsc/topologies/uw_mesh/dms/vRT/vecs/vRT

Nothing was dropped — I checked the written file, and the per-field section and its global vector are both still there, one group deeper. Only the reader's existence check stayed at the root, so for a file written with petsc_reload=True it answered "no PETSc-native payload". The #640 guard then refused the good path along with the bad:

RuntimeError: read_timestep: the saved field 'vRT' holds 15 coordinates
carrying more than one DOF ... The file carries no PETSc-native payload:
re-write it with Mesh.write_timestep(..., petsc_reload=True).

— advising the flag the caller had already passed.

The fix

Both changes are reader-side, in what the guard is given:

  • _petsc_payload_name searches both layouts. It is only an existence check, and a file written by either version has to stay readable.
  • _guard_coincident_dofs no longer assumes a v1.1 file has no payload. It looked the payload up for legacy files only, while read_timestep calls _read_native_payload on a v1.1 file immediately afterwards — so the guard refused the files the native path was about to read correctly. The saved-vs-live row comparison stays legacy-only: v1.1 shares one compact geometry across fields, so there is no per-field saved cloud and saved_rows is 0 by construction.

The test that was missing

test_0864 caught the symptom; nothing pinned the cause, so a relocation presented as a confusing refusal instead of a located failure. test_the_payload_is_found_wherever_the_writer_put_it asserts the section and the vector are on disk and that the existence check finds them.

Tier B because it is proven, not assumed: with the single-location lookup restored it fails, and so does the round-trip test; with the fix both pass.

Verification

test_0864_split_checkpoint_roundtrip 6 passed
+ test_0005, test_0007, test_0008, test_0012, test_0020 (checkpoint and snapshot) 50 passed
check_test_coverage.py all 352 files reachable

How this got in

#756's own CI was green, and its run was 51m46s against roughly 1h30m for its neighbours — worth a look at whether its test job was collecting the full set. The general lesson is the #828 one again: the question is not "do the new tests pass" but does CI exercise the configuration this change affects — here, a file-layout change with the reader's expectations written down in a different file.

Issue state: #640 stays open and fixed-in-development is accurate again once this merges. It should not be closed by the release sweep until then.

Underworld development team with AI support from Claude Code

🤖 Generated with Claude Code

https://claude.ai/code/session_01E87Q7KrpapxeQiLD1RiNXv

#756 moved the reload payload from the file root into /restart/petsc.
The writer still emits the per-field section AND its global vector, one
group deeper -- both are on disk -- but the reader's existence check
looked only at the root, so it reported "no PETSc-native payload" for a
file written WITH petsc_reload=True. The #640 guard then refused the
good path along with the bad, telling the caller to re-write the file
with the flag they had already used, and split-mesh checkpoints could
not be read back at all.

Measured: tests/test_0864_split_checkpoint_roundtrip.py is 5 passed at
42f9347 (the merge before #756) and 1 failed at f414da6 (after it).
It is tier_a and reachable from scripts/test.sh, so CI covers it.

Two reader-side changes, both in the guard's inputs:

- _petsc_payload_name searches "topologies" and
  "restart/petsc/topologies". It is only an existence check, and a file
  written by either version has to stay readable.
- _guard_coincident_dofs no longer assumes a v1.1 file has no payload.
  It looked the payload up only for legacy files while read_timestep
  calls _read_native_payload on a v1.1 file immediately afterwards, so
  the guard refused the very files the native path was about to read
  correctly. The saved-vs-live row comparison stays legacy-only: v1.1
  shares one compact geometry across fields (#756), so there is no
  per-field saved cloud and saved_rows is 0 by construction.

New test, tier B because it is proven: with the single-location lookup
restored, test_the_payload_is_found_wherever_the_writer_put_it fails and
so does the round-trip test; with the fix both pass. It asserts the
section and the vector are on disk and that the existence check finds
them, so a future relocation fails with the reason rather than as a
confusing refusal.

test_0864 6 passed. With test_0005, test_0007, test_0008, test_0012 and
test_0020 (the checkpoint and snapshot suite): 50 passed.
check_test_coverage.py: all 352 files reachable.

Underworld development team with AI support from Claude Code

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01E87Q7KrpapxeQiLD1RiNXv
Copilot AI balanced review requested due to automatic review settings October 13, 2026 05:44

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@lmoresi
lmoresi merged commit 909a98e into development Oct 13, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants