Skip to content

Can the IB neighbourhood hand-off append the same patch twice when a rank fills several neighbour slots? #2044

Open

Description

@sbryngelson

This is a code-reading observation, not a reproducer. I could not build a test case on master that reaches the path (see the last section), so please treat the question as open: does the neighbourhood hand-off still drop a patch that arrives more than once in a single unpack loop?

The path. In s_handoff_ib_ownership (src/simulation/m_ibm.fpp, master 62f658ad):

num_ibs_pre = num_ibs                                                   ! 1943
...
    call s_get_neighborhood_idx(tmp_patch%gbl_patch_id, j, num_ibs_pre) ! 1951
    if (j < 0) then
        num_ibs = num_ibs + 1
        patch_ib(num_ibs) = tmp_patch                                   ! 1955
    else
        patch_ib(j)%owner_rank = tmp_patch%owner_rank
    end if
...
call s_merge_ib_lookup(num_ibs_pre)                                     ! 1965

Three properties together are what prompt the question:

  1. the lookup search is bounded to the pre-loop count — s_get_neighborhood_idx sets hi = num_entries (m_ib_patches.fpp:753), and the caller passes num_ibs_pre, so entries appended during this loop are not searchable;
  2. appending at 1955 does not register the patch in the lookup; registration happens only afterwards, at 1965;
  3. s_merge_ib_lookup (m_ib_patches.fpp:812) is a straight sorted merge of the new arrivals into the existing keys, with no duplicate check.

So if the same gbl_patch_id is unpacked twice within one call — which happens when a rank fills more than one neighbour slot of the sender, e.g. two ranks across a periodic direction, where each rank is both the +1 and the -1 neighbour of the other — it looks absent both times and is appended twice. The lookup then holds two entries with the same key, and the binary search returns whichever it lands on.

Why I think it is worth a look. The equivalent hole in the previous (direct-indexed ib_gbl_idx_lookup) design produced a visible failure for us: moving IBs lost all force after crossing a rank seam. That was #1973, which registered the patch at append time so copies after the first were dropped. #1973 was closed rather than merged, so whatever its merits, the protection is not in master, and #1916 has since rewritten this region.

What I could not do. I could not produce a reproducer on master:

  • our own periodic moving-sphere array case sets patch_ib%thermal_bc and patch_ib%surface_reaction, which master's namelist does not have;
  • the obvious substitute — a triply periodic box of moving spheres at 1 vs 2 ranks — stops at step 8 with Prohibited condition: bounds_error / Ghost Point and Image Point on Different Processors, before the hand-off question arises. That check appears to fire on the periodic wrap even on a single rank, which may itself be worth a separate look.

So I have not measured anything here, and I am not claiming a defect. If the design already prevents a patch being unpacked twice in one call (for instance if the send side guarantees one copy per receiving rank), then this is a non-issue and I would be glad to know why.

Prepared with Claude Code.

Activity

  1. sbryngelson commented on Oct 13, 2026

    @sbryngelson
    MemberAuthor

    One clarification on the history above, so nothing is read into it that is not there.

    I wrote that #1973 was "closed rather than merged, so whatever its merits, the protection is not in master". That could be taken to mean the PR was reviewed and declined. It was not. It received no review comments — the only comment on it is the automated line-count bot — and it was closed from my own side on 2026-10-10, as part of a decision to stop carrying a long tail of open PRs and keep that work on a downstream branch instead. Nobody upstream weighed in either way.

    So the fix was withdrawn by me, not turned down, and the symptom it addressed (moving IBs losing all force after crossing a rank seam) was real and observed at the time, in the previous direct-indexed lookup design. #1916 then rewrote the region. That is the whole of the history; the question in this issue stands or falls on the current code, not on #1973.

    Prepared with Claude Code.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions