Skip to content

[MRG] Fix empty and degenerate EMD inputs without native crashes - #904

Open
xihaian251 wants to merge 4 commits into
PythonOT:masterfrom
xihaian251:fix/emd-empty-input-crash
Open

xihaian251 wants to merge 4 commits into
PythonOT:masterfrom
xihaian251:fix/emd-empty-input-crash

Conversation

@xihaian251

Copy link
Copy Markdown
Contributor

Summary

ot.emd / ot.emd2 kill the interpreter on an empty cost matrix (issue #901):
NetworkSimplexSimple::setDenseCostMatrix reads D[0] and then loops with
for (ArcsType i = 1; i != _arc_num; ++i). With _arc_num == 0,
D[0] is already out of bounds and the loop continues scanning beyond the
buffer (until a fault or unsigned wraparound), causing SIGBUS/SIGSEGV instead
of a Python exception.

The same degenerate supports expose three further native hazards: &weights1[0]
on an empty std::vector, an uninitialised in_arc in the block-search pivot
rule, and unchecked marginal/potential/edge buffers in the Cython entry points
(emd_c, emd_c_sparse, emd_c_lazy). A one-sided empty support also makes
ot.emd_1d segfault.

This PR makes the affected exact-EMD entry points handle these inputs safely,
at the layer where the necessary information is available. In the tested
non-degenerate cases, the numerical results are unchanged.

Fixes #901.

Root cause

Public API → Cython → C++:

  1. ot.emd/ot.emd2 default an empty marginal to nx.ones(n) / n, which is
    0/0 for n == 0, then rescale b with b * a.sum() / b.sum() (0/0).
    With a non-empty cost matrix and zero total mass on both sides, the active
    support filter hands a (0, 0) sub-matrix to the solver.
  2. emd_c passes the raw pointers to EMD_wrap. With no positive weight, n = m = 0, full_support is true, and the dense-cost fast path calls
    setDenseCostMatrix(D, 0): _max_cost = D[0] is an out-of-bounds read and the
    i != _arc_num loop scans beyond the buffer. This is the reported crash.
  3. EMD_wrap also evaluates &weights1[0] / &weights2[0] on empty vectors
    before any solver call (undefined behaviour), and ot.emd on a sparse matrix
    with no stored entry reaches findEnteringArc() with zero arcs, which
    dereferences the never-initialised NetworkSimplexSimple::in_arc
    (init() does not set it; only warmstartInit() does) and segfaults.
  4. The Cython entry points blindly trust that marginals, warm-start potentials,
    coordinates and edge endpoints match the native buffers, so a direct caller
    can turn a shape mismatch into an out-of-bounds read.

Changes

ot/lp/_network_simplex.py (public API policy)

  • A cost matrix with an empty dimension ((0, 0), (0, n), (n, 0), dense or
    sparse) raises ValueError before any conversion: an empty marginal cannot be
    defaulted to a uniform distribution ([] is the documented sentinel for
    "uniform").
  • b is rescaled per histogram with a safe denominator
    (b * mass_a / np.where(mass_b != 0, mass_b, 1.0)). With nonnegative
    marginals, a positive-mass source and a zero-mass target remain infeasible;
    a zero-mass source instead rescales a nonzero target to zero when
    check_marginals=False. Each histogram in a batch is handled independently.
    The formula is unchanged when mass_b != 0, including negative totals.
  • A non-finite total mass raises ValueError instead of silently being treated
    as an empty support.
  • center_ot_dual no longer divides by zero when both marginals are empty, so
    the returned potentials stay finite.

ot/lp/emd_wrap.pyx (Cython boundary)

  • emd_c, emd_c_sparse, emd_c_lazy: reject marginals, initial potentials and
    edge arrays whose lengths do not match the problem; emd_c_sparse also rejects
    endpoints outside the marginals and non-equal edge array lengths. The
    pre-existing "empty marginal means uniform" fallback of emd_c/emd_c_lazy is
    preserved.
  • emd_1d_sorted: an empty support raises ValueError instead of reading
    u_weights[0] and writing out of bounds (reachable from ot.emd_1d).

ot/lp/EMD_wrapper.cpp (native entry points)

  • EMD_wrap / EMD_wrap_sparse: return before building any graph when one side
    has no positive weight. Both sides empty is the trivial problem (OPTIMAL,
    cost 0, empty plan); one side empty is INFEASIBLE. This also removes the
    &weights[0] undefined behaviour.
  • EMD_wrap_sparse: reject endpoints outside the marginals before indexing the
    reverse maps.

ot/lp/network_simplex_simple.h (solver)

  • setDenseCostMatrix: no read of D[0] and no scan when _arc_num == 0, and
    the scan condition is i < _arc_num.
  • init() initialises in_arc deterministically; the pivot rule checks for
    absent eligible arcs before using it.
  • BlockSearchPivotRule::findEnteringArc returns false as soon as no arc has a
    negative reduced cost, instead of scaling its tolerance with the unset
    _in_arc. The old code was equivalent whenever it did not read uninitialised
    memory; this is a strict improvement and saves the useless scan.

ot/lp/solver_1d.py (public 1-D API)

  • ot.emd_1d rejects an empty support before defaulting the weights to uniform,
    instead of dividing by zero and crashing in the native 1-D sweep.

test/test_ot.py — 11 regression tests (empty problems, zero total mass on
dense and sparse inputs, one-sided zero mass, batches with an empty histogram,
ot.emd_1d, and the three native entry points).

RELEASES.md — one entry under 0.9.8dev / Closed issues.

Tests

Environment: Windows 11, Python 3.13.1, NumPy 2.5.4, SciPy 1.18.1, Cython 3.3.0,
MSVC 19.44; worktree at 0b6f8d5, ot.lp.emd_wrap rebuilt in place from the
worktree sources with the extension spec of setup.py (the binary postdates
every native source; MSVC /GL builds are not bit-reproducible, so no artifact
hash is quoted). Linux side: WSL Ubuntu 24.04, gcc 13.3, AddressSanitizer.

command result
pytest test/test_ot.py -o addopts="" -q 49 passed, 9 skipped (with the JAX backend installed)
pytest test/ -o addopts="" -q 2294 passed, 551 skipped, 4 xfailed, 7 failed
full 39-case degenerate-input matrix, each case in its own subprocess 0 crashes (was 7 crashes + 4 exceptions)
LP oracle (scipy.optimize.linprog/HiGHS): 102 comparisons out of 110 sampled problems (8 sampled sparse masks forbade all transport, the oracle returned infeasible and the trial was skipped); dense, sparse mask, rescaled marginals, zero mass, uniform contract all pass
AddressSanitizer build (Linux, gcc 13.3) on the 38 matrix cases existing at build time (the 39th case added later exercises Python code only) 0 reports
same ASan build of the unpatched tree, ot.emd2([], [], np.zeros((0, 0))) heap-buffer-overflow … network_simplex_simple.h:932 in setDenseCostMatrix
performance, 400×400 dense / 1600-edge sparse small-sample medians differed by roughly 10–18%; repeated-run noise prevents a firm regression estimate

The numbers above come from the final run with the JAX backend installed; the same suite run before JAX was installed reported 1527 passed, 139 skipped, 4 xfailed and 6 failed, i.e. the same only-BSP failures.

The 7 reported failures are test_bspot.py (5) and
test_solvers.py::test_solve_sample_bsp (NumPy and JAX). The local environment
could not fetch the deps/eigen submodule (gitlab.com returned 5xx), so the
BSP extension was not built. These tests fail with NotImplementedError from
a local placeholder and do not enter the modified EMD paths. The full suite
therefore did not pass on this machine; the affected EMD regression tests
did pass. The placeholder is not part of the proposed patch.

Each new test was run against a second worktree built from the same base commit
without the patch: test_emd_empty_problem_raises,
test_emd_sparse_no_transport_edges, test_emd_c_empty_support and
test_emd_1d_empty_support segfault the interpreter; test_emd_zero_mass,
test_emd_zero_mass_sparse, test_emd_zero_mass_one_sided,
test_emd_c_shape_mismatch, test_emd_c_sparse_invalid_edges and
test_emd_c_lazy_shape_mismatch fail on assertions. test_emd_empty (the
existing uniform-marginal contract) and
test_emd2_batch_with_empty_histogram (a regression guard for behaviour that was
already correct) pass on both.

Compatibility

  • ot.emd([], [], M) on a non-empty M keeps its "uniform marginals" meaning;
    test_emd_empty is unchanged and passes.
  • All tested non-degenerate cases were bit-identical to an unpatched
    build. The rescaling expression is unchanged for mass_b != 0, and
    center_ot_dual is unchanged when the total mass is positive. Cases checked
    include the uniform contract, partial zeros, non-unit mass, integer weights,
    sparse transport, and warm starts.
  • Dimension and mass mismatch still raise the same AssertionErrors
    (test_emd_dimension_and_mass_mismatch unchanged).
  • Behaviour that changes, all previously undefined or NaNs:
    • empty cost matrix: crash → ValueError;
    • zero total mass on a non-empty matrix: zero plan with NaN potentials and a
      spurious "infeasible" warning → zero plan, cost 0, finite potentials,
      no warning. Dense-path dual feasibility was checked; sparse-path potentials
      for zero-weight nodes are not necessarily dual feasible with negative costs.
      The zero plan is the unique feasible plan and is optimal;
    • one-sided zero mass with check_marginals=False: if the source has
      positive mass but the target total is zero, the solver reports
      INFEASIBLE with a zero plan and finite potentials; if the source has
      zero mass but the target total is positive, the existing target-rescaling
      convention produces zero target mass and the optimal zero plan;
    • non-finite marginals with check_marginals=False: undefined results →
      ValueError;
    • ot.emd_1d with an empty support: ValueError("negative dimensions are not allowed") or a segfault → ValueError naming the empty support;
    • direct calls to emd_c/emd_c_sparse/emd_c_lazy with mismatched buffers:
      out-of-bounds reads → ValueError.
  • Invalid marginals that are not part of this issue keep their previous
    behaviour: negative entries still reach the solver (which reports
    INFEASIBLE when it sees them), and a negative total target mass is rescaled
    exactly as before (np.where(mass_b != 0, mass_b, 1) only replaces the
    division by zero, so the result is unchanged whenever mass_b != 0).
  • Sparse duals for zero-weight nodes remain 0 (they are never repaired by the
    sparse path, as documented in emd_c); with negative costs they are therefore
    not necessarily dual feasible. This is pre-existing and unchanged; only dense
    inputs get repaired potentials.

Not in this PR

  • ot.emd2_lazy and ot.lp.emd_1d's handling of zero weights (not empty
    supports): emd2_lazy still computes b * a.sum() / b.sum() and returns NaN
    potentials when both masses are zero. It does not crash (its native entry point
    has an early return) and is left for a separate change to keep this PR focused.
  • The _grid EMD path of [WIP] Add exact EMD-L1 solver for Cartesian grid histograms #863 is a different code path and is not touched.

AI disclosure

This change was prepared with an AI coding assistant (Qoder) used for source
reading, patch drafting, test generation, and running the build/test/sanitizer
matrix. Three independent AI review passes were run against the draft: a
maintainer-style review (API, compatibility, tests), a native memory-safety
review, and a re-review of the revised patch. The first two found real defects
that were fixed before this version: the uninitialised in_arc was still
dereferenced on the pivot rule's block-boundary path, the batched emd2
aborted a whole batch on one empty histogram, and ot.emd_1d/emd_c_lazy had
their own crashes. The re-review approved the revision and confirmed that
the tested non-degenerate results were bit-identical to master. No human code review was
performed. All commands and results in the Tests section were executed on the
author's machine and are reproducible from the branch.

- ot.emd/ot.emd2: reject a cost matrix with an empty dimension with a
  ValueError before any conversion, and reject non-finite marginal masses
- rescale b per histogram with a safe denominator: when both marginals have a
  zero total mass, return the exact zero plan with finite potentials instead
  of NaNs, while a batch keeps solving its non-degenerate histograms
- ot.emd_1d: reject an empty support instead of crashing in the 1-D sweep
- emd_c/emd_c_sparse/emd_c_lazy: validate marginals, initial potentials,
  edge array lengths and edge endpoints against the native buffers
- EMD_wrap/EMD_wrap_sparse: return before building a graph when one side has
  no positive weight (OPTIMAL for an empty problem, INFEASIBLE otherwise),
  which also removes the &weights[0] access on empty vectors
- NetworkSimplexSimple: do not read D[0] in setDenseCostMatrix when there is
  no arc, initialise in_arc, and return from findEnteringArc as soon as no
  arc has a negative reduced cost
- add 11 regression tests covering the empty, zero-mass, degenerate-batch,
  1-D and direct native entry points

Closes PythonOT#901
@xihaian251

Copy link
Copy Markdown
Contributor Author

Hi,

The only failing required check appears to be ci/circleci: build_docs, and the failure looks unrelated to this patch.

In [CircleCI job #4273](https://circleci.com/gh/PythonOT/POT/4273), the Get Python running step timed out after 964 seconds while downloading the 539 MB PyTorch wheel. The subsequent make html step never started, so Sphinx was not executed.

All 16 GitHub Actions jobs passed, including doctests and the source build. I also compared a minimal Sphinx build of the modified docstrings against the base commit; both produced the same warning, with no new warnings introduced by this PR.

I don't appear to have permission to rerun the CircleCI workflow. Could a maintainer with access please rerun it from the failed job?

If the rerun reveals an actual documentation issue, I'll address it.

Thanks!

@xihaian251

Copy link
Copy Markdown
Contributor Author

ci/circleci: build_docs (job 4273) hasn't been rerun since 2026-10-29 — its commit status still shows created_at == updated_at == 2026-10-29T03:04:12Z, and the paired build_docs artifact status reads "No artifacts found".

The failure is in the "Get Python running" step (964 s timeout fetching the 539 MB PyTorch wheel), so make html never started and Sphinx was never reached on CircleCI. Docs for this exact commit do build: Actions run 2310 ("Build doc") is green on b1d459e with Get Python running, Check installation, Generate HTML docs and upload-artifact all passing. All 16 GitHub Actions checks are green, including doctests and build_from_source.

For what it's worth, #911's build_docs passed on CircleCI today, so the job itself is healthy and this looks like a one-off download timeout on this run rather than anything about the patch.

Could someone with CircleCI access rerun job 4273, or is the passing "Build doc" run enough to consider build_docs satisfied? The branch is one commit behind master with no conflicts — happy to update it if that helps.

@codecov

codecov Bot commented Nov 5, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.11%. Comparing base (700fc19) to head (81f07ce).

Additional details and impacted files
@@            Coverage Diff             @@
##           master     #904      +/-   ##
==========================================
+ Coverage   97.09%   97.11%   +0.01%     
==========================================
  Files         130      130              
  Lines       27229    27340     +111     
==========================================
+ Hits        26437    26550     +113     
+ Misses        792      790       -2     
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@xihaian251

Copy link
Copy Markdown
Contributor Author

Hi all,

The earlier CircleCI build_docs timeout is now resolved. All checks on the current PR head are green (GitHub Actions: 18 passed, 1 intentionally skipped; CircleCI documentation build and artifact: passed). GitHub also reports no merge conflicts.

This PR addresses the interpreter crash in #901: setDenseCostMatrix previously accessed D[0] when there were no transport arcs, followed by a non-terminating scan. The patch adds native zero-arc guards and validates the relevant Cython boundary buffers.

Public emd/emd2 calls now reject empty-dimensional cost matrices, while problems with nonempty supports and zero mass on both sides return the zero plan with finite dual potentials. The regression tests pass in upstream CI, including tests against the compiled native extension.

Would a maintainer be able to review this when convenient? I'm happy to address any feedback.

Thanks!

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ot.emd / ot.emd2 crash the interpreter on empty inputs (out-of-bounds loop in setDenseCostMatrix)

2 participants