Repository navigation
[MRG] Fix empty and degenerate EMD inputs without native crashes - #904
xihaian251 wants to merge 4 commits into
Conversation
- ot.emd/ot.emd2: reject a cost matrix with an empty dimension with a ValueError before any conversion, and reject non-finite marginal masses - rescale b per histogram with a safe denominator: when both marginals have a zero total mass, return the exact zero plan with finite potentials instead of NaNs, while a batch keeps solving its non-degenerate histograms - ot.emd_1d: reject an empty support instead of crashing in the 1-D sweep - emd_c/emd_c_sparse/emd_c_lazy: validate marginals, initial potentials, edge array lengths and edge endpoints against the native buffers - EMD_wrap/EMD_wrap_sparse: return before building a graph when one side has no positive weight (OPTIMAL for an empty problem, INFEASIBLE otherwise), which also removes the &weights[0] access on empty vectors - NetworkSimplexSimple: do not read D[0] in setDenseCostMatrix when there is no arc, initialise in_arc, and return from findEnteringArc as soon as no arc has a negative reduced cost - add 11 regression tests covering the empty, zero-mass, degenerate-batch, 1-D and direct native entry points Closes PythonOT#901
|
Hi, The only failing required check appears to be In [CircleCI job #4273](https://circleci.com/gh/PythonOT/POT/4273), the All 16 GitHub Actions jobs passed, including doctests and the source build. I also compared a minimal Sphinx build of the modified docstrings against the base commit; both produced the same warning, with no new warnings introduced by this PR. I don't appear to have permission to rerun the CircleCI workflow. Could a maintainer with access please rerun it from the failed job? If the rerun reveals an actual documentation issue, I'll address it. Thanks! |
|
The failure is in the "Get Python running" step (964 s timeout fetching the 539 MB PyTorch wheel), so For what it's worth, #911's Could someone with CircleCI access rerun job 4273, or is the passing "Build doc" run enough to consider |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #904 +/- ##
==========================================
+ Coverage 97.09% 97.11% +0.01%
==========================================
Files 130 130
Lines 27229 27340 +111
==========================================
+ Hits 26437 26550 +113
+ Misses 792 790 -2 🚀 New features to boost your workflow:
|
|
Hi all, The earlier CircleCI This PR addresses the interpreter crash in #901: Public Would a maintainer be able to review this when convenient? I'm happy to address any feedback. Thanks! |
Summary
ot.emd/ot.emd2kill the interpreter on an empty cost matrix (issue #901):NetworkSimplexSimple::setDenseCostMatrixreadsD[0]and then loops withfor (ArcsType i = 1; i != _arc_num; ++i). With_arc_num == 0,D[0]is already out of bounds and the loop continues scanning beyond thebuffer (until a fault or unsigned wraparound), causing SIGBUS/SIGSEGV instead
of a Python exception.
The same degenerate supports expose three further native hazards:
&weights1[0]on an empty
std::vector, an uninitialisedin_arcin the block-search pivotrule, and unchecked marginal/potential/edge buffers in the Cython entry points
(
emd_c,emd_c_sparse,emd_c_lazy). A one-sided empty support also makesot.emd_1dsegfault.This PR makes the affected exact-EMD entry points handle these inputs safely,
at the layer where the necessary information is available. In the tested
non-degenerate cases, the numerical results are unchanged.
Fixes #901.
Root cause
Public API → Cython → C++:
ot.emd/ot.emd2default an empty marginal tonx.ones(n) / n, which is0/0forn == 0, then rescalebwithb * a.sum() / b.sum()(0/0).With a non-empty cost matrix and zero total mass on both sides, the active
support filter hands a
(0, 0)sub-matrix to the solver.emd_cpasses the raw pointers toEMD_wrap. With no positive weight,n = m = 0,full_supportis true, and the dense-cost fast path callssetDenseCostMatrix(D, 0):_max_cost = D[0]is an out-of-bounds read and thei != _arc_numloop scans beyond the buffer. This is the reported crash.EMD_wrapalso evaluates&weights1[0]/&weights2[0]on empty vectorsbefore any solver call (undefined behaviour), and
ot.emdon a sparse matrixwith no stored entry reaches
findEnteringArc()with zero arcs, whichdereferences the never-initialised
NetworkSimplexSimple::in_arc(
init()does not set it; onlywarmstartInit()does) and segfaults.coordinates and edge endpoints match the native buffers, so a direct caller
can turn a shape mismatch into an out-of-bounds read.
Changes
ot/lp/_network_simplex.py(public API policy)(0, 0),(0, n),(n, 0), dense orsparse) raises
ValueErrorbefore any conversion: an empty marginal cannot bedefaulted to a uniform distribution (
[]is the documented sentinel for"uniform").
bis rescaled per histogram with a safe denominator(
b * mass_a / np.where(mass_b != 0, mass_b, 1.0)). With nonnegativemarginals, a positive-mass source and a zero-mass target remain infeasible;
a zero-mass source instead rescales a nonzero target to zero when
check_marginals=False. Each histogram in a batch is handled independently.The formula is unchanged when
mass_b != 0, including negative totals.ValueErrorinstead of silently being treatedas an empty support.
center_ot_dualno longer divides by zero when both marginals are empty, sothe returned potentials stay finite.
ot/lp/emd_wrap.pyx(Cython boundary)emd_c,emd_c_sparse,emd_c_lazy: reject marginals, initial potentials andedge arrays whose lengths do not match the problem;
emd_c_sparsealso rejectsendpoints outside the marginals and non-equal edge array lengths. The
pre-existing "empty marginal means uniform" fallback of
emd_c/emd_c_lazyispreserved.
emd_1d_sorted: an empty support raisesValueErrorinstead of readingu_weights[0]and writing out of bounds (reachable fromot.emd_1d).ot/lp/EMD_wrapper.cpp(native entry points)EMD_wrap/EMD_wrap_sparse: return before building any graph when one sidehas no positive weight. Both sides empty is the trivial problem (
OPTIMAL,cost 0, empty plan); one side empty is
INFEASIBLE. This also removes the&weights[0]undefined behaviour.EMD_wrap_sparse: reject endpoints outside the marginals before indexing thereverse maps.
ot/lp/network_simplex_simple.h(solver)setDenseCostMatrix: no read ofD[0]and no scan when_arc_num == 0, andthe scan condition is
i < _arc_num.init()initialisesin_arcdeterministically; the pivot rule checks forabsent eligible arcs before using it.
BlockSearchPivotRule::findEnteringArcreturnsfalseas soon as no arc has anegative reduced cost, instead of scaling its tolerance with the unset
_in_arc. The old code was equivalent whenever it did not read uninitialisedmemory; this is a strict improvement and saves the useless scan.
ot/lp/solver_1d.py(public 1-D API)ot.emd_1drejects an empty support before defaulting the weights to uniform,instead of dividing by zero and crashing in the native 1-D sweep.
test/test_ot.py— 11 regression tests (empty problems, zero total mass ondense and sparse inputs, one-sided zero mass, batches with an empty histogram,
ot.emd_1d, and the three native entry points).RELEASES.md— one entry under 0.9.8dev / Closed issues.Tests
Environment: Windows 11, Python 3.13.1, NumPy 2.5.4, SciPy 1.18.1, Cython 3.3.0,
MSVC 19.44; worktree at
0b6f8d5,ot.lp.emd_wraprebuilt in place from theworktree sources with the extension spec of
setup.py(the binary postdatesevery native source; MSVC
/GLbuilds are not bit-reproducible, so no artifacthash is quoted). Linux side: WSL Ubuntu 24.04, gcc 13.3, AddressSanitizer.
pytest test/test_ot.py -o addopts="" -qpytest test/ -o addopts="" -qscipy.optimize.linprog/HiGHS): 102 comparisons out of 110 sampled problems (8 sampled sparse masks forbade all transport, the oracle returned infeasible and the trial was skipped); dense, sparse mask, rescaled marginals, zero mass, uniform contractot.emd2([], [], np.zeros((0, 0)))heap-buffer-overflow … network_simplex_simple.h:932 in setDenseCostMatrixThe numbers above come from the final run with the JAX backend installed; the same suite run before JAX was installed reported 1527 passed, 139 skipped, 4 xfailed and 6 failed, i.e. the same only-BSP failures.
The 7 reported failures are
test_bspot.py(5) andtest_solvers.py::test_solve_sample_bsp(NumPy and JAX). The local environmentcould not fetch the
deps/eigensubmodule (gitlab.com returned 5xx), so theBSP extension was not built. These tests fail with
NotImplementedErrorfroma local placeholder and do not enter the modified EMD paths. The full suite
therefore did not pass on this machine; the affected EMD regression tests
did pass. The placeholder is not part of the proposed patch.
Each new test was run against a second worktree built from the same base commit
without the patch:
test_emd_empty_problem_raises,test_emd_sparse_no_transport_edges,test_emd_c_empty_supportandtest_emd_1d_empty_supportsegfault the interpreter;test_emd_zero_mass,test_emd_zero_mass_sparse,test_emd_zero_mass_one_sided,test_emd_c_shape_mismatch,test_emd_c_sparse_invalid_edgesandtest_emd_c_lazy_shape_mismatchfail on assertions.test_emd_empty(theexisting uniform-marginal contract) and
test_emd2_batch_with_empty_histogram(a regression guard for behaviour that wasalready correct) pass on both.
Compatibility
ot.emd([], [], M)on a non-emptyMkeeps its "uniform marginals" meaning;test_emd_emptyis unchanged and passes.build. The rescaling expression is unchanged for
mass_b != 0, andcenter_ot_dualis unchanged when the total mass is positive. Cases checkedinclude the uniform contract, partial zeros, non-unit mass, integer weights,
sparse transport, and warm starts.
AssertionErrors(
test_emd_dimension_and_mass_mismatchunchanged).ValueError;spurious "infeasible" warning → zero plan, cost 0, finite potentials,
no warning. Dense-path dual feasibility was checked; sparse-path potentials
for zero-weight nodes are not necessarily dual feasible with negative costs.
The zero plan is the unique feasible plan and is optimal;
check_marginals=False: if the source haspositive mass but the target total is zero, the solver reports
INFEASIBLEwith a zero plan and finite potentials; if the source haszero mass but the target total is positive, the existing target-rescaling
convention produces zero target mass and the optimal zero plan;
check_marginals=False: undefined results →ValueError;ot.emd_1dwith an empty support:ValueError("negative dimensions are not allowed")or a segfault →ValueErrornaming the empty support;emd_c/emd_c_sparse/emd_c_lazywith mismatched buffers:out-of-bounds reads →
ValueError.behaviour: negative entries still reach the solver (which reports
INFEASIBLEwhen it sees them), and a negative total target mass is rescaledexactly as before (
np.where(mass_b != 0, mass_b, 1)only replaces thedivision by zero, so the result is unchanged whenever
mass_b != 0).sparse path, as documented in
emd_c); with negative costs they are thereforenot necessarily dual feasible. This is pre-existing and unchanged; only dense
inputs get repaired potentials.
Not in this PR
ot.emd2_lazyandot.lp.emd_1d's handling of zero weights (not emptysupports):
emd2_lazystill computesb * a.sum() / b.sum()and returns NaNpotentials when both masses are zero. It does not crash (its native entry point
has an early return) and is left for a separate change to keep this PR focused.
_gridEMD path of [WIP] Add exact EMD-L1 solver for Cartesian grid histograms #863 is a different code path and is not touched.AI disclosure
This change was prepared with an AI coding assistant (Qoder) used for source
reading, patch drafting, test generation, and running the build/test/sanitizer
matrix. Three independent AI review passes were run against the draft: a
maintainer-style review (API, compatibility, tests), a native memory-safety
review, and a re-review of the revised patch. The first two found real defects
that were fixed before this version: the uninitialised
in_arcwas stilldereferenced on the pivot rule's block-boundary path, the batched
emd2aborted a whole batch on one empty histogram, and
ot.emd_1d/emd_c_lazyhadtheir own crashes. The re-review approved the revision and confirmed that
the tested non-degenerate results were bit-identical to master. No human code review was
performed. All commands and results in the Tests section were executed on the
author's machine and are reproducible from the branch.