SPF: ordered bucket queue, reusable workspace, fixed-routing reuse (0.11.0)

.gitignore

@@ -224,3 +224,6 @@ __marimo__/
 
 # Personal Superset lifecycle overrides
 .superset/config.local.json
+
+# Performance harness run logs
+dev/perf/**/*.log

.pre-commit-config.yaml

@@ -4,8 +4,11 @@ repos:
     hooks:
       - id: check-merge-conflict
       - id: check-yaml
+      # Recorded measurement artefacts and review patches must stay byte-exact.
       - id: end-of-file-fixer
+        exclude: ^dev/(research/|perf/.*/(results|review-[^/]+)/)
       - id: trailing-whitespace
+        exclude: ^dev/(research/|perf/.*/(results|review-[^/]+)/)
       - id: mixed-line-ending
       - id: debug-statements
 

CHANGELOG.md

@@ -5,6 +5,14 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.11.0] - 2026-10-02
+
+### Changed
+
+- **Shortest Paths**: `shortest_paths` / `shortest_paths_to` (and therefore `spf`, `spf_to`, max-flow, `FlowPolicy` and KSP spur searches) now run Dijkstra on a monotone bucket queue whenever the graph's edge costs fit 65,536 buckets (`max_cost / gcd(costs) + 1`), which covers unit, uniform and small-integer cost graphs including zero-cost pseudo attachments; wider or int64-offset costs keep the binary heap. The active bucket is ordered by the same `(-bottleneck, node)` key as the heap, so the pop order and every output (distances, predecessor order, single-path capacity ties, early exit) are bit-identical; a differential test in the C++ suite enforces this. Scratch arrays are now a thread-local workspace reset per search (only the touched nodes when few were touched), which removes the O(N) initialisation that dominated small destination-limited searches on large graphs. Measured sequentially on an M4 Max: 1.4–1.8x on weighted random graphs, 1.1–1.3x on unit-cost graphs, 2.4–3.0x on near-destination queries; path graphs (frontier of one node) are about 1.3x slower in absolute terms of a few microseconds, and graphs that fall back to the heap (`W > 65536`) are within a few percent of the previous heap on full-source searches while their small destination-limited searches are still about 2.8x faster. `StrictMultiDiGraph` exposes `max_cost()` / `cost_gcd()`; the C++ functions take an optional trailing `SpfQueue` for tests, and `NGRAPH_CORE_SPF_QUEUE=heap|bucket` overrides the choice process-wide. The search also skips settled neighbours before scanning their parallel edges and sizes the predecessor output from live per-node counts, which makes dense unit-cost fabrics about 1.3x faster; a frontier of one node is held inline, which halves the path-graph penalty.
+- **Max-Flow**: `calc_max_flow` and `FlowState::place_max_flow` with `require_capacity=false` computed the identical cost-only shortest-path DAG on every tier; it is now computed once per call. Results are unchanged.
+- **Internals**: `FlowPolicy` no longer copies the path DAG when returning a bundle, and the `spf` / `spf_to` bindings move the DAG into the Python result instead of copying it.
+
 ## [0.10.0] - 2026-09-13
 
 ### Fixed

README.md

@@ -110,6 +110,9 @@ bindings/python/        # pybind11 bindings
 python/netgraph_core/   # Python package
 tests/cpp/              # C++ tests (googletest)
 tests/py/               # Python tests (pytest)
+dev/                    # Developer scripts (not shipped): checks, coverage,
+dev/perf/               #   performance tooling and recorded A/B/A results,
+dev/research/           #   research prototypes and their evidence
 ```
 
 ## Development
@@ -123,13 +126,17 @@ make cpp-test   # C++ tests only
 make cov        # Combined coverage report (C++ + Python)
 ```
 
+Performance work (harnesses, old/new/old protocol, recorded measurements) is
+described in [dev/perf/README.md](dev/perf/README.md).
+
 ## Environment Variables
 
 | Variable | Effect |
 | --- | --- |
 | `NGRAPH_CORE_PROFILE=1` | Enable profiling of C++ hot paths (`profiling_dump()` / `profiling_reset()`). |
 | `NGRAPH_CORE_BATCH_THREADS` | Worker threads for `batch_max_flow` (default: hardware concurrency). Set to `1` when calling from your own worker pool. |
 | `NGRAPH_CORE_SENSITIVITY_THREADS` | Worker threads for `sensitivity_analysis` (default: hardware concurrency). |
+| `NGRAPH_CORE_SPF_QUEUE` | Frontier queue for SPF: `heap` forces the reference binary heap, `bucket` prefers the bucket queue where the graph is eligible. Unset (default) selects automatically; both give bit-identical results. Read once per process; for benchmarking and diagnosis. |
 
 ## Requirements
 

bindings/python/module.cpp

@@ -209,12 +209,12 @@ PYBIND11_MODULE(_netgraph_core, m, py::mod_gil_not_used()) {
           py::array_t<std::int64_t> dist_arr(res.first.size());
           auto* out = dist_arr.mutable_data();
           for (std::size_t i=0;i<res.first.size();++i) out[i] = (res.first[i]==maxc) ? static_cast<std::int64_t>(maxc) : static_cast<std::int64_t>(res.first[i]);
-          return py::make_tuple(std::move(dist_arr), res.second);
+          return py::make_tuple(std::move(dist_arr), std::move(res.second));
         } else if (dtype == "float64") {
           py::array_t<double> dist_arr(res.first.size());
           auto* out = dist_arr.mutable_data();
           for (std::size_t i=0;i<res.first.size();++i) out[i] = (res.first[i]==maxc) ? std::numeric_limits<double>::infinity() : static_cast<double>(res.first[i]);
-          return py::make_tuple(std::move(dist_arr), res.second);
+          return py::make_tuple(std::move(dist_arr), std::move(res.second));
         } else {
           throw py::value_error("dtype must be 'float64' or 'int64'");
         }
@@ -258,12 +258,12 @@ PYBIND11_MODULE(_netgraph_core, m, py::mod_gil_not_used()) {
           py::array_t<std::int64_t> dist_arr(res.first.size());
           auto* out = dist_arr.mutable_data();
           for (std::size_t i=0;i<res.first.size();++i) out[i] = (res.first[i]==maxc) ? static_cast<std::int64_t>(maxc) : static_cast<std::int64_t>(res.first[i]);
-          return py::make_tuple(std::move(dist_arr), res.second);
+          return py::make_tuple(std::move(dist_arr), std::move(res.second));
         } else if (dtype == "float64") {
           py::array_t<double> dist_arr(res.first.size());
           auto* out = dist_arr.mutable_data();
           for (std::size_t i=0;i<res.first.size();++i) out[i] = (res.first[i]==maxc) ? std::numeric_limits<double>::infinity() : static_cast<double>(res.first[i]);
-          return py::make_tuple(std::move(dist_arr), res.second);
+          return py::make_tuple(std::move(dist_arr), std::move(res.second));
         } else {
           throw py::value_error("dtype must be 'float64' or 'int64'");
         }

dev/perf/README.md

@@ -0,0 +1,21 @@
+# Performance tooling
+
+Developer-only tools for measuring NetGraph-Core. Nothing here ships in the
+wheel or the sdist, and nothing here is run by CI; the repository gates
+(`make check-ci`, `make sanitize-test`) remain the acceptance tests.
+
+- [`spf_queue/`](spf_queue/): the shortest-path frontier queue and workspace
+  work (plan, parity harness, old-versus-new bit-gate, NetGraph-level
+  benchmark, recorded A/B/A results and the independent review). Start with
+  [`spf_queue/PLAN.md`](spf_queue/PLAN.md).
+- [`../benchmark_profiling_overhead.py`](../benchmark_profiling_overhead.py):
+  SPF and placement micro-benchmark used for the PGO training run
+  (`make install-pgo`) and for measuring profiling overhead.
+
+Conventions for performance claims (see `AGENTS.md`): verify correctness
+first (bit-identical outputs where the contract is exact, certificates for
+flows), then measure old/new/old in fresh processes on one quiet machine,
+record load, hashes and raw samples, and treat differences inside the
+run-to-run band as inconclusive. The research record behind the queue work,
+including the Apple GPU investigation that preceded it, is in
+[`../research/`](../research/).