Skip to content

Streamline: the rebuilt Core (phases 0 to 1e and the cleanup) - #12

Draft
networmix wants to merge 12 commits into
mainfrom
streamline
Draft

Streamline: the rebuilt Core (phases 0 to 1e and the cleanup)#12
networmix wants to merge 12 commits into
mainfrom
streamline

Conversation

@networmix

@networmix networmix commented Oct 12, 2026 •
edited
Loading

Copy link
Copy Markdown
Owner

Draft: not to be merged yet. NetGraph still calls the 0.11 API; this merges after NetGraph's migration to it. Opened now to run CI on every platform and to keep the work off one machine.

Summary

A rebuilt Core (breaking, version 0.12.0): one numeric domain (capacities in integer units of an explicit resolution), one minimum-cost flow core, one placement engine. It replaces the 0.11 API; CHANGELOG.md (Unreleased) lists every change.

  • Phase 0: the integration branch and the version.
  • Phases 1a to 1e: the graph on integer units with stable edge ids, the min-cost flow core, the integer ledger and placement kernels, the policy engine, evaluators and DagCache, the public search, KSP and the frozen cache, and graph construction by counting passes.
  • Cleanup: bindings split by subject, a generated and checked type stub, docs/semantics.md stating the API's rules (each with a test), dead code and old experiments removed, defects found on the way fixed.
  • dev/perf/: a minimal A/B/A timing tool with its own tests (one macOS CI job).

Verification (local, Apple M4 Max)

  • Lint and types; 303 C++ tests with the private test hooks, without them, under ASan/UBSan and under ThreadSanitizer; 492 Python tests (399 passed and 93 hook-only skips without the hooks); the type stub against the built module.
  • A recorded corpus of 29,266 queries gives the same results as the accepted phase 1e build, with the hooks on and off (only the field naming the producing build differs).
  • A/B/A timing against the 1e build, 51 rows: no row slower beyond the measurement noise; flow-core rows 3-4 % faster. Against the phase 0 build (df3b894, before the rebuild), graph construction takes 21-91 % less time and the timed SPF rows 9-82 % less.

Not yet run: CI on Linux, Windows and CentOS (this PR), NetGraph integration.

🤖 Generated with Claude Code

networmix and others added 12 commits October 3, 2026 18:05
@networmix @claude
Phase 0: integration branch with the API batch (spf_dag, EdgePriority…
76d8c0d 
…, from_hops, cost metadata)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@networmix @claude
Phase 0: version 0.12.0 on the integration branch; lint excludes for …
df3b894 
…review artefacts

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@networmix @claude
Phase 1a: graph on integer units with B′ edge ids and the min-cost fl…
b476639 
…ow core

Delete the old library (FlowState, the old FlowGraph and FlowPolicy, the backend,
the old search entry points and their bindings: 411 symbols, 65 files) and build
the first piece of the streamlined Core against APPROACH revision 25:

- StrictMultiDiGraph(num_nodes, src, dst, capacity_units, cost, resolution): capacities
  as int64 units of an explicit resolution, costs >= 1, edges in the canonical
  (src, cost, dst, input) order so the forward CSR slot is the public EdgeId,
  input_index() as the only map, to_units (half up) and as_float, validation in the
  contract's order (resolution, unit values, shapes, costs, bounds) before any narrowing.
- The flow core: successive shortest paths with potentials on reduced costs, Dinic
  phases per tier, forward and reverse arcs interleaved in ascending EdgeId at every
  node, virtual terminals, a fresh solve from zero on the view, the two-branch
  behaviour for an input cap of 0, the closure seeded at the real sources; exposed to
  the recorder through the private _place_global_units entry with the typed payload
  (placed, cost_distribution, edge_flows, min_cut, potentials with NEVER_REACHED = -1,
  six int64 counters).
- 38 C++ tests (every validation error, B′ order, the review witnesses: attained value,
  update-before-drain, two Dinic phases in one tier, the legal maximum, the zero-view
  cap, masks and cost factors) and 154 Python tests.

Gates (dev/design/2026-10-streamline/gates/1a/, uncommitted design tree): workspace
check and sanitizers green; independent oracle population 3,821 pass / 0 fail;
cross-check against the Python reference on 30,000 random graphs identical; the
equivalence run against the G0 baseline 0 failures under the 1a manifest rules;
inventory: 0 Core callers of any deleted symbol (NetGraph callers pending until
Phase 2); construction and max-flow A/B/A rows 23 to 48 percent faster than df3b894.
Reviewed adversarially (review-1a, review-1a-fixes): D1 to D4 fixed or settled in the
contract.

NetGraph is red from this commit until its migration converges (Phase 2), by decision
of 2026-10-04.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@networmix @claude
Phase 1b: the integer ledger, the placement kernels and the internal …
231eb85 
…search

Build the second piece of the streamlined Core on the 1a graph and flow core, against
APPROACH revision 27:

- FlowGraph: the integer ledger (records as snapshots in FlowId order, residual and
  allocated views, bottleneck over member edges, reset only unbound), the lifetime table
  (bound without flows, bound with flows, orphaned records; copy, move, destroy), epochs,
  ids never reused; a staged, exception-safe commit (every throwing step before one
  no-throw publication, an epoch-stamped touched set, all-or-nothing scratch growth) with
  fault-injection tests.
- The three kernels in units: MaxFlow as a fresh minimum-cost solve on the view
  V_i = R0 + own_i with arcs in B' order; Weighted with membership by priority at
  creation, frozen weights and highest-averages apportionment; EqualBalanced with round
  robin on a fixed cohort; Admission.Lossy on EB and Weighted (per-call offers, transit
  drops, +inf injection); split-graph pruning; T_max by bisection with an exact bracket
  and a checked 62-probe bound.
- PredDAG (from_csr with complete validation, predecessors in ascending via EdgeId),
  Path(g, hops) with to_dag(), Eligibility with intersect.
- The internal search: the unique shortest-path DAG over the eligible member graph,
  anycast pruning, the staged canonical single path with PathSelection projection.
- Bindings, _docs.py and stubs for the public classes; 100 C++ tests and 200 Python tests.

Gates (dev/design/2026-10-streamline/gates/1b/, uncommitted design tree): workspace
check and sanitizers green; oracle population 3,821 pass / 0 fail; check_spf
3,820/3,820; prototype cross-check identical on 45 fixtures and 16,172 random cases;
909 corpus Manual placements equal their references; the equivalence run 0 failures
under the 1b rules (no observable change before the policies of 1c); inventory: 0 Core
callers of any deleted symbol; A/B/A: construction and max-flow rows 22 to 47 percent
faster than df3b894, the five kernel rows 5 to 36 percent faster after the one-pass
PredDAG-native route and plan builder; a per-process bimodality of the
proportional-grid row (an out-of-line NodeMap::add in the flow core's subset solve whose
callee-saved register reloads stalled in some process layouts) fixed by an allocation-free,
noexcept, inlined add with a guard test, both grid rows unimodal (-33 and -4 percent).
Reviewed adversarially (review-1b, review-1b-fixes, review-1b-fixes2,
review-1b-bimodality): D1 to D3, N1, N2 and R15 fixed; D3 settled in the contract
(revision 26), R16 in revision 27.

NetGraph stays red until its migration converges (Phase 2).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@networmix @claude
Phase 1c: the policy engine, the evaluators and the DagCache; the 1a …
2088f3b 
…to 1c tree coherent and groomed

Build the third piece of the streamlined Core on the 1a graph, the 1b ledger and kernels,
against APPROACH revision 42, then make the whole 1a to 1c tree coherent and groom its public
interface before 1d:

- FlowPolicy with the four routing modes (CostOnly, CapacityAware, Global, Pinned),
  DiminishingReturns, the cost ceiling (Absolute, Stretch), explicit placement, admission and
  selection; place(volume, outputs) -> PlacementResult (placed, remaining, termination,
  local outcomes per roster source, the two event-accumulating flags, drops, the cumulative
  cost curve, edge flows, residual, bottleneck, min cut); remove(), records(), record(id),
  placed_demand(), usable_bundles(); the roster fixed at construction (9.1); the per-source
  transaction of 9.2 (cumulative round-robin offers, stages 1 to 4 on a per-thread working
  overlay, growth never repeating a route on a multipath DAG, T_max probes within the
  62-probe bound); the termination reduction and the flag table of 9.4; the epoch law and
  lifetimes of section 8; Global as the cumulative minimum-cost flow on V = W + own with the
  canonical decomposition and per-source records; every output prepared before one atomic
  commit, including the Python result objects (all-or-nothing across worker threads and the
  binding, APPROACH 13, with its two stated limits: the pybind11 object machinery, and a
  supplied building DagCache as a memo).
- Sparse record bundles over member nodes (R16): the public CSR shape on demand, unchanged
  supplied bundles shared, the builder's grouping by a stable counting sort.
- DagCache in building mode: key (root set, direction, Eligibility uid), value the full
  static support, statistics (hits, misses, entries, bytes, live values), clear().
- Evaluators: capacity, batch_capacity (arguments first, then each pair completely in pair
  order; the lowest failing index; explicit per-pair sequences length-checked; Pinned
  rejected), sensitivity (Global positive-flow candidates, the other models every allowed
  edge of at least one unit, signed losses, unevaluable with a reason); workers with
  private caches allocated before any thread starts, noexcept worker bodies, every thread
  joined before the lowest-index failure is rethrown; the exact shortcuts (Global loss from
  the baseline's maximum flow, CostOnly off-route edges skipped).
- One min-cost flow core with the deletion re-solve as a mode (APPROACH 2.2): the deletion
  state as a value, working potentials as published potentials plus the running sum of the
  deletion deltas, 128-bit working arithmetic within the stated bounds, the three
  fixed-origin repairs with the failed-search reuse; it replaces the separate warm solver
  and the 1b single-bundle driver (_PinnedFlow, src/pinned_flow.cpp); a one-bundle Pinned
  FlowPolicy stands in for the latter.
- The coherence pass of the whole tree: no dead code, no wrong comment, no leaked
  abstraction, one implementation per rule; profiling removed; the test probes and trace
  behind the CMake option NETGRAPH_CORE_PRIVATE_HOOKS (OFF for wheels, ON for the
  development build and the tests; the OFF build exports no hook, probe or trace symbol);
  transactional copy assignment of the deletion state; validating constructors for every
  enumeration-typed option (std::invalid_argument naming the field).
- The public interface groomed to one scheme: Graph (was StrictMultiDiGraph), FlowLedger
  (was FlowGraph) taken by FlowPolicy as ledger=, OutputOptions (was Outputs),
  DiminishingReturns (was Diminishing), MAX_FLOWS_PER_CALL in Python (kMaxFlowsPerCall in
  C++); the public headers split so each says what it holds (graph.hpp, eligibility.hpp,
  flow_record.hpp, flow_ledger.hpp, flow_policy.hpp, evaluators.hpp, dag_cache.hpp,
  options.hpp with the two placement enums, pred_dag.hpp, types.hpp); Counter128
  self-contained, so no public header includes a detail/ header; the private headers and
  the test files on one naming scheme; the policy fixture data in tests/data/ with a
  maintained exporter; no alias, no re-export, no adapter of a retired spelling.
- The kernel's round robin as one inline callback template (one implementation of the split; the
  span form a wrapper), so allocate and the Lossy step run in one pass: the coherence pass's
  two-pass form had cost kernel.eb.grid 11 percent against df3b894; results unchanged.
- The kernel plan builder fills its arrays by index into pre-sized vectors instead of push_back:
  under Xcode 27's libc++ 22 the push_back form cost the kernel rows about ten percent
  (emplace_back's new fast-path and slow-path lambdas); results unchanged.
- Bindings, _docs.py stubs and the Python package for every class and function
  (keyword-only configuration objects, keep-alive of ledger, Eligibility and cache, the GIL
  released in the evaluators); pytest collects tests/ only; 157 C++ tests (134 ordinary,
  23 in the separate fault executable) and 266 Python tests, with fault-injection sweeps
  of every growth, publication and deletion path at 1, 2 and 4 workers and at every Python
  allocation position.

Gates (dev/design/2026-10-streamline/gates/1c/, uncommitted design tree), on the final
groomed build with the private hooks on and off: workspace check and sanitizers green; the
fault executable under the sanitizers; the warning-set builds with 0 diagnostics; the
literal equivalence run 0 failures (31,788 unchanged, 28,224 new-api, 14,778 rule, 13,701
new-only; the rest invalid-input, deferred to 1d or absent by rule); oracle population
13,901 pass / 0 fail (ksp and spf queries wait for 1d); every returned value checked and
proven by mutation (114,222 corruptions of the accepted results, 0 undocumented survivors);
inventory CHECK OK on 1,249 rows, 0 Core callers of any deleted symbol; prototype parity
identical on the 1b and 1c drivers (45 and 59 fixtures, 8,412 random cases); every recorded
result of the groomed build identical to the pre-grooming build's (20,960 of 20,960) and to
the frozen 1c baseline; A/B/A against df3b894 on the final tree (both sides Apple clang 21 build
2100.3.34.2, Xcode 27), 15 of 15 rows: construction -43 and -52 percent, the kernels -3 to
-28 (kernel.eb.grid -1.8 within tolerance), capacity -28 and -24, max-flow -13 and -14,
sensitivity -8 (the deletion re-solve replacing the separate warm solver) and -92, the
CapacityAware cohort rows -61 and -99.6; the new-model rows as target-versus-target A/A, 3 of 3. Reviewed adversarially (review-1c, review-1c-fixes, review-1c-r29,
review-resolve 1 to 3, review-audit-1abc, review-coherence, review-coherence-rc,
review-naming 1 to 4, review-grooming, review-timing-fixes): F1 to F5 fixed (a worker-thread allocation that
escaped to std::terminate, Python publication after the commit, batch validation order, the
explicit empty eligibility list, the Global zero call's cumulative curve); the coherence
audits' findings fixed and RC1 to RC3 closed; the tooling's twelve gate holes (W11) closed
by W12 before these gates ran.

NetGraph stays red until its migration converges (Phase 2).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@networmix @claude
Two small speed-ups, results unchanged
a48ca9d 
- Weighted quota division: a 64-bit division when the dividend fits
  (kernel.weighted.clos 6 to 8 percent faster).
- Graph constructor: stop computing the GCD once it reaches 1
  (construct.clos about 4 percent faster).

Both measured A/B/A against the 1c build; every result byte-identical;
a test for each.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@networmix @claude
Phase 1d: the public search, KSP and the frozen DagCache
3e1f3ea 
One traversal core now runs every search. The public spf searches from a
set of roots, forward or reverse, optionally stopping at the nearest
target. ksp lists the K shortest paths in the contract's order, and
resolve_to_paths unfolds a DAG into paths. The DagCache gains a frozen
mode that every worker of an evaluator can share. Hits are published
when the call succeeds; a call ending in DagCacheMiss counts one miss.
The Python bindings cover all of it, and the old search entry points are gone.

CostOnly placement builds a new flow's route, record DAG and kernel plan
in one pass over the cached support, with a few smaller cuts. NetGraph's
MSD loop now takes about 3 % less time than 0.12 instead of 35 % more;
results are unchanged.

Evidence: after the independent review's fixes, the final gates pass
with hooks on and off; the timed SPF rows take 9 to 82 % less time than 0.12.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@networmix @claude
Phase 1e: graph construction by counting passes
c1c9de7 
The Graph constructor numbers the edges with stable counting passes
instead of a full sort. Costs are ranked through a hash table that
reuses the graph's own arrays; above 65,536 distinct costs, or when the
table probes too long, the costs are sorted instead.

Results are unchanged; construction takes 21 to 91 % less time than
0.12. An empty NumPy view from a graph accessor now also keeps its
graph alive.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@networmix @claude
Cleanup: the rebuilt Core made coherent, documented and tested
16e5100 
The bindings are split by subject, the type stub is generated from the
module and checked, and docs/semantics.md states the API's rules, each
with a test. A double free when a result object could not be wrapped
is fixed, and inputs too large for the flow core are now refused.
Dead code and the old research and timing material are gone.

Results match accepted 1e with the private hooks on and off; only the
field that names the producing build differs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@networmix @claude
dev/perf: a minimal A/B/A timing tool
cb4dd8e 
One driver times the Core's rows as old build, new build, old build
again on one machine, so drift shows, and checks that both builds give
the same results. Its own tests run with make perf-tool-test and on one
macOS CI job, so a library change that breaks the tool fails there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@networmix @claude
Portability fixes found by the first CI run
a6a24c0 
- The private test hooks build on Python 3.11 (no PyErr_GetRaisedException
  before 3.12) and keep the exception object and its traceback.
- The rank oracle uses += so the portable 128-bit test integer (MSVC) compiles.
- The two large fault sweeps get a 600 s ctest timeout; they finish but run
  long under sanitizers on small CI machines.
- The native allocation sweep accepts a failure that the standard library
  absorbs (GNU stable_partition's optional buffer) only with the exact
  reference result, state and live count; its discard control now compares
  a real reference result.

Library code and results are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@networmix @claude
Negative controls independent of the platform and the tracer
b61fa9e 
- The few-allocations control plants a call that makes no native
  allocation on any platform; reset's count depends on the standard library.
- The Python allocation sweep's controls advance at snapshot boundaries, so a
  tracer that fails before a call enters its body cannot shift them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@networmix