Skip to content

Multigraphs via an edge index and edge properties/metadata - #81

Open
zblanco wants to merge 25 commits into
bitwalker:mainfrom
zblanco:zw/multigraph-indexes
Open

zblanco wants to merge 25 commits into
bitwalker:mainfrom
zblanco:zw/multigraph-indexes

Conversation

@zblanco

@zblanco zblanco commented Aug 20, 2024 •

Copy link
Copy Markdown
Contributor

Adds opt-in partition indexes and edge properties for #18 and #54. Graphs already distinguish parallel edges by label; Graph.new(multigraph: true) adds an index so selected edge kinds can be queried without scanning unrelated neighbors.

API and storage

  • partition_by receives an edge and returns a list of keys, defaulting to fn edge -> [edge.label] end. For example: Graph.new(multigraph: true, partition_by: fn edge -> [edge.weight] end).
  • Edge queries accept by: (one partition or a list) and where: (an edge predicate). BFS, DFS, Dijkstra, A*, and Bellman-Ford support partition filtering.
  • The index is %{partition => %{vertex_id => MapSet.t(edge_key)}}. Incident queries begin with the selected endpoint's indexed candidates; relabeling and deletion update affected memberships while retaining partitions shared by parallel labels.
  • Existing %{label => weight} records remain unchanged. Optional properties live separately in Graph.edge_properties and are exposed as %Graph.Edge{properties: %{}}.
  • Indexing trades extra storage and mutation work for narrower queries. Retrieval cost depends on selected endpoint pairs and their parallel labels.

Reconciliation with upstream

Merged upstream main at 0215768 and ported the implementation improvements from the multigraph fork at 5ede725, retaining the Graph namespace and :libgraph package. Fork release configuration, renamed modules, development-server dependencies, and generated artifacts are excluded. Upstream CI, dependency lock, package metadata, and minimum Elixir requirement are retained.

The integration combines upstream's incident-local vertex mutations, exact numeric identity, custom-identifier pathfinding/subgraphs, self-loop handling, traversal, serialization, and documentation fixes with the fork's partition-local queries and mutation optimizations. Regression tests additionally cover:

  • Changed-ID and coalescing vertex replacement, including undirected orientation, edge properties, and complete index reconstruction.
  • Split and bulk edge deletion cleaning up properties and partition memberships.
  • Indexed numeric identities and custom-identifier subgraphs.
  • Filtered shortest paths with custom identifiers, cheapest parallel-edge relaxation, and reverse undirected traversal using property-based partitions.

Validation

Local validation of integration commit 772ea9e (followed by a2297aa, which removes an unreachable clause flagged by Elixir 1.20):

  • Exact-commit detached checkout, Elixir 1.19.5 / OTP 28.5: 117 doctests, 15 properties, 174 tests; 0 failures.
  • Elixir 1.18.3 / OTP 27.2 full suite: 306 checks; 0 failures.
  • Formatting, compilation with warnings as errors, documentation with warnings as errors, and local Hex package build pass.
  • Dialyzer passes on Elixir 1.18.3 / OTP 27.2. An additional OTP 28 run reports MapSet opacity warnings; the same 10 warnings reproduce on unmodified upstream 0215768.
  • The Elixir 1.20 follow-up also passes formatting, compilation with warnings as errors, and all 32 focused index/pathfinding regression tests.
  • GitHub CI passes on final head a2297aa: Elixir/OTP 1.10/23, 1.13/24, 1.16/26, 1.18/27, and 1.20/29; formatting, compile, Dialyzer, documentation, and package checks; and workflow security. BEAM matrix and quality run, workflow security run.

The included bench/partition_indexes.exs was also run sequentially against old PR head 3228065 and integration commit 772ea9e on Elixir 1.19.5 / OTP 28.5, with four schedulers, sizes 128/512/2048, and three measured trials. At 2,048 history entries (6,146 total edges), median times were:

Operation Old PR Reconciled
500 incoming partition queries 300.164 ms 0.420 ms
500 outgoing partition queries 142.133 ms 0.441 ms
500 independent single-edge relabels 86.653 ms 2.570 ms
Relabel all 2,048 pending edges 175.105 ms 6.243 ms

These are synthetic workload measurements with fixed selected degree, not general application performance claims. Raw measurements remain outside the repository.

@zblanco
zblanco marked this pull request as ready for review June 24, 2025 17:05
@zblanco

zblanco commented Jun 24, 2025

Copy link
Copy Markdown
Contributor Author

I've been using this PR inside https://github.com/zblanco/runic for some time now as a way to keep causal runtime edges produced during DAG executions from increasing the dataflow traversal costs. It hasn't need changes and tests pass for 1.18 and might be worth reviewing.

I developed this locally on Elixir 1.18+ which entailed some changes to doctests related to ordering of some results but this leaves older versions broken in CI. Not 100% on a preferred course of action - I would rather find a way to keep older versions compatible so that's what I'm looking into now.

This test discrepancy may be related to OTP updates as it's not related to the actual changes.

1) test sizeof/1 (Graph.UtilsTest)
Error:      test/utils_test.exs:8
     match (=) failed
     code:  assert 456 = sizeof(String.duplicate("bar", 128))
     left:  456
     right: 440
     stacktrace:
       test/utils_test.exs:10: (test)

@zblanco

zblanco commented Feb 21, 2026

Copy link
Copy Markdown
Contributor Author

The library I've been building that depends on this PR's multigraph functionality: https://github.com/zblanco/runic is about ready to be released on Hex with some other libraries on top of that also looking to be released.

Latest commits should add multigraph interaction to the rest of graph mutations and traversal / pathfinding.

There are some included benchmarks that show space / time trade offs at different graph sizes with the adjacency index. I also benchmarked this PR against main:

Benchmark main map edge properties separate map edge properties
create 10k 10.69ms 11.61ms (+8%) 10.74ms
create 100k 170.36ms 180.94ms (+6%) 173.75ms
create 1M 3.72s 4.09s (+10%) 3.96s
topsort 7.39ms 8.03ms (+9%) 7.23ms
k_core 1.77s 2.14s (+21%) 1.87s

The original changes I made had edge metadata changed to %{label => %{weight: w, properties: %{}}} but this added overhead and some minor performance regressions so I moved it back to %{label => weight} and stored in a separate edge_properties map on the graph struct - only populated when used.

mix bench.multigraph results:

➜  libgraph git:(zw/multigraph-indexes) ✗ mix bench.multigraph       
Operating System: Linux
CPU Information: AMD Ryzen 7 5800X 8-Core Processor
Number of Available Cores: 16
Available memory: 62.74 GB
Elixir 1.18.3
Erlang 27.2

Benchmark suite executing with the following configuration:
warmup: 2 s
time: 10 s
memory time: 5 s
reduction time: 0 ns
parallel: 1
inputs: 10k vertices, 50k edges, 50 labels, 1k vertices, 5k edges, 10 labels
Estimated total run time: 2.27 min

Benchmarking indexed lookup (multigraph by:) with input 10k vertices, 50k edges, 50 labels ...
Benchmarking indexed lookup (multigraph by:) with input 1k vertices, 5k edges, 10 labels ...
Benchmarking indexed out_edges (multigraph by:) with input 10k vertices, 50k edges, 50 labels ...
Benchmarking indexed out_edges (multigraph by:) with input 1k vertices, 5k edges, 10 labels ...
Benchmarking scan all edges + filter (no multigraph) with input 10k vertices, 50k edges, 50 labels ...
Benchmarking scan all edges + filter (no multigraph) with input 1k vertices, 5k edges, 10 labels ...
Benchmarking scan out_edges + filter (no multigraph) with input 10k vertices, 50k edges, 50 labels ...
Benchmarking scan out_edges + filter (no multigraph) with input 1k vertices, 5k edges, 10 labels ...
warning: using map.field notation (without parentheses) to invoke function Benchee.Conversion.Count.base_unit() is deprecated, you must add parentheses instead: remote.function()
  (benchee 1.1.0) lib/benchee/conversion/scale.ex:165: Benchee.Conversion.Scale.do_best_unit/3
  (benchee 1.1.0) lib/benchee/conversion.ex:63: Benchee.Conversion.units/2
  (benchee 1.1.0) lib/benchee/formatters/console/run_time.ex:90: Benchee.Formatters.Console.RunTime.render/2
  (benchee 1.1.0) lib/benchee/formatters/console.ex:134: Benchee.Formatters.Console.generate_output/3
  (elixir 1.18.3) lib/enum.ex:1714: Enum."-map/2-lists^map/1-1-"/2


##### With input 10k vertices, 50k edges, 50 labels #####
Name                                              ips        average  deviation         median         99th %
indexed out_edges (multigraph by:)          1543.31 K        0.65 μs ±10047.10%        0.55 μs        0.91 μs
scan out_edges + filter (no multigraph)      648.08 K        1.54 μs  ±3613.50%        1.19 μs        3.56 μs
indexed lookup (multigraph by:)                1.27 K      786.65 μs    ±26.92%      810.52 μs      938.08 μs
scan all edges + filter (no multigraph)      0.0360 K    27812.55 μs    ±33.33%    25003.63 μs    89080.10 μs

Comparison: 
indexed out_edges (multigraph by:)          1543.31 K
scan out_edges + filter (no multigraph)      648.08 K - 2.38x slower +0.90 μs
indexed lookup (multigraph by:)                1.27 K - 1214.04x slower +786.00 μs
scan all edges + filter (no multigraph)      0.0360 K - 42923.31x slower +27811.91 μs

Memory usage statistics:

Name                                       Memory usage
indexed out_edges (multigraph by:)           0.00108 MB
scan out_edges + filter (no multigraph)      0.00317 MB - 2.92x memory usage +0.00208 MB
indexed lookup (multigraph by:)                 1.03 MB - 946.52x memory usage +1.02 MB
scan all edges + filter (no multigraph)        32.58 MB - 30071.13x memory usage +32.58 MB

**All measurements for memory usage were the same**

##### With input 1k vertices, 5k edges, 10 labels #####
Name                                              ips        average  deviation         median         99th %
indexed out_edges (multigraph by:)           868.61 K        1.15 μs  ±3411.98%        0.91 μs        2.50 μs
scan out_edges + filter (no multigraph)      818.72 K        1.22 μs  ±3518.01%        0.97 μs        2.90 μs
indexed lookup (multigraph by:)                3.01 K      331.95 μs    ±22.96%      320.27 μs      428.02 μs
scan all edges + filter (no multigraph)        0.68 K     1467.22 μs    ±16.28%     1418.30 μs     1914.55 μs

Comparison: 
indexed out_edges (multigraph by:)           868.61 K
scan out_edges + filter (no multigraph)      818.72 K - 1.06x slower +0.0701 μs
indexed lookup (multigraph by:)                3.01 K - 288.34x slower +330.80 μs
scan all edges + filter (no multigraph)        0.68 K - 1274.44x slower +1466.07 μs

Memory usage statistics:

Name                                       Memory usage
indexed out_edges (multigraph by:)              1.70 KB
scan out_edges + filter (no multigraph)         2.65 KB - 1.56x memory usage +0.95 KB
indexed lookup (multigraph by:)               500.35 KB - 295.14x memory usage +498.66 KB
scan all edges + filter (no multigraph)      3341.46 KB - 1971.00x memory usage +3339.77 KB

**All measurements for memory usage were the same**
  • Indexed out_edges is 2.4x faster than scan at 10k scale, index is only really useful when graphs are large with varying edge kinds.

  • Global edges queries Indexed lookup is 35x faster than scanning all edges at 10k scale (787μs vs 27,812μs) and uses 32x less memory (1MB vs 33MB).

  • The indexed lookup advantage grows with graph size — from ~4x at 1k to ~35x at 10k for global edge queries — confirming O(1)-ish index access vs O(E) scan.

Only really matters when multigraphs are actually needed with varying kinds of edges. My use case required workflow runtime memory stored as edges so this prevents dataflow traversal costs from growing with long running workflows.

This includes some minor test changes to get CI passing across the Elixir/OTP versions that broke some list comparison ordering in doc tests and charlist representations but it might be worth just dropping older version support and keeping the doctests clean.

@bitwalker let me know if you want any changes and/or need time to review - we can point repos at a temporary fork if needed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant