data-pipelines: DAG shown at both levels — collapsed modules and expanded tables

[dimitri-yatsenko]

Follow-up to #263. The restyle replaced the module-level illustration with a generated table-level diagram, which made the prose inaccurate — it still said "nodes represent Python modules" while the figure's nodes are tables.

The DAG at both levels

src/explanation/data-pipelines.md now shows the same pipeline at both levels, using dj.Diagram's collapse():

• Module level (collapsed) — pipeline-modules-collapsed.svg, one node per schema/module with its table count; each edge represents a bundle: the foreign key references between the two schemas' tables plus the Python import dependency between their modules.
• Table level (expanded) — the existing pipeline-modules.svg; nodes are tables, dashed clusters group each module, and each edge is an individual foreign key constraint.

Closes the description gap and demonstrates the collapse feature in the docs. Both figures are generated from the same schema set, adaptive light/dark.

Also in this PR

The change touches 25 files; the rest is the work that makes the two figures consistent with the surrounding docs:

• Example unification — imaging-schema.svg replaces scan-schema.svg, so the schema set behind the two new figures is the one the surrounding pages already use. relational-workflow-model.md, index.md, and rwm-pipeline.svg follow.
• Spec table — reference/specs/diagram.md § Visual Encoding: the notation and theme hex triples brought in line with _DIAGRAM_THEMES. This is the half of Diagram notation: edge-weight rule is stated incorrectly; document the modernized style #246 that docs(#246): correct the diagram edge-weight rule #247 deferred (tier palette, entity groups, direction), documenting what #1544 actually shipped rather than the hex values Diagram notation: edge-weight rule is stated incorrectly; document the modernized style #246 proposed.
• 14 regenerated notebooks — embedded diagram SVG output only, refreshed to the current notation. No re-execution: connection banners and all other cell outputs are untouched, so the check-notebook-versions guard stays green on 2.3.

Diagram generator

The three figures on these pages were committed SVGs with no recoverable source — the 14-table pipeline behind them existed nowhere in the repo, so re-verifying them against a future release meant reconstructing the schemas by reading the picture.

• scripts/pipeline_example/ — that pipeline, as four modules with one schema each, which is the module↔schema correspondence the page describes.
• scripts/gen_pipeline_diagrams.py — renders the three figures from it. --check reports any committed figure that differs and exits non-zero, so Diagram notation: edge-weight rule is stated incorrectly; document the modernized style #246's release-time step becomes a command rather than a manual audit.

Verified to reproduce the committed figures' nodes, tiers, shapes, edge weights and styles, tooltips, clusters, and labels. The committed SVGs are unchanged by this PR — the generator is added alongside them, not used to overwrite them.

Three caveats are documented in the module docstring, two of which are findings worth their own follow-up:

• The generator needs an empty database. The schema names must be unprefixed for the cluster labels to match, so on a server already holding a session schema the diagram silently picks up the foreign tables. This bit during development.
• Tooltip padding varies by pydot version —   from whichever pydot produced the committed figures, literal spaces from 4.0.1. Whole-file diff, no visual change. Nothing pins pydot.
• The collapsed lab → session edge is traversal-order dependent. Diagram._collapse_graph gives a collapsed edge the attributes of whichever foreign key in its bundle is visited first, with no aggregation. Where a bundle mixes a primary and a secondary foreign key — as lab → session does, bundling Subject → Session and User → Session — it renders solid or dashed on order alone. The committed figure is solid; the generator produces dashed. Both come out of the same renderer.

That last one has a consequence for the prose on this page, flagged for review rather than silently changed — see the review thread.

Verification

• Prose checked line by line against the generated SVGs: 14 tables → 4 collapsed nodes; reference feeds both session and imaging; the two session → imaging foreign keys are Scan → ScanQuality and ScanInfo → MotionCorrection; the two lab → session foreign keys are Subject → Session and User → Session.
• imaging-schema.svg carries all three master-part edges, including Segmentation.Roi → Fluorescence.Trace.
• diagram.md hex triples match _DIAGRAM_THEMES post-#1544.

Release-time re-verification

The figures and the spec table encode notation from datajoint-python #1534/#1544, which the latest release (v2.3.2) predates, and datajoint-python#1545 will change the collapsed figure again. That recurring re-verification is tracked separately in #266 — once per major.minor, following #210.

Closes #246.

[MilagrosMarin]

The prose/figure mismatch is genuinely fixed, and I verified the new text against the generated SVGs — 14 tables → 4 nodes is exact, the collapsed edge set matches (reference feeding both session and imaging), the session → imaging pair (Scan → ScanQuality, ScanInfo → MotionCorrection) is correct, and imaging-schema.svg has all three master-part edges including Segmentation.Roi → Fluorescence.Trace. The diagram.md hex triples match _DIAGRAM_THEMES post-#1544 verbatim.

One correction, and the reason I'm marking this rather than commenting: the lab → session pair is given as "Subject → Session and Session → User" — the second is reversed. The generated SVG has User -> Session, and Lab -> User places User in the lab schema.

It's a one-word fix, but it's load-bearing in context:

• the collapsed figure directly above has no session → lab edge, which it would need if Session → User were real;
• the same paragraph argues that collapsing every module's tables "must itself yield a DAG" — Session → User alongside lab → session would put a cycle in exactly the claim being made.

The preceding session → imaging sentence gets parent→child right, so it reads as a slip rather than a different convention. Should be Subject → Session and User → Session.

Two notes, neither blocking:

The description covers essentially two files while the PR touches 25 — 14 regenerated notebooks (SVG output only, no re-execution — I checked) plus the spec-table rewrite and the example unification. All of it is coherent work, but worth reflecting in the body since that becomes the squash message.

The figures and the new spec table encode post-#1534/#1544 notation, and the latest release (v2.3.2, 2026-07-21) predates both merges — so this documents colors the released library doesn't produce. Explicitly not introduced here: #255/#256/#259/#263 already landed figures from the unreleased renderer. But it's now at spec depth rather than just figures, so worth confirming the release-time re-verification plan along the #210 lines.

[dimitri-yatsenko]

Thanks — you were right to block on this rather than leave it as a comment, and the reasoning is the part I want to keep.

The reversed edge — fixed in 4458ae7. User → Session. I confirmed it the way you did: the table-level SVG has User -> Session and Lab -> User, which places User in the lab schema. Your two consequences are what make it more than a typo — the collapsed figure has no session → lab edge it would need, and the same paragraph asserts that collapsing every module's tables must itself yield a DAG, so the reversed form contradicts its own claim two sentences later. A reader checking the figure against the prose would have hit it immediately.

Scope in the body — done. You're right that the description covered two files while the change touches 25, and that the body becomes the squash message. Rewritten to name the example unification (imaging-schema.svg replacing scan-schema.svg, so the schema set behind both new figures is the one the surrounding pages use), the diagram.md spec-table rewrite, and the 14 regenerated notebooks — with your observation that they're SVG output only, no re-execution, so the version guard stays green on 2.3.

Release-time re-verification — recorded in #246 (#246 (comment)). Your distinction is the right one: figures from the unreleased renderer already landed in #255/#256/#259/#263, but this PR is the first to put that notation at spec depth, where a reader is entitled to treat the table as normative for the version they installed. #246 now stays open until the renderer changes ship, with two closing steps — re-verify the edge-style rows and theme hex triples against _DIAGRAM_THEMES in the released package, then regenerate the committed SVGs against it. Once per major.minor, following #210.

Ready for another look when you have a moment.

[MilagrosMarin]

All three verified. The fix is one line and correct — User → Session, no stragglers, and the sentence now reads parent→child consistently across both pairs.

The rewritten description covers the real scope, and I'd single out the notebook line ("SVG output only … banners untouched, so the guard stays green") — that's the detail a future reader needs to know they can skip re-verifying 14 files.

#246 is the right home for the release gating, and the closing condition is sharper than what I raised: two concrete steps, gated on the renderer shipping, once per major.minor rather than per patch. Keeping the figures-vs-spec-depth distinction in the issue itself is what makes it actionable later.

Approving.

[dimitri-yatsenko]

Folded the diagram generator into this PR (bce9e26), and building it turned up something on this page's prose that needs a call before merge.

The generator

scripts/pipeline_example/ defines the 14-table pipeline as four modules, one schema each; scripts/gen_pipeline_diagrams.py renders the three figures from it, with --check for CI. It reproduces the committed figures' nodes, tiers, shapes, edge weights and styles, tooltips, clusters, and labels. The committed SVGs are unchanged here — I didn't want to overwrite figures you'd verified by hand.

This was worth doing because the figures had no recoverable source. The pipeline behind them was nowhere in the repo, so the #246 release-time step was going to mean re-deriving 14 table definitions from the picture. Now it's --check.

What it exposed

The prose claim about edge weight is wrong, and it isn't wrong because of anything in this PR.

Diagram._collapse_graph builds a collapsed edge like this:

if not new_graph.has_edge(new_src, new_dest):
    new_graph.add_edge(new_src, new_dest, **data)

The collapsed edge takes the attributes of whichever foreign key in the bundle is visited first. There is no aggregation over the bundle. Two consequences:

1. Weight doesn't track bundle size. The page says "A heavier edge carries a larger bundle — below, the lab → session and session → imaging edges each bundle two separate foreign keys." Both bundle exactly two, but in the committed figure lab → session is 0.75 and session → imaging is 2. The weight is inherited from an arbitrary bundle member, so it still means what Diagram notation: edge-weight rule is stated incorrectly; document the modernized style #246 says it means — 1:1 vs multi-valued for that one member — not bundle size. session → imaging is thick because Scan → ScanQuality happens to be 1:1.

2. Solid vs dashed on a mixed bundle is order-dependent. lab → session bundles Subject → Session (primary) and User → Session (secondary). Committed renders solid; the generator renders dashed. Same renderer, same schemas — different traversal order. Stable within an environment, which is why this never surfaced.

Note this is the same edge as your blocking correction, from the other direction: you caught the prose misdescribing its direction, and the figure turns out to be unstable in how it draws it.

What I'd like your read on

Three options, and I don't want to pick unilaterally since it changes what the page claims:

1. Fix the prose here, drop the bundle-size reading of weight, and file the collapse behavior against datajoint-python. Smallest change, keeps this PR landing.
2. Fix the renderer first so a collapsed edge aggregates its bundle (thick only if every member is 1:1; dashed only if every member is secondary — which is the rule the committed figure accidentally satisfies), then regenerate. Correct, but blocks this PR on a datajoint-python release and makes the figures diverge from the released library even further.
3. Land the generator and the direction fix now, leave the weight sentence, and handle both in a follow-up against Diagram notation: edge-weight rule is stated incorrectly; document the modernized style #246.

I lean 1 — the sentence is load-bearing for a reader checking the figure, which is exactly the failure mode you caught in the direction slip, and the renderer fix shouldn't gate a docs correction.

Also two smaller things from the docstring: the generator needs an empty database (unprefixed schema names are required for the cluster labels, so it silently absorbs foreign tables if a session schema already exists — this bit me mid-development), and tooltip padding is   from whichever pydot produced the committed figures versus literal spaces from 4.0.1. Nothing pins pydot, so a byte comparison will be noisy across environments; --check compares bytes, and the docstring says to look at the pydot version first.

[MilagrosMarin]

Verified all of it. diagram.py:1387 is first-wins with no aggregation, and the figure bears out the consequence: lab → session and session → imaging both bundle exactly two foreign keys and render 0.75 and 2 respectively. So "a heavier edge carries a larger bundle" is contradicted by the figure directly beneath it. Per-FK styles confirm your reading — Scan → ScanQuality and ScanInfo → MotionCorrection are both 1:1, Subject → Session and User → Session both multi.

On the dash instability: narrower than it might read. Only lab → session has a mixed bundle. session → imaging's two members agree, and both reference edges are single-FK bundles (ScannerModel → Scan, SegmentationMethod → Segmentation). One of four edges is order-dependent — still real, still worth fixing, but the blast radius on --check is one edge.

Agreed on option 1, and I'd rule out 3 explicitly: shipping a sentence you know the figure contradicts is worse than the direction slip was, because that one was accidental. Option 2 gates a docs correction on a renderer behavior change that needs its own design call — what should a mixed bundle render as? — and widens the gap #246 already tracks.

One refinement on how to fix the prose. Dropping the bundle-size reading isn't quite enough: softening to "weight encodes cardinality" would still mislead, since on a collapsed edge it's the cardinality of one arbitrary bundle member. I'd not attribute meaning to weight or dash in the collapsed figure at all — describe it as one edge per schema pair, and keep the weight/dash semantics with the table-level figure where they're well defined. Otherwise the wrong claim becomes a vaguer claim that's still unsupported.

For the follow-up: worth including the dash instability alongside the weight semantics, since it means the committed figure isn't byte-reproducible for reasons unrelated to pydot padding — which lands on #246's "regenerate the committed SVGs" step.

The generator is a real improvement regardless. Re-deriving 14 table definitions from a picture at release time was the alternative.

[dimitri-yatsenko]

Option 1 it is, and your refinement was the right call — I'd drafted exactly the softening you warned against ("weight encodes cardinality") before rereading your point that on a collapsed edge it's the cardinality of one arbitrary member. Vaguer and still unsupported.

Prose fixed in 4be3067. The collapsed view now attributes no meaning to weight or style: one edge per schema pair, recording that a bundle exists. Weight and style semantics moved to the table-level paragraph, which now points at reference/specs/diagram.md rather than restating the rule. Your correction on blast radius is right and I've stopped implying otherwise — only lab → session has a mixed bundle; session → imaging's two members agree and both reference edges are single-FK.

On option 2 — the design call has been made, so it's no longer a blocker; it's just a separate PR. datajoint/datajoint-python#1545.

The rule: an edge between two collapsed nodes claims none of the per-foreign-key properties, and every bundle edge renders identically (solid, penwidth 2). Two points that sharpened it while implementing:

• It isn't about schemas but about collapsed nodes. A collapsed node can stand for any subset of tables — Diagram(schema).collapse() + Diagram(OneTable) leaves the remainder as a group — so keying the rule on "collapsed schema" would have been too narrow.
• An edge with only one collapsed end is not a bundle and keeps its own style. It still names a single table, so cardinality and primary-vs-secondary remain well defined there. Verified: one end expanded renders 0.75 solid for a multi-valued FK and 0.75 dashed for a secondary one, while both-ends-collapsed goes uniform.

I chose uniform styling over aggregating the bundle (thick only if every member is 1:1, dashed only if all secondary). Aggregation is well-defined and order-independent too, but it gives the same visual cue a second meaning that holds only at the collapsed level. One meaning per cue.

The order-dependence reproduces cleanly, which answers your reproducibility point: identical structure, opposite declaration order, penwidth=2 solid versus penwidth=0.75 dashed before the fix, both 2 solid after. So you're right that the committed figure isn't byte-reproducible for reasons unrelated to pydot padding — I've noted on #1545 that collapsed figures here need regenerating once it ships, which lands on #246's existing regenerate step.

Not regenerating the committed SVGs in this PR. The fix isn't merged or released, and the figures shouldn't move ahead of the renderer that docs builds against. When it does land, --check will flag pipeline-modules-collapsed.svg and the four edges become uniform — which the prose as now written already describes correctly, so that regeneration is figures-only.

[dimitri-yatsenko]

Correcting something I told you two comments up: I said the release-time re-verification was "recorded in #246, which stays open until the renderer changes ship." Wrong place.

#246's edge-weight half shipped in #247 back on Aug 13 — the commit referenced the issue without closing it, so it has been sitting open with that part done. Its remaining half is the one #247 explicitly deferred: tier palette, entity groups, and the left-to-right/no-arrowhead direction. That is the diagram.md § Visual Encoding rewrite in this PR. So #246 closes with #265, and the body now says Closes #246 rather than repeating #247's mistake of merely referencing it.

The recurring re-verification is a maintenance step, not #246's ask, and hanging it there would have kept the issue open indefinitely past the work it describes. It now lives in #266, with your two refinements built in: the pydot padding diff called out as a non-signal, and the collapsed-edge order-dependence recorded separately from it, since that one genuinely broke byte-reproducibility until datajoint-python#1545.

Worth flagging for your review of the spec table here: the hex triples #246 proposed are not what shipped. It suggested Manual #E7F3EC / #2F7D5B; #1544 adopted the brand tier colors instead, so the renderer emits #e8f0e9 / #3e7a52. The table in this PR documents the renderer, not the proposal — which is the right way round, but it does mean the table won't match #246 if you read them side by side.

[MilagrosMarin]

All three landed, and the prose fix is the version I was hoping for — "a collapsed edge records only that the bundle exists … read at the table level, not here" attributes nothing rather than attributing something vaguer, and the table-level paragraph now carries the full notation explicitly. Zero residue of the bundle-size claim, User → Session correct, body covers all 25 files, and #246/#266 are split the right way: #247 did ship the edge-weight half back on Aug 13, so closing #246 here is right, and the recurring step belongs in its own issue.

The generator is the part that will keep paying off — --check instead of re-deriving 14 table definitions from a picture is exactly the difference between a release step someone does and one they skip.

One thing to fold in before datajoint-python#1545 ships: that PR gives bundle edges penwidth 2 solid, which is what § Edge Styles here assigns to "1:1 dependency" + "primary foreign key," and the section states "line weight encodes cardinality, and only cardinality — it is binary." That becomes untrue in collapsed views, and the table has no mention of bundles. Since #1545 carries no docs, this table is the only normative home for it — a row or a sentence would do. Cheap to add here while the PR is open, or fold into #266 if you'd rather keep this one closed.

Approving — this doesn't block, since the spec only goes stale when #1545 ships.