eliminate the workspace-module-generate helper

[vito]

Generation used to run in a nested Go helper: build a container, start a privileged nested session, ask the engine for the module's generated-context changeset, then throw that changeset away and export its two sides as loose directories, so the Dang side could re-import them, re-stage them into a fork, and re-derive the diff the engine had already computed.

PolyfillDirectoryDiff — the type the helper's output was read back into — was {before, after}, which is exactly core.Changeset. It was a hand-rolled reimplementation of the type being discarded.

The engine can now be asked directly:

generate: Changeset! {
  core.generatedContextChangeset
}

core: ModuleSource! {
  ws.moduleSource(workspaceRef)
}

Workspace.moduleSource also removes the need to compute the module's include set, since it resolves the workspace overlay rootfs itself. That takes with it the config parsing for both dagger-module.toml and legacy dagger.json, and the recursive walk over local directory dependencies.

The helper's other two entry points also fall away without their public API changing: core is now a direct Workspace.moduleSource call, and workspaceView is served from ModuleSource.contextDirectory — the same workspace-rooted layout, no longer filtered to the module's include set. workspaceView and sourceRootPath stay because they are public API consumed together outside this repo (sdk-sdk's mod-test contract is target(workspaceView, sourceRootPath)); an earlier revision of this branch dropped them on a "no in-repo consumers" test, which is the wrong test for public API.

Net: 1023 lines deleted, and per module a container build, a privileged nested session, two full directory exports, two re-imports, two withDirectory copies, and a redundant diff all stop happening.

Companion PR

dagger/dang-sdk#9 adapts generate-all to the bare Changeset! return: per-module changesets are merged with Changeset.withChangesets instead of folding workspace forks through fork.merge. The two land in lockstep; the dang-sdk side pins polyfill@fix-generate until this merges.

Why this doesn't revert #5

#5's module-path scoping guarded the fork pipeline, and this PR deletes the hazard rather than just the guard:

• The old pipeline diffed against a different base than the engine generated from: the helper's after carried staged dependency codegen while the fork's before was the pristine workspace, so each module's changeset contained its dependencies' generated files. fork.merge then overlaid full before/after snapshots (withDirectory(".", other), last-wins), which is how one module's staged before leaked into the merged base and canceled the dependency's own additions — or conflicted on apply.
• The engine changeset diffs the generated context against the same base it generated from (generatedDir.changes(from: contextDir)). Dependency codegen is either absent from both sides or staged into both — either way it cancels before it can enter the delta, which only ever carries this module's own generated files. For a Dang module the delta is literally its own dagger.json.
• Merging happens at the delta level now: Changeset.withChangesets applies each changeset as a patch onto a merged base and fails on overlap instead of silently last-winning.

generate-merge-check pins this: two sibling Go modules sharing a local dependency produce changesets that each stay inside their own subtree and whose octopus merge is the exact union of the parts.

Requires an engine change

Blocked on dagger/dagger#13745, specifically the outerEnvFile commit.

Workspace.moduleSource produces a dir-kind module source. Loading one used to reach for host.findUp(".env") unconditionally, which module code has no session for, so it blocked until the context deadline and then failed with failed to get requester session: context deadline exceeded. That is what the "temporary workaround" comment on the helper was about — the nested privileged session existed to borrow a host session, not for any module-source semantics.

Caller cwd

The engine roots a module's generated context at the workspace, but a returned changeset is applied relative to the caller cwd (changeset.Export(ctx, ".")). Generating from a subdirectory therefore wrote every path nested under that cwd a second time — running from a module directory produced the whole workspace-root path beneath it instead of editing the module's own files. The changeset is now re-rooted onto the cwd when the two differ; selecting a subdirectory is metadata only, so this does not copy anything or reintroduce staging.

Workspace.cwd exists as of v1.0.0-0, so the findUp(".", from: ".") workaround for reading the cwd is gone too.

Known gap: module discovery includes the nearest enclosing module, which may sit above the cwd. Its generated files then fall outside the cwd subtree, which cwd-relative changesets cannot express at all. generate now raises with the offending paths when that happens, so the gap is loud rather than a silent no-op; the remaining policy decision is whether discovery should exclude such modules or keep refusing them.

Testing

dagger check passes — all 18 checks, including a new generate-from-subdir-check that builds a synthetic workspace entered at a subdirectory and asserts the changeset carries cwd-relative paths (via a cwd-relative module ref, with no workspace-root-prefixed leftovers), and a new generate-merge-check covering two sibling modules sharing a local dependency. The four module-config failures previously noted here no longer reproduce on the current engine build.

Verified end to end against a real workspace: dagger generate dagger-dang-sdk:generate-all -y run from core/integration/testdata/modules/dang/counter in the dagger repo now rewrites that module's own dagger.json and touches nothing else.

🤖 Generated with Claude Code

[TomChv]

Overall this is a good deletion and the mechanism is right — PolyfillDirectoryDiff really was a hand-rolled core.Changeset, and asking the engine directly removes a container build, a privileged nested session, two exports, two re-imports and a redundant diff per module. Commits are cleanly separated (baseline migration isolated from behavior) and the generate comment explains why rather than what.

Two things I'd want resolved before merge — a silent revert of #5, and an under-declared engine requirement. Details inline; the cross-cutting points are below.

---

Breaking API change is unflagged

generate now returns Changeset! instead of PolyfillWorkspaceFork!, so callers lose .merge for composing several modules' generation with other workspace edits. The body mentions verifying dagger-dang-sdk:generate-all end to end but doesn't link the companion PR — worth linking so they land in lockstep, and worth confirming generate-all can still combine results the way it needs now that it's a bare changeset.

Modules above the cwd: is it a drop or a crash?

The body calls this a silent drop. Worth confirming — if the enclosing module's generated context doesn't span the cwd subtree, generated.after.directory(cwd) may error with "no such file or directory" rather than dropping.

Either way, silent is the worse outcome: the user runs dagger generate, sees "nothing to do", and the module never regenerates. A raise until the policy decision is made costs one line and turns a wrong answer into a legible one.

---

Not concerns

• Dropping the 141 lines of Go unit tests is correct — that logic moved into the engine and shouldn't be re-tested here.
• withDirectoryDiff survives with an e2e consumer (.dagger/modules/e2e/workspace.dang:79), so it isn't orphaned by the generate rewrite.
• The baseline commit's noise (pub removal, toJSON→JSON.encode, JSON!→Dagger.JSON!, .{x}→.{{x}}) is a mechanical language migration, cleanly isolated.

[TomChv]

This reverts the scoping fix from #5, and the body doesn't mention it.

Rooting at cwd means that at cwd == "." this returns core.generatedContextChangeset unfiltered — the whole workspace-rooted generated context. That is exactly what 1662706 (#5, three commits back) changed away from:

Root the generated context at the module's own path and stage it on an empty fork base, so each module's changeset contains only its own files.

…because cwd-rooting at the workspace root "captured the whole workspace — including any staged dependency codegen — [so] the staged dependencies leak across modules: they cancel out (a dependency's codegen goes missing) or conflict on apply ('already exists')".

The root: path argument, the empty-fork base, and the comment explaining all of it are deleted here.

The mechanism isn't identical, to be fair — #5's failure ran through the fork's before snapshot, and there's no fork base now, so before comes straight from the engine. But two sibling modules sharing a local dep still each carry that dep's generated files in their own changeset, so the overlap on merge still needs explaining away.

Could you either state why #5's failure mode no longer applies, or restore module-path scoping? Restoring it means keeping the module subtree selection and the cwd-relative prefix — which is what the withDirectoryDiff staging was providing.

[vito]

🤖 ⚔ 🤖 : I went and checked this against the engine rather than argue from the diff, and the failure mode from #5 can't reach this code — the PR deletes the machinery it lived in, not just the guard.

The #5 leak needed two ingredients, both fork-pipeline artifacts:

1. A before that disagreed with the engine's base. The helper's after carried staged dependency codegen while the fork's base was the pristine workspace, so each module's changeset contained its dependencies' generated files.
2. Snapshot merging. PolyfillWorkspaceFork.merge was before.withDirectory(".", other.before) (and the same for after) — a full-tree, last-wins overlay. That's how one module's dep-staged before leaked into the merged base and canceled the dependency's own additions, or conflicted on apply.

Neither exists now:

• generatedContextChangeset is generatedDir.changes(from: contextDir), where both sides share the module's own context as base (runGeneratedContext in core/schema/modulesource.go). Codegen runs for this module only — dependencies are loaded just to feed their schemas to it — so the delta only ever contains this module's own files. Dependency codegen is either absent from both sides or staged into both; either way it cancels before it can enter the changeset. For a Dang module the delta is literally its own dagger.json.
• Merging is delta-level now: Changeset.withChangesets converts each changeset to a patch and octopus-merges the patches onto a common base (identical before directories are deduped by content digest), failing on overlap instead of silently last-winning (core/changeset.go).

So sibling modules sharing a local dep each carry only their own files. That's pinned by generate-merge-check as of e1c2a5a: two sibling Go modules sharing a local dep produce changesets that each stay inside their own subtree, and the merge is asserted to be the exact union of the parts — which would catch both leak symptoms (a conflict, or cancellation dropping paths from the union).

The fair part of this comment: none of that was written down. dc6da86 states the invariant on generate, and the PR body now has a "Why this doesn't revert #5" section. The cwd-relative-prefix half of what withDirectoryDiff provided is what the re-root does.

[TomChv]

Nothing here covers the case #5 fixed. fixtures/generate is a single module (app) with no local dependency and no sibling, so dagger check passing says nothing about multi-module merge.

Compounding it: Asserts.added (utils.dang:38) is membership-only. Neither this check nor generateCheck asserts exhaustiveness, so extra leaked paths pass silently — including the nesting bug this check is meant to pin down, if it ever regressed partially. Asserts.paths already does exact-set matching and looks like the right tool here.

Suggested: a fixture with two sibling modules sharing a local dep, generate both, merge, assert the exact path set.

Separately, line 39 uses .moduleSource("/app"). The path users actually hit from a module directory is the cwd-relative "." — both resolve to path = "app", but only the relative form exercises moduleSourcePath's cwd-joining branch.

[vito]

Added in e1c2a5a: fixtures/generate-merge holds two sibling Go modules sharing a local dep. The check generates all three, asserts each changeset stays inside its own subtree, and asserts the octopus merge is the exact union of the parts. Asserts.paths does the exact-set comparison as suggested — against the union of the parts rather than a hardcoded list, since the Go SDK's generated file manifest changes across versions.

The subdir check now addresses the module as ., exercising moduleSourcePath's cwd-joining branch, and asserts no changed path carries the workspace-root prefix — the doubled-path signature, exhaustively this time.

[TomChv]

This under-declares the requirement. dagger/dagger#13745 is still open, and the outerEnvFile commit is a hard prerequisite. A user on exactly v1.0.0-0 doesn't get a version error — they get failed to get requester session: context deadline exceeded after a full context-deadline hang.

Bump to whichever release carries #13745, and hold the merge on it landing.

Side note: .dagger/modules/e2e/dagger-module.toml requires v1.0.0-beta.7, which by semver prerelease ordering is newer than v1.0.0-0 — so the root module is now the loosest declaration in the repo. Pre-existing, but the two should probably agree.

[vito]

Root engineVersion is now v1.0.0-beta.7 (7e811bd), matching the e2e module. Agreed on holding the merge on dagger/dagger#13745 — that's the "Blocked on" in the body; there's no released version carrying it to declare yet, so the real bump happens once that release exists.

[TomChv]

The old version handled null, "/", and the prefix trim; this drops to two cases. If ws.cwd can ever return "", the result is "", which fails the cwd == "." guard in PolyfillModuleSource.generate and the cwd == "." branch in PolyfillWorkspace.moduleSourcePath — both then silently take the wrong branch.

Fine if the schema types it String! and guarantees a leading /. Can you confirm?

[vito]

🤖 ⚔ 🤖 : Confirmed — the schema types it String! and the engine can't produce "": Workspace.cwd goes through workspaceAPIPath (core/schema/workspace.go), which maps "" and "." to "/" and everything else to a leading-slash path. The two cases here are exhaustive, so the cwd == "." guards downstream can't be skipped.

[TomChv]

The doc says "inside the workspace view", but workspaceView is removed in this PR. sourceRootPath also has no in-repo consumers — the same reasoning used to justify removing workspaceView. Drop it too, or reword the comment.

[vito]

Dropped in 9839b2a — it fails the same "no consumers" test that removed workspaceView, and path is still there if a consumer ever needs the raw value.

[vito]

Walking that back: both sourceRootPath and workspaceView are restored in 5256266. They're public API consumed together outside this repo — sdk-sdk's mod-test contract is target(workspaceView, sourceRootPath), mounting the view at /work and running the module at /work/<sourceRootPath> — so "no in-repo consumers" is the wrong test for either of them, including the workspaceView removal this PR originally shipped. workspaceView is now served from ModuleSource.contextDirectory (same workspace-rooted layout, no longer filtered to the include set), the doc on sourceRootPath is accurate again since the view exists, and a new workspace-view-check pins the pair's contract.

[kpenfound]

Confirmed this will fix the migration issue I ran into with source in a parent dir and not loaded. Details of that issue noted here dagger/dagger#13743 (comment)

[vito]

🤖 ⚔ 🤖 : On the two cross-cutting points:

Breaking API / lost .merge: the companion PR is dagger/dang-sdk#9, now linked in the body. It's worth reading for what it shows about the "lost" composition, though: the only consumer of fork-merge composition was generate-all itself, and fork.merge was a pair of withDirectory overlays — last-wins, so two modules generating the same path silently kept whichever ran last. Changeset.withChangesets is the same composition with conflicts made loud. Nothing callers could correctly do with .merge is lost; the silent part is.

Drop or crash: it's a drop, not a crash — a dir-kind module source keeps the full workspace overlay rootfs as its context (loadModuleSourceContext has no Dir case, so no include filtering applies), and the caller's cwd is a real directory in that rootfs, so .directory(cwd) resolves on both sides. The conclusion stands regardless, so it's no longer silent: as of 40bf509, generate raises with the offending paths whenever generated changes fall outside the cwd. Checking the changed paths rather than the module root also catches a generator that writes outside its own subtree.

(The #5 question is answered on the inline thread; short version: the engine diffs against the same base it generates from and merges at the delta level, so dependency codegen can't enter a sibling's changeset — now pinned by generate-merge-check.)