Skip to content

fix(apps): bind volume mounts from RUNPOD_MOUNTS when the runtime did not - #626

Merged
justinwlin merged 4 commits into
mainfrom
fix/volume-path-self-configure
Oct 22, 2026
Merged

justinwlin merged 4 commits into
mainfrom
fix/volume-path-self-configure

Conversation

@justinwlin

@justinwlin justinwlin commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

Flash → SDK 2.0 fix stack

Status as of 2026-10-22. 1–5 are merged; 10 is stacked on 9. Hover a link for its status.

# PR Fixes Needs
1 #627 ✅ merged: CI e2e fixed on main —
2 #623 ✅ merged: torch pulled in by peft/accelerate no longer bundled —
3 #624 ✅ merged: requested GPU devices are pinned, not whole pools —
4 #625 ✅ merged: rp flash dev deletes its endpoints when stopped by a signal —
5 👉 #626 (this PR) ✅ merged: vol.path works on deployed workers —
6 #628 ❌ closed as a duplicate of #641, which was also closed unmerged. The API key is still copied into every endpoint —
7 — Release runpod 2.0.1 with 2–5 (release PR #594). 2.0.0rc2 (2026-10-20) predates them 2–5 merged ✅
8 runpod-workers/sdk-runtime#1 Rebuild runtime images (dev sessions run an old SDK) independent
9 runpod/runpod-plugins-official#89 Plugin docs for the fixed SDK (flash lane → SDK 2.0) 2.0.1 on PyPI
10 runpod/runpod-plugins-official#90 Plugin VolumeCache guidance: global volumes, measured numbers, golden paths 08/09 (stacked on #89) runpod/runpod-plugins-official#89

In deployed apps, NetworkVolume.path raised volume '...' is not mounted in the current execution even though the volume was attached at /runpod-volume.

Cause

vol.path reads bindings that the worker runtime installs by calling _configure_mounts before user code (sdk-runtime mounts.py, added 2026-09-29). The published runtime images (runpod/queue:py3.12-latest and the rest) were pushed on 2026-09-22 and don't contain that hook. docker run against the image shows no mounts.py, and its queue/worker.py never calls configure_mounts. The bindings are never installed.

Change

Deployed workers run the client's vendored runpod (unless RUNPOD_PACKAGE_SPEC overrides the package), so the first vol.path lookup now installs the bindings from the RUNPOD_MOUNTS provisioning env itself when the runtime hasn't. A runtime that configured mounts still takes precedence. Invalid JSON raises VolumeError.

This removes the SDK's dependence on the image for vol.path. The images still need a rebuild for dev sessions, which run on the image's own runpod.

Testing

CPU queue writing through vol.path / "probe.txt" main runpod-flash 1.21 This PR
Deployed on today's runtime images VolumeError: volume '...' is not mounted in the current execution no vol.path; apps used the literal /runpod-volume {'path': '/runpod-volume', 'read': 'ok'}
How we tested and compared
  • Root cause. docker run runpod/queue:py3.12-latest shows no mounts.py, and its queue/worker.py never calls configure_mounts. The images predate the hook.
  • Live, before and after. The same app, with a network volume attached, was deployed from main (raised the error above) and from this branch (returned the path). The app and volume were deleted afterward.
  • Unit tests cover binding from RUNPOD_MOUNTS when the runtime did not, the runtime's configuration winning, and invalid JSON. pytest tests: 1482 passed, coverage 91.66%.
Review (Codex, two rounds)
Finding Resolution
The flag was set before the bindings were published, so a concurrent first lookup could read empty bindings Fixed in dee2d20 by publishing the bindings first.
"Deployed workers run the client's vendored runpod" has an exception: RUNPOD_PACKAGE_SPEC skips the overlay Description corrected below.
No concurrent-lookup test, and no automated old-image worker test Open. Old-image behavior was checked live.

Comment thread tests/test_apps/test_volume.py Fixed
Comment thread tests/test_apps/test_volume.py Fixed
Comment thread tests/test_apps/test_volume.py Fixed
Comment thread tests/test_apps/test_volume.py Fixed
… not

vol.path reads bindings the worker runtime installs through
_configure_mounts before user code. Runtime images published before
that hook never call it, so on them every deployed NetworkVolume
attached but vol.path raised "not mounted in the current execution".

Deployed workers run the client's vendored runpod package, so the
first vol.path lookup now installs the bindings from the RUNPOD_MOUNTS
provisioning env itself when the runtime has not. A runtime that
configured mounts still takes precedence.
@justinwlin
justinwlin force-pushed the fix/volume-path-self-configure branch from 317e3fd to dee2d20 Compare October 20, 2026 15:48
@justinwlin
justinwlin marked this pull request as ready for review October 20, 2026 15:48
Comment thread tests/test_apps/test_volume.py Fixed
Comment thread tests/test_apps/test_volume.py Fixed

@KAJdev KAJdev left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

Drops the second import of runpod.apps.volume that CodeQL flagged. The
globals are module-level, so they are patched by name, not on NetworkVolume.
@justinwlin
justinwlin merged commit fc4a6c7 into main Oct 22, 2026
10 checks passed
@justinwlin
justinwlin deleted the fix/volume-path-self-configure branch October 22, 2026 16:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants