Repository navigation
# Description - #4138
Merged
Merged
# Description#4138
Conversation
copybara-service
Bot
requested review from
A9isha,
NicoGrande,
NuojCheng,
RissyRan,
SurbhiJainUSC,
abhinavclemson,
aireenmei,
bvandermoon,
darisoy,
dipannita08,
gagika,
gobbleturk,
hengtaoguo,
igorts-git,
jesselu-google,
jiangjy1982,
khatwanimohit,
richjames0,
shralex,
suexu1025 and
vipannalla
as code owners
June 10, 2026 21:03
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
copybara-service
Bot
force-pushed
the
test_930062734
branch
from
June 12, 2026 18:26
19c0ea4 to
35c4347
Compare
copybara-service
Bot
force-pushed
the
test_930062734
branch
2 times, most recently
from
June 12, 2026 21:35
6cce657 to
1e578e3
Compare
copybara-service
Bot
force-pushed
the
test_930062734
branch
2 times, most recently
from
June 15, 2026 21:27
f8bb854 to
4aedab8
Compare
copybara-service
Bot
force-pushed
the
test_930062734
branch
12 times, most recently
from
June 23, 2026 18:42
2f05415 to
c80d8ce
Compare
copybara-service
Bot
requested review from
jshin1394 and
liudangyi
as code owners
June 23, 2026 18:42
copybara-service
Bot
force-pushed
the
test_930062734
branch
7 times, most recently
from
June 23, 2026 23:50
7fba3bb to
646d3a5
Compare
This PR fixes a `ValueError: can only convert an array of size 1 to a Python scalar` that occurs in `RemoteIteratorWrapper` during state save/restore on multi-device topologies (size > 1). It also adds validation to ensure colocated Python data input is only used with Pathways (single controller) enabled, and replaces incorrect usages of `jax.local_devices()` with `global_mesh.devices`. # Root Cause 1. **ValueError in save/restore**: `RemoteIteratorWrapper.save_state` and `restore_state` were attempting to shape the step value array using `self.dummy_array.shape` and shard it across devices. On topologies with more than 1 device, this resulted in a partitioned array. When this partitioned array was passed to the local iterator, attempting to unpack it to a Python scalar (e.g. via `.item()` or direct conversion) failed because JAX does not allow converting partitioned arrays of size > 1 to Python scalars. 2. **Incorrect Device Resolution**: `RemoteIteratorWrapper` was using `jax.local_devices()` to determine CPU/TPU devices. Under Pathways (single-controller), all devices in the cluster are virtualized as local to the JAX client, meaning `jax.local_devices()` returns all devices (including inactive ones during elastic scale-down), which is incorrect for sharding and shape calculations. 3. **Missing Validation**: `colocated_python_data_input` relies on Pathways single-controller mode, but there was no validation enforcing this constraint, which could lead to cryptic failures if misconfigured. # Solution 1. **Replicated Scalar for Step**: Modified `RemoteIteratorWrapper.save_state` and `restore_state` in `multihost_dataloading.py` to pass the training step as a replicated 0D JAX scalar array (global shape `()`) with replicated sharding (`NamedSharding` with `PartitionSpec()`). This ensures the array has size 1 on all devices and can be safely converted to a Python scalar by the local iterator. 2. **Use Global Mesh Devices**: Replaced `jax.local_devices()` with `global_mesh.devices` (via `tuple(global_mesh.devices.flat)`) in `RemoteIteratorWrapper.__init__` to ensure it only uses the active devices defined by the global mesh, handling elastic scaling correctly. 3. **Config Validation**: Added a check in `types.py` to raise a `ValueError` if `colocated_python_data_input` is enabled but `enable_single_controller` is false. # Tests Added new unit tests in `third_party/py/maxtext/tests/unit/multihost_dataloading_test.py` to verify the fixes: 1. `test_remote_iterator_wrapper_save_state`: Parameterized over different mesh shapes (1, 2, and 4 devices). Instantiates `RemoteIteratorWrapper` and verifies that calling `save_state` successfully writes the state to a JSON file without raising `ValueError`. 2. `test_remote_iterator_wrapper_restore_state`: Parameterized over different mesh shapes. Verifies that `restore_state` successfully restores the state from a JSON file and resumes iteration correctly. These tests are configured to run with `XLA_FLAGS="--xla_force_host_platform_device_count=4"` via the `BUILD` target to simulate multi-device environments. # Checklist Before submitting this PR, please make sure (put X in square brackets): - [X] I have performed a self-review of my code. For an optional AI review, add the `gemini-review` label. - [X] I have necessary comments in my code, particularly in hard-to-understand areas. - [X] I have run end-to-end tests tests and provided workload links above if applicable. - [X] I have made or will make corresponding changes to the doc if needed. PiperOrigin-RevId: 936996365
copybara-service
Bot
force-pushed
the
test_930062734
branch
from
June 24, 2026 00:50
646d3a5 to
4f33eba
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR fixes a
ValueError: can only convert an array of size 1 to a Python scalarthat occurs inRemoteIteratorWrapperduring state save/restore on multi-device topologies (size > 1). It also adds validation to ensure colocated Python data input is only used with Pathways (single controller) enabled, and replaces incorrect usages ofjax.local_devices()withglobal_mesh.devices.Root Cause
RemoteIteratorWrapper.save_stateandrestore_statewere attempting to shape the step value array usingself.dummy_array.shapeand shard it across devices. On topologies with more than 1 device, this resulted in a partitioned array. When this partitioned array was passed to the local iterator, attempting to unpack it to a Python scalar (e.g. via.item()or direct conversion) failed because JAX does not allow converting partitioned arrays of size > 1 to Python scalars.RemoteIteratorWrapperwas usingjax.local_devices()to determine CPU/TPU devices. Under Pathways (single-controller), all devices in the cluster are virtualized as local to the JAX client, meaningjax.local_devices()returns all devices (including inactive ones during elastic scale-down), which is incorrect for sharding and shape calculations.colocated_python_data_inputrelies on Pathways single-controller mode, but there was no validation enforcing this constraint, which could lead to cryptic failures if misconfigured.Solution
RemoteIteratorWrapper.save_stateandrestore_stateinmultihost_dataloading.pyto pass the training step as a replicated 0D JAX scalar array (global shape()) with replicated sharding (NamedShardingwithPartitionSpec()). This ensures the array has size 1 on all devices and can be safely converted to a Python scalar by the local iterator.jax.local_devices()withglobal_mesh.devices(viatuple(global_mesh.devices.flat)) inRemoteIteratorWrapper.__init__to ensure it only uses the active devices defined by the global mesh, handling elastic scaling correctly.types.pyto raise aValueErrorifcolocated_python_data_inputis enabled butenable_single_controlleris false.Tests
Added new unit tests in
third_party/py/maxtext/tests/unit/multihost_dataloading_test.pyto verify the fixes:test_remote_iterator_wrapper_save_state: Parameterized over different mesh shapes (1, 2, and 4 devices). InstantiatesRemoteIteratorWrapperand verifies that callingsave_statesuccessfully writes the state to a JSON file without raisingValueError.test_remote_iterator_wrapper_restore_state: Parameterized over different mesh shapes. Verifies thatrestore_statesuccessfully restores the state from a JSON file and resumes iteration correctly.These tests are configured to run with
XLA_FLAGS="--xla_force_host_platform_device_count=4"via theBUILDtarget to simulate multi-device environments.Checklist
Before submitting this PR, please make sure (put X in square brackets):
gemini-reviewlabel.