Skip to content

Fix: Gemma 3 & 4 Base Model and Flax Linen Decoding - #4066

Merged
copybara-service[bot] merged 1 commit into
mainfrom
jackyf/gemma3_4-base-decoding-fixes
Jul 2, 2026
Merged

copybara-service[bot] merged 1 commit into
mainfrom
jackyf/gemma3_4-base-decoding-fixes

Conversation

@RexBearIU

@RexBearIU RexBearIU commented Jun 4, 2026 •

Copy link
Copy Markdown
Collaborator

Description

This PR introduces architectural configurations for Gemma3 and Gemma4, and implements critical decoding layer KV-cache propagation fixes for JAX/NNX
(nnx_decoders.py) and Flax Linen (decoders.py) pipelines.

Before this change, Flax Linen scanned blocks (_apply_gemma3_scanned_blocks and _apply_gemma4_scanned_blocks) lacked mechanisms to propagate
intermediate KV-caches across scanned layers under nn.scan, causing end-to-end decodes to fail.

Key Changes:
- Flax Linen Decoder (src/maxtext/layers/decoders.py): Added kv_caches and attention_metadata propagation across Gemma3/4 scanned blocks
via a stack/unstack mapping over nn.scan.
- JAX/NNX Decoder (src/maxtext/layers/nnx_decoders.py): Unified dynamic KV-cache carry updates within scanned block execution.
- Model Scaffolding (gemma3.py, gemma4.py): Added kv_cache routing through scannable blocks and layer inputs.

Tests

  • Validated via end-to-end decoding with Gemma3-4B on an 8-device TPU mesh.
  • Run unit tests:
python3 -m pytest tests/post_training/unit/lora_utils_test.py

Checklist

Before submitting this PR, please make sure (put X in square brackets):

  • I have performed a self-review of my code. For an optional AI review, add the gemini-review label.
  • I have necessary comments in my code, particularly in hard-to-understand areas.
  • I have run end-to-end tests tests and provided workload links above if applicable.
  • I have made or will make corresponding changes to the doc if needed, including adding new documentation pages to the relevant Table of Contents (toctree directive) as explained in our documentation.

@codecov

codecov Bot commented Jun 4, 2026 •

Copy link
Copy Markdown

@RexBearIU
RexBearIU force-pushed the jackyf/gemma3_4-base-decoding-fixes branch 3 times, most recently from 355c904 to 59ca028 Compare June 12, 2026 08:15
@RexBearIU
RexBearIU force-pushed the jackyf/gemma3_4-base-decoding-fixes branch 3 times, most recently from fca7f7c to 338f2fc Compare June 18, 2026 09:09
@RexBearIU
RexBearIU force-pushed the jackyf/gemma3_4-base-decoding-fixes branch 6 times, most recently from 8bf05e7 to d96b8a6 Compare June 30, 2026 08:11
for i in range(scan_length):
start_idx = i * block_len
for offset, updated_item in enumerate(unstacked[i]):
kv_caches[start_idx + offset] = updated_item

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is kv_caches a python list, not jax array? jax array is not mutable. If so, use if isinstance(kv_caches) List to make sure kv_caches is a list

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, good catch. kv_caches is indeed a Python list containing JAX PyTrees. Since JAX arrays/PyTrees are immutable but we mutate the outer list structure in update_kv_caches_after_scan, I've added explicit isinstance(kv_caches, list) checks and validation tests to ensure it behaves correctly. I just pushed the fixes.

@RexBearIU
RexBearIU force-pushed the jackyf/gemma3_4-base-decoding-fixes branch from d96b8a6 to dbcc192 Compare July 1, 2026 08:10
@RexBearIU
RexBearIU requested a review from xibinliu as a code owner July 1, 2026 08:10
@RexBearIU
RexBearIU force-pushed the jackyf/gemma3_4-base-decoding-fixes branch from dbcc192 to 08239eb Compare July 2, 2026 03:04
@copybara-service
copybara-service Bot merged commit 4a932fb into main Jul 2, 2026
52 of 53 checks passed
@copybara-service
copybara-service Bot deleted the jackyf/gemma3_4-base-decoding-fixes branch July 2, 2026 06:38
@aireenmei aireenmei mentioned this pull request Jul 29, 2026
4 tasks done
chiajunglien added a commit that referenced this pull request Nov 4, 2026
The Llama3.1-70B vLLM decodes in the RL and SFT E2E tests printed
gibberish for two independent reasons.

1. LlamaDecoderLayer returned (layer_output, None) whenever scan_layers
   was set. When vLLM serves a scanned model, the decoder statically
   unrolls the scanned layers, passes each layer its own kv_cache and
   keeps the cache each layer returns, so every KV cache update was
   dropped. The first token, computed in the attention call that also
   writes the prompt's keys and values, was right; every later token
   attended to a stale cache ("Big the the the ..."). This broke the
   step-3 decodes, which load the scanned post-training checkpoints.
   Return the KV cache under scan_layers as well, as Qwen3DecoderLayer
   already does and as #4066 did for Gemma3. Outside vLLM, attention
   returns the kv_cache it was given (None), so nothing else changes.

2. vllm_decode samples with decode_sampling_temperature (1.0 by
   default). The tests decode the base meta-llama/Llama-3.1-70B
   checkpoint with the Instruct chat template, which gives incoherent
   text at temperature 1.0. Decode greedily in the Llama3.1-70B RL and
   SFT scripts, as the vllm_decode usage example does.

Add a unit test that a scanned decoder layer returns the KV cache its
attention returns.

Bug: b/558524540
Bug: b/558524529
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants