Skip to content

fix(testing): fall back to the MaxText tokenizer in the logit checker's golden mode - #6025

Closed
ryanliuCienet wants to merge 1 commit into
mainfrom
ryan/fix-logit-checker-sentencepiece-fallback
Closed

ryanliuCienet wants to merge 1 commit into
mainfrom
ryan/fix-logit-checker-sentencepiece-fallback

Conversation

@ryanliuCienet

@ryanliuCienet ryanliuCienet commented Oct 29, 2026 •

Copy link
Copy Markdown
Collaborator

Description

d81f8fab1 ("add Qwen3 VL tests, update logit checker, and golden logit generator", #4330) moved the AutoTokenizer.from_pretrained call in tests/utils/forward_pass_logit_checker.py out of the --run_hf_model branch so the multimodal golden-logits path could use it. Since then every golden-logits run must load an HF tokenizer, although that mode only uses it to print the top-k tokens. 1fb4be117 ("fix(testing): resolve tokenizer loading error and model_prefix scope bug in logit checker", #4438) later turned the load into a loop over --hf_model_path, tokenizer_path and HF_IDS[model_name], which repaired the models that have an HF_IDS entry, but models whose tokenizer_path is a SentencePiece file and that are absent from HF_IDS (gemma-2b, gemma-7b, llama2-7b) still cannot satisfy any candidate and die before comparing anything:

OSError: It looks like the config file at '.../tokenizers/tokenizer.gemma' is not a valid JSON file.

In tests/end_to_end/tpu/{gemma/2b,llama2/7b} this stayed hidden behind earlier failures in the same scripts, most recently the generate_param_only_checkpoint PRNG mismatch (b/576849994, fixed in #6026), which runs first; this change is only reachable once that one is in.

Changes

tests/utils/forward_pass_logit_checker.py only:

  • When every AutoTokenizer candidate fails and --run_hf_model is not set, build the tokenizer with MaxText's own build_tokenizer (the same tokenizer_path/tokenizer_type the model was trained with) and log the fallback.
  • Golden-logits mode only calls decode on the tokenizer, so the top-k printout now decodes a one-element list (decode([tok_id])), which HF tokenizers (int | list[int]) and MaxText's own tokenizers both accept, and the llama3.1/mixtral pad_token = eos_token assignment is guarded with hasattr, since MaxText tokenizers have no eos_token.

--run_hf_model is excluded on purpose: it tokenizes the prompts for both models and needs a real HF tokenizer, and there --hf_model_path always loads first. Adding HF_IDS entries instead was rejected: pyconfig uses that dict to default tokenizer_path, so it would switch every tokenizer_path-less command for those models to a gated HF download.

BUGS: b/579251787

Related: b/576849994 (the PRNG failure that masked this one; fixed in #6026).

Tests

Lint and tokenizer check

pylint 10.00/10, pyink --check clean. build_tokenizer(...).decode([id]) returns the expected text for both SentencePiece files shipped in the test image (tokenizer.gemma -> ' world', tokenizer.llama2 -> 'world'), and neither has an eos_token attribute, hence the guard.

E2E on TPU v5p-8

maxtext_end_to_end DAG (dev Composer env), cluster auto-v5p-8-bodaborg, image maxtext_jax_stable:2026-10-27 + this branch and #6026 (maxtext_testing_liuryan:testing_all-6ef6677e1), tasks maxtext-stable-gemma-2b-v5p-8 (test_gemma.sh) and maxtext-stable-llama2-7b-v5p-8 (test_llama2_7b.sh).

Image gemma-2b golden check llama2-7b golden check
Current prod image (before #6026 landed) OSError: ... tokenizer.gemma is not a valid JSON file, EXIT_CODE=1 not reached (PRNG mismatch first)
Prod image + #6026 only same OSError, EXIT_CODE=1 (this is the next failure once #6026 is in) not run
This PR + #6026 Falling back to the MaxText TokenizerType.SENTENCEPIECE tokenizer at .../tokenizer.gemma; max KL 3.67e-03 / 3.00e-03 / 1.98e-03 (bound 0.015); whole script passes, EXIT_CODE=0 same fallback for tokenizer.llama2; max KL 6.55e-05 on all three prompts; check passes

The llama2-7b script then fails further down in llama_mistral_mixtral_orbax_to_hf (AttributeError: No attribute 'params' in State, the standalone converter was never ported to the NNX state); that is unrelated to the checker and fixed in #6035 (b/579252268).

Logs (Cloud Logging, project cloud-tpu-multipod-dev):

Checklist

Before submitting this PR, please make sure (put X in square brackets):

  • I have performed a self-review of my code. For an optional AI review, add the gemini-review label.
  • I have necessary comments in my code, particularly in hard-to-understand areas.
  • I have run end-to-end tests and provided workload links above if applicable.
  • I have made or will make corresponding changes to the doc if needed, including adding new documentation pages to the relevant Table of Contents (toctree directive) as explained in our documentation.

@gemini-code-assist

Copy link
Copy Markdown

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request addresses an issue where certain models, such as gemma-2b and llama2-7b, failed during golden-logit generation because their tokenizers could not be loaded via AutoTokenizer. By providing a fallback to the native MaxText tokenizer, the logit checker can now successfully proceed with printing top-k tokens for these models, while maintaining strict requirements for the --run_hf_model path.

Highlights

  • Tokenizer Fallback Mechanism: Introduced a fallback to the native MaxText tokenizer in the logit checker's golden mode when AutoTokenizer fails to load, specifically for models using SentencePiece or tiktoken files.
  • Decode-Only Adapter: Implemented a _DecodeOnlyTokenizer adapter class to ensure compatibility with the existing top-k token logging logic without requiring a full HuggingFace tokenizer instance.
New Features

🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

@github-actions github-actions Bot added the skip-ci-tests Add this label to the PR to skip running CI tests label Oct 29, 2026
@github-actions

Copy link
Copy Markdown
Contributor

🤖 Hi @ryanliuCienet, I'm running an automatic MaxText review of this pull request. You can track my progress in the logs. To request another review later, add the gemini-review label or comment @gemini-cli /review.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📋 Review Summary

Verdict: Spec met, but the implementation includes unnecessary AI slop that should be simplified.

Top issues:

  1. 🟠 [Complexity] _DecodeOnlyTokenizer is a single-use wrapper that papers over a trivial interface mismatch. → Delete the wrapper class, pass [tok_id] to decode in get_top_k_tokens_scores, and guard the pad_token assignment on line 420 with if hasattr(tokenizer, 'eos_token'):.
Inferred spec & validation
  • Sources: PR description, code before/after.
  • Problem: AutoTokenizer fails to load the tokenizer for models like gemma-2b that use a SentencePiece file and lack HF_IDS fallback, breaking golden-logits test runs.
Spec item Implementation Validating test
R1 (stated): In golden-logits mode (--run_hf_model is False) and when AutoTokenizer fails, fall back to the MaxText tokenizer via build_tokenizer. ✅ tests/end_to_end/tpu/{gemma/2b,llama2/7b} ✅ (E2E run validates testing script fallback)
C1 (inferred): Do not fallback if --run_hf_model is set, since it requires a real HF tokenizer. ✅ preserved tests/end_to_end/tpu/{gemma/2b,llama2/7b} ✅

@codecov

codecov Bot commented Oct 29, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@ryanliuCienet ryanliuCienet changed the title Fall back to the MaxText tokenizer in the logit checker's golden mode fix(testing): fall back to the MaxText tokenizer in the logit checker's golden mode Oct 29, 2026
# Description

Root cause: #d81f8fab1 ("add Qwen3 VL tests, update logit checker, and golden
logit generator") moved the `AutoTokenizer.from_pretrained` call in
`forward_pass_logit_checker` out of the `--run_hf_model` branch so the
multimodal golden-logits path could use it. Since then every golden-logits run
must load an HF tokenizer, although that mode only uses it to print the top-k
tokens. #1fb4be117 later turned the load into a loop over `--hf_model_path`,
`tokenizer_path` and `HF_IDS[model_name]`, which repaired the models that have
an `HF_IDS` entry, but models whose `tokenizer_path` is a SentencePiece file
and that are absent from `HF_IDS` (gemma-2b, gemma-7b, llama2-7b) still cannot
satisfy any candidate and die before comparing anything:

    OSError: It looks like the config file at '.../tokenizers/tokenizer.gemma' is not a valid JSON file.

In `tests/end_to_end/tpu/{gemma/2b,llama2/7b}` this stayed hidden behind
earlier failures in the same scripts, most recently the
`generate_param_only_checkpoint` PRNG mismatch (b/576849994, fixed in
#<PRNG_PR_NUMBER>), which runs first; this change is only reachable once
that one is in.

When every `AutoTokenizer` candidate fails and `--run_hf_model` is not set,
build the tokenizer with MaxText's own `build_tokenizer` (the same
`tokenizer_path`/`tokenizer_type` the model was trained with). Golden-logits
mode only calls `decode` on it, so the top-k printout now decodes a one-element
list, which HF and MaxText tokenizers both accept, and the llama3.1/mixtral
`pad_token` assignment is guarded with `hasattr`, since MaxText tokenizers have
no `eos_token`. `--run_hf_model` is excluded on purpose: it tokenizes the
prompts for both models and needs a real HF tokenizer, and there
`--hf_model_path` always loads first. Adding `HF_IDS` entries instead was
rejected: `pyconfig` uses that dict to default `tokenizer_path`, so it would
switch every `tokenizer_path`-less command for those models to a gated HF
download.

# Tests

- `pylint` 10.00/10, `pyink --check` clean.
- `build_tokenizer(...).decode([id])` returns the expected text for both
  SentencePiece files in the test image (`tokenizer.gemma`, `tokenizer.llama2`);
  HF `PreTrainedTokenizerBase.decode` accepts `int | list[int]`.
- E2E on v5p-8 via the `maxtext_end_to_end` DAG with a test image built from
  this branch: pending (fill in workload links for gemma-2b and llama2-7b).

# Checklist

Before submitting this PR, please make sure (put X in square brackets):
- [X] I have performed a self-review of my code. For an optional AI review, add the `gemini-review` label.
- [X] I have necessary comments in my code, particularly in hard-to-understand areas.
- [X] I have run end-to-end tests and provided workload links above if applicable.
- [X] I have made or will make corresponding changes to the doc if needed, including adding new documentation pages to the relevant Table of Contents (toctree directive) as explained in [our documentation](https://maxtext.readthedocs.io/en/latest/development.html#adding-new-documentation-files).
@ryanliuCienet
ryanliuCienet force-pushed the ryan/fix-logit-checker-sentencepiece-fallback branch from 7cbc616 to a690405 Compare October 29, 2026 09:12
@ryanliuCienet
ryanliuCienet deleted the ryan/fix-logit-checker-sentencepiece-fallback branch November 2, 2026 04:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

skip-ci-tests Add this label to the PR to skip running CI tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant