Repository navigation
fix(testing): fall back to the MaxText tokenizer in the logit checker's golden mode - #6025
ryanliuCienet wants to merge 1 commit into
Conversation
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request addresses an issue where certain models, such as gemma-2b and llama2-7b, failed during golden-logit generation because their tokenizers could not be loaded via AutoTokenizer. By providing a fallback to the native MaxText tokenizer, the logit checker can now successfully proceed with printing top-k tokens for these models, while maintaining strict requirements for the --run_hf_model path. Highlights
New Features🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
|
🤖 Hi @ryanliuCienet, I'm running an automatic MaxText review of this pull request. You can track my progress in the logs. To request another review later, add the |
There was a problem hiding this comment.
📋 Review Summary
Verdict: Spec met, but the implementation includes unnecessary AI slop that should be simplified.
Top issues:
- 🟠 [Complexity]
_DecodeOnlyTokenizeris a single-use wrapper that papers over a trivial interface mismatch. → Delete the wrapper class, pass[tok_id]todecodeinget_top_k_tokens_scores, and guard thepad_tokenassignment on line 420 withif hasattr(tokenizer, 'eos_token'):.
Inferred spec & validation
- Sources: PR description, code before/after.
- Problem:
AutoTokenizerfails to load the tokenizer for models likegemma-2bthat use a SentencePiece file and lackHF_IDSfallback, breaking golden-logits test runs.
| Spec item | Implementation | Validating test |
|---|---|---|
R1 (stated): In golden-logits mode (--run_hf_model is False) and when AutoTokenizer fails, fall back to the MaxText tokenizer via build_tokenizer. |
✅ | tests/end_to_end/tpu/{gemma/2b,llama2/7b} ✅ (E2E run validates testing script fallback) |
C1 (inferred): Do not fallback if --run_hf_model is set, since it requires a real HF tokenizer. |
✅ preserved | tests/end_to_end/tpu/{gemma/2b,llama2/7b} ✅ |
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
# Description
Root cause: #d81f8fab1 ("add Qwen3 VL tests, update logit checker, and golden
logit generator") moved the `AutoTokenizer.from_pretrained` call in
`forward_pass_logit_checker` out of the `--run_hf_model` branch so the
multimodal golden-logits path could use it. Since then every golden-logits run
must load an HF tokenizer, although that mode only uses it to print the top-k
tokens. #1fb4be117 later turned the load into a loop over `--hf_model_path`,
`tokenizer_path` and `HF_IDS[model_name]`, which repaired the models that have
an `HF_IDS` entry, but models whose `tokenizer_path` is a SentencePiece file
and that are absent from `HF_IDS` (gemma-2b, gemma-7b, llama2-7b) still cannot
satisfy any candidate and die before comparing anything:
OSError: It looks like the config file at '.../tokenizers/tokenizer.gemma' is not a valid JSON file.
In `tests/end_to_end/tpu/{gemma/2b,llama2/7b}` this stayed hidden behind
earlier failures in the same scripts, most recently the
`generate_param_only_checkpoint` PRNG mismatch (b/576849994, fixed in
#<PRNG_PR_NUMBER>), which runs first; this change is only reachable once
that one is in.
When every `AutoTokenizer` candidate fails and `--run_hf_model` is not set,
build the tokenizer with MaxText's own `build_tokenizer` (the same
`tokenizer_path`/`tokenizer_type` the model was trained with). Golden-logits
mode only calls `decode` on it, so the top-k printout now decodes a one-element
list, which HF and MaxText tokenizers both accept, and the llama3.1/mixtral
`pad_token` assignment is guarded with `hasattr`, since MaxText tokenizers have
no `eos_token`. `--run_hf_model` is excluded on purpose: it tokenizes the
prompts for both models and needs a real HF tokenizer, and there
`--hf_model_path` always loads first. Adding `HF_IDS` entries instead was
rejected: `pyconfig` uses that dict to default `tokenizer_path`, so it would
switch every `tokenizer_path`-less command for those models to a gated HF
download.
# Tests
- `pylint` 10.00/10, `pyink --check` clean.
- `build_tokenizer(...).decode([id])` returns the expected text for both
SentencePiece files in the test image (`tokenizer.gemma`, `tokenizer.llama2`);
HF `PreTrainedTokenizerBase.decode` accepts `int | list[int]`.
- E2E on v5p-8 via the `maxtext_end_to_end` DAG with a test image built from
this branch: pending (fill in workload links for gemma-2b and llama2-7b).
# Checklist
Before submitting this PR, please make sure (put X in square brackets):
- [X] I have performed a self-review of my code. For an optional AI review, add the `gemini-review` label.
- [X] I have necessary comments in my code, particularly in hard-to-understand areas.
- [X] I have run end-to-end tests and provided workload links above if applicable.
- [X] I have made or will make corresponding changes to the doc if needed, including adding new documentation pages to the relevant Table of Contents (toctree directive) as explained in [our documentation](https://maxtext.readthedocs.io/en/latest/development.html#adding-new-documentation-files).
7cbc616 to
a690405
Compare
Description
d81f8fab1 ("add Qwen3 VL tests, update logit checker, and golden logit generator", #4330) moved the
AutoTokenizer.from_pretrainedcall intests/utils/forward_pass_logit_checker.pyout of the--run_hf_modelbranch so the multimodal golden-logits path could use it. Since then every golden-logits run must load an HF tokenizer, although that mode only uses it to print the top-k tokens. 1fb4be117 ("fix(testing): resolve tokenizer loading error and model_prefix scope bug in logit checker", #4438) later turned the load into a loop over--hf_model_path,tokenizer_pathandHF_IDS[model_name], which repaired the models that have anHF_IDSentry, but models whosetokenizer_pathis a SentencePiece file and that are absent fromHF_IDS(gemma-2b, gemma-7b, llama2-7b) still cannot satisfy any candidate and die before comparing anything:In
tests/end_to_end/tpu/{gemma/2b,llama2/7b}this stayed hidden behind earlier failures in the same scripts, most recently thegenerate_param_only_checkpointPRNG mismatch (b/576849994, fixed in #6026), which runs first; this change is only reachable once that one is in.Changes
tests/utils/forward_pass_logit_checker.pyonly:AutoTokenizercandidate fails and--run_hf_modelis not set, build the tokenizer with MaxText's ownbuild_tokenizer(the sametokenizer_path/tokenizer_typethe model was trained with) and log the fallback.decodeon the tokenizer, so the top-k printout now decodes a one-element list (decode([tok_id])), which HF tokenizers (int | list[int]) and MaxText's own tokenizers both accept, and the llama3.1/mixtralpad_token = eos_tokenassignment is guarded withhasattr, since MaxText tokenizers have noeos_token.--run_hf_modelis excluded on purpose: it tokenizes the prompts for both models and needs a real HF tokenizer, and there--hf_model_pathalways loads first. AddingHF_IDSentries instead was rejected:pyconfiguses that dict to defaulttokenizer_path, so it would switch everytokenizer_path-less command for those models to a gated HF download.BUGS: b/579251787
Related: b/576849994 (the PRNG failure that masked this one; fixed in #6026).
Tests
Lint and tokenizer check
pylint10.00/10,pyink --checkclean.build_tokenizer(...).decode([id])returns the expected text for both SentencePiece files shipped in the test image (tokenizer.gemma->' world',tokenizer.llama2->'world'), and neither has aneos_tokenattribute, hence the guard.E2E on TPU v5p-8
maxtext_end_to_endDAG (dev Composer env), clusterauto-v5p-8-bodaborg, imagemaxtext_jax_stable:2026-10-27+ this branch and #6026 (maxtext_testing_liuryan:testing_all-6ef6677e1), tasksmaxtext-stable-gemma-2b-v5p-8(test_gemma.sh) andmaxtext-stable-llama2-7b-v5p-8(test_llama2_7b.sh).OSError: ... tokenizer.gemma is not a valid JSON file,EXIT_CODE=1OSError,EXIT_CODE=1(this is the next failure once #6026 is in)Falling back to the MaxText TokenizerType.SENTENCEPIECE tokenizer at .../tokenizer.gemma; max KL 3.67e-03 / 3.00e-03 / 1.98e-03 (bound 0.015); whole script passes,EXIT_CODE=0tokenizer.llama2; max KL 6.55e-05 on all three prompts; check passesThe llama2-7b script then fails further down in
llama_mistral_mixtral_orbax_to_hf(AttributeError: No attribute 'params' in State, the standalone converter was never ported to the NNX state); that is unrelated to the checker and fixed in #6035 (b/579252268).Logs (Cloud Logging, project
cloud-tpu-multipod-dev):Checklist
Before submitting this PR, please make sure (put X in square brackets):
gemini-reviewlabel.