fix: 让 TIGER 示例可运行并新增 MovieLens 脚本 - #245
Conversation
- split run modes into generate-toy-data/train/test/all (+ prepare-data for MovieLens) - align generated and read data paths; drop hardcoded absolute ckpt_path default - add semantic-id tokens and resize embeddings before training - load tokenizer/config/model from ckpt_path at test time, move inputs to device - fix TIGERModel for transformers>=5 (model_parallel guard, Trainer processing_class) - add run_tiger_movielens.py with toy data and a ratings.dat prepare-data mode - add TigerSeqDataset/Trie/collate unit tests and a toy end-to-end smoke test - document TIGER in generative.md (zh/en) and add tiger_reproduction.md (zh/en) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
…kpt load Addresses review on datawhalechina#245: - prepare-data: add --vocab_path to reuse the shared HSTU/HLLM vocab.pkl so inter.json item ids are the same token ids as the item embeddings and RQ-VAE semantic ids (TIGER/HSTU/HLLM share one id space). Falls back to the standalone movie_id->1..N mapping when omitted. - test(): load the model with the checkpoint's own config and resize only on a tokenizer mismatch; previously a grown vocab_size was passed to from_pretrained, raising an embedding-size mismatch before the resize could run. - Keep training from scratch (random init) per the TIGER paper; correct the misleading "fine-tune T5" wording in docstrings and the zh/en reproduction docs. - Make the MovieLens toy data a deterministic +1 walk so the toy run shows non-trivial accuracy once training works. - test(): set use_cache=True to speed up constrained beam search. - tests: add a vocab-aligned prepare-data test.
|
1、关于 "fine-tune T5" 的措辞 TIGER 原论文里 T5 编码器-解码器就是从头随机初始化训练的(语义 ID 词表不是自然语言,预训练的 NL 权重没有意义),官方参考实现 NonameUntitled/tiger 也是 T5ForConditionalGeneration(config) 随机初始化。所以 train() 用 TIGERModel(config) 从 --base_model 的 config 构建架构、随机初始化,是符合论文的;--base_model 只提供架构/config 与 tokenizer,不加载预训练权重。 因此我保留了训练代码,只把 PR 描述和文档里误导性的 "fine-tune T5" 改成“从头训练(随机初始化,不加载预训练权重)”,并在 train() docstring 和复现 blog 里写明了这一点,避免后续读者误解。 2、关于 test() 的 embedding size mismatch 这是真 bug,已修复。原因正如你所说:之前把已经增长过的 vocab_size 传进 from_pretrained,会在加载阶段就因 embedding size 不匹配报错,根本到不了后面的 resize。 现在改为:用 checkpoint 自带的 config 加载模型(TIGERModel.from_pretrained(ckpt),不再传入外部 config),embedding 尺寸与保存权重天然一致;之后只在 tokenizer 与 embedding 行数确实不一致时(例如喂进一个原始 T5 checkpoint 的兜底情形)才 resize_token_embeddings。这样既修了崩溃,也保留了原来的兜底逻辑。 |
背景
PR 3(TIGER Runnable Example)。原
run_tiger_amazon_books.py存在一组导致无法稳定运行的问题,且缺少 MovieLens 脚本、测试与文档。本 PR 修复这些问题并补齐 MovieLens 支持。主要改动
TIGER 示例修复(
run_tiger_amazon_books.py)generate-toy-data/train/test/all(--mode)ckpt_path默认值,回退到--output_diradd_tokens语义 ID 并resize_token_embeddings(原代码漏掉,导致<a_1>被切成子词)ckpt_path加载 tokenizer/config/model,并把输入张量搬到对应 device模型兼容性(
torch_rechub/models/generative/tiger.py)model_parallel/device_map(DataParallel 守卫),Trainer的tokenizer→processing_class按签名自动适配新增 MovieLens 脚本(
run_tiger_movielens.py)测试(
tests/test_tiger_example.py)TigerSeqDataset三种模式切分、get_new_tokens/get_all_items、collate 的 PAD→-100 屏蔽、Trie前缀约束prepare-data+ 一个真实 T5 的端到端 smoke(离线自动 skip)文档
docs/{zh,en}/models/generative.md新增 TIGERModel 小节docs/{zh,en}/blog/tiger_reproduction.md工作流文档并注册 VitePress 侧边栏测试结果
pytest tests/(排除 slow 与缺失可选依赖 pyarrow/onnx 的用例):49 passed / 41 skipped / 0 failed--mode all:MovieLens 与 Amazon 均端到端跑通并输出 hit@k / ndcg@k注意事项
semantic_ids.json仍需用 RQ-VAE 生成,且 item id 必须与inter.json对齐--base_model google-t5/t5-small(规范仓库名)