diff --git a/.github/workflows/torchtitan-quick-start.yml b/.github/workflows/torchtitan-quick-start.yml index 8f7a8b57a..06dca9c81 100644 --- a/.github/workflows/torchtitan-quick-start.yml +++ b/.github/workflows/torchtitan-quick-start.yml @@ -14,22 +14,23 @@ # twice: once from PyPI (pure-python wheel, proves the binary route) # and once from source (`git clone` at the latest release tag + # `uv pip install -e .`) — the training commands need the upstream -# tree for the config registry (`llama3_debugmodel_ce_loss`) and the -# `--override.imports` mechanism. pytorch/torchtitan publishes GitHub +# tree for the recipe-style config API (`--module X --config Y`) and the +# `--override` mechanism. pytorch/torchtitan publishes GitHub # Releases, so the engine's default release_source=releases works — no # special source=tags override needed. # -# torch / torch_npu both pinned to 2.14.0: torch_npu 2.14.0 requires -# CANN 9.2.0 (an older CANN 9.1.0 image fails with a missing -# libhccl.so / missing aclnn symbols), so the image below is the -# 9.2.0 one. triton-ascend 3.6.0 comes from Huawei Cloud's ascend -# `test` index and is installed with --no-deps so the CUDA-only -# community `triton` wheel never lands in the environment. -# -# The modelscope volume caches the `snapshot_download` demo of the -# doc's tokenizer section between runs (the training itself uses -# torchtitan's bundled tests/assets/tokenizer, so no checkpoint is -# downloaded). +# torch 2.15.0 + torch_npu 2.15.0.dev20261116 (Ascend's nightly channel): +# torchtitan v0.4.0 is developed against torch 2.15 (its own CI pins +# torch==2.15.0) and uses torch-2.15-only APIs — the two +# torch.distributed.pipelining symbols the optional Dist-MoE package +# imports eagerly, and FSDP2's MixedPrecisionPolicy(param_dtype_override_fn=). +# The newest *released* torch_npu is 2.14.0, which pins torch==2.14.0.*, +# so the nightly build is the only one that matches torch 2.15. +# torch_npu 2.14.0 requires CANN 9.2.0 (an older CANN 9.1.0 image fails +# with a missing libhccl.so / missing aclnn symbols), so the image below +# is the 9.2.0 one. triton-ascend comes from Ascend's nightly channel +# (its `/test` sub-path is gone) and is installed with --no-deps so the +# CUDA-only community `triton` wheel never lands in the environment. name: torchtitan-quick-start @@ -80,21 +81,17 @@ jobs: # `torchrun --nproc_per_node=2` (FSDP shard=2 + HCCL), and the # single-card section asserts `torch.npu.device_count() == 2`. test_runner: '["linux-aarch64-a2-2"]' - # CANN 9.2.0 + Python 3.12 base image — torch_npu 2.14.0 needs - # CANN 9.2.0 (the 9.1.0 image fails with a missing libhccl.so). + # CANN 9.2.0 + Python 3.12 base image — torch_npu needs CANN 9.2.0 + # (the 9.1.0 image fails with a missing libhccl.so). image: swr.cn-south-1.myhuaweicloud.com/ascendhub/cann:9.2.0-910b-ubuntu22.04-py3.12 - # Persist the modelscope cache across runs: the doc's tokenizer - # section demonstrates `snapshot_download`, and re-downloading the - # Llama 3 tokenizer on every guard run is pure waste. - container_options: >- - --volume=/data/ci-cache/modelscope/torchtitan:/root/.cache/modelscope - # 180 min budget. The doc installs torch / torch_npu / modelscope / - # triton-ascend / torchtitan, clones the upstream tree, writes the - # NPU override module, then runs a 2-step single-card training and - # a 2-step dual-card FSDP training. The first cold-cache run also - # compiles a handful of dataloader Triton kernels through the - # bisheng compiler (serial, TORCHINDUCTOR_COMPILE_THREADS=1, to - # stay under the runner cgroup memory limit). + # 180 min budget. The doc installs torch / torch_npu / triton-ascend / + # torchtitan, clones the upstream tree, writes the NPU override module + # + recipe, then runs a 2-step single-card training and a 2-step + # dual-card FSDP training. The first cold-cache run also compiles the + # model's compiled regions and a handful of dataloader Triton kernels + # through the bisheng compiler (serial, + # TORCHINDUCTOR_COMPILE_THREADS=1, to stay under the runner cgroup + # memory limit). timeout_minutes: 180 upstream_repo: pytorch/torchtitan # Doc URL points to the upstream Ascend/docs repo. {0} is filled by the diff --git a/sources/torchtitan/quick_start.md b/sources/torchtitan/quick_start.md index 107d2b1bf..dd299bf7e 100644 --- a/sources/torchtitan/quick_start.md +++ b/sources/torchtitan/quick_start.md @@ -4,10 +4,10 @@ 本文在昇腾 NPU 上跑通 TorchTitan 的安装与训练链路: -- **安装**:用 uv 装 PyPI 上的二进制 wheel,或从上游最新 release tag 源码安装。 -- **NPU 兼容 override**:用 torchtitan 官方 override 机制替换 attention 栈(RoPE + attention),训练前先断言替换组件与上游组件数值等价。 -- **单卡训练**:`llama3_debugmodel` 配置真跑 2 步,验证配置解析、dataloader 构造、forward + backward 整条链路。 -- **多卡训练**:双卡 FSDP shard 训练,验证 HCCL 集合通信与 DTensor 参数分布。 +- **安装**:用 uv 装 PyPI 上的二进制 wheel,再从**同一个 release tag** 源码安装(可编辑模式),训练用源码树。 +- **NPU 兼容 override**:用 torchtitan 官方 override 机制替换 attention 后端(上游的 flex / varlen 后端在 NPU 上跑不起来),训练前先断言替换组件与上游组件数值等价。 +- **单卡训练**:本文自带的 recipe(llama3 `debugmodel`,~6 M 参数)真跑 2 步,验证配置解析、dataloader 构造、forward + backward 整条链路。 +- **多卡训练**:双卡 FSDP shard + bfloat16 训练,验证 HCCL 集合通信与 DTensor 参数分布。 多卡训练需要至少 2 张 NPU。 @@ -21,7 +21,7 @@ Atlas 900 A2 / A3 训练系列产品或者 Ascend 950 系列产品,并按需 在跑本文档**之前**,你的机器上需要已经装好并可用: -- 可用的 Python 环境(3.11 及以上,TorchTitan v0.3.0 要求 `requires-python >= 3.11`) +- 可用的 Python 环境(3.11 及以上,TorchTitan 要求 `requires-python >= 3.11`) - 可用的 CANN(参考[快速安装昇腾环境](https://ascend.github.io/docs/sources/ascend/quick_install.html)) ```{admonition} Note @@ -46,12 +46,50 @@ swr.cn-south-1.myhuaweicloud.com/ascendhub/cann:9.2.0-910b-ubuntu22.04-py3.12 | --- | --- | | Python | 3.12 | | CANN | 9.2.0 | -| torch | 2.14.0+cpu | -| torch_npu | 2.14.0 | -| triton-ascend | 3.6.0 | -| modelscope | 1.40.2 | -| torchtitan | 最新 release | -| 训练配置 | 单卡 / 多卡均使用上游默认配置 `torchtitan/models/llama3/config_registry.py::llama3_debugmodel`(默认 `ChunkedLossWrapper` loss;基于 debugmodel:dim=256 / 6 层 / 16 head / vocab 2048,~6 M 参数;多卡为 FSDP shard=2 + bf16) | +| torch | 2.15.0+cpu | +| torch_npu | 2.15.0.dev20261116(Ascend nightly 源) | +| triton-ascend | 3.6.0+dev20261112(Ascend nightly 源) | +| torchtitan | 最新 stable release(二进制 wheel 与源码安装用同一个 tag,当前 v0.4.0) | +| 训练配置 | 本文自带的 recipe `npu_recipe.py::llama3_npu_debugmodel`(单卡 fp32)/ `llama3_npu_debugmodel_bf16`(多卡 bf16):模型是 llama3 `debugmodel`(dim=256 / 6 层 / 16 head / vocab 2048,~6 M 参数),数据是上游自带的 `tests/assets/c4_test`,tokenizer 是上游自带的 `tests/assets/tokenizer`,不下载任何权重 | + +```{admonition} Note +:class: note +**torch_npu 装的是 nightly**:torchtitan v0.4.0 是对着 torch 2.15 开发的(它自己的 CI 钉 +`torch==2.15.0`),要用到 `torch.distributed.pipelining` 的 +`analyze_pipeline_activation_liveness` / `PipelineStageInfo` 和 FSDP2 的 +`MixedPrecisionPolicy(param_dtype_override_fn=...)` —— 都是 torch 2.15 才有的符号。而昇腾发布的 +`torch_npu` 最新只到 2.14.0(并且把 `torch` 钉死在 `2.14.0.*`),所以 2.14.0 是正式发布栈的天花板: +在那套栈上 torchtitan 的 import 链会在建模之前就断掉,FSDP 阶段也会报 +`MixedPrecisionPolicy.__init__() got an unexpected keyword argument 'param_dtype_override_fn'`。 +Ascend 的 nightly 源提供与 torch 2.15 配套的 `torch_npu`,本文就装那一版。 +``` + +```{admonition} Note +:class: note +**上游的配置是 recipe 式的**:torchtitan 不再提供 `torchtitan/models/*/config_registry.py` 里的配置注册表, +配置改成「一个返回 `Trainer.Config` 的普通 Python 函数」,命令行用 `--module <模块> --config <函数名>` 选择; +`--training.steps` / `--comm.mode` / `--dump-folder` 这类通用开关也一并消失,只保留 +`--module` / `--config` / `--override` / `--comm-backend` / `--output-dir` / `--resume-step` / `--print-config`。 +所以本文自带一个 recipe(`npu_recipe.py`),把模型、数据、步数写死,不依赖上游的配置文件 —— +换上游版本时只要 recipe 里的 API 还对得上就行。 +``` + +```{admonition} Note +:class: note +**recipe 里有一处编译区留在 eager**:上游默认编译 `loss` / `fused_binary_activation` / `rope` +三个区域,但 `loss` 区域的图以 `aten.nll_loss_forward` 结尾,inductor 在 NPU 上降不了它 —— +只能注册 fallback,而 torch 2.15 在 **`CI` 环境变量存在时**(GitHub Actions 会设)把 +「a decomposition exists, we should switch to it」从警告升级成硬断言: + +``` +torch._inductor.exc.InductorError: AssertionError: make_fallback +(aten.nll_loss_forward.default): a decomposition exists, we should switch to it. +``` + +所以 recipe 把 `loss`(以及 `moe_aux_loss`)留在 eager:eager 路径由 NPU 自己的 kernel 处理 loss, +上游的 `ChunkedLossWrapper` 原样就能跑。其余编译区照旧开着 —— 编译过的 `rope` 区域正是上游 +`ComplexRoPE` 能在 NPU 上跑的原因(编译路径 gather 的是 cache 的实数视图)。 +``` ## 前置安装 @@ -74,7 +112,7 @@ npu-smi info | 0 | 0000:41:00.0 | 0 0 / 0 2922 / 32768 | +===========================+===============+====================================================+ | NPU Chip | Process id | Process name | Process memory(MB) | -+========================---+---------------+----------------------------------------------------+ ++---------------------------+---------------+----------------------------------------------------+ | No running processes found in NPU 5 | +---------------------------+---------------+----------------------------------------------------+ ``` @@ -99,8 +137,17 @@ Python 3.12.xxx 装 `torch` + `torch_npu`: ```shell #test-setup -uv pip install -f https://mirrors.aliyun.com/pytorch-wheels/cpu torch==2.14.0 -uv pip install torch_npu==2.14.0 +uv pip install -f https://mirrors.aliyun.com/pytorch-wheels/cpu torch==2.15.0 +uv pip install --no-deps --find-links https://repo.huaweicloud.com/ascend/repos/pypi/nightly/torch-npu/ torch_npu==2.15.0.dev20261116 +uv pip install numpy +``` + +```{admonition} Note +:class: note +`torch_npu` 的 nightly 源(`https://repo.huaweicloud.com/ascend/repos/pypi/nightly/torch-npu/`)是**扁平目录** +而不是 PEP 503 索引,所以用 `--find-links` 而不是 `--extra-index-url`。`--no-deps` 是必需的: +wheel 把 `torch` 钉在 `2.15.0.*`,不加这个开关 pip 会把上面装好的 CPU 版 torch 换成 PyPI 上的 +CUDA 版;被 `--no-deps` 一起挡掉的 `numpy` 单独装(`torch_npu` 的另一个依赖)。 ``` 检查 torch / torch_npu 是否装好且 NPU 设备可用(下面的命令用 Python 执行): @@ -116,8 +163,8 @@ print('count:', torch.npu.device_count()) 输出结果如下: ```shell #test-result id="check-torch" -torch= 2.14.0+cpu -torch_npu= 2.14.0 +torch= 2.15.0+cpu +torch_npu= 2.15.0.dev20261116 is_available: True count: 2 ``` @@ -127,29 +174,18 @@ count: 2 如果 `import torch_npu` 失败,回到 [Ascend PyTorch 安装文档](https://gitcode.com/Ascend/pytorch) 检查 torch / torch_npu / CANN 三方兼容矩阵。 ``` -### 安装 modelscope - -```shell #test-setup -uv pip install modelscope==1.40.2 -``` - -打印安装版本(下面的命令用 Python 执行): - -```python #test id="modelscope-install" -import modelscope -print('modelscope', modelscope.__version__) -``` - -输出结果如下: - -```shell #test-result id="modelscope-install" -modelscope 1.40.2 -``` - ### 安装 triton-ascend +`triton-ascend` 是 Ascend 的 Triton fork(上游编译区域要用它),发布在华为云的 Ascend PyPI 源: +正式版在 `https://repo.huaweicloud.com/ascend/repos/pypi/triton-ascend/`,nightly 版在 +`https://repo.huaweicloud.com/ascend/repos/pypi/nightly/triton-ascend/`。本文装 nightly 版, +和上面的 `torch_npu` nightly 配套。`--no-deps` 是必需的:triton-ascend 的 wheel 声明了社区版 +`triton` 依赖,装上后两个包会同时写 `triton/` 目录,混装会在 `import triton` 时报 +`cannot import name 'Language'`。 + ```shell #test-setup -uv pip install --no-deps --extra-index-url https://repo.huaweicloud.com/ascend/repos/pypi/test triton-ascend==3.6.0 +set -euo pipefail +uv pip install --no-deps --find-links https://repo.huaweicloud.com/ascend/repos/pypi/nightly/triton-ascend/ triton-ascend==3.6.0+dev20261112 uv pip install pybind11 ``` @@ -163,17 +199,27 @@ print('triton-ascend', version('triton-ascend')) 输出结果如下: ```shell #test-result id="triton-install" -triton-ascend 3.6.0 +triton-ascend 3.6.0+dev20261112 ``` ## 安装 torchtitan ### 通过 uv 安装 -torchtitan 上 PyPI 有 `py3-none-any` 的纯 Python wheel(450 KB),不依赖 native build——`uv pip install torchtitan` 直接装最新 stable release,不用管版本号: +torchtitan 上 PyPI 有 `py3-none-any` 的纯 Python wheel,不依赖 native build——`uv pip install torchtitan` 直接装最新 stable release,不用管版本号: ```shell #test-setup -uv pip install --extra-index-url https://mirrors.aliyun.com/pypi/simple/ torchtitan +printf 'attn-gym\n' > /tmp/torchtitan-excludes.txt +uv pip install --excludes /tmp/torchtitan-excludes.txt --extra-index-url https://mirrors.aliyun.com/pypi/simple/ torchtitan +``` + +```{admonition} Note +:class: note +`--excludes` 排除的是 `attn-gym`:上游把 `attn-gym[linear,cudnn]==0.0.17` 写成了硬依赖, +而它的 `cudnn` / `linear` extra 会拉 `cuda-python>=13` 与 `nvidia-cutlass-dsl[cu13]` —— 两者都是 CUDA 专有组件, +在昇腾 NPU 上既用不到,也会白占几个 GB 磁盘。`attn-gym` 本体是纯 Python wheel,它提供的只是 CUDA attention +kernel;torchtitan 对此有官方降级支持(`torchtitan/models/common/attn_gym.py` 用占位 kernel 兜底, +只有真的要用 GDN / KDA 这类线性注意力模型时才会报错),所以在 NPU 上直接排除它最省事。 ``` 验证装上的版本(下面的命令用 Python 执行): @@ -194,33 +240,42 @@ torchtitan xxx xxx 表示最新的版本号。 ``` - - - ### 从源码安装 - - 克隆上游仓库、`uv pip install -e .` 安装: ```shell #test-setup load="upstream_ref>>ref" +set -euo pipefail git clone --depth 1 --branch https://github.com/pytorch/torchtitan.git cd torchtitan -uv pip install -e . +uv pip install --excludes /tmp/torchtitan-excludes.txt -e . ``` ```{admonition} Note :class: note -`` 替换为 torchtitan 当前的最新 release 版本(如 `v0.3.0`)。 +`` 是上一步解析出的 release tag(当前 `v0.4.0`)。用 release tag 而不是 `main`: +recipe 用到的 API(`Trainer.Config` 的字段、`GQAttention.Config` 的结构、override 机制)在 +release 上是稳定的,`main` 上游随时可能改。 + +`uv pip install -e .` 会把上一步装的二进制 wheel 换成可编辑安装(同一个 distribution 名)。 +`--excludes` 沿用上一节排除 `attn-gym` 的理由:源码安装同样会解析它的硬依赖。 ``` 验证装上的版本(下面的命令用 Python 执行,需在克隆出来的源码目录里跑): @@ -242,57 +297,71 @@ xxx 表示最新的版本号。 ## 跑训练 -### 下载 tokenizer +### 安装 NPU override 模块与 recipe -```shell #test-setup id="modelscope-download-tokenizer" store="ms_tokenizer_path" -python -c "from modelscope import snapshot_download; print(snapshot_download('LLM-Research/Llama-3.2-1B', allow_patterns=['*.json', '*.model', 'tokenizer*']))" | tail -n 1 -``` - -验证 tokenizer 关键文件都落盘: +override 模块和 recipe 都是本仓(Ascend/docs)里的普通 Python 文件,放在 torchtitan 仓库**之外**, +通过 `PYTHONPATH` 挂进去: -```shell #test id="modelscope-verify-tokenizer" load="ms_tokenizer_path>>ms_tokenizer_path" -ls -``` - -```shell #test-result id="modelscope-verify-tokenizer" fuzzy='...' -config.json -configuration.json -generation_config.json -original -special_tokens_map.json -tokenizer.json -tokenizer_config.json -``` +| 文件 | 作用 | +| --- | --- | +| `tests/torchtitan/npu_compat.py` | override 模块本身:注册一个 `npu_attention` 工厂,替换 `GQAttention.Config` 节点 | +| `tests/torchtitan/npu_recipe.py` | quick-start 的 recipe:返回 `Trainer.Config`(模型 / 数据 / 步数 / loss) | +| `tests/torchtitan/check_npu_overrides.py` | 等价性自检脚本(CPU):断言替换进去的组件与上游组件数值等价 | -文件名是 Llama 3 tokenizer 必备文件,确认 snapshot_download 命中正确。 +override 模块由 torchtitan 官方的 override 机制激活(`--override npu_compat.npu_attention`), +**不修改上游源码**——torchtitan 的 checkout 保持 git clean。 ```{admonition} Note :class: note -`` 替换为上一步 `snapshot_download` 打印的模型落盘路径。 +`${DOC_REF:-main}` 是被本文档自身所在的 ref(CI 里由测试注入 `DOC_REF`,手工执行时默认 `main`), +保证拉到的 override 模块与本文档来自同一个版本;走 contents API 而不是 raw.githubusercontent.com: +NPU runner 到 raw 端点不通,到 api 端点通(base class 里也是这么取文档的);仓库是 public,匿名即可, +`GH_TOKEN` 存在时带上以提高限流额度。 ``` -### 安装 NPU override 模块 - -override 模块是本仓(Ascend/docs)里的两个普通 Python 文件,放在 torchtitan 仓库**之外**,通过 `PYTHONPATH` + `--override.imports module.function` 激活: - -| 文件 | 作用 | -| --- | --- | -| `tests/torchtitan/npu_compat.py` | override 模块本身:注册一个 `npu_attention` 工厂,替换 `GQAttention.Config` 节点 | -| `tests/torchtitan/check_npu_overrides.py` | 等价性自检脚本(CPU):断言替换进去的组件与上游组件数值等价 | - ```shell #test-setup mkdir -p "$HOME/tt-npu-overrides" BASE="https://api.github.com/repos/Ascend/docs/contents/tests/torchtitan" REF="${DOC_REF:-main}" AUTH=() [ -n "${GH_TOKEN:-}" ] && AUTH+=(-H "Authorization: Bearer $GH_TOKEN") -for f in npu_compat.py check_npu_overrides.py; do +for f in npu_compat.py npu_recipe.py check_npu_overrides.py; do curl -fsSL --retry 3 --retry-connrefused --retry-delay 2 \ -H "Accept: application/vnd.github.raw" "${AUTH[@]}" \ "$BASE/$f?ref=$REF" -o "$HOME/tt-npu-overrides/$f" done ``` +### torch 2.15 的两个符号 + +torchtitan v0.4.0 会**顺带**导入它可选的 Dist-MoE 支持,而那个包的 runtime 模块第一行就是: + +```python +from torch.distributed.pipelining import ( + analyze_pipeline_activation_liveness, + PipelineStageInfo, +) +``` + +这两个符号是 **torch 2.15.0** 才加的;FSDP2 的 mixed-precision policy 在 2.15 也多了一个按参数 +覆盖 dtype 的回调(上游 `torchtitan/distributed/fsdp.py` 里的 +`MixedPrecisionPolicy(param_dtype_override_fn=...)`)。本文装的是与 torch 2.15 配套的 `torch_npu` +nightly,上面两个符号都是原生存在的——**不需要任何兼容垫片**。下面这条命令确认这一点: + +```shell #test id="torch-compat" +cd torchtitan && PYTHONPATH="$HOME/tt-npu-overrides" python -c "import torch, torch.distributed.pipelining as p; from torch.distributed.fsdp import MixedPrecisionPolicy; print('analyze_pipeline_activation_liveness:', hasattr(p, 'analyze_pipeline_activation_liveness')); print('PipelineStageInfo:', hasattr(p, 'PipelineStageInfo')); print('param_dtype_override_fn:', MixedPrecisionPolicy(param_dtype=torch.bfloat16, param_dtype_override_fn=None).param_dtype)" +``` + +输出结果如下: + +```shell #test-result id="torch-compat" +analyze_pipeline_activation_liveness: True +PipelineStageInfo: True +param_dtype_override_fn: torch.bfloat16 +``` + +### 验证 override 与 recipe + 验证 override 工厂完成注册: ```shell #test id="override-register" @@ -305,64 +374,77 @@ cd torchtitan && PYTHONPATH="$HOME/tt-npu-overrides" python -c "import npu_compa ['npu_compat.npu_attention'] ``` -再断言替换进去的组件与上游组件**数值等价**(CPU 即可跑,不需要 NPU):RoPE 与上游 `ComplexRoPE` 的输出必须逐位一致,attention 与朴素实现的逐文档 causal 掩码注意力必须一致。 +再断言替换进去的组件与上游组件**数值等价**(CPU 即可跑,不需要 NPU):attention 与朴素实现的逐文档 +causal 掩码注意力必须一致,同时确认 override **不动 RoPE**(llama3 保持上游的 `ComplexRoPE`, +qwen3 保持上游的 `CosSinRoPE`)。 ```shell #test id="override-equivalence" -python "$HOME/tt-npu-overrides/check_npu_overrides.py" +PYTHONPATH="$HOME/tt-npu-overrides" python "$HOME/tt-npu-overrides/check_npu_overrides.py" ``` 输出结果如下: ```shell #test-result id="override-equivalence" -npu override equivalence: rope OK, mask OK +npu override equivalence: mask OK, metadata OK, attention selection OK +``` + +验证 recipe 能被 `--module` 选中,并返回一份可训练的 `Trainer.Config`: + +```shell #test id="recipe-import" +cd torchtitan && PYTHONPATH="$HOME/tt-npu-overrides" python -c "import npu_recipe; c = npu_recipe.llama3_npu_debugmodel(); print('recipe ok, steps =', c.training.steps)" +``` + +输出结果如下: + +```shell #test-result id="recipe-import" +recipe ok, steps = 2 ``` ### 单卡训练 -在 1 张 NPU 上用 `torchrun` 真跑 2 步 `debugmodel`(上游自带的最小配置,~6 M 参数,单卡足够),验证完整的训练链路。`--override.imports` 激活上一步的 NPU 兼容替换,日志会打印 `Applied 6 override(s)`。多卡训练见下一节「多卡训练」(8B 规模不在本文档验证范围): +在 1 张 NPU 上用 `torchrun` 真跑 2 步 `debugmodel`(~6 M 参数,单卡足够),验证完整的训练链路。 +`--override` 激活上一步的 NPU 兼容替换,日志会打印 `Applied 6 override(s)`(6 层各替换一次)。 +多卡训练见下一节「多卡训练」(8B 规模不在本文档验证范围): -```shell #test id="torchtitan-train-debug" load="upstream_ref>>ref" -cd torchtitan && git checkout -f -PYTHONPATH="$HOME/tt-npu-overrides" \ +```shell #test id="torchtitan-train-debug" +cd torchtitan && PYTHONPATH="$HOME/tt-npu-overrides" \ ASCEND_RT_VISIBLE_DEVICES=0 \ TORCHINDUCTOR_COMPILE_THREADS=1 \ torchrun --nproc_per_node=1 \ --rdzv_backend c10d \ --rdzv_endpoint="localhost:0" \ -m torchtitan.train \ - --module llama3 \ - --config llama3_debugmodel \ - --override.imports npu_compat.npu_attention \ - --comm.mode default \ - --training.steps 2 \ - --training.local-batch-size 1 \ - --training.seq-len 256 \ - --metrics.log-freq 1 \ - --metrics.disable-color-printing \ - --dump-folder /tmp/torchtitan-quickstart + --module npu_recipe \ + --config llama3_npu_debugmodel \ + --override npu_compat.npu_attention \ + --output-dir /tmp/torchtitan-quickstart ``` 输出结果类似如下: +torchtitan v0.4.0 起每个模块用自己的 logger(`logging.getLogger(__name__)`),日志格式是 +`[titan] <时间> - - <级别> - <消息>`。入口是 `python -m torchtitan.train`,所以入口自己打的 +两行日志(版本号、`Process group destroyed`)logger 名是 `__main__`;模块里的日志则是模块路径 +(`torchtitan.trainer`、`torchtitan.config.override`、`torchtitan.observability.metrics`): + ```shell #test-result id="torchtitan-train-debug" fuzzy='xxx' fuzzy='...' -[titan] xxx - root - INFO - torchtitan version: xxx -[titan] xxx - root - INFO - [Override] npu_compat.npu_attention: model_spec.model.layers.0.attention GQAttention.Config -> NpuGQAttention.Config -... -[titan] xxx - root - INFO - Applied 6 override(s) -[titan] xxx - root - INFO - Building llama3 debugmodel +[titan] xxx - __main__ - INFO - torchtitan version: xxx +[titan] xxx - torchtitan.config.override - INFO - [Override] npu_compat.npu_attention: model.layers.0.attention GQAttention.Config -> NpuGQAttention.Config +[titan] xxx - torchtitan.config.override - INFO - Applied 6 override(s) +[titan] xxx - torchtitan.observability.metrics - INFO - step: xxx +[titan] xxx - torchtitan.trainer - INFO - Training completed +[titan] xxx - __main__ - INFO - Process group destroyed ... -[titan] xxx - root - INFO - Sleeping 2 seconds for other ranks to complete -[titan] xxx - root - INFO - Training completed -[titan] xxx - root - INFO - Process group destroyed ``` ### 多卡训练 -在 2 张 NPU 上用 `torchrun` 起真分布式训练,同样真跑 2 步:FSDP 把模型参数和优化器状态分片到两张卡,并叠 `--training.dtype bfloat16` 验证混合精度。`HCCL_NPU_SOCKET_PORT_RANGE` 用于错开两个 rank 的 HCCL 端口(否则会报 `EI0020` 端口冲突): +在 2 张 NPU 上用 `torchrun` 起真分布式训练,同样真跑 2 步:FSDP 把模型参数和优化器状态分片到两张卡, +并叠 bf16 参数(`llama3_npu_debugmodel_bf16`)验证混合精度。`HCCL_NPU_SOCKET_PORT_RANGE` 用于错开两个 rank 的 +HCCL 端口(否则会报 `EI0020` 端口冲突): -```shell #test id="torchtitan-train-2card" load="upstream_ref>>ref" -cd torchtitan && git checkout -f -PYTHONPATH="$HOME/tt-npu-overrides" \ +```shell #test id="torchtitan-train-2card" +cd torchtitan && PYTHONPATH="$HOME/tt-npu-overrides" \ ASCEND_RT_VISIBLE_DEVICES=0,1 \ HCCL_NPU_SOCKET_PORT_RANGE=60000-60200 \ TORCHINDUCTOR_COMPILE_THREADS=1 \ @@ -372,32 +454,25 @@ torchrun --nproc_per_node=2 \ --local-ranks-filter 0 \ --tee 3 \ -m torchtitan.train \ - --module llama3 \ - --config llama3_debugmodel \ - --override.imports npu_compat.npu_attention \ - --comm.mode default \ - --training.dtype bfloat16 \ - --training.steps 2 \ - --training.local-batch-size 1 \ - --training.seq-len 256 \ - --metrics.log-freq 1 \ - --metrics.disable-color-printing \ - --dump-folder /tmp/torchtitan-quickstart-2card + --module npu_recipe \ + --config llama3_npu_debugmodel_bf16 \ + --override npu_compat.npu_attention \ + --output-dir /tmp/torchtitan-quickstart-2card ``` 输出结果类似如下: -```shell #test-result id="torchtitan-train-2card" -[default0]:[titan] ... - root - INFO - torchtitan version: ... -[default0]:[titan] ... - root - INFO - Building device mesh with parallelism: ... -[default0]:[titan] ... - root - INFO - Applied 6 override(s) -[default0]:[titan] ... - root - INFO - Building llama3 debugmodel -... -[default0]:[titan] ... - root - INFO - Synchronizing and adjusting timeout for all ProcessGroups to 0:01:40 +`--tee 3` 把每个 worker 的 stdout/stderr 复制到控制台,行首加 `[<角色名>]:` 前缀(这里就是 +`[default0]:`);`--local-ranks-filter 0` 只让 rank 0 的日志上控制台——rank 1 照常参与训练和集合通信, +只是它的日志不打印,所以下面的输出不会被两个 rank 交错: + +```shell #test-result id="torchtitan-train-2card" fuzzy='xxx' fuzzy='...' +[default0]:[titan] xxx - __main__ - INFO - torchtitan version: xxx +[default0]:[titan] xxx - torchtitan.config.override - INFO - Applied 6 override(s) +[default0]:[titan] xxx - torchtitan.observability.metrics - INFO - step: xxx +[default0]:[titan] xxx - torchtitan.trainer - INFO - Training completed +[default0]:[titan] xxx - __main__ - INFO - Process group destroyed ... -[default0]:[titan] ... - root - INFO - Sleeping 2 seconds for other ranks to complete -[default0]:[titan] ... - root - INFO - Training completed -[default0]:[titan] ... - root - INFO - Process group destroyed ``` ## 外部链接 diff --git a/tests/torchtitan/check_npu_overrides.py b/tests/torchtitan/check_npu_overrides.py index c0d5f0734..b07d20207 100644 --- a/tests/torchtitan/check_npu_overrides.py +++ b/tests/torchtitan/check_npu_overrides.py @@ -2,14 +2,16 @@ Run: ``python check_npu_overrides.py`` -Asserts that the two components the NPU overrides substitute in are -numerically equivalent to the upstream components they stand in for: +Asserts that the component the NPU override substitutes in is numerically +equivalent to the upstream component it stands in for: -* ``RealGatherComplexRoPE`` vs upstream ``ComplexRoPE`` -- same cache math, - same rotation, only the gather kernel differs. -* ``BlockCausalSDPA`` vs a naive per-document causal reference -- the +* ``NpuBlockCausalSDPA`` vs a naive per-document causal reference -- the masking semantics upstream gets from its BlockMask. +and that the override only touches the attention backend: the RoPE stays +upstream's -- ``ComplexRoPE`` for llama3 / deepseek_v3, ``CosSinRoPE`` for +qwen3. + Prints one stable line on stdout (the quick-start doc matches it); the measured numbers go to stderr so CI logs keep them. """ @@ -23,59 +25,75 @@ sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) -from npu_compat import BlockCausalSDPA, RealGatherComplexRoPE # noqa: E402 +from npu_compat import ( # noqa: E402 + NpuBlockCausalMetadata, + NpuBlockCausalSDPA, +) -from torchtitan.models.common.rope import ComplexRoPE # noqa: E402 +from torchtitan.models.common.rope import ( # noqa: E402 + ComplexRoPE, + CosSinRoPE, +) TOL = 1e-5 def _naive_block_causal(q, k, v, mask, scale): - """Reference attention: masked softmax over the (q, kv) grid.""" - q2, k2, v2 = q.transpose(1, 2), k.transpose(1, 2), v.transpose(1, 2) + """Reference attention: masked softmax over the (q, kv) grid. + + ``q`` / ``k`` / ``v`` are ``(T, H, D)`` -- packed tokens, like upstream's + inner attentions. ``mask`` is ``(1, 1, T, T)``. + """ + q2 = q.transpose(0, 1).unsqueeze(0) + k2 = k.transpose(0, 1).unsqueeze(0) + v2 = v.transpose(0, 1).unsqueeze(0) att = (q2.float() @ k2.float().transpose(-1, -2)) * scale att = att.masked_fill(~mask, float("-inf")).softmax(-1) - return (att @ v2.float()).transpose(1, 2) + return (att @ v2.float()).squeeze(0).transpose(0, 1) -def check_rope() -> float: - """RealGatherComplexRoPE must match upstream ComplexRoPE exactly.""" - torch.manual_seed(0) - dim, max_seq_len = 16, 128 - shared = dict(dim=dim, max_seq_len=max_seq_len, theta=500000.0, scaling="llama") - upstream = ComplexRoPE.Config(**shared).build() - ours = RealGatherComplexRoPE.Config(**shared).build() - - q = torch.randn(2, 8, 4, dim) - k = torch.randn(2, 8, 4, dim) - positions = torch.tensor([[0, 1, 2, 3, 0, 1, 2, 3], [0, 1, 2, 3, 4, 5, 6, 7]]) - - out_upstream, _ = upstream(q, k, positions) - out_ours, _ = ours(q, k, positions) - diff = (out_upstream - out_ours).abs().max().item() - assert diff < TOL, f"rope mismatch: max_abs_diff={diff}" - return diff +def check_metadata() -> str: + """The backend config must hand the per-document positions to the kernel. + + The trainer builds the attention metadata through the *config* + (``config.build_attention_metadata(...)``), not through the built + module: if the replacement config does not override it, the kernel + never sees the positions and the run dies with "requires per-document + positions". + """ + positions = torch.tensor([0, 1, 2, 0, 1]) + metadata = NpuBlockCausalSDPA.Config().build_attention_metadata(positions) + assert isinstance(metadata, NpuBlockCausalMetadata), type(metadata) + assert metadata.positions is positions + return "positions metadata OK" def check_mask() -> float: - """BlockCausalSDPA must match a naive per-document causal reference.""" + """NpuBlockCausalSDPA must match a naive per-document causal reference.""" torch.manual_seed(0) - batch, seqlen, heads, head_dim = 2, 16, 4, 8 + num_tokens, heads, head_dim = 16, 4, 8 scale = head_dim**-0.5 - q = torch.randn(batch, seqlen, heads, head_dim) - k = torch.randn(batch, seqlen, heads, head_dim) - v = torch.randn(batch, seqlen, heads, head_dim) - # two documents in the first sample, one in the second - positions = torch.tensor([list(range(6)) + list(range(10)), list(range(16))]) - - attention = BlockCausalSDPA.Config().build() - out = attention(q, k, v, positions=positions, scale=scale, enable_gqa=False) + q = torch.randn(num_tokens, heads, head_dim) + k = torch.randn(num_tokens, heads, head_dim) + v = torch.randn(num_tokens, heads, head_dim) + # two documents in this microbatch + positions = torch.tensor([0, 1, 2, 3, 4, 5, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) + + attention = NpuBlockCausalSDPA.Config().build() + out = attention( + q, + k, + v, + attention_metadata=NpuBlockCausalMetadata(positions=positions), + scale=scale, + enable_gqa=False, + ) - doc = torch.cumsum((positions == 0).to(torch.int32), dim=1) - 1 + doc = torch.cumsum((positions == 0).to(torch.int64), dim=0) - 1 mask = ( - (doc[:, :, None] == doc[:, None, :]) - & torch.ones(seqlen, seqlen, dtype=torch.bool).tril() - ).unsqueeze(1) + (doc[:, None] == doc[None, :]) + & torch.ones(num_tokens, num_tokens, dtype=torch.bool).tril() + ).unsqueeze(0).unsqueeze(0) out_ref = _naive_block_causal(q, k, v, mask, scale) diff = (out - out_ref).abs().max().item() assert diff < TOL, f"mask mismatch: max_abs_diff={diff}" @@ -87,16 +105,61 @@ def check_mask() -> float: return diff +def check_rope_untouched() -> str: + """The override must not touch the RoPE. + + Upstream's ``ComplexRoPE`` (llama3 / deepseek_v3) and ``CosSinRoPE`` + (qwen3) stay as they are: the complex64 cache gather only exists in + the eager path, and upstream compiles the ``rope`` region by default -- + the compiled path gathers the cache's real view, which NPU can do. + """ + from torchtitan.config.override import OverrideConfig, apply_overrides + from torchtitan.config.parallelism import ParallelismConfig + from torchtitan.models.llama3 import build_model_config as llama3_build + from torchtitan.models.qwen3 import build_model_config as qwen3_build + + import npu_compat # noqa: F401 (registers the override) + + parallelism = ParallelismConfig() + llama = llama3_build( + "debugmodel", + seq_len=256, + attn_backend="varlen", + parallelism=parallelism, + ) + qwen = qwen3_build( + "debugmodel", + seq_len=256, + attn_backend="varlen", + parallelism=parallelism, + ) + for spec in (llama, qwen): + apply_overrides( + OverrideConfig(imports=["npu_compat.npu_attention"]), spec + ) + + llama_rope = llama.layers[0].attention.rope + qwen_rope = qwen.layers[0].attention.rope + assert isinstance(llama_rope, ComplexRoPE.Config), type(llama_rope) + assert type(qwen_rope) is CosSinRoPE.Config, type(qwen_rope) + assert isinstance( + llama.layers[0].attention.inner_attention, NpuBlockCausalSDPA.Config + ), type(llama.layers[0].attention.inner_attention) + return f"llama3={type(llama_rope).__name__}, qwen3={type(qwen_rope).__name__}" + + def main() -> int: - rope_diff = check_rope() mask_diff = check_mask() + metadata = check_metadata() + selection = check_rope_untouched() print( - f"npu override equivalence: rope OK, mask OK", + "npu override equivalence: mask OK, metadata OK, attention selection OK", file=sys.stdout, ) print( - f"rope max_abs_diff={rope_diff:.3e} (tol {TOL:.0e}), " - f"mask max_abs_diff={mask_diff:.3e} (tol {TOL:.0e})", + f"mask max_abs_diff={mask_diff:.3e} (tol {TOL:.0e}), " + f"metadata: {metadata}, " + f"attention selection: {selection}", file=sys.stderr, ) return 0 diff --git a/tests/torchtitan/npu_compat.py b/tests/torchtitan/npu_compat.py index 86ca52e7e..fe627f4c8 100644 --- a/tests/torchtitan/npu_compat.py +++ b/tests/torchtitan/npu_compat.py @@ -1,109 +1,89 @@ -"""NPU compatibility overrides for torchtitan v0.3.0 on Ascend. +"""NPU compatibility override for torchtitan v0.4.0 on Ascend. Activated through torchtitan's official override mechanism -(``--override.imports``); no upstream source file is modified -- the -torchtitan checkout stays git clean. - -One override replaces one config node (``GQAttention.Config``) because the -override framework rejects *nested* overrides (an override claiming an -ancestor of another override's node raises). That single replacement -carries both NPU fixes: - -1. **RoPE** -- upstream ``ComplexRoPE`` keeps a complex64 cache and indexes - it with the position ids; ``aclnnIndex`` has no ``DT_COMPLEX64`` - implementation. ``RealGatherComplexRoPE`` performs the same gather in - real space (the real / imaginary halves as float32) and hands the result - back to upstream's own rotation code, so the values -- including the - llama frequency scaling and the adjacent-dimension pairing convention -- - are upstream's. Upstream's ``CosSinRoPE`` is *not* a substitute: it - uses the rotate-half pairing convention and rejects - ``scaling="llama"`` outright. - -2. **Attention** -- upstream builds the per-document block-causal mask - only for the Flex / Varlen backends, and neither compiles on NPU. - Upstream's SDPA does compile, but upstream forbids it as a - language-model backend (``get_attention_config("sdpa")`` raises) because - it cannot consume per-document positions. ``BlockCausalSDPA`` feeds - SDPA an explicit dense block-causal mask instead, which keeps upstream's - masking semantics: a token attends to the tokens of its own document - that precede it. - -Both replacements are numerically equivalent to the upstream components -they stand in for. ``check_npu_overrides.py`` (same directory) asserts -that on CPU; the quick-start doc runs it before the training steps. - -Upstream private helpers used below (``_maybe_wrap_positions``, -``_maybe_check_max_pos``, ``_reshape_for_broadcast``): if upstream renames -them this module fails loudly at import time. +(``--override npu_compat.npu_attention``); no upstream source file is +modified -- the torchtitan checkout stays git clean. + +One override replaces one config node (``GQAttention.Config``) and swaps one +of its children -- the **attention backend**: + +* upstream ships two backends for language models: + ``FlexInnerAttention`` (``torch.nn.attention.flex_attention``) and + ``VarlenInnerAttention`` (``torch.nn.attention.varlen``). Neither works + on NPU: the varlen one calls ``aten::_flash_attention_forward``, which + torch_npu does not implement, and the flex one fails while building its + block mask (``torch._inductor.exc.InductorError: SympifyError``). + Upstream's SDPA does run on NPU, but upstream forbids it as a + language-model backend (``get_attention_config("sdpa")`` raises) because + it cannot consume per-document positions. + ``NpuBlockCausalSDPA`` is an ``InnerAttention`` that rebuilds the dense + block-causal mask from the per-document positions and hands it to SDPA, + which keeps upstream's masking semantics: a token attends to the tokens + of its own document that precede it. + +Everything else stays upstream's: + +* **RoPE** -- upstream's ``ComplexRoPE`` keeps a complex64 cache and indexes + it with the position ids; ``aclnnIndex`` has no ``DT_COMPLEX64`` + implementation, so the *eager* gather cannot run on NPU. It does not have + to: upstream compiles the ``rope`` region by default + (``local_compile_regions``), and the compiled path gathers the cache's + real view (float32) instead of the complex one -- which NPU can do. So + the RoPE stays upstream's, and no RoPE replacement is needed. +* **Loss / compiled regions** -- upstream's ``ChunkedLossWrapper`` and its + compiled ``loss`` / ``fused_binary_activation`` regions run on the + torch_npu nightly stack as they are. + +The replacement is numerically equivalent to the upstream component it +stands in for. ``check_npu_overrides.py`` (same directory) asserts that on +CPU; the quick-start doc runs it before the training steps. """ from dataclasses import dataclass +from typing import NamedTuple import torch import torch.nn.functional as F from torchtitan.config.override import derive, override -from torchtitan.models.common.attention import GQAttention -from torchtitan.models.common.rope import ( - ComplexRoPE, - _maybe_check_max_pos, - _maybe_wrap_positions, - _reshape_for_broadcast, -) -from torchtitan.protocols.module import Module - - -class RealGatherComplexRoPE(ComplexRoPE): - """``ComplexRoPE`` whose position gather happens in real space. - - Upstream's ``ComplexRoPE._reshape_cache`` indexes the complex64 cache - directly (``rope_cache[positions]``), which NPU cannot do. Here the - cache is viewed as float32 (real / imaginary interleaved), gathered as - a plain float32 tensor, and converted back to complex64 before - upstream's ``apply_rotary_emb`` runs. The gathered values are - bit-identical to upstream's; only the kernel that produces them - differs. - """ - - @dataclass(kw_only=True, slots=True) - class Config(ComplexRoPE.Config): - pass - - def _reshape_cache(self, query, positions=None): - positions = _maybe_wrap_positions(positions, query) - if positions is not None: - _maybe_check_max_pos(positions, max_valid_pos=self.cache.shape[0] - 1) - # complex64 (max_seq_len, dim/2) -> float32 (max_seq_len, dim) with - # real/imag interleaved -> two contiguous (max_seq_len, dim/2) caches. - real = self.cache.view(torch.float32) - cos_cache = real[:, 0::2].contiguous() - sin_cache = real[:, 1::2].contiguous() - # Same broadcast rules as upstream: the pair shape is - # (batch, seq, heads, head_dim / 2). - pair_shape = (*query.shape[:-1], query.shape[-1] // 2) - cos = _reshape_for_broadcast(cos_cache, pair_shape, positions) - sin = _reshape_for_broadcast(sin_cache, pair_shape, positions) - # Back to the complex64 layout upstream's rotation expects. - return torch.stack([cos, sin], dim=-1).flatten(-2).view(torch.complex64) +from torchtitan.models.common.attention import GQAttention, InnerAttention def _block_causal_mask( - positions: torch.Tensor, seqlen: int, device: torch.device + positions: torch.Tensor, num_tokens: int ) -> torch.Tensor: - """Dense ``(B, 1, L, L)`` bool mask: same document, not in the future. + """Dense ``(1, 1, T, T)`` bool mask: same document, not in the future. Mirrors upstream's per-document (block causal) masking: positions reset to 0 at every document boundary, so tokens of one document must not - attend to another document's tokens. + attend to another document's tokens. Upstream derives the document ids + the same way (``get_document_mask_mod``). + """ + if positions.dim() != 1: + raise ValueError( + f"block-causal masking expects 1-D positions, got {positions.shape}" + ) + doc = torch.cumsum((positions == 0).to(torch.int64), dim=0) - 1 # (T,) + same_doc = doc[:, None] == doc[None, :] # (T, T) + causal = torch.ones( + num_tokens, num_tokens, dtype=torch.bool, device=positions.device + ).tril() + return (same_doc & causal).unsqueeze(0).unsqueeze(0) + + +class NpuBlockCausalMetadata(NamedTuple): + """Per-microbatch metadata handed to ``NpuBlockCausalSDPA``. + + Upstream's inner attentions receive a backend specific metadata object + (a ``BlockMask`` for flex, cumulative sequence lengths for varlen). The + NPU backend needs the per-document positions themselves, because it + rebuilds the dense block-causal mask. """ - pos = positions if positions.dim() > 1 else positions.unsqueeze(0) - doc = torch.cumsum((pos == 0).to(torch.int32), dim=1) - 1 # (B, L) - same_doc = doc[:, :, None] == doc[:, None, :] # (B, L, L) - causal = torch.ones(seqlen, seqlen, dtype=torch.bool, device=device).tril() - return (same_doc & causal).unsqueeze(1) + positions: torch.Tensor -class BlockCausalSDPA(Module): + +class NpuBlockCausalSDPA(InnerAttention): """``scaled_dot_product_attention`` with an explicit block-causal mask. Upstream's ``ScaledDotProductAttention`` refuses an ``attention_masks`` @@ -113,100 +93,86 @@ class BlockCausalSDPA(Module): """ @dataclass(kw_only=True, slots=True) - class Config(Module.Config): - pass + class Config(InnerAttention.Config): + def build_attention_metadata( + self, + positions: torch.Tensor, + *, + padding_mask: torch.Tensor | None = None, + max_num_documents: int | None = None, + max_context_length: int | None = None, + ) -> NpuBlockCausalMetadata: + del padding_mask, max_num_documents, max_context_length + return NpuBlockCausalMetadata(positions=positions) def __init__(self, config: Config) -> None: super().__init__() def forward( self, - q_BLNH: torch.Tensor, - k_BLNH: torch.Tensor, - v_BLNH: torch.Tensor, + q_THK: torch.Tensor, + k_THK: torch.Tensor, + v_THV: torch.Tensor, *, - attention_masks=None, - positions: torch.Tensor | None = None, + attention_metadata: NpuBlockCausalMetadata | None = None, scale: float | None = None, enable_gqa: bool = False, **kwargs, ) -> torch.Tensor: - if positions is None: - raise ValueError("BlockCausalSDPA requires positions") - q_BNLH = q_BLNH.transpose(1, 2) - k_BNLH = k_BLNH.transpose(1, 2) - v_BNLH = v_BLNH.transpose(1, 2) - mask = _block_causal_mask(positions, q_BLNH.shape[1], q_BLNH.device) - out_BNLH = F.scaled_dot_product_attention( - q_BNLH, - k_BNLH, - v_BNLH, + if attention_metadata is None: + raise ValueError("NpuBlockCausalSDPA requires per-document positions") + positions = attention_metadata.positions + mask = _block_causal_mask(positions, q_THK.shape[0]) + out_1HTV = F.scaled_dot_product_attention( + q_THK.transpose(0, 1).unsqueeze(0), + k_THK.transpose(0, 1).unsqueeze(0), + v_THV.transpose(0, 1).unsqueeze(0), attn_mask=mask, scale=scale, is_causal=False, enable_gqa=enable_gqa, ) - return out_BNLH.transpose(1, 2) + return out_1HTV.squeeze(0).transpose(0, 1) class NpuGQAttention(GQAttention): - """``GQAttention`` that forwards ``positions`` to the inner attention. + """``GQAttention`` wired for NPU. - Upstream's ``GQAttention.forward`` consumes ``positions`` for the RoPE - but does not pass it on, so an inner attention that has to rebuild the - per-document mask (everything but Flex / Varlen) cannot see it. The - body below is upstream v0.3.0's ``GQAttention.forward`` with exactly - one addition: ``positions=positions`` in the inner-attention call. + Upstream's ``GQAttention.forward`` already hands the per-document + metadata to the inner attention, and the NPU inner attention rebuilds + the block-causal mask from it -- so the forward is upstream's, unchanged. + This subclass exists so the override has a distinct config class (and a + distinct name in the override log). """ @dataclass(kw_only=True, slots=True) class Config(GQAttention.Config): pass - def forward( - self, - x_BLD: torch.Tensor, - attention_masks, - positions: torch.Tensor | None = None, - ) -> torch.Tensor: - xq_BLNH, xk_BLNH, xv_BLNH = self.qkv_linear(x_BLD) - - # Optional QK normalization (before RoPE, per Qwen3) - if self.q_norm is not None or self.k_norm is not None: - assert self.q_norm is not None and self.k_norm is not None - xq_BLNH = self.q_norm(xq_BLNH) - xk_BLNH = self.k_norm(xk_BLNH) - - # Apply rotary embeddings - xq_BLNH, xk_BLNH = self.rope(xq_BLNH, xk_BLNH, positions) - - # inner_attention returns (B, L, N, H) - out_BLNH = self.inner_attention( - xq_BLNH, - xk_BLNH, - xv_BLNH, - attention_masks=attention_masks, - positions=positions, - scale=self.scaling, - enable_gqa=self.enable_gqa, - ).contiguous() - out_BLD = out_BLNH.flatten(2) - return self.wo(out_BLD) - @override( target=GQAttention.Config, description=( - "NPU attention stack: real-space RoPE gather (aclnnIndex has no " - "DT_COMPLEX64) + SDPA with an explicit block-causal mask" + "NPU attention backend: SDPA with an explicit block-causal mask " + "(upstream's flex / varlen backends do not run on NPU)" ), ) def npu_attention(cfg: GQAttention.Config) -> NpuGQAttention.Config: new = derive(cfg, NpuGQAttention.Config) - # complex64 RoPE cache -> same values, gathered in real space - new.rope = derive(new.rope, RealGatherComplexRoPE.Config) - # flex -> SDPA with a dense block-causal mask. The config class must - # not be a Flex / Varlen config, otherwise the trainer would try to - # build a BlockMask (that path does not compile on NPU). - new.inner_attention = BlockCausalSDPA.Config() + # flex / varlen -> SDPA with a dense block-causal mask. The config class + # must not be a Flex / Varlen config, otherwise the trainer would try to + # build a BlockMask (that path does not work on NPU). + # + # derive() keeps the fields the two configs share -- notably + # ``sharding_config`` (llama3 puts the local-SPMD sharding of q/k/v on + # the inner attention config), which a fresh Config() would silently drop. + new.inner_attention = derive(cfg.inner_attention, NpuBlockCausalSDPA.Config) return new + + +__all__ = [ + "NpuBlockCausalMetadata", + "NpuBlockCausalSDPA", + "NpuGQAttention", + "npu_attention", +] diff --git a/tests/torchtitan/npu_recipe.py b/tests/torchtitan/npu_recipe.py new file mode 100644 index 000000000..a2effa200 --- /dev/null +++ b/tests/torchtitan/npu_recipe.py @@ -0,0 +1,156 @@ +"""Quick-start recipe for torchtitan (latest release) on Ascend NPU. + +torchtitan's config system is recipe-based: a recipe is a plain Python +function that returns a ``Trainer.Config``, and the CLI selects it with +``--module --config ``. The CLI deliberately exposes a +small surface -- there is no ``--training.steps`` / ``--comm.mode`` / +``--dump-folder`` knob any more -- so a quick-start needs a recipe of its +own instead of patching upstream's config tree. + +This module provides the tiny recipe the quick-start trains with: + +* llama3 ``debugmodel`` (dim 256 / 6 layers / 16 heads / vocab 2048, ~6 M + parameters); +* 2 training steps; +* one 2048-token microbatch per step; +* the bundled ``tests/assets/c4_test`` dataset -- no download, no network. + +Everything else is upstream's: the loss is upstream's ``ChunkedLossWrapper`` +around a plain ``CrossEntropyLoss``, and the model's compiled regions +(``loss`` / ``fused_binary_activation`` / ``rope``) are the ones upstream +compiles by default -- they run on the torch_npu nightly stack as they are. + +The recipe lives outside the torchtitan checkout (next to the NPU override +modules) and is activated with ``PYTHONPATH``:: + + PYTHONPATH="$HOME/tt-npu-overrides" torchrun ... -m torchtitan.train \\ + --module npu_recipe --config llama3_npu_debugmodel \\ + --override npu_compat.npu_attention --output-dir /tmp/torchtitan-quickstart +""" + +from __future__ import annotations + +from torchtitan.components.data import ConcatThenSplitPackingConfig, GrainDataLoader +from torchtitan.components.data.text_datasets import DATASETS +from torchtitan.components.loss import ChunkedLossWrapper, CrossEntropyLoss +from torchtitan.components.optim import AdamW, Optim, OptimizersContainer +from torchtitan.config import TrainingConfig +from torchtitan.config.parallelism import ParallelismConfig +from torchtitan.distributed.activation_remat import DefaultActivationRematConfig +from torchtitan.models.common.config_utils import decoder_vocab_size +from torchtitan.models.llama3 import build_model_config +from torchtitan.observability.metrics import MetricsProcessor +from torchtitan.trainer import Trainer + +SEQ_LEN = 256 +"""Training context length (also the RoPE cache length).""" + +NUM_TOKENS_PER_MICROBATCH = 8 * SEQ_LEN +"""Token slots per data-parallel rank per microbatch.""" + +STEPS = 2 +"""How many train steps to run.""" + +UNLOWERABLE_COMPILE_REGIONS = ("loss", "moe_aux_loss") +"""Compiled regions upstream enables that torch_npu cannot lower. + +``loss`` -- the region's graph ends in ``aten.nll_loss_forward``, which +inductor cannot lower for NPU: it registers a *fallback* instead, and +torch 2.15 turns "a decomposition exists, we should switch to it" from a +warning into a hard assertion **whenever the ``CI`` env var is set** +(``torch/_inductor/lowering.py`` guards it with ``os.getenv("CI")``) -- so +on a CI runner the compiled loss region dies with:: + + torch._inductor.exc.InductorError: AssertionError: make_fallback + (aten.nll_loss_forward.default): a decomposition exists, ... + +Keeping the region eager avoids the whole question: the eager NPU kernel +handles the loss (and upstream's ``ChunkedLossWrapper`` runs fine that way). + +``moe_aux_loss`` -- deepseek_v3's MoE router compiles a reduction that dies +in inductor's SIMD scheduling on NPU (``SympifyError``); it is not part of +llama3, but the filter is shared with the examples track. + +The remaining regions (``fused_binary_activation`` / ``rope`` / ...) lower +fine on the torch_npu nightly stack and stay compiled -- the compiled +``rope`` region is also what makes upstream's own ``ComplexRoPE`` work on +NPU (it gathers the cache's real view instead of the complex one). +""" + +MAX_NUM_DOCUMENTS = max(16, (NUM_TOKENS_PER_MICROBATCH + 511) // 512) +"""Upper bound on document segments packed into one microbatch. + +Same estimate upstream's own recipes use (``c4_document_capacity``): ~16 +document slots per 512 tokens, at least 16 for short batches. It has to be +an upper bound because the packed rows are padded to it. +""" + + +def _debugmodel_config(*, bf16: bool = False) -> Trainer.Config: + """Build the quick-start config. + + ``bf16`` switches the FSDP parameter dtype to bfloat16 (mixed + precision); the fp32 variant keeps the parameters in float32. + """ + parallelism = ParallelismConfig() + model_config = build_model_config( + "debugmodel", + seq_len=SEQ_LEN, + attn_backend="varlen", + parallelism=parallelism, + ) + # The compiled regions torch_npu cannot lower stay eager (see + # UNLOWERABLE_COMPILE_REGIONS). + model_config.local_compile_regions = [ + region + for region in model_config.local_compile_regions + if region not in UNLOWERABLE_COMPILE_REGIONS + ] + return Trainer.Config( + loss=ChunkedLossWrapper.Config( + loss_fn=CrossEntropyLoss.Config( + global_vocab_size=decoder_vocab_size(model_config), + ), + ), + hf_assets_path="./tests/assets/tokenizer", + model=model_config, + optim=Optim.Config( + optimizer=OptimizersContainer.Config( + optimizers=[AdamW.Config(pattern=r".*", lr=8e-4)] + ) + ), + training=TrainingConfig( + num_tokens_per_microbatch_per_dp_rank=NUM_TOKENS_PER_MICROBATCH, + max_context_length=model_config.max_context_length, + steps=STEPS, + mixed_precision_param="bfloat16" if bf16 else "float32", + ), + dataloader=GrainDataLoader.Config( + max_num_documents=MAX_NUM_DOCUMENTS, + dataset=ConcatThenSplitPackingConfig(dataset=DATASETS["c4_test"]), + shuffle=False, + ), + metrics=MetricsProcessor.Config(log_freq=1, disable_color_printing=True), + parallelism=parallelism, + checkpointer=None, + activation_remat=DefaultActivationRematConfig(), + validator=None, + ) + + +def llama3_npu_debugmodel() -> Trainer.Config: + """Single-card quick-start: 2 steps of the llama3 debugmodel, fp32.""" + return _debugmodel_config() + + +def llama3_npu_debugmodel_bf16() -> Trainer.Config: + """Dual-card quick-start: same recipe with bfloat16 parameters.""" + return _debugmodel_config(bf16=True) + + +__all__ = [ + "SEQ_LEN", + "STEPS", + "llama3_npu_debugmodel", + "llama3_npu_debugmodel_bf16", +] diff --git a/tests/torchtitan/test_npu_overrides.py b/tests/torchtitan/test_npu_overrides.py index ff24376ca..6d5164fea 100644 --- a/tests/torchtitan/test_npu_overrides.py +++ b/tests/torchtitan/test_npu_overrides.py @@ -21,7 +21,7 @@ HERE = os.path.dirname(os.path.abspath(__file__)) SCRIPT = os.path.join(HERE, 'check_npu_overrides.py') -EXPECTED = 'npu override equivalence: rope OK, mask OK' +EXPECTED = 'npu override equivalence: mask OK, metadata OK, attention selection OK' class NpuOverrideEquivalenceTests(unittest.TestCase): diff --git a/tests/torchtitan/test_quick_start_ascend.py b/tests/torchtitan/test_quick_start_ascend.py index 57c0f393b..1afcf19b2 100644 --- a/tests/torchtitan/test_quick_start_ascend.py +++ b/tests/torchtitan/test_quick_start_ascend.py @@ -8,36 +8,56 @@ Scope note: the doc body covers the smoke path for **stock torchtitan** running under ``torch_npu`` PrivateUse1 dispatch. ``torch`` / ``torch_npu`` / -``modelscope`` / ``triton-ascend`` / ``torchtitan`` are all installed by the -doc body via ``uv pip install`` (versions per the -[Ascend PyTorch Compatibility Matrix](https://gitcode.com/Ascend/pytorch/blob/main/COMPATIBILITY.en.md): -torch==2.14.0 / torch_npu==2.14.0 / CANN==9.2.0) — ``torch``'s CPU build -only exists in the Aliyun ``pytorch-wheels/cpu`` find-links source (PyPI's -linux wheels are all CUDA builds), ``torch_npu`` comes from the default -index, which in CI is the in-cluster PyPI cache gateway. ``triton-ascend`` -comes from Huawei Cloud's ascend ``test`` index (the community ``triton`` -wheel is CUDA-only and must stay out of the environment). ``torchtitan`` is -installed **twice**: first from PyPI (pure-python wheel) to prove the -binary route, then from source (``git clone`` pytorch/torchtitan at the -latest release tag + ``uv pip install -e .``) because the training commands -need the upstream tree (config registry + the ``--override.imports`` -mechanism). +``triton-ascend`` / ``torchtitan`` are all installed by the doc body via +``uv pip install`` (torch==2.15.0 / torch_npu==2.15.0.dev20261116 / +CANN==9.2.0) — ``torch``'s CPU build only exists in the Aliyun +``pytorch-wheels/cpu`` find-links source (PyPI's linux wheels are all CUDA +builds). ``torch_npu`` and ``triton-ascend`` come from Ascend's **nightly** +channel (flat wheel listings, used as find-links): torchtitan v0.4.0 needs +torch >= 2.15 APIs (``torch.distributed.pipelining``'s +``analyze_pipeline_activation_liveness`` / ``PipelineStageInfo`` and FSDP2's +``MixedPrecisionPolicy(param_dtype_override_fn=...)``), and the newest +*released* ``torch_npu`` is 2.14.0, which pins ``torch==2.14.0.*`` — the +nightly build is the only one that matches torch 2.15, so no compatibility +shim is needed (the doc asserts the three symbols exist instead). The +community ``triton`` wheel is CUDA-only and must stay out of the +environment. ``torchtitan`` is installed **twice**: first from PyPI +(pure-python wheel, latest release, with ``attn-gym`` excluded because its +``cudnn`` / ``linear`` extras pull CUDA-only wheels) to prove the binary +route, then from source (``git clone`` pytorch/torchtitan at the **latest +release tag** + ``uv pip install -e .``) because the training commands need +the upstream tree. + +The doc tracks the **latest upstream release** (``UPSTREAM_REF``, injected by +the engine from ``/releases/latest``; locally the doc falls back to the +GitHub API). Since v0.4.0 torchtitan's config system is recipe-based: a +recipe is a plain function returning a ``Trainer.Config``, selected with +``--module --config ``, and the CLI no longer exposes +``--training.steps`` / ``--comm.mode`` / ``--dump-folder`` (nor does +``torchtitan/models/*/config_registry.py`` exist any more). The doc +therefore ships its own recipe (``tests/torchtitan/npu_recipe.py``) next to +the NPU override modules, mounted into the upstream tree via ``PYTHONPATH``; +the upstream checkout stays git clean (no source patching). The doc verifies: * package + sub-module imports (``torchtitan``, ``torchtitan.config.override``, ``torchtitan.models.common.attention``, ``torchtitan.models.common.rope``) -* the NPU compatibility overrides register correctly (ComplexRoPE -> - CosSinRoPE, FlexAttention -> SDPA) +* the torch 2.15 APIs torchtitan v0.4.0 needs are present in the installed + torch (the two ``torch.distributed.pipelining`` symbols and the FSDP2 + ``MixedPrecisionPolicy(param_dtype_override_fn=...)`` keyword) +* the NPU compatibility override registers correctly (upstream's Flex / + Varlen attention backends -> SDPA with an explicit block-causal mask; the + RoPE stays upstream's) +* the recipe is importable and returns a trainable ``Trainer.Config`` * single-card training (``torchrun --nproc_per_node=1``, 2 real steps over the Llama 3 debug model) with NPU dispatch, which fails fast if the - attention / RoPE kernels did not register + attention kernels did not register * dual-card training (``torchrun --nproc_per_node=2``, FSDP shard=2 + bf16), covering HCCL collectives and DTensor parameter sharding -Model download / ModelScope cache is only used to demonstrate the -``snapshot_download`` API — training itself uses torchtitan's bundled -``tests/assets/tokenizer``, so no checkpoint is fetched. +Training uses torchtitan's bundled assets (``tests/assets/tokenizer`` and +``tests/assets/c4_test``), so no checkpoint or dataset is downloaded. """ from __future__ import annotations @@ -87,30 +107,30 @@ class TestQuickStartAscend(MarkdownDocTestBase, unittest.TestCase): ``prepare_environment`` only does runner-side scaffolding (CANN env + CUDA exclusion list + ``uv`` bootstrap); ``torch`` / ``torch_npu`` / - ``modelscope`` / ``triton-ascend`` / ``torchtitan`` are all installed by - the doc's ``#test-setup`` / ``#test`` blocks, so the test exercises the - exact install path users get. + ``triton-ascend`` / ``torchtitan`` are all installed by the doc's + ``#test-setup`` / ``#test`` blocks, so the test exercises the exact + install path users get. """ # 60 min per command. torchtitan's smallest meaningful command — - # `python -m torchtitan.train --module llama3 --config - # llama3_debugmodel --override.imports npu_compat.npu_attention - # --training.steps 2 --comm.mode default` — needs to: clone torchtitan - # (~5 MB sparse), `uv pip install -e .`, pull the NPU override module - # ($HOME/tt-npu-overrides/npu_compat.py + check_npu_overrides.py, from - # this repo at the doc's own ref, swapped in via torchtitan's official - # @override mechanism — no upstream source patching), assert the - # override is numerically equivalent to the upstream components it - # stands in for, then run a 2-step HCCL training that builds the - # Llama 3 debug model (~256 dim / 6 layers) on NPU with SDPA attention - # (aclnn flash attention + an explicit block-causal mask; the flex path - # hits torch_npu 2.14.0 lowering gaps, see the doc's compat section). - # 60 minutes: first cold-cache run still compiles a handful of - # dataloader Triton kernels through the bisheng compiler (serial, - # TORCHINDUCTOR_COMPILE_THREADS=1, to stay under the runner cgroup - # memory limit) plus the model build/init chain. The loss is - # upstream's default (`llama3_debugmodel` -> ChunkedLossWrapper); the - # 2026-09 meta-tensor leak no longer reproduces on torch 2.14. + # `python -m torchtitan.train --module npu_recipe --config + # llama3_npu_debugmodel --override npu_compat.npu_attention` — needs to: + # clone torchtitan at the latest release tag (~5 MB sparse), + # `uv pip install -e .`, pull the NPU override module + recipe + # ($HOME/tt-npu-overrides/{npu_compat,npu_recipe,check_npu_overrides}.py, + # from this repo at the doc's own ref, swapped in via torchtitan's + # official @override mechanism — no upstream source patching), assert the + # override is numerically equivalent to the upstream component it stands + # in for, then run a 2-step HCCL training that builds the Llama 3 debug + # model (~256 dim / 6 layers) on NPU with SDPA attention (aclnn flash + # attention + an explicit block-causal mask; upstream's flex / varlen + # backends do not run on NPU). 60 minutes: the first cold-cache run + # still compiles the model's compiled regions (loss / activation / rope) + # plus a handful of dataloader Triton kernels through the bisheng + # compiler (serial, TORCHINDUCTOR_COMPILE_THREADS=1, to stay under the + # runner cgroup memory limit) plus the model build / init chain. The loss + # is upstream's own (ChunkedLossWrapper around a plain CrossEntropyLoss) + # — it runs as it is on the torch_npu nightly stack. DEFAULT_COMMAND_TIMEOUT = 3600 USER_AGENT = 'ascend-docs/quick-start' @@ -181,10 +201,10 @@ class TestQuickStartAscend(MarkdownDocTestBase, unittest.TestCase): def prepare_environment(cls) -> None: """Source CANN env + write the CUDA exclusion list + install uv. - ``torch`` / ``torch_npu`` / ``modelscope`` / ``triton-ascend`` / - ``torchtitan`` are intentionally NOT pre-installed here: the doc's - ``#test-setup`` blocks install them, so a broken install block - fails loudly instead of being masked by a pre-installed copy. + ``torch`` / ``torch_npu`` / ``triton-ascend`` / ``torchtitan`` are + intentionally NOT pre-installed here: the doc's ``#test-setup`` + blocks install them, so a broken install block fails loudly instead + of being masked by a pre-installed copy. """ # 0) CANN env: source set_env.sh and merge the env stream into # os.environ