diff --git a/.github/goldens-constraints.txt b/.github/goldens-constraints.txt
index 9d81f28..c3f8815 100644
--- a/.github/goldens-constraints.txt
+++ b/.github/goldens-constraints.txt
@@ -3,3 +3,6 @@ Pillow==12.3.0
numpy==2.4.6
texture2ddecoder==1.0.6
astc-encoder-py==0.1.12
+fonttools==4.67.0
+freetype-py==2.5.1
+scipy==1.18.1
diff --git a/.github/scripts/smoke_wheel.py b/.github/scripts/smoke_wheel.py
index 84aa7a2..354258c 100644
--- a/.github/scripts/smoke_wheel.py
+++ b/.github/scripts/smoke_wheel.py
@@ -14,7 +14,8 @@
import nnnotes
import ournotes_sim
-from nnnotes import deckdata, musicdata
+from nnnotes import musicdata
+from nnnotes.musicdata import deckdata
assert Path(nnnotes.__file__).resolve().is_relative_to(Path(sys.prefix).resolve())
deck = musicdata.Deck(workers=1, aptitude=False)
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 0b8bf00..1245c80 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -32,7 +32,7 @@ jobs:
- name: Install (with the deck model and the fonts extra)
run: |
python -m pip install --upgrade pip
- python -m pip install -e ".[test,deck,fonts]"
+ python -m pip install -e ".[test,deck,fonts,remote]"
- name: Lint
run: |
python -m ruff check .
@@ -55,7 +55,7 @@ jobs:
- name: Install
run: |
python -m pip install --upgrade pip
- python -m pip install -e ".[test,deck,fonts]"
+ python -m pip install -e ".[test,deck,fonts,remote]"
- name: Type check
run: python -m pyright
@@ -65,6 +65,9 @@ jobs:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v7
+ - uses: actions/setup-node@v7
+ with:
+ node-version: "24"
- uses: actions/setup-python@v7
with:
python-version: "3.13"
@@ -75,7 +78,7 @@ jobs:
- name: Install (pinned)
run: |
python -m pip install --upgrade pip
- python -m pip install -c .github/goldens-constraints.txt -e ".[test]"
+ python -m pip install -c .github/goldens-constraints.txt -e ".[test,fonts]"
- name: Golden records
env:
GOLDENS_STRICT: "1"
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 3e0f008..43b5f68 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -4,6 +4,39 @@ All notable changes to this project are listed here. Versions follow [Semantic V
over the command line (commands, options, exit statuses and `--json` reports), the configuration keys, the JSON
schemas in `nnnotes.schemas` and the documented Python API. Before 1.0 a minor version can change any of them.
+## [0.3.0](https://github.com/MetaSekaiLab/nnnotes/compare/v0.2.5...v0.3.0) - 2026-10-19
+
+### Compatibility
+
+- Configure persistent storage with `[paths] depot` and optional `[depot] remote`. Manage it with `depot plan`, `depot run`, `depot verify` and `depot gc`.
+- Exported products use `nnnotes.product/1` manifests and shared object references. Browser playback uses ournotes-player 0.2.x.
+- Build named sites with `site build --name NAME -o DIR`. Copy selected source bundles with `bundle pull -o DIR`.
+- The `deck` extra uses ournotes-sim 0.0.12.
+
+### Features
+
+- Report per-object S3 deletion failures during depot collection.
+- Keep PNG and font task results distinct across native compression library implementations.
+- Export episode documents, shader files and cue-sheet audio through depot tasks and retained file layouts.
+- Share cached results when adding remote storage, verify remote-only content, and retain recent results with their dependencies during collection.
+- Validate player package compatibility and publish chart metadata and requested language references consistently.
+- Ship schemas for shared product root documents and prefab objects, with numeric font and transform contracts.
+- Share catalog versions and labels through depot remotes, with verified catalog files fetched on demand.
+- Retain census planning results, emitted task descriptions and standalone task runs as depot collection roots.
+- Store content, task inputs, results and catalog versions in a persistent depot with optional remote storage, verified collection and bounded source downloads.
+- Release recoverable bundle downloads after successful conversion batches under the configured budget and command leases.
+- Assemble Live2D, room, spot, live, story and UI products from shared object tables and explicit manifest references.
+- Generate localized live and story text products with game or supplied fonts, shared glyph atlases and material records.
+- Build named sites through depot product, player read-set, asset and index tasks.
+- Retain unselected regional topics, publish immutable product dependencies in order, and restore complete sites from their saved depot layouts.
+- Offer Live variant products from chart manifests and select story UI/font languages explicitly.
+
+### Architecture
+
+- Site construction uses shared product requests, named depot catalogs and ordered layout publication. Player page collection and catalog model queries have dedicated interfaces.
+- Conversion memoization is bounded within each process, with persistent reuse provided by depot task results.
+- Product assembly consumes stored class-converter outputs. Font lookup, texel packing and material construction share object-based implementations.
+
## [0.2.1](https://github.com/MetaSekaiLab/nnnotes/compare/v0.2.0...v0.2.1) - 2026-10-13
### Build
@@ -195,5 +228,3 @@ schemas in `nnnotes.schemas` and the documented Python API. Before 1.0 a minor v
- List MetaMiku as an author ([cc43e13](https://github.com/MetaSekaiLab/nnnotes/commit/cc43e132875206d02d8665b19c06b27255ab0037))
- Point the project links at MetaSekaiLab/nnnotes ([a2b8304](https://github.com/MetaSekaiLab/nnnotes/commit/a2b83040bcf06771028bbfd5e90c10c225bde3e9))
- English package readme; check commits on pull requests only ([e602dd5](https://github.com/MetaSekaiLab/nnnotes/commit/e602dd5ebf6dda0f18da5ace54a02dbb893a796f))
-
-
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index 807a9d9..7bda8be 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -32,7 +32,9 @@ mkdocs build --strict # the documentation site
- **Golden records.** `tests/golden/` records the output of each stage version. When a change alters a stage's
output, raise that stage's (or converter's) version and rewrite the records with
`GOLDENS_UPDATE=1 pytest tests/pipeline/test_goldens.py`, using the library versions of
- `.github/goldens-constraints.txt` (`pip install -c .github/goldens-constraints.txt -e ".[test]"`).
+ `.github/goldens-constraints.txt` (`pip install -c .github/goldens-constraints.txt -e ".[test,fonts]"`).
+ Node.js runs the synthetic read-set fixture. CI uses Node.js 24 and Python 3.13 for these records.
+ PNG-dependent records also use the native `Pillow.zlib` identity listed in their library versions.
- **Command reference.** `docs/cli.md` is generated from the parser. After changing a command or an option, run
`python -m nnnotes.cli.reference docs/cli.md`; a test fails while the file is out of date.
diff --git a/README.md b/README.md
index 84ce615..84751bd 100644
--- a/README.md
+++ b/README.md
@@ -1,100 +1,99 @@
-# nnnotes？！
+# nnnotes
[English](https://github.com/MetaSekaiLab/nnnotes/blob/main/README.md) | [简体中文](https://github.com/MetaSekaiLab/nnnotes/blob/main/README.zh-CN.md)
-nnnotes is an offline data toolkit for the game files of BanG Dream! Our Notes. It reads Addressables catalogs,
-downloads and decrypts asset bundles, downloads and decodes master data, and exports stories, Live2D models, spots,
-shaders, CRI audio and live charts as structured JSON and common file formats. `nnnotes site build` writes the static
-site that [ournotes-player](https://github.com/empty-sekai/ournotes-player) plays, and `nnnotes music-data` one JSON
-file with every song and chart. The name was inspired by sssekai.
+nnnotes turns your BanG Dream! Our Notes game files into structured data, media and static player sites.
+It reads Addressables catalogs, bundles, master data and APK resources; exports stories, Live2D models, live
+charts, rooms, UI, shaders and audio; and assembles sites for
+[ournotes-player](https://github.com/empty-sekai/ournotes-player).
-This is an unofficial fan project, not affiliated with the game's developer or operator. The repository contains no
-game assets, keys or server addresses: you supply the game files, the keys and the server addresses in your own
-configuration, and the exports stay in the local directories you choose.
+A persistent **depot** stores converted objects, task results and catalog versions. Exports share these results,
+and named sites retain their selected products across builds. A depot can also share work through a directory
+or S3-compatible object storage.
-**Documentation: **
+**[Documentation](https://metasekailab.github.io/nnnotes/)** ·
+[Quick start](https://metasekailab.github.io/nnnotes/quickstart/) ·
+[Commands](https://metasekailab.github.io/nnnotes/cli/)
## Install
+Python 3.11 or newer:
+
```sh
-pip install nnnotes # Python 3.11+
-pip install 'nnnotes[fonts]' # the game's fonts (--fonts game) and the story text of site build
-pip install 'nnnotes[deck]' # the deck model's chart statistics of music-data
+pip install nnnotes==0.3.0
+pip install 'nnnotes[fonts]==0.3.0' # story and UI fonts
+pip install 'nnnotes[remote]==0.3.0' # S3-compatible depot storage
+pip install 'nnnotes[deck]==0.3.0' # music-data chart statistics
```
-nnnotes also needs the game's `base.apk`, decoded master data for the commands that read it, and the external tools
-[vgmstream](https://vgmstream.org/) and [FFmpeg](https://ffmpeg.org/) for audio and video (Node.js 20+ and a built
-ournotes-player for `site build`).
+Choose the extras your commands need; they can be combined as `nnnotes[fonts,remote,deck]==0.3.0`.
+Audio/video conversion uses [vgmstream](https://vgmstream.org/) and [FFmpeg](https://ffmpeg.org/).
+Site builds additionally require Node.js 22+ and a built **ournotes-player 0.2.x** package.
+The deck extra installs `ournotes-sim==0.0.12`.
+
+## First export
-## Quick start
+Supply your own client files, keys and server addresses through the configuration wizard:
```sh
-nnnotes config init --user # asks for each setting (keys without echo) and writes the per-user config file
-nnnotes config check # where each setting comes from and whether it is valid, never its value
+nnnotes config init --user
+nnnotes config check
nnnotes master download --latest -o work/master-bin
nnnotes master decode work/master-bin -o work/master
-nnnotes --master work/master export story 10462 -o out/story_10462
-nnnotes site build -o out/site --all --player path/to/ournotes-player
+nnnotes config set paths.master work/master
+nnnotes export episode 10462 -o out/episode.json
+nnnotes export story 10462 --fonts game -o out/story
+```
+
+Set `paths.depot` to a persistent directory and `paths.apk` to your game APK. The wizard reads keys without echo;
+`config check` reports setting names, origins and validity. Story export requires the `fonts` extra and audio
+tools. See the [quick start](https://metasekailab.github.io/nnnotes/quickstart/) for prerequisites and output layout.
+
+## Build a player site
+
+```sh
+npm install ournotes-player@0.2.0
+nnnotes site build --name stories -o out/site --story 10462 --fonts game --player node_modules/ournotes-player
+node node_modules/ournotes-player/scripts/validate-data.mjs out/site
+python -m http.server 8000 --directory out/site
```
-Every command prints a short summary, or with `--json` its report as JSON. Scripts set the config file without
-questions: `nnnotes config init --user --no-input --set SECTION.KEY=VALUE ...`.
-
-## Commands
-
-| Command | What it does |
-| --- | --- |
-| `config init\|set\|unset\|check\|path` | write, edit and check the config file |
-| `servers` | the game API's server list: regions with their CDN and API roots |
-| `catalog keys\|browse` | the addressable keys of a region's catalog; a local web page to browse catalogs and bundles |
-| `catalog versions\|import\|fetch\|diff` | catalog versions kept in the store, and what changed between two |
-| `bundle pull` | fetch and decrypt the bundle closure of keys into the cache |
-| `master version\|download\|decode` | the master data version a region serves, its download and its decoding to JSON |
-| `export assets` | every object of the selected bundles in common formats, through the content-addressed store |
-| `export episode\|story\|live\|live2d\|spot\|room\|shader\|audio\|ui` | one subject: an ADV episode, a story, a live, a Live2D model, a spot, a room, shaders, a cue sheet, the UI library |
-| `apk settings\|hca-key` | the player's graphics settings and the CRI HCA keycode of `base.apk` |
-| `store plan\|run\|verify` | what `export assets` would run and why, single tasks, the store's check |
-| `voices list\|search\|summary\|get` | the character and story voices of the master data |
-| `site build` | the static site of ournotes-player: charts, Live2D models and stories, in several regions and languages |
-| `music-data` | every song and chart in one JSON file, with the deck model's chart statistics |
-
-The [command reference](https://metasekailab.github.io/nnnotes/cli/) lists every option.
-
-## Documentation
-
-- [Quick start](https://metasekailab.github.io/nnnotes/quickstart/): from an empty config to an exported story.
-- How-to guides: [configure nnnotes](https://metasekailab.github.io/nnnotes/configuration/),
- [export stories and lives](https://metasekailab.github.io/nnnotes/export/),
- [build an ournotes-player site](https://metasekailab.github.io/nnnotes/site/),
- [export every asset](https://metasekailab.github.io/nnnotes/assets/),
- [the Japanese release](https://metasekailab.github.io/nnnotes/jp/),
- [music data](https://metasekailab.github.io/nnnotes/music-data/).
-- Reference: [commands](https://metasekailab.github.io/nnnotes/cli/),
- [command outputs](https://metasekailab.github.io/nnnotes/commands/),
- [JSON schemas](https://metasekailab.github.io/nnnotes/schemas/),
- [Python API](https://metasekailab.github.io/nnnotes/api/),
- [contracts](https://metasekailab.github.io/nnnotes/contracts/), [stages](https://metasekailab.github.io/nnnotes/stages/),
- [views](https://metasekailab.github.io/nnnotes/views/).
-- Explanation: [architecture](https://metasekailab.github.io/nnnotes/architecture/),
- [determinism](https://metasekailab.github.io/nnnotes/determinism/),
- [validation](https://metasekailab.github.io/nnnotes/validation/).
-
-Exports are deterministic for a given installation: the same input with the same versions of nnnotes, its Python
-dependencies (UnityPy, Pillow, numpy, ...) and the external tools gives byte-identical files; JSON is UTF-8 with LF
-line endings and non-finite numbers written as `1e999`. Other versions can encode the same content into other bytes
-(PNG files written by another Pillow version can differ in their bytes while their pixels are equal).
-
-## Changelog and versions
-
-[CHANGELOG.md](https://github.com/MetaSekaiLab/nnnotes/blob/main/CHANGELOG.md) lists the changes of every version.
-Versions follow [Semantic Versioning](https://semver.org/) over the command line, the configuration keys, the JSON
-schemas and the documented Python API.
-
-## Contributing
-
-Setup, checks and commit conventions are in
-[CONTRIBUTING.md](https://github.com/MetaSekaiLab/nnnotes/blob/main/CONTRIBUTING.md).
-
-## License
-
-MIT, see [LICENSE](https://github.com/MetaSekaiLab/nnnotes/blob/main/LICENSE).
+Open `http://localhost:8000/story/`. Live2D, MotionSync and Spine features require the corresponding browser
+runtimes supplied by the embedding page. See the [site guide](https://metasekailab.github.io/nnnotes/site/).
+Use `--pair MUSIC_ID:DIFFICULTY` for a chart, `--live2d KEY` for a model, and `--story-languages ja,en` for selected
+story languages. Reusing a site name updates selected topics and retains the others.
+
+## Choose a workflow
+
+| Task | Commands | Guide |
+|---|---|---|
+| Configure inputs | `config init`, `config check`, `servers` | [Configuration](https://metasekailab.github.io/nnnotes/configuration/) |
+| Inspect or retain catalogs | `catalog keys`, `browse`, `versions`, `import`, `fetch`, `diff` | [Commands](https://metasekailab.github.io/nnnotes/cli/) |
+| Copy source bundles | `bundle pull` | [Asset export](https://metasekailab.github.io/nnnotes/assets/) |
+| Export a subject | `export episode`, `story`, `live`, `live2d`, `spot`, `room`, `shader`, `audio`, `ui` | [Exports](https://metasekailab.github.io/nnnotes/export/) |
+| Convert selected assets | `export assets`, `depot plan`, `depot run` | [Asset export](https://metasekailab.github.io/nnnotes/assets/) |
+| Check and retain stored work | `depot verify`, `depot gc` | [Depot](https://metasekailab.github.io/nnnotes/depot/) |
+| Export song and chart statistics | `music-data` | [Music data](https://metasekailab.github.io/nnnotes/music-data/) |
+| Find character voices | `voices list`, `search`, `summary`, `get` | [Commands](https://metasekailab.github.io/nnnotes/cli/) |
+
+Commands print summaries; `--json` selects a structured report. The reference describes exit statuses and options.
+
+## Data and compatibility
+
+Product exports use versioned manifests and shared object records. Read the
+[format reference](https://metasekailab.github.io/nnnotes/formats/),
+[product reference](https://metasekailab.github.io/nnnotes/products/) and
+[schemas](https://metasekailab.github.io/nnnotes/schemas/) to consume them.
+Task identities include input contents, options and conversion implementations, including native encoders.
+Byte-identical output requires matching inputs and library/tool implementations, including Pillow and its native
+compression backend. JSON uses UTF-8 and LF; infinite values use `1e999` or `-1e999`.
+[Determinism](https://metasekailab.github.io/nnnotes/determinism/) explains reproducibility across installations.
+
+Versions cover the CLI, configuration, schemas and documented Python API. Before 1.0, a minor version can change
+these interfaces. See [CHANGELOG.md](https://github.com/MetaSekaiLab/nnnotes/blob/main/CHANGELOG.md) and the [architecture](https://metasekailab.github.io/nnnotes/architecture/).
+
+## Contributing and license
+
+[CONTRIBUTING.md](https://github.com/MetaSekaiLab/nnnotes/blob/main/CONTRIBUTING.md) describes development and synthetic tests. nnnotes is licensed under
+[MIT](https://github.com/MetaSekaiLab/nnnotes/blob/main/LICENSE). This is an unofficial fan project, unaffiliated with the game's developer or operator.
+Game resources, keys and server configuration are supplied by each user and are subject to their owners' rights.
diff --git a/README.zh-CN.md b/README.zh-CN.md
index 243f5f2..413a3ea 100644
--- a/README.zh-CN.md
+++ b/README.zh-CN.md
@@ -1,96 +1,95 @@
-# nnnotes？！
+# nnnotes
[English](https://github.com/MetaSekaiLab/nnnotes/blob/main/README.md) | [简体中文](https://github.com/MetaSekaiLab/nnnotes/blob/main/README.zh-CN.md)
-nnnotes 是 BanG Dream! Our Notes 游戏文件的离线数据工具包：读取 Addressables catalog，下载并解密资源包，下载并解码
-masterdata，再把剧情、Live2D 模型、据点、着色器、CRI 音频和谱面导出成结构化 JSON 与通用格式文件。
-`nnnotes site build` 写出 [ournotes-player](https://github.com/empty-sekai/ournotes-player) 播放的静态站点，
-`nnnotes music-data` 写出包含全部歌曲与谱面的单个 JSON 文件。命名灵感来自 sssekai。
+nnnotes 将你自己的 BanG Dream! Our Notes 游戏文件导出为结构化数据、媒体文件和静态播放器站点。
+它读取 Addressables 目录、资源包、主数据和 APK 资源，导出剧情、Live2D、谱面、房间、UI、着色器与音频，
+并为 [ournotes-player](https://github.com/empty-sekai/ournotes-player) 组装站点。
-本项目为非官方爱好者项目，与游戏的开发和运营方无关。仓库不包含任何游戏资源、密钥或服务器地址：游戏文件、密钥和服务器地址
-都由使用者在自己的配置中提供，导出结果只保存在使用者指定的本地目录。
+持久化的 **depot** 保存转换后的对象、任务结果和目录版本。各次导出共享这些结果；命名站点在构建之间保留
+所选内容。depot 也可以通过共享目录或兼容 S3 的对象存储在机器之间复用工作。
-**文档（英文）：**
+**[文档](https://metasekailab.github.io/nnnotes/)** ·
+[快速开始](https://metasekailab.github.io/nnnotes/quickstart/) ·
+[命令参考](https://metasekailab.github.io/nnnotes/cli/)
## 安装
+需要 Python 3.11 或更新版本：
+
```sh
-pip install nnnotes # Python 3.11+
-pip install 'nnnotes[fonts]' # 游戏字体（--fonts game）与 site build 的剧情文字
-pip install 'nnnotes[deck]' # music-data 的组卡模型谱面统计
+pip install nnnotes==0.3.0
+pip install 'nnnotes[fonts]==0.3.0' # 剧情与 UI 字体
+pip install 'nnnotes[remote]==0.3.0' # 兼容 S3 的 depot 远端
+pip install 'nnnotes[deck]==0.3.0' # music-data 谱面统计
```
-此外还需要游戏的 `base.apk`；读取 masterdata 的命令需要解码后的 masterdata；音频与视频需要外部工具
-[vgmstream](https://vgmstream.org/) 和 [FFmpeg](https://ffmpeg.org/)（`site build` 另需 Node.js 20+ 与构建好的
-ournotes-player）。
+按需选择附加依赖，也可以组合为 `nnnotes[fonts,remote,deck]==0.3.0`。
+音视频转换需要 [vgmstream](https://vgmstream.org/) 与 [FFmpeg](https://ffmpeg.org/)。
+站点构建另需 Node.js 22+ 和已构建的 **ournotes-player 0.2.x** 包。
+`deck` 附加依赖安装 `ournotes-sim==0.0.12`。
+
+## 第一次导出
-## 快速上手
+通过配置向导提供自己的客户端文件、密钥与服务地址：
```sh
-nnnotes config init --user # 逐项询问设置（密钥不回显），写入用户配置文件
-nnnotes config check # 每项设置的来源与是否有效，不显示设置值
+nnnotes config init --user
+nnnotes config check
nnnotes master download --latest -o work/master-bin
nnnotes master decode work/master-bin -o work/master
-nnnotes --master work/master export story 10462 -o out/story_10462
-nnnotes site build -o out/site --all --player path/to/ournotes-player
+nnnotes config set paths.master work/master
+nnnotes export episode 10462 -o out/episode.json
+nnnotes export story 10462 --fonts game -o out/story
+```
+
+将 `paths.depot` 设为持久目录，`paths.apk` 设为游戏 APK。向导输入密钥时不回显；`config check` 显示配置项名称、
+来源及有效性。剧情导出需要 `fonts` 附加依赖及音频工具。完整前置条件和输出结构见
+[快速开始](https://metasekailab.github.io/nnnotes/quickstart/)。
+
+## 构建播放器站点
+
+```sh
+npm install ournotes-player@0.2.0
+nnnotes site build --name stories -o out/site --story 10462 --fonts game --player node_modules/ournotes-player
+node node_modules/ournotes-player/scripts/validate-data.mjs out/site
+python -m http.server 8000 --directory out/site
```
-每个命令输出简短摘要，加 `--json` 时输出 JSON 报告。脚本中可以不经提问写好配置文件：
-`nnnotes config init --user --no-input --set 段.键=值 ...`。
-
-## 命令
-
-| 命令 | 作用 |
-| --- | --- |
-| `config init\|set\|unset\|check\|path` | 写入、修改和检查配置文件 |
-| `servers` | 游戏 API 的服务器列表：各区服及其 CDN 与 API 根地址 |
-| `catalog keys\|browse` | 区服 catalog 的资源键；在本地网页中浏览 catalog 与资源包 |
-| `catalog versions\|import\|fetch\|diff` | store 中保存的 catalog 版本，以及两个版本之间的变化 |
-| `bundle pull` | 下载并解密资源键的依赖闭包到缓存 |
-| `master version\|download\|decode` | 区服当前的 masterdata 版本、下载，以及解码为 JSON |
-| `export assets` | 经内容寻址的 store，把所选资源包中的每个对象导出为通用格式 |
-| `export episode\|story\|live\|live2d\|spot\|room\|shader\|audio\|ui` | 单个对象：ADV 剧情集、完整剧情、谱面、Live2D 模型、据点、房间、着色器、cue sheet、UI 库 |
-| `apk settings\|hca-key` | `base.apk` 中的画面设置与 CRI HCA 密钥 |
-| `store plan\|run\|verify` | `export assets` 将运行什么及原因、单独运行任务、检查 store |
-| `voices list\|search\|summary\|get` | masterdata 中的角色语音与剧情语音 |
-| `site build` | ournotes-player 静态站点：谱面、Live2D 模型与剧情，支持多个区服与语言 |
-| `music-data` | 全部歌曲与谱面的单个 JSON 文件，附组卡模型的谱面统计 |
-
-[命令参考](https://metasekailab.github.io/nnnotes/cli/)列出了所有参数。
-
-## 文档
-
-- [快速上手](https://metasekailab.github.io/nnnotes/quickstart/)：从空配置到导出一集剧情。
-- 操作指南：[配置](https://metasekailab.github.io/nnnotes/configuration/)、
- [导出剧情与谱面](https://metasekailab.github.io/nnnotes/export/)、
- [构建 ournotes-player 站点](https://metasekailab.github.io/nnnotes/site/)、
- [导出全部资源](https://metasekailab.github.io/nnnotes/assets/)、
- [日服](https://metasekailab.github.io/nnnotes/jp/)、
- [music-data](https://metasekailab.github.io/nnnotes/music-data/)。
-- 参考：[命令](https://metasekailab.github.io/nnnotes/cli/)、
- [命令输出](https://metasekailab.github.io/nnnotes/commands/)、
- [JSON schema](https://metasekailab.github.io/nnnotes/schemas/)、
- [Python API](https://metasekailab.github.io/nnnotes/api/)、
- [契约](https://metasekailab.github.io/nnnotes/contracts/)、[stage](https://metasekailab.github.io/nnnotes/stages/)、
- [视图](https://metasekailab.github.io/nnnotes/views/)。
-- 原理：[架构](https://metasekailab.github.io/nnnotes/architecture/)、
- [确定性](https://metasekailab.github.io/nnnotes/determinism/)、
- [验证](https://metasekailab.github.io/nnnotes/validation/)。
-
-在同一套环境下导出是确定的：同样的输入，在 nnnotes、Python 依赖库（UnityPy、Pillow、numpy 等）和外部工具版本都相同时，
-导出文件逐字节相同；JSON 一律 UTF-8、LF 换行，非有限数写作 `1e999`。版本不同时，相同内容可能写成不同的字节（不同版本的
-Pillow 写出的 PNG 像素相同，字节可能不同）。
-
-## 版本记录与版本号
-
-每个版本的改动见 [CHANGELOG.md](https://github.com/MetaSekaiLab/nnnotes/blob/main/CHANGELOG.md)。版本号遵循
-[语义化版本](https://semver.org/lang/zh-CN/)，约束的接口是命令行、配置键、JSON schema 和文档中的 Python API。
-
-## 参与贡献
-
-开发环境、检查与提交规范见
-[CONTRIBUTING.md](https://github.com/MetaSekaiLab/nnnotes/blob/main/CONTRIBUTING.md)。
-
-## License
-
-MIT，见 [LICENSE](https://github.com/MetaSekaiLab/nnnotes/blob/main/LICENSE)。
+打开 `http://localhost:8000/story/`。Live2D、MotionSync 和 Spine 功能需要嵌入页面提供对应的浏览器运行时，
+具体见[站点指南](https://metasekailab.github.io/nnnotes/site/)。
+用 `--pair MUSIC_ID:DIFFICULTY` 选择谱面，`--live2d KEY` 选择模型，`--story-languages ja,en` 选择剧情语言。
+复用同一站点名会更新本次选中的内容，并保留其他内容。
+
+## 选择工作流
+
+| 任务 | 命令 | 指南 |
+|---|---|---|
+| 配置输入 | `config init`、`config check`、`servers` | [配置](https://metasekailab.github.io/nnnotes/configuration/) |
+| 查看与保存目录 | `catalog keys`、`browse`、`versions`、`import`、`fetch`、`diff` | [命令](https://metasekailab.github.io/nnnotes/cli/) |
+| 复制源资源包 | `bundle pull` | [资源导出](https://metasekailab.github.io/nnnotes/assets/) |
+| 导出主题 | `export episode`、`story`、`live`、`live2d`、`spot`、`room`、`shader`、`audio`、`ui` | [导出](https://metasekailab.github.io/nnnotes/export/) |
+| 转换指定资源 | `export assets`、`depot plan`、`depot run` | [资源导出](https://metasekailab.github.io/nnnotes/assets/) |
+| 检查与保留存储结果 | `depot verify`、`depot gc` | [Depot](https://metasekailab.github.io/nnnotes/depot/) |
+| 导出歌曲与谱面统计 | `music-data` | [音乐数据](https://metasekailab.github.io/nnnotes/music-data/) |
+| 查找角色语音 | `voices list`、`search`、`summary`、`get` | [命令](https://metasekailab.github.io/nnnotes/cli/) |
+
+命令默认打印摘要；`--json` 输出结构化报告。退出状态和各项参数见命令参考。
+
+## 数据与兼容性
+
+产物通过版本化清单和共享对象记录描述。消费这些数据时，请查阅
+[格式](https://metasekailab.github.io/nnnotes/formats/)、[产物](https://metasekailab.github.io/nnnotes/products/)和
+[Schema](https://metasekailab.github.io/nnnotes/schemas/)参考。
+任务标识包括输入内容、选项和转换实现，其中也包括原生编码器。
+字节一致的输出需要相同输入及库和工具实现，其中包括 Pillow 及其原生压缩后端。
+JSON 使用 UTF-8 和 LF，无穷值写为 `1e999` 或 `-1e999`。
+[确定性说明](https://metasekailab.github.io/nnnotes/determinism/)解释不同安装环境中的可复现性。
+
+版本覆盖 CLI、配置、Schema 和文档中的 Python API。1.0 之前，次版本可以改变这些接口。
+改动见 [CHANGELOG.md](https://github.com/MetaSekaiLab/nnnotes/blob/main/CHANGELOG.md)，实现分层见[架构](https://metasekailab.github.io/nnnotes/architecture/)。
+
+## 贡献与许可
+
+[CONTRIBUTING.md](https://github.com/MetaSekaiLab/nnnotes/blob/main/CONTRIBUTING.md) 说明开发和合成测试流程。nnnotes 采用 [MIT](https://github.com/MetaSekaiLab/nnnotes/blob/main/LICENSE) 许可。
+这是非官方爱好者项目，与游戏开发或运营方无关。游戏资源、密钥和服务配置由各用户自行提供，并受其权利人的权利约束。
diff --git a/docs/api.md b/docs/api.md
index 9b04a4d..ecb2d1a 100644
--- a/docs/api.md
+++ b/docs/api.md
@@ -38,6 +38,5 @@ raises `GameApiError`. Other names are internal and can change in any version.
## Site and schemas
-::: nnnotes.site.webmodel
::: nnnotes.schemas
diff --git a/docs/architecture.md b/docs/architecture.md
index e259fe3..a257b19 100644
--- a/docs/architecture.md
+++ b/docs/architecture.md
@@ -10,12 +10,12 @@ subpackage of `nnnotes` and imports only the layers below it.
|---|---|---|
| Command line | `nnnotes.cli` | one module per command group; parses options, reads the configuration, prints summaries and `--json` reports |
| Products | `nnnotes.site`, `nnnotes.musicdata` | the ournotes-player site (charts, models, stories, indexes); the music data file and its deck statistics |
-| Pipeline | `nnnotes.pipeline` | stages, task descriptions and results, the content-addressed store, plans, layouts, the catalog versions, the views of the master data |
+| Pipeline | `nnnotes.pipeline` | stages, task descriptions and results, the content-addressed depot, plans, layouts, the catalog versions, the views of the master data |
| Content | `nnnotes.content` | the game's subjects as exports: ADV episodes and stories, lives and scores, spots, rooms, the UI library |
-| Unity objects | `nnnotes.unity` | bundle reading, the typetree exporter, the player data, Live2D prefabs, text styles and TextMesh Pro fonts |
+| Unity objects | `nnnotes.unity` | bundle reading, object ids, atomic texture/sprite/mesh/glTF/PNG conversions, class converters (`nnnotes.unity.convert`), clip bindings (`nnnotes.unity.bindings`), script type trees (`nnnotes.unity.scripttypes`), player data, text styles, font protocols and texel packing |
| File formats | `nnnotes.formats` | formats read from bytes: CRI ACB/AWB and HCA, USM movies, CRI Lips data, shaders, motions |
| Sources | `nnnotes.source` | the configuration, the APK, Addressables catalogs and bundles, the game API, master data, languages, `Session` |
-| Core | `nnnotes.core` | the one JSON writer and the process caches |
+| Core | `nnnotes.core` | the one JSON writer, canonical JSON and its hashes, media types, the process caches and the downloads cache |
The order is a contract: the `layers` contract of [import-linter](https://import-linter.readthedocs.io/) in
`pyproject.toml` fails the build when a module imports a layer above its own, and `nnnotes.site` and
@@ -27,27 +27,36 @@ and print, and the work they run is in the library modules below them.
Every setting comes from the configuration ([sources and precedence](configuration.md#sources-and-precedence)).
`nnnotes.source.session.Session` turns one configuration into the data sources a command needs: the catalog of a
region, its decoded master data and the master data version it serves, the APK, the bundle and master data keys and
-the store root. It reads a setting when it is first needed and raises a `ConfigError` naming the setting when it is missing or wrong;
+the depot root. It reads a setting when it is first needed and raises a `ConfigError` naming the setting when it is missing or wrong;
the command line turns that error into exit status 2.
The catalog (`nnnotes.source.catalog.Catalog`) resolves an addressable key to its bundle and the bundle's
dependency closure, takes a bundle from the APK when the APK has it and from the CDN otherwise, and keeps fetched and
-decrypted bundles in the cache directory.
-
-## Two ways to export
-
-- **One subject.** `export episode`, `export story`, `export live`, `export live2d`, `export spot`, `export room`,
- `export shader`, `export audio`, `export ui`, `site build` and `music-data` read what one subject needs through the
- Session and write their output directory directly ([Export stories and lives](export.md)).
-- **Every asset.** `export assets` splits the work into stages ([stages](stages.md)). A task is one stage applied to
- one subject (a bundle, a content id, a view); its key covers exactly what its output depends on: the stage and
- converter versions, the input contents and the options ([contracts](contracts.md)). Results and their files are
- stored once by content hash in the store, and the output directory is laid out from the store. A second run finds
- every unchanged result in the store and only lays out what changed; `store plan` says what would run and why
- ([asset export](assets.md)).
-
-Both ways use the same readers below the pipeline layer, so a texture, shader, mesh or audio stream is converted by
-one implementation whichever command asks for it.
+decrypted bundles in the depot's `downloads/` directory.
+
+## Product and asset requests
+
+`export assets` runs stages whose tasks are keyed by their input contents, options, stage versions and converter identities. Results and files are stored in the depot; layouts place their content into output directories. `depot plan` reports which tasks and placements would change.
+
+`export live`, `export story`, `export live2d`, `export spot`, `export room` and `export ui` request the products of selected subjects through that same pipeline. Stories first produce an episode, then request its referenced products. Shared objects and unchanged products reuse committed tasks.
+
+Class converters emit object records and binary files. Product assemblers read those stored records through an object table and identify shared dependencies by object id and product manifest. Text products use the same font lookup, material and in-memory atlas functions. A product request contains its complete object and media dependencies before assembly.
+
+`site build` uses the same product requests, then runs player read-set, asset, document and index tasks. A named catalog retains unselected topics and references immutable products. Publication writes dependencies before entry manifests and indexes. The successful run stores the complete layout so a new output directory can be restored through the depot.
+
+The site's `Builder` composes product snapshots, catalog indexes and player page files. Product assembly and conversion belong to the pipeline and content layers. `catalog_models` queries model subjects from a source catalog.
+
+Converters use bounded process-local memoization for repeated reads and computations. Persistent reuse is keyed by depot tasks and their content inputs. Clearing a process memo releases its values; a later task can reuse its committed depot result.
+
+The coordinator observes successful object batches after worker readers close. Their recoverable bundle downloads
+can then be released toward the configured budget under the command's exclusive lease. Preliminary episode
+requests retain inputs until the expanded product request is known. Stored conversion results remain available
+for assembly and layouts.
+
+`export episode`, `export shader` and `export audio` request depot tasks and project their results into files.
+Episode assembly reads the common object table, shader files reuse converted Shader artifacts, and audio files
+reuse `cri.audio` streams. `music-data` uses Session to obtain catalog and master inputs for its tabular export.
+Commands record their accessed depot content and results as retention roots; dry runs leave no command record.
## Outputs
diff --git a/docs/assets.md b/docs/assets.md
index 4e089bf..33c23a5 100644
--- a/docs/assets.md
+++ b/docs/assets.md
@@ -5,29 +5,29 @@ meshes as binary glTF, text assets and fonts as their stored bytes, prefabs, mat
class as JSON. It also writes the semantic views of the master data (cards, characters, stamps, …) with the files of
the objects they name. The work is split into stages ([stages.md](stages.md)); each stage runs one task per subject
(a bundle, a view, …), and a task's result is stored under a key that covers exactly what its output depends on
-([contracts.md](contracts.md)). A second run over the same data finds every result in the store and only
+([contracts.md](contracts.md)). A second run over the same data finds every result in the depot and only
rewrites what changed.
```
nnnotes export assets -o out/assets
nnnotes export assets -o out/cards --select group:membercard --views cards
-nnnotes store plan --select key:Image/Jacket/
-nnnotes store plan --check && echo "up to date"
+nnnotes depot plan --select key:Image/Jacket/
+nnnotes depot plan --check && echo "up to date"
```
## Commands
```
-nnnotes export assets -o OUT [--layout original[,cas]] [--select group:G|key:PREFIX|bundle:GLOB ...] [--views all|none|a,b]
- [--store DIR] [--workers N] [--memory GiB] [--fetch-workers N] [--stored-catalog L|SHA]
+nnnotes export assets -o OUT [--layout original[,cas][,product]] [--select group:G|key:PREFIX|bundle:GLOB ...] [--views all|none|a,b]
+ [--depot DIR] [--workers N] [--memory GiB] [--fetch-workers N] [--stored-catalog L|SHA]
[--png-level N] [--flac-level N] [--only-class C,...] [--link auto|clone|hard|copy] [--strict]
[--dry-run] [--explain]
-nnnotes store plan [the selection and parameters of export] [-o OUT] [--json] [--since RUN] [--check] [--census]
+nnnotes depot plan [the selection and parameters of export] [-o OUT] [--json] [--since RUN] [--check] [--census]
[--emit-tasks DIR] [--why TASK]
-nnnotes store run TASK.json|DIR [...] [--store DIR] [--fetch] [--force]
+nnnotes depot run TASK.json|DIR [...] [--depot DIR] [--fetch] [--force]
nnnotes catalog versions [--json] | import FILE [--label L] [--apk-catalog FILE] [--resource-version V] | fetch [--label L]
| diff OLD NEW [--json]
-nnnotes store verify [--quick] [--workers N]
+nnnotes depot verify [--quick] [--workers N]
```
### export assets
@@ -37,43 +37,48 @@ nnnotes store verify [--quick] [--workers N]
| `-o OUT` | output directory |
| `--select SEL` | what to export (repeatable, the union; default everything): `group:G` the bundles of an Addressables group (the stable bundle name up to `_assets_`, else up to its first `_`), `key:PREFIX` the bundles and raw files the catalog keys under `PREFIX` load, `bundle:GLOB` the bundles and raw files whose stable name matches |
| `--views` | the semantic views to build ([views.md](views.md)): `all`, `none` or names, comma-separated; default `all` when master data is set, else `none` |
-| `--layout` | `original`, `cas` or both (below) |
-| `--store DIR` | the store (`[paths] store`; default `/store`) |
+| `--layout` | `original`, `cas`, `product`, comma-separated (below) |
+| `--depot DIR` | the depot (`[paths] depot`) |
| `--workers N` | worker processes (default: the CPUs the process may run on, its affinity mask as `taskset` or a container's cpuset sets it; `0`: every task in this process) |
| `--memory GiB` | memory budget (default: 80 % of the physical memory where it can be read, else none) |
-| `--fetch-workers N` | parallel downloads of bundles not in the cache (default 8) |
-| `--stored-catalog` | a catalog version of the store (`catalog versions`) instead of the current catalog |
+| `--fetch-workers N` | parallel downloads of bundles not downloaded yet (default 8) |
+| `--stored-catalog` | a catalog version of the depot (`catalog versions`) instead of the current catalog |
| `--png-level N` | zlib level 0-9 of the PNG files (default 6); part of the task keys |
| `--flac-level N` | FLAC compression level 0-12 of the CRI audio and the movies' audio (default 8; every level decodes to the same samples; 12 writes files 0.2-1.3 % smaller in about five times the encoding time); part of the task keys |
-| `--only-class C,...` | export only objects of these Unity classes; part of the task keys |
-| `--link` | how the layout files are made from the store (below): `auto` (default; `[export] link`), `clone`, `hard` or `copy` |
+| `--only-class C,...` | export only the units whose root object is of these Unity classes (a prefab by `GameObject`, [stages.md](stages.md#units)) |
+| `--link` | how the layout files are made from the depot (below): `auto` (default; `[export] link`), `clone`, `hard` or `copy` |
| `--strict` | exit 1 when a view has gaps: a required role whose address names no catalog key or no fitting sub-object (empty values and roles that do not apply are not gaps); in `voices`, a voice whose cue sheet has no catalog key or whose decoded sheet lacks its cue, and a story episode without a catalog key |
| `--dry-run` | print the plan and stop |
-| `--explain` | print every setting of the run first: store, context, selection, pipeline, stages not installed, parameters, workers, memory budget, recycling, layouts |
+| `--explain` | print every setting of the run first: depot, context, selection, pipeline, stages not installed, parameters, workers, memory budget, recycling, layouts |
A run resolves the context (region, language, catalog version) and indexes the catalog, identifies the selected
-bundles (the sha256 of each decrypted bundle, remembered by path, size and modification time; bundles not in the
-cache are fetched), takes the census of each bundle, links the scripts and addresses, exports the bundles, runs the
+bundles (the sha256 of each decrypted bundle, remembered by path, size and modification time; bundles not
+downloaded yet are fetched into the depot's `downloads/`), takes the census of each bundle, links the scripts and
+addresses, exports the units (one task per hierarchy or object outside any, [stages.md](stages.md#units)), runs the
stages that need those results (atlas sprites, views), writes the layouts and the reports. The selected raw files
-(CRI audio and movies) not in the cache are fetched while the bundle stages run. The census also covers
-the bundles the selected ones depend on, so script classes and atlas textures in other bundles resolve.
-
-Tasks run in spawned worker processes that stay alive between tasks (no more of them start than a stage has tasks to
-run). The longest tasks (by estimated CPU time) start first; a task starts only while the estimated peak memory of
-the running tasks and its own fits the budget, and a task estimated above the budget runs alone. A worker is
-replaced after 500 tasks or when its resident memory after a task is above 3 GiB, or above 40 % of its part of the
-memory budget (the budget divided by the workers) when that is less, but not below 256 MiB: a worker keeps much of
-its largest task's memory, which the budget does not count (`--explain` prints the threshold). The estimates come
-from each stage's cost model, scaled per stage by the costs measured in earlier runs of the store
-(`/costs/calibration.json`, updated after every export; a task's measured peak counts from its worker's
-memory at its start). None of this changes an output: a run with one worker and a run with many write the same store
+(CRI audio and movies) not downloaded yet are fetched while the bundle stages run. The census also covers
+the bundles the selected ones depend on, so script classes and atlas textures in other bundles resolve, and the
+export covers the units of those bundles that the selected ones reference.
+
+Tasks run in spawned worker processes that stay alive between tasks. The tasks of a stage that read the same
+loaded data (a stage's batch, such as one bundle) form one job: one worker runs them one after another and loads the
+data once. Every other task is a job alone. No more workers start than a stage has jobs to run. The longest jobs (by
+estimated CPU time, the sum over their tasks) start first; a job starts only while the estimated peak memory of the
+running jobs and its own (the largest of its tasks') fits the budget, and a job estimated above the budget runs
+alone. A worker is replaced after a job once it has run 500 tasks, or when its resident memory at the end of a job
+is above 3 GiB, or above 40 % of its part of the memory budget (the budget divided by the workers) when that is
+less, but not below 256 MiB: a worker keeps much of its largest task's memory, which the budget does not count
+(`--explain` prints the threshold). The estimates come
+from each stage's cost model, scaled per stage by the costs measured in earlier runs of the depot
+(`/costs/calibration.json`, updated after every export; a task's measured peak counts from its worker's
+memory at its start). None of this changes an output: a run with one worker and a run with many write the same depot
and the same files.
A failed object is reported with its task's result and stays failed until something in its key changes. A task that
fails as a whole (an exception, an input that cannot be read, a worker that dies) leaves no result and runs again
-next time.
+next time; when a worker dies, the tasks of its job after the failed one run in another worker.
-### store plan
+### depot plan
What `export assets` with the same flags would do, without running or fetching anything: every task with its status
(`hit`: its result is stored, `run`, `unknown`: it reads the result of a task that would run first, or of a bundle
@@ -86,25 +91,32 @@ gives per stage its counts and its first 20 tasks that are not hits; `--json` ha
| `--json` | the plan document (`nnnotes.plan/1`) |
| `--check` | exit 1 when anything would run or a layout file would change, else 0 |
| `--census` | first fetch the bundles without a census and take it, so the plan knows every bundle's classes |
-| `--emit-tasks DIR` | write the tasks that can run now as task descriptions `DIR/.json` |
+| `--emit-tasks DIR` | write the tasks that can run now as task descriptions `DIR/.json`; retain their layout and stored input dependencies in the depot |
| `--why TASK` | one task: its status, reasons, and its key parts beside those of the previous run |
Reasons: `new-input`, `input-changed`, `stage-version`, `atom-changed`, `params-changed`, `context-changed`,
-`previous-failed`, `result-missing`, `forced`; hits are `unchanged` or `cached` (the result is in the store from
+`previous-failed`, `result-missing`, `forced`; hits are `unchanged` or `cached` (the result is in the depot from
another run); unknown tasks are `pending` on a task, or on `fetch:` for a bundle whose content is not known
yet. See [contracts.md](contracts.md#plan-nnnotesplan1).
-### store run
+Census planning writes a run record for its completed tasks. Emitted descriptions have a saved task layout whose
+input references keep stored inputs reachable during collection. Interrupted pipeline runs also retain completed
+task results in their run records.
-Runs task descriptions (`nnnotes.task/1`, as `plan --emit-tasks` writes them) into a store: each is checked (its key
+### depot run
+
+Runs task descriptions (`nnnotes.task/1`, as `plan --emit-tasks` writes them) into a depot: each is checked (its key
is recomputed, its stage must be installed at its version), its inputs are read by content id through their
-locators (the store, the cache, a file, or with `--fetch` the catalog), and its result is stored exactly as `export assets`
-would store it. Several tasks run one after another in one process; stages that do not read bundles never load
+locators (the depot, its downloads, a file, or with `--fetch` the catalog), and its result is stored exactly as
+`export assets` would store it. Several tasks run one after another in one process; stages that do not read bundles never load
UnityPy. `--force` runs tasks whose result is stored again (the new result must be identical).
+Each attempted task has a run record, selected by `depot.run` and its task id. The JSON report's `runs` array names
+these records. Collection applies `--keep` separately to the recorded versions of each task.
+
### catalog versions, import, fetch, diff
-The catalog versions of a store: `import` a catalog file (with the APK's catalog from `[paths] apk` or
+The catalog versions of a depot: `import` a catalog file (with the APK's catalog from `[paths] apk` or
`--apk-catalog`), `fetch` the current catalog of the configured region and language, `list` them, and `diff` two of
them (labels, version ids or sha256 prefixes): keys added, removed and changed, bundles and raw files by stable name,
the keys that reach a changed file, and, when a run over each version has stored its address table, the objects by
@@ -112,14 +124,27 @@ stable address (an object keeps its stable address while its object id changes w
is labelled by `--label`, else by its resource version, else by the first 12 hex digits of its sha256. `fetch`
records the resource version it downloads (none for `main`); `import` takes `--resource-version`, else the JP
catalog's source version, else the version in a file name `catalog__.bin` as `fetch` and the
-cache name them (none for `main`). `export assets` imports the current catalog when it is not imported yet.
+downloads name them (none for `main`). `export assets` imports the current catalog when it is not imported yet.
-### store verify
+### depot verify
Checks that every stored content has the sha256 its name says (`--quick`: sizes only) and that every result names
its own key, has the key of its parts and names stored contents of the right size. Exit 1 when a problem is found. `--workers` hashes in
parallel (default: the CPUs the process may use; `0`: one at a time).
+### depot gc
+
+Removes what no kept run reaches. The kept runs are, for each selection, the `--keep N` latest (default: every run;
+the latest run of a selection is always kept). A kept run reaches the results of its tasks, the contents those
+results name (their artifacts, and the inputs their task keys name when the depot holds them) and its layout
+manifests with the files they list. Results and contents younger than `--grace SECONDS` (default 86400) also stay;
+a recent result keeps its older inputs and artifacts. Other runs, results (with their measured costs) and contents are removed.
+`--dry-run` counts what would be removed. A layout made of hard links or clones keeps its files. `--remote`
+collects the depot's remote the same way, by the runs on the remote and the same grace period. Remote collection
+does not populate the local cache. gc runs only when
+no other command uses the depot (it exits with an error otherwise), and commands that start meanwhile wait for it.
+The downloads are bounded separately, by `[depot] downloads` ([commands.md](commands.md#downloads)).
+
## Layouts
A layout places the artifacts of a run as files; it never changes their bytes. With one layout the files are
@@ -135,7 +160,8 @@ no longer wanted, never touches other files, and continues an interrupted write
- other objects under the same container path: `stem[].`, the Addressables sub-object form
(`…/member_full[square].png`);
- objects without a container path: `_bundles///~.`;
-- secondary artifacts beside the primary one: `..` (`….meta.json`, `….blob.m_Data.moc3`);
+- secondary artifacts beside the primary one: `..` (`….blob.m_Data.moc3`), `.` for a
+ role that is its extension (`….png.json`, the `json` of a texture or sprite);
- a container path present in two bundles: `stem@.ext` for each;
- names Windows refuses, reserved device names and trailing dots are escaped as `%XX`; paths that coincide ignoring
case get `~<8 hex>` and are listed in the layout report;
@@ -150,28 +176,34 @@ manifest; the run manifest and the printed summary name the method and why it wa
| `--link` | the files | extra space | an edit of a file |
|---|---|---|---|
-| `clone` | clones (reflinks) of the store objects: Linux FICLONE (Btrfs, XFS, bcachefs, …), macOS APFS `clonefile` | none until a file is written (a quota that counts each file's blocks, such as an XFS project quota, counts them in full) | changes that file only |
-| `hard` | hard links to the store objects (same file system) | none | fails, except for the superuser: the store objects are read-only (on Windows a delete fails too) |
+| `clone` | clones (reflinks) of the depot objects: Linux FICLONE (Btrfs, XFS, bcachefs, …), macOS APFS `clonefile` | none until a file is written (a quota that counts each file's blocks, such as an XFS project quota, counts them in full) | changes that file only |
+| `hard` | hard links to the depot objects (same file system) | none | fails, except for the superuser: the depot objects are read-only (on Windows a delete fails too) |
| `copy` | copies | the whole layout again | changes that file only |
| `auto` (default) | `clone` where the file system makes clones, else `hard` on the same file system, else `copy` | | |
`auto` tries each method once, on a temporary file, before the run starts. An explicit `clone` or `hard` that does
not work there is a usage error (exit 2) before anything runs; a file that fails with the chosen method is reported
(summary, layout report, exit 1), left out of the manifest and placed by the next run: no file falls back to
-another method. The store writes its objects read-only; a layout clears the attribute of a file it owns before it
-replaces or removes it where the system needs that (Windows), and the store object stays read-only.
+another method. The depot writes its objects read-only; a layout clears the attribute of a file it owns before it
+replaces or removes it where the system needs that (Windows), and the depot object stays read-only.
`cas` is `assets/.` per distinct content, plus the manifest. Each layout follows from the other: `cas`
from the `original` manifest alone, `original` from `cas` and the results. The views are the exception: `original`
has them with the paths of their objects, `cas` the view documents they are made from.
+`product` places the products of the selection ([products](products.md)): every product stage makes its products
+of the catalog keys the selected bundles hold (the Live2D stages: each model key), and the layout writes each
+product's manifest (`models/.json`, `cubism/.json`) with its files and the files of its objects at
+`assets/.`. The product stages run only with this layout. Without `original` or `cas`, the run exports
+only the objects the products reach, not every object of the selection.
+
## Reports
`OUT/_reports/`:
| File | Content |
|---|---|
-| `run.json` | the run manifest (`nnnotes.run/1`): context, selection, parameters, stage versions, every task with its key and status, the layout manifests, a summary; also stored as `/runs/.json` |
+| `run.json` | the run manifest (`nnnotes.run/1`): context, selection, parameters, stage versions, every task with its key and status, the layout manifests, a summary; also stored as `/runs/.json` |
| `coverage.json` | per class the census total and the count of every item status (`exported`, `contained`, `generic`, `unsupported`, `failed`, and `missing`: objects no task covered; an object two tasks cover, such as a sprite whose image `sprite.crop` makes, counts once); per reason code its count and five examples |
| `failures.json` | failed tasks and failed objects, with their codes and messages |
| `sources.json` | files of the selection that could not be read: `source.absent` (a file the catalog names that is not there, or an APK file without `[paths] apk`), `source.error` (the download failed) |
@@ -185,28 +217,28 @@ and the reports directory.
| Command | 0 | 1 | 2 | 3 | 4 |
|---|---|---|---|---|---|
| `export assets` | complete (unsupported objects allowed) | a task, an object or a download failed (reports written); with `--strict` also view gaps | usage or settings | aborted: a setting or tool a task needs is missing, or interrupted; results written so far stay valid | |
-| `store plan` | | with `--check`: something would run or change | usage or settings | | |
-| `store run` | every task ran or hit | a task failed or its result is partial | usage or settings | | a task is incompatible (edited, unknown stage or version); nothing of the batch runs |
-| `store verify` | no problem | problems found | usage or settings | | |
+| `depot plan` | | with `--check`: something would run or change | usage or settings | | |
+| `depot run` | every task ran or hit | a task failed or its result is partial | usage or settings | | a task is incompatible (edited, unknown stage or version); nothing of the batch runs |
+| `depot verify` | no problem | problems found | usage or settings | | |
## Running tasks elsewhere
-Task descriptions carry everything a task needs; any machine with the same nnnotes version and access to the store
+Task descriptions carry everything a task needs; any machine with the same nnnotes version and access to the depot
can run them. A task can only be described once the content of its inputs is known, so the bundles are identified
first (`plan --census` fetches and hashes them and takes their census). Then plan the tasks that can run now, run
them in batches on as many workers as you like, and plan again until nothing is left; `export assets` finally writes the
layouts and reports (every task is a hit by then).
```sh
-store=/data/nnnotes-store
-nnnotes store plan --store "$store" --census > /dev/null
+depot=/data/nnnotes-depot
+nnnotes depot plan --depot "$depot" --census > /dev/null
for round in 1 2 3 4 5 6 7 8; do
rm -rf tasks
- nnnotes store plan --store "$store" --emit-tasks tasks > /dev/null
+ nnnotes depot plan --depot "$depot" --emit-tasks tasks > /dev/null
ls tasks/*.json > /dev/null 2>&1 || break # nothing left to run
- ls tasks/*.json | xargs -n 50 -P 16 nnnotes store run --store "$store" --fetch
+ ls tasks/*.json | xargs -n 50 -P 16 nnnotes depot run --depot "$depot" --fetch
done
-nnnotes export assets --store "$store" -o out/assets
+nnnotes export assets --depot "$depot" -o out/assets
```
Each round runs the tasks whose inputs are known: the catalog index and the censuses, then the script and address
diff --git a/docs/cli.md b/docs/cli.md
index 1b38fe7..e7de8bd 100644
--- a/docs/cli.md
+++ b/docs/cli.md
@@ -11,10 +11,9 @@ to standard error. Exit status 2 is a usage or settings error.
| `--config` `FILE` | TOML config file (else NNNOTES_CONFIG, else ./nnnotes.toml) |
| `--region` `REGION` | region: a [servers.] table ([catalog] region) |
| `--language` `LANG` | catalog and client language: ja, en, zh-Hant, zh-Hans or ko ([catalog] language) |
-| `--catalog` `FILE` | catalog .bin file ([paths] catalog; else downloaded into the cache) |
+| `--catalog` `FILE` | catalog .bin file ([paths] catalog; else downloaded into the depot) |
| `--catalog-version` `VERSION` | the resource version of an international catalog ([catalog] version; else the version the game API serves when an API root is configured, otherwise main) |
-| `--cache` `DIR` | cache directory ([paths] cache) |
-| `--store` `DIR` | store directory ([paths] store; default /store) |
+| `--depot` `DIR` | depot directory ([paths] depot) |
| `--master` `DIR` | decoded master data directory ([paths] master) |
| `--apk` `FILE` | base.apk ([paths] apk) |
| `--ffmpeg` `EXE` | ffmpeg executable ([paths] ffmpeg; else on PATH) |
@@ -97,7 +96,7 @@ browse the configured regions' catalogs and bundles in a local web page
## nnnotes catalog versions
-the catalog versions imported into the store (--json: nnnotes.catalogs/1)
+the catalog versions imported into the depot (--json: nnnotes.catalogs/1)
## nnnotes catalog import
@@ -112,7 +111,7 @@ import a catalog file (with the APK's catalog when [paths] apk is set)
## nnnotes catalog fetch
-fetch the region's current catalog and import it into the store
+fetch the region's current catalog and import it into the depot
| Argument | Description |
| --- | --- |
@@ -120,7 +119,7 @@ fetch the region's current catalog and import it into the store
## nnnotes catalog diff
-keys, bundles, raw files (and objects) that differ between two catalog versions of the store (--json: nnnotes.catalog-diff/1)
+keys, bundles, raw files (and objects) that differ between two catalog versions of the depot (--json: nnnotes.catalog-diff/1)
| Argument | Description |
| --- | --- |
@@ -129,11 +128,12 @@ keys, bundles, raw files (and objects) that differ between two catalog versions
## nnnotes bundle pull
-fetch the bundle closure of keys into the cache (prints the cached bundle files)
+fetch the bundle closure of keys and copy the decrypted bundles to a directory (prints the bundle files)
| Argument | Description |
| --- | --- |
| `KEY` ... | addressable keys (`catalog keys` lists them) |
+| `-o`, `--out` `OUT` | directory the bundles are copied to, under their file names (required) |
## nnnotes master version
@@ -164,23 +164,23 @@ Exactly one of `--version`, `--latest`.
## nnnotes export assets
-every object of the selected bundles -> common formats (the store and its layouts)
+every object of the selected bundles -> common formats (the depot and its layouts)
| Argument | Description |
| --- | --- |
| `-o`, `--out` `OUT` | output directory (required) |
| `--select` `SEL` | group:, key: or bundle: (repeatable; default: everything) |
| `--views` `all\|none\|a,b` | semantic views to build (default: all when master data is set, else none) |
-| `--stored-catalog` `LABEL\|SHA` | a catalog version of the store (`catalog versions`; default: the current catalog) |
+| `--stored-catalog` `LABEL\|SHA` | a catalog version of the depot (`catalog versions`; default: the current catalog) |
| `--png-level` `N` | PNG compression level 0-9 (changes the output) |
| `--flac-level` `N` | FLAC compression level 0-12 of the CRI audio (default 8; every level decodes to the same samples; changes the files) |
-| `--only-class` `C,...` | export only objects of these classes |
+| `--only-class` `C,...` | export only the units whose root object is of these classes |
| `--workers` `N` | worker processes (default: the CPUs this process may use; 0: this process) |
| `--memory` `GiB` | memory budget of the running tasks and workers (default: 80 % of the physical memory) |
| `--fetch-workers` `N` | parallel downloads (default 8) |
-| `--layout` `LAYOUT` | original, cas or original,cas (default original) |
+| `--layout` `LAYOUT` | original, cas, product, comma-separated (default original; product: the products of the selection) |
| `--explain` | print every setting of the run first |
-| `--link` `LINK` | how the layout files are made from the store: auto (default; [export] link): a clone where the file system makes them, else a hard link (read-only), else a copy; or that method only (one of `auto`, `clone`, `hard`, `copy`) |
+| `--link` `LINK` | how the layout files are made from the depot: auto (default; [export] link): a clone where the file system makes them, else a hard link (read-only), else a copy; or that method only (one of `auto`, `clone`, `hard`, `copy`) |
| `--strict` | exit 1 when a view has gaps |
| `--dry-run` | print the plan, run nothing |
@@ -195,63 +195,90 @@ an ADV episode -> JSON
## nnnotes export story
-an ADV episode -> story directory (episode, audio, scene, UI, media, videos) + its Live2D models
+an ADV episode -> the story product and the products it plays with (scene, assets, Live2D models, cue sheets, videos, chat, UI, fonts, CRI Lips)
| Argument | Description |
| --- | --- |
| `adv_id` | MasterAdv id of the episode |
+| `--story-languages` `LANG,...` | text languages, comma-separated (default: [catalog] language) |
+| `--fonts` `FONTS` | glyph source: open font files (default) or game TMP assets (one of `open`, `game`) |
+| `--font` `LANG=PATH` | font file for a language, or emoji=PATH (repeatable; [paths] fonts.) |
+| `--format` `FORMAT` | web audio format of the cue sheets (default aac) (one of `aac`, `flac`, `mp3`, `opus`, `vorbis`) |
+| `--flac-level` `N` | FLAC compression level 0-12 of the decoded sounds (default 8; every level decodes to the same samples) |
| `-o`, `--out` `OUT` | output directory (required) |
-| `--models` `DIR` | directory of the Live2D models, one / each (default: OUT/../live2d) |
-| `--force` | export the models again whose directory exists |
-| `--format` `FORMAT` | audio file format (default flac) (one of `flac`, `ogg`, `wav`) |
-| `--flac-level` `N` | FLAC compression level 0-12 of --format flac (default 8; every level decodes to the same samples) |
-| `--no-audio` | do not decode the cue sheets |
-| `--fonts` `FONTS` | open (default): text layout and style only, no font data; game: also the game's TMP fonts (needs the 'fonts' extra) (one of `open`, `game`) |
+| `--stored-catalog` `LABEL\|SHA` | a catalog version of the depot (`catalog versions`; default: the current catalog) |
+| `--workers` `N` | worker processes (default: the usable CPUs) |
+| `--link` `LINK` | as export assets --link (one of `auto`, `clone`, `hard`, `copy`) |
+| `--explain` | print every setting of the run first |
+| `--dry-run` | print the plan, run nothing |
## nnnotes export live
-a live (music + difficulty) -> self-contained live directory
+a live (music + difficulty) -> the live product and the products it plays with (scene, note highway, sounds, music)
| Argument | Description |
| --- | --- |
| `music_id` | MasterLiveMusic id |
| `--difficulty` `DIFFICULTY` | chart difficulty (default expert) (one of `easy`, `normal`, `hard`, `expert`) |
-| `--format` `FORMAT` | audio file format (default flac) (one of `flac`, `ogg`, `wav`) |
-| `--flac-level` `N` | FLAC compression level 0-12 of --format flac (default 8; every level decodes to the same samples) |
-| `--fonts` `FONTS` | start canvas text: open (default): layout and style only, no font data (your fonts are not used); game: also the game's TMP fonts (needs the 'fonts' extra) (one of `open`, `game`) |
+| `--format` `FORMAT` | web audio format of the BGM (default aac) (one of `aac`, `flac`, `mp3`, `opus`, `vorbis`) |
+| `--flac-level` `N` | FLAC compression level 0-12 of the decoded sounds (default 8; every level decodes to the same samples) |
| `--band` `BAND` | band of the LightWeight background and start timeline (default: band of the music's first vocal character) |
| `--leader-card` `LEADER_CARD` | deck centre MasterMemberCard id (its character's band) |
| `--live-option` `OPTION[=VALUES]` | also export the files of a Live option's variants (repeatable): MirrorChart, MeasureLineDisplay, NoteDesignId, NoteEffectId, LiveQuality, NoteSePatternId (every value of the master data, or =v,v...), defaults (the option defaults and ranges only) or all |
+| `--live-languages` `CODES` | comma-separated UI languages, or all (default: catalog language) |
+| `--fonts` `FONTS` | glyph source: open font files (default) or game TMP assets (one of `open`, `game`) |
+| `--font` `LANG=PATH` | font file for a language (repeatable; [paths] fonts.) |
| `-o`, `--out` `OUT` | output directory (required) |
+| `--stored-catalog` `LABEL\|SHA` | a catalog version of the depot (`catalog versions`; default: the current catalog) |
+| `--workers` `N` | worker processes (default: the usable CPUs) |
+| `--link` `LINK` | as export assets --link (one of `auto`, `clone`, `hard`, `copy`) |
+| `--explain` | print every setting of the run first |
+| `--dry-run` | print the plan, run nothing |
At most one of `--band`, `--leader-card`.
## nnnotes export live2d
-a Live2D model -> runtime model directory
+a Live2D model -> its Cubism SDK files, or the player's model
| Argument | Description |
| --- | --- |
| `key` | model key Character/Live2D///model/, or its model id |
| `-o`, `--out` `OUT` | output directory (required) |
+| `--player` | write the player's model (models/.json and assets/) instead of the Cubism SDK files |
+| `--stored-catalog` `LABEL\|SHA` | a catalog version of the depot (`catalog versions`; default: the current catalog) |
+| `--workers` `N` | worker processes (default: the usable CPUs) |
+| `--link` `LINK` | as export assets --link (one of `auto`, `clone`, `hard`, `copy`) |
+| `--explain` | print every setting of the run first |
+| `--dry-run` | print the plan, run nothing |
## nnnotes export spot
-a spot -> spot.json + Spine + room.glb + shaders
+a spot -> spot.json + Spine files + room.glb + room.json
| Argument | Description |
| --- | --- |
| `spot_id` | MasterHomeSpot id |
| `-o`, `--out` `OUT` | output directory (required) |
+| `--stored-catalog` `LABEL\|SHA` | a catalog version of the depot (`catalog versions`; default: the current catalog) |
+| `--workers` `N` | worker processes (default: the usable CPUs) |
+| `--link` `LINK` | as export assets --link (one of `auto`, `clone`, `hard`, `copy`) |
+| `--explain` | print every setting of the run first |
+| `--dry-run` | print the plan, run nothing |
## nnnotes export room
-a background prefab -> glb
+a spot background -> room.glb + room.json
| Argument | Description |
| --- | --- |
-| `key` | addressable key of the prefab |
-| `-o`, `--out` `OUT` | output .glb file (required) |
+| `key` | background key Spot//Background/ |
+| `-o`, `--out` `OUT` | output directory (required) |
+| `--stored-catalog` `LABEL\|SHA` | a catalog version of the depot (`catalog versions`; default: the current catalog) |
+| `--workers` `N` | worker processes (default: the usable CPUs) |
+| `--link` `LINK` | as export assets --link (one of `auto`, `clone`, `hard`, `copy`) |
+| `--explain` | print every setting of the run first |
+| `--dry-run` | print the plan, run nothing |
## nnnotes export shader
@@ -286,8 +313,11 @@ the APK's UI prefabs -> an offline UI prefab library for ournotes-player/ui
| `--key` `KEY` | exact APK catalog key (repeatable) |
| `--prefix` `PREFIX` | APK UI key prefix (default EmbUI/) |
| `--limit` `LIMIT` | maximum number of keys |
-| `--no-dependencies` | omit additional prefab roots and controllers |
-| `--force` | re-export the selected keys |
+| `--stored-catalog` `LABEL\|SHA` | a catalog version of the depot (`catalog versions`; default: the current catalog) |
+| `--workers` `N` | worker processes (default: the usable CPUs) |
+| `--link` `LINK` | as export assets --link (one of `auto`, `clone`, `hard`, `copy`) |
+| `--explain` | print every setting of the run first |
+| `--dry-run` | print the plan, run nothing |
## nnnotes apk settings
@@ -305,7 +335,7 @@ find the HCA keycode of the CRI audio (never printed)
| --- | --- |
| `--write` `DIR` | write it as DIR/.hcakey (the file vgmstream reads) |
-## nnnotes store plan
+## nnnotes depot plan
what `export assets` would run and why, without running it (--json: nnnotes.plan/1)
@@ -314,14 +344,14 @@ what `export assets` would run and why, without running it (--json: nnnotes.plan
| `-o`, `--out` `OUT` | output directory of the export (counts the layout files to write) |
| `--select` `SEL` | group:, key: or bundle: (repeatable; default: everything) |
| `--views` `all\|none\|a,b` | semantic views to build (default: all when master data is set, else none) |
-| `--stored-catalog` `LABEL\|SHA` | a catalog version of the store (`catalog versions`; default: the current catalog) |
+| `--stored-catalog` `LABEL\|SHA` | a catalog version of the depot (`catalog versions`; default: the current catalog) |
| `--png-level` `N` | PNG compression level 0-9 (changes the output) |
| `--flac-level` `N` | FLAC compression level 0-12 of the CRI audio (default 8; every level decodes to the same samples; changes the files) |
-| `--only-class` `C,...` | export only objects of these classes |
+| `--only-class` `C,...` | export only the units whose root object is of these classes |
| `--workers` `N` | worker processes (default: the CPUs this process may use; 0: this process) |
| `--memory` `GiB` | memory budget of the running tasks and workers (default: 80 % of the physical memory) |
| `--fetch-workers` `N` | parallel downloads (default 8) |
-| `--layout` `LAYOUT` | original, cas or original,cas (default original) |
+| `--layout` `LAYOUT` | original, cas, product, comma-separated (default original; product: the products of the selection) |
| `--explain` | print every setting of the run first |
| `--since` `RUN` | explain against this run (default: the latest of the selection) |
| `--check` | exit 1 when anything would run or change |
@@ -329,9 +359,9 @@ what `export assets` would run and why, without running it (--json: nnnotes.plan
| `--emit-tasks` `DIR` | write the runnable tasks as task descriptions |
| `--why` `TASK` | the reasons and key parts of one task |
-## nnnotes store run
+## nnnotes depot run
-run task descriptions (nnnotes.task/1) into the store
+run task descriptions (nnnotes.task/1) into the depot
| Argument | Description |
| --- | --- |
@@ -339,7 +369,7 @@ run task descriptions (nnnotes.task/1) into the store
| `--fetch` | fetch inputs through their catalog locators |
| `--force` | run tasks whose result is stored (it must not change) |
-## nnnotes store verify
+## nnnotes depot verify
check every stored content and result
@@ -348,6 +378,17 @@ check every stored content and result
| `--quick` | check sizes only, do not hash the contents |
| `--workers` `N` | parallel hashing (default the CPUs this process may use; 0: one at a time) |
+## nnnotes depot gc
+
+remove the runs, results and contents no kept run reaches
+
+| Argument | Description |
+| --- | --- |
+| `--keep` `N` | keep the N latest runs of each selection (default: every run; the latest run of a selection is kept) |
+| `--dry-run` | count what would be removed, remove nothing |
+| `--grace` `SECONDS` | keep recent results and their inputs and contents in both depots (default: 86400 seconds) |
+| `--remote` | also collect the depot's remote, by its runs (its results and contents younger than a day stay) |
+
## nnnotes voices list
the voices matching the filters (the text column in the --language language, default [catalog] language)
@@ -405,32 +446,32 @@ live charts, Live2D models and stories -> static site for ournotes-player (share
| Argument | Description |
| --- | --- |
| `-o`, `--out` `SITE` | site directory (required) |
+| `--name` `NAME` | persistent site name in the depot (default: default) |
| `--player` `PLAYER` | ournotes-player checkout (built) or installed package ([paths] player) |
| `--pair` `PAIR` | : (repeatable) |
| `--all` | every MasterLiveMusic x difficulty |
-| `--player-only` | rewrite the player files, charts.json and models.json only |
-| `--reingest-json` | store every chart's and model's JSON files again |
+| `--player-only` | publish the saved catalog with current player pages |
| `--live2d` `MODEL` | Live2D model id or key Character/Live2D///model/ (repeatable) |
| `--all-live2d` | every Live2D model of the catalog |
| `--format` `FORMAT` | audio format of the charts and stories (default aac) (one of `aac`, `opus`, `vorbis`, `mp3`, `flac`) |
| `--no-audio` | export no audio files |
| `--compress` `COMPRESS` | encoding of the compressible assets (JSON, shaders, moc3, ...) where it makes them smaller: gzip (default), br (brotli) or none (one of `gzip`, `br`, `none`) |
-| `--force` | rebuild charts, models and stories whose manifest exists (stories: also their models) |
-| `--tmp` `TMP` | directory for the temporary live and model builds (default .tmp) |
-| `--workers` `N` | parallel music / story / model processes (default a quarter of the CPUs, up to 8; models up to 4; 0: this process) |
-| `--read-workers` `N` | chart read sets run at a time (default half the CPUs, up to 16) |
+| `--workers` `N` | product task worker processes; 0: this process |
+| `--memory` `GiB` | product task memory budget |
+| `--fetch-workers` `N` | parallel source downloads |
+| `--stored-catalog` `LABEL\|SHA` | a stored catalog version of the depot |
| `--story` `ADV_ID` | story episode: its MasterAdv id (repeatable) |
| `--all-stories` | every story episode (MasterAdv) of the master data |
-| `--story-languages` `LANGS` | languages of the stories' text, comma-separated (default: ja,en,zh-Hant,zh-Hans,ko) |
+| `--story-languages` `LANGS` | languages of the stories' text, comma-separated (default: the catalog language) |
| `--font` `LANG=PATH` | font file the story text of a language is drawn with (repeatable; [paths] fonts.); emoji=PATH: the colour emoji font the emoji sprites are drawn from ([paths] fonts.emoji) |
| `--site-region` `REGION` | a region the site serves: a [servers.] table (repeatable; the first is the base; default: [catalog] region) |
| `--all-regions` | every configured region |
-| `--fonts` `FONTS` | text of the start canvas and the stories: open (default): the start canvas has layout and style only, the stories glyphs from your font files (--font); game: the game's TMP fonts (needs the 'fonts' extra) (one of `open`, `game`) |
+| `--fonts` `FONTS` | text of the start canvas and stories: open (default): glyphs from your font files (--font); game: the game's TMP fonts (needs the 'fonts' extra) (one of `open`, `game`) |
| `--band` `BAND` | band of the LightWeight background and start timeline (default: band of the music's first vocal character) |
| `--leader-card` `LEADER_CARD` | deck centre MasterMemberCard id (its character's band) |
| `--live-option` `OPTION[=VALUES]` | also export the files of a Live option's variants (repeatable): MirrorChart, MeasureLineDisplay, NoteDesignId, NoteEffectId, LiveQuality, NoteSePatternId (every value of the master data, or =v,v...), defaults (the option defaults and ranges only) or all |
-At most one of `--pair`, `--all`, `--player-only`, `--reingest-json`.
+At most one of `--pair`, `--all`, `--player-only`.
At most one of `--live2d`, `--all-live2d`.
diff --git a/docs/commands.md b/docs/commands.md
index a5188e5..6969eef 100644
--- a/docs/commands.md
+++ b/docs/commands.md
@@ -5,11 +5,11 @@ nnnotes [options]
```
Global options (before or after the command name): `--config`, `--region`, `--language`, `--catalog`,
-`--catalog-version`, `--cache`, `--store`, `--master`, `--apk`, `--ffmpeg`, `--vgmstream`, `--node` and `--json`,
+`--catalog-version`, `--depot`, `--master`, `--apk`, `--ffmpeg`, `--vgmstream`, `--node` and `--json`,
plus `--version` and `--help`. They set the settings described in [configuration.md](configuration.md), which also
lists the settings each command needs. `nnnotes --help` prints the options of a command;
-[cli.md](cli.md) lists every command with its options. The asset export commands (`export assets`, `store plan`,
-`store run`, `store verify` and the catalog versions of the store) are described in [assets.md](assets.md).
+[cli.md](cli.md) lists every command with its options. The asset export commands (`export assets`, `depot plan`,
+`depot run`, `depot verify` and the catalog versions of the depot) are described in [assets.md](assets.md).
Commands that write files take the output path with `-o/--out` (required). A command prints a short summary on
standard output; with `--json` it prints its report as JSON (UTF-8). Every JSON file is written by one
@@ -38,26 +38,53 @@ Write the config file from the template (asking in a terminal, else from `--set`
check every setting's origin and format, and list where the config file is looked up. No `config` command prints a
setting's value. Details in [configuration.md](configuration.md#writing-the-config-file).
-## Cache
+## Downloads
-`bundle pull` and every extractor read bundles through the cache (`[paths] cache`):
+The commands read the game files from the depot's `downloads/` directory (`[paths] depot`), which keeps the files
+under their own names:
```
-/catalog_main_.bin the legacy main catalog (explicit main or no API/version pin)
-/bundles/ bundles of the dependency closures, decrypted (UnityFS)
-/raw/ raw CDN files stored as they are (e.g. CRI cue sheet data)
-/catalogs// catalogs downloaded by `catalog browse`
-/international///
- versioned catalog, bundles and raw files; isolated from old main caches
+/downloads/catalog_main_.bin the main catalog (explicit main or no API/version pin)
+/downloads/bundles/ bundles of the dependency closures, decrypted (UnityFS)
+/downloads/raw/ raw CDN files stored as they are (e.g. CRI cue sheet data)
+/downloads/catalogs// catalogs downloaded by `catalog browse`
+/downloads/international///
+ versioned catalog, bundles and raw files of one CDN and version
```
-Files already in the cache are not downloaded again. When `[paths] apk` is set, the APK's own catalog is merged with
-the region's and bundles that ship inside the APK are read from it.
+A file is downloaded when a command first needs it and read from `downloads/` while it is there. When `[paths] apk`
+is set, the APK's own catalog is merged with the region's and bundles that ship inside the APK are read from it.
+
+The bundles and raw files are a cache bounded by `[depot] downloads`. Every command that reads the depot holds a
+lease in `/leases/` while it runs; when the last running command ends, the bundles and raw files used least
+recently are removed until the rest fit the budget:
+
+| `[depot] downloads` | kept once no command runs |
+|---|---|
+| unset or `0` | no bundles or raw files |
+| a size, e.g. `20GiB` | the most recently used ones, up to that size |
+| `all` | every one |
+
+Catalog files are kept. A removed file is downloaded again when a command needs its bytes. The depot records the
+identity (sha256 and size) of every bundle and raw file it reads, so `export assets` plans with files whose name
+carries the catalog's hash without downloading them again.
+
+During product and asset requests, a completed object-conversion batch makes its bundle downloads eligible for
+release toward the same budget. Readers finish before release; active batches, failed batches and preliminary
+episode requests stay protected. A batch already present in the depot can release its downloaded input as well.
+Only files recoverable through an explicit catalog or depot locator are released, and their contents are checked
+against the input identity. `all` retains downloads throughout the run; `0` releases each eligible batch without
+scanning the download directory. With a finite size, finished files are evicted in LRU order and active files can
+temporarily exceed the budget. Later requests can download a released bundle again.
+
+Commands may run at the same time on one depot: no command removes downloads while another one holds a lease, and a
+command that starts while one trims waits for it. The lease of a process that ended without releasing it is dropped
+by the next command; a lease written on another host counts for a day.
For international regions with a configured API, catalog-based commands discover `resource_version` first and
download `catalog__.bin`. `catalog fetch` labels the bytes with the version used to
select that file, rather than querying a newer label after downloading main. Use `--catalog-version VERSION` or
-the [version settings](configuration.md) to pin builds and use cached data offline. An explicit `--catalog FILE`
+the [version settings](configuration.md) to pin builds and use downloaded data offline. An explicit `--catalog FILE`
bypasses discovery. JP keeps its existing Version-driven path.
## catalog keys
@@ -82,11 +109,11 @@ it, decrypted; `bundles/` lists every bundle directly.
## bundle pull
```
-nnnotes bundle pull KEY [KEY ...]
+nnnotes bundle pull KEY [KEY ...] -o DIR
```
-Fetches the bundle closure (the key's location and all its dependencies) of each key into the cache and prints the
-cached path of every bundle.
+Fetches the bundle closure (the key's location and all its dependencies) of each key, copies the decrypted bundles
+to `DIR` under their file names, and prints the path of every copied bundle.
## servers
@@ -156,136 +183,94 @@ row of a Movie / Clip row's video id.
## export story
```
-nnnotes export story ADV_ID -o OUT [--models MODELS] [--force] [--format flac|ogg|wav] [--flac-level N] [--no-audio]
- [--fonts open|game]
-```
-
-One ADV episode as a directory, and its Live2D models in a models directory:
-
-```
-MODELS// every Live2D model of the episode (MODELS: `--models`, default OUT/../live2d), the files
- of a site's model manifest (see `site build`): model.json, moc3, prefab, the atlas pages and the
- shader programs its drawables use
-OUT/episode.json as `export episode`
-OUT/audio// every cue sheet of the episode, one file per cue + cues.json, streams.json
-OUT/scene.json player graphics, cameras, ADV fields, volumes, settings and stages
-OUT/frames.json Frame prefabs by asset name: uGUI, Animator controllers and clips, UI particle systems
-OUT/effects.json particle effect prefabs by asset name, `instances` (effect name -> asset name)
-OUT/posteffects.json PostEffect volume profiles by asset name
-OUT/stills.json Still prefabs by asset name
-OUT/talkwindows.json talk window prefabs of the TalkWindow rows
-OUT/chat.json chat window prefabs, icons and stamps (sprites), the MasterAdvChat rows, shared chat texts,
- sounds and cue sheet rows
-OUT/textures/, shaders/ textures and shaders of the scene and of the files above
-OUT/videos/ Movie / Clip videos as WebM + videos.json (video id -> file, video row, size, frame rate)
-OUT/ui/ ADV UI: ui.json (front canvas, still / frame / video canvases and their camera, letterbox),
- packed textures, UI shaders, rule transitions
-OUT/crilips/ crilips.json + crilips.bin: the CRI Lips analysis data, when a voice reaches the analysis
-OUT/story.json index of the above (a file the episode does not need is null and not written)
-```
-
-story.json `models` maps the key of each model to its id (`` of `Character/Live2D///model/`)
-and `modelsDir` is the path from OUT to MODELS. A model directory that exists is used as it is, so the stories of one
-models directory export each model once; `--force` exports the episode's models again. The printed summary lists
-the models exported (`modelsBuilt`) and those used as they were (`modelsSkipped`). Several `export story` processes may
-share a models directory: a model appears in it in one rename, whole, and a model two processes export at once is
-kept from the first to finish (the other's export, the same bytes, is dropped and counted in `modelsSkipped`).
-
-`--format` (default `flac`) is the audio format and `--flac-level` its FLAC compression level (as for `export audio`);
-`--no-audio` leaves the cue sheets undecoded (no `audio/`, `audio`
-in story.json is empty). With audio, `crilips/` holds the weights of the game's CRI Lips mouth analysis, read from the
-CRI Lips library in `[paths] apk` (or in the arm64-v8a split next to it, `split_config.arm64_v8a.apk`), when a voice
-reaches that analysis: a lip-synced voice while a model has no MotionSync controller (model.json `motionSync`), or a
-voice that other speakers follow (story.json `crilips`; ournotes-player docs/crilips.md). Videos keep the VP9 stream
-of the game's USM file and carry its ADX audio as Opus (FFmpeg,
-`[paths] ffmpeg`); the USM streams are unmasked with the CRI key read from `[paths] apk`. TextMesh Pro text in
-frames, talk windows and chat keeps its layout and style; with `--fonts open` its font and sprite assets and text
-materials are only named (no atlas is written), with `--fonts game` they are exported. The command stops before
-writing anything when the episode uses resources that are not in the catalog or resource kinds that are not
-supported (`timeline`). Prints the index with a summary.
-
-The shared chat sound rows (`chat.json` `sounds`, `cueSheets`) name the cue sheet `AdvSe_Common`. In the version
-1.0.1 data nothing holds it: no catalog language (`ja`, `en`, `zh-Hant`, `zh-Hans`, `ko`) has a key
-`Cri/Sound/AdvSe_Common` or `EmbCri/Sound/AdvSe_Common` (remote or APK catalog), the APK has no ACB file of that
-name, and it is not a Resources asset. The story therefore has no audio for these rows; the rows are kept as they
-are. Every cue sheet of the episodes' own `-SoundCueSheet` shards has its `Cri/Sound/` key.
-
-`ui/ui.json` holds the parts of the game's ADV widget the story draws: the front canvas (talk window, speaker plate,
-location caption, title, rule transition, curtains, flash, subtitles caption, next indicator, menu entry button, menu
-panel and video buttons, backlog, choices, tap areas), the canvases of stills, frames and videos with the screen
-image and the camera they render with (`videoAndStillCamera`), the UI camera that renders the frame canvas
-(`uiCamera`), and the letterbox bands; per node its transform, rect
-and uGUI components, the views' serialized references as node paths (the menu view with its fast-forward sprites),
-the other game components' serialized fields (`behaviours`), the widget's canvas sort orders (`widget`), the ADV
-screen's dialogs (`dialogs`: the skip confirm and common dialog prefabs) and their texts of the language
-(`masterIdTexts`), the chat phone (`chatWidget`), the text
-records of the episode's chat windows (`chatTexts`) with their status texts (`chatStatusTexts`). The talk windows
-sit under `TalkView`: the default window,
-then each other window the episode's TalkWindow rows attach; when one of them dims and blurs the screen behind its
-talk (`UICenterTalkWindow`), the front canvas' `CenterTalkBackdrop` node, the renderer's blur settings (`blur`) and
-the blur shader come with it.
-
-`ui/` follows the client language `[catalog] language` (`ja`, `en`, `zh-Hant`, `zh-Hans` or `ko`): the localized
-fonts and materials, the line spacing LocalizeText applies, and, with `--fonts game`, the characters of the lines
-(Talk, Location and Subtitles rows, ruby readings included), speaker names, title and UI texts in that language
-(`ui/ui.json` `language`: `mode`, `field`, `lineSpacing`).
-
-`--fonts` (default `open`) chooses how `ui/` handles text. Both modes write a text record `textStyle` for each text
-node of `ui/ui.json`, and a document-level `textStyle` that holds the units and the line metrics of each font role.
-With `open`, `ui/` contains no font data: no font atlases, glyph tables, font materials or text shader. `game` also
-exports the game's TextMesh Pro fonts: per node the serialized text settings with the localized font and material
-(`text`), and in the document the font assets reduced to the characters the episode shows (`fonts`), runtime glyphs,
-text materials, `glyphCoverage`, `tmpSettings` and the text shader. `game` needs the optional `fonts` dependencies
-(`pip install -e ".[fonts]"`).
-
-The `ui/ui.json` text record (`textStyle` of a node; the node's `rect` and transform give its layout box):
-
-| Field | Value |
-|---|---|
-| `class`, `enabled` | TMP component class, enabled flag |
-| `fontRole` | font slot of the game's localized fonts: `primary` (main text face), `number` (Latin / numeral face) |
-| `materialType` | material style name of the text (`Default`, `OutlineAdvCommon`, ...) |
-| `localized`, `textKey` | LocalizeText enabled; its master text id, or null when the runtime sets the text (talk, speaker, location and title texts come from `episode.json`, one field per language) |
-| `text`, `richText`, `parseControlCharacters` | serialized text, rich text tags on, `\n`-style escapes parsed |
-| `fontSize`, `autoSize` | size in canvas units; auto-size `enabled`, `min`, `max`, `maxCharWidthAdjust`, `maxLineSpacingAdjust` |
-| `fontStyle`, `fontWeight` | style flags (`bold`, `italic`, `underline`, `strikethrough`, `lowerCase`, `upperCase`, `smallCaps`, `superscript`, `subscript`, `highlight`), weight 100-900 |
-| `alignment` | `horizontal` (`left`, `center`, `right`, `justified`, `flush`, `geometry`), `vertical` (`top`, `middle`, `bottom`, `baseline`, `geometry`, `capline`) |
-| `wrapping`, `overflow` | `noWrap` / `normal` / `preserveWhitespace` / `preserveWhitespaceNoWrap`; `overflow` / `ellipsis` / `masking` / `truncate` / `scrollRect` / `page` / `linked` |
-| `margin` | `left`, `top`, `right`, `bottom` insets of the rect |
-| `lineSpacing` | `serialized`, `applied` (the value for the export language), `byLanguage` (the per-language line spacing LocalizeText applies; null when not localized) |
-| `paragraphSpacing`, `characterSpacing`, `wordSpacing` | spacing in 1/100 em |
-| `characterHorizontalScale`, `kerning`, `rightToLeft`, `orthographic` | as serialized |
-| `color`, `colorMode`, `colorGradient`, `overrideHtmlColors` | vertex colour, gradient mode, four-corner gradient (null when off) |
-| `face` | `color` (fill = `color` x `face.color`), `dilateEm`, `boldDilateEm` (glyph edge moved outwards), `softnessEm` (edge ramp width) |
-| `outline` | null, or `color`, `widthEm` (band on each side of the glyph edge: stroke width 2 x `widthEm`, drawn over the fill), `softnessEm` |
-| `underlay` | null, or `color`, `offsetEm` [x, y] (+x right, +y down), `dilateEm`, `softnessEm`, `inner` (shadow inside the glyph) |
-
-`*Em` values are fractions of the drawn font size, converted from the text material's distance-field properties the
-way the TMP shader applies them. The document-level `textStyle.roles` gives, per role, `lineHeightEm`, `ascentEm`,
-`descentEm` (line pitch = `lineHeightEm` + `lineSpacing` / 100 em) and the font's `spacingOffset` / `boldSpacing`
-(1/100 em) for the export language.
+nnnotes export story ADV_ID -o OUT [--format FORMAT] [--flac-level N] [--stored-catalog LABEL|SHA] [--workers N]
+ [--story-languages LANG,...] [--fonts open|game] [--font LANG=PATH]
+ [--link auto|clone|hard|copy] [--explain] [--dry-run]
+```
+
+One ADV episode and the products it plays with, in the `product` layout ([products](products.md#stories)):
+
+```
+OUT/stories/.json the story (`story`): episode.json and story.json
+OUT/storyscenes/default.json the ADV scene (`story.scene`)
+OUT/storyassets/.json each stage, frame, effect, post effect, still, timeline, transition, chat window,
+ chat icon and stamp the episode loads (`story.asset`; : its key, "/" as "~")
+OUT/models/.json each Live2D model of the episode (`live2d.model`)
+OUT/cuesheets/.json each cue sheet of the episode, decoded (`story.sound`)
+OUT/videos/.json each video of the episode, as WebM (`story.video`; : its asset, "/" as "~")
+OUT/storychat/default.json the shared chat rows, when the episode shows chat (`story.chat`)
+OUT/storyui/.json the story UI in each requested language (`story.ui`)
+OUT/storyfonts/_.json the episode's font records and glyph pages (`story.fonts`)
+OUT/storysimpleui/.json the Overlay talk-window and system-message UI (`story.simpleui`)
+OUT/storyhosts/.json the Overlay host screen (`story.host`)
+OUT/spots/.json a home host's spot and Spine files (`spot.files`)
+OUT/rooms/.json a home host's background (`room.gltf`)
+OUT/crilips/default.json the CRI Lips analysis data, when a voice can reach it (`story.crilips`)
+OUT/assets/. the files the manifests name
+```
+
+`episode.json` is the episode as `export episode` writes it, each resource given by its kind and address.
+`story.json` names the episode's chat rows, its effect instances and, in `parts`, the manifest of each product above
+it plays with. The command runs twice: it asks for the story, reads its `story.json`, then asks for the story and
+every product `parts` names and places them. It stops with a usage error before the second run when a key of such a
+product is not in the catalog. The episode's talk windows are listed in `episode.json` `resources`. The videos are
+the `movie.webm` of `cri.movie` with `format` `webm`: VP9 as stored, re-encoded when the video is in another codec or
+has an alpha stream.
+
+A product several stories play with (the scene, a stage, a model, a cue sheet) is one product: stories exported into
+one directory share its manifest and files, and the depot keeps every result for the next export.
+
+`--story-languages` selects text languages, separated by commas; the default is `[catalog] language`.
+`--fonts open` (the default) generates glyph pages from the font files in `[paths.fonts]` or the repeatable
+`--font LANG=PATH` flags. Supply each requested language's font and any fonts needed by its fallback chain.
+`--font emoji=PATH` supplies a bitmap emoji font. Without it, open emoji sprites retain their layout with empty
+images. `--fonts game` uses the client's TMP glyphs and generates the runtime glyphs its font assets require.
+Both sources use font settings from the client's assets and require the `fonts` extra. Font files are explicit
+content-addressed inputs, so changing their bytes rebuilds the corresponding font product.
+
+Overlay episodes also request their home or live-result host, the simple UI in each requested language, and,
+for home hosts, the shared spot and room products. The host's master-data relationship is recorded in
+`story.json` `host`; the episode font product covers simple UI text and any area title in both font modes.
+
+`--format` (default `aac`) is the web audio format of the cue sheets: `aac` (AAC-LC in MP4, `cues.json` records each
+file's encoder delay), `opus`, `vorbis`, `mp3`, or `flac` (the decoded FLAC as it is); `--flac-level` is the FLAC
+compression level of the decoded sounds (as for `export audio`). `crilips/` holds the weights of the game's CRI Lips
+mouth analysis, read from the CRI Lips library in `[paths] apk` (or in the arm64-v8a split next to it,
+`split_config.arm64_v8a.apk`); `story.json` names it when a voice of the episode is lip-synced, and the models'
+MotionSync (model.json `motionSync`) decides whether a player runs the analysis.
+
+The shared chat sound rows (`chat.json` `sounds` and `cueSheets` of `storychat/default.json`) name the cue sheet
+`AdvSe_Common`. In the version 1.0.1 data nothing holds it: no catalog language (`ja`, `en`, `zh-Hant`, `zh-Hans`,
+`ko`) has a key `Cri/Sound/AdvSe_Common` or `EmbCri/Sound/AdvSe_Common` (remote or APK catalog), the APK has no ACB
+file of that name, and it is not a Resources asset. The story therefore has no audio for these rows; the rows are
+kept as they are. Every cue sheet of the episodes' own `-SoundCueSheet` shards has its `Cri/Sound/` key.
## export live2d
```
-nnnotes export live2d MODEL -o OUT
+nnnotes export live2d MODEL -o OUT [--player]
```
-A Live2D (Cubism) model prefab as a Cubism runtime directory. `MODEL` is the model's key
+A Live2D (Cubism) model as the Cubism SDK files other tools read. `MODEL` is the model's key
`Character/Live2D///model/` or its id `` (`nnnotes catalog keys --prefix Character/Live2D/` lists
the keys):
```
OUT/.moc3 the model's moc3
-OUT/textures/*.png atlas pages
-OUT/.prefab.json the whole prefab: every GameObject and component, motions and expressions inlined
-OUT/.exp3.json expressions
+OUT/textures/.png the atlas pages the drawables draw with, by name
OUT/.physics3.json physics (not written for a model without physics)
-OUT/motions/.motion3.json motions (and motions/_fades.json)
-OUT/.model3.json file references, EyeBlink / LipSync groups
+OUT/.exp3.json expressions
+OUT/motions/.motion3.json motions
+OUT/.model3.json file references, the model's fade times per motion, EyeBlink / LipSync groups
```
-Needs `[paths] apk` (component classes are resolved through the APK). Prints a JSON summary.
+With `--player`, the model the player reads instead: the product layout of `live2d.model`, `OUT/models/.json`
+and the files it names under `OUT/assets/` ([products](products.md#live2d-models)).
+
+The command runs the depot's pipeline for the model alone (the census of the bundles of its key, the objects its
+prefab reaches, the product) like `export assets`, and writes the same reports in `OUT/_reports/`; it takes
+`--stored-catalog`, `--workers`, `--link`, `--explain` and `--dry-run` as `export assets` does. Needs `[paths] apk`
+(the component classes and the mask materials are in the APK).
## export spot
@@ -296,42 +281,26 @@ nnnotes export spot SPOT_ID -o OUT
```
OUT/spot.json master row, names, tap-talk episodes, situation settings, tap targets, Spine characters
OUT/spine/ Spine skeletons (.json or .skel), atlases, atlas page PNGs
-OUT/room.glb the spot's background as binary glTF (as `export room`), with room.json
-OUT/shaders/ shaders of the spot's bundles (as `export shader`)
+OUT/room.glb the spot's background as binary glTF (as `export room`)
+OUT/room.json its summary
```
-Needs `[paths] apk` and master data (`MasterHomeSpot`, `MasterText`, `MasterStoryHomeSpotTapTalkEpisode`). `room.glb` and
-`room.json` are the `export room` export of the spot's background.
-
-A reference the situation prefab leaves empty is written as null: a Spine character with no `_animation` has null
-`skeletonData`, `animation` and `world`, and a tap target with no `_focus` has a null `focusWorld`.
+The spot whose `MasterHomeSpot` row has the id `SPOT_ID`: the products `spot.files` of its situation
+(`_situationAssetPath`) and `room.gltf` of its background (`_backgroundAssetPath`), their files at their own paths
+([products](products.md#spots)). The command runs the depot's pipeline for the two products alone, as `export live2d`
+does, writes the same reports in `OUT/_reports/` and takes the same options. Needs `[paths] apk` (the Spot and Spine
+component classes are in the APK) and master data (`MasterHomeSpot`, `MasterText`,
+`MasterStoryHomeSpotTapTalkEpisode`).
## export room
```
-nnnotes export room KEY -o OUT.glb
+nnnotes export room KEY -o OUT
```
-A spot background prefab as binary glTF: every mesh baked into prefab space, converted to glTF's right-handed space,
-textures embedded, materials translated from the shader's render state. Objects inactive in the prefab are kept with
-`extras.unityActive = false`. A summary (`meshCount`, materials, textures, samplers) is written next to it as
-`OUT.json`.
-
-Materials and textures are told apart by serialized file and path id (the prefab's bundle and its dependencies can
-reuse a path id). Each texture is one glTF texture with its own sampler, and textures whose PNG bytes are equal share
-one glTF image. The summary's `textures` has one entry per glTF texture, in glTF order (the index a material's
-`baseColorTexture` names): the Unity texture's name.
-
-The render state is that of the shader's first pass for the material: blend factors (colour and alpha), blend
-operations, colour mask, culling, depth write, depth test and alpha to mask. A state the shader takes from a
-property (`Blend [_SrcBlend] [_DstBlend]`, `Cull [_Cull]`, `ZWrite [_ZWrite]`, ...) is the material's value of that
-property, else the shader's default for it. Each glTF material keeps the resolved state in
-`extras.unityRenderState` (and `OUT.json` in `materials[].renderState`); a shader or a state glTF cannot express
-stops the command. A submesh whose material slot is empty is not written (there is no material to draw it with): the
-summary lists each as `nullMaterialSubmeshes` (`mesh`, object `path` in the prefab, `submesh` index), and a mesh
-left with no submesh is not written. A MeshFilter with no mesh, or a mesh with no triangles (no index data), draws
-nothing and is not written; the summary lists the object path in `nullMeshFilters`, or the mesh and object path in
-`meshesWithoutTriangles`.
+A spot background as binary glTF, `OUT/room.glb`, with its summary `OUT/room.json`: the product `room.gltf` of the
+background key `KEY` (`Spot//Background/`; [products](products.md#spots)). Runs and reports as
+`export spot`.
## export shader
@@ -419,266 +388,122 @@ nnnotes apk settings -o OUT.json
The game's render settings from the APK's boot data as JSON: `colorSpace`, `defaultPipeline`, `qualityLevels`,
`qualityPerPlatform`, `pipelines`, `renderers`, `postProcessData`. Needs `[paths] apk`. The MonoBehaviours are read
with the type trees that ship with nnnotes; an APK whose classes do not match them stops the command with exit
-status 2 and a line naming the class, the game version and the Unity version (as `export story`, `export live` and `site build`).
+status 2 and a line naming the class, the game version and the Unity version (as `export story` and `site build`).
## export live
```
-nnnotes export live MUSIC_ID -o OUT [--difficulty easy|normal|hard|expert] [--format flac|ogg|wav]
- [--flac-level N] [--fonts open|game] [--band BAND | --leader-card CARD_ID]
+nnnotes export live MUSIC_ID -o OUT [--difficulty easy|normal|hard|expert] [--format aac|flac|mp3|opus|vorbis]
+ [--flac-level N] [--band BAND | --leader-card CARD_ID]
[--live-option OPTION[=VALUES] ...]
+ [--live-languages LANG,...] [--fonts open|game] [--font LANG=PATH]
```
-One chart (music + difficulty, default `expert`) as a self-contained directory, the format ournotes-player reads:
+One chart (music + difficulty, default `expert`) and the products it plays with, in the product layout
+([products](products.md#lives)):
```
-OUT/score/ the chart as shipped, the converted runtime notes, master rows (title and band
- names in every language), summary.json
-OUT/audio// the BGM cue sheet decoded per cue + cues.json, streams.json
-OUT/audio/live-audio.json the live's sounds (BGM, note SE, live SE: the start and finish cheers and the finish
- sound of every result, all perfect, full combo, assist full combo and clear) with
- their CRI routing; their cue sheets decoded next to the BGM
-OUT/livescene/ scene graph, cameras, lane, background, start timeline, textures and shaders
-OUT/livenotes/ note, line and effect prefabs, skins, clips, particle systems, textures and shaders
-OUT/liveui/ the start canvas in [catalog] language: strings, text records or fonts, sprites
-OUT/live.json index of the above
+OUT/lives/_[_].json the live: the chart as shipped, its converted notes (also mirrored),
+ the music's master rows, the jacket, the manifest paths of the parts
+OUT/scenes/default.json the live scene: scene graph, cameras, lane, UI assets, the start
+ timeline and LightWeight background of every band, the built-in
+ graphics settings
+OUT/notes/default.json the note highway: note, line and lane effect prefabs, settings
+OUT/noteskins/.json a note skin
+OUT/noteeffects/.json a note effect set
+OUT/sounds/.json the note and live sounds with their CRI routing and volumes
+OUT/music/.json the BGM in a web format
+OUT/liveui/__.json localized start-canvas text, glyph pages and image bindings
+OUT/assets/. every file the manifests name, once
```
-`--format` (default `flac`) is the format of every decoded cue sheet and `--flac-level` its FLAC compression level
-(as for `export audio`).
-
-The start canvas follows the client language `[catalog] language` (`ja`, `en`, `zh-Hant`, `zh-Hans` or `ko`): its
-text table column, fonts and line spacing. `--fonts open` (default) writes each text's layout and style (size,
-alignment, wrapping, spacing, colours, outline / underlay from its material) and only the names of the game's fonts
-and materials; `--fonts game` also writes the game's TextMesh Pro fonts reduced to the characters shown, with their
-atlas textures (needs the optional `fonts` dependencies: `pip install 'nnnotes[fonts]'`).
+`--format` (default `aac`) is the web format of the BGM: `aac` (AAC-LC in MP4), `opus`, `vorbis`, `mp3`, or `flac`,
+the decoded FLAC itself. `--flac-level` is the FLAC compression level the cue sheets are decoded at (every level
+decodes to the same samples).
The game takes the band of the background and start timeline from the player's deck centre. Without a deck the
band is `--band`, or the band of the character of `--leader-card` (a `MasterMemberCard` id), or by default the band
-of the music's first vocal character; the choice is recorded in `livescene/scene.json`. Prints the index with a
-summary.
-
-The directory holds what a live reads with the game's default options (option preset 1 of a fresh profile).
-`--live-option` (repeatable) adds the files of other values of the options that select files, so that a player can
-offer them; the options are named as in the game (the names ournotes-player's settings use):
+of the music's first vocal character; `live.json` `band` records the choice, and the live's manifest path holds the
+band only when one of the two flags is given. A band whose start timeline or background the catalog does not hold is
+a usage error.
+
+`--live-languages` selects UI languages separated by commas (`all` selects all client languages); the default is
+`[catalog] language`. `--fonts open` (default) generates glyph pages from `[paths.fonts]` or repeatable
+`--font LANG=PATH` flags. Supply fonts for the requested language and its fallback languages. `--fonts game` uses
+the client's TMP assets. Both modes require the `fonts` extra and use the client's font metrics and layout
+settings. Font bytes participate in the task key. The UI's canvas and difficulty icons are in `live.scene`, and
+its jacket and text shaders are in the UI product's object table. `live.json` names UI products by language in
+`parts.ui`.
+
+The products hold what a live reads with the game's default options (option preset 1 of a fresh profile), and always
+the mirrored notes, the bar line view and the option tables. `--live-option` (repeatable) adds the products of other
+values of the options that select files, so that a player can offer them; the options are named as in the game (the
+names ournotes-player's settings use):
| `--live-option` | Adds |
|---|---|
-| `MirrorChart` | `score/.mirror.notes.json`, the chart converted with lanes and flick directions mirrored (the game mirrors while it converts the chart); `live.json` `notesMirror` |
-| `NoteDesignId` | the other note skins (`MasterLiveNoteSkin`): `livenotes/notes.json` `noteSkins` (skin asset name -> a record like `noteSkin`) and `settings.skins` (`NoteDesignId` -> skin asset name) |
-| `NoteEffectId` | the other note effect sets (`MasterLiveNoteEffectSkin`; the lane effects are the same for every set): their assets in `livenotes/notes.json` `assets`, and `settings.effects` (`NoteEffectId` -> effect set name) |
-| `LiveQuality` | the qualities of `MasterLiveQualitySettings`; at the Low quality (2) the game loads the `Light` variant of a note effect set where the catalog has one (else the set itself), whose assets are added (and `settings.effects`) |
-| `NoteSePatternId` | the other note sound sets (`MasterLiveNoteSe` groups): `audio/live-audio.json` `noteSe.groups` (set -> `LiveNoteSeType` -> sound id) and their sounds, decoded next to the others |
-| `MeasureLineDisplay` | the bar lines: `livenotes/notes.json` `prefabs.bar_line_view`, the bar line view the live scene's bar line container instantiates (a node list like the note view prefabs), and its sprite's texture; the bar times and the BPM and time signature changes are in the score's notes (`barLineTimeMs`, `bpmChanges`, `barChanges`) with or without the option |
-| `defaults` | only `settings.optionDefaults` / `optionRanges` of every option the player offers (note timing, the live and note sound volumes and mutes, the note sound set and per-type sounds, ...) |
+| `NoteDesignId` | the other note skins (`MasterLiveNoteSkin`): a `live.noteskin` product each |
+| `NoteEffectId` | the other note effect sets (`MasterLiveNoteEffectSkin`; the lane effects are the same for every set): a `live.noteeffect` product each |
+| `LiveQuality` | the qualities of `MasterLiveQualitySettings`; at the Low quality (2) the game loads the `Light` variant of a note effect set where the catalog has one (else the set itself), whose `live.noteeffect` products are added |
+| `NoteSePatternId` | the other note sound sets (`MasterLiveNoteSe` groups): the `live.sounds` product `NoteSePatternId=`, which holds them and the default set |
+| `MirrorChart`, `MeasureLineDisplay`, `defaults` | nothing: the products always hold their files |
| `all` | every option above with every value |
An option without values offers every value the master data has; `NAME=v,v,...` offers those values (the default
-value is always included; a value the master data does not have is a usage error). Any option also writes the
-`defaults` tables. Without `--live-option` the directory holds the files of the default options only.
+value is always included; a value the master data does not have is a usage error).
+
+The command runs the depot's pipeline for these products alone, as `export live2d` does, writes the same reports in
+`OUT/_reports/` and takes `--stored-catalog`, `--workers`, `--link`, `--explain` and `--dry-run` as `export assets`
+does. Needs `[paths] apk` (the built-in graphics and sound settings and the CRI ACF are in the APK), master data,
+`vgmstream` and `ffmpeg`.
## site build
+```sh
+nnnotes site build --name NAME -o SITE [--pair MUSIC_ID:DIFFICULTY | --all]
+ [--live2d MODEL | --all-live2d] [--story ADV_ID | --all-stories]
+ [--player DIR] [--format aac|opus|vorbis|mp3|flac] [--no-audio]
+ [--compress gzip|br|none] [--workers N] [--memory GiB] [--fetch-workers N]
+ [--stored-catalog LABEL|SHA] [--site-region REGION | --all-regions]
+ [--fonts open|game] [--story-languages LANGS] [--font LANG=PATH]
+ [--band N | --leader-card ID] [--live-option OPTION[=VALUES]]
+nnnotes site build --name NAME -o SITE --player-only
+```
+
+The [site guide](site.md) covers selection, fonts, read sets and hosting. Chart, model and story requests execute the same depot product stages as their export commands. `--name` (default `default`) is a persistent namespace in the depot. Partial selections retain other topics; complete selections replace that product kind in the requested regions.
+
+```text
+SITE/
+ index.html, chart-list.js, ... chart list and player page
+ live2d/, story/, songs/ player pages and bundles
+ charts.json, models.json,
+ stories.json listings and localized facts
+ site.json complete nnnotes.site/1 catalog
+ charts/.json live product entry manifests
+ models/.json model product entry manifests
+ stories/.json story product entry manifests
+ charts//.json,
+ stories//.json regional variants
+ products/.json immutable nnnotes.product/1 manifests
+ assets/.[.gz|.br] shared asset bytes
+ manifest.json output ownership
```
-nnnotes site build -o SITE [--pair MUSIC_ID:DIFFICULTY [--pair ...] | --all] [--live2d MODEL [--live2d ...] | --all-live2d]
- [--story ADV_ID [--story ...] | --all-stories] [--story-languages LANG,...] [--font LANG=PATH ...]
-nnnotes site build -o SITE (--player-only | --reingest-json)
- [--player DIR] [--format aac|opus|vorbis|mp3|flac] [--no-audio] [--compress gzip|br|none]
- [--force]
- [--tmp DIR] [--workers N] [--read-workers N] [--band BAND | --leader-card CARD_ID]
- [--site-region REGION [--site-region ...] | --all-regions] [--fonts open|game]
- [--live-option OPTION[=VALUES] ...]
-```
-
-Builds or updates a static [ournotes-player](https://github.com/empty-sekai/ournotes-player) site of charts, Live2D
-models, stories or any of them:
-
-```
-SITE/index.html, chart-list.js ... the player's chart list page; ?music=&difficulty= plays one chart
-SITE/ournotes-player.element.min.js the player's built bundle (and its source map)
-SITE/charts.json chart index: listing facts (texts in every language), regions,
- manifest path, sizes
-SITE/charts/_.json chart manifest: every path the player reads -> {asset, size[, stored]},
- or {parts: [[key, asset, size[, stored]], ...], size} for a large JSON
- object
-SITE/charts//.json the manifest of a region whose chart files differ (see Regions)
-SITE/models.json Live2D model index: id, key, group, canvas, manifest path, sizes; with
- master data the character and its names
-SITE/models/.json model manifest, the same entry forms as a chart manifest
-SITE/live2d/ the player's Live2D model page and its bundle (when the player has them)
-SITE/stories.json story index: titles and story groups in every language, commands,
- languages, manifest path, sizes, regions
-SITE/stories/.json story manifest: the common files and one file group per language, the
- same entry forms as a chart manifest; the model manifests of its models
- and the path to the site root
-SITE/stories//.json the manifest of a region whose story files differ (see Regions)
-SITE/story/ the player's story page and its bundle (when the player has them)
-SITE/assets/.[.gz|.br] content-addressed files shared by all charts, models and stories
- (see --compress)
-```
-
-- `--pair` (repeatable; `:` or `_`) adds the given charts (a chart that
- no region of the site has a `MasterLiveMusicScore` row for is a usage error); `--all` adds every music and
- difficulty that has a `MasterLiveMusicScore` row (in the master data of any of the site's regions).
-- `--live2d` (repeatable; a model id `` or key `Character/Live2D///model/`) adds the given
- Live2D models; `--all-live2d` adds every model key of the catalog. A model's id is its ``. Its manifest lists
- the files the player's Live2D viewer and story page read: `model.json` (index: moc3, prefab, textures, shader index,
- the Cubism mask materials and `motionSync`, whether the prefab's root has the MotionSync controller with its CRI
- audio input), the moc3 and prefab as `export live2d` writes them, the atlas pages the drawables use, and the GLSL ES 3.00
- programs of the Live2D shaders that the drawables' materials select, each keyword set also with
- `_ADDITIONAL_LIGHTS_VERTEX` (the story player adds it at quality 4), and the mask shader's for a model with masked
- drawables. Charts and models can be added in one run; models are built first.
-- `--story` (repeatable; a `MasterAdv` id) adds the given story episodes (an id no region of the site has is a usage
- error); `--all-stories` adds every `MasterAdv` episode. See [Stories](#stories) below. Stories are built after
- models and charts; the Live2D models the stories to build use are built before them, as with `--live2d` (listed in
- `models.json`, names from the master data as for `--live2d`).
-- A chart, model or story whose manifest exists is skipped unless `--force` (with stories, their models too). A
- model manifest whose `model.json` is of an older format (without `motionSync`) is outdated and built again; the
- summary lists such models in `modelsRebuilt` (`storyModels.modelsRebuilt` for the models of stories). Assets no
- chart, no model and no story (common or language file, or model manifest it names) references are removed.
-- `--player-only` rewrites the player files, `charts.json`, `models.json` and `stories.json` only; `--reingest-json`
- stores every chart's and model's JSON files again from the site's own assets (then rewrites the player files and
- the indexes; story manifests are left as they are).
-- `--player`: the ournotes-player checkout (after its build) or installed package; it must contain
- `scripts/read-set.mjs`, `dist/ournotes-player.element.min.js` and `examples/chart-list/index.html`. The read-set
- script runs under Node.js to list the files the player reads for each chart; only those are stored. When it also has
- `examples/live2d/index.html`, that page and `dist/ournotes-player.live2d.element.min.js` go to `SITE/live2d/`;
- when it has `examples/story-list/index.html` and `dist/ournotes-player.story.element.min.js`, they go to
- `SITE/story/` (a build that adds stories stops when the page is there without its bundle).
-- `--compress` (default `gzip`): how the compressible assets are stored: JSON files (split parts included), GLSL,
- moc3, atlas, skel, bin, wav and glb files. With `gzip` or `br` (brotli, quality 11) such a file is stored encoded
- when that makes it smaller: `assets/..gz` (or `.br`), named by the SHA-256 of the decoded bytes; its
- entry's `size` is the decoded size and `stored` the size of the stored file (a part: `[key, asset, size, stored]`).
- Other files, and every file with `none`, are stored as they are, without `stored`. `--reingest-json` stores the
- JSON files again with the given `--compress`.
-- Hosting: the player decodes a `.gz` or `.br` asset when the bytes it receives have the entry's `stored` length and
- uses them as they are when they have its `size`. A gzip site can be served as plain files; `Content-Encoding:
- gzip` on its `.gz` files is optional (with it the browser decodes them, without it the player does). A brotli site
- needs `Content-Encoding: br` on its `.br` files for Chromium-based browsers, which cannot decode brotli in the
- page.
-- `--format` (default `aac`) is the BGM format of the charts and the format of every story sound (music, sound
- effects, voices); the charts' note SE, cheers and voices stay FLAC. `--no-audio` stores no audio (the player then
- runs the chart silent on its own clock, and plays stories without their sounds; story videos keep their sound
- track).
-- `--workers`: parallel music processes (default a quarter of the CPUs, up to 8), model processes (default up
- to 4) and story processes (default a quarter of the CPUs, up to 8); `0` or `1` builds in this process. `--read-workers`: chart read sets run at a time, each a Node.js process
- (default half the CPUs, up to 16). With a player whose read-set script lists a chart's files from its plan
- (without stepping the chart) and serves many charts from one process, the plans run in one long-lived Node.js
- process per read-set slot, ended with the build; the full simulations that check a sample of the plans still run
- in a process each.
-- `--tmp`: directory for the temporary live and model builds (default `SITE.tmp`); the build directories in it are
- removed after use. `SITE.tmp/cache/` is kept: decoded cue sheets, encoded PNGs, shader dumps and read sets, each
- stored under a hash of everything it was made from (input bytes, settings, the external tools and libraries in use
- and nnnotes' own code), so a later build with the same `--tmp` reuses what is unchanged and writes the same files
- faster. Deleting it is
- safe at any time outside a build; the next build then makes everything again.
-- `--band` / `--leader-card`: as for `export live`, for every chart. `--fonts`: for charts as for `export live` (the site stores
- no start canvas files); for stories where the glyphs of the story text come from (see Stories).
-- `--live-option` (repeatable): as for `export live`, for the charts of this run (with `--pair` or `--all`). The live
- directories carry the variants' files, and a chart's read set is the union of its read set with the default
- options and its read sets with the settings of each offered variant (every combination of the offered mirror,
- note skin, effect set and quality values, the bar lines on, and each other note sound set; the player's read-set
- script needs its `--settings`); only the files a variant reads are stored, the shared ones once per site. A chart
- manifest lists the offered qualities as `options` (`{"LiveQuality": [0, 1, 2]}`); the other options are offered by
- the chart's files. Each variant adds a read-set plan per chart (`all` makes several dozen), so such a build takes
- longer. Charts whose manifest exists keep the options they were built with unless `--force`.
-- Models need `[paths] apk` (component classes and the mask materials are read from the APK). Master data is
- optional: when it is configured (of the site's first region), each model that a `MasterCharacterCostume` row maps
- to a character (its `_live2dPath` is the key after `Character/Live2D/`) gets, in its manifest's `model` and its
- `models.json` entry, `character` (the `MasterCharacter` id), `names` (`{language: the character's name}`, from the
- `MasterText` row of `_nameTextID`, the languages that have a text) and `label` (the name in `[catalog] language`).
- A key the rows map to two characters gets none. The manifests of skipped models get this build's fields; without
- master data the fields are not written and skipped manifests keep theirs.
-- `--site-region` (repeatable) / `--all-regions`: the regions the site serves (see Regions); default: the one
- `[catalog] region`. The global `--region` sets `[catalog] region`.
+
+Product manifests retain logical file paths, object IDs and file roles. Their declared dependencies point to immutable products. Public aliases have roots relative to their depth. Asset entries support whole and split JSON, with decoded sizes and optional encoded sizes.
+
+The complete catalog and layout are committed to the depot after publication succeeds. `--player-only` restores that catalog to the chosen output with current player pages. A configured remote supports restoration from a fresh local depot. Layout reachability keeps the retained products and assets available to depot collection.
+
+Read-set tasks freeze player sources, product inputs, Node identity and settings. Plans are checked against full chart simulations on the first chart and a deterministic sample. Plan failures fall back to full simulation; mismatches switch subsequent reads to full. Variants union every offered setting combination's read set. Structured documents and their image/audio/WebGL2 dependencies remain available; additional binary file roles are selected from the reads.
+
+Publication orders assets before immutable products, aliases/pages, and indexes. Retired owned files are collected after replacements succeed. Failures preserve the journal and previous successful-run pointer; retry the request after correcting the error. Product failures stop publication and appear in the run report. `--json` prints structured task and placement counts.
### Stories
-A story's manifest lists what the player's story page reads, in the layout of `export story` (below the site's `assets/`):
-`story.json`, `episode.json`, `scene.json`, the scene shaders (GLSL ES 3.00 programs only), textures, per cue sheet
-`cues.json` and the file of each cue in the `--format` (an AAC file's encoder delay in
-`cues.json` as `encoderDelay`; the CRI category names a cue plays in as `categories`, e.g. `["Voice", "VoiceConfig"]`), the media files the episode uses and its videos; per language (`--story-languages`,
-default `ja,en,zh-Hant,zh-Hans,ko`) the story UI in that language (`ui/ui.json`), its font data (`ui/fonts.json` with
-glyph pages under `ui/fonts/`) and `ui/languages.json`. UI files that are equal in every language are common files.
-`scene.json` and `ui/ui.json` are stored split per top-level key, so the parts stories and languages share are stored
-once. The Live2D models are the site's: the manifest names the model manifest of each model the story uses
-(`models`, `{id: "models/.json"}`, relative to the site root; `root` is the path from the manifest to the site
-root), and `stories.json` counts their files in `size.models`. The format is described in the player's
-`docs/story-data-format.md`.
-
-- Story text is laid out with TextMeshPro's rules from TextMeshPro font assets holding exactly the characters the
- episode shows in the language (its text table, title and UI labels). `--fonts open` (default) generates them from
- a font file per language: `--font LANG=PATH` (repeatable) or `fonts.` in the `[paths]` table of the config
- file (`NNNOTES_PATHS_FONTS_`, e.g. `NNNOTES_PATHS_FONTS_ZH_HANT`); every language of `--story-languages`
- needs one. Any OpenType or TrueType font (the first face of a collection) with the characters of the language
- works. The game's fallback chains are mirrored: where the game takes a character from a fallback font asset of
- another language's font (Korean text falls back to the Japanese font, for example), the generated asset falls
- back to an asset of that language's font file, which then needs to be given too (the build stops with the setting
- it needs when a character reaches it); characters no font file of the chain has are listed in `ui/fonts.json`
- `coverage.missing`. Characters the game's own font assets lack are drawn as the game draws them, as its missing
- glyph (U+25A1 where the game's font has it), not from the font file (`coverage.missingGlyph`). The simple talk
- window of an Overlay story is handled the same way. Each generated asset takes the
- point size, padding, style settings and render mode of the game font asset the texts use in that language, so the
- game's text materials apply unchanged; those values, the game's fallback chains, the characters the game's assets
- have and the text materials come from the table of them shipped with nnnotes (`src/nnnotes/unity/fontsettings.json`,
- `fontsettings.FORMAT`), so an open-font export reads no game font asset, material or font bundle;
- its face info and glyph metrics come from the font file (FreeType, no
- hinting), its distance field from the generator `--fonts game` uses for runtime glyphs (supersampled render modes
- at up to 8x). The asset records the font's family, version, license and SHA-256. `--fonts game` uses the game's
- font assets instead (the `fonts` extra); the format is the same. No font file is stored in the site other than as
- the generated glyph pages.
-- Emoji are laid out as the game lays them out: its UI texts (the talk and chat texts) draw a character their font
- assets lack from the game's emoji sprite asset, and emoji sequences (several code points with U+200D or U+FE0F)
- become sprite tags through the game's emoji search. The layout keeps the game's sprite metrics; the images come
- from a colour emoji font of your own: `--font emoji=PATH` or `fonts.emoji` in the `[paths]` table
- (`NNNOTES_PATHS_FONTS_EMOJI`), a font with PNG bitmap glyphs (CBDT or sbix tables). Noto Color Emoji (SIL Open Font
- License 1.1) is the tested font. Only the sprites an episode's texts can draw are generated, each at the game's
- sprite size, into one page under `ui/fonts/`; `ui/fonts.json` records the emoji font as it records a text font.
- Without an emoji font the sprites keep their layout with empty glyphs, and `ui/fonts.json`
- `coverage.sprites.missing` lists them. `--fonts game` uses the game's sprite asset.
-- Stories need `[paths] apk` and the optional `fonts` dependencies (`pip install 'nnnotes[fonts]'`) in both modes.
-- The story index's groups come from the story tables of the master data (`MasterStoryEpisode` and its chapter,
- `MasterStoryFriendshipEpisode`, `MasterStoryHomeSpotTapTalkEpisode`, `MasterStoryLiveResultEpisode`,
- `MasterHomeSpot`), with chapter, character and spot names in every language.
-- An Overlay episode (`MasterAdv._playbackMode` 1: the game plays it in its simple ADV player over the screen that
- opened it) is listed with `playbackMode` 1. With `--fonts open` its story also has the host screen (`host/`: the
- home spot or the live result screen of its story group) and, per language, the simple talk window
- (`ui/simple/ui.json`, `ui/simple/fonts.json` and its glyph pages); the manifest names them in `host`. The host has
- no game-font variant: with `--fonts game` an Overlay story has neither.
-- With `--fonts open` the chat window texts of an episode with chat rows get their bindings in `ui/fonts.json`
- `chatTexts`, and the font assets also hold the chat windows' status texts and the texts they format at run time.
- The text nodes of the episode's frames get theirs in `frameTexts`, and the font assets also hold the texts a frame
- that receives texts is given at run time (the texts of its Frame rows' `TargetTextIDs`, as the frame formats them).
-- Stories that fail are listed in the printed summary and in `SITE.story-failures.json`; the exit status is then 1.
+Story entry products reference their episode, scene, models, assets, cue sheets, videos, localized UI/font products and optional host products. The selected languages default to the catalog language. Story models also appear in the model listing. `--no-audio` retains cue metadata while omitting the sound streams; playback uses silent timing.
### Regions and languages
-One site serves several regions and every language:
-
-- The regions serve the same catalog for a language, so a chart's files depend on the region's master data only.
- Regions whose chart tables (the master tables the chart build reads) are identical share one manifest,
- `charts/.json`, built once from the first region's data. A region with other chart tables is built on its own
- into `charts//.json`; such a manifest whose files equal the shared one's is dropped and the region joins
- the shared manifest. Models read no master data: one build serves every region. Stories follow the charts' rule
- with the master tables the story build reads (`stories/.json`, `stories//.json`); a region offers
- the stories of its `MasterAdv`.
-- Each region's master data: `[servers.] master` (else `[paths] master`, for at most one of the regions; the
- global `--master` flag is refused with more than one region). A region offers the charts its master data has a
- `MasterLiveMusicScore` row for.
-- Chart files come from the catalog of `[catalog] language`; the listing texts come from the text tables in every
- language: a manifest's `chart` has `title` and `bands` in `[catalog] language` (`language`), and `titles` /
- `bandNames` in `ja`, `en`, `zh-Hant`, `zh-Hans` and `ko`. The manifest's `regions` lists the regions it serves;
- regions accumulate over builds (a skipped chart gains the regions of the build), so a region is dropped by building
- the site again from scratch.
-- `charts.json` has one entry per manifest (an id appears once per region) with `regions`, `titles`, `bandNames`,
- and at the top `language` (the default listing language: `[catalog] language` of the latest build), `languages`
- and `regions` (`id`, `name` from `[servers.] name`, `languages` from `[servers.] languages`). The
- chart list page switches with `?region=&lang=`.
-
-The same inputs with the same versions of nnnotes, its libraries and tools (and the same font files) give
-byte-identical outputs; the encoded assets also need the same zlib and brotli versions. Charts that fail are listed
-in the printed summary and in `SITE.failures.json`, models that fail in the summary and in
-`SITE.model-failures.json`, stories in `SITE.story-failures.json`; the exit status is then 1.
+Configured regions use their own master data. Products with identical immutable manifests and listing metadata share an alias and list all regions they serve. Other versions use regional paths. JP stories/models and international releases use separate sites; conflicting contents for the same model ID require separate sites. Chart and story listings include localized titles, supported languages and region metadata for the player pages.
## music-data
diff --git a/docs/configuration.md b/docs/configuration.md
index 2cfcb5a..a5419a3 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -59,7 +59,7 @@ No `config` command prints a setting's value. The config files these commands wr
The global flags go before or after the command name:
```bash
-nnnotes --config ~/nnnotes.toml --cache /data/nnnotes-cache bundle pull
+nnnotes --config ~/nnnotes.toml --depot /data/nnnotes-depot bundle pull -o bundles
NNNOTES_PATHS_MASTER=/data/master nnnotes export episode 10462 -o episode.json
```
@@ -86,7 +86,7 @@ exist when they are given.
| `[master] key` | `NNNOTES_MASTER_KEY` | — | Rijndael-256 key of the master data files: 32 bytes as 64 hex digits |
| `[master] iv` | `NNNOTES_MASTER_IV` | — | Rijndael-256 CBC initialization vector: 32 bytes as 64 hex digits |
| `[catalog] region` | `NNNOTES_CATALOG_REGION` | `--region` | name of one `[servers.]` table |
-| `[catalog] language` | `NNNOTES_CATALOG_LANGUAGE` | `--language` | catalog language: the `` of `catalog__.bin`: `ja`, `en`, `zh-Hant`, `zh-Hans` or `ko`; also the client language of `export live`, `export story` and `site build` (the text field, fonts and line spacing of their UI) and of the model labels of `site build --live2d` |
+| `[catalog] language` | `NNNOTES_CATALOG_LANGUAGE` | `--language` | catalog language: the `` of `catalog__.bin`: `ja`, `en`, `zh-Hant`, `zh-Hans` or `ko`; also the client language of `site build` (the text field, fonts and line spacing of its story UI) and of the model labels of `site build --live2d` |
| `[catalog] version` | `NNNOTES_CATALOG_VERSION` | `--catalog-version` | optional international resource version; unset: query the configured region API, or use legacy `main` when no API is configured; JP uses its own discovery |
| `[servers.] name` | `NNNOTES_SERVERS__NAME` | — | label of the region in `catalog browse` (default: the region name) |
| `[servers.] provider` | `NNNOTES_SERVERS__PROVIDER` | — | `international` or `jp`; empty selects `jp` for the region named jp, international otherwise |
@@ -101,17 +101,22 @@ exist when they are given.
| `[bootstrap] api` | `NNNOTES_BOOTSTRAP_API` | — | API root that serves the server list (`nnnotes servers`), same format |
| `[client] version` | `NNNOTES_CLIENT_VERSION` | — | client version sent to the game's API, e.g. `1.0.1`; unset: the `versionName` of `[paths] apk` |
| `[paths] catalog` | `NNNOTES_PATHS_CATALOG` | `--catalog` | a catalog `.bin` file to read instead of discovering/downloading a catalog; explicit files stay usable offline |
-| `[paths] cache` | `NNNOTES_PATHS_CACHE` | `--cache` | cache directory (created when missing) |
-| `[paths] store` | `NNNOTES_PATHS_STORE` | `--store` | store directory of the asset export ([assets.md](assets.md)); unset: `<[paths] cache>/store` |
+| `[paths] depot` | `NNNOTES_PATHS_DEPOT` | `--depot` | depot directory (created when missing): the downloaded game files, the results of the stages, catalog versions and runs ([commands.md](commands.md#downloads), [contracts.md](contracts.md#depot)) |
| `[paths] master` | `NNNOTES_PATHS_MASTER` | `--master` | decoded master data directory, one `
.json` per table; the flag overrides `[servers.] master` |
-| `[paths] apk` | `NNNOTES_PATHS_APK` | `--apk` | the game's `base.apk` (the story export reads the CRI Lips library from it, or from `split_config.arm64_v8a.apk` next to it) |
+| `[paths] apk` | `NNNOTES_PATHS_APK` | `--apk` | the game's `base.apk` (the story export reads the CRI Lips library from it, or from `split_config.arm64_v8a.apk` next to it; the asset export its built-in Unity files) |
| `[paths] ffmpeg` | `NNNOTES_PATHS_FFMPEG` | `--ffmpeg` | `ffmpeg` executable; unset: `ffmpeg` on `PATH` |
| `[paths] vgmstream` | `NNNOTES_PATHS_VGMSTREAM` | `--vgmstream` | `vgmstream-cli` executable; unset: `vgmstream-cli` on `PATH` |
| `[paths] node` | `NNNOTES_PATHS_NODE` | `--node` | Node.js executable; unset: `node` on `PATH` |
| `[paths] player` | `NNNOTES_PATHS_PLAYER` | `--player` (of `site build`) | ournotes-player checkout (built) or installed package |
| `[paths] fonts.` | `NNNOTES_PATHS_FONTS_` (e.g. `NNNOTES_PATHS_FONTS_ZH_HANT`) | `--font =PATH` (of `site build`) | font file (OpenType or TrueType) the story text of a language is drawn with (`--fonts open`); also read for the other story languages whose game fonts fall back to that language's font |
| `[paths] fonts.emoji` | `NNNOTES_PATHS_FONTS_EMOJI` | `--font emoji=PATH` (of `site build`) | colour emoji font (PNG bitmap glyphs: CBDT or sbix) the stories' emoji sprites are drawn from (`--fonts open`); Noto Color Emoji (SIL Open Font License 1.1) is the tested font. Optional: without it the sprites keep their layout with empty glyphs, listed in `ui/fonts.json` `coverage.sprites.missing` |
-| `[export] link` | `NNNOTES_EXPORT_LINK` | `--link` (of `export assets`) | how `export assets` makes the layout files from the store: `auto` (unset), `clone`, `hard` or `copy` ([assets.md](assets.md)) |
+| `[depot] downloads` | `NNNOTES_DEPOT_DOWNLOADS` | — | download-cache budget, applied to finished conversion batches and when the last command ends: a size (`500MB`, `20GiB`; units B, KB, MB, GB, TB and KiB, MiB, GiB, TiB), `all`, or `0` (unset) ([commands.md](commands.md#downloads)) |
+| `[depot] remote` | `NNNOTES_DEPOT_REMOTE` | — | a remote of the depot shared by the machines that use it: `s3:///` (S3-compatible object storage; `pip install 'nnnotes[remote]'`) or `file:///` ([contracts.md](contracts.md#depot-remote)) |
+| `[depot] endpoint` | `NNNOTES_DEPOT_ENDPOINT` | — | endpoint URL of the S3-compatible object storage; unset: AWS S3 |
+| `[depot] access_key_id` | `NNNOTES_DEPOT_ACCESS_KEY_ID` | — | access key id of the object storage; unset: boto3 looks for credentials itself (`AWS_ACCESS_KEY_ID`, `~/.aws`) |
+| `[depot] secret_access_key` | `NNNOTES_DEPOT_SECRET_ACCESS_KEY` | — | secret access key of the object storage |
+| `[depot] signing_region` | `NNNOTES_DEPOT_SIGNING_REGION` | — | region the requests to the object storage are signed for; unset: `us-east-1` |
+| `[export] link` | `NNNOTES_EXPORT_LINK` | `--link` (of `export assets`) | how `export assets` makes the layout files from the depot: `auto` (unset), `clone`, `hard` or `copy` ([assets.md](assets.md)) |
Hex values are case-insensitive; a `0x` prefix and surrounding whitespace are ignored. All values are TOML strings
except `languages` (array of strings).
@@ -136,36 +141,36 @@ nnnotes --region tw --catalog-version 1.0.0.201 export live2d MODEL_ID -o out/mo
```
Alternatively set `NNNOTES_SERVERS_TW_CATALOG_VERSION` in the build environment; workers inherit this pin.
-`--catalog-version` selects a remote resource release; `--stored-catalog LABEL|SHA` of `export assets` and `store
-plan` selects a catalog version of the store. To inspect the historical main catalog, explicitly pin `main`.
-Versioned caches are isolated by CDN root and resource version; a cached pin needs the CDN setting to identify
-its directory but performs no API/catalog download. A missing versioned file never falls back to main.
+`--catalog-version` selects a remote resource release; `--stored-catalog LABEL|SHA` of `export assets` and `depot
+plan` selects a catalog version of the depot. To inspect the historical main catalog, explicitly pin `main`.
+Versioned downloads are isolated by CDN root and resource version; a pin whose catalog is downloaded needs the CDN
+setting to identify its directory but performs no API/catalog download. A missing versioned file never falls back to main.
## What each command needs
-"Catalog" below means: `[paths] cache`; either `[paths] catalog` or `[catalog] language` (the catalog is then
-downloaded into the cache on first use); and, for what must be downloaded, `[catalog] region` and that region's
+"Catalog" below means: `[paths] depot`; either `[paths] catalog` or `[catalog] language` (the catalog is then
+downloaded into the depot on first use); and, for what must be downloaded, `[catalog] region` and that region's
`cdn`, `[bundle] key` and `nonce_seed`. `[paths] apk` is optional for the catalog: when it is set, the APK's own catalog and bundles are merged
in, and bundles that ship inside the APK can only be read with it.
"Master data" means the decoded master data directory of the region: the `--master` flag, else
`[servers.] master` of `[catalog] region`, else `[paths] master`.
-`--fonts game` (`export story`, `export live`, `site build`) also needs the optional `fonts` dependencies (`pip install 'nnnotes[fonts]'`),
+`--fonts game` (`site build`) also needs the optional `fonts` dependencies (`pip install 'nnnotes[fonts]'`),
and so does `site build --story` / `--all-stories` with either font source (it generates the TextMesh Pro font assets of the
story text); it is not a setting.
| Command | Settings |
|---|---|
-| `catalog keys` | `[paths] cache`; `[paths] catalog`, or `[catalog] language` (plus region and `cdn` while the catalog is not in the cache); no bundle key |
-| `catalog browse` | `[paths] cache`, `[bundle] key` + `nonce_seed`, and per region `cdn` + `languages` (`name` optional) |
+| `catalog keys` | `[paths] depot`; `[paths] catalog`, or `[catalog] language` (plus region and `cdn` while the catalog is not downloaded); no bundle key |
+| `catalog browse` | `[paths] depot`, `[bundle] key` + `nonce_seed`, and per region `cdn` + `languages` (`name` optional) |
| `bundle pull` | catalog |
| `master decode` | `[master] key` + `iv` |
| `master download` | `[catalog] region` and that region's `cdn`; `--latest` also that region's `api` and the client version |
| `master version` | `[catalog] region` and that region's `api`; the client version: `[client] version` or `[paths] apk` |
| `servers` | `[bootstrap] api`; the client version: `[client] version` or `[paths] apk` |
| `export episode` | catalog, master data |
-| `export story` | catalog, `[catalog] language`, master data, `[paths] apk`, `ffmpeg` (audio and videos), `vgmstream` (not with `--no-audio`) |
+| `export story` | catalog, master data, `[paths] apk`, `vgmstream`, `ffmpeg` |
| `export live2d` | catalog, `[paths] apk` |
| `export spot` | catalog, master data, `[paths] apk` |
| `export room` | catalog |
@@ -173,19 +178,20 @@ story text); it is not a setting.
| `export audio` | catalog, `[paths] apk`, `vgmstream`, `ffmpeg` (not for `--format wav`) |
| `apk hca-key` | `[paths] apk` |
| `apk settings` | `[paths] apk` |
-| `export live` | catalog, `[catalog] language`, master data, `[paths] apk`, `vgmstream`, `ffmpeg` |
-| `site build --pair` / `--all` | as `export live`, plus `[paths] player`, `node`; with `--site-region` / `--all-regions` each region's `[servers.]` table (its `cdn` for what must be downloaded) and master data (`[servers.] master`; `[paths] master` for at most one region) |
+| `export live` | catalog, master data, `[paths] apk`, `vgmstream`, `ffmpeg` |
+| `site build --pair` / `--all` | as `export live`, plus `[catalog] language`, `[paths] player`, `node`; with `--site-region` / `--all-regions` each region's `[servers.]` table (its `cdn` for what must be downloaded) and master data (`[servers.] master`; `[paths] master` for at most one region) |
| `site build --live2d` / `--all-live2d` | catalog (bundles from the CDN of the site's first region), `[paths] apk`, `[paths] player`; not `node`; master data only for the model names (optional: without it `models.json` has no names) |
| `site build --player-only` / `--reingest-json` | `[paths] player` |
| `music-data` | catalog, `[master] key` + `iv` (not with `--decoded-master`); `--master-files` also `[catalog] region`; `--apk-master` also `[paths] apk`; `--decoded-master` also `[catalog] region` and master data with its `MasterManifest.json`; `[paths] apk` (optional otherwise) for the client version |
-| `export assets`, `store plan` | the store (`[paths] store` or `[paths] cache`); catalog (bundles are fetched into the cache); `[paths] apk` for the bundles inside the APK (without it they are reported as `source.absent`); master data for `--views`; with `--stored-catalog` a catalog version of the store instead of the current catalog |
-| `store run` | the store; `[paths] cache` for inputs located in the cache; `--fetch` also what fetching needs (region, `cdn`, bundle key, `[paths] apk`) |
-| `catalog versions` / `import` / `diff`, `store verify` | the store; `import` reads the APK's catalog from `[paths] apk` when it is set |
-| `catalog fetch` | the store, `[catalog] region` and `language`, that region's `cdn`; its `api` (optional) for the resource version label |
+| `export assets`, `depot plan` | catalog (bundles are fetched into the depot); `[paths] apk` for the bundles inside the APK (without it they are reported as `source.absent`) and its built-in Unity files ([stages.md](stages.md#the-apks-built-in-files)); master data for `--views`; with `--stored-catalog` a catalog version of the depot instead of the current catalog |
+| `depot run` | the depot; `--fetch` also what fetching needs (region, `cdn`, bundle key, `[paths] apk`) |
+| `catalog versions` / `import` / `diff`, `depot verify` | the depot; `import` reads the APK's catalog from `[paths] apk` when it is set |
+| `catalog fetch` | the depot, `[catalog] region` and `language`, that region's `cdn`; its `api` (optional) for the resource version label |
-The region, its `cdn`, `[bundle] key` and `nonce_seed` are read only when a file must be downloaded: when the
-catalog and every bundle a command needs are in the cache, none of them is needed. A download that needs a missing
-one stops the command with one line naming the file and the setting (see Errors).
+The region, its `cdn`, `[bundle] key` and `nonce_seed` are read only when a file must be downloaded. A product
+request can read a downloaded catalog and its bundle files without these settings. Use `[depot] downloads = all`
+for an offline sequence of requests; a bounded budget can release a bundle that a later request needs again.
+A download that needs a missing setting stops the command with one line naming the file and setting (see Errors).
## Errors
@@ -194,7 +200,7 @@ the setting, its environment variable and its flag where there is one; it never
```
nnnotes: setting catalog.region is not set: give it as `region` in the [catalog] table of the config file, the environment variable NNNOTES_CATALOG_REGION or --region
-nnnotes: .bundle is not in the cache: setting bundle.key is not set: give it as `key` in the [bundle] table of the config file, the environment variable NNNOTES_BUNDLE_KEY
+nnnotes: .bundle is not downloaded: setting bundle.key is not set: give it as `key` in the [bundle] table of the config file, the environment variable NNNOTES_BUNDLE_KEY
nnnotes: setting bundle.key: must be 16 bytes (32 hex digits)
nnnotes: setting paths.apk: file not found
nnnotes: ffmpeg not found on PATH: give its path as `ffmpeg` in the [paths] table of the config file, the environment variable NNNOTES_PATHS_FFMPEG or --ffmpeg
@@ -230,7 +236,7 @@ nnnotes: game API call Version to [servers.tw] api failed: UNAVAILABLE (server u
can hold several alternatives separated by `|`, and any one of them can be used as `cdn` / `api`.
- **Client version**: the version of your game client (`versionName` of its `base.apk`). The game's API rejects a
version older than the one it accepts (`game error code CLIENT_UPDATE_REQUIRED`).
-- **`[paths] apk`**: the `base.apk` of your own installation of the game. `apk settings`, `export story`, `export live` and `site build`
+- **`[paths] apk`**: the `base.apk` of your own installation of the game. `apk settings`, `export story` and `site build`
read MonoBehaviours of its boot data with type trees that ship with nnnotes, one set per Unity version (currently
game version 1.0.1, Unity 6000.3.12f1). With an APK whose classes do not match them, these commands stop with
exit status 2 and a line naming the class, the game version and the Unity version.
diff --git a/docs/contracts.md b/docs/contracts.md
index c9e0f2d..9c47051 100644
--- a/docs/contracts.md
+++ b/docs/contracts.md
@@ -11,12 +11,26 @@ Every document names its format in a `schema` field (`nnnotes.result/1`). Within
added, never renamed, removed or given another meaning; readers ignore fields they do not know. A breaking change
is a new major version (`nnnotes.result/2`).
+Episode font data uses the `format` field `nnnotes.story-fonts/1`, described in
+[story-fonts.schema.json](https://github.com/MetaSekaiLab/nnnotes/blob/main/src/nnnotes/schemas/story-fonts.schema.json)
+and [story products](products.md#stories).
+
+Live start UI data uses the `format` field `nnnotes.live-ui/1`, described in
+[live-ui.schema.json](https://github.com/MetaSekaiLab/nnnotes/blob/main/src/nnnotes/schemas/live-ui.schema.json)
+and [live start UI products](products.md#live-start-ui).
+
+Overlay story products use `nnnotes.story-host/1` and `nnnotes.story-simple-ui/1` in their `format` fields:
+[story-host.schema.json](https://github.com/MetaSekaiLab/nnnotes/blob/main/src/nnnotes/schemas/story-host.schema.json)
+and [story-simple-ui.schema.json](https://github.com/MetaSekaiLab/nnnotes/blob/main/src/nnnotes/schemas/story-simple-ui.schema.json).
+
Stages carry their own version. A change of what a stage writes for the same inputs is a new stage version, which
changes the key of every task of that stage (golden tests fail when a stage's output changes while its version
stays).
## canon/1
+The canonical JSON (`nnnotes.core.canon`) of every document nnnotes writes:
+
- UTF-8, LF line endings, one trailing newline, non-ASCII characters as they are.
- Contract documents (everything on this page): object keys sorted, indent 1.
- Object documents (the JSON of one Unity object): the object's own field order, indent 1.
@@ -34,14 +48,15 @@ stays).
| object | `:` | `CAB-2f1e…:-4153…`, `unity default resources:10` |
| artifact of an object | `