Repository navigation
Add allocator-bounded BPE tokenization into caller storage - #1148
Draft
Baiju Meswani (baijumeswani) wants to merge 2 commits into
Draft
Baiju Meswani (baijumeswani) wants to merge 2 commits into
Baiju Meswani (baijumeswani) wants to merge 2 commits into
Conversation
Transfer the result buffers rather than deep-copying the rvalue argument. This removes two allocations for a single encoded row without changing its token IDs or lifetime. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a32500b6-be93-4206-a257-11406b00a3f3
Prepare immutable lookup views at model initialization, propagate per-call scratch allocators through the shared BPE algorithm, and reject unsupported fallback paths before encoding. Preserve atomic output and cover quota recovery, independent concurrent calls and shared-library access. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a32500b6-be93-4206-a257-11406b00a3f3
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add opt-in byte-level BPE tokenization into caller-owned output with an enforced per-call scratch allowance. This is a prerequisite for reserving media-preparation resources before tokenization in ONNX Runtime GenAI; it is independently usable and does not add video processing or change existing tokenizer APIs.
OrtxPrepareBoundedTokenizationbuilds immutable tokenizer-owned lookup views during initialization.OrtxTokenizeIntoBufferpropagates a per-call PMR allocator through the shared BPE implementation, including Unicode conversion, splitting, byte buffers, token output and merge-node storage.The quota bounds requested live input-dependent scratch, not RSS. Caller buffers, immutable tokenizer-owned lookup data, allocator overhead and fixed error/status bookkeeping are excluded. There is no global allocator override or shared mutable per-call quota.
Validation
The exact committed slice was built separately from all video edits in a clean Linux C++17 Release worktree. All 163 tokenizer regression cases pass (two existing profiling cases are disabled), and both separately linked shared-library API cases pass. Coverage includes quota exhaustion/recovery, atomic output, concurrent independent quotas and unsupported backends.
The integrated GenAI candidate also passed 36 complete Qwen/Ling token-ID comparisons against the legacy path, 108 Qwen image/video cases, 39 Flash cases and the reduced-Ling lifecycle checks. Those are integration receipts, not proof of total Engine memory accounting or complete three-model qualification.
Draft status
Windows shared-library ABI execution remains to be run on a Windows machine. This draft does not claim complete Engine media-memory accounting, encoder numerical qualification, or full Ling model enablement.