[林SO (Linxiushen)]

Describe your changes

JOIN_SLIDING_WINDOW currently carries tokens over from the end of the last emitted window when tokenization moves to another processing_batch_size batch. The next window starts only stride tokens later, so this discards its overlapping context. Token IDs, labels and sample counts consequently depend on the processing batch size. For example, 24 tokens in four 6-token rows with max_seq_len=8, stride=3 and processing_batch_size=1 produce four windows instead of the six produced with one processing batch.

Carry over tokens from the next scheduled window start (begin_loc + step), while continuing to slice each complete max_seq_len window. Retain all accumulated tokens when no complete window fits. Non-overlapping JOIN continues to advance by max_seq_len.

The regression uses the existing offline FakeTokenizer with distinct token IDs and checks exact windows, attention masks and context-masked labels across three row layouts, processing batch sizes 1/2/1024, both joined strategies and optional max_samples. On the unchanged production base, 12 of the 36 cases fail; all 36 pass with the fix.

Validation on Windows CPU, offline:

• python -m pytest test/data_container/test_text_generation.py -q: 50 passed.
• python -m pytest test/data_container/test_dataset.py test/data_container/test_data_config.py test/data_container/test_template.py -q: 9 passed.
• Changed-file lintrunner -a --skip PYLINT, followed by changed-file lintrunner including PYLINT: passed.
• git diff --check: passed.

These are native data preprocessing tests using the existing deterministic tokenizer and in-memory datasets. The full suite, real model/tokenizer execution and GPU execution were not run. AI assistance was used for implementation, tests and review.

Checklist before requesting a review

• Add unit tests for this change.
• Make sure all tests can pass. Relevant tests above pass; the full suite was not run.
• Update documents if necessary. No configuration or API changes.
• Lint and apply fixes to your code by running lintrunner -a (changed files; PYLINT also run separately).
• Is this a user-facing change? If yes, give a description of this change to be included in the release notes.

Release note: Preserve sliding-window overlap across tokenizer processing batches so text-generation preprocessing produces consistent samples and labels when processing_batch_size changes.

(Optional) Issue link

Found while inspecting the native preprocessing path; no separate issue.

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

🟢 Approval recommended

No active review issues are reported for the joined sliding-window preprocessing change.

No open issues 🎉

What changed in this PR

This PR fixes joined sliding-window preprocessing so token windows and labels remain consistent across processing batch sizes.

Changes:

• Preserve overlapping tokens by carrying them forward from the next scheduled window start, while retaining full-length window slices.
• Add regression tests covering row layouts, processing batch sizes, joined strategies, and optional sample limits.

File	Description
test/​data_container/​test_text_generation.py	Adds 36 cases checking exact windows, attention masks, labels, and sample counts.
olive/​data/​component/​text_generation.py	Corrects sliding-window carryover while preserving non-overlapping JOIN behavior.

_{Review effort: Balanced}