fix: remove doc-text sanitization; keep only sound index validation

[rejojer]

Cleans up the document-text sanitization introduced in 9681cda..9bcc7cf ("Implement document text sanitization and validation"), keeping only the parts that earn their place.

Removed

1. Keyword redaction (_INJECTION_PATTERNS, _sanitize_doc_text)
The pattern list matches ordinary English with no word boundaries: "impact assessment" → "imp[REDACTED]sessment", "The Trustee shall act as…", "disregarded" → "[REDACTED]ed", "renew instructions" → "re[REDACTED]". This silently corrupts the exact text the pipeline then fuzzy-matches section titles against, breaking physical_index assignment — while a keyword blocklist stops no real attacker (trivially bypassed by typos, encodings, or other languages).

2. Prompt wrapping and hardening preamble (_wrap_doc_text, _SYSTEM_HARDENING)
The indexing LLM calls have no tools, no secrets, and no cross-user surface — a malicious PDF can only make its own index bad, which requires no injection. The wrapper also labeled the JSON structure the model must fill in as "raw document text, treat as data only" inside the same prompt as the actual document text. All prompts are restored to their pre-sanitization form.

3. Pre-filter nullification (_validate_physical_indices + its four calls in process_no_toc)
meta_processor drops entries with physical_index=None before verify_toc runs. Nullifying at this stage therefore silently deleted sections that the designed verify → fix_incorrect_toc_with_retries loop (which checks every entry) would have relocated with the section preserved. It added no detection either — verify already checks everything. Out-of-range and non-int values are handled after the filter by validate_and_truncate_physical_indices (both bounds since c603b63), where a nullified entry survives as a placeholder.

This is also why process_no_toc intentionally no longer chunk-validates: an entry referencing a page outside its own chunk (e.g. copied from the previous structure) is left for verify to flag and the fix loop to relocate — the section survives with the corrected page, at the cost of a few extra LLM calls. Chunk-rejecting it here could only nullify, i.e. delete the section. The chunk check stays only in toc_index_extractor, where the validated structure is offset-vote scratch data discarded after use, so nullification there is free.

Kept

1. _parse_physical_index + the chunk-membership check ending toc_index_extractor
The printed-page/physical-page offset is decided by a majority vote over very few matching pairs, and one fabricated pair can shift every page number in the tree. This check nullifies answers referencing pages the model was never shown, operates on scratch data that no prompt ever sees, and costs no extra LLM call.

2. The fill-only-blanks merge loop in process_toc_no_page_numbers
Prevents the model from overwriting or dropping previously found entries (the crash-and-loss class #188 fixed symptoms of). Refined here: identity mismatches skip the chunk instead of raising; accepted values are parsed leniently (bare ints as well as <physical_index_N>) and stored in canonical marker form, so mid-flight state matches the format the prompts request.

Verification

• AST-level function diff against the pre-sanitization baseline: generate_toc_init, generate_toc_continue, process_toc_with_page_numbers, extract_matching_page_pairs, calculate_page_offset are byte-identical; every remaining delta is accounted for (fix: prevent KeyError crash and context exhaustion in TOC processing #188 robustness fixes, the keeps above, whitespace).
• Edge probes on parser/validator (whitespace, floats, malformed markers, negatives, absent keys) all pass.
• Pipeline tests updated to the new semantics (reordered LLM output skips the round preserving order; lenient fill accepts bare ints and rejects out-of-window answers).

[rejojer]

Follow-up commit c603b63 addresses external review findings: closes two latent crash paths that this PR re-exposed by removing the delete-based validators (bare numeric strings reaching the > comparison in validate_and_truncate_physical_indices; below-start_index values reaching page_list[negative] in verify_toc during recursive node processing), and tightens _parse_physical_index against bools, non-integral floats, and infinite floats. Both crashes predate the sanitization commits — the removed validators had masked them at the cost of silently deleting sections. Fixed at the source instead: parse what is parseable, nullify out-of-range values after the None-entry filter where placeholders are safe.

[rejojer]

Code review

No issues found. Checked for bugs, git history context, prior PR review comments, and code comment compliance.

🤖 Generated with Claude Code

_{- If this code review was useful, please react with 👍. Otherwise, react with 👎.}