[fallenmi]

Summary

Preserve Unicode characters inside streamed JSON responses instead of treating them as record separators.

HTTPX iter_lines()/aiter_lines() use universal str.splitlines() semantics. That splits valid JSON strings containing raw U+0085, U+2028 or U+2029 before json.loads(), producing an “Unterminated string” error. The new regressions reproduce this through public Client.chat() and AsyncClient.generate() calls with deterministic one-byte chunks.

This keeps HTTPX's public iter_text()/aiter_text() decoding and changes only record framing to LF, accepting CRLF. Partial records are accumulated as pieces and joined once. Existing charset/default-encoding behavior is retained as backwards compatibility; Latin-1 is not being presented as NDJSON-conforming.

Verification

On Python 3.11.11 / Pydantic 2.10.6:

• Base with these tests: 124 passed, 3 failed, with the three Unicode cases failing at JSON parsing.
• Patched full suite and doctests: 127 passed, with both HTTPX 0.28.1 and the declared minimum 0.27.0.
• An additional independent public-API matrix: 94 passed on each version, covering all five sync/async streaming methods, one-byte chunks, final unterminated records, CRLF, escaped text, gzip, response errors/statuses, charset/default encoding and context-manager closure. Base: 44 failed / 50 passed.
• Ruff 0.17.1 check and format: passed (46 files). git diff --check: passed.
• Tests ran with private-home and external-network access denied; only local fixtures/mocked transports were used. No live Ollama, model or cloud calls.
• The full Python/OS CI matrix was not run locally. Python 3.8 syntax was checked, not its runtime.

Scope / overlaps

The NDJSON framing rules use LF or CRLF; JSON strings permit these Unicode characters. CR-only framing and other non-NDJSON separators are no longer treated as record boundaries. Final records without a newline remain accepted.

This does not add blank-line skipping (#689/#393) or connection-error wrapping (#741). Those open PRs touch the same loops but retain universal line splitting; a rebase may be needed depending on merge order. No combined runtime compatibility with their complete branches is claimed.

AI assistance

Claude Code prepared the implementation and regressions. Codex independently reproduced the base failure, found and rejected a charset regression in the first proposal, and verified this refined patch before publication.