Skip to content

fix(render): encode non-BMP characters as UTF-16 surrogate pairs in AsciiJSON - #4728

Open
amitmishra11 wants to merge 1 commit into
gin-gonic:masterfrom
amitmishra11:fix/ascii-json-non-bmp-surrogate-pairs
Open

fix(render): encode non-BMP characters as UTF-16 surrogate pairs in AsciiJSON#4728
amitmishra11 wants to merge 1 commit into
gin-gonic:masterfrom
amitmishra11:fix/ascii-json-non-bmp-surrogate-pairs

Conversation

@amitmishra11

Copy link
Copy Markdown

Bug

AsciiJSON silently corrupts any Unicode code point above U+FFFF (emoji, CJK Extension B, etc.). The escaping loop uses fmt.Appendf(buf, "\u%04x", r), but %04x is a minimum width, not a fixed width. A non-BMP rune produces a 5-digit escape such as ὠ0 instead of the required 4-digit form. JSON parsers read exactly 4 hex digits for \u, so the 5th digit becomes literal text and the decoded string is silently wrong.

Example: rendering the string consisting of U+1F600 (grinning face emoji):

Before fix: AsciiJSON output is {"msg":"ὠ0"}, json.Unmarshal gives "xE0" (wrong)
After fix: AsciiJSON output is {"msg":"😀"}, json.Unmarshal gives the original emoji (correct)

Reported in #4688.

Root cause

Per RFC 8259 section 7, code points outside the BMP must be encoded as a UTF-16 surrogate pair (\uHHHH\uLLLL). The existing code uses a single \uXXXX token regardless of the code point size.

Fix

  • Import unicode/utf16.
  • In the AsciiJSON.Render loop, add a branch for r > 0xFFFF that calls utf16.EncodeRune and emits two four-digit \uXXXX escapes.
  • BMP characters (U+0080-U+FFFF) continue to use a single \uXXXX escape as before.

Test

Added TestRenderAsciiJSONNonBMP in render/render_test.go. It checks several emoji and a CJK Extension B character for:

  1. Pure-ASCII output (every byte < 128).
  2. Correct json.Unmarshal round-trip (decoded string equals the original input).

…sciiJSON

AsciiJSON used fmt.Appendf with "\u%04x" to escape all non-ASCII runes.
For code points above U+FFFF the format produces a 5+ digit sequence (e.g.
ὠ0 for U+1F600), which is syntactically valid JSON but wrong: a JSON
\u escape is exactly 4 hex digits, so decoders interpret the 5th digit as
literal text and the recovered string is silently corrupted.

Per RFC 8259 section 7, code points above U+FFFF must be encoded as a
UTF-16 surrogate pair (\uD800-\uDBFF followed by \uDC00-\uDFFF). Use
unicode/utf16.EncodeRune to compute the pair and emit two \uXXXX escapes.

Add TestRenderAsciiJSONNonBMP to verify that emoji and other non-BMP
characters survive an AsciiJSON -> json.Unmarshal round-trip unchanged.

Fixes gin-gonic#4688

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@codecov

codecov Bot commented Jul 3, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.33%. Comparing base (3dc1cd6) to head (ed773a4).
⚠️ Report is 289 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #4728      +/-   ##
==========================================
- Coverage   99.21%   98.33%   -0.89%     
==========================================
  Files          42       48       +6     
  Lines        3182     3176       -6     
==========================================
- Hits         3157     3123      -34     
- Misses         17       43      +26     
- Partials        8       10       +2     
Flag Coverage Δ
?
--ldflags="-checklinkname=0" -tags sonic 98.31% <100.00%> (?)
-tags go_json 98.25% <100.00%> (?)
-tags nomsgpack 98.30% <100.00%> (?)
go-1.18 ?
go-1.19 ?
go-1.20 ?
go-1.21 ?
go-1.25 98.33% <100.00%> (?)
go-1.26 98.33% <100.00%> (?)
macos-latest 98.33% <100.00%> (-0.89%) ⬇️
ubuntu-latest 98.33% <100.00%> (-0.89%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant