Skip to content

fix(json): fix escapeIndex string escaping on big-endian architectures - #175

Open
AdamMagued wants to merge 1 commit into
segmentio:masterfrom
AdamMagued:fix-issue-170-escapeindex-bigendian
Open

AdamMagued wants to merge 1 commit into
segmentio:masterfrom
AdamMagued:fix-issue-170-escapeindex-bigendian

Conversation

@AdamMagued

Copy link
Copy Markdown

Fixes #170

Problem

On big-endian architectures such as s390x, string escaping produces invalid JSON when strings contain characters that require escaping (such as quotes, backslashes, control characters) within the first four bytes of an 8-byte chunk.

There were two defects in escapeIndex:

  1. Word Indexing: In 'for _, n := range chunks', the chunk index was ignored when returning the match index. An escape match in chunk i > 0 returned offset rather than i*8 + offset.
  2. Byte Indexing on Big-Endian Architectures: When stringToUint64 casts string memory to uint64 on big-endian systems, byte 0 is positioned in the most significant byte (bits 56..63) and byte 7 in the least significant byte (bits 0..7). Evaluating 'bits.TrailingZeros64(mask&msb)/8' produced 7 - k instead of k, pointing past the character and causing the encoder to emit unescaped characters.

Solution

  • When endianness == 1 (big-endian host), reverse the 64-bit word using bits.ReverseBytes64. This normalizes the byte layout so byte 0 is in the least significant byte, matching the LSB-first TrailingZeros64 calculation and arithmetic borrow direction.
  • Add chunk index offset i*8 to the returned index.
  • Add unit tests in json/escape_test.go validating byte indexing, chunk offsets, edge cases, native execution, and simulated big-endian word layouts.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

escapeIndex() breaks string escaping on big-endian (s390x)

1 participant