fix(csv-stringify): quote every field matching a global quoted_match regexp - #498
Open
Jaybhade wants to merge 1 commit into
Open
fix(csv-stringify): quote every field matching a global quoted_match regexp#498Jaybhade wants to merge 1 commit into
Jaybhade wants to merge 1 commit into
Conversation
…regexp
`quoted_match` regexps are supplied by the user and the same objects are
reused for every field of every record, but they were applied with
`RegExp.prototype.test`. That advances `lastIndex` on a global or sticky
regexp, so a match in one field moved the offset the next field started
matching from, and fields that should have been quoted were not:
stringify([["1", "2"], ["3", "4"]], { quoted_match: /\d/g })
// '"1",2\n"3",4' instead of '"1","2"\n"3","4"'
The option is documented as quoting all fields matching the regular
expression, and the result should not depend on where the previous field
happened to match. The failure is intermittent in practice: a field that
does not match resets `lastIndex` to 0, so an interleaved non-matching
column hides it, and it also leaks state to the caller's regexp.
Test the patterns with `String.prototype.search`, which searches from the
start of the value and restores `lastIndex`. It is equivalent to `test`
for the stateless regexps that already worked, verified by differentially
sweeping both predicates over 1819 stateless pattern/value pairs with no
disagreement. The two duplicated match blocks become one helper, next to
`emits_separator`, so they cannot drift apart; it also drops the
per-field intermediate array the `filter` used to allocate.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
quoted_matchaccepts regexps from the user, and the same regexp objects are reused for every field of every record. They were applied withRegExp.prototype.test, which advanceslastIndexon a global or sticky regexp, so where one field matched decided where the next field started matching:Every matching field is documented — and typed, in
index.d.ts— as quoted: "quote all fields matching a regular expression". Whether a field gets quoted should not depend on the offset at which the previous one happened to match, and it should not depend on how many times the stringifier has been called: passing the same regexp to twostringifycalls carrieslastIndexfrom the first into the second.Two things make this easy to miss, and unpleasant when it does bite:
lastIndexto 0, so one interleaved non-matching column restores correct behaviour.quoted_match: [/^0\d+$/g]over[["1","01234"],["2","02345"]]is completely correct, because theidcolumn resets the state every row. The bug only shows once matching fields land next to each other, which makes it look data-dependent rather than reproducible.quoted_matchis the mechanism for quoting that the delimiter/quote/record-delimiter checks cannot infer — preserving leading zeros, keeping1-2from being read as a date, stopping long digit strings from becoming scientific notation. Dropping it on alternate rows leaves a file that parses fine and means something else in the consumer.Fix
Test the patterns with
String.prototype.search, which searches from the start of the value and restoreslastIndexafterwards, so the caller's regexp is left as it was found. For the stateless regexps that already worked it is equivalent totest.The two duplicated match blocks are folded into one helper next to
emits_separator, so they cannot drift apart — the empty-value branch carried the sametestcall. That branch is not observably affected today, since a zero-length match does not advancelastIndex, but it is the same expression and now shares the same fix. The helper also drops the intermediate array thefilterallocated per field.Verification
test-based predicate against the newsearch-based one over 1819 pattern/value pairs — 14 non-global regexps and 6 strings, singly and in two-element arrays, against 17 values (empty, unicode, astral, multiline, 50-char) — 0 disagreements. Repeating one value four times, the old predicate returned inconsistent results in 35 of 119 global/sticky cases, the new one in 0.test/option.quoted_match.ts, each red before the change with the output noted in a trailing comment: across records, within one record, and one that pre-setslastIndexto show the caller's regexp is no longer consumed. The existing file could not have caught this — every case used a single record and a non-global regexp.npx tsc --noEmitclean;packages/csv-stringify209 passing / 1 pending,packages/csv25 passing.prettier --checkandeslintclean on both changed files.api.callback"catch error in end handler, see Fix a bug where errors could leak on extremely large stream chunks #386",Sample api.async.iterator.jsand the memory-heavySample api.sync.memory.jsalso fail on an unmodifiedmaster, so they are unrelated to this change.Found while fuzzing the
stringify→parseround trip. Happy to split the helper extraction out of the fix, or to resetlastIndexon the caller's regexp instead of usingsearch, if you prefer either shape.