Skip to content

fix(csv-stringify): quote every field matching a global quoted_match regexp - #498

Open
Jaybhade wants to merge 1 commit into
adaltas:masterfrom
Jaybhade:fix/quoted-match-regexp-state
Open

fix(csv-stringify): quote every field matching a global quoted_match regexp#498
Jaybhade wants to merge 1 commit into
adaltas:masterfrom
Jaybhade:fix/quoted-match-regexp-state

Conversation

@Jaybhade

@Jaybhade Jaybhade commented Aug 4, 2026

Copy link
Copy Markdown

quoted_match accepts regexps from the user, and the same regexp objects are reused for every field of every record. They were applied with RegExp.prototype.test, which advances lastIndex on a global or sticky regexp, so where one field matched decided where the next field started matching:

const { stringify } = require("csv-stringify/sync");

stringify([["1", "2"], ["3", "4"]], { quoted_match: /\d/g });
// '"1",2\n"3",4'       expected '"1","2"\n"3","4"'

stringify([["1", "2", "3", "4"]], { quoted_match: /\d/g });
// '"1",2,"3",4'        expected '"1","2","3","4"'

Every matching field is documented — and typed, in index.d.ts — as quoted: "quote all fields matching a regular expression". Whether a field gets quoted should not depend on the offset at which the previous one happened to match, and it should not depend on how many times the stringifier has been called: passing the same regexp to two stringify calls carries lastIndex from the first into the second.

Two things make this easy to miss, and unpleasant when it does bite:

  • It is self-hiding. A field that does not match resets lastIndex to 0, so one interleaved non-matching column restores correct behaviour. quoted_match: [/^0\d+$/g] over [["1","01234"],["2","02345"]] is completely correct, because the id column resets the state every row. The bug only shows once matching fields land next to each other, which makes it look data-dependent rather than reproducible.
  • The missed quotes are the ones that were asked for. quoted_match is the mechanism for quoting that the delimiter/quote/record-delimiter checks cannot infer — preserving leading zeros, keeping 1-2 from being read as a date, stopping long digit strings from becoming scientific notation. Dropping it on alternate rows leaves a file that parses fine and means something else in the consumer.

Fix

Test the patterns with String.prototype.search, which searches from the start of the value and restores lastIndex afterwards, so the caller's regexp is left as it was found. For the stateless regexps that already worked it is equivalent to test.

The two duplicated match blocks are folded into one helper next to emits_separator, so they cannot drift apart — the empty-value branch carried the same test call. That branch is not observably affected today, since a zero-length match does not advance lastIndex, but it is the same expression and now shares the same fix. The helper also drops the intermediate array the filter allocated per field.

Verification

  • No behaviour change for stateless patterns. Swept the old test-based predicate against the new search-based one over 1819 pattern/value pairs — 14 non-global regexps and 6 strings, singly and in two-element arrays, against 17 values (empty, unicode, astral, multiline, 50-char) — 0 disagreements. Repeating one value four times, the old predicate returned inconsistent results in 35 of 119 global/sticky cases, the new one in 0.
  • Three regression tests in test/option.quoted_match.ts, each red before the change with the output noted in a trailing comment: across records, within one record, and one that pre-sets lastIndex to show the caller's regexp is no longer consumed. The existing file could not have caught this — every case used a single record and a non-global regexp.
  • npx tsc --noEmit clean; packages/csv-stringify 209 passing / 1 pending, packages/csv 25 passing. prettier --check and eslint clean on both changed files.
  • On this machine api.callback "catch error in end handler, see Fix a bug where errors could leak on extremely large stream chunks #386", Sample api.async.iterator.js and the memory-heavy Sample api.sync.memory.js also fail on an unmodified master, so they are unrelated to this change.

Found while fuzzing the stringifyparse round trip. Happy to split the helper extraction out of the fix, or to reset lastIndex on the caller's regexp instead of using search, if you prefer either shape.

…regexp

`quoted_match` regexps are supplied by the user and the same objects are
reused for every field of every record, but they were applied with
`RegExp.prototype.test`. That advances `lastIndex` on a global or sticky
regexp, so a match in one field moved the offset the next field started
matching from, and fields that should have been quoted were not:

    stringify([["1", "2"], ["3", "4"]], { quoted_match: /\d/g })
    // '"1",2\n"3",4'  instead of  '"1","2"\n"3","4"'

The option is documented as quoting all fields matching the regular
expression, and the result should not depend on where the previous field
happened to match. The failure is intermittent in practice: a field that
does not match resets `lastIndex` to 0, so an interleaved non-matching
column hides it, and it also leaks state to the caller's regexp.

Test the patterns with `String.prototype.search`, which searches from the
start of the value and restores `lastIndex`. It is equivalent to `test`
for the stateless regexps that already worked, verified by differentially
sweeping both predicates over 1819 stateless pattern/value pairs with no
disagreement. The two duplicated match blocks become one helper, next to
`emits_separator`, so they cannot drift apart; it also drops the
per-field intermediate array the `filter` used to allocate.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant