[3.15] gh-109638: Fix exponential time in csv.Sniffer for doubled quotes

[serhiy-storchaka]

The regular expression which detects a doubled quote character lets its runs match the quote character too, so every quote can be taken either as part of a run or as one of the three matched quotes. With three unbounded quantifiers over a region of quotes and delimiters the search is O(n^5) -- 254 s at n=100 in the report.

>>> sample = '"",' * 100 + '"' * 100 + '0' + '"' * 100 + '0'
>>> csv.Sniffer().sniff(sample)   # minutes

The runs now exclude the quote character and are matched possessively, which leaves exactly one way to read any input.

   n      before      after
  60    6.1300 s    0.0002 s
2000        --      0.0009 s

The runs also no longer exclude the delimiter and the line break, which the original did for no reason a reader shares -- so a doubled quote is now found in a field containing them as well, e.g. ,"All-Weather Dining Table, Round 48""",. That is the bulk of the behaviour change: 42 of 560 corpus files, and against ground truth doublequote=True improves from 25.8% to 36.7% and doublequote=False from 77.0% to 78.7%, with delimiter and quotechar unchanged.

Note #109639 proposed an atomic group on the first run only. It removes the time but also the detection: doublequote becomes unconditionally False (0 of 126 annotated files, vs 32 for 3.15).

main is not affected: the sniffer was rewritten there in gh-83273.

• Issue: csv.Sniffer regular expression has significant backtracing #109638

[serhiy-storchaka]

Note that the widened detection -- a doubled quote is now found in a field which contains the delimiter or a line break -- changes results on 42 of 560 files in the CSVsniffer corpora. Against that corpus's ground truth the change is positive on both sides: doublequote=True 25.4% -> 37.3%, doublequote=False 77.0% -> 78.5%.

There are two alternatives. #154865 changes no results, but leaves the search polynomial (O(n^5) -> O(n^4), still four and a half minutes on a 2 KB sample). Or, since 3.15 is still in beta, the new engine can be backported from main, which fixes this issue along with seven others; it changes results on 122 of 560 files.

[Punisheroot]

Thanks again for identifying the residual polynomial behavior in my original patch. I reworked and reopened #154865 at a45aca2, using the possessive quote-exclusion approach from this PR as the pre-filter, then tested the current #154868 head (92b3133) against broader correctness and scaling cases.

I found two additional cases not covered by the original reproducer.

Empty-delimiter scaling

When delimiter == '', the left boundary in the current pattern becomes (?:|^). The empty alternative lets re.search() retry the pattern at every position, leaving quadratic behavior on single-column samples.

Post-rebase Windows x64 Release medians for:

sample = '"a"\n' + ' ' * n

Sample size	#154868	Revised #154865
4,004 B	2.000 ms	0.044 ms
8,004 B	7.686 ms	0.088 ms
16,004 B	29.598 ms	0.175 ms
32,004 B	123.973 ms	0.352 ms
64,004 B	472.371 ms	0.710 ms

The revised patch uses separate non-empty- and empty-delimiter patterns, so the empty case can only start at \A, CR, or LF.

Cross-field false positives

An unanchored match can start at a delimiter inside one quoted field and finish in another. For example:

sample = '",","",","'

This is three quoted fields containing ,, an empty string, and ,; it has no doubled quote, but the current #154868 pattern reports doublequote=True.

The revised patch keeps the possessive pre-filter, then uses an anchored linear pattern to rule out a complete well-formed sample without doubled quotes. It also handles CRLF/CR boundaries and literal quote characters in unquoted fields.

Generated-input results

I generated two independent batches of 500,000 semantically labelled samples using seeds 0xA11CE and 0xBADC0DE. Coverage included comma, semicolon, tab, pipe, space, and the Sniffer's internal empty delimiter; both quote characters; LF/CRLF/CR; mixed quoted and unquoted fields; embedded delimiters and line breaks; doubled quotes; and literal quotes in unquoted fields. Ground truth came from the values used to construct and escape each field.

Implementation	False negatives	False positives
#154868 (92b3133)	52,905	6,208
Revised possessive pre-filter alone	0	742
Revised #154865 (a45aca2)	0	0

I also exhaustively enumerated 506,996 valid combinations: the revised implementation had 0 false negatives and 0 false positives.

On the 548 corpus files available in my local checkout, newline-normalized input produced no decision differences from #154868. With raw newlines there were three differences: two agreed with the annotations and one did not; I am reporting the last one conservatively as an annotated regression.

The revised implementation remains effectively equivalent to #154868 on the original gh-109638 family:

Sample size	#154868	Revised #154865
80,002 B	0.938 ms	0.964 ms
160,002 B	1.865 ms	1.920 ms
320,002 B	3.737 ms	3.861 ms
640,002 B	7.499 ms	7.471 ms

There is one measured trade-off: when the first real doubled quote is very late in a 128 KB comma-delimited sample, #154868 took 1.870 ms and the revision took 2.497 ms because it performs the structural validation pass. Both remain linear.

The complete implementation rationale, rejected alternatives, corpus accounting, and post-rebase validation are documented in the reopening comment on #154865. I would appreciate your thoughts, particularly on the separate empty-delimiter handling and the anchored no-doublequote validation.

[serhiy-storchaka]

Thanks to @Punisheroot for finding in #154865 that the pattern reported a doubled quote for '",","",","', which is three quoted fields ,, an empty one and ,. It required a doubled quote character in the middle, so it could match from a delimiter inside one quoted field to a quote in another one.

Now whole quoted fields are matched and their bodies are tested separately. No file of 560 in the corpora changes, so the numbers above still hold; nothing in the test suite noticed either, hence the added test.

[Punisheroot]

Thanks to @Punisheroot for finding in #154865 that the pattern reported a doubled quote for '",","",","', which is three quoted fields ,, an empty one and ,. It required a doubled quote character in the middle, so it could match from a delimiter inside one quoted field to a quote in another one.

Now whole quoted fields are matched and their bodies are tested separately. No file of 560 in the corpora changes, so the numbers above still hold; nothing in the test suite noticed either, hence the added test.

Thanks for updating the pattern and for the credit. I tested the current head
(e78bb80) and found two remaining record-boundary cases.

A doubled quote in the last field of a CRLF record is missed:

sample = 'x,"a""b"\r\ny,"c"\r\n'
csv.Sniffer().sniff(sample).doublequote  # False, expected True

Conversely, this valid CR-separated CSV contains no doubled quote but reports
True:

sample = '"",","\r"",","'
list(csv.reader(io.StringIO(sample, newline="")))
# [['', ','], ['', ',']]

This appears to be because ^/$ with re.MULTILINE do not fully represent
CR/CRLF record boundaries, allowing a match to cross records.

The empty-delimiter case also remains approximately quadratic:

sample = '"a"\n' + ' ' * n

On Windows x64 Release, the current head grows from 2.30 ms at 4 KB to
432.42 ms at 64 KB.

#154865 handles CR/CRLF and the empty delimiter separately. Would you consider
adding these cases to this PR’s regression tests?