Repository navigation
Conversation
guybedford
left a comment
There was a problem hiding this comment.
Thanks, the analysis is right: Slice is 12 bytes and shorthand literal exports (a,) can produce one every two code units, so 8 * len is exactly the worst-case bound, and the tests reproduce the trap on main.
Rather than scaling the fixed reservation though, I'd prefer to bound-check in the allocator and grow on demand, which is what es-module-lexer does now: sa() snapshots analysis_limit = __builtin_wasm_memory_size(0) << 16, and each _add* calls an inline ensureAnalysisCapacity(sizeof(Slice)) that falls into a noinline growAnalysis doing __builtin_wasm_memory_grow when analysis_head + size > analysis_limit. The JS side can then keep len * 4 (or lower) as a floor. Wasm memory never shrinks, so with 8 * len every parse of a large bundle permanently pins 8x its size for a case that almost never occurs; growing on demand keeps retained memory proportional to actual results, at the cost of one compare per recorded export. The JS wrapper only touches wasm.memory.buffer before parseCJS, so in-wasm growth detaching the buffer is safe.
This needs a wasm rebuild, so happy to help with that if useful. The tests here are good as the regression either way.
Dense object exports can trap in WebAssembly because the host reserves four bytes per UTF-16 code unit, below the result-list requirement. A result node needs 12 bytes and can occur every two code units. Reserve eight bytes per unit, including the source. This doubles the input-based capacity allowance. Application measurements show no consistent CPU regression. Signed-off-by: Ruben Bridgewater <ruben.bridgewater@datadoghq.com>
Dense shorthand exports produce a 12-byte record for every two code units, so a source-sized analysis allowance can overflow. Reserving for that worst case retains excess memory on sparse inputs. A 1,048,596-code-unit sparse input retains 4,259,840 bytes instead of 8,454,144 bytes. Dense shorthand parses cost 2.18-3.99% more time (Node 22.23.3, V8 12.4.254.21-node.57, seven alternating trials, trimmed means, two fresh processes).
f45d1a0 to
07b764e
Compare
The proportional analysis allowance wastes linear memory on ordinary modules with few exports. Parsing a 296-file Mocha, Babel, and Terser corpus retains 2.19 MiB of Wasm linear memory instead of 4.25 MiB.
guybedford
left a comment
There was a problem hiding this comment.
Thanks, this looks good to me. Verified locally:
- Confirmed the bug on main: a fresh instance parsing
module.exports={+'v,'.repeat(2047)+v};throwsRuntimeError: memory access out of bounds. The new tests fail 12 cases against main and pass here in all three modes. - Both this PR's and main's committed
lib/lexer.wasmare byte-identical to a rebuild with the pinned docker toolchain. - Since the JS now reserves source only, the terminator can land in the last two bytes of memory. I probed ~13,800 inputs placing every keyword prefix the lexer matches at that boundary and saw no traps, which checks out since
str_eqNshort-circuits at the terminator. - Bench is within noise versus main.
Growing on demand from Wasm rather than over-reserving the worst case (8 bytes per code unit) in JS is the right trade-off here, since analysis is tiny for real files.
One adjacent pre-existing issue noted inline that this PR may want to fold in since it is already touching the reservation line, otherwise happy for it to be a follow-up.
| // sa(len) writes a terminator at index len. | ||
| let memory = wasm.memory.buffer; | ||
| const extraMem = (wasm.__heap_base.value || wasm.__heap_base) + len * 4 - memory.byteLength; | ||
| const extraMem = (wasm.__heap_base.value || wasm.__heap_base) + (len + 1) * 2 - memory.byteLength; |
There was a problem hiding this comment.
Pre-existing, not introduced here, but this is the line that encodes it: len is source.length + 1, so sa(len) writes the terminator at index source.length + 1, and index source.length is never written by either copy path. It keeps whatever the previous parse left there, so results depend on the prior call:
parse('exports.a=1;');
parse('exports.a'); // { exports: ['a'] } — a fresh instance gives []I reproduced this on both main and this branch, and verified against this PR's wasm that terminating at source.length (i.e. sa(source.length) with a reservation of (source.length + 1) * 2) fixes it. Could be folded in here with a test, or as a follow-up.
| void* analysis_base; | ||
| void* analysis_head; | ||
| // The end of a full Wasm32 memory does not fit in uintptr_t. | ||
| uint64_t analysis_head; |
There was a problem hiding this comment.
Nit, non-blocking: the 64-bit head/limit only matter when memory is exactly 4 GiB (where memory_size << 16 wraps to 0 in u32). A uintptr_t pair with the check written as analysis_limit - analysis_head < size is simpler and only loses that edge, which ends in a trap either way. Fine to keep as is.
Dense shorthand exports can exhaust analysis capacity because a 12-byte record can occur every two code units. Grow Wasm memory before writing beyond capacity.
A 296-file Mocha, Babel, and Terser corpus retains 2.19 MiB of Wasm linear memory.