Repository navigation
perf: keep the value stack's height in a register across tail-call handlers - #79
matthargett wants to merge 6 commits into
Conversation
7bb7a50 to
29679a7
Compare
originally reported in #79 Signed-off-by: Henry <mail@henrygressmann.de>
based on the fix in #79 Signed-off-by: Henry <mail@henrygressmann.de>
|
I just pushed a fix for the function call panics from this to next. I do see a good performance improvement overall, and I’d definitely like to bring in more of these changes. I’m just a bit hesitant to add more nightly features right now, so I might shuffle the flags around to separate tail-call dispatch from a more general nightly feature. I’ll probably do some more testing in this direction and pull the changes in incrementally rather than all at once. |
29679a7 to
fc1f817
Compare
On arm64_32 (watchOS) the Rust ABI passes the 8-byte Instruction by reference, so every dispatch stores it and the next handler loads it back. The C convention passes it in a register, and `C-unwind` still lets a host function's panic unwind. Other targets keep the Rust ABI.
The height of a value-stack lane is its own field, over a Vec that holds every slot the lane has reached. Slots above the height hold stale values that are written before they are read again. The tail-call handlers can then pass the height between them in a register and write it back without unsafe code. Entering a function writes its locals, and its operand-stack reservation if that is at most 64 slots, so later entries at that height compare once. A larger reservation stays capacity, and a push writes each slot the first time it reaches it, so a function whose deep branch rarely runs keeps no pages for it; once the locals have slots, later entries compare and go on. With 64 stores of a module whose untaken branch declares a 30000-deep i64 stack, each store keeps next's footprint (35 KB on macOS, 16-19 KB on glibc) instead of 273 KB and 250 KB. v128.any_true and i8x16.all_true test the vector with integer arithmetic: LLVM scalarized their iterator forms once pushes gained that path. Only builds with debug assertions check indices against the height. CI also runs the tests optimized with them, keeping the release profile's wrapping arithmetic. Entering a function zeroes one or two locals with direct stores, and only the ones in between with memset.
The executor takes the store's value stack when it is created and gives it back when it is dropped, and around host calls, which read their arguments from the store and may call back into wasm. Every handler receives the executor as a `&mut`, so a stack access reached through it needs one load fewer (the store pointer), and the compiler knows that no store to a slot or to memory changes a stack's height, so it can keep the heights in registers within a handler.
…lers Each tail-call handler receives the height of the 32-bit value stack, which most instructions push to or pop from, as its last argument. It writes the height to the stack before its body and reads it back before dispatching, and with the stack in the executor the compiler can keep it in a register in between, as it does in most handlers. A handler no longer loads the height its predecessor stored, a store-to-load round trip through memory on every dispatch. The Unbudgeted handlers now take six integer arguments and the Bounded ones five, which stay in registers on arm64 (eight) and x86-64 System V (six). Windows x64 passes four, so there the height goes on the stack along with the Instruction, which already does without this change.
…ows x64 Windows' calling convention passes four integer arguments in registers, so every dispatch stored the Unbudgeted handlers' last two and the next handler loaded them back. On Windows x64 the handlers now use the System V convention, which passes six, so the 32-bit height stays in a register. The instruction still goes through the stack: under the System V rules this enum is passed in memory. A CI job runs the tests with tail calls on Windows.
Windows x64 passes four integer arguments in registers. The bounded handlers (fuel and time budgets) take four besides the 32-bit height, so passing the height put a fifth argument on the stack under either convention: the height under the Rust ABI, the instruction under System V, which passes this enum in memory. On Windows x64 the bounded handlers keep the Rust ABI and leave the height out: they read it from the stack before dispatching, as before the height was passed. The unbudgeted handlers keep the System V convention and the height.
d9a4749 to
1e2d47a
Compare
I figured out how to do this without needing nightly features on Windows x64: the unbudgeted handlers now use extern "sysv64-unwind", which gives us some extra home registers on that platform. It's not completely putting all the hot references that we are able to accomplish on arm64_32 and arm64, or even x64 Linux, so there are still more stack references in the disassembly, but the overall uplift on all the bnechmarks is still pretty awesome :D
happy to split it up, but before you were saying the smaller incremental PRs weren't an obvious win across all the benchmark/target scenarios. let me know if I should keep the integrated PR, but with better commit splits and per-commit benchmark result tables, or some other way that makes your life easier :) |
Stack, in merge order: #74 → #81, which is #79's six commits on top of #74. #79, this PR is the same six commits on
nextalone. All three share one table layout for side-by-side comparison.The stack-height half of #78: the tail-call handlers pass the 32-bit value stack's height to each other in a register instead of through memory. On
nextevery handler loads the height its predecessor just stored. That store-to-load round trip is on every dispatch's critical path, and once enough handler pairs alias through it the CPU stops predicting them, so loads wait or the pipeline flushes. On the iPhone 12 the height commit removed 88–97% of the memory-order flushes. The rest of the series makes the calling conventions keep the height and the instruction in registers on the targets where they didn't.Results
CoreMark is −12.9% averaged over the five chips. These are the rows where the series gains least. GC binary trees, which mostly allocates, moves within ±3%. The other bench-suite fixtures and WasmBench rows are faster too.
Change in cycles per call against
nextwithnightly-tail-calls, median of five interleaved runs per build. i7-8650U: a Surface Book 2 (Kaby Lake, 2017) on Windows x64, pinned to one core, core cycles counted with VTune. A10X: Apple TV 4K (1st gen). A12, A13: iPhone XS Max and iPhone SE, performance cores. M4: performance cores. argon2id, compress and decompress arecargo bench-suite's fixtures, the EH parser and GC binary trees are WasmBench rows, and CoreMark runs with a clock that advances a fixed step per read, so every build runs the same iterations.Commits
Each builds and passes the tests on its own, and none needs a nightly feature.
extern "C-unwind", so the 8-byteInstructiontravels in a register; the Rust ABI passes it by reference there.Vec, which only grows; slots above the height are stale until written. The handlers can then write the height back withoutunsafe.extern "sysv64-unwind", six argument registers instead of four. Without this, commits 1–4 are up to 8% slower on Windows.nexton Windows.For discussion
C-unwindon arm64_32 andsysv64-unwindon Windows x64 (stable, and host panics still unwind). Under System V rustc passesInstructionin memory, which is why x86-64 Linux keeps the Rust ABI.extern "rust-preserve-none"would give twelve or more argument registers everywhere, but it's nightly-only.next's 35 KB per store instead of 273 KB).tests/host_nested_panic.rs).next, because a fifth argument spills under either convention.