Improves test suite performance by 8.5-25x [experiment!] - #22917
Improves test suite performance by 8.5-25x [experiment!]#22917NickSdot wants to merge 24 commits into
Conversation
github-actions
Bot
added
Extension: openssl
Extension: curl
Extension: opcache
Extension: pdo (core)
Extension: pdo_dblib
Extension: phar
Extension: shmop
Extension: snmp
Extension: sockets
Extension: sodium
Extension: standard
Extension: sysvmsg
Extension: sysvshm
SAPI: cli
Category: CI
labels
|
Didn't expect this to be smooth in the very first CI run; will look into these. |
|
Interesting work! Related: #20980 Please submit each commit as a separate PR to make reviewing easier (maybe not all at once to avoid spamming the repos. Start with the most impacting ones). I will review at least some of them. |
Thanks, Arnaud! Would you, for now, be fine with reviewing the individual commit diffs here? Like, so that we can pre-decide a few things here first, and based on that I am happily splitting up into individual PRs. The commits table in the PR desc brings you straight to the same diff you would have in a dedicated PR. #20980 is indeed (admittedly naively) solved by this change. By decreasing batch size, much longer running batches get automatically more rare. Though, this is something that surely could be further improved in the future. |
@arnaud-lb I stand corrected here. Didn't expect you commenting directly on the commits but here in the changes; my force pushes to keep the related changes all in one commit break the relation to the comments in that case. I already cherry picked #22939 and #22934. Will will follow up with more. |
|
I will keep answering here for meta things not related to only one specific PR.
|
2 participants
While working on #22799 I was bothered by the test suite being so slow. CI needs ~45 minutes from push to finish; running tests locally also takes quite a while. I was wondering why that is and whether it must be like that. Hence, I decided to run an experiment to find out if we could make the developer experience notably better.
Note
This experiment was fully executed by Codex. I was throwing some money on OpenAI to let GPT 5.6 Sol pursue the
/goalto make the test suite "massively" faster. This PR is the result of Codex trying exactly that -- for 14 hours straight. Consequently, I will not pretend that I understand the implications of every proposal -- and I also did not review everything in detail yet. Though, I did spend plenty of time to confirm the claims and reworking things.The result are interesting enough to open this clearly flagged as LLM-executed experiment, hoping that it can serve as a discussion starter to find out whether anything can be picked up. Would love to get feedback from people with more experience than me, to pin down which of these are legit ideas and which are dead-ends right away.
Extracted PRs
All canges are now extracted to logical PRs as outlined below.
Surfaced fixes extracted to separate PRs
Results
Values in seconds; measured on macOS. The full suite improvements are already awesome on their own. But targeted runs also show up to 25x faster results (
-j10vs.-j10), many at least 2x faster, and very few slightly slower.Full Suite
Partial Runs
The -j1 row compares the master default with the explicit sequential opt-out.
Misc
existing, internal
--repeatflag does exactly that.-j2, and parallel not being the default. I, however, could not find out what these reasons are; hence, I kept changes Codex proposed.MAX_CONCURRENCY, but it shows gains.