Fix multithreading reproducibility - #98
Merged
lrobion merged 2 commits intoAug 10, 2026
Merged
Conversation
lrobion
force-pushed
the
fix-multithreading-reproducibility
branch
from
August 8, 2026 00:40
44a44b8 to
a9e68ce
Compare
Contributor
Author
|
Rebased on the #92, added back safe parallelism... The slowdown is negligible on my machine (issl_140 example for 4h, mean of 3 runs):
All results are bitwise identical with main running on 1 CPU. For the moment calculations I changed the loop order to make parallelism safe where each thread computes a row of moments in the 2D grid so they do not overlap but still keep continuous data access patterns in the innermost loop |
lrobion
force-pushed
the
fix-multithreading-reproducibility
branch
from
August 10, 2026 20:26
fa1596f to
959b1a4
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #19. Multithreaded simulations are now bitwise reproducible, adds CI checking bitwise reproducibility between two runs (1 and 4 threads) to ensure this does not happen again.
Multithreaded simulations even using a fixed random seed were not reproducible. This was for three reasons:
LAGRIDPlumeModel.cppand two inAerosol.cpp(including the moment calculation)Aerosol.cppthat due to floating point non-associativity lead to different results. This floating point noise is magnified because the shear applied to the contrail depends on finding the maximum index ofxODwith depends on multiple non-deterministic reductions. Eventually the numerical noise is sufficient to shift the index of the maxxODwhich changes the shear which then leads to a different simulation grid.This addresses all 3 issues but
comes at a signficant performance drop (40% slower simulations on 4h of the ISSL 140 test) due to the non-parallel reductions(see new comments). This performance drop can be reduced further if we don't repeat these (now slower) reductions multiple times per timestep (see #97).I think once we address the repeated calculations, we can decide if the performance drop is still significant enough that want to look into this more. Further work on the performance of this should probably happen after #92 is merged as it also deals with performance in
Aerosol.cpp.