Skip to content

Fix multithreading reproducibility - #98

Merged
lrobion merged 2 commits into
MIT-LAE:mainfrom
lrobion:fix-multithreading-reproducibility
Aug 10, 2026
Merged

Fix multithreading reproducibility#98
lrobion merged 2 commits into
MIT-LAE:mainfrom
lrobion:fix-multithreading-reproducibility

Conversation

@lrobion

@lrobion lrobion commented May 14, 2026

Copy link
Copy Markdown
Contributor

Closes #19. Multithreaded simulations are now bitwise reproducible, adds CI checking bitwise reproducibility between two runs (1 and 4 threads) to ensure this does not happen again.

Multithreaded simulations even using a fixed random seed were not reproducible. This was for three reasons:

  1. Multithreaded access to the global RNG state in updating the temperature perturbation. Even if the RNG state is consistent, the threads access it in a non-deterministic way making the temperature perturbation different even run.
  2. Plain race condition in a conversion factor in LAGRIDPlumeModel.cpp and two in Aerosol.cpp (including the moment calculation)
  3. Parallel reductions in Aerosol.cpp that due to floating point non-associativity lead to different results. This floating point noise is magnified because the shear applied to the contrail depends on finding the maximum index of xOD with depends on multiple non-deterministic reductions. Eventually the numerical noise is sufficient to shift the index of the max xOD which changes the shear which then leads to a different simulation grid.

This addresses all 3 issues but comes at a signficant performance drop (40% slower simulations on 4h of the ISSL 140 test) due to the non-parallel reductions (see new comments). This performance drop can be reduced further if we don't repeat these (now slower) reductions multiple times per timestep (see #97).

I think once we address the repeated calculations, we can decide if the performance drop is still significant enough that want to look into this more. Further work on the performance of this should probably happen after #92 is merged as it also deals with performance in Aerosol.cpp.

@lrobion
lrobion force-pushed the fix-multithreading-reproducibility branch from 44a44b8 to a9e68ce Compare August 8, 2026 00:40
@lrobion

lrobion commented Aug 8, 2026

Copy link
Copy Markdown
Contributor Author

Rebased on the #92, added back safe parallelism... The slowdown is negligible on my machine (issl_140 example for 4h, mean of 3 runs):

Threads Main (s) New (s) Change
1 195.8 198.6 +1.4%
2 121.4 120.1 -1.1%
4 76.9 76.7 -0.2%
8 74.6 74.7 +0.1%

All results are bitwise identical with main running on 1 CPU. For the moment calculations I changed the loop order to make parallelism safe where each thread computes a row of moments in the 2D grid so they do not overlap but still keep continuous data access patterns in the innermost loop

@lrobion
lrobion force-pushed the fix-multithreading-reproducibility branch from fa1596f to 959b1a4 Compare August 10, 2026 20:26
@lrobion
lrobion merged commit fa1b556 into MIT-LAE:main Aug 10, 2026
2 checks passed
@lrobion
lrobion deleted the fix-multithreading-reproducibility branch August 10, 2026 20:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Inconsistent Results When "OpenMP Num Threads" is Greater Than 1

1 participant