Perf regression: 3D_performance_test regresses 31% from v4.9.3 #2088

Summary

The performance numbers in https://mflowcode.github.io/documentation/expectedPerformance.html are no longer reproducible as of 47568a0. I am getting 0.468ns on H200 PCIe, which has identical memory bandwidth to GH200 and should come close to that (0.32ns). Another team independently reproduced 0.41ns on H200 SXM in #2087.

I have identified a few reasons. Some of them are concrete enough to fix, while others I am still finding out why.

One of the reasons identified was the HLLC kernel. The HLLC Riemann solver kernel on master runs 1.85-2.0x slower than v4.9.3 on NVIDIA GPUs, with local-memory traffic up 5.5x/13x (loads/stores) higher. On examples/3D_performance_test tested on H200 PCIe, NVHPC 26.9, with case optimization, the kernel time per call goes from 1.42-1.48 ms to 2.72-2.83 ms.

Bisecting this shows it is not one regression but a recurring one. The same local-memory-spilling kernel issue has been introduced, fully cured, and re-introduced three times over the last four months.

The grind also rose from 0.346ns to 0.43ns between v4.9.3 and pre-#1414 master with an identical HLLC kernel, so there must be more regression outside the HLLC kernel that I have yet to identify.

History

All of below are measured on the same H200 PCIe GPU, same compiler (NVHPC 26.9, CUDA 13.4.2), same case-optimized build of examples/3D_performance_test.

The Riemann hot-path decomposition introduced the spilling shape (#1556, then #1572 at its worst: 242 registers, 1.9x kernel time), removing the dead capabilities fully cured it (#1675 partial, #1713 made it back to v4.9.3 numbers), #1414 re-introduced it when hypoelasticity came back (its kernel's local-memory traffic is identical to #1675), #1811 reshaped it (264 gangs, 177 regs) and is the one PR that the CI failed to catch (see below), #1870 reverted the shape, and the thermochemistry-generation rewrites flipped it back and then worsened it (#1915 back to the #1811 shape, #1919 added 28% local-memory traffic on top)

Root cause of HLLC kernel regression

The kernel assembles its fluxes through seven per-cell work arrays indexed by the field loop (present since the decomposition, re-created by #1414's dual pass):

real(wp), dimension(sys_size) :: U_L, U_R
real(wp), dimension(sys_size) :: F_L, F_R, F_star_L, F_star_R, F_HLLC

Field-loop-indexed arrays cannot be register-promoted, so the flux assembly runs through thread-local memory rather than registers. ncu confirms these are not register spills (Local Memory Spilling Requests: 0). The arrays live in local memory by construction. The v4.9.3/#1713 implementation uses scalar star-state variables exclusively and shows 12.2M store sectors where the current kernel shows 161.2M.

On benchmarking CI case

The CI bench case (benchmarks/5eq_rk3_weno3_hllc, --gbpp 4 sees the same story with #1811 as the visible step, since it never ran before #1713:

I ran these myself. None of this was caught by CI, because PR #1811 (like #1823) has no benchmark workflow runs (the self job only triggers on approved reviews or maintainer-authored PRs, so author pushes skip it and the run stays vacuously green); after merge, the PR-vs-master comparison is blind to merged regressions because both master and PR regressed.

Reproducing

git checkout 0c849aa0
./mfc.sh build -i benchmarks/5eq_rk3_weno3_hllc/case.py -t pre_process simulation --gpu -j 32
./mfc.sh run -i benchmarks/5eq_rk3_weno3_hllc/case.py -t pre_process simulation --gpu --no-build --gbpp 4 -n 2
git checkout 808df619   # same commands; grind steps 0.48x -> 0.51x

# pick a commit I listed above in the table
git checkout <commit>
./mfc.sh build -i examples/3D_performance_test/case.py -t pre_process simulation \
    --gpu --case-optimization -j 32
./mfc.sh run -i examples/3D_performance_test/case.py -t pre_process simulation \
    --gpu --case-optimization --no-build -n 1
ncu -k "regex:hllc_riemann_solver" --launch-count 1 --kill yes \
    --metrics launch__registers_per_thread,gpu__time_duration.sum,l1tex__t_sectors_pipe_lsu_mem_local_op_ld.sum,l1tex__t_sectors_pipe_lsu_mem_local_op_st.sum \
    build/install/<slug>/bin/simulation

Environment

• NVIDIA H200 NVL (GH100, 132 SMs, CC 9.0), driver r580 (CUDA 13.4)
• NVHPC 26.9-0, CUDA 13.4.2, hpcx 2.50 Open MPI 5
• Flags: -fast -O3 -Mipa=fast,inline -march=native -gpu=cc90 -acc -Mfprelaxed -Minline (as generated by ./mfc.sh --gpu --case-optimization)

Fix

I have implemented optimisations that fix issues as far as I can identify (HLLC kernel regression being one). I will include them in a PR soon as part of NTU Team Supernova's contribution for the SC26 Student Cluster Competition.

As an additional suggestion for the maintainers, such regressions would not have been missed by the CI if the benchmark CIs were run on all PRs, including those opened by the author. Additionally, the benchmark CI should include this 3D performance test case, since that is what is written in the documentation as a reference.