Repository navigation
Chemistry on generic amdflang builds is capped at ten species by a literal array bound #1848
Description
Generic amdflang builds cap chemistry at ten species. The bound is a literal dimension(10) repeated across 21 declarations in nine files, swapped in for dimension(num_species) by the USING_AMD fypp guard when case optimization is off, and it is stated as a constraint in m_checker_common.fpp:
@:PROHIBIT(chemistry .and. num_species > 10, "num_species > 10 for AMDFLang when Case optimization is off")Two problems.
The check never runs. m_thermochem.f90 is generated per mechanism with concrete bounds, so an eleven-species mechanism gives a dimension(11) dummy against a dimension(10) actual and the build dies first:
error: Actual argument array has fewer elements (10) than dummy
repeated across m_chemistry.fpp, m_cbc.fpp, m_riemann_state.fpp, m_ibm.fpp and the three Riemann solvers. The reader gets a wall of argument-mismatch errors rather than the sentence explaining the limit. Hit while adding an eleven-species mechanism in #1821.
Raising the literal is not free. These are per-thread arrays in the hot kernels. m_riemann_solver_hllc.fpp:63 alone declares eleven of them:
real(wp), dimension(10) :: Ys_L, Ys_R, Xs_L, Xs_R, Gamma_iL, Gamma_iR, Cp_iL, Cp_iR, R_species, h_iL, h_iR110 doubles per thread at the current bound, 220 at dimension(20). On gfx90a a kernel drops from three waves per SIMD to two at 170 registers, so a bound raised for one mechanism is paid by every generic-build chemistry case on that backend.
Worth deciding between:
- Raise the bound and measure what it costs the chemistry kernels.
- Keep it and make the limit legible: a fypp-time failure carrying the message above, so the constraint is stated rather than inferred from argument mismatches.
- Drop the guards once the amdgpu compiler bug they work around is fixed, which removes the class.
Case-optimized builds are unaffected: the guard is not MFC_CASE_OPTIMIZATION and USING_AMD, so those take dimension(num_species) and any species count works.
Activity
sbryngelson commentedon Sep 11, 2026
Measured option 1. Same case (
C4EB58A8, 2D reacting mixing layer), same mechanism, generic chemistry GPU build both sides, amdflang OpenMP offload on MI210. Numbers are the 5-equation pure-fluid HLLC kernel:dimension(10)(master)dimension(20)Both build and pass the test, so raising the bound is correct. It costs 66 registers and 3.5x the scratch.
No occupancy tier is lost: waves per SIMD is
floor(512 / roundup(vgpr, 8)), so 260 -> 264 and 326 -> 328 are both one wave. Chemistry is already register-bound enough that the bound cannot make the wave count worse.The cost is spill traffic instead, and that is not negligible — 396 to 1372 bytes per thread in the hottest chemistry kernel, paid by every generic-build case on this backend so that mechanisms above ten species can compile.
Raising the literal also surfaced three latent shape mismatches that the bound had been masking, in code that assumed the padded array was exactly
num_specieslong: a whole-array assignmentW_species(:) = molecular_weights(:), the combinationYs_g(:) = 2*Ys_s(:) - Ys_IP(:), and two species arrays declared outside theUSING_AMDguard entirely. Each has to be pinned to1:num_speciesor carry the guarded bound. That fragility is itself an argument for option 3: the guards require every new species array to get both halves right, and the failure only appears on generic AMD builds.sbryngelson commentedon Sep 11, 2026
Timing for the same comparison, since the register and scratch figures above do not settle it. Same case, same GPU, back to back through the normal harness, MFC's own reported average:
dimension(10)(master)dimension(20)About 2 percent slower. One run each, and that is close to run-to-run variation, so it is better read as "no large penalty" than as an exact figure.
That is much cheaper than the static numbers suggested, and it revises what I wrote above. Two reasons it costs so little: the wave count does not move (260 and 326 both round to one wave per SIMD on gfx90a), and this case is dominated by the reaction and transport kernels rather than by HLLC, so the extra spill lands off the critical path.
So option 1 is more defensible than the register delta implies, at least for a case shaped like this one. A case where HLLC dominates would be the one to check before deciding, since that is the kernel carrying eleven of these arrays.
Give the surface species arrays the same bound as the ones they are c…
Give the surface species arrays the same bound as the ones they are c…