After v8.4.1 code merge, rap/summer hrrrv5 CI tests have differences

[clark-evans]

After the v8.4.1 code merge, all of the rap/summer HRRRv5 CI tests return differences in the diagnostic files - but not at all forecast times. A sample of the output is below:

DIFFER : VARIABLE : w_velocity_mean : POSITION : [0,0] : VALUES : -nan <> -nan
DIFFER : VARIABLE : w_velocity_mean : POSITION : [0,0] : VALUES : -nan <> -nan
DIFFER : VARIABLE : w_velocity_mean : POSITION : [0,0] : VALUES : -nan <> -nan
DIFFER : VARIABLE : w_velocity_mean : POSITION : [0,0] : VALUES : -nan <> -nan
DIFFER : VARIABLE : w_velocity_mean : POSITION : [0,0] : VALUES : -nan <> -nan
DIFFER : VARIABLE : w_velocity_mean : POSITION : [0,0] : VALUES : -nan <> -nan
--------------------------------------------------
Comparing /__w//run_bl/history.2024-08-15_21.00.00.nc to /__w//run_rt/history.2024-08-15_21.00.00.nc  PASS
Comparing /__w//run_bl/history.2024-08-15_18.00.00.nc to /__w//run_rt/history.2024-08-15_18.00.00.nc  PASS
Comparing /__w//run_bl/history.2024-08-16_00.00.00.nc to /__w//run_rt/history.2024-08-16_00.00.00.nc  PASS
Comparing /__w//run_bl/history.2024-08-15_23.00.00.nc to /__w//run_rt/history.2024-08-15_23.00.00.nc  PASS
Comparing /__w//run_bl/history.2024-08-15_18.10.00.nc to /__w//run_rt/history.2024-08-15_18.10.00.nc  PASS
Comparing /__w//run_bl/history.2024-08-15_21.30.00.nc to /__w//run_rt/history.2024-08-15_21.30.00.nc  PASS
Comparing /__w//run_bl/history.2024-08-15_20.30.00.nc to /__w//run_rt/history.2024-08-15_20.30.00.nc  PASS
Comparing /__w//run_bl/history.2024-08-15_22.30.00.nc to /__w//run_rt/history.2024-08-15_22.30.00.nc  PASS
Comparing /__w//run_bl/history.2024-08-15_21.20.00.nc to /__w//run_rt/history.2024-08-15_21.20.00.nc  PASS
Comparing /__w//run_bl/history.2024-08-15_20.00.00.nc to /__w//run_rt/history.2024-08-15_20.00.00.nc  PASS
Comparing /__w//run_bl/history.2024-08-15_20.40.00.nc to /__w//run_rt/history.2024-08-15_20.40.00.nc  PASS
Comparing /__w//run_bl/history.2024-08-15_22.20.00.nc to /__w//run_rt/history.2024-08-15_22.20.00.nc  PASS
Comparing /__w//run_bl/history.2024-08-15_20.20.00.nc to /__w//run_rt/history.2024-08-15_20.20.00.nc  PASS
Comparing /__w//run_bl/history.2024-08-15_19.40.00.nc to /__w//run_rt/history.2024-08-15_19.40.00.nc  PASS
Comparing /__w//run_bl/history.2024-08-15_22.40.00.nc to /__w//run_rt/history.2024-08-15_22.40.00.nc  PASS
Comparing /__w//run_bl/history.2024-08-15_21.40.00.nc to /__w//run_rt/history.2024-08-15_21.40.00.nc  PASS
Comparing /__w//run_bl/history.2024-08-15_22.10.00.nc to /__w//run_rt/history.2024-08-15_22.10.00.nc  PASS
Comparing /__w//run_bl/history.2024-08-15_19.20.00.nc to /__w//run_rt/history.2024-08-15_19.20.00.nc  PASS
Comparing /__w//run_bl/history.2024-08-15_21.10.00.nc to /__w//run_rt/history.2024-08-15_21.10.00.nc  PASS
Comparing /__w//run_bl/history.2024-08-15_22.00.00.nc to /__w//run_rt/history.2024-08-15_22.00.00.nc  PASS
Comparing /__w//run_bl/history.2024-08-15_19.10.00.nc to /__w//run_rt/history.2024-08-15_19.10.00.nc  PASS
Comparing /__w//run_bl/history.2024-08-15_23.30.00.nc to /__w//run_rt/history.2024-08-15_23.30.00.nc  PASS
Comparing /__w//run_bl/history.2024-08-15_18.40.00.nc to /__w//run_rt/history.2024-08-15_18.40.00.nc  PASS
Comparing /__w//run_bl/history.2024-08-15_23.20.00.nc to /__w//run_rt/history.2024-08-15_23.20.00.nc  PASS
Comparing /__w//run_bl/history.2024-08-15_18.30.00.nc to /__w//run_rt/history.2024-08-15_18.30.00.nc  PASS
Comparing /__w//run_bl/history.2024-08-15_18.20.00.nc to /__w//run_rt/history.2024-08-15_18.20.00.nc  PASS
Comparing /__w//run_bl/history.2024-08-15_23.40.00.nc to /__w//run_rt/history.2024-08-15_23.40.00.nc  PASS
Comparing /__w//run_bl/history.2024-08-15_19.00.00.nc to /__w//run_rt/history.2024-08-15_19.00.00.nc  PASS
Comparing /__w//run_bl/history.2024-08-15_20.10.00.nc to /__w//run_rt/history.2024-08-15_20.10.00.nc  PASS
Comparing /__w//run_bl/history.2024-08-15_19.30.00.nc to /__w//run_rt/history.2024-08-15_19.30.00.nc  PASS
Comparing /__w//run_bl/history.2024-08-15_23.10.00.nc to /__w//run_rt/history.2024-08-15_23.10.00.nc  PASS
Comparing /__w//run_bl/diag.2024-08-15_19.00.00.nc to /__w//run_rt/diag.2024-08-15_19.00.00.nc  NOT IDENTICAL
Comparing /__w//run_bl/diag.2024-08-15_18.00.00.nc to /__w//run_rt/diag.2024-08-15_18.00.00.nc  PASS
Comparing /__w//run_bl/diag.2024-08-15_18.48.00.nc to /__w//run_rt/diag.2024-08-15_18.48.00.nc  NOT IDENTICAL
Comparing /__w//run_bl/diag.2024-08-15_22.00.00.nc to /__w//run_rt/diag.2024-08-15_22.00.00.nc  PASS
Comparing /__w//run_bl/diag.2024-08-15_18.36.00.nc to /__w//run_rt/diag.2024-08-15_18.36.00.nc  NOT IDENTICAL
Comparing /__w//run_bl/diag.2024-08-15_18.24.00.nc to /__w//run_rt/diag.2024-08-15_18.24.00.nc  NOT IDENTICAL
Comparing /__w//run_bl/diag.2024-08-15_20.00.00.nc to /__w//run_rt/diag.2024-08-15_20.00.00.nc  NOT IDENTICAL
Comparing /__w//run_bl/diag.2024-08-16_00.00.00.nc to /__w//run_rt/diag.2024-08-16_00.00.00.nc  PASS
Comparing /__w//run_bl/diag.2024-08-15_18.12.00.nc to /__w//run_rt/diag.2024-08-15_18.12.00.nc  NOT IDENTICAL
ALL TESTS PASSED, BUT OUTPUT DIFFERS FROM BASELINE.
6

This issue did not appear before the v8.4.1 code merge, and it does not appear with the gfs/winter or rap/winter HRRRv5 CI tests.

The v8.4.1 code merge did include a change to the streams.atmosphere file for just this test. We changed the diagnostic frequency from 12 min to 10 min, which was while we were using the shorter time step for this test to get it to pass. However, we never changed it back, and I don't recall whether the diagnostic frequency change was tied to using the shorter time step (as it appears from the discussion history) or for another reason. Tagging @guoqing-noaa and @dustinswales for awareness.

[clark-evans]

For what it's worth, our streams.atmosphere suggests output every 10 min, but I'm only seeing output every 12 min. Not entirely sure what's going on, but I also have conference brain...