[develop] Fix for nightly runs

[EdwardSnyder-NOAA]

DESCRIPTION OF CHANGES:

The AWS Jenkins nightly runs are failing. Initially it was due to lack of staged SRW data, but that was resolved, which led to failures in the make_sfc_climo task. This failure was captured in issue #1328. Troubleshooting and testing revealed that the executables where built with a newer GNU version than what is used at runtime, causing the executables to fail. The easiest solution is to load this missing GNU library before the executables run.

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

TESTS CONDUCTED:

• derecho.intel
• gaeac6.intel
• hercules.intel
• orion.intel
• ursa.gnu
• ursa.intel
• NOAA Cloud (indicate which platform)
• Jenkins
• fundamental test suite
• comprehensive tests (specify which if a subset was used)

DEPENDENCIES:

DOCUMENTATION:

ISSUE:

This resolves issue #1328.

CHECKLIST

• My code follows the style guidelines in the Contributor's Guide
• I have performed a self-review of my own code using the Code Reviewer's Guide
• I have commented my code, particularly in hard-to-understand areas
• My changes need updates to the documentation. I have made corresponding changes to the documentation
• My changes do not require updates to the documentation (explain).
• My changes generate no new warnings
• New and existing tests pass with my changes
• Any dependent changes have been merged and published

LABELS (optional):

A Code Manager needs to add the following labels to this PR:

• Work In Progress
• bug
• enhancement
• documentation
• release
• high priority
• run_ci
• run_we2e_fundamental_tests
• run_we2e_comprehensive_tests
• help wanted

CONTRIBUTORS (optional):

@benkozi

[MichaelLueken]

@EdwardSnyder-NOAA -

Thanks for making the necessary fix to allow the SRW to run on CSPs! The addition to the NOAA Cloud python_srw.lua file are similar to those made in the Gaea C6 python_srw.lua file and it looks good to me.

Approving now.

[RatkoVasic-NOAA]

LGTM

[MichaelLueken]

The Gaea C6 Jenkins runner is down, so I will manually run the coverage WE2E tests on that platform.

[MichaelLueken]

The Gaea C6 coverage WE2E tests have successfully passed:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used 
----------------------------------------------------------------------------------------------------
custom_ESGgrid_NewZealand_3km_20260415091027                       COMPLETE              70.37
grid_RRFS_CONUScompact_13km_ics_HRRR_lbcs_RAP_suite_RRFS_sas_2026  COMPLETE              36.93
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR_2026041509  COMPLETE              39.81
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15_thompson  COMPLETE             435.01
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_HRRR_suite_HRRR_2026041  COMPLETE              45.62
smoke_dust_grid_RRFS_CONUS_3km_suite_HRRR_gf_20260415091131        COMPLETE            1205.56
2020_CAPE_20260415091147                                           COMPLETE              41.65
2020_easter_storm_20260415091203                                   COMPLETE              41.48
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE            1916.43

Awaiting completion of the rest of the Jenkins tests now.

[MichaelLueken]

The Jenkins tests have successfully passed on Hercules, Orion, Ursa GNU, and Ursa Intel. Still awaiting WE2E tests to complete on Derecho.

The FunctionalWorkflowTaskTests stage has failed on Azure, and is still running on PC AWS - https://jenkins.epic.oarcloud.noaa.gov/blue/organizations/jenkins/ufs-srweather-app%2Fpipeline/detail/PR-1372/1/pipeline/719

[MichaelLueken]

@EdwardSnyder-NOAA -

It looks like the modification that was made to allow the WE2E tests to run on NOAA Cloud isn't working for the WorkflowTaskTests stage, which tests the ability to run a test using stand-alone scripts (the Azure test has failed and the PC AWS test has been running for close to an hour). We will need to address this, as the nightly builds run this test stage before the skill-score test. On Azure, it looks like it ran for 8 minutes, which could be enough time to reach the make_sfc_climo task. Unfortunately, outside of noting that the stage failed, and how long it took to fail, it isn't clear what task failed and why it failed. Please check on this failure. If you need assistance, please let me know.

[EdwardSnyder-NOAA]

@EdwardSnyder-NOAA -

It looks like the modification that was made to allow the WE2E tests to run on NOAA Cloud isn't working for the WorkflowTaskTests stage, which tests the ability to run a test using stand-alone scripts (the Azure test has failed and the PC AWS test has been running for close to an hour). We will need to address this, as the nightly builds run this test stage before the skill-score test. On Azure, it looks like it ran for 8 minutes, which could be enough time to reach the make_sfc_climo task. Unfortunately, outside of noting that the stage failed, and how long it took to fail, it isn't clear what task failed and why it failed. Please check on this failure. If you need assistance, please let me know.

Thank you @MichaelLueken for reporting this failure. The reason why it is failing is because the missing GNU library isn't being loaded during the WorkflowTaskTests stage, which uses the stand-alone scripts. My change was only in the noaacloud task module file, which doesn't get loaded with the stand-alone scripts. To resolve this failure, I would need to make changes the build noaa cloud module file since that is what is being loaded with the stand-alone scripts. I'll make this change shortly.

[MichaelLueken]

Thanks, @EdwardSnyder-NOAA! I can confirm that the Jenkins tests successfully completed for Derecho, Gaea C6, Hercules, Orion, Ursa GNU, and Ursa Intel. I will rerun the tests on CSPs now.

[MichaelLueken]

The Jenkins tests have successfully passed on Azure:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used 
----------------------------------------------------------------------------------------------------
grid_RRFS_CONUScompact_25km_ics_RRFS_lbcs_RRFS_suite_RRFS_sas_202  COMPLETE              22.63
deactivate_tasks_20260415194415                                    COMPLETE               2.76
specify_template_filenames_20260415194446                          COMPLETE              18.72
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_RAP_20260415194  COMPLETE              40.65
grid_RRFS_AK_3km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR_20260415194635  COMPLETE             309.75
grid_SUBCONUS_Ind_3km_ics_FV3GFS_lbcs_FV3GFS_suite_WoFS_v0_202604  COMPLETE              41.50
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE             436.01

[MichaelLueken]

On PW AWS, three tests passed, while three tests failed:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used 
----------------------------------------------------------------------------------------------------
2019_hurricane_barry_20260415231958                                DEAD                  14.72
community_20260415232019                                           COMPLETE              33.50
grid_RRFS_AK_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_plot_20260  DEAD                  36.96
grid_RRFS_CONUS_25km_ics_GSMGFS_lbcs_GSMGFS_suite_GFS_v15p2_20260  COMPLETE              22.62
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_20260415  DEAD                  59.26
grid_SUBCONUS_Ind_3km_ics_FV3GFS_lbcs_FV3GFS_suite_WoFS_v0_202604  COMPLETE              36.67
----------------------------------------------------------------------------------------------------
Total                                                              DEAD                 203.73

Will rerun the three failed tests to see if they successfully pass.

[MichaelLueken]

@EdwardSnyder-NOAA -

The three tests failed once again in the run_fcst task. I will attempt to run these tests one by one. I'll let you know if the tests continue to fail.

[MichaelLueken]

The 2019_hurricane_barry WE2E test has failed in the run_fcst task once again. The test fails after running for 62 seconds:

04/16/26 14:42:49 UTC :: FV3LAM_wflow.xml :: Cycle 201907120000, Task run_fcst_mem000, jobid=137, in state DEAD (FAILED), ran for 62.0 seconds, exit status=1, try=1 (of 1)

The Jenkins run can be found - https://jenkins.epic.oarcloud.noaa.gov/blue/organizations/jenkins/ufs-srweather-app%2Fpipeline/detail/PR-1372/6/pipeline

[EdwardSnyder-NOAA]

These three experiments are failing in the run_fcst task on AWS due to a variable not being allocated properly (see error message below).

forrtl: severe (408): fort: (8): Attempt to fetch from allocatable variable LAT when it is not allocated

Image              PC                Routine            Line        Source
ufs_model          0000000004F2DC6F  module_ozphys_mp_         165  module_ozphys.F90
ufs_model          000000000AD00B57  gfs_phys_time_var         281  GFS_phys_time_vary.fv3.F90
ufs_model          00000000068A5135  ccpp_fv3_gfs_v16_        2603  ccpp_fv3_gfs_v16_time_vary_cap.F90
ufs_model          0000000005FD0C0C  ccpp_fv3_gfs_v16_         187  ccpp_fv3_gfs_v16_cap.F90
ufs_model          0000000004B63405  ccpp_static_api_m        1068  ccpp_static_api.F90
ufs_model          0000000004B35E6C  ccpp_driver_mp_cc         118  CCPP_driver.F90
ufs_model          0000000001A31B37  atmos_model_mod_m         730  atmos_model.F90
ufs_model          000000000143A017  module_fcst_grid_         836  module_fcst_grid_comp.F90
ufs_model          0000000000A47230  Unknown               Unknown  Unknown

I dug through the code and cannot find the source of this error. What is even more frustrating is that these same experiments pass with my PW AWS instance. The only difference between my PW AWS configuration and Kris's is he using a different lustre filesystem. So I reran one of Kris's failed experiments on contrib (the same filesystem for me) and it still throws the same error. Here are some troubleshooting methods that I tried and still failed:

• increase the OMP_STACKSIZE_RUN_FCST size to 64 GB
• run the fcst task on one node (all three failed experiments needs more then one node to run; the passing experiments used only one node)
• Created a cluster based off of Kris's AWS configuration and was able to run one of the failed experiments successfully

My intuition tells me there is something wrong with Kris's profile or configuration since the experiments passed with my AWS configuration. I'm unsure how to precede, so any guidance or suggestions are welcomed.

[MichaelLueken]

@EdwardSnyder-NOAA -

Thanks for digging into this.

I'd recommend replacing the 2019_hurricane_barry, grid_RRFS_AK_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_plot, and grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16 test configurations from tests/WE2E/machine_suites/coverage.noaacloud.aws file, with two or three tests from the tests/WE2E/machine_suites/coverage.noaacloud.gcp file. The skill-score test, grid_SUBCONUS_Ind_3km_ics_FV3GFS_lbcs_FV3GFS_suite_WoFS_v0, is successfully completing, so the nightly test should successfully run.

[EdwardSnyder-NOAA]

@EdwardSnyder-NOAA -

Thanks for digging into this.

I'd recommend replacing the 2019_hurricane_barry, grid_RRFS_AK_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_plot, and grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16 test configurations from tests/WE2E/machine_suites/coverage.noaacloud.aws file, with two or three tests from the tests/WE2E/machine_suites/coverage.noaacloud.gcp file. The skill-score test, grid_SUBCONUS_Ind_3km_ics_FV3GFS_lbcs_FV3GFS_suite_WoFS_v0, is successfully completing, so the nightly test should successfully run.

@MichaelLueken -

I shuffled around the tests, and confirmed that the new tests for AWS do pass. Feel free to rerun the Jenkins job again.

[MichaelLueken]

Thanks, @EdwardSnyder-NOAA! I can confirm that the updated coverage test suite successfully passes on PW AWS:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used 
----------------------------------------------------------------------------------------------------
community_20260424171335                                           COMPLETE              24.47
grid_RRFS_CONUS_25km_ics_GSMGFS_lbcs_GSMGFS_suite_GFS_v15p2_20260  COMPLETE              26.18
MET_verification_only_vx_20260424171418                            COMPLETE               0.33
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_RAP_20260424171  COMPLETE             196.97
grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_HRRR_suite_HRRR_2026042417150  COMPLETE              36.97
grid_SUBCONUS_Ind_3km_ics_FV3GFS_lbcs_FV3GFS_suite_WoFS_v0_202604  COMPLETE              35.17
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE             320.09

Merging this PR now.