[develop] Add fire-weather test case, LBC forecast-cycle bridging, and a more robust container module-loading approach

[natalie-perlin]

DESCRIPTION OF CHANGES:

1. New WE2E test case: custom_ESGgrid_CalFire_1km - support of fire weather wind prediction capabilities

Adds tests/WE2E/test_configs/custom_grids/config.custom_ESGgrid_CalFire_1km.yaml — a 1 km custom ESGgrid case over the Ventura/Santa Barbara, CA area, initialized from the December 2017 Thomas Fire period (DATE_FIRST_CYCL/LAST_CYCL: 2017121400), using HRRR ICs/LBCs and the FV3_WoFS_v0 physics suite. It doubles as a real-world exercise of item 2: EXTRN_MDL_LBCS_MAX_FCST_HRS: 18 / EXTRN_MDL_LBCS_BRIDGE_INTVL_HRS: 6 forces LBC bridging to kick in for its 24 h forecast, since non-synoptic HRRR cycles only extend to 18 h.

This link shows a figure with the outer model domain (~3 deg x 3 deg) and the inner square indicating the output domain focused on the fire (100 x 100 points)

This link shows an unexpected shift in winds that happened around 9:30am local time and caused firefighter fatalities. SRW model predicts wind shift around that time (after 10am)

A link to the Thomas Fire 2017 incident reporting: https://www.fire.ca.gov/incidents/2017/12/4/thomas-fire

2. LBC forecast-cycle bridging (scripts/exregional_get_extrn_mdl_files.sh, ush/config_defaults.yaml)
   Adds two new task_get_extrn_lbcs options:

EXTRN_MDL_LBCS_MAX_FCST_HRS — the max forecast length a single external-model cycle actually provides (e.g. 48 h for HRRR). Leave unset when one cycle already covers the full run.
EXTRN_MDL_LBCS_BRIDGE_INTVL_HRS — how often (in hours) to check for a fresher on-schedule cycle of the same external model.
When EXTRN_MDL_LBCS_MAX_FCST_HRS < FCST_LEN_HRS, get_extrn_lbcs now retrieves LBCs in EXTRN_MDL_LBCS_BRIDGE_INTVL_HRS-sized chunks. Before each chunk it checks whether a fresher on-schedule cycle is available and switches to it if so (always preferring the freshest guidance); if the fresher cycle isn't ready yet (e.g. real-time operation), it falls back to extending whichever cycle is currently in use, up to that cycle's own EXTRN_MDL_LBCS_MAX_FCST_HRS. This check repeats every interval for the whole run — there's no permanent opt-out once a cycle is missed. Retrieved per-cycle data and its retrieve_data.py summary file live in fcst_cycle_ subdirectories under the LBC staging directory, and get relabeled/symlinked (bridge.f###...) and merged into the task's single downstream summary file, keeping exregional_make_lbcs.sh unaffected.

3. More robust container module loading (devbuild.sh, modulefiles/wflow_container.lua, tests/WE2E/run_we2e_tests.py, ush/config_defaults.yaml)
   Previously, PLATFORM=container builds baked a fixed set of environment variables (PATH, LD_LIBRARY_PATH, ESMFMKFILE, CRTM_FIX, etc.) into a ufs-srw.env file and passed it to the container via singularity/apptainer exec --env-file. This is replaced with loading the same Lmod modules used to build the code (module load ${MODULE_FILE}, e.g. build_container_gnu) inside the container at run time, before handing off to the actual binary — a single source of truth for the build/run environment rather than a separately-maintained env file that can drift out of sync.

Documentation has been updated in support of these changes.

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

TESTS CONDUCTED:

New run has been tested on several platforms:

• derecho.intel
• gaeac6.intel
• hercules.intel
• orion.intel
• ursa.gnu
• ursa.intel
• NOAA Cloud (indicate which platform)
• Jenkins
• fundamental test suite
• comprehensive tests (specify which if a subset was used)
• NOAA Cloud AWS with a GNU container

DEPENDENCIES:

DOCUMENTATION:

ISSUE:

CHECKLIST

• My code follows the style guidelines in the Contributor's Guide
• I have performed a self-review of my own code using the Code Reviewer's Guide
• I have commented my code, particularly in hard-to-understand areas
• My changes need updates to the documentation. I have made corresponding changes to the documentation
• My changes do not require updates to the documentation (explain).
• My changes generate no new warnings
• New and existing tests pass with my changes
• Any dependent changes have been merged and published

LABELS (optional):

A Code Manager needs to add the following labels to this PR:

• Work In Progress
• bug
• enhancement
• documentation
• release
• high priority
• run_ci
• run_we2e_fundamental_tests
• run_we2e_comprehensive_tests
• help wanted

CONTRIBUTORS (optional):

LOGS:

WE2E_summary_AWS_gnu_container_20260822003818.txt
WE2E_summary_ursa_20260824131233.txt
WE2E_summary_gaeac6_20260824094158.txt
WE2E_summary_hercules_20260824092758.txt
WE2E_summary_orion_20260824105903.txt

[MichaelLueken]

@natalie-perlin -

I have completed my initial review of the changes. The documentation changes look good. I will be testing the comprehensive WE2E tests on Ursa to ensure that there are no problems (as well as the new CalFire test). The only question I have is with modulefiles/wflow_container.lua. Line 11 has been commented out. If this line is still required for platforms that use Singularity, then the modification is fine. If it is no longer necessary at all, it would be best to remove the line entirely.

[MichaelLueken]

The new custom_ESGgrid_CalFire_1km WE2E test successfully passed on Gaea C6:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used 
----------------------------------------------------------------------------------------------------
custom_ESGgrid_CalFire_1km_20260831101035                          COMPLETE             276.42
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE             276.42

The coverage WE2E tests successfully passed on Gaea C6:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used 
----------------------------------------------------------------------------------------------------
custom_ESGgrid_NewZealand_3km_20260831105642                       COMPLETE              72.21
grid_RRFS_CONUScompact_13km_ics_HRRR_lbcs_RAP_suite_RRFS_sas_2026  COMPLETE              37.34
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR_2026083110  COMPLETE              39.21
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15_thompson  COMPLETE             434.30
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_HRRR_suite_HRRR_2026083  COMPLETE              47.24
smoke_dust_grid_RRFS_CONUS_3km_suite_HRRR_gf_20260831105754        COMPLETE            1249.25
2020_CAPE_20260831105812                                           COMPLETE              46.57
2020_easter_storm_20260831105830                                   COMPLETE              42.04
grid_SUBCONUS_Ind_3km_ics_FV3GFS_lbcs_FV3GFS_suite_WoFS_v0_202608  COMPLETE              41.38
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE            2009.54

Skill Score: 0.99617
+ [[ 0.99617 < 0.700 ]]
Congrats! You pass check!

[MichaelLueken]

While testing on Ursa, I realized that the latest HEAD of develop wasn't merged to the feature/lbcs_fcst_cycles branch.

The new custom_ESGgrid_CalFire_1km WE2E test was once again tested on Gaea C6 and successfully passed:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used 
----------------------------------------------------------------------------------------------------
custom_ESGgrid_CalFire_1km_20260901090715                          COMPLETE             275.98
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE             275.98

The coverage WE2E on Gaea C6 was run and all successfully passed:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used 
----------------------------------------------------------------------------------------------------
custom_ESGgrid_NewZealand_3km_20260901095141                       COMPLETE              75.77
grid_RRFS_CONUScompact_13km_ics_HRRR_lbcs_RAP_suite_RRFS_sas_2026  COMPLETE              40.15
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR_2026090109  COMPLETE              43.19
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15_thompson  COMPLETE             447.07
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_HRRR_suite_HRRR_2026090  COMPLETE              47.96
smoke_dust_grid_RRFS_CONUS_3km_suite_HRRR_gf_20260901095252        COMPLETE            1249.39
2020_CAPE_20260901095310                                           COMPLETE              41.20
2020_easter_storm_20260901095328                                   COMPLETE              41.84
grid_SUBCONUS_Ind_3km_ics_FV3GFS_lbcs_FV3GFS_suite_WoFS_v0_202609  COMPLETE              41.45
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE            2028.02

Skill Score: 0.99617
+ [[ 0.99617 < 0.700 ]]
Congrats! You pass check!

The UFS Fire WE2E tests have successfully passed on Gaea C6:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used 
----------------------------------------------------------------------------------------------------
UFS_FIRE_multifire_two-way-coupled_20260901105437                  COMPLETE              16.12
UFS_FIRE_one-way-coupled_20260901105448                            COMPLETE              26.08
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE              42.20

The AQM WE2E test has successfully passed on Gaea C6:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used 
----------------------------------------------------------------------------------------------------
aqm_grid_AQM_NA13km_suite_GFS_v16_20260901111857                   COMPLETE            3235.46
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE            3235.46

With this, testing is complete. I will now approve this PR.

[MichaelLueken]

@natalie-perlin -

Thanks for letting me know about the intent of line 11 in modulefiles/wflow_container.lua! My tests have successfully completed, so I will now approve this PR.

[EdwardSnyder-NOAA]

Tested the new fire experiment on PW AWS and it passed:

Experiment name                                                  | Status    | Core hours used
----------------------------------------------------------------------------------------------------
custom_ESGgrid_CalFire_1km_20260901153126                          COMPLETE             198.70
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE             198.70

Also, tested the fundamental WE2E testing suite there as well:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used
----------------------------------------------------------------------------------------------------
grid_RRFS_CONUScompact_25km_ics_RRFS_lbcs_RRFS_suite_RRFS_sas_202  COMPLETE              38.86
custom_ESGgrid_Central_Asia_3km_20260901191709                     COMPLETE              78.05
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v17_p8_plot  COMPLETE              66.99
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_HRRR_suite_HRRR_2026090  COMPLETE             105.27
grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_RAP_suite_WoFS_v0_20260901191  COMPLETE              39.54
grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_GFS_v16_2026090119182  COMPLETE              64.55
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE             393.26

[natalie-perlin]

@MichaelLueken - could you please hold off on merging this PR?
I'd like to do some fine-tuning of the Documentation related to containers today, doing it at the moment - it will be ready within 30min-1h!

[MichaelLueken]

@MichaelLueken - could you please hold off on merging this PR? I'd like to do some fine-tuning of the Documentation related to containers today, doing it at the moment - it will be ready within 30min-1h!

@natalie-perlin - Final testing still needs to be completed before this PR will be merged. Once Derecho returns from maintenance, I will launch the tests. You have time to fine-tune the container documentation. To be on the safe side, please let me know once you are complete. Thank you very much!

[natalie-perlin]

A note regarding the following change in ContainerQuickstart.rst:

singularity exec rocky9-oneapi2024.2-ss192 cp /opt/*.sh .

had to be changed back to

singularity exec rocky9-oneapi2024.2-ss192 cp /opt/intel-sandbox.sh . 
singularity exec rocky9-oneapi2024.2-ss192 cp /opt/compilers_cp.sh .

The reason:
Replaced the /opt/*.sh wildcard with explicit file names because shell globbing is performed by the host shell before singularity exec/apptainer exec enters the container. As a result, the wildcard cannot match files that exist only inside the running container instance.