[develop] Add MPI for UFS_FIRE, add a check for incorrectly placed envvars: in config file, remove obsolete FV3_RRFS_v1beta suite - #1376
Merged
MichaelLueken merged 12 commits intoMay 22, 2026
Conversation
…ILE for specifying netCDF input filename (defaults to previous behavior)
…ad. Use FIRE_NUM_TASKS: 2 in multifire test. Remove unnecessary vars from pass_settings (they are under envvars now)
mkavulich
requested review from
EdwardSnyder-NOAA,
JeffBeck-NOAA,
MichaelLueken,
RatkoVasic-NOAA,
WeirAE,
benkozi,
chan-hoo,
christinaholtNOAA,
gsketefian,
gspetro-NOAA,
jdkublnick,
jkbk2004,
natalie-perlin,
panll,
priyarpillai,
rickgrubin-noaa,
selbigmtnwx23 and
ywangwof
as code owners
May 9, 2026 18:53
13 tasks
* parm/FV3.input.yml - Remove FV3_RRFS_v1beta * parm/diag_table.FV3_RRFS_v1beta - Deleted * parm/field_table.FV3_RRFS_v1beta - Deleted * scripts/exregional_make_ics.sh - Remove FV3_RRFS_v1beta * scripts/exregional_make_lbcs.sh - Remove FV3_RRFS_v1beta * sorc/CMakeLists.txt - Remove FV3_RRFS_v1beta * tests/WE2E/machine_suites/comprehensive.derecho - Remove FV3_RRFS_v1beta * tests/WE2E/machine_suites/comprehensive.hera.gnu - Remove FV3_RRFS_v1beta * tests/WE2E/machine_suites/comprehensive.hera.intel - Remove FV3_RRFS_v1beta * tests/WE2E/machine_suites/comprehensive.noaacloud - Remove FV3_RRFS_v1beta * tests/WE2E/machine_suites/comprehensive.orion - Remove FV3_RRFS_v1beta * tests/WE2E/machine_suites/comprehensive.ursa.gnu - Remove FV3_RRFS_v1beta * tests/WE2E/machine_suites/comprehensive.ursa.gnu - Remove FV3_RRFS_v1beta * tests/WE2E/test_configs/grids_extrn_mdls_suites_community/config.grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta.yaml - Deleted * ush/experiment.jsonschema - Remove FV3_RRFS_v1beta * ush/setup.py - Remove FV3_RRFS_v1beta
mkavulich
changed the title
FV3_RRFS_v1beta suite
Collaborator
Author
|
@MichaelLueken This PR now includes the changes from #1375 |
MichaelLueken
requested changes
May 11, 2026
MichaelLueken
left a comment
MichaelLueken
left a comment
Collaborator
There was a problem hiding this comment.
While attempting to launch the comprehensive tests on Gaea C6, the following two WE2E tests failed to generate their workflows:
deactivate_tasks
pregen_grid_orog_sfc_climo
The error message from the logs is as follows:
Traceback (most recent call last):
File "/gpfs/f6/bil-fire8/scratch/Michael.Lueken/ufs-srweather-app/gaeac6/tests/WE2E/./run_we2e_tests.py", line 740, in <module>
run_we2e_tests(srw_dir, user_args)
File "/gpfs/f6/bil-fire8/scratch/Michael.Lueken/ufs-srweather-app/gaeac6/tests/WE2E/./run_we2e_tests.py", line 275, in run_we2e_tests
expt_dir = generate_FV3LAM_wflow(
^^^^^^^^^^^^^^^^^^^^^^
File "/gpfs/f6/bil-fire8/scratch/Michael.Lueken/ufs-srweather-app/gaeac6/tests/WE2E/../../ush/generate_FV3LAM_wflow.py", line 75, in generate_FV3LAM_wflow
expt_config = setup(ushdir, user_config_fn=config, debug=debug)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/gpfs/f6/bil-fire8/scratch/Michael.Lueken/ufs-srweather-app/gaeac6/tests/WE2E/../../ush/setup.py", line 414, in setup
expt_config = load_config_for_setup(ushdir, default_config_fp, user_config_fp)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/gpfs/f6/bil-fire8/scratch/Michael.Lueken/ufs-srweather-app/gaeac6/tests/WE2E/../../ush/setup.py", line 120, in load_config_for_setup
if envvars:=default_config[section].get("envvars"):
~~~~~~~~~~~~~~^^^^^^^^^
File "/gpfs/f6/bil-fire8/scratch/Michael.Lueken/ufs-srweather-app/gaeac6/conda/envs/srw_app/lib/python3.12/collections/__init__.py", line 1135, in __getitem__
raise KeyError(key)
KeyError: 'rocoto'
These two tests add rocoto: sections to deactivate tasks. Looking in the ush/config_defaults.yaml file, there is no rocoto: section, which is likely leading to this failure.
Please ensure that all current WE2E tests (in the comprehensive WE2E suites) run following the updated configuration checks. Thanks!
Collaborator
|
Outside of the two tests that fail to generate experiments, the comprehensive WE2E tests successfully passed on Gaea C6: The UFS Fire WE2E tests successfully passed on Gaea C6: The AQM WE2E test successfully passed on Gaea C6: |
Collaborator
|
Similarly, the comprehensive WE2E tests successfully passed on Ursa (with the exception of the two tests which can't generate experiments): The UFS Fire WE2E tests successfully passed on Ursa: The AQM WE2E test successfully passed on Ursa: |
Collaborator
|
The comprehensive WE2E tests (with the exception of the two tests that won't generate experiments) have successfully passed on Hercules: One of the two UFS Fire WE2E tests failed on Hercules: The Using The AQM WE2E test successfully passed on Hercules: @mkavulich Please double check the |
Collaborator
|
The coverage WE2E tests (with Similar to Hercules, the The The AQM WE2E test successfully passed on Derecho: |
Collaborator
|
Outside of the The UFS Fire WE2E tests successfully passed on Orion: The AQM WE2E has successfully passed on Orion: |
Collaborator
Author
|
@MichaelLueken sorry for the delay, I have pushed a fix for the The fundamental test suite also passed on Hercules. I saw the same failure as you with the |
Collaborator
|
Thanks, @mkavulich! I'll double check the |
Collaborator
|
The On Ursa, the two UFS Fire WE2E tests are successfully passing: As well as Derecho: And Hercules: Approving this PR now. |
MichaelLueken
approved these changes
May 19, 2026
MichaelLueken
left a comment
MichaelLueken
left a comment
Collaborator
There was a problem hiding this comment.
Thank you very much, @mkavulich, for addressing the issues with the WE2E tests! All tests are now passing as expected.
Approving this PR now.
Collaborator
|
Ursa GNU coverage WE2E tests successfully passed: As well as UFS Fire WE2E tests: |
Collaborator
|
The entire comprehensive WE2E tests were ran and all passed on Gaea C6: Both UFS Fire WE2E tests successfully passed on Gaea C6: The AQM WE2E test successfully passed: |
Collaborator
|
I ran tests on Ursa, Hercules and Derecho. Ursa (GNU): Hercules (INTEL): But, when I ran Ursa (GNU) comprehensive test, I got this: Are we running GNU comprehensive tests on Ursa? Any idea on this error? |
Collaborator
|
No, I don't run the comprehensive WE2E for Ursa GNU. That issue has been occurring since uwtools was integrated into the SRW App's configuration layer. I have no idea what is causing it or how to correct it. |
Collaborator
Great, my Derecho test is finished as well. |
RatkoVasic-NOAA
approved these changes
May 21, 2026
MichaelLueken
added
the
run_we2e_coverage_tests
Collaborator
|
The Otherwise, the WE2E coverage tests have successfully completed on Gaea C6, Orion, AWS PW, and Ursa GNU. Awaiting the tests to complete for Azure, Derecho, and Hercules. |
Collaborator
|
The coverage WE2E tests successfully passed on Azure and Derecho. The tests on Hercules timed out and were aborted by Jenkins. Looking at the
Will manually run coverage WE2E tests on Hercules, then merge this PR. |
Collaborator
|
WE2E coverage tests on Hercules have successfully passed: Merging this PR now. |
github-project-automation
Bot
moved this from In Review
to Done
in SRW App's PR Reviewers
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
DESCRIPTION OF CHANGES:
This PR updates to the latest UFS Weather Model hash, which includes the addition of MPI capability for the CFBM fire model (UFS_FIRE). The MPI parallelization is activated by setting
FIRE_NUM_TASKSto a value greater than the default of 1. In addition, a new settingFIRE_GEO_EM_FILEis provided in case the user wants to use an input fire geo_em file with a different name. This supersedes the oldFIRE_INPUT_DIRwhich assumed the filename would begeo_em.d01.nc, though the default values are set so that config files using the old setting will still work as expected.This PR also adds a check for if the user incorrectly places a setting outside of the envvars: section for all config variables. For example, if the user were to set the following fire settings (note that
FIRE_INPUT_DIRis incorrectly not placed in theenvvars:subsection):the experiment generation will fail with the following descriptive message:
This will work on all sections, not just
fire:.And finally, this commit incorporates changes from #1375, removing the obsolete
FV3_RRFS_v1betaCCPP suite.Type of change
TESTS CONDUCTED:
Ran Fire tests on Hera, Derecho, and Hercules. Also ran fundamental tests on Hera. Ran a series of fire tests on Hercules with various MPI and OpenMP settings for the fire model; the results are documented in the users guide. Also attempted to generate experiments with several variations of incorrectly placed config variables; these resulted in the expected errors.
DEPENDENCIES:
None
DOCUMENTATION:
Documentation is updated for new
fire:optionsCHECKLIST