Container and community platform support for UFS WM build-and-test workflow - #3268
Container and community platform support for UFS WM build-and-test workflow#3268natalie-perlin wants to merge 30 commits into
Conversation
|
This is so exciting. Thanks @natalie-perlin Can you explain a little more the intent/distinction of the On this, I hesitate to diverge in testing scripts/infrastructure. |
|
I see that this is marked as high priority. For the build and run failures, please start issues to follow up |
| test cases. Users will likely need to modify the testing configuration for their own | ||
| computing platform, and are encouraged to tailor it further to suit their own modeling | ||
| goals beyond running pre-defined tests. | ||
|
|
There was a problem hiding this comment.
@natalie-perlin I would move this note to the bottom of the "Supported Platforms & Compilers" section (i.e., after current line 28), since it relates to where/how users choose to build & run the model. I'd also delete the info about the RT framework because it is already in the container chapter and is more relevant there. Better to keep the note short and to-the-point.
| .. note:: | |
| For instructions on using software containers to build and run the UFS Weather Model | |
| on any community platform, see :numref:`Chapter %s <container-rt-tests>`. | |
| The container workflow provides a portable, reproducible software environment and | |
| does not require a site-specific software stack. |
files: tests/fv3_conf/fv3_qsub.IN_container, tests/fv3_conf/fv3_slurm.IN_container, tests/fv3_conf/compile_qsub.IN_container, tests/fv3_conf/compile_slurm.IN_container, tests/module-setup.sh
@DeniseWorthen - I see two separate issues here:
At this stage, there is no viable community-oriented option for running the UFS Weather Model on compute systems outside the NOAA-supported platforms. This PR addresses item 1 first. From a developer perspective, I still rely on NOAA platforms for continuous UFS Weather Model development and improvement. However, having access to a broader range of platforms provides a useful way to treat those systems as proxies for possible community platforms. It also allows us to model a community-oriented workflow more realistically. The proposed workflow supports running an RT-type set of configurations and tests either with the software container option or on a custom community platform, assuming the required software stack has already been built and installed natively. In the recent updates to this PR, the scripts and configuration files were renamed to community.sh and community.conf to avoid confusion with the existing rt.sh workflow and to preserve consistency for the regularly run RTs on NOAA Tier 1 platforms. For item 2, transitioning Tier 1 machines to containers may still be a viable option, but it requires additional testing first so that containers can be fairly compared with the current natively installed stack options. This PR provides a more transparent workflow for performing such comparisons on selected platforms before any container-based approach is incorporated into the more rigorous rt.sh workflow. The updated community.sh script can now be run with the -p flag. In that mode, the workflow runs without a container, using the natively installed software stack on the platform specified in the community.conf configuration file. This makes it easy to switch between the container option and the native-stack platform option on the same machine, as tested on Ursa, by changing the MACHINE_ID and likely the RUNDIR_ROOT variables in community.conf. For the container-based workflow, the submission would be: For the native-stack workflow, after updating MACHINE_ID and RUNDIR_ROOT, the submission would be: My suggestion is to allow this hybrid workflow option first. Adapting rt.sh to support container-based full-RT runs on NOAA Tier 1 platforms should be handled in a separate PR focused specifically on rt.sh changes (will do next). It may be helpful to have a working version of the container-based community workflow in place first, since that experience can inform how container support is later integrated into rt.sh. |
|
NOTE: A description of the PR will be updated ASAP, following the changes of a workflow script and a configuration file renamed from "rt_container.sh" and "rt_container.conf" to "community.sh" and "community.conf", correspondingly. Documentation has been updated in the repository with these changes. |
|
The names community.sh and community.conf seem too vague. Also, I wouldn't name a file Have you had anyone from the broader community test this setup? One problem I see is there is no option to create a baseline, so that's one place folks would get stuck on their platforms |
@NickSzapiro-NOAA - thanks for the comments! The containers and the approach of launching containers have been tested and shared with EMC representatives and other UFS WM users and EPIC/developers since the beginning of the year. While it was not the same set of scripts shared, the key commands for launching containers in the compile and run phases remained the same. The scripts are constantly being adapted to 1) feedback from different developers, 2) to simplify the changes needed across platforms, and 3) retain resemblance to the RT workflow. There is intentionally no goal or option to create any baselines, since this appears to be primarily needed by code developers and is generally not the purpose of running UFS WM tests for the community. The current After community users verify that the pre-configured tests run successfully, they can more confidently use the code to suit their modeling needs. I have run a different set of tests, with different runscript flags, on all the NOAA Tier 1 platforms, to varying degrees. Since the tests require significant compute resources, it did not seem practical to rerun all tests on all platforms after each workflow change, following feedback or improvements; some were run only on selected platforms, selected tests or inline options. I would gladly rerun a suggested set of configurations, tests, workflow/script options, platforms, and include log files from running or other suggested names in place of the current ones. Regular log files from running rt.sh could also be included, but it possibly make more sense to do it at a later step, since no changes to rt.sh/rt.conf were involved. Please feel free to comment on any of the issues!. |
|
The Description of this PR has been updated. |
|
I'm struggling to understand the real purpose of this PR. The PR title and corresponding issue suggest the main goal is to allow the community (people without access to any Tier-1 platform) to run the ufs-weather-model tests on any computer system that supports Apptainer. Fine. That's a useful feature. Note the goal is to run tests (specifically the model configurations available in the current regression test system). Not just any arbitrary model configuration but one of the pre-configured tests. Then why do we need a new script to do that? We already have a script that runs pre-configured tests. It called rt.sh. I do not see what the new script (community.sh) can do that rt.sh cannot do. Of course rt.sh would need to be configured to support the container environment by setting several variables in rt.sh and/or default.sh. PR description says: "This PR introduces a complete, portable community-friendly workflow for building and running the UFS Weather Model (WM)" What makes What makes "A standalone driver script (tests/community.sh) and companion configuration file (tests/comminity.conf) compile the UFS WM inside the container and run selected RT cases sequentially, without requiring Rocoto or ECFlow installed on a community platform." The driver script (community.sh) is not standalone. It requires the regression test system to do anything useful, such as compiling the model, configuring the job card, preparing configuration files, getting the correct input data from a holding area, submitting a job card to a scheduler, monitoring the job's execution, and determining if model execution was successful. etc. All of this functionality is provided by the rt system. It (community.sh) is just another driver script that parses user options and configuration file listing the tests it will execute, and that's all. Just like the rt.sh. It's a driver script, a slightly different driver script, using a .conf file with slightly different syntax. In my opinion, the above description is misleading. It sounds like if users just have My suggestion is to consolidate the features that the new |
|
@DusanJovic-NOAA - thank you for the comments. The title of the PR may need to be adapted from its original version. The suggestions:
Yes, I agree that describing community.sh as a “standalone script” is not fully precise, since it relies on other scripts to build and run the RTs, including run_compile.sh, compile.sh, run_test.sh, rt_utils.sh, module-setup.sh, and default_vars.sh. The main challenge, as I see it from a community perspective, is that rt.sh and rt.conf contain substantial amounts of hard-coded Tier 1 platform configurations and other options tailored specifically to those platforms. To make the workflow more accessible to the broader community, a more transparent/easily configurable input file with community-specific configuration variables, separate from rt.sh and rt.conf, appeared to me as a reasonable alternative. That was the purpose of the One possible option would be to allow Are there any suggestions for an acceptable input file extension? A YAML file might be the easiest format from a readability perspective, but it would require a specific python/conda environment to parse it, rather than allowing the workflow to run directly from a simple Bash environment. |
@jkbk2004 @ulmononian @EdwardSnyder-NOAA Can any of you speak to this? I had heard talk of moving to container-based RT testing, but I haven't heard much lately, and I believe it's a completely separate issue/effort from what Natalie is doing here. |
|
@natalie-perlin $ cat container.def
MACHINE_ID=container
INPUTDATA_ROOT=/path/to/input_data
TPN=128
SCHEDULER=
MPI_LAUNCH=mpiexec
RT_COMPILER=gnu
PARTITION=my_partition
QUEUE=debug
....In rt.sh add a new option, ./rt.sh -p container.def -l rt_container.conf -a <account> -w ....Then in rt.sh if it does not recognize the current machine and if the user specified the source ${PLATFORM_DEF}After this there should be no changes to the logic of what rt does, the container is just another machine with all variables defined that are needed to successfully compile and execute the tests. If we want users in the container to execute just a subset of all tests currently listed in rt.conf, there are two options, either crate |
@gspetro-NOAA - There was an effort to move to containerizing spack-stack builds on our supported Tier-1 machines but this effort has been stopped. You're correct in saying that the work here is different from the containerizing spack-stack effort, which is to provide an option for users to run the UFS WM on non-Tier 1 machines using a spack-stack container. |
natalie-perlin
changed the title
|
@natalie-perlin Just wanted to check in on this PR. I was under the impression you were doing some work to implement Dusan's suggestions, but I could be mistaken. |
on Container and Community workflow using community.sh
…s and container/community workflows
|
Updated the branch with the latest changes from develop. Also revised the documentation in the following sections:
The main idea reflected in these revisions, based on the discussion in the comments above, is: " community.sh is intended for portability testing on a contributor’s own platform. It gives the UFS-WMcommunity members and users outside the core UFS WM development team a simple starting point for building and running the model. This platform may be a Tier 1 system, another HPC center, a cloud instance, or a laptop/workstation, with or without container software. Running the tests successfully with community.sh confirms that the model has been ported, built, and run successfully, rather than reproducing the full baseline-comparison capability maintained on Tier 1... " Regarding the idea of treating a container as a regular machine from the perspective of rt.sh: this approach may have been considered during the initial testing, when the code was built / run by interactively shelling into a container on a single node. However, it is not further implemented in rt.sh. The main reasons are:
Treating the container itself as a separate platform, however, would still require specifying many characteristics of the underlying host system. This exact rationale underlies the Therefore, adding a container option to rt.sh would not address the primary community-use case of this PR. |
|
Hi @natalie-perlin . Thanks for these contributions and apologies for the extended review I applaud bridging "a simple starting point for building and running the model" and "test that the model has been ported, built, and run successfully". rt.sh must support the latter on Tier 1 systems and is open to further container options/platforms. To test whether a run is successful, the capability here relies on reproducing baselines generated via rt.sh. It seems that future versions would also need capabilities to create baselines and run tests in parallel. |
Removed a line left from merging/resolving conflicts
|
Thank you for the comments. I think an important distinction in this discussion is between regression validation and community portability validation. The purpose of the current RT/rt.sh framework is to determine whether code changes preserve expected UFS Weather Model results under controlled configurations, using maintained baselines and supported NOAA Tier-1 platforms. For pull requests, that is exactly what is needed: a selected set of regression tests is run across the supported Tier-1 systems and compared against authoritative baselines. The community use case addressed by this PR is different. A contributor working on another HPC system, cloud platform, workstation, or other non-Tier-1 environment primarily needs to answer a different question: Can UFS WM be configured, built, and successfully run on this platform? Selected RT configurations are useful for this purpose because they already provide known, well-defined model configurations and existing build/run infrastructure. However, reusing these test cases does not necessarily mean that the contributor is performing a formal regression test. On a community platform, a successful test may mean that the model builds, runs to completion, and, where appropriate, reproduces the same result on repeated runs on that platform. Maintaining or generating authoritative Tier-1 baselines is a different requirement. In other words:
I agree that container support in rt.sh can be valuable, particularly for running the official regression-test workflow in containers on supported Tier-1 systems. I also agree that a community workflow should not grow into a second, duplicate regression-testing framework. Baseline generation and comparison, PR qualification, full RT-suite orchestration, and other regression-framework functionality should remain with rt.sh. However, adding container support to This distinction also seems particularly important given that the repository is hosted under the ufs-community organization. One of the broader objectives of UFS is to enable participation and contributions from the modeling community beyond the institutions operating the Tier-1 systems. A practical path for contributors to build and run UFS WM on their own platforms is therefore not simply an alternative way of running NOAA's regression infrastructure; it is part of lowering the barrier to participating in UFS development. From that perspective, I think it would be useful to keep/consider two related but distinct capabilities:
The key architectural question is therefore not necessarily whether both workflows can technically be implemented through rt.sh, but whether external contributors should need to understand and conform to the Tier-1 regression framework simply to determine whether UFS WM builds and runs on their platform. I would also suggest avoiding the wording that community users are necessarily “running RTs.” A more precise description may be that they are running selected UFS reference/test cases derived from the RT suite to verify a successful port. That keeps the useful common infrastructure while preserving the distinction between a runnable test case and the formal regression-validation process. |
|
Thanks @natalie-perlin . I think we can find an alignment that benefits everyone. The instructions I see in the PR description are largely for Tier 1 platforms, mostly Hercules. Not all problems have to be addressed in a PR. |
|
A note about this corrections in the docs: was changed to: Replaced the wildcard with explicit file names because host-side shell globbing cannot resolve files that exist only inside the container. Shell globbing is performed by the host shell before singularity |
6 participants
Commit Queue Requirements:
test_changes.listindicates which tests, if any, are changed by this PR. Committest_changes.list, even if it is empty.(?) The code has been reviewed, edited, and validated by NWS staff.
UPDATED DESCRIPTION (1 July 2026)
Description:
This PR introduces a complete, portable community-friendly build-and-test workflow for the UFS Weather Model (WM) on any computing platform with either Singularity/Apptainer container software or natively installed software stack (use flag "-p" with the
community.shscript). In a container-based workflow, all prerequisite software libraries are included in a pre-built container image, eliminating the need for a site-specific spack-stack installation. For the natively installed software stack libraries, use the "-p" version with thecommunity.shscript, which assumes that the required software stack has already been built on the tested system. See Documentation: Chapter 4 for the required adaptations of the modulefiles and job scheduler scripts.The Regression Test (RT) framework is adopted as a convenient starting point, following the example of
rt.shto help test it for users familiar with the standard RT workflow. A driver script (tests/community.sh) and companion configuration file (tests/comminity.conf) compile the UFS WM inside the container and run selected RT cases sequentially, without requiring Rocoto or ECFlow installed on a community platform. After verifying that a number of standard configurations and tests run successfully on a tested platform, users may adapt the configuration to suit their modeling needs.Testing with the pre-staged compilers with spack-stack-1.9.2 has been done on all the NOAA Tier 1 platforms with the following containers (intel-based or/and GNU-based), including 25 built UFS WM configurations and ~51 tests:
Ursa (intel, GNU)
Gaea-c6 (Intel, GNU)
Hercules (Intel, GNU)
Orion (Intel, GNU)
AWS (Intel, GNU)
Azure (Intel, GNU)
Derecho (GNU)
List of configurations built and tests run is given in
Intel-based containers are usually all pass, except for some runtime timeout (e.g., on AWS or Azure or for GNU container on some platforms, e.g. for the test cpld_control_gfsv17).
GNU-based containers fail to build UFS-WM for the following configurations:
atmaero
datm_cdeps_land
hafsw
hafs_mom6w
and failing for the following tests for which the fv3.exe were built successfully:
cpld_debug_gefs_gnu
cpld_dcp_gefs_gnu
cpld_control_gefs_gnu
Testing Instructions for Hercules
The following steps describe how to test this PR on Hercules (NOAA RDHPC). Hercules requires minimal changes to the sample
community.confbecause the container image and RT input data are already pre-staged by the EPIC team.1. Check out the code
Before the PR is merged, use the feature branch from the author's fork:
2. Adapt the required host-side runtime modulefile
On Hercules, only the
singularitymodule needs to be loaded, adapt the modulefiles/ufs_container.runtime.lua:3. Edit
tests/community.conf- an example for Hercules/MSU machineOpen
tests/community.conf.For a container-based workflow,
the only required change for Hercules is updating
RUNDIR_ROOT(header line 3) to a path writable by your own user account(e.g., on
/work2or/work). If you are not in theepicproject group,also update
ACCNR(header line 2).Minimal edit example (replace
<your-username>with your actual Hercules username):To test with the GNU compiler and image instead of Intel, change header line 1 to:
For a native-stack forkflow, replace "container" in the in the Header line 1 by "hercules". Update the RUNDIR_ROOT and/or ACCNR as mentioned earlier.
4. Build configurations and run tests
A container-based workflow is a default option. Add the "-p" flag when using the native-stack workflow.
From the repository root:
To run a single test case (for a quick check):
To compile only (skip test execution):
5. Check results
Progress is printed to the terminal as each job completes. Full Slurm output
(stdout/stderr) is written under
${RUNDIR_ROOT}/logs/. A PASS/FAIL summaryis printed at the end; the script exits with status
0on success and1if any compile or test failed.
A convenience symlink
tests/run_diris created automatically and points toRUNDIR_ROOT, making it easy to navigate to the run directory:Commit Message:
Priority:
Git Tracking
UFSWM:
Sub component Pull Requests:
UFSWM Blocking Dependencies:
Documentation:
Changes
Regression Test Changes (Please commit test_changes.list):
Input data Changes:
Library Changes/Upgrades:
Testing Log:
Testing Remarks:
Some logs included from running the
./rt_container.shscript:Ursa Intel container: log.tests.001.txt
Ursa GNU container (less configs and tests than for Intel):
log.tests.001.txt (compile jobs and test jobs)
log.tests.002.txt (no compile jobs, just running tests)
NOAA AWS Intel container: log.tests.001.txt
NOAA AWS GNU container: log.tests.all.txt
NOAA Azure Intel container: log.tests.all.txt - 5 tests did not complete due to time limit/timeout.
A test successfully completed after increasing runtime limit: log.test1.txt