[develop] Updated option to use PLATFORM=container instead of singularity - #1373
Conversation
… feature/platform-container
|
While attempting to build the GNU container on Ursa, I encounter the following: All of the executables are in the |
Michael, thank you for the comments above. It looks like the errors from the build step, after shelling into the container. When you shelled into the container. It looks like something to do with naming of the "--bin-dir" specified in the devbuild.sh line. What was the command line you used to build the SRW App? I may need to include more cases for naming conventions... As to the I have retested the build that had the following: And the final step of the build was as follows: |
|
@MichaelLueken - minding your last comment, I adapted the |
|
Thanks for checking the The fundamental WE2E tests were run on Gaea C6 and all tests successfully passed: The UFS Fire WE2E tests successfully passed on Gaea C6: The AQM WE2E test successfully passed on Gaea C6: |
There was a problem hiding this comment.
I was able to get the executables to successfully copy from the bin/ directory to the exec/ directory. The container experiment successfully passed on Ursa. I ran the fundamental, UFS Fire, and AQM WE2E tests on Gaea C6.
The path to the machine files should be:
ush/machine/*.yaml
I have marked the two locations where machines is used rather than machine. Once these changes have been made, I will approve these changes.
Thank you, Michael, for finding this elusive typo! Corrected it, rebuilt the html docs successfully, and pushed the updates to the PR. |
MichaelLueken
left a comment
There was a problem hiding this comment.
Thanks, @natalie-perlin!
Approving PR now.
|
Several verification WE2E tests have failed on Ursa GNU and Ursa Intel due to issues encountered with the service nodes while attempting to retrieve data from HPSS. I will be resubmitting the failed jobs and will add the logs to the PR once they successfully complete. |
|
Two tests failed while running the WE2E coverage tests on Azure: Resubmitting |
|
Jenkins tests have successfully completed on Derecho, Gaea C6, Hercules, Orion, and PW AWS. |
|
The two Azure WE2E coverage tests that previously failed have passed during the retest: Thus, Jenkins testing has successfully completed for Azure. |
|
The WE2E coverage tests have successfully passed for Ursa GNU: |
|
The WE2E coverage tests have successfully completed on Ursa Intel: All WE2E coverage tests have successfully completed. Moving forward with merging this PR now. |
DESCRIPTION OF CHANGES:
INSTRUCTIONS FOR TESTING
The containerized SRW App community test has been successfully run with both GNU and Intel containers on the following platforms: Ursa, Gaea-c6, Hercules, Orion, NOAA Cloud AWS (Rocky9), and NOAA Cloud Azure (Rocky9).
1. Locate a singularity/apptainer container in a shared location (a) or build a GNU container from a Docker repository (b).
(a) Use existing containers in a shared location. Common locations are as follows:
(*) NOAA Cloud AWS and Azure - only using EPIC resources via ParallelWorks, and when the storage is attached to the AWS or Azure cluster used for testing.
For GNU container use the following:
export IMG=<full-container-path>/rocky9-gcc13-ss192-ompi416.sifFor Intel container use the following:
export IMG=<full-container-path>/rocky9-oneapi2024.2-ss192.sif(b)
Build a GNU singularity (*.sif) container image from a Docker Hub repository. It could be done interactively on a service partition on Ursa, or on login nodes on other systems. A singularity or apptainer module may need to be loaded.
Use singularity or apptainer command depending on the module loaded (apptainer given as an example):
2. Checkout the SRW feature/platform-container branch and sub-modules:
3. Shell into the container to build the SRW Application.
Use
singularityorapptainerdepending on availability on the platform of your choice, as outlined for the Tier 1 platforms below:Ursa: apptainer (no need to load modules)
Gaea-c6: apptainer (no need to load modules)
Hersules: singularity (module load singularity)
Orion: singularity (module load singularity)
noaacloud: singularity (no need to load modules)
Bind-mount host directories into a container, listing each directory with a preceding "-B" flag. The following directories need to be bind-mounted on each Tier 1 platform:
Ursa: /scratch3, /scratch4
Gaea-c6: /gpfs, /ncrc/home2
Hercules/Orion: /work, /work2, /local
NOAA Cloud AWS, Azure (ParallelWorks): /contrib, /lustre(*)
(*) if staging the run or containers locally on /lustre
When shelled into the container, terminal prompt Apptainer> or Singularity> appears.
For example, shelling into a container on Gaea-c6 and building the SRW App for the
gnucontainer looks as follows:Use
singularityorapptainerdepending on availability, andgnuorinteldepending on the container used.Conda environments will be installed, followed by building of the SRW App. A wrapper script (srw.sh)and
a run-time environment file (ufs-srw.env) are prepared and written to the main ./ufs-srweather-app during the build. When all has been finished, type
exitto exit from the interactive container shell4. Prepare a workflow module file, all located in
./modulefilesdirectory.There is a wflow_container.lua file that you may want to copy and keep for reference.
Examples below assume standard platform names are used instead of :
ursa, gaeac6, orion, hercules, noaacloud
On Tier 1 systems,
use and copy a wflow_.lua* module file for the corresponding platform:
cp wflow_<platform>.lua wflow_container.luaLoad the singularity or apptainer module on platforms that need it to use the software. For example, add loading the to wflow_hercules.lua and wflow_orion.lua:
load("singularity")5. Adapt configuration files for the platform and for the community test.
a Prepare a configuration file:
Make the following changes in ./ush/config.yaml:
ACCOUNT: epic(set account/project name)COMPILER: gnu(set compiler to whether gnu or intel)USE_CRON_TO_RELAUNCH: false(set to true if use of cron is allowed on the machine)EXPT_SUBDIR: test_community(change the experiment name directory if needed)b Prepare a machine file:
You could modify the
NCORES_PER_NODEin ./ush/machine/container.yaml if needed.Make the following changes to container.yaml.
a Replace the values of following variables:
Adapt the variable
SCHED_NATIVE_CMDas follows:On Gaea-c6:
SCHED_NATIVE_CMD: --clusters=c6.Remove the line with this variable from the container.yaml for Hercules, Orion, Ursa, NOAA Cloud.
Note: AQM, NEXUS have not been tested with the container
6. Generate a workflow and launch the workflow:
Export EXPTDIR as suggested in the prompt and start the test, e.g.:
Resubmit periodically and recheck the progress with rocoto:
Type of change
TESTS CONDUCTED:
A community test has been run on all Tier 1 Platforms and has successfully completed on all except Derecho.
Derecho fails in
make_ics/make_lbcsstages.More container integration work is needed for Derecho and Ursa to utilize fast interconnect/hpc-x.
LOGS
logs from rocotostat after test completion:
rocotostat_test_community_ursa_gnu.txt
rocotostat_test_community_ursa_intel.txt
rocotostat_test_community_gaeac6_gnu_2.txt
rocotostat_test_community_gaeac6_intel_2.txt
rocotostat_test_community_hercules_gnu_2.txt
rocotostat_test_community_hercules_intel_2.txt
rocotostat_test_community_orion_gnu_2.txt
rocotostat_test_community_orion_intel_2.txt
rocotostat_test_community_noaacloud_aws_rocky9_gnu.txt
rocotostat_test_community_noaacloud_aws_rocky9_intel.txt
rocotostat_test_community_noaacloud_aws_rocky8_gnu.txt
rocotostat_test_community_noaacloud_aws_rocky8_intel.txt
rocotostat_test_community_noaacloud_azure_gnu.txt
rocotostat_test_community_noaacloud_azure_intel.txt
DEPENDENCIES:
DOCUMENTATION:
Documentation is being prepared, and will be included with this PR asap.
ISSUE:
NB: A PR in progress to the NOAA-RDHPCS Documentation (https://docs.rdhpcs.noaa.gov/), which uses a link to a Chapter on Containers: https://docs.rdhpcs.noaa.gov/software/containers/ . The page updated when the PR is merged: NOAA-RDHPCS/noaa-rdhpcs.github.io#741
So this PR is not a dependency, but rather informational supplement to the SRW documentation that contains a link.
CHECKLIST
LABELS (optional):
A Code Manager needs to add the following labels to this PR:
CONTRIBUTORS (optional):