Spack-stack and UFS-SRW Installation on non-NOAA HPC Systems

[cigdemakan]

Hello All,

We have been trying to build and install spack-stack and UFS-SRW app on our HPC system following the steps on the user guide's page. So far we haven't been able to install either of them. I also tried the container-based version of UFS-SRW, but that did not work as well. So, I was wondering if anyone here has successfully installed spack-stack and the UFS-SRW app on a non-NOAA HPC system. If so, would you mind sharing your experience with us?

[DeniseWorthen]

tagging @AlexanderRichert-NOAA.

[AlexanderRichert-NOAA]

@natalie-perlin @RatkoVasic-NOAA @rickgrubin-tomorrow would you be able to advise here in terms of spack-stack installation and container issue(s)?

[rickgrubin-noaa]

The issue description is vague in that there's nothing specific or concrete with respect to "haven't been able to install" something.

I'll only reference spack-stack here.

@cigdemakan, the following documentation is helpful:

• Generating new site configs
• Manual software installations (primarily for git-lfs; if you can install via dnf / yum / etc., that's easier)
  • see also Generating new site configs
• you'll also want to have some sort of environment module software: tcl or Lmod

Generally speaking, when creating a set of configuration files for a new host, I'll start with an existing host (see configs/sites/tier{1, 2}) and make a copy of provided config files that most closely resembles the host on which I want to build spack-stack. This will require walking through the site config's yaml files and changing settings for compilers, MPI, system-provided tools (e.g. binutils, etc.) to match the new host's settings.

Afer that, it's trial and error with Create local environment to get a concretized environment and stack installation. Once you have a configuration that concretizes and installs, you Load the spack-stack environment and then build your application(s) against that stack.

This process will require some patience as it will be repetitive; I would suggest that issues solely relating to spack-stack be created in JCSDA/spack-stack.

[cigdemakan]

Thank you very much for all the information and sorry for not being very specific. Here is a specific question, container-based UFS-SRW app:
I read the documentation and also watched the tutorial on epic.noaa.gov. Based on the documentation page, the only prerequisites are intel compiler, mpi and singularity which we have on our cluster. I was able to download the img file and build the container. The problem starts in the next step, which is staging the container with the following line:
./stage-srw.sh -c=intel/2022.1.2 -m=impi/2022.1.2 -p=hera -i=$img
In this example and in the tutorial, this line works because this is a NOAA system where everything is already set up. If you try to run this on a system not listed on the documentation page, you will get an error saying "module file not found". There is no mention of this on the documentation page. So, I eventually figured out that I can add a YAML file and a module file for the system I am trying to set up under ubuntu22.04-intel-ue-1.6.0-srw-dev/opt/ufs-srweather-app in the img file. I looked at the existing files for other systems and created my own module files. The next issue I noticed is the need for the spack-stack modules. All the existing module files load srw_common, which basically loads all the necessary libraries that come with spack-stack. So, one does need spack-stack installed even for the container based version. Again, there is no explanation of this in the documentation. That's when I realized there is a spack-stack installation under the img. Right now, I will try to load the libraries using that, but I am not sure if it will work. My point is overall, the documentation is insufficient for non-NOAA system installers, and trial and error is not a good way to figure out things in my opinion. That's why I am again asking if someone has an example of running the container-based version on a non-NOAA HPC system from scratch, like what I am trying to do. That would help a lot.

[gspetro-NOAA]

@cigdemakan There are several issues here. Firstly, the SRW App container includes spack-stack in it, last I knew, so there should be no need to build it separately. @EdwardSnyder-NOAA can perhaps confirm that. The modules are there; there's just some kind of disconnect in finding them, unless something has significantly changed with the containers recently. Secondly, the documentation for building the SRW App in a container is here. It includes instructions for what to do on Level 2-4 systems (i.e., non-NOAA systems). For example, you'll likely need to convert the container to a writable sandbox, and make sure that the topmost directory on your system is also added to the container. There may be additional changes required. As you said, you'd probably need to create a wflow_<platform> file to load the proper modules. (Most people do this by adapting an existing file with a similar configuration to their system.)
@jdkublnick , I believe in the usual building/running section, we used to mention that users on level 2-4 systems need to create their own workflow yamls and such on generic Linux/Mac platforms. I presume we used to link to that info from the container chapter, but regardless, we definitely don't now, so the container chapter will probably need an update to state that. I'd confer with Eddie first tho.
@cigdemakan I have not personally built the SRW App via container on a non-NOAA system recently, but there are teams who have succeeded in getting it to run on AWS (e.g., @natalie-perlin @EdwardSnyder-NOAA and @kbooker79 might know more about when/which version/what tweaks). I know @benjamin-cash has had some success with containers on Frontera, though probably not with the SRW App specifically, since his research is more global. Feel free to contribute documentation suggestions/tweaks back to the repo once you get things working. We primarily work on NOAA systems and cannot account for every permutation of possible systems and tweaks, but hopefully one of the folks I mentioned can get things running, and user contributions help out the next person!

[benjamin-cash]

Hi @cigdemakan @gspetro-NOAA - I haven't tried installing the SRW via containers recently, but I have built the various components of the global-workflow, including the weather model, by externalizing the modules from the container and then just setting up my cluster as a new system. This is similar to the approach you would use if you had built spack-stack on your system and then tried to install the model (so not using the containerized version of the SRW-app) without the (severe) challenges of trying to build spack-stack directly.

Looking at your original post, if you have found the places where you need to update the SRW config files to find the spack-stack modules on your system then I think that this approach will work for you. If you would like, I have some tutorial slides that I've put together for my own group that I'd be happy to share if you would like.

[cigdemakan]

@gspetro-NOAA thank you very much for the information! The link you shared is the one I was looking at and referring to, and one would definitely need to create a workflow_ file for a new system.

@benjamin-cash the tutorial slides would be really helpful - thank you very much in advance!

[benjamin-cash]

Here you go!
containerized module instructions.pptx

This will take you up through installing the spack-stack modules. I'm afraid I haven't looked into modifying SRW to recognize when it is on a new system yet, but I will probably recognize some of the errors that will crop up when you do.

When you inevitably run into an issue with these instructions because I left something out, just let me know what the error was and I will do my best to diagnose. :)

[jdkublnick]

Thank you for the assistance, everyone @gspetro-NOAA @benjamin-cash @rickgrubin-noaa. This discussion has been moved into the SRW repository. Please feel free to come back to this discussion with any updates/questions about your work, @cigdemakan. Any progress you make could definitely help others in the future!

[cigdemakan]

@benjamin-cash thank you very much! I really appreciate it.
I thank everyone for their assistance and will post an update when I make a progress.