Build JEDI environment on Gaea/C6.

[lliu1220]

Seeking support on building JEDI environment on C6. JEDI environment is currently only configured on C5 according to:
https://spack-stack.readthedocs.io/en/1.5.1/PreConfiguredSites.html

Were able to load most similar spack-stack modules of C5 on C6:
module load PrgEnv-intel/8.3.3
module load intel-classic/2023.1.0
module load cray-mpich/8.1.25
module load python/3.9.12

module use /lustre/f2/dev/wpo/role.epic/contrib/spack-stack/c5/modulefiles
module load ecflow/5.8.4
module load mysql/8.0.31

module use /lustre/f2/dev/wpo/role.epic/contrib/spack-stack/c5/spack-stack-1.5.1/envs/unified-env/install/modulefiles/Core
module load stack-intel/2023.1.0
module load stack-cray-mpich/8.1.25
module load stack-python/3.10.8

But the actual JEDI environment cannot be loaded:
module load jedi-fv3-env
module load ewok-env
module load soca-env
module load sp

for instance, when I tried to load JEDI FV3 ENvironemnt after preconfiguration, I got the following error:
home2/Ling.Liu> module load jedi-fv3-env
Lmod has detected the following error:
The following module(s) are unknown:
"jedi-fv3-env"

[travissluka]

spack stack 1.8 is supported on C6 , see https://spack-stack.readthedocs.io/en/1.8.0/PreConfiguredSites.html#noaa-rdhpcs-gaea-c6

[gspetro-NOAA]

Hi @lliu1220,

I believe @RatkoVasic-NOAA is looking into this for you in more detail. We also received your email, but I think it's best to continue the conversation here so that others can benefit from it.

Best,
Gillian Petro

[gspetro-NOAA]

I am posting @ulmononian 's email reply here so that others can benefit:

Hi Ling,

Currently, only spack-stack/1.6.0 and 1.8.0 are installed on Gaea C6. spack-stack versions installed on C5 will not work correctly on C6.

Gaea C5's OS and PrgEnv were just updated, so we are currently working to re-build the stacks on C5. Then, we will install spack-stack/1.5.1 on Gaea C6. We can notify you when this installation is ready for use. In the meantime, if you would like to try spack-stack/1.6.0 on C6 (this is the version currently used in the UFS-WM develop branch), it can be loaded as follows:

module load PrgEnv-intel/8.5.0
module load intel-classic/2023.2.0
module load cray-mpich/8.1.29
module load python/3.11

module use /ncrc/proj/epic/spack-stack/c6/spack-stack-1.6.0/envs/unified-env/install/modulefiles/Core
module load stack-intel/2023.2.0
module load stack-cray-mpich/8.1.29

To load the jedi, ewok, and soca environments, please also do:

module load jedi-fv3-env
module load ewok-env
module load soca-env

I hope this helps in the interim. As I mentioned, we will let you know when 1.5.1 is available on C6. Please let us know if you have additional questions.

Thank you,
Cameron

[lliu1220]

Thank you @gspetro-NOAA @travissluka @ulmononian .
I have tested on C6 on my end, so most of the modules loaded well except there is no: python/3.11
So I had substituted with python/3.10.13.
Then the ecbuild ran into a python error:
If you are familiar with this error and how to avoid it please let me know.

--------------------------------------------------------------------------------------
-- Configuring done
CMake Error at ioda/src/engines/ioda/CMakeLists.txt:316 (target_link_libraries):
  Target "ioda_engines" links to:
    Python3::Python
  but the target was not found.  Possible reasons include:

    * There is a typo in the target name.
    * A find_package call is missing for an IMPORTED target.
    * An ALIAS target is missing.
CMake Error at ioda/src/engines/ioda/python/CMakeLists.txt:28 (target_link_libraries):
  Target "_ioda_python" links to:
    Python3::Module
  but the target was not found.  Possible reasons include:
    * There is a typo in the target name.
    * A find_package call is missing for an IMPORTED target.
    * An ALIAS target is missing.
-- Generating done
CMake Generate step failed.  Build files cannot be regenerated correctly.

[ulmononian]

@lliu1220 please try module load stack-python before you load the jedi/ewok/soca environments, and then try to build again and let us know if anything changes.

[lliu1220]

So , I turned off IODA-ENGINE and the compile worked (just loading stack-python wouldn't do).
however, when I try to run a simple testcase, this is the error I got:

roms-jedi/test> srun -M c6 -A drsa-hurr3 -t 00:05:00 -p batch -N 1 -n 1 test_romsjedi_geometry testinput/geometry.yaml 
srun: job 207216790 queued and waiting for resources
srun: job 207216790 has been allocated resources
OOPS Starting 2024-10-23 14:11:45 (UTC-0400)
Configuration input file is: testinput/geometry.yaml
Full configuration is:YAMLConfiguration[path=testinput/geometry.yaml, root={geometry => {project_dir => Data/wc13 , roms_stdinp => roms_wc13_20040103.in , fields metadata => Data/fields_metadata.yaml , ng => 1 , write to gmsh => false , gmsh filename => wc13.msh}}]
OOPS_STATS ObjectCountHelper started.
OOPS_STATS Run start                                - Runtime:      0.08 sec,  Memory: total:    98.50 Mb, per task: min =    98.50 Mb, max =    98.50 Mb
Run: Starting oops::Test running test::Geometry<ROMSJEDI>
Running 2 tests:
Running case 0: interface/Geometry/testConstructor ...
--------------------------------------------------------------------------------
 Model Input Parameters:  ROMS/TOMS version 4.2  
                          Wednesday - October 23, 2024 -  2:11:45 PM
--------------------------------------------------------------------------------
forrtl: severe (153): allocatable array or pointer is not allocated
Image              PC                Routine            Line        Source             
libroms.so         00007F5A7243398A  get_metadata_mod_         976  get_metadata.f90
libroms.so         00007F5A72134230  mod_ncparam_mp_in        1175  mod_ncparam.f90
libroms.so         00007F5A7251DF9E  read_phypar_              240  read_phypar.f90
libroms.so         00007F5A72456729  inp_par_mod_mp_in         101  inp_par.f90
libroms.so         00007F5A720E39AF  roms_kernel_mod_m         139  roms_kernel.f90
libromsjedi.so     00007F5A8BC91A37  roms_geom_mod_mp_     Unknown  Unknown
libromsjedi.so     00007F5A8BC7D94D  roms_geom_init_f9     Unknown  Unknown
libromsjedi.so     00007F5A8BC644E5  _ZN8romsjedi8Geom     Unknown  Unknown
test_romsjedi_geo  000000000040ECD8  Unknown               Unknown  Unknown
test_romsjedi_geo  000000000040E87C  Unknown               Unknown  Unknown
test_romsjedi_geo  0000000000413598  Unknown               Unknown  Unknown
test_romsjedi_geo  0000000000412FC7  Unknown               Unknown  Unknown
test_romsjedi_geo  000000000040AAB4  Unknown               Unknown  Unknown
liboops.so         00007F5A8421D608  _ZN4oops3Run7exec     Unknown  Unknown
test_romsjedi_geo  000000000040A889  Unknown               Unknown  Unknown
libc-2.31.so       00007F5A71EDE24D  __libc_start_main     Unknown  Unknown
test_romsjedi_geo  000000000040A74A  Unknown               Unknown  Unknown
[CRAYBLAS_WARNING] **Application linked against multiple cray-libsci libraries**
srun: error: c6n0200: task 0: Exited with exit code 153
srun: Terminating StepId=207216790.0

[ulmononian]

@lliu1220 not sure if this will work w/ this specific problem, but please try doing the following before compiling/running (in the ufs-wm, it is done in the modulefiles, which are loaded before compile and runtime):

unload("darshan-runtime")
unload("cray-libsci")

[gspetro-NOAA]

@lliu1220 Just wanted to check in and see if @ulmononian 's suggestions resolved the problem for you or if you're still seeing issues?

[lliu1220]

@gspetro-NOAA
@ulmononian 's suggestions are helpful, I was able to load the environment and build ROMS-JEDI following these suggestions.
The new error message above is a srun error, which has not yet been resolved.
I have reached out to C6 helpdesk and awaiting for a response.

[ulmononian]

@lliu1220 did you have any luck trying

unload("darshan-runtime")
unload("cray-libsci")

before running the model? you may have to recompile with the cray-libsci unloaded as well, and then unload before running.

[lliu1220]

@ulmononian I did try what you suggested on the cray libsci, it didn't matter; the other module was never loaded.
C6 updated python to :

python/3.10.13
stack-python/3.10.13

then the compile issue is outstanding again:

-- The following OPTIONAL packages have not been found:

 * gsibec (required version >= 1.2.1)
 * bufr_query
 * Python3
 * crtm (required version >= 2.4.0)
 * rttov (required version >= 12.1.0)
 * ropp-ufo
 * geos-aero
 * oasim

-- Configuring done
CMake Error at ioda/src/engines/ioda/CMakeLists.txt:316 (target_link_libraries):
  Target "ioda_engines" links to:

    Python3::Python

  but the target was not found.  Possible reasons include:

    * There is a typo in the target name.
    * A find_package call is missing for an IMPORTED target.
    * An ALIAS target is missing.



CMake Error at ioda/src/engines/ioda/python/CMakeLists.txt:28 (target_link_libraries):
  Target "_ioda_python" links to:

    Python3::Module

  but the target was not found.  Possible reasons include:

    * There is a typo in the target name.
    * A find_package call is missing for an IMPORTED target.
    * An ALIAS target is missing.



-- Generating done
CMake Generate step failed.  Build files cannot be regenerated correctly.

[ulmononian]

@lliu1220 can you please reiterate what the current problem is? it sounded like it was compiling, but failing to run, but now you again cannot compile? if you share the build/run steps, i can ask around to see if we can find a solution.

[lliu1220]

@ulmononian @RatkoVasic-NOAA, adding @IGRivin -
I just tried to compile and tested again.
NOW I was able to compile with python binding turned OFF, so I don't run into IODA engine error.- But this could cause problem later, so far the error was unrelated.
Then when I test it. I ran into this different SRUN error. Maybe my srun line wasn't correct?

srun -M c6 -A drsa-hurr3 -t 00:05:00 -p patch -N 1 -n 1 test_romsjedi_geometry testinput/geometry.yaml

I tried two partitions:

C6 and batch;
es and eslogin

If any of you ever had this srun error please write something here. thank you

roms-jedi/test> srun -M c6 -A drsa-hurr3 -t 00:05:00 -p patch -N 1 -n 1 test_romsjedi_geometry testinput/geometry.yaml
srun: job 67852101 queued and waiting for resources
srun: job 67852101 has been allocated resources
srun: error: Task launch for StepId=67852101.0 failed on node gaea62: Header lengths are longer than data received
srun: error: Application launch failed: Header lengths are longer than data received
srun: Job step aborted
roms-jedi/test> 

Here is my entire module load for your testing, notice loading stack python will automatically update python to the latest:

module load PrgEnv-intel/8.5.0
module load intel-classic/2023.2.0
module load cray-mpich/8.1.29

module use /ncrc/proj/epic/spack-stack/c6/spack-stack-1.6.0/envs/unified-env/install/modulefiles/Core
module load stack-intel/2023.2.0
module load stack-cray-mpich/8.1.29
module load stack-python

module load jedi-fv3-env
module load ewok-env
module load soca-env

ecbuild -DMPIEXEC_EXECUTABLE=/usr/bin/srun -DMPIEXEC_NUMPROC_FLAG="-n" -DUSE_ECCODES=ON -DBUILD_IODA_CONVERTERS=OFF -DBUILD_PYTHON_BINDINGS=OFF  -DENABLE_LORENZ95_MODEL=OFF -DENABLE_QG_MODEL=OFF -DBUNDLE_SKIP_ROPP-UFO=OFF  -DCMAKE_BUILD_TYPE=Release ../Bundle_wc13

[lliu1220]

@gspetro-NOAA @ulmononian @travissluka @IGRivin
So we worked with C6 helpdesk and found out that ROMS won't work with intel/2023.2.0

Please see following discussion for proof:
https://www.myroms.org/forum/viewtopic.php?p=24931&hilit=forrtl%3A+severe+153+%3A#p24931

I saw you mentioned spack-stack 1.5.1 will become available on C6, may I ask if it will be with a different compiler, and when it will become available?

Thank you.

[RatkoVasic-NOAA]

@lliu1220 on C6 spack-stack/1.5.1 is available only with intel/2023.2.0
/ncrc/proj/epic/spack-stack/c6/spack-stack-1.5.1/envs/unified-env/install/modulefiles/Core

I see one other classic intel compiler available: intel/2022.1.0
We can try to install spack-stack/1.5.1 with 2022.1.0 on Gaea-C6. I'll let you know if that one installed OK (1.5.1 is soon going away, and cannot spend too much time on that).

[lliu1220]

@RatkoVasic-NOAA
Ok, Thank you.
Our colleague has tested intel/2024.0 would work with ROMS, we aren't sure about 2022.1.0, but would be good to have an option to test.

module load PrgEnv-intel/8.5.0 cray-hdf5 cray-netcdf
module swap intel/2024.0

So looking forward to it, thanks.

[RatkoVasic-NOAA]

OK, I'll try with intel/2024.0

[RatkoVasic-NOAA]

Installation on Gaea C6, spack-stack-1.5.1 using Intel compiler 2024.0 didn't work:
/ncrc/proj/epic/spack-stack/c6/spack-stack-1.5.1/envs/ue-intel-2024.0/

[lliu1220]

Thank you for your response @RatkoVasic-NOAA .
Is it possible to try spack-stack/1.5.1 with 2022.1.0 also?
Many thanks.
P.S. If you have written instruction handy and able to share, I could try install 1.5.1 /2022.1.0 on my own as well, so to save your time.

[RatkoVasic-NOAA]

@lliu1220
Here are things to do to install spack-stack/1.5.0 on Gaea-C6:

git clone --recursive https://github.com/JCSDA/spack-stack.git -b release/1.5.1 spack-stack-1.5.1
cd spack-stack-1.5.1
module use /ncrc/proj/epic/spack-stack/modulefiles
module load ecflow/5.8.4
. setup.sh
spack stack create env --site gaea-c6 --template unified-dev --name unified-env
cd envs/unified-env
spack env activate .

      Edit: spack.yaml
      replace line: - compilers: ['%aocc', '%apple-clang', '%gcc', '%intel']
      with:         - compilers: ['%intel']
      Edit: site/compilers.yaml (replace 2023.2.0 with version you want, two places)
      Edit: site/packages.yaml  (replace 2023.2.0 with version you want, two places)

spack concretize 2>&1 | tee log.concretize
spack install  --verbose > log.install 2>&1 &
spack module lmod refresh
spack stack setup-meta-modules

You can also try different MPI implementations, in file site/packages.yaml, default option is cray-mpich/8.1.29, but we have available cray-mpich/8.1.28 and cray-mpich/8.1.30

[lliu1220]

@RatkoVasic-NOAA
Thank you for sharing, I was able to move forward with this instruction until
spack install --verbose > log.install 2>&1 &

so it looks like 1.5.1 /2022.1.0 failed as well. the log.install is copied here with permission to access:
/autofs/ncrc-svm1_home2/Ling.Liu/log.install

copy here in case anyone is familiar with this IndexError:

Traceback (most recent call last):
  File "/autofs/ncrc-svm1_proj/drsa-hurr3/Ling.Liu/spack-stack-1.5.1/spack/lib/spack/spack/build_environment.py", line 1048, in _setup_pkg_and_run    pkg, dirty=kwargs.get("dirty", False), context=context
  File "/autofs/ncrc-svm1_proj/drsa-hurr3/Ling.Liu/spack-stack-1.5.1/spack/lib/spack/spack/build_environment.py", line 779, in setup_package
    set_wrapper_variables(pkg, env_mods)
  File "/autofs/ncrc-svm1_proj/drsa-hurr3/Ling.Liu/spack-stack-1.5.1/spack/lib/spack/spack/build_environment.py", line 515, in set_wrapper_variables
    update_compiler_args_for_dep(dspec)
  File "/autofs/ncrc-svm1_proj/drsa-hurr3/Ling.Liu/spack-stack-1.5.1/spack/lib/spack/spack/build_environment.py", line 509, in update_compiler_args_for_dep
    _prepend_all(include_dirs, query.headers.directories)
  File "/autofs/ncrc-svm1_proj/drsa-hurr3/Ling.Liu/spack-stack-1.5.1/spack/lib/spack/spack/spec.py", line 1236, in __get__
    value = f()
  File "/autofs/ncrc-svm1_proj/drsa-hurr3/Ling.Liu/spack-stack-1.5.1/spack/lib/spack/spack/spec.py", line 1225, in <lambda>
    callbacks_chain.append(lambda: getattr(pkg, self.attribute_name))
  File "/autofs/ncrc-svm1_proj/drsa-hurr3/Ling.Liu/spack-stack-1.5.1/spack/var/spack/repos/builtin/packages/cray-mpich/package.py", line 112, in headers
    return hdrs
  File "/autofs/ncrc-svm1_proj/drsa-hurr3/Ling.Liu/spack-stack-1.5.1/spack/lib/spack/llnl/util/filesystem.py", line 1856, in __getitem__
    return self.files[item]
IndexError: list index out of range

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib64/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib64/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/autofs/ncrc-svm1_proj/drsa-hurr3/Ling.Liu/spack-stack-1.5.1/spack/lib/spack/spack/build_environment.py", line 1067, in _setup_pkg_and_run
    package_context = get_package_context(tb)
  File "/autofs/ncrc-svm1_proj/drsa-hurr3/Ling.Liu/spack-stack-1.5.1/spack/lib/spack/spack/build_environment.py", line 1256, in get_package_context
    func = getattr(obj, tb.tb_frame.f_code.co_name, "")
  File "/autofs/ncrc-svm1_proj/drsa-hurr3/Ling.Liu/spack-stack-1.5.1/spack/var/spack/repos/builtin/packages/cray-mpich/package.py", line 112, in headers
    return hdrs
  File "/autofs/ncrc-svm1_proj/drsa-hurr3/Ling.Liu/spack-stack-1.5.1/spack/lib/spack/llnl/util/filesystem.py", line 1856, in __getitem__
    return self.files[item]
IndexError: list index out of range

[RatkoVasic-NOAA]

@lliu1220 if spacck-stack-1.5.1 is not working for JEDI on Gaea-C6, and that stack version is to supported anymore, I suggest to go with existing 1.8.0, or even better, wait for 1.9.0 which will be released in next couple of weeks.

[lliu1220]

I have checked spack-stack-1.8.0 is compiled with the same intel 2023.2.0, which does not work with ROMS
I saw installation attempt of spack-stack-1.9.0 with 2024.2, when will this become available?
/autofs/ncrc-svm1_proj/epic/spack-stack/spack-stack-1.9.0

[RatkoVasic-NOAA]

@lliu1220 We are in the process of installing 1.9.1 (1.9.0 will be skipped). We hope by the end of next week to have 1.9.1 installations on all machines.