SRW App doesn't run on PW AWS

[EdwardSnyder-NOAA]

Expected behavior

WE2E tests should pass on PW AWS.

Current behavior

However, when running WE2E tests, I'm getting this shared library error in the make_sfc_climo task:

 All executables will be submitted with command \'mpiexec -np $nprocs\'.
/contrib/Edward.Snyder/1321/ufs-srweather-app/exec/sfc_climo_gen: /contrib/Edward.Snyder/1321/ufs-srweather-app/conda/lib/libstdc++.so.6: version `GLIBCXX_3.4.32' not found (required by /contrib/Edward.Snyder/1321/ufs-srweather-app/exec/sfc_climo_gen)

I attempted to resolve it by following the solution (setting LD_PRELOAD) found in a similar issue on Gaea C5 but it didn't work. Given this is a tier 2 machine and that it was missing UFS_SRW_data for months, I'll tackle it later given other pressing responsibilities.

Machines affected

PW AWS

Steps To Reproduce

To produce, build the model given these instructions and run a WE2E test as seen here. It should fail in the make_sfc_climo task.

Detailed Description of Fix (optional)

Additional Information (optional)

Possible Implementation (optional)

Output (optional)

[benkozi]

Just FYI, a user reported a similar issue on Gaea-C6. We were able to resolve via:

conda activate srw_app
conda activate base
conda update libstdcxx-ng
conda activate srw_app

It was with an older SRW branch/fork, so we didn't pursue it since it hadn't appeared in the head of develop yet. However, that might not be the case anymore. With a quick look, I was not able to determine why LD_PRELOAD was not working. I also was not able to quickly determine if the issue is introduced at compile or runtime.

[EdwardSnyder-NOAA]

More investigation showed the reason for the libstdc++.so.6: version GLIBCXX_3.4.32 not found error is that the executables were built with a newer GNU C++ library than what is present at runtime, which is using the conda srw_app env. The SRW App loads the conda srw_app env last which causes those library paths to be stored first in the LD_LIBRARY_PATH variable, superseding the host library paths. This explains why updating libstdcxx-ng in the base conda worked (which worked for me here too). Also, this same error is present on PW Azure as well.

There are two solutions here: (1) update the base conda libstdcxx-ng in the environment.yml file or (2) add the host library path as a module task file. I would lean towards solution 2, since it has only occurred on Gaea C6 and PW AWS/Azure, and is dependent on what GNU C++ library is used during the compilation, which is machine dependent.

Solution 2 would requires this line setenv("LD_PRELOAD", "/apps/gnu/gcc-13.2.0/lib64/libstdc++.so.6") to be added at the end of this python task files: ufs-srweather-app/modulefiles/tasks/noaacloud/python_srw.lua. This solution does work on AWS and Azure.

Work dir on AWS: /contrib/Edward.Snyder/1321/expt_dirs/grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_RAP_suite_WoFS_v0
Work dir on Azure: /contrib/Edward.Snyder/srw-v3/expt_dirs/community