Syncing updated UFS_UTILS scripts for GFSv17 fix files

[drnimbusrain]

Description

This is in regards to SRW-App and using updated UFS_UTILS scripts for GFSv17 physics related fix files (e.g. sfc_climo and orog files). This is somewhat related to the issue Barry raised of UFS_UTILS not pointing to auth repo, but our main issue may not exactly be related to this point.

The main issue in using SRW-App [develop] for GFSv17 physics relates to a similar issue with UFS_UTILS and generation of the correct 'sfc_climo' and 'orog' fix files:

1. In SRW-App [develop] the UFS_UTILS external points to Michael's hash 069b11c, which uses the sfc_climo_gen script with later updates to use input sfc_climo and orog files needed for updated GFSv17 physics, but in current SRW-App [develop] the main scripts are not synced up to use such files (e.g., the exregional_make_sfc_climo_sh), which are different (see missing 'soil_color.clm.0.05.nc' file and other file differences) . Thus using the SRW-App [develop] and initial tasks (i.e., make_sfc_climo, make_orog) to generate the sfc_climo on our AQMv8 grid are not correct for use with updated GFSv17 physics, which we are required to use in future testing and implementations.

2. I also don't know if the make_orog scripts are synced up on the SRW-App side to match needed updated files for supporting GFSv17 (e.g., for issues related to updated GWD parameterizations).

3. We also note that the EPIC fix files related to sfc_climo and orog are relatively old on Gaea C6 (date stamped 2022; /gpfs/f6/bil-fire8/world-shared/UFS_SRW_data/develop/fix/fix_sfc_climo/ (or fix_orog/ or fix_am/)), so ultimately using SRW-App [develop] with older scripts and these files preclude us from consistently testing GFSv17 physics with fix files. Using the GFSv17 physics option in SRW-App runs OK for AQMv8 testing, but the input fix files for sfc_climo and orog are likely not fully consistent.

4. Including the SRW-App script changes for sfc_climo and orog necessitates updates the namelist to read in the UFS_UTILS produced files (via fixed_files_mapping), which is currently missing soil_color for example for GFSv17 consistency.

Is this something EPIC can help us work on updating so that we can continue to use SRW-App to test AQMv8 developments with GFSv17 and community support? Overall, we would like to still use SRW-App for some AQMv8 development and run the initial forecast tasks (e.g. make_sfc_climo, make_orog) to generate the correct fix files for GFSv17, but right now it seems we cannot with current [develop].

Solution

Two main tasks - one for synchronizing the SRW scripts and a second for a data refresh.

Requirements**

Updates to system tools/scripts and data inputs (possibly).

Acceptance Criteria (Definition of Done)

Using synced UFS_UTILS scripts and fix file data to consistently test updated GFSv17 physics in SRW-App.

Dependencies (optional)

#1340

@benkozi

[MichaelLueken]

@drnimbusrain -

The only path forward, at this point, is updating the version of UFS_UTILS to point to the authoritative repository within a branch of the noaa-oar-arl fork. I have completed the necessary changes to use the 6c05801 hash of ufs-community/UFS_UTILS in my feature/aqmv8_p1.2 branch.

Should I open a PR into the feature/aqmv8_p1.2 branch in the noaa-oar-arl fork, with these changes? Unfortunately, due to the inability to run experiments using RAP, HRRR, and RRFS physics schemes while using the HEAD of UFS_UTILS, it is impossible to update to the authoritative UFS_UTILS in the authoritative develop branch of the ufs-srweather-app.

[benkozi]

@MichaelLueken Does the UFS_UTILS compatibility issue prevent us from updating the SRW scripts and refreshing fix files?

[drnimbusrain]

Thank you @MichaelLueken, and thanks @benkozi for noting that. It is my question as well. Seems we could update within the SRW scripts and files alone, without having to point to UFS_UTILS authoritative repository. I am open to either path forward.

[MichaelLueken]

@drnimbusrain and @benkozi -

1. The soil_color fixed file is not currently available in the fix files available from EPIC. So, this will need to be added before including this file in the fort.41 file in scripts/exregional_make_sfc_climo.sh. Currently, the available fixed files for make_sfc_climo are:

(srw_app) [Michael.Lueken@ufe01 fix_sfc_climo]$ ls
facsf.1.0.nc		     snowfree_albedo.4comp.0.05.nc    substrate_temperature.1.0.nc	vegetation_type.igbp.0.05.nc	    vegetation_type.modis.igbp.conus.0.01.nc  vegetation_type.viirs.igbp.conus.0.01.nc
leaf_area_index.0.05.nc      soil_type.statsgo.0.03.nc	      substrate_temperature.2.6x1.5.nc	vegetation_type.igbp.conus.0.01.nc  vegetation_type.viirs.igbp.0.03.nc
maximum_snow_albedo.0.05.nc  soil_type.statsgo.0.05.nc	      vegetation_greenness.0.144.nc	vegetation_type.modis.igbp.0.03.nc  vegetation_type.viirs.igbp.0.05.nc
slope_type.1.0.nc	     soil_type.statsgo.conus.0.01.nc  vegetation_type.igbp.0.03.nc	vegetation_type.modis.igbp.0.05.nc  vegetation_type.viirs.igbp.0.1.nc

2. For orog fixed files, the current authoritative UFS_UTILS is expecting three files. These files are available in fix_orog available on RDHPCs:

(srw_app) [Michael.Lueken@ufe01 fix_orog]$ ls
GlobalLakeDepth.dat   GlobalLakeStatus.txt  TOP8M_slm.80I1.asc	convert.f90		 gmted2010.30sec.int	  gtopo30_gg.fine.nh	makefile			 topography.antarctica.ramp.30s.nc
GlobalLakeDepth.txt   TOP8M_avg.20I4.asc    clmgrb		gmted2010.30sec.flt	 gmted2010.30sec.int.ctl  landcover.umd.30s.nc	run.lsf				 topography.gmted2010.30s.nc
GlobalLakeStatus.dat  TOP8M_max.20I4.asc    clmgrb.index	gmted2010.30sec.flt.ctl  gtopo30_gg.fine	  landcover30.fixed	thirty.second.antarctic.new.bin

Please note that we can only point to the different fixed files for the make_orog task if we use the authoritative UFS_UTILS, otherwise, we need to continue using what is currently in the scripts/exregional_make_orog.sh script.

3. The updated orog fixed files are up-to-date, but require to use the HEAD of the authoritative UFS_UTILS repository:

Mar  6  2024 landcover.umd.30s.nc
Mar  6  2024 topography.antarctica.ramp.30s.nc
Mar  6  2024 topography.gmted2010.30s.nc

This will also appear for the soil_color fixed file once it is added.

4. The parm/fixed_files_mapping.yaml file would need to be updated to link in the new soil_color fixed file for make_sfc_climo.

[MichaelLueken]

@drnimbusrain and @benkozi -

Looking more closely into soil_color, this appears to only be used in NOAH-MP. This isn't exercised in the SRW App, it is only used in Land DA. It isn't clear to me that the soil_color fixed file is required for AQM or SRW. We should make sure that workflows outside of Land DA use soil_color before adding this into the SRW App.

[drnimbusrain]

@drnimbusrain and @benkozi -

Looking more closely into soil_color, this appears to only be used in NOAH-MP. This isn't exercised in the SRW App, it is only used in Land DA. It isn't clear to me that the soil_color fixed file is required for AQM or SRW. We should make sure that workflows outside of Land DA use soil_color before adding this into the SRW App.

Thanks @MichaelLueken Maybe @barlage can comment more on use of soil_color (and other available sfc_climo and orog files pointed to at the HEAD of UFS_UTILS) for areas outside of Land DA. He did mention we should be using it with Noah-MP and GFSv17 physics in our AQMv8 development.