'get_extrn_ics' fails on Discover w/community input data

[ssinno]

 While I've made much headway on NASA's NCCS Discover platform, the 'get_extrn_ics' task fails due to errors outlined below.  This run is utilizing the 'fix.tar.gz' and 'gst_data.tgz' datasets as specified in section 3.2.3 of the documentation.  It appears to my less-than-enlightened eye that the data itself is being rejected by hard-coded values in the 'JREGIONAL_GET_EXTRN_MDL_FILES' job.  Is my inference here off-target, or is there perhaps alternate/newer sample data available?  

(srw_app) [ssinno@discover33 test_community]egrep -A 6 "^ERROR" log/get_extrn_ics_2019061518.log
ERROR:
From function: "source"
In file: "JREGIONAL_GET_EXTRN_MDL_FILES"
Full path to file: "/gpfsm/dnb34/ssinno/srw/ufs-srweather-app/jobs/JREGIONAL_GET_EXTRN_MDL_FILES"
NETCDF is not available for this date:: However, NEMSIO is available:: Output from the specified external model (EXTRN_MDL_NAME) is not availa- ble for the specified cycle date and time (EXTRN_MDL_CDATE) because the latter is earlier than the implementation date of this model: EXTRN_MDL_NAME = "FV3GFS" CDATE_min = "2021032100" EXTRN_MDL_CDATE = "2019061518"
Exiting with nonzero status.

[ssinno]

Apologies, meant to include this additional bit of info

sed -n '211,217 p' jobs/JREGIONAL_GET_EXTRN_MDL_FILES

The transition date from the GSMGFS to the FV3GFS was 2019061212, i.e.

this was the first official forecast with the FV3GFS. However, paral-

lel runs with the FV3GFS go back to 2018121500. So we set the first

EXTRN_MDL_CDATE for the FV3GFS to this date and time.

CDATE_min="2018121500"
CDATE_min_netcdf="2021032100"
CDATE_max_nemsio="2021032018"

[MichaelLueken]

Hello @ssinno,

The task_get_extrn_ics section in your config.yaml file should include the following for the community experiment:

task_get_extrn_ics:
  envvars:
    EXTRN_MDL_NAME_ICS: FV3GFS
    FV3GFS_FILE_FMT_ICS: grib2
    USE_USER_STAGED_EXTRN_FILES: true
    EXTRN_MDL_SOURCE_BASEDIR_ICS: "/path/to/UFS_SRW_data/develop/input_model_data/<model_type>/<data_type>/${yyyymmddhh}"

If your config.yaml file looks like the above (with the path to the input model data set appropriately for Discover), please provide your config.yaml file so that I can try and see what might be happening.

[ssinno]

Seemed a promising suggestion as I was indeed missing defs for 

EXTRN_MDL_NAME_ICS & FV3GFS_FILE_FMT_ICS but unfortunately adding them did
not yield different results, even after purging & re-generating the 'test_community'
directory via 'generate_FV3LAM_wflow.py'.

config.yaml is attached, as well as the relevant logfile and the machine config

residing under ufs-weather-app/ush/machine/

get_extrn_ics_2019061518.log
discover.yaml
config.yaml

[MichaelLueken]

Hello @ssinno,

Are you using the authoritative develop branch, or are you using the SRW v3.0.0 release?

Looking in your community experiment's config.yaml file, I see from the task_run_fcst and task_plot_allvars sections that the config.community.yaml file from the authoritative develop branch was used to generate the experiment. However, both the task_get_extrn_ics and task_get_extrn_lbcs sections suggest the version is the SRW v3.0.0 release.

Given that the error message in the get_extrn_ics_2019061518.log states that there is no NETCDF version available, and NETCDF is the default file format for both ICs and LBCs, it looks like the default FV3GFS_FILE_FMT_ICS setting in config_defaults.yaml is being used rather than the setting in your config.yaml file.

Please try updating your config.yaml file as follows:

metadata:
  description: >-
    Sample community config
user:
  RUN_ENVIR: community
  MACHINE: Discover
  ACCOUNT: s2827
workflow:
  USE_CRON_TO_RELAUNCH: false
  EXPT_SUBDIR: test_community
  CCPP_PHYS_SUITE: FV3_GFS_v16
  PREDEF_GRID_NAME: RRFS_CONUS_25km
  DATE_FIRST_CYCL: '2019061518'
  DATE_LAST_CYCL: '2019061518'
  FCST_LEN_HRS: 12
  PREEXISTING_DIR_METHOD: rename
  VERBOSE: true
  COMPILER: gnu
task_get_extrn_ics:
  envvars:
    EXTRN_MDL_NAME_ICS: FV3GFS
    FV3GFS_FILE_FMT_ICS: grib2
    USE_USER_STAGED_EXTRN_FILES: true
    EXTRN_MDL_SOURCE_BASEDIR_ICS: "/discover/nobackup/ssinno/srw/DATA/UFS/input_model_data/FV3GFS/grib2/2019061518"
task_get_extrn_lbcs:
  envvars:
    EXTRN_MDL_NAME_LBCS: FV3GFS
    LBC_SPEC_INTVL_HRS: 6
    FV3GFS_FILE_FMT_LBCS: grib2
    USE_USER_STAGED_EXTRN_FILES: true
    EXTRN_MDL_SOURCE_BASEDIR_LBCS: "/discover/nobackup/ssinno/srw/DATA/UFS/input_model_data/FV3GFS/grib2/2019061518"
task_run_fcst:
  QUILTING: true
  fv3:
    execution:
      batchargs:
        walltime: 02:00:00
task_plot_allvars:
  envvars:
    COMOUT_REF: ""
global:
  DO_ENSEMBLE: false
  NUM_ENS_MEMBERS: 2

Please delete your current experiment and regenerate using generate_FV3LAM_wflow.py once the above changes have been made and see if the experiment will run.

[ssinno]

Many thanks!  I'm using ufs-srw-v3.0.0 release, happy to switch to the dev branch if recommended.  I'm out of the office today and may not have an opportunity to re-exec for a bit but will certainly try soon!

[ssinno]

   Definitive progress, but the finish line continues to elude.  Got up to 'run_fcst_mem000' which dies with the error below, full log attached.  Worth noting that 'PE_MEMBER01' is actually defined in test_community/var_defns.yaml but I guess isn't making it to the run?  Should I perhaps add that to config.yaml, or simply set it via 'export' in my working shell?   

---from attached log---
History not written to native grid or dynamically-generated grids not used. No hala0 grid link created.
/gpfsm/dnb34/ssinno/srw/ufs-srweather-app/scripts/exregional_run_fcst.sh: line 918: PE_MEMBER01: unbound variable
FATAL ERROR:
ERROR:
From function: "source"
In file: "JREGIONAL_RUN_FCST"
Full path to file: "/gpfsm/dnb34/ssinno/srw/ufs-srweather-app/jobs/JREGIONAL_RUN_FCST"
Call to ex-script corresponding to J-job "JREGIONAL_RUN_FCST" failed.
Exiting with nonzero status.
run_fcst_mem000_2019061518.log

[MichaelLueken]

Good morning @ssinno,

I've not encountered an issue where PE_MEMBER01 is an unbound variable, especially if it is present in the var_defns.yaml file. It wouldn't hurt to add the calculated PE_MEMBER01 from the var_defns.yaml file into your config.yaml file, delete the experiment, then regenerate the community experiment.

I'll also note that a final decision regarding EPIC's continued support for the SRW App will be made this Thursday afternoon. If adding PE_MEMBER01 to your config.yaml file doesn't work, then I would recommend not pursuing the porting of the SRW App to Discover at this time.

[ssinno]

An explicit addition of PE_MEMBER01 in config.yaml proved sufficient to cross the finish line.   I've attached the final yaml config.  Much obliged! 

config.yaml