Errors for making ICS using HRRR for predefined RRFS_CONUS_3km

[danrosen25]

@JeffBeck-NOAA
task_make_ics is failing with a FATAL ERROR: IN FieldRegridStore on Jet. I looked at the ESMF PET logs and they each haver errors similar to the error below. I attached (but renamed as .txt the config.yaml) file. Is there something wrong in my configuration file? I took a look through other issues but the only similar issue was using a custom grid, I'm using a predefined grid RRFS_CONUS_3km

ERROR Destination id=1387453 NOT found in weight matrix.
ERROR ESMCI_Mesh_Regrid_Glue.C:363 ESMCI_regrid_create() Arguments are incompatible  - - There exist destination points (e.g. id=1387453) which can't be mapped to any source cell
ERROR ESMCI_Mesh_Regrid_Glue.C:586 ESMCI_regrid_create() Arguments are incompatible  - Internal subroutine call returned Error
ERROR ESMCI_MeshCap.C:1599 MeshCap::regrid_create() Arguments are incompatible  - Internal subroutine call returned Error
ERROR ESMF_Regrid.F90:368 ESMF_RegridStore Arguments are incompatible  - Internal subroutine call returned Error
ERROR ESMF_FieldRegrid.F90:1202 ESMF_FieldRegridStoreNX Arguments are incompatible  - Internal subroutine call returned Error
ERROR atmosphere.F90:190 Arguments are incompatible  - Passing error in return code

config.txt

Notes:
I'm working off SRW@d93f6d1bb3ea8d57e8af696ffa5b193d735b3bd6 with the following UFS_UTILS revision.

[ufs_utils]
repo_url = https://github.com/ufs-community/UFS_UTILS
hash = 4d2768c

[JeffBeck-NOAA]

@danrosen25, I see that you're using HRRR data, so you'll need to use the RRFS_CONUScompact_3km grid, since the HRRR domain is too small to initialize the RRFS_CONUS_3km grid. The following error means that the external model data and chosen grid aren't compatible:

ERROR ESMCI_Mesh_Regrid_Glue.C:363 ESMCI_regrid_create() Arguments are incompatible - - There exist destination points (e.g. id=1387453) which can't be mapped to any source cell

[danrosen25]

@JeffBeck-NOAA I switched to FV3GFS for the ICS data and that fixed the make ICS issue. The forecast still didn't finish because it crashed in 'range_check(PT UPDATE ...)' due to a SIGTERM in MPI AllReduce for FV3 during the 'atmosphere_state_update' step. I don't know what this is about.

[gspetro-NOAA]

Hi @danrosen25 !

Just wanted to see if you were able to resolve this issue offline with Jeff or if you need further assistance.

Best,
Gillian Petro

[JeffBeck-NOAA]

@danrosen25, @gspetro-NOAA, that error looks like a random run-time failure that may resolve itself when the run_fcst job is submitted a second time.