chgres_cube runs out of memory when using C1152 input

[jswhit]

Running chgres_cube on hercules/orion using C1152 warm start restarts runs out of memory. I'm running on 6 nodes with 1 task per node. The last output it prints before the OOM killer gets it is CALL FieldScatter FOR INPUT GRID U.

Is this a known issue, and if so is there a known workaround?

[jswhit]

fort.41 (the namelist) looks like this:

&config
  mosaic_file_target_grid="/work/noaa/gsienkf/whitaker/global-workflow/fix/orog/C768/C768_mosaic.nc"
  fix_dir_target_grid="/work/noaa/gsienkf/whitaker/global-workflow/fix/orog/C768/sfc"
  orog_dir_target_grid="/work/noaa/gsienkf/whitaker/global-workflow/fix/orog/C768"
  orog_files_target_grid="C768.mx025_oro_data.tile1.nc","C768.mx025_oro_data.tile2.nc","C768.mx025_oro_data.tile3.nc","C768.mx025_oro_data.tile4.nc","C768.mx025_oro_data.tile5.nc","C768.mx025_oro_data.tile6.nc"
  vcoord_file_target_grid="/work/noaa/gsienkf/whitaker/global-workflow/fix/am/global_hyblev.l128.txt"
  mosaic_file_input_grid="/work/noaa/gsienkf/whitaker/global-workflow/fix/orog/C1152/C1152_mosaic.nc"
  orog_dir_input_grid="/work/noaa/gsienkf/whitaker/global-workflow/fix/orog/C1152"
  orog_files_input_grid="C1152.mx025_oro_data.tile1.nc","C1152.mx025_oro_data.tile2.nc","C1152.mx025_oro_data.tile3.nc","C1152.mx025_oro_data.tile4.nc","C1152.mx025_oro_data.tile5.nc","C1152.mx025_oro_data.tile6.nc"
  data_dir_input_grid="/work/noaa/gsienkf/whitaker/ICSDIR/C1152/gdas.20250910/00/model/atmos/restart"
  atm_files_input_grid="NULL"
  atm_core_files_input_grid="fv_core.res.tile1.nc","fv_core.res.tile2.nc","fv_core.res.tile3.nc","fv_core.res.tile4.nc","fv_core.res.tile5.nc","fv_core.res.tile6.nc","fv_core.res.nc"
  atm_tracer_files_input_grid="fv_tracer.res.tile1.nc","fv_tracer.res.tile2.nc","fv_tracer.res.tile3.nc","fv_tracer.res.tile4.nc","fv_tracer.res.tile5.nc","fv_tracer.res.tile6.nc"
  sfc_files_input_grid="sfc_data.tile1.nc","sfc_data.tile2.nc","sfc_data.tile3.nc","sfc_data.tile4.nc","sfc_data.tile5.nc","sfc_data.tile6.nc"
  nst_files_input_grid="NULL"
  grib2_file_input_grid="NULL"
  geogrid_file_input_grid="NULL"
  varmap_file="NULL"
  wam_parm_file="NULL"
  cycle_year=2025
  cycle_mon=09
  cycle_day=15
  cycle_hour=00
  convert_atm=.true.
  convert_sfc=.true.
  convert_nst=.true.
  input_type="restart"
  tracers="sphum","liq_wat","o3mr","ice_wat","rainwat","snowwat","graupel"
  tracers_input="sphum","liq_wat","o3mr","ice_wat","rainwat","snowwat","graupel"
  regional=0
  halo_bndy=0
  halo_blend=0
  sotyp_from_climo=.true.
  vgtyp_from_climo=.true.
  vgfrc_from_climo=.true.
  minmax_vgfrc_from_climo=.true.
  tg3_from_soil=.false.
  lai_from_climo=true.
  external_model="GFS"
  nsoill_out=4
  thomp_mp_climo_file="/work/noaa/gsienkf/whitaker/global-workflow/fix/am/Thompson_MP_MONTHLY_CLIMO.nc"
  wam_cold_start=.false.
 /

Note that

/work/noaa/gsienkf/whitaker/global-workflow/fix/orog -> /work2/noaa/global/role-global/fix/orog/20240917