Run_fcst is failing on grid_spec ntiles on Linux

[jacobsmithsfd]

Here's some background info:
Platform: Linux
SRW version: develop
config file.odt
linux config.odt
I'm using a bash shell and the only module I have loaded is wflow_linux.
Compiler + MPI combination being used: GCC 13.2.0 MPI 4.2.3

The issue is at the workflow task run_fcst
I’m getting the error when run_fcst is looking at info from the grid_spec file: FATAL from PE 0: fv_grid_tools(ge
t_symmetry): ntiles should be 6.
I’ve done an ncdump -h on several mosaic files and they all show tiles of 1 regardless of where the mosaic file originates, the repository or generated locally. From what I can tell, the number of tiles that I need to have is 1 but grid_tools wants to match it to 6. It seems like there's something that either isn't getting passed into fv_grid_tools or I have something wrong with my configuration settings but I'm not sure where to go from here and any help is appreciated.

Thanks you!

[gspetro-NOAA]

Hi @jacobsmithsfd !

Sorry for the delay in getting back to you, and thanks for all the info you provided! Would it be possible to also send us your log file(s)? And in your experiment directory, is ntiles set to 1 or 6 in the input.nml file? If it's set to 6, then something is wrongly being passed there/calculated. I'm also wondering if there could be an issue with the grid you're defining, but I am going to reach out to our technical experts for further guidance.

Best,
Gillian Petro

[jacobsmithsfd]

Hi Gillian, I'll get those files to you soon. I'm away from my desk until early next week.

[gspetro-NOAA]

Great! In the meantime, I asked @MichaelLueken if he'd seen similar errors before, and he hasn't. However, he checked the staged C403_mosaic.halo* files and the mosaic files that are created in make_grid, and they all have ntiles equal to 1 in our experiments. For global/cubed-sphere forecasts, ntiles must be 6, but for any other domain (such as the community experiment regional domain), ntiles should be set to 1. So you may want to see if you're indicating the cubed-sphere domain rather than regional. But hopefully seeing the log files will help us to track down where the problem is coming from!

[jacobsmithsfd]

Hi Gillian,
Sorry it took longer to gather the log files than anticipated but here they are. Input.nml from the experiment directory pulls ntiles from gridspec.nc. Gridspec.nc is made from one of the mosaic files and has ntiles = 1 from the staged files. I attached screenshots from the ncdump of the mosaic grid files.
Let me know if you find anything in the log files. I'll continue trying to look for the cause as well.
Thanks,
Jake

![grid mosaic ncdump halo3](https://github.com/user-attachments/assets/cca44b4f-b2f

c-4870-9f1c-53136fb16700)
warnfile.000000.out.txt

FV3LAM_wflow.log
get_extrn_ics_2019061518.log
get_extrn_lbcs_2019061518.log
make_ics_mem000_2019061518.log
make_lbcs_mem000_2019061518.log
run_fcst_mem000_2019061518.log
log.generate_FV3LAM_wflow.txt
logfile.000000.out.txt
launch_FV3LAM_wflow.txt
input.nml.txt
model_configure.txt
ufs.configure.txt

[jacobsmithsfd]

I rebuilt the entire SRW app using openmpi rather than MPICH and I am no longer running into this problem. I'll attribute it to an MPI issue failing to pass correct data.