GridDiag wrapper output

[mpm-meto]

I'm trying to debug a GridDiag wrapper but the only error line I get is
03/04 13:55:19.002 metplus (command_builder.py:255) ERROR: (runtime_freq_wrapper.py:108) A problem occurred trying to obtain input files
03/04 13:55:19Z run-METplus-metplus: ERROR: (runtime_freq_wrapper.py:108) A problem occurred trying to obtain input files
03/04 13:55:19.002 metplus.GridDiag (met_util.py:181) ERROR: GridDiag had 1 error.
03/04 13:55:19Z run-METplus-metplus.GridDiag: ERROR: GridDiag had 1 error.

I am at a complete loss as to how to fix this. I have a sense it's because the format strings are wrong but there is no hint of what it is trying to construct given the info I provide so I have no idea which direction to try next. Most of the options/combinations I've tried do not have any effect.

What I can say is that GridDiag works as I've tried it without using the wrapper.

Forecast files are called 20190302T000Z_*.nc and obs files are called *_20190302.nc
Each of these files contains 24 fields for each hour of that day from 01 to 00Z the following day. The forecast is order by lead time from t+1 to t+24 (from a 00Z init). The observed files also contain 24 fields corresponding to the valid time.

The wrapper is set up as follows
LOOP_BY = INIT

The init time begin and end times, increment, and last init hour.

INIT_TIME_FMT = %Y%m%d%H
INIT_BEG = 2019030200
INIT_END = 2019030200

This is the step-size. Increment in seconds from the begin time to the end time

INIT_INCREMENT = 1H

LEAD_SEQ = begin_end_incr(1,24,1)

LOG_GRID_DIAG_VERBOSITY = 4

frequency to run the tool; valid options include:

RUN_ONCE, RUN_ONCE_PER_INIT_OR_VALID, RUN_ONCE_PER_LEAD, RUN_ONCE_FOR_EACH

GRID_DIAG_RUNTIME_FREQ = RUN_ONCE_PER_LEAD

GRID_DIAG_DESC = India_km4p4_RA1T

Configuration file

GRID_DIAG_CONFIG_FILE =GridDiag_joint.ltng

BOTH_VAR1_NAME = "number_of_lightning_flashes"
BOTH_VAR1_LEVELS = ({valid?fmt=%Y%m%d%H},,)
FCST_VAR1_OPTIONS = set_attr_name = "model_ltng"; set_attr_level = "sfc"; n_bins = 20; range = [0, 20];
OBS_VAR1_OPTIONS = set_attr_name = "obs_ltng"; set_attr_level = "sfc"; n_bins = 20; range = [0, 20];

FCST_GRID_DIAG_INPUT_DIR = {INPUT_BASE_FC}/forecast_sequential
OBS_GRID_DIAG_INPUT_DIR = {INPUT_BASE_OB}/OBS_on_FC_grid_total

GRID_DIAG_OUTPUT_DIR = {OUTPUT_BASE}/lightning

[filename_templates]

FCST_GRID_DIAG_INPUT_TEMPLATE = {init?fmt=%Y%m%d}T0000Z_model.nc
OBS_GRID_DIAG_INPUT_TEMPLATE = Gridded_obs_{valid?fmt=%Y%m%d}.nc

GRID_DIAG_OUTPUT_TEMPLATE = grid_diag_model_{valid?fmt=%Y%m%d%02H}.nc

I don't get to the point where it echoes the MET command that it tries to run as in this example from PCPcombine. I don't know if this line exists for GridDiag but it would be useful if it could be added? I could at least see what kind of filename it's trying to build...

03/03 15:34:25.542 metplus INFO: COMMAND: pcp_combine -v 5 -add /scratch/prods_op_gl-mn_20220201_00.nc 'name="stratiform_rainfall_amount"; level="(0,,),(1:,,)";' -name "stratiform_rainfall_amount" /scratch/METplus/output/met_tool_wrapper/PCPCombine/PCPCombine_subtract/global/prods_op_gl-mn_2022020100_f006_A06.nc

Referring to the grid_diag output I can get when not running with the wrapper, can I confirm that the final bin contains all data that are bigger than the range specified... the code gives a warning:
WARNING:
WARNING: process_series() -> the range of the model_ltng/sfc data (0, 23) falls outside the configuration file range (0, 20)!
WARNING:
WARNING:

Further on, there may be clues as to why the wrapper fails. The observation files were based on the model forecast.. and a dummy file was used to construct the metadata... is this the problem perhaps?
DEBUG 1: Default Config File: /data/users/cfrd/MET_requirements/MET10.1.0Beta6/share/met/config/GridDiagConfig_default
DEBUG 1: User Config File: tmp_GridDiag.cfg
DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new Met2dDataFile object of type "FileType_NcCF".
DEBUG 4: NcCfFile::open() -> parsing units for the time variable "hours since 1970-01-01 00:00:00"
DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time unit string "hours since 1970-01-01 00:00:00"
DEBUG 4: as a reference time of 19700101_000000 and 3600 second(s) per time step.
DEBUG 3: NcCfFile::open() -> read time from the bounds variable "time_bnds"
DEBUG 4: NcCfFile::open() -> parsing units for the forecast_reference_time variable "hours since 1970-01-01 00:00:00"
DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time unit string "hours since 1970-01-01 00:00:00"
DEBUG 4: as a reference time of 19700101_000000 and 3600 second(s) per time step.
DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new Met2dDataFile object of type "FileType_NcCF".
DEBUG 4: NcCfFile::open() -> parsing units for the time variable "hours since 1970-01-01"
DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time unit string "hours since 1970-01-01"
DEBUG 4: as a reference time of 19700101_000000 and 3600 second(s) per time step.
DEBUG 3: NcCfFile::open() -> read time from the bounds variable "time_bnds"
DEBUG 4: NcCfFile::open() -> parsing units for the forecast_reference_time variable "hours since 1970-01-01 00:00:00"
DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time unit string "hours since 1970-01-01 00:00:00"
DEBUG 4: as a reference time of 19700101_000000 and 3600 second(s) per time step.
DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo object of type "FileType_NcCF".
DEBUG 3: Parsed set_attr_name = "model_ltng"
DEBUG 3: Parsed set_attr_level = "sfc"
DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo object of type "FileType_NcCF".
DEBUG 3: Parsed set_attr_name = "obs_ltng"
DEBUG 3: Parsed set_attr_level = "sfc"
DEBUG 3: Grid Definition: Projection: Rotated Lat/Lon Nx: 1200 Ny: 1200 rot_lat_ll: -24.000 rot_lon_ll: 24.000 delta_rot_lat: 0.040 delta_rot_lon: 0.040 wrapLon: false true_lat_south_pole: -72.000 true_lon_south_pole: -85.000 aux_rotation: 0.000
DEBUG 2: Processing masking regions.
DEBUG 3: Including 1440000 of the 1440000 grid points in the analysis.
DEBUG 2: Initializing model_ltng/sfc histogram with 20 bins from 0 to 20.
DEBUG 2: Initializing obs_ltng/sfc histogram with 20 bins from 0 to 20.
DEBUG 2: Initializing model_ltng/sfc_obs_ltng/sfc joint histogram with 20 x 20 bins.
DEBUG 1: Processing 2 data fields from 1 input file(s).
DEBUG 2: Processing series entry 1 of 1.
DEBUG 2: Reading field model_ltng/sfc data from file: /data/users/frmm/india/lightning/forecast_sequential//20190302T0000Z_India_km4p4_RA1T_pverb.nc
DEBUG 3: Processing series entry 1: number_of_lightning_flashes(0,,)
DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new Met2dDataFile object of type "FileType_NcCF".
DEBUG 4: NcCfFile::open() -> parsing units for the time variable "hours since 1970-01-01 00:00:00"
DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time unit string "hours since 1970-01-01 00:00:00"
DEBUG 4: as a reference time of 19700101_000000 and 3600 second(s) per time step.
DEBUG 3: NcCfFile::open() -> read time from the bounds variable "time_bnds"
DEBUG 4: NcCfFile::open() -> parsing units for the forecast_reference_time variable "hours since 1970-01-01 00:00:00"
DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time unit string "hours since 1970-01-01 00:00:00"
DEBUG 4: as a reference time of 19700101_000000 and 3600 second(s) per time step.
DEBUG 4:
DEBUG 4: Grid information:
DEBUG 4: Projection: Rotated Lat/Lon
DEBUG 4: Nx: 1200
DEBUG 4: Ny: 1200
DEBUG 4: rot_lat_ll: -24.000
DEBUG 4: rot_lon_ll: 24.000
DEBUG 4: delta_rot_lat: 0.040
DEBUG 4: delta_rot_lon: 0.040
DEBUG 4: wrapLon: false
DEBUG 4: true_lat_south_pole: -72.000
DEBUG 4: true_lon_south_pole: -85.000
DEBUG 4: aux_rotation: 0.000
DEBUG 4:
DEBUG 4: Data plane information:
DEBUG 4: plane min: 0
DEBUG 4: plane max: 0
DEBUG 4: valid time: 20190302_010000
DEBUG 4: lead time: 010000
DEBUG 4: init time: 20190302_000000
DEBUG 4: accum time: 010000
DEBUG 4:
DEBUG 2: Reading field obs_ltng/sfc data from file: /data/users/frmm/india/lightning/OBS_on_FC_grid_total//Gridded_IITM_data_20190302.nc
DEBUG 3: Processing series entry 1: number_of_lightning_flashes(0,,)
DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new Met2dDataFile object of type "FileType_NcCF".
DEBUG 4: NcCfFile::open() -> parsing units for the time variable "hours since 1970-01-01"
DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time unit string "hours since 1970-01-01"
DEBUG 4: as a reference time of 19700101_000000 and 3600 second(s) per time step.
DEBUG 3: NcCfFile::open() -> read time from the bounds variable "time_bnds"
DEBUG 4: NcCfFile::open() -> parsing units for the forecast_reference_time variable "hours since 1970-01-01 00:00:00"
DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time unit string "hours since 1970-01-01 00:00:00"
DEBUG 4: as a reference time of 19700101_000000 and 3600 second(s) per time step.
DEBUG 4:
DEBUG 4: Grid information:
DEBUG 4: Projection: Rotated Lat/Lon
DEBUG 4: Nx: 1200
DEBUG 4: Ny: 1200
DEBUG 4: rot_lat_ll: -24.000
DEBUG 4: rot_lon_ll: 24.000
DEBUG 4: delta_rot_lat: 0.040
DEBUG 4: delta_rot_lon: 0.040
DEBUG 4: wrapLon: false
DEBUG 4: true_lat_south_pole: -72.000
DEBUG 4: true_lon_south_pole: -85.000
DEBUG 4: aux_rotation: 0.000
DEBUG 4:
DEBUG 4: Data plane information:
DEBUG 4: plane min: 0
DEBUG 4: plane max: 6
DEBUG 4: valid time: 20190302_010000
DEBUG 4: lead time: -4550000
DEBUG 4: init time: 20190321_000000
DEBUG 4: accum time: 010000
DEBUG 4:
DEBUG 2: Processed model_ltng/sfc data with range (0, 0) into bins with range (0, 20).
DEBUG 2: Processed obs_ltng/sfc data with range (0, 6) into bins with range (0, 20).
DEBUG 1: Output file: /scratch/frmm/MET/output/grid_diag/lightning//grid_diag_India_km4p4_RA1T_2019030200.nc

One might need (or want) to override some or all of the time info in the file and use the time info in the filename instead. Here I want to ignore the date but keep the hour. Clearly the MET tool doesn't care, but the METplus wrapper does.

[georgemccabe]

Hi @mpm-meto,

GridDiag wrapper does not read separate FCST and OBS data like other MET tools do. The config variable to set to read in files is GRID_DIAG_INPUT_TEMPLATE, not FCST_GRID_DIAG_INPUT_TEMPLATE and OBS_GRID_DIAG_INPUT_TEMPLATE. Setting LOG_LEVEL = DEBUG in your user config file may have given you a hint as to why you could not find any files, but I did notice that there was no check if GRID_DIAG_INPUT_TEMPLATE is not set to report a useful error message. I just added that so it will be available for the upcoming 4.1.0 release. This should make it more clear why it couldn't find any files.

You will want to pass in the templates to read as a list:

GRID_DIAG_INPUT_TEMPLATE = {INPUT_BASE_FC}/forecast_sequential/{init?fmt=%Y%m%d}T0000Z_model.nc, {INPUT_BASE_OB}/OBS_on_FC_grid_total/Gridded_obs_{valid?fmt=%Y%m%d}.nc

If there was a common parent directory for all of the file templates, you could set that in GRID_DIAG_INPUT_DIR.

Some other things I noticed about your configuration that I should mention:

1. Since GridDiag does not read FCST and OBS separately, then the value you set for OBS_VAR1_OPTIONS will not be used. You will want to only set FCST_ or BOTH_ variables for this tool.

2. Since you are using GRID_DIAG_RUNTIME_FREQ = RUN_ONCE_PER_LEAD you should only reference 'lead' in the output filename template. The wildcard character '*' will be substituted for {valid?fmt=%Y%m%d%02H} (it should actually replace '*' with 'all' in the output file path but there was a bug in the logic that I fixed today). RUN_ONCE_PER_LEAD is used to run once for each forecast lead processing many init/valid times for each. The init/valid time is not actually known in this case, since there could be many different times.

Since your use case is only running a single init time and 24 forecast leads, you should use:

GRID_DIAG_RUNTIME_FREQ = RUN_ONCE_FOR_EACH

The valid time will be computed properly for each init/lead combination.

[mpm-meto]

Thanks @georgemccabe for the detailed response! I will go away and try that. I was being too clever I think in thinking it would take FCST and OBS separately. Referring to your comment on the OPTIONS bit, I just want to try to understand how you would translate the following dictionary entry into METplus. Both are model variables, but not the same. Would it be a case of specifying FCST_VAR1_OPTIONS and FCST_VAR2_OPTIONS? You could do the same using observations. In that case calling them FCST_VAR1 and FCST_VAR2 may be somewhat misleading but if this is the way to get it to work, then there is a way forward.
data = {
field = [
{
name = "number_of_lightning_flashes";
level = [ "(23,,)" ];
n_bins = 20;
range = [0, 20];
set_attr_name = "model_ltng";
set_attr_level = "sfc";
} ,
{
name = "stratiform_rainfall_amount";
level = ["(23,,)"];
n_bins = 20;
range = [0, 100];
set_attr_name = "model_precip";
set_attr_level = "sfc";
}
];
}
Previously we worked out that if the names are the same, that the attr_name needs to be set to differentiate between the two comparators, i.e. when the two fields are identical without having different attr_name fields (as per the example in the original discussion). Even in this instance invoking BOTH would not work. Right?

[georgemccabe]

In GridDiag wrapper, only one data source is provided to the tool and the METplus config variable identifier used is FCST_. Setting the BOTH_ variables essentially set the same values for both FCST_ and OBS_ variables, but the OBS_ variables are ignored by GridDiag. This means that FCST_ and BOTH_ can be used interchangeably to define field info to be read by GridDiag.

In your example, there are 2 fields defined in the data.field list. The first is defined with VAR1 and the 2nd is defined with VAR2.

Field 1:

{
name = "number_of_lightning_flashes";
level = [ "(23,*,*)" ];
n_bins = 20;
range = [0, 20];
set_attr_name = "model_ltng";
set_attr_level = "sfc";
}

is translated to:

FCST_VAR1_NAME = number_of_lightning_flashes
FCST_VAR1_LEVELS = "(23,\*,\*)"
FCST_VAR1_OPTIONS = n_bins = 20; range = [0, 20]; set_attr_name = "model_ltng"; set_attr_level = "sfc";

and Field 2:

{
name = "stratiform_rainfall_amount";
level = ["(23,*,*)"];
n_bins = 20;
range = [0, 100];
set_attr_name = "model_precip";
set_attr_level = "sfc";
}

is translated to:

FCST_VAR1_NAME = stratiform_rainfall_amount
FCST_VAR1_LEVELS = "(23,\*,\*)"
FCST_VAR1_OPTIONS = n_bins = 20; range = [0, 100]; set_attr_name = "model_precip"; set_attr_level = "sfc";

If the field names are the same, then I believe you will have to add them as a separate VAR using a different value for set_attr_name so that they produce 2 different output fields.

[georgemccabe]

Hi @mpm-meto, I am just following up on this discussion to see if your question has been answered or if you need additional assistance.

[georgemccabe]

Hi @mpm-meto, I am going to mark this as answered and lock the conversation to prevent future posts. Please create a new discussion topic if you need additional assistance with this tool.