Python 3.12 FutureWarnings

[ShannonShields-NOAA]

EMC's original testing showed that the upgrade from Python 3.10 to Python 3.12 resulted in a lot of “FutureWarnings,” which are identified during NCO and EIB EE2 Reviews. Here is an example from /gpfs/f6/drsa-precip3/world-shared/Shannon.Shields/verif_global_standalone_newss_202307_anom/tmp/verif_global.875900/batch_jobs/verif_global_grid2grid_step2.out on GaeaC6:

python /gpfs/f6/drsa-precip3/world-shared/Shannon.Shields/EMC_verif-global_step2_ss193/EMC_verif-global/ush/plots/grid2grid/grid2grid_plots.py
/gpfs/f6/drsa-precip3/world-shared/Shannon.Shields/EMC_verif-global_step2_ss193/EMC_verif-global/ush/plots/grid2grid/verif_global_util.py:1294: FutureWarning: Setting an item of incompatible dtype is deprecated and will raise an error in a future version of pandas. Value 'V12.0.1' has dtype incompatible with float64, please explicitly cast to a compatible dtype first.
  model_num_df.loc[(model_num_name, valid_date)] = (

This example is from the EMC_verif-global spack-stack 1.9.3 package (which uses Python 3.11), but this exact same FutureWarning also occurs in EVS when trying to use Python 3.12. In both verification packages, this FutureWarning pops up in the plotting step associated with the respective util.py script (verif_global_util.py for EMC_verif-global and subseasonal_util.py for subseasonal EVS). Please note that this FutureWarning is not isolated to the EVS subseasonal component, and occurs in other components of EVS.
It seems that the FutureWarning originates from the following line of code (EVS example):
https://github.com/ShannonShields-NOAA/EVS/blob/feature/subseasonal_python_test/ush/subseasonal/subseasonal_util.py#L4944
Jason started talking to NCO about accepting “FutureWarnings” log files. However, what else can we do about “FutureWarnings” in EVS v2.1 log files?
Method 1: Python script import warnings warnings.simplefilter(action='ignore', category=FutureWarning)
Method 2: Environment Variables export PYTHONWARNINGS=”ignore::FutureWarning”
George notes that this seems like a good option since it’d be a one-liner in the run script.
This topic was discussed in the METplus Users Telecon on Dec. 22, 2025.

[georgemccabe]

Hi @ShannonShields-NOAA, it looks like this warning is occurring because the value from the VERSION column, e.g. V12.0.1, is being added to the model_num_df DataFrame, which has been initialized with np.nan, so it assumes that all of the values set are of dtype float64.

Since the values in the MET stat output can be type str or float64, you will need to create the model_num_df DataFrame to allow other dtypes.

I think you should be able remove this warning by adding the dtype=object argument to the function call on lines 4902-4903:

Change this:

        model_num_df = pd.DataFrame(np.nan, index=model_num_df_index,
                                    columns=met_version_line_type_col_list)

to this:

        model_num_df = pd.DataFrame(np.nan, index=model_num_df_index,
                                    columns=met_version_line_type_col_list, dtype=object)

[ShannonShields-NOAA]

Thank you so much @georgemccabe ! That logic makes sense to me and Ho-Chun was able to confirm this fix in his EMC_verif-global development. Hopefully this should also fix the same warnings we are seeing in our EVS development with Python 3.12.

[ShannonShields-NOAA]

One more clarification @georgemccabe: Under the for loop https://github.com/ShannonShields-NOAA/EVS/blob/feature/subseasonal_python_test/ush/subseasonal/subseasonal_util.py#L4915, is setting the following still correct (no other changes needed) in the build_df function:

if col_idx < float_idx:
     df_dtype_dict[col] = str
else:
      df_dtype_dict[col] = np.float64

[georgemccabe]

Hi @ShannonShields-NOAA,

It looks like df_dtype_dict, which defines the data type for each column, is used to set the data types for model_stat_file_df.

See line 4923:

                model_stat_file_df = model_stat_file_df.astype(df_dtype_dict)

It is not used to set the data types for model_num_df, which was causing the future warnings. I can't say for certain if there are any future warnings caused by model_stat_file_df.

The code block you shared sets the data type to str if the column is before (left of) the TOTAL column and sets the data type to np.float64 for the TOTAL column and all columns after.
Question for @JohnHalleyGotway: Is it safe to assume that all columns after the TOTAL column are numeric values?

Looking at the Grid-Stat output information, it looks like the columns after TOTAL are typically some numeric value (integer or double), but that may not be true of all MET stat output files.

There are a few columns BEFORE the TOTAL column that are not strings, like INTERP_PNTS (integer) and ALPHA (double). While it is inaccurate that these values are not strings, I don't think interpreting these values as strings would cause the future warning.

I suppose this all depends on how this data is being used by the script. It is imperative that the data types are specified explicitly for each column? If not, you could consider settings the data type to object for all columns in model_stat_file_df:

                model_stat_file_df = model_stat_file_df.astype(df_dtype_dict)

I'm not familiar enough with this script to know the answer to that. This approach should resolve the future warnings. However, a quick search tells me that setting the data type to object can result in slower performance, higher memory use, and loss of specialized functionality for different data types.

It sounds like pandas has some functions that can handle data types intelligently, like convert_dtypes() and infer_objects(). I have not used these methods, but they may be a good alternative to trying to guess the correct data types in the script.

[ShannonShields-NOAA]

Ok, thank you @georgemccabe ! I believe the change to model_num_df as you originally suggested should address the FutureWarnings. EMC upper management is still discussing options with NCO about this topic regarding EVS. We greatly appreciate your assistance!

[JohnHalleyGotway]

@ShannonShields-NOAA, we chatted about this discussion during a METplus project meeting today. @georgemccabe pointed out that the you're assuming all the .stat columns of TOTAL (i.e. column 25) or greater, are numeric. That is true MOST of the time, but NOT ALL of the time.

For example, the OBS_SID and OBS_QC columns in the MPR line type, SEEPS_MPR line type, and ORANK line type are actually strings, not numbers. We actually updated the documentation for the last version to explicitly state the data type for each output column.

I'll also note that many of the Tropical Cyclone line types written by TC-Pairs also contain strings.

So, if you're reading MPR or ORANK .stat output or .tcst output from TC-Pairs, I'd recommend NOT assuming all columns are numeric.

[ShannonShields-NOAA]

Just wanted to provide an update on this discussion. For EVS subseasonal component, adding the following lines to the top (import section) of subseasonal_plots.py and subseasonal_util.py resolved the FutureWarnings seen in the log files:

import warnings
warnings.filterwarnings("ignore", message=".*dtype.*", category=FutureWarning)

[georgemccabe]

Hi @ShannonShields-NOAA, thanks for the update. Ignoring the warnings works for now, but I will warn (no pun intended) that this can catch up to you when you are forced to move to a newer version of Python or its packages and those warnings become errors. We have been dealing with the same issues upgrading to pandas 3 and Python 3.14 internally.

I will note that I recently created a GitHub issue dtcenter/MET#3384 to add a data type lookup table for MET output that will be used by METdataio (dtcenter/METdataio#416) to explicitly set the data types when reading MET output into Python. This enhancement can be applied to your scripts to help resolve these issues.