Feature 2963 fire wx use cases by CPKalb · Pull Request #3164 · dtcenter/METplus

CPKalb · 2025-11-26T22:21:44Z

Pull Request Testing

Describe testing already performed for these changes:

Have tested the use cases to verify that they work and are producing the expected output in GHA. Also have verified that the documentation builds successfully
Recommend testing for the reviewer(s) to perform, including the location of input datasets, and any additional instructions:

Review the documentation for these use cases. There was also a small change made to the contributors guide with instructions when adding data for more than one use case. Those changes can be found at the links below:
MODEMultivar_fcstHRRR_fcstOnly_CreekFire
MTD_fcstWRF_obsMMA_416Fire
PointStat_fcstHRRR_obsMADIS_FrazierFire
Contributors Guide Change

Also, make sure the use cases run successfully in GHA. For the 3 new fire weather use cases (MODEMultivar_fcstHRRR_fcstOnly_CreekFire.conf, MTD_fcstWRF_obsMMA_416Fire.conf, and PointStat_fcstHRRR_obsMADIS_FrazierFire.conf), there should now be new output truth data and no other errors. For the 2 ML use cases (medium_range/PointStat_fcstCREDIT_GFS_obsGDAS_6hrRealtime.conf and medium_range/GridStat_fcstCREDIT_GFS_obsGFS_6hrRealtime.conf), the output data should have moved locations but there should be no other errors.

Do these changes include sufficient documentation updates, ensuring that no errors or warnings exist in the build of the documentation? [Yes or No]
Yes
Do these changes include sufficient testing updates? [Yes or No]
Yes
Will this PR result in changes to the test suite? [Yes or No]

If yes, describe the new output and/or changes to the existing output:

Yes. There will be new output for the 3 new fire weather use cases. In addition, the output for two other use cases has changed (the paths are different). These paths were changed because it was found that they were inconsistent with the documentation for the use case.
Do these changes introduce new SonarQube findings? [Yes or No]

If yes, please describe:
Yes. I expect that SonarQube will flag the reformatting Python script, reformat_linetype.py, because it is too similar to other use cases that also have reformatting and plotting scripts. We have issue 2851 to address this, but the path forward will be decided when we determine exactly what direction we are taking with the METplus Analysis Tools
Please complete this pull request review by [Fill in date].

12/8/2025

Pull Request Checklist

See the METplus Workflow for details.

Add any new Python packages to the METplus Components Python Requirements table.
For any new datasets, an entry to the METplus Verification Datasets Guide.
Review the source issue metadata (required labels, projects, and milestone).
Complete the PR definition above.
Ensure the PR title matches the feature or bugfix branch name.
Define the PR metadata, as permissions allow.
Select: Reviewer(s) and Development issue
Select: Milestone as the version that will include these changes
Select: Coordinated METplus-X.Y Support project for bugfix releases or METplus-Wrappers-X.Y.Z Development project for official releases
After submitting the PR, select the ⚙️ icon in the Development section of the right hand sidebar. Search for the issue that this PR will close and select it, if it is not already selected.
After the PR is approved, merge your changes. If permissions do not allow this, request that the reviewer do the merge.
Close the linked issue and delete your feature or bugfix branch from GitHub.

j-opatz

Reviewed documentation and completed a push for some spelling, sentence structure.

Use cases look good, GHA show failures from expected points (new output, new path to data, etc.).

* update version for development toward next release first beta * Updating RC1 release instructions for MET to create the vX.Y input test data directory prior to creating the main_vX.Y-ref branch. * prevent error when input version requested does not exist in lookup table * handle if input version is develop * Feature 2741 update use cases (#2795) * updating template first pass * removing backslash * removing extra # symbol * removing spaces. Keep as is * removing spaces for consistency * adding changes to match template * fixing spacing * fixing spacing take 2 * Per #2741, correcting typo in OUPUT_BASE in template file * Per #2741, correcting typo in OUPUT_BASE * Per #2741, updating sections based on template and template description * Updated language in several sections, needs additional work on output and config file * updating for consistency * Updating file based on suggestions from George * Updating with suggestions from George * Updating file based on suggestions from George * reviewing with Julie * Per #2741 attempting to resolve errors * Per #2741, updating template to fix rendering problems * Per #2741, fixing formatting * Per #2741 attempting to resolve warnings * Per #2741, resolving error * Resolving formatting problems with template * Fixing formatting problems * updating for consistency * fixing formatting * removing extra spaces * Updating formatting * updating for consistency * adding METplus Workflow times * modifications * Update GridStat_fcstCESM_obsGFS_ConusTemp.py Added some information to the use case * Update MODE_fcstCESM_obsGPCP_AsianMonsoonPrecip.py Updates * Added an extra blank line * updating for consistency * fixing spacing * adding space * removing Update content * updating for consistency * adding a period * updating for consistency * updating for consistency * updating for consistency * updating for consistency * adding return * replacing underlines * replacing return? * fixing blank line * more attempts to fix warnings * updating for consistency * adding double colons and stuff * trying stuff * trying extra spacing * Changing Datasets Location info to match template * Per #2741, updating formatting and wording * Per #2741, updating after reviewing * Per #2741, attempting to resolve errors * updating for consistency * fixing brackets and section breaks * adding return * Per #2741, updating after reviewing * updating for consistency * fixing spacing * adding a pound * Updating wording for User Scripting * Updating the text for User Scripting * Updating User Script section * Updating text for User Scripting * Updating text * adding location information * adding location information * updated first cloud file, will use for remaining cloud use cases * updated language in numerous use case doc files * updated final marine use case --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> Co-authored-by: Christina Kalb <kalb@ucar.edu> Co-authored-by: j-opatz <jopatz@ucar.edu> * fix typo * update timeline to include actual date of rc1 release * added missing script to create mp_analysis conda environment for use case tests * add mp_analysis conda env to workflow to create conda environments for DockerHub * Feature 2741 running metplus (#2801) * marine_and_cryosphere directories * fixing underlining * trying to fix warning msg. * trying to fix warning msg. again * Removing space before i.e. * updating medium_range and pbl directories * deleting empty file * removing lots of old text * precipitation files * removing junk file * removing extra # * add return * updating 3 dirs s2s, space_weather and unstructured_grids * updating short_range files * adding tc_and_extra_tc files * removing defunct -c option * Updating Datasets section * Added needed return character before "#Pass" * Added /path/to/user_system.conf to end of run_metplus.py line * Added a comma after "command line" * Updating file path * Added fields for empty Satasets section --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Updated HERA information * Resolving error * Updated jet and hera * prevent error in the command to pull the changelog file when it does not exist yet * Feature 2751 develop release updates (#2798) * Adding period to end of sentence. * Release Guide (develop) - updating existing builds (#2806) * update existing builds for upcoming coordinated 6.1 (beta1) release * added template for existing builds for next dev cycle and added it to ignore list * Update release guide instructions to set the new location of the Existing Builds page on the Downloads page. Update instructions for creating a new existing builds page to instead copy the template and replace values in the develop branch * added a step to update the development timeline info for an official release * Updating existing_builds files * changed Docker Hub to DockerHub * added step to update the DockerHub info on the Existing Builds page * change version to official release because RAL-IT doesn't install development releases on RAL machines --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * prevent error when merge commit messages are too long * prevent error when change summary text contains backticks * Update METplus-6.0.0 release date from 12/11/24 to 12/18/24. * Feature 2741 dropdowns (#2808) * marine_and_cryosphere directories * fixing underlining * trying to fix warning msg. * trying to fix warning msg. again * Removing space before i.e. * updating medium_range and pbl directories * deleting empty file * removing lots of old text * precipitation files * removing junk file * removing extra # * add return * updating 3 dirs s2s, space_weather and unstructured_grids * updating short_range files * adding tc_and_extra_tc files * removing defunct -c option * added dropdown menus for marine_and_cryosphere * fixing problems and dropdowns for medium_range * fixing new problems * dropdowns for pbl and precip * adding double colons * Attempting to resolve errors * dropdowns for s2s and s2s_mjo * adding dropdowns for short_range files * final dropdown directories * Update docs/use_cases/model_applications/marine_and_cryosphere/GridStat_fcstRTOFS_obsGHRSST_climWOA_sst.py removing note Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Apply suggestions from code review adding all of the suggestions Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Update GridStat_fcstRTOFS_obsGHRSST_climWOA_sst.py ensuring literal include is in dropdown menu * Update GridStat_fcstRTOFS_obsOSTIA_iceCover.py ensuring literal include is in dropdown * ensuring literal include is in dropdown menu * marine_and_cryosphere MET Configuration dropdown fixes * remaining dropdown directories for MET Configuration * Julie changing dropdowns Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * HEAD needed to be removed * There are not MET tools used in this use case * There are no MET tools used in this use case * Apply suggestions from code review Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Per #2741, few small changes * updates from pull request discussion * removing space Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * fixing dropdown Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * removing bash Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * update from Julie Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Fixing error * lots of typos Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Adding period * Adding colon * Added period * Added period * Added period * typos and more Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Feature #511 lists in command line single config overrides (#2815) * exclude metplus/scripts dir from PyCharm project to prevent incorrect duplicate code warnings with files that are sym linked * to expand on #2772, updating instructions to include note to set tmp directory for apptainer to prevent issues pulling large images that require a lot of temp space * per #511, add unit test for expected behavior to support comma-separated lists in a command line single config override that should fail until fix is made * add a test to ensure that the -c argument is properly ignored since it is can be used in old use cases that were created when the argument was required * per #511, add support for command line single config overrides to include values that are lists. Simplify logic to parse arguments to strip out -c/--config/-config arguments and skip check/error if argument is invalid because it is already handled in the metplus_config setup step that parses the arguments * added unit test to ensure that an invalid command line argument causes the appropriate failure from run_metplus.py * Feature metplus 2780 dedication (#2819) * adding In Memoriam section * Changing to "This Coordinated Release is dedicated to" removing version * adding links * fixing Randy's link * Per #2780, change case of words and add comma after year --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Feature #2816 enhance workflow dispatch testing (#2818) * per #2816, update title of workflow displayed in Actions tab to display the repo that triggered the workflow on workflow_dispatch events. Preserve the behavior of pull_request and push events by displaying the PR title or head commit message respectively * fix logic for run-name * Update event info to note that the workflow was triggered manually if the repo/title was not provided. Update description of workflow_dispatch argument to describe what it can be used for when triggering manually. Only run step to build URL to commit that triggered workflow if it was triggered from an external repo * add optional argument for workflow_dispatch to specify the title of the workflow run * revert change to repository argument description since title argument was added to handle custom workflow run titles * Feature #2814 develop - doc single config overrides (#2825) * per #2814, add documentation to describe how to set a single config variable on the command line * fixed typos and formatting issues * move new content to top of chapter and reword intro sentence * change label and add link to user env var section * per #2814, added info and examples for single config overrides with a list of values * Sorts the list of FLUXNET csv files returned from glob. (#2836) * Bugfix #2830 develop fix missing log output (#2841) * Per #2830, skip closing of log handlers for METplusConfig objects that are created for copying values for process list instances so they are not closed before the end of the run. * remove some output directories after tests are run * Updated METexpress versions in component_versions.py (#2842) * added script to regroup release notes for official release by parsing dev release notes. improve naming of drop downs for generating dev release notes to be consistent * improve script to handle different formatting for categories with no issues * per suggestion from @bikegeek, output 'None' if there were no items under a category * reorder instructions to follow more natural progression * update instructions for updating ReadTheDocs based on the changes to the RTD web interface * add next coordinated release to version lookup table * move instructions to create directory for data for new dev cycle on DTC web server to be done for the rc1 release instead of after the official release * update versions of packages that have vulnerabilities * Feature #2586 GenVxMask improvements (#2833) * resolve some SonarQube complaints * per #2586, added function with tests to properly parse list of command line arguments that can now contain comma-separated lists that should not be split up into separate items * add support for {app}_{data_type}_FILE_WINDOW_BEGIN/END, e.g. GEN_VX_MASK_OBS_FILE_WINDOW_BEGIN. This just adds support for an additional variation of the config variable names * add support for an empty label for input templates * update wrapper to be consistent with other wrappers wrt finding input files, progress towards #2492. Allow file window range to be specified separately for mask and input files. Other cleanup to move towards consistent wrappers with fewer wrapper-specific overrides of functions like get_command * update unit tests to align with changes for #2492 * add documentation for config variables that are newly supported to allow file window range to be specified separately for mask and input files * renamed GEN_VX_MASK_OBS variables to be GEN_VX_MASK_INPUT as suggested by @JohnHalleyGotway in PR review * fix logic to properly read input files by handling inputs that support multiple inputs with labels (used by GridDiag and UserScript wrappers) and typical inputs (all other wrappers). Prior to this change only input templates that have the FCST or OBS identifier were read properly via get_input_templates * Feature 2628 documentation updates (#2817) * Feature #2844 release_guide_dropdowns (#2847) * Work in progress adding instructions dropdowns to the release guide * Per #2844, more dropdowns * Per #2844, add instruction dropdowns for all MET instructions * Per #2844, more dropdowns. * Per #2844, use instruction dropdowns throughout * Per #2844, remove METexpress AWS instruction since its no longer relevant. * Remove accidentally committed .DS_Store file. * Update the MET Release Guide instructions about the dependent library tar files. * Minor formatting change * update URL for posting sample data * Doc-only change to rename the 'Code support' section as 'User support' based on 1/9/2025 decision at the METplus-Analysis Tools project meeting * fixed typo * Add 2 new 'alert' labels to flag changes that modify config options and/or output formats. * Update the common labels by adding the 'pull request:' prefix, one new reporting label and update scripts to run /bin/bash instead of /bin/sh, which now points to dash on seneca. * Feature #2827 SeriesAnalysis gradient (#2835) * Add 'requestor: NOAA/NWS' label for use in METplus Discussions. * Feature #2758 SonarQube (1) (#2866) * test that this file is not needed anymore because RTD controls the version selector * various changes to appease SonarQube complaints * remove deprecated docker command * resolve more SonarQube code smells * update URL for posting sample data * Add note to the Release and Contributor's Guide pointing readers to the develop branch content. * Updating common_labels.txt in the METplus develop branch * Update the details of the 'pull request' labels. * Feature #2781 PairStat wrapper (#2872) * update release guide for METplus to create data location for next release on mohawk when creating rc1 release instead of after official release * cleanup * per #2781, start first implementation of PairStat wrapper * appease SonarQube by increasing code coverage for component versions script * add new wrapper to lookup dictionary * per #2781 add wrapped MET config file * use runtime freq logic to find input files * start unit tests (broken) * reorder assert to align expected and actual values with PyCharm notation * update unit tests * rename field info variable 'field' to 'pairs' to match Pair-Stat naming convention. Add command line arguments to pair_stat command * reorder assert to align actual and expected in PyCharm * refactor function to get start/end times to prevent error when using INIT/VALID_LIST and use time_generator to be more consistent and reduce duplicate code * change many wrappers to be consistent with finding input files. Errors are now thrown when any input file is not found, checking other input types even when another input was not found. This increased errors reported in unit tests, so updated tests to reflect this. * remove line that is not needed * fix bug introduced with recent changes and run use cases that failed to test that the fix is correct * fix multi-variate MODE run to pass all fields to the call instead of just the first one * update SeriesAnalysis test to use logic that is actually used when running * turn off use cases that now succeed and turn on use case to test fix to multi-variate mode * turn off use case after confirming that it now runs successfully * refactor logic to satisfy SonarQube to reduce cognitive complexity and remove unused imports * remove functions that are not used * handle -pairs flag like other command line args for input files. Add documentation to new functions * updates to new wrapper, add documentation, basic use case stubs. create function for duplicate code to handle land_mask and topo_mask in PointStat and PairStat wrappers * remove unused import * remove config variable that is no longer used in MET * remove match_month support from climo_mean/stdev * remove obs_valid_beg/end from command line args because it is not supported by pair_stat * remove test config files that are not being used * configure basic use case to run command that is being used to test/develop the app * error if invalid tool name is provided * add missing tests * remove support for config variables that did not make it into the final implementation of the MET pair_stat tool * fix formatting on a few config values * correct input file info * turn on use case to test * fix command line argument for output directory that changed since the last test * refactor and clean up to reduce SonarQube issues * update use case to prevent redundant runs of SeriesAnalysis(run_two) by only setting custom loop list for the first instance of SeriesAnalysis * Fix bug that prevents correct field information from being set when multiple fields are read from the same file. Removed unused function * per feedback in PR #2872, remove support for setting output_prefix * add PairStat to list of wrappers that use the MODEL config variable * per feedback in PR #2872, modify wrapper behavior to change output path from a directory to a filename base that will be used to write output files with various extensions, e.g. _mpr.txt or .stat * remove output_prefix from tables * regroup met_tool_wrapper use cases * Feature #2877 v6.1.0-beta1 release (#2878) * update version for beta1 release * add release notes for beta1 * update script to help generate release notes to add a set of issues to be marked as pass because no extenral testing required, removed SonarQube complaints about duplicate string literals * added missing issue * rename 'Internal' category to 'Build, repository, and test' to be consistent with MET * update version for development towards beta2 * fix list * Feature #2880 SeriesAnalysis allow time templates in field name (#2881) * per #2880, allow filename template tags to be substituted in the fcst/obs name -- previously it was only supported in the level value * remove support for running SeriesAnalysis wrapper with RUNTIME_FREQ=RUN_ONCE_FOR_EACH because this will not produce any useful output. The default setting of RUN_ONCE_PER_INIT_OR_VALID will be used instead * Update version v12.0.1 MET (#2884) * add support for overriding METplotpy, METcalcpy, and METdataio branch for testing workflow * Update version 12.0.2 MET (#2896) * use develop branch version of the component versions script to determine version numbers for creating docker images for releases * updated configuration (#2901) * fixed typo * Feature 2891 develop existing builds modulefiles (#2909) * Per #2891, updated for 6.1.0 release with coming soon messages added * Per #2891, updating all modulefiles for 6.1.0 * Per #2891, fixing formatting * Feature 2741 datsets and workflow (#2839) * new branch testing * marine and cryosphere datasets and workflow * medium_range datasets and workflow * pbl and precep files for datasets and workflow * s2s_mjo and s2s files datasets and workflow * Per #2741, updated METplus Workflow section * first 5 short_range files * Per #2741, rearranged and modified information for consistency. * short_range directories starting with m or p * short_range/UserScript files * space_weather files * tc_and_extra_tc and unstructured_grids files * removing extra # * fixes Julie suggested * more fixes and new updates * second pass of short_range files * last directories clean up * Tidy up UGRID use case doc file. * Minor changes to tc_and_extra_tc docs. * Minor changes to space_weather docs. * Minor changes to short_range docs. * Minor changes to s2s_mjo docs. * Minor changes to s2s docs. * Minor changes to precipitation docs. * Minor changes to pbl docs. * Minor changes to medium_range docs. * Minor changes to marine_and_cryo docs. * Clarifying on climo dataset. * Adds INIT_END and INIT_INCREMENT to workflow. * Removes unused conf item. * Adds other timing control elements for completeness. * Removes unused conf item. * Removes unused conf item and switches min/max for readability. * Adds additional timing control conf items for completeness. * INIT_END is not set so changed to None. * Removes 2014 for VALID_BEG because it actually uses data from 2016, and updated the information to reflect that. * Adds other timing control for consistency. * Adds other timing control for consistency. * Adds other timing control for consistency. * Clarifies that despite LEAD_SEQ not being set, it does use multiple forecast leads because it uses output from another use case. * Clean up CyclonePlotter METplus Workflow. * Adds clarity about the 60h lead time. * Adds the best description of timing control I could think of, since it's a bit confusing for the TC- tools. * Adds timing info for more clarity and consistency with template. * Per #2741, correcting formatting --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> Co-authored-by: Daniel Adriaansen <dadriaan@ucar.edu> * add run_metplus.py to bin directory upon install * Revert "add run_metplus.py to bin directory upon install" This reverts commit d762570a803e4c8ed87c4ead5630a2a6c4967638. * Feature #2697 Python 3.12.0 - using Python 3.10 still (#2918) * Per #2697, update conda environment creation scripts to create v6.1 environments that use Python 3.12.0. Specified python package versions to match what was used to create METbaseimage, but left the rest of the packages versionless to allow mamba/conda to install the desired versions. Once envs have successfully been created, I will need to go back and add the version numbers that were used to make sure future envs will be created consistently * change how cartopy feature download script is called -- see https://github.com/SciTools/cartopy/pull/2263 * get older set of tar_files that includes jasper version needed to install gfdl tracker * revert back to using py3.10.4 for spacetime environment because pyngl is not supported in py3.12.0 * remove args now that SSL cert for dtcenter site is renewed * Per #2697, update version of conda environments to use envs created using python 3.12.0 * added scipy to python embedding environment because it is now a requirement of MET Python Embedding * remove deprecated import * update swpc env script * add option to specify branch to use for METplotpy/METcalcpy/METdataio * add workflow dispatch inputs to override METplotpy/METcalcpy/METdataio branch for use case testing, changed default values to empty string to determine develop or main_vX.Y * simplify METplus Analysis branch override * do not pull METplus image from DockerHub before building to see if it will properly pull the latest changes from MET that are not being incorporated in the METplus image * fix warnings in python script * fix error using numpy 2.X * added multi-variate mode quick search keyword so use case is more discoverable * updated config settings with new variables * add back pull and use cache of previously generated docker image since I determined it was not the cause of the MET content not being updated properly * revert back override of MET image * changes to appease SonarQube * removed duplicate entries * rearrange args to appease SonarQube * Updating METplus-6.1.0 Development Timeline * Feature #2923 SonarQube for 6.1.0-beta2 (#2924) * fix sonarqube findings * update test to ensure it fails as expected even if the OUTPUT_BASE is set in the defaults * force https protocol to satisfy SQ * copy explicit files/dirs into Docker image instead of using a wildcard to satisfy SonarQube -- tested that METplus unit tests can still run successfully instead a Docker container that was created using the copy method * prevent future warning * Feature 2744 nrl gfs goes (#2926) * Use case config file from @briannen. * Documentation template for this use case. * Changes GRIBv2 record numbers to match subsetted forecast files. * Updates to add new use case to automation. * Updates input locations to the correct place where sample data are stored. * Adds use case gallery image. * Removes apostrophe from Point2GridConfig_wrapped MET config file and adds literalinclude of this file to the Point2Grid met_tool_wrapper use case documentation. * Use case documentation file. * Updates to remove using grib record numbers because they could vary, which necessitated using three FCST/OBS vars for each cloud level. Also used CUSTOM_LOOP_LIST to process both goes16 ans goes18 for each run, and switched to only a single forecast for demonstration purposes. * Changes docker attempts to 20 instead of 10. * Minor tweaks to docs from PR review. * Feature #2697 python 3.12 round 2 (#2930) * fix unit test to work with python 3.12 and 3.10 * per #2697, update python version to 3.12 for GHA workflows, create python requirements files for 3.12 while preserving 3.10 versions to allow for testing of multiple versions * update recommended python version to 3.12.0 * per #2697, run unit tests using both py3.10 and py3.12 * add missing comma * add creation of conda env used for development for METplus wrappers (includes METplus wrappers requirements, unit test reqs, and documentation reqs) * use scripts for python ersions to ensure 3.10 is not truncated to 3.1 * update version of python to use for RTD to 3.12 * added version numbers for the python package requirements that were used when they were created without specifying versions to ensure that the same environment will be created if the env is regenerated * added instructions to create the metplus_dev environment so it can easily be created locally * update required python for install * update conda environments for use case tests to use versions that were created with python 3.12 * disable spacetime use case because pyngl is not supported in python 3.12 * change MJO use cases to use mp_analysis conda env instead of spacetime * update python package versions * install METplotpy and METcalcpy automatically if mp_analysis environment is used * use conda to clone an existing conda env because the --clone argument doesn't appear to work using a newer version of mamba (v2.0) * Feature 2745 nrl gfs asos (#2922) * Add NRL GFS-ASOS use case * Moves use case from met_tool_wrapper to model_applications and renames use case conf file and directoy. * Latest versions to use. * Changes variable names to have stn_ prefix since latitude and longitude were being overwritten with zeros, and also multiples cloud fraction by 100 to make it a percentage to match the forecast. * Adds checking to see if it's a fraction variable or not before converting to percent (assuming the other variable is altitude), and simplifies the height/level used so that observation is a single value from the obs file. * Scripts to read GFS data with Python embedding and interpolate vertically from cloud base pressure to cloud base height. * Latest changes for Python embedding of forecast data to compare cloud base height with ASOS, add MPR output, and shift around various settings to the config subgroups. * Removes INIT_INCREMENT because only one INIT time is being processed. * Turns off MPR output. * Adds reading of topography from the model to convert the geopotential height field from meters MSL to meters AGL to compare with observations. * Adds valid time to obs filename so it dynamically selects the correct date for the input file. * Adds use case documentation gallery image. * Renames gallery image filename and updates path in docs file. * Changes to run for a single lead time, and removes python embedding for the forecast for low_cloud_base_altitude in favor of the ceiling field from GFS. Based on comments in NOAA's UPP code, ceiling is the cloud base height for cloud fraction > 50%, and this variable is from the local surface rather than from mean sea level. If no ceiling is found a value of 20,000m is used. * Fixes incorrect quotes in the use case documentation template for the gallery image path. * Fixes incorrect quotes in sphinx gallery image file path. * Switches back to Python embedding for the low_cloud_base_altitude since bonafide ceiling was not available in the METAR obs file we are using. * GHA/CI integration after adding sample data. * Updates to input dirs. * Correct input directory for obs. * Avoids namespace conflict with standard conf item. * Adjusts documentation for correct forecast leads. * Removes MET_PYTHON_EXE from config. * Removes verbosity settings from testing. * Removes setting OBTYPE. * Adds more comments to interpolation function. * Adds many comments about read_asos_ceil.py. * Adds info about the third Python embedding script gfs_025_interp_funcs.py. --------- Co-authored-by: Tracy Hertneky <hertneky@ucar.edu> Co-authored-by: j-opatz <jopatz@ucar.edu> * fix use case script to work with cfgrib 0.9.11.0 and 0.9.15.0 using changes from @DanielAdriaansen * update instructions to download cartopy static files * Feature #2863 initial conda recipe (#2900) * initial implementation of conda recipe to install MET executables and METplus wrappers * add setup tools to fix incorrect METplus wrappers on install * prevent conflict when isatty is defined in multiple places * update version of MET to 12.0.1 to include pb2nc bugfix * remove specific versions of compilers that are no longer needed * use printf to properly add line break * turn off fortran optimization to prevent issues with NHC-provided code * remove setting z lib flag because it shouldn't be necessary to compile MET * remove fortran flags that are not needed * skip if attempting to build on windows * explicitly use gcc 12 for linux because atlas fails to compile using gcc 14. Clean up comments to be more clear why dependencies are needed * update website URL and change maintainer to GitHub username * prep for bugfix release * update MET v12.0.2 hashes * remove unused env vars and sym link of ar * add back env vars that are needed, clean up * set env var to easily access the parm directory for use case files * specify version for libopenblas because latest version 0.3.29 caused eckit to not find the library and created additional library dependencies that were not listed in the recipe * added more about info and changed maintainers item to correct name * update license info * add tests to ensure MET executables and required python imports are available * clean up comments so they can be easily removed when submitting to conda-forge * formatted changes based on conda-forge linter * add cmake because it was missing from conda-forge build * specify different dynamic library name for netcdf C++ to match what is available for osx-64 * change specific gcc compiler version line * revert change to try to handle differing netcdf c++4 dynamic library names * change wget to curl because wget is not available in conda-forge env * formatting for conda forge linter * skip stdlib c for osx-arm64 to prevent error, get latest netcdf-cxx4 to fix issue with lib naming on osx-64, specify python version as 3.10 * add license files for library dependencies * specify linux compiler versions * fixed typo in fortran version spec * add make and pip to build reqs * fix c stdlib logic by specifying versions and names, move pip to host reqs, capitalize True for skip windows * add install of run_metplus.py in path * clean up comments to be more accurate * changed format of exact version requirements * add instructions to build/test a conda recipe * set parm base based on python version * specify python version and numpy requirements * changes suggested by @JohnHalleyGotway in the pull request review * made formatting fixes suggested by @jprestop in the PR review * per suggestion from @jprestop code review, moved Conda Recipe chapter up to be chapter 3 * update c_stdlib verisons to match https://github.com/conda-forge/conda-forge-pinning-feedstock/blob/main/recipe/conda_build_config.yaml * updates based on feedback from conda-forge PR * get eckit/atlas from metview on conda-forge instead of installing it * change tests back to checking if exes exist because they return 1 when run without arguments and cause a failure * add description * Feature 2743 nrl gfs cloud analysis (#2850) * Initial commit of Use Case * Adding use case image * Updating documentation paths * Updating processing * Fixed typo * Updated documentation typos * Trying to get use case tests to run * Fixed typo in use case name * Turned on MODE run for testing * More testing * Updating data path * Trying updated docker settings * Updating text in .conf file * Testing time for all mode runs * Moving use case to new area * Adding masking to the use case * Fixing mask location * Adding CI overrides * Updating process list * Fixing Wrong lead times * Update docs/use_cases/model_applications/clouds/GridStat_fcstGFS_obsGFS_cloudFracLayer.py Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> * Update docs/use_cases/model_applications/clouds/GridStat_fcstGFS_obsGFS_cloudFracLayer.py Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> * Update docs/use_cases/model_applications/clouds/GridStat_fcstGFS_obsGFS_cloudFracLayer.py Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> * updating to be clouds #8 * Update use_case_groups.json * Updating to be clouds #8 * Some refactoring of the UserScript section. * Missed an indent. * Removing line with space confusing the RST formatter. * Minor clarification about three MODE entries and only one GridStat for the PROCESS_LIST. * Fixed error in use_case_groups.json * Updates to fix one sonar cube error * Changing use case to false * Minor changes to UserScript wording. * Addresses logger variable formatting issue. --------- Co-authored-by: Christina Kalb <kalb@seneca.rap.ucar.edu> Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> * updated examples to reflect new paths on seneca, cleaned up instructions to install METplus analysis tools in conda envs locally and consolidated them in one place to reduce redundant instructions * Feature 2771 subprojects (#2942) * Per #2771, attempting to add MET User's Guide to ToC * Per #2771, updating link to MET User's Guid and adding link for MET Contributor's Guide * Per #2771, adding COMPONENTS section * Per #2771, playing with menu options * Per #2771, playing with menu options * Per #2771, try to add scroll bar functionality to ToC * Per #2771, attempt two to add scroll bar * Per #2771, attempt three add scroll bar for ToC * Per #2771, attempt four to get scroll bar for ToC * Per #2771, attempt to make logo area sticky * Per #2771, attempt to fix problems with sticky logo * Per #2771, attempting to fix overlap of sticky logo over ToC * Per #2771, attempting to fix overlap of sticky logo over ToC * Per #2771, attempting to fix overlap of sticky logo over ToC * Per #2771, attempting to fix overlap of sticky logo over ToC * Per #2771, ensures that when an anchor is clicked, the selected element is not hidden behind the sticky header * Per #2771, attempt 2 - ensure that when an anchor is clicked, the selected element is not hidden behind the sticky header * Per #2771, menu updates * Per #2771, hide the sticky logo so that it behaves like the original (disappearing into the hamburger menu) * Per #2771, replace reference to feature branch with latest * Feature 2937 update data physics tendency (#2938) * Update config file to match updated data * redirect output to OUTPUT_BASE/tmp_500hPa.png because the refactored script will save the output to the directory where the command was invoked. This may cause issues when running the use case in a CI environment * Update config file to match updated data * Replaced plot with new plot using updated data. * Updated plot using latest data * Updated for refactored code that doesn't require additional args from the command line. Also redirect output to the OUTPUT_BASE directory because the plotting script stores the output to the directory from where the script was invoked. * updates to conf files for the vertical profile plotting * Updated thumbnails for new physics tendency plots generated from new data * updates made to reflect changes due to new data * Modified instructions on location of output file * remove the redirection command, it does not work * Update UserScript_fcstFV3_fcstOnly_PhysicsTendency_Planview.py removed text that was erroneously copied * pass output directory to script to write output in correct location * remove unused imports --------- Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Feature #2923 SonarQube 6.1.0-beta2 round 2 (#2946) * sq fix * move duplicate script into common directory and update conf files to call script from new location -- removed duplicate code SQ complaints * handle SQ complaints for duplicate code and code smells * fix paths to new locations of common scripts in documentation * refactor to resolve SonarQube complaints * more SQ - specify exception and start to reduce cognitive complexity * clean up SQ complaints * remove a few more code smells * more code smell cleanup * Per #2944, add support for fcst/obs_valid/init_beg/end/inc/exc/hour and remove obs_window for PairStat wrapper (#2945) * update apt to prevent imagemagick install failure and don't cancel other python version unit test if one fails * fix imagemagick install by updating apt packages * Feature #2941 v6.1.0-beta2 (#2950) * add release notes * update version for beta2 release * update beta2 release date * update info about DockerHub images for beta2 * update date of release * Update docs/Users_Guide/release-notes.rst Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * per suggestion from @jprestop, add info about python version upgrade to the upgrade instructions * disable doxygen by default * fix formatting of note --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * update version for dev towards rc1 * Add requestor and reporting labels for NOAA/EPIC, as requested by @michelleharrold. * Feature 2953 installation files (#2958) * Per #2953, update Orion file for Python 3.12 * Per #2953, updating Existing Builds page for installation on Orion * Per #2953, updating Casper file for Python 3.12 * Per #2953, adding script for METplus-6.1 using Python 3.12 * Per #2953, updated installation information for Casper * Per #2953, updated compiler for Orion * Per #2953 updating compiler in Existing Builds for Orion * Per #2953, added qhull needed for atlas on hera * Per #2953, remove qhull as it was actually causing problems on hera * Per #2953, updating hera file for Python 3.12 * Per #2953, updating with hera information * Per #2953, updating jet file for Python 3.12 * Per * Per #2953, adding internal/scripts/installation to paths-ignore * Feature #2560 Update Fire use case (#2956) * add quotation marks around file path if it includes spaces * update use case to read WRF fire field on subgrid directly instead of using Python Embedding * update instructions for updating web server data to use MET scripts * add a test for python embedding input * fix issue introduced where double quotes are added * prevent incorrect error when running PyEmbedIngest * fix URL link * update use case doc to note that Python Embedding is no longer used * remove keyword to set up Python Embedding in automated tests because it is no longer used * per PR review suggestion from Copilot, add comment to describe why variable is set * override output units/level Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * fix formatting of set_attr options --------- Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * added suggestions from conda-forge code review * force using python 3.10 locally for now * fix skip of any python except 3.10 and change env var name to prevent conflict with reserved env var used by METplus to set parm base * remove dependencies that don't need to be listed because they are made available by other dependencies * add MET license file * remove GS Fonts due to licensing issues and it is optional * prevent warning for numpy because it is actually used * add back libopenblas for linux only * Feature 2746 sfs use cases (#2954) * Adding Soil Moisture averaging case * Adding a new use case to compare soil moisture of SFS-GSL vs ERA5. The use case uses a python embedding file to read the SFS-GSL files. * Adding documentation to the soil moisture use case to compare SFS-GSL against ERA5 * Adding a thumbnail for the use case * Adding updated soil moisture use case * Updating title line * Adding Use Case image * Updating docs * Trying to fix use case image * Updating documentation * Updating documentation * Fixing documenation * Updating documentation * Updating directory * Adding use case data * Adding the s2s_soil_moisture directory * Updating documentation location * Adding Documentation README * Adding config override for actions * Fixing typo * Testing Config overrides * Testing new timeframe * Changing CI overrides * Renamed files and moved them to s2s_soil_moisture directory * Updated file to change ERA to ERA5Land * Moved file to the correct directory and updated * Changed file name * Fixed directory name * Fixed directory name * Fixed directory name * Added test for the soil moisture use case * Adding another use case * Setting tests to false for pull requests * Fixed image file name * Testing sonar cube change * Turning a test back on to check a change * Update sfs_gsl_model_wrapper.py * Testing a sonar cube update * Updated documentation as per new template * Updated documentation as per new template * Gallery path update * Update use_case_groups.json * Turning tests off for PR * Removed boilerplate language * Changing log level --------- Co-authored-by: Mrinal Biswas <biswas@ucar.edu> Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> * add example of using the allow_empty extra arg keyword * change minimum supported version of python to 3.10.4 because the wrappers use logic that requires that version and we are unable to run the unit tests for earlier versions in GitHub Actions * fix tests that check python versions * update function that checks if a value is a number to properly handle numpy masked values (MaskedConstant) * fix incorrect label * Feature 2904 tif fuse case (#2987) * first pass over doc, basic files. Needs config update and finalize doc * add rioxarray conda environment * update out-of-date info * added image, updated docs, attempt to resolve compile issue * removing space for error * updated remaining files, adjusted docs * fix incorrect naming for another use case that had errors in use case doc rendering * fix typos and formatting * fix path to thumbnail image * add py_embed keyword to tell automated test suite to set MET_PYTHON_EXE to the version of python in the rioxarray conda env * fix SonarQube complaint "Fix this attribute access on a value that can be 'None'." despite the function cannot return None because it will exit the script if an invalid input is provided --------- Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Hotfix to Release Guide instructions for MET to fix a minor formatting issue. * change version of sphinx-rtd-theme to a version that is available on conda-forge * Feature #2962 RMWAnalysis wrapper (#2985) * per #2962, begin implementation of RMWAnalysis wrapper * start basic use case conf * read BOTH_VAR<n> variables into data field * change pycharm python interpreter * ignore file generated by building docs locally * update use case to read input data * Per #2962, add documentation, unit tests, and basic use case for RMWAnalysis wrapper * per #2962, added thumbnail for use case documentation * fix incorrect formatting * per feedback from @JohnHalleyGotway, change 5 config variables to support lists of values instead of a single string * Bugfix #2890 develop - prevent incorrect error for skipped time (#2988) * per #2890, refactor Example wrapper to support all EXAMPLE_RUNTIME_FREQ settings to expand its use and reproduce the bug to be addressed * add docs for setting runtime freq for Example wrapper * prevent crash checking an init or valid time that is a wildcard * clean up function docs * Per #2890, add more checks for skipping times so wrappers don't incorrectly report an error if an init/valid time is skipped. Create function to handle skip checks, logging run time, and adding instance/custom to templates to reduce duplicate code. Change skip logging from DEBUG to INFO so it is more clear what is being skipped. Rename run_at_time function to be more clear of its purpose * update metplus_dev environment to include the documentation requirements * Per #2890, added unit tests for example wrapper to increase code coverage on new code. Improved error reporting if incorrect time configs are set * Feature #2984 v6.1.0-rc1 (#2991) * add release notes for rc1 * update DockerHub rc1 releases * fix indentation * Per #2984, updating title of issue 2863 --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * change deprecated maintainer to label * prevent GHA 'System.IO.IOException: No space left on device' error by removing data from original location as well * remove docker image after pushing to DockerHub to free up disk space * add new workflow to only run and update the truth data for use case groups that had differences in the last non-ref branch run. runs all use case groups if there is no previous non-ref run to handle first main_vX.Y-ref run before main_vX.Y has been created. also supports workflow dispatch to force update of all truth data as needed * remove problematic run-name logic because PR merge that will trigger the run has the information we need * remove logic to update truth data from testing workflow since it is now handled in update_truth_data.yml * rename job so workflow looks more similar to testing * rename workflow so it is clear that it updates the *-ref branch instead of actually updating the truth data on DockerHub * add release note for missing bugfix issue * push fix when no latest run for main_v6.1 branch exists * do not run anything in testing workflow for -ref branches * update version for development towards 6.2.0-rc1 * Feature 2932 use case amdar pbl (#2989) * Enhanced AMDAR PBLH METplus use case * Fixes title underline length error. * Adds missing .csv for airport filename in the conf file. * Reverts addition of .csv suffix since the Python embedding script adds it, and adjusts Python embedding script locations to use PARM_BASE instead of INPUT_BASE. * Changes the NumPy where statement to use the string nan instead of the NumPy nan constant to match the data type of the tailNumber column at this point. The string nan is used also on the subsequent line for valid_tails. --------- Co-authored-by: Daniel Adriaansen <dadriaan@ucar.edu> * Feature #2998 develop - updating truth data instructions (#3001) * per #2998, update instructions for updating truth data to note that the latest develop branch run needs to finish running. also cleaned up some out-of-date info * add note to delete the temporary branch generated for updating develop-ref * enhance pytest fixture so it will work with wrappers that don't have class variable WRAPPER_ENV_VAR_KEYS * clean up tests * Feature 2953 installation files (#3004) * Per #2953, update Orion file for Python 3.12 * Per #2953, updating Existing Builds page for installation on Orion * Per #2953, updating Casper file for Python 3.12 * Per #2953, adding script for METplus-6.1 using Python 3.12 * Per #2953, updated installation information for Casper * Per #2953, updated compiler for Orion * Per #2953 updating compiler in Existing Builds for Orion * Per #2953, added qhull needed for atlas on hera * Per #2953, remove qhull as it was actually causing problems on hera * Per #2953, updating hera file for Python 3.12 * Per #2953, updating with hera information * Per #2953, updating jet file for Python 3.12 * Per * Per #2953, adding internal/scripts/installation to paths-ignore * Per #2953, updating existing builds page. * Bugfix #2980 develop - PCPCombine 3 year accum bug (#3005) * per #2980, add test to recreate bug when building a 3 year accumulation using 1 year accums * Per #2980, fix bug in determining accumulation seconds used to find last valid time to check for an accumulation -- The search time minus the output accumulation is used as the reference time to compute the number of seconds of the input accumulation. The input accumulation is subtracted from this time to determine the number of seconds, which depending on the time can cross over a leap year when the actual accumulation calculations do not. This fix adds the input accumulation to the 'first' time to ensure that it is correctly calculated * Feature #3008 Support lead months/years in template substitution (#3009) * clean up code formatting * Per #3008, add support for months and years in {lead} filename template tags. Added tests to ensure correct behavior occurs. Rename format_hms function to be more clear of what it actually does now * reduce SonarQube findings by reducing cognitive complexity. Clean up documentation * Feature 2857 support (#3012) * Per #2857, updated User Support section and also modified case of other headers * Per #2857, resolved duplicate explicit target name warning. * Per 2857, added information about the team drive where data is uploaded to and labelling for organization or institution. * add swpc_metpy env to GHA workflow to create conda envs * pull changes from metplus-feedstock * reduce required python version to match metplus/PYTHON_VERSION_MIN * add met_tool_wrapper use case to existing group * Feature 2999 improve diffs for met tests (#3019) * added some temp code to check for .log files being diffed * removed checks for .log files / reverted * added some more detail to output, including number files * changed calc_transport.log to calc_transport.out and updated ref in docs * skip .log files * changes to diff_text_lines * added temp check for files without extensions * added temp check for files without extensions / fixed * don't compare files without extensions, e.g. file_list in MET tests * forgot to hit save after deleting a couple lines before the last commit... * improved handling of nc files * enhanced numeric diffs for nc files * implemented logic for setting tiny values to zero as pper MET feature 3134 * implemented diff checking on sig figs * slight tweak to order of equality checks * added debug line * more debugging * fixed _round_sig_figs * fixed bug in _set_zero * another fix * reinstated rounding precision for diffs after trying only sig fig rouding * bumped SIG_FIG for diffs to 7 * fix to _round_sig_figs * simplified _is_number() * some tidying * refactor/cleanup to remove code smells reported by SonarQube * more cleanup * added unit tests for skip file extensions logic and new equal comparison functions * add comments to describe each list of file extensions --------- Co-authored-by: Natalie Babij <nbabij@seneca.rap.ucar.edu> Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * add config variables for setting file_type in the climo_mean/stdev dictionaries (#3027) * Feature 2921 develop sphinx macports (#3025) * Per #2921, adding documentation for setting up a sphinx environment using MacPorts * Per #2921, adding context to explain why a developer may need to run these commands. * Feature #2575 TCI use case - generic model (#2768) * Generalizes METplus config file entries relevant to the FCST dataset used to compute TCI. * per #2575, rename variables, scripts, etc. to avoid using model-specific wording to encourage users to modify this use case to use other models * cleanup * merged (old) develop and resolved conflicts * fixed incorrect merge * update use case doc file to more closely match use case documentation template * replace bad quotation mark character and fix indentation * fix formatting * more formatting changes * fix formatting to match template for METplus Workflow section and moved use case specific config variable descriptions inside dropdown for python scripts * Removes errant include of Python embedding script, tidies up glossary RST, and adds special note under the dataset section that if a user has GRIB or other data without a time dimension, they need to somehow create a data object or file with a time dimension in order to use the fcst_tci Python embedding script since it assumes there is a time dimension on the forecast data. --------- Co-authored-by: Daniel Adriaansen <dadriaan@ucar.edu> * Feature #2980 refactor PCPCombine file search (#3017) * per #2980, refactor PCPCombine wrapper to find input files like other RuntimeFreq wrappers to be consistent. Clean up logic to remove duplicate code. Update pytests to match changes to wrapper * fix typo in name of temporary var list * clean up logic for SonarQube * handle if temp var (field) list is None or an empty list * fixed bug where USER_DEFINED method was reporting an error when it shouldn't be because it doesn't gather any files * refactored logic into function to reduce duplicate code * move call to set_environment_variables right before call to app so environment variables are not printed if the command will not be run * update ExtractTiles error count if regrid_data_plane has any errors so entire run fails from error * add unit test to capture bug where ExtractTiles is failing * fix bug from previous commit by settings output template/dir from FCST/OBS * update test to run just FCST and just OBS instead of always both * change input template config to legacy supported OBS_REGRID_DATA_PLANE_TEMPLATE (instead of OBS_REGRID_DATA_PLANE_INPUT_TEMPLATE) to reproduce bug in RegridDataPlane_multi_field_one_file use case * clean up formatting complaints * reduce cognitive complexity of function. use utility function instead of calling regex logic explicitly. make returns consistent * fix bug in RegridDataPlane -- this wrapper supports setting a generic FCST/OBS_REGRID_DATA_PLANE_TEMPLATE that can be used as both _INPUT_TEMPLATE and _OUTPUT_TEMPLATE. This is used in one of the met_tool_wrapper examples but was not supported by RuntimeFreq's get_input_templates, so I modified the function to look for generic TEMPLATE if INPUT_TEMPLATE was not set * reset level to be used in the output filename template substitution back to the output accumulation aka lookback seconds * Only require input template to be set for run methods other than USER_DEFINED and SUM. * add unit test for USER_DEFINED run method * refactor to reduce duplicate code * add pytest fixtures to reduce duplicate code in unit tests * improve/expand SUPPRESS_WARNINGS logic * add argument to skip logging when input files cannot be found and just return True/False * clean up formatting * refactor RegridDataPlane wrapper to use RuntimeFreq input file finding logic and remove functions that are no longer needed * update unit tests to match changes to wrapper * fix ExtractTiles to not report errors when files are not found (previously reported warnings) * reduce cognitive complexity * split copyable environment variable commands onto their own lines so it is easier to read and share with others for debugging purposes * refactor ExtractTiles wrapper to find files using RuntimeFreq logic since it calls RegridDataPlane to create the tiles. Update unit tests to reflect changes and ensure that failures are captured properly * clean up formatting * clean up formatting * Feature 3014 sfs updates (#3020) * Adding graphic * Updating python embedding script location * Updating environment * Updating environment * Removing bad path * Adding CI override * Updating documentation * Updating documenttion again * Take 3 * Another documentation update * Updating use case * Updating the order of processes * Updating documentation * Cleaning up unused variables * Updating documentation * Adding climatology. Testing will fail b/c haven't uploaded data * Adding Series Analysis climatology and updating documentation * Another doc update * Fixing valid_beg * fixing time * Fixing documentation typo * Fixed another typo * Fixing log verbosity and block size * Updating some code * Fixing typo * Testing removal of file_type * Removing file type * Setting tests to false for pull requests * minor grammar, spelling updates * Fixing increment * Fixing Log level * Update GridStat_fcstSFSGSL_obsERA5Land_SoilMoisture.conf * editing plot title --------- Co-authored-by: Christina Kalb <kalb@seneca.rap.ucar.edu> Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Feature 2723 use case sep climates (#3030) * added config and docs. Added use case to testing files. Still needs img, testing * updated config with file_type, use case img * Small grammar change --------- Co-authored-by: Christina Kalb <kalb@seneca.rap.ucar.edu> * Feature 2390 ctp hum index (#2981) * Initial Python embedding script for computing CTP from GDAS prepbufr radiosondes (obs). * Cleanup of extra debugging and development blocks. * Initial Python embedding script for computing HI from GDAS prepbufr radiosondes (obs). * Minor changes to print statements. * Major cleanup. * Removes check for highest pressure since that could vary. Check was moved to function in METcalcpy. * Conf file for this use case. * Python embedding script for FCST for CTP. * Changes for UserScript and GenVxMask. * Changes for function signature to take a station_id argument. * Python embedding script to compute Humidity Index from UFS/HR1 forecast data. * Script to create a lat/lon text file for GenVxMask to read and create a gridded mask file to use, based on radiosonde station locations in a GDAS prepBUFR file. * Updates to account for new station_id argument by calc_ctp function in METcalcpy. * Major changes to workflow using multiprocessing for computing CTP from UFS/HR1 forecast data. * Removes erroneous code from script that this script was copied from. * Removes unused code from development and print statements from development. * Removes print statements used during development. * Removes unused code from development and print statements from development. * Updated use case configuration file including config sections for each metric, GenVxMask options, and UserScript options. * Turns down verbosity for PointStat. * Adds print statement that the Python embedding script is starting for log purposes. * Adds print statement to log when it is starting to each Python embedding script, and also removes the interp argument from calc_humidity_index for obs to use the default value in METcalcpy. * Adds debug message for printing the station being processed, with the default turned off. * Major updates to documentation file for use case. * Updates to the UserScript to allow a user to control which sites are included in the mask file for GenVxMask. * Cleanup of use case config gile to remove unused items. * Switches to F strings, wraps some print statements in DEBUG, and switches to NumPy max/min. * Removes unused functions from forecast Python embedding scripts. * Switches to F strings and wraps some print statements in DEBUG flag. * Switches to F strings and adds script name to print statements. * Adds space around print statements to aid log file inspection. * Adds DEBUG level as a command line argument and wraps last remaining print statement in DEBUG flag. This can now be controlled via the METplus config file. * Adds controlling DEBUG level for observation Python embedding scripts. * Updates documentation URL in config file. * Changes subsequent processing steps to read from OUTPUT_DIR, adjusts for subset of global grid to demonstrate functionality, and adjusts input data paths to be relative to INPUT_BASE and the use case category structure for GHA. * Updates for sample data and running the use case in GHA. * Typo in the JSON file. * Changes the MET_PYTHON_DIR to the installed location rather than the source code location. * Adds user script to create graphic from PointStat MPR output. * Adds use case gallery image and updates observation dataset info in documentation. * Fixes path to config file include in documentation. * Adds new user script for plotting to the documentation file. * Updates list of expected output files to include UserScript outpout and PB2NC output. * Changes to eliminate code smells. * Finalizes forecast dataset description and scientific objective. * Adds info on additional Python packages required for Python embedding and UserScripts. * Removes extra #. * Removes uneccesary imports in user script. * Adds more clarity that METcalcpy is a required Python package for Python embedding. * Moves the setting of MET_PYTHON_EXE into the configuration file, as the user should ensure the version of Python in their environment at the time of running the use case meets the requirements for both Python embedding and Python UserScripts. * Removes MET_PYTHON_EXE since it will not be needed if a user has compiled MET against a version of Python that has the requirements for Python embedding. MET_PYTHON_EXE should be set in the users' local conf file, and as described in the documentation, simply be set to python3. * Adds updated note for users in the Running METplus section that provides guidance on how to ensure the correct Python is used for the various components (Python embedding, Python UserScripts). * prevent exception when trying to compare a masked (nan) value to a number * turn off use case * added workflow dispatch to workflow to allow rebuild of Docker images periodically to fix any security vulnerabilities in upstream images * added catch for log10(0) (#3043) Co-authored-by: Natalie Babij <nbabij@seneca.rap.ucar.edu> * Feature #2577 develop - WRF Hurricane Matthew use case (#3036) * Feature #2577 WRF Hurricane Matthew use case (#3015) * per #2577, add use case files * turn off use case that shouldn't be on * fix path to data and scripts * fix path to plotting scripts * added scripts to create wrf_plot conda env that contains the METplus Analysis requirements and the wrf package. change use case to use that env and obtain METdataio and METplotpy * fix name of wrf python package * override input and output directories in WRF plotting script * clean up script re: SonarQube * renaming variables for SonarQube * turn off 100-m wind speed plot because zlev files are not included in the input data * added missing closing single quote * add version of package that was obtained for reproducibility * refactor WRF plotting script to split logic into functions to reduce cognitive complexity * try creating output directory before running reformatter * METdataio reformatter does not accept environment variables in its config file, so attempting to change directory to output directory and find input data relative to that directory * change approach to reference plot directory * add WRF and MADIS file types to quick search keywords and start doc for new use case * fix formatting for output file lists and correct info about output directories (use case path are added for automated tests, but are relative to {OUTPUT_BASE} if run by a user) * write log file to logs directory and rename it to include .log extens…

* Add requestor and reporting labels for NOAA/EPIC, as requested by @michelleharrold. * Feature 2953 installation files (#2958) * Per #2953, update Orion file for Python 3.12 * Per #2953, updating Existing Builds page for installation on Orion * Per #2953, updating Casper file for Python 3.12 * Per #2953, adding script for METplus-6.1 using Python 3.12 * Per #2953, updated installation information for Casper * Per #2953, updated compiler for Orion * Per #2953 updating compiler in Existing Builds for Orion * Per #2953, added qhull needed for atlas on hera * Per #2953, remove qhull as it was actually causing problems on hera * Per #2953, updating hera file for Python 3.12 * Per #2953, updating with hera information * Per #2953, updating jet file for Python 3.12 * Per * Per #2953, adding internal/scripts/installation to paths-ignore * Feature #2560 Update Fire use case (#2956) * add quotation marks around file path if it includes spaces * update use case to read WRF fire field on subgrid directly instead of using Python Embedding * update instructions for updating web server data to use MET scripts * add a test for python embedding input * fix issue introduced where double quotes are added * prevent incorrect error when running PyEmbedIngest * fix URL link * update use case doc to note that Python Embedding is no longer used * remove keyword to set up Python Embedding in automated tests because it is no longer used * per PR review suggestion from Copilot, add comment to describe why variable is set * override output units/level Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * fix formatting of set_attr options --------- Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * added suggestions from conda-forge code review * force using python 3.10 locally for now * fix skip of any python except 3.10 and change env var name to prevent conflict with reserved env var used by METplus to set parm base * remove dependencies that don't need to be listed because they are made available by other dependencies * add MET license file * remove GS Fonts due to licensing issues and it is optional * prevent warning for numpy because it is actually used * add back libopenblas for linux only * Feature 2746 sfs use cases (#2954) * Adding Soil Moisture averaging case * Adding a new use case to compare soil moisture of SFS-GSL vs ERA5. The use case uses a python embedding file to read the SFS-GSL files. * Adding documentation to the soil moisture use case to compare SFS-GSL against ERA5 * Adding a thumbnail for the use case * Adding updated soil moisture use case * Updating title line * Adding Use Case image * Updating docs * Trying to fix use case image * Updating documentation * Updating documentation * Fixing documenation * Updating documentation * Updating directory * Adding use case data * Adding the s2s_soil_moisture directory * Updating documentation location * Adding Documentation README * Adding config override for actions * Fixing typo * Testing Config overrides * Testing new timeframe * Changing CI overrides * Renamed files and moved them to s2s_soil_moisture directory * Updated file to change ERA to ERA5Land * Moved file to the correct directory and updated * Changed file name * Fixed directory name * Fixed directory name * Fixed directory name * Added test for the soil moisture use case * Adding another use case * Setting tests to false for pull requests * Fixed image file name * Testing sonar cube change * Turning a test back on to check a change * Update sfs_gsl_model_wrapper.py * Testing a sonar cube update * Updated documentation as per new template * Updated documentation as per new template * Gallery path update * Update use_case_groups.json * Turning tests off for PR * Removed boilerplate language * Changing log level --------- Co-authored-by: Mrinal Biswas <biswas@ucar.edu> Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> * add example of using the allow_empty extra arg keyword * change minimum supported version of python to 3.10.4 because the wrappers use logic that requires that version and we are unable to run the unit tests for earlier versions in GitHub Actions * fix tests that check python versions * update function that checks if a value is a number to properly handle numpy masked values (MaskedConstant) * fix incorrect label * Feature 2904 tif fuse case (#2987) * first pass over doc, basic files. Needs config update and finalize doc * add rioxarray conda environment * update out-of-date info * added image, updated docs, attempt to resolve compile issue * removing space for error * updated remaining files, adjusted docs * fix incorrect naming for another use case that had errors in use case doc rendering * fix typos and formatting * fix path to thumbnail image * add py_embed keyword to tell automated test suite to set MET_PYTHON_EXE to the version of python in the rioxarray conda env * fix SonarQube complaint "Fix this attribute access on a value that can be 'None'." despite the function cannot return None because it will exit the script if an invalid input is provided --------- Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Hotfix to Release Guide instructions for MET to fix a minor formatting issue. * change version of sphinx-rtd-theme to a version that is available on conda-forge * Feature #2962 RMWAnalysis wrapper (#2985) * per #2962, begin implementation of RMWAnalysis wrapper * start basic use case conf * read BOTH_VAR<n> variables into data field * change pycharm python interpreter * ignore file generated by building docs locally * update use case to read input data * Per #2962, add documentation, unit tests, and basic use case for RMWAnalysis wrapper * per #2962, added thumbnail for use case documentation * fix incorrect formatting * per feedback from @JohnHalleyGotway, change 5 config variables to support lists of values instead of a single string * Bugfix #2890 develop - prevent incorrect error for skipped time (#2988) * per #2890, refactor Example wrapper to support all EXAMPLE_RUNTIME_FREQ settings to expand its use and reproduce the bug to be addressed * add docs for setting runtime freq for Example wrapper * prevent crash checking an init or valid time that is a wildcard * clean up function docs * Per #2890, add more checks for skipping times so wrappers don't incorrectly report an error if an init/valid time is skipped. Create function to handle skip checks, logging run time, and adding instance/custom to templates to reduce duplicate code. Change skip logging from DEBUG to INFO so it is more clear what is being skipped. Rename run_at_time function to be more clear of its purpose * update metplus_dev environment to include the documentation requirements * Per #2890, added unit tests for example wrapper to increase code coverage on new code. Improved error reporting if incorrect time configs are set * Feature #2984 v6.1.0-rc1 (#2991) * add release notes for rc1 * update DockerHub rc1 releases * fix indentation * Per #2984, updating title of issue 2863 --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * change deprecated maintainer to label * prevent GHA 'System.IO.IOException: No space left on device' error by removing data from original location as well * remove docker image after pushing to DockerHub to free up disk space * add new workflow to only run and update the truth data for use case groups that had differences in the last non-ref branch run. runs all use case groups if there is no previous non-ref run to handle first main_vX.Y-ref run before main_vX.Y has been created. also supports workflow dispatch to force update of all truth data as needed * remove problematic run-name logic because PR merge that will trigger the run has the information we need * remove logic to update truth data from testing workflow since it is now handled in update_truth_data.yml * rename job so workflow looks more similar to testing * rename workflow so it is clear that it updates the *-ref branch instead of actually updating the truth data on DockerHub * add release note for missing bugfix issue * push fix when no latest run for main_v6.1 branch exists * do not run anything in testing workflow for -ref branches * update version for development towards 6.2.0-rc1 * Feature 2932 use case amdar pbl (#2989) * Enhanced AMDAR PBLH METplus use case * Fixes title underline length error. * Adds missing .csv for airport filename in the conf file. * Reverts addition of .csv suffix since the Python embedding script adds it, and adjusts Python embedding script locations to use PARM_BASE instead of INPUT_BASE. * Changes the NumPy where statement to use the string nan instead of the NumPy nan constant to match the data type of the tailNumber column at this point. The string nan is used also on the subsequent line for valid_tails. --------- Co-authored-by: Daniel Adriaansen <dadriaan@ucar.edu> * Feature #2998 develop - updating truth data instructions (#3001) * per #2998, update instructions for updating truth data to note that the latest develop branch run needs to finish running. also cleaned up some out-of-date info * add note to delete the temporary branch generated for updating develop-ref * enhance pytest fixture so it will work with wrappers that don't have class variable WRAPPER_ENV_VAR_KEYS * clean up tests * Feature 2953 installation files (#3004) * Per #2953, update Orion file for Python 3.12 * Per #2953, updating Existing Builds page for installation on Orion * Per #2953, updating Casper file for Python 3.12 * Per #2953, adding script for METplus-6.1 using Python 3.12 * Per #2953, updated installation information for Casper * Per #2953, updated compiler for Orion * Per #2953 updating compiler in Existing Builds for Orion * Per #2953, added qhull needed for atlas on hera * Per #2953, remove qhull as it was actually causing problems on hera * Per #2953, updating hera file for Python 3.12 * Per #2953, updating with hera information * Per #2953, updating jet file for Python 3.12 * Per * Per #2953, adding internal/scripts/installation to paths-ignore * Per #2953, updating existing builds page. * Bugfix #2980 develop - PCPCombine 3 year accum bug (#3005) * per #2980, add test to recreate bug when building a 3 year accumulation using 1 year accums * Per #2980, fix bug in determining accumulation seconds used to find last valid time to check for an accumulation -- The search time minus the output accumulation is used as the reference time to compute the number of seconds of the input accumulation. The input accumulation is subtracted from this time to determine the number of seconds, which depending on the time can cross over a leap year when the actual accumulation calculations do not. This fix adds the input accumulation to the 'first' time to ensure that it is correctly calculated * Feature #3008 Support lead months/years in template substitution (#3009) * clean up code formatting * Per #3008, add support for months and years in {lead} filename template tags. Added tests to ensure correct behavior occurs. Rename format_hms function to be more clear of what it actually does now * reduce SonarQube findings by reducing cognitive complexity. Clean up documentation * Feature 2857 support (#3012) * Per #2857, updated User Support section and also modified case of other headers * Per #2857, resolved duplicate explicit target name warning. * Per 2857, added information about the team drive where data is uploaded to and labelling for organization or institution. * add swpc_metpy env to GHA workflow to create conda envs * pull changes from metplus-feedstock * reduce required python version to match metplus/PYTHON_VERSION_MIN * add met_tool_wrapper use case to existing group * Feature 2999 improve diffs for met tests (#3019) * added some temp code to check for .log files being diffed * removed checks for .log files / reverted * added some more detail to output, including number files * changed calc_transport.log to calc_transport.out and updated ref in docs * skip .log files * changes to diff_text_lines * added temp check for files without extensions * added temp check for files without extensions / fixed * don't compare files without extensions, e.g. file_list in MET tests * forgot to hit save after deleting a couple lines before the last commit... * improved handling of nc files * enhanced numeric diffs for nc files * implemented logic for setting tiny values to zero as pper MET feature 3134 * implemented diff checking on sig figs * slight tweak to order of equality checks * added debug line * more debugging * fixed _round_sig_figs * fixed bug in _set_zero * another fix * reinstated rounding precision for diffs after trying only sig fig rouding * bumped SIG_FIG for diffs to 7 * fix to _round_sig_figs * simplified _is_number() * some tidying * refactor/cleanup to remove code smells reported by SonarQube * more cleanup * added unit tests for skip file extensions logic and new equal comparison functions * add comments to describe each list of file extensions --------- Co-authored-by: Natalie Babij <nbabij@seneca.rap.ucar.edu> Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * add config variables for setting file_type in the climo_mean/stdev dictionaries (#3027) * Feature 2921 develop sphinx macports (#3025) * Per #2921, adding documentation for setting up a sphinx environment using MacPorts * Per #2921, adding context to explain why a developer may need to run these commands. * Feature #2575 TCI use case - generic model (#2768) * Generalizes METplus config file entries relevant to the FCST dataset used to compute TCI. * per #2575, rename variables, scripts, etc. to avoid using model-specific wording to encourage users to modify this use case to use other models * cleanup * merged (old) develop and resolved conflicts * fixed incorrect merge * update use case doc file to more closely match use case documentation template * replace bad quotation mark character and fix indentation * fix formatting * more formatting changes * fix formatting to match template for METplus Workflow section and moved use case specific config variable descriptions inside dropdown for python scripts * Removes errant include of Python embedding script, tidies up glossary RST, and adds special note under the dataset section that if a user has GRIB or other data without a time dimension, they need to somehow create a data object or file with a time dimension in order to use the fcst_tci Python embedding script since it assumes there is a time dimension on the forecast data. --------- Co-authored-by: Daniel Adriaansen <dadriaan@ucar.edu> * Feature #2980 refactor PCPCombine file search (#3017) * per #2980, refactor PCPCombine wrapper to find input files like other RuntimeFreq wrappers to be consistent. Clean up logic to remove duplicate code. Update pytests to match changes to wrapper * fix typo in name of temporary var list * clean up logic for SonarQube * handle if temp var (field) list is None or an empty list * fixed bug where USER_DEFINED method was reporting an error when it shouldn't be because it doesn't gather any files * refactored logic into function to reduce duplicate code * move call to set_environment_variables right before call to app so environment variables are not printed if the command will not be run * update ExtractTiles error count if regrid_data_plane has any errors so entire run fails from error * add unit test to capture bug where ExtractTiles is failing * fix bug from previous commit by settings output template/dir from FCST/OBS * update test to run just FCST and just OBS instead of always both * change input template config to legacy supported OBS_REGRID_DATA_PLANE_TEMPLATE (instead of OBS_REGRID_DATA_PLANE_INPUT_TEMPLATE) to reproduce bug in RegridDataPlane_multi_field_one_file use case * clean up formatting complaints * reduce cognitive complexity of function. use utility function instead of calling regex logic explicitly. make returns consistent * fix bug in RegridDataPlane -- this wrapper supports setting a generic FCST/OBS_REGRID_DATA_PLANE_TEMPLATE that can be used as both _INPUT_TEMPLATE and _OUTPUT_TEMPLATE. This is used in one of the met_tool_wrapper examples but was not supported by RuntimeFreq's get_input_templates, so I modified the function to look for generic TEMPLATE if INPUT_TEMPLATE was not set * reset level to be used in the output filename template substitution back to the output accumulation aka lookback seconds * Only require input template to be set for run methods other than USER_DEFINED and SUM. * add unit test for USER_DEFINED run method * refactor to reduce duplicate code * add pytest fixtures to reduce duplicate code in unit tests * improve/expand SUPPRESS_WARNINGS logic * add argument to skip logging when input files cannot be found and just return True/False * clean up formatting * refactor RegridDataPlane wrapper to use RuntimeFreq input file finding logic and remove functions that are no longer needed * update unit tests to match changes to wrapper * fix ExtractTiles to not report errors when files are not found (previously reported warnings) * reduce cognitive complexity * split copyable environment variable commands onto their own lines so it is easier to read and share with others for debugging purposes * refactor ExtractTiles wrapper to find files using RuntimeFreq logic since it calls RegridDataPlane to create the tiles. Update unit tests to reflect changes and ensure that failures are captured properly * clean up formatting * clean up formatting * Feature 3014 sfs updates (#3020) * Adding graphic * Updating python embedding script location * Updating environment * Updating environment * Removing bad path * Adding CI override * Updating documentation * Updating documenttion again * Take 3 * Another documentation update * Updating use case * Updating the order of processes * Updating documentation * Cleaning up unused variables * Updating documentation * Adding climatology. Testing will fail b/c haven't uploaded data * Adding Series Analysis climatology and updating documentation * Another doc update * Fixing valid_beg * fixing time * Fixing documentation typo * Fixed another typo * Fixing log verbosity and block size * Updating some code * Fixing typo * Testing removal of file_type * Removing file type * Setting tests to false for pull requests * minor grammar, spelling updates * Fixing increment * Fixing Log level * Update GridStat_fcstSFSGSL_obsERA5Land_SoilMoisture.conf * editing plot title --------- Co-authored-by: Christina Kalb <kalb@seneca.rap.ucar.edu> Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Feature 2723 use case sep climates (#3030) * added config and docs. Added use case to testing files. Still needs img, testing * updated config with file_type, use case img * Small grammar change --------- Co-authored-by: Christina Kalb <kalb@seneca.rap.ucar.edu> * Feature 2390 ctp hum index (#2981) * Initial Python embedding script for computing CTP from GDAS prepbufr radiosondes (obs). * Cleanup of extra debugging and development blocks. * Initial Python embedding script for computing HI from GDAS prepbufr radiosondes (obs). * Minor changes to print statements. * Major cleanup. * Removes check for highest pressure since that could vary. Check was moved to function in METcalcpy. * Conf file for this use case. * Python embedding script for FCST for CTP. * Changes for UserScript and GenVxMask. * Changes for function signature to take a station_id argument. * Python embedding script to compute Humidity Index from UFS/HR1 forecast data. * Script to create a lat/lon text file for GenVxMask to read and create a gridded mask file to use, based on radiosonde station locations in a GDAS prepBUFR file. * Updates to account for new station_id argument by calc_ctp function in METcalcpy. * Major changes to workflow using multiprocessing for computing CTP from UFS/HR1 forecast data. * Removes erroneous code from script that this script was copied from. * Removes unused code from development and print statements from development. * Removes print statements used during development. * Removes unused code from development and print statements from development. * Updated use case configuration file including config sections for each metric, GenVxMask options, and UserScript options. * Turns down verbosity for PointStat. * Adds print statement that the Python embedding script is starting for log purposes. * Adds print statement to log when it is starting to each Python embedding script, and also removes the interp argument from calc_humidity_index for obs to use the default value in METcalcpy. * Adds debug message for printing the station being processed, with the default turned off. * Major updates to documentation file for use case. * Updates to the UserScript to allow a user to control which sites are included in the mask file for GenVxMask. * Cleanup of use case config gile to remove unused items. * Switches to F strings, wraps some print statements in DEBUG, and switches to NumPy max/min. * Removes unused functions from forecast Python embedding scripts. * Switches to F strings and wraps some print statements in DEBUG flag. * Switches to F strings and adds script name to print statements. * Adds space around print statements to aid log file inspection. * Adds DEBUG level as a command line argument and wraps last remaining print statement in DEBUG flag. This can now be controlled via the METplus config file. * Adds controlling DEBUG level for observation Python embedding scripts. * Updates documentation URL in config file. * Changes subsequent processing steps to read from OUTPUT_DIR, adjusts for subset of global grid to demonstrate functionality, and adjusts input data paths to be relative to INPUT_BASE and the use case category structure for GHA. * Updates for sample data and running the use case in GHA. * Typo in the JSON file. * Changes the MET_PYTHON_DIR to the installed location rather than the source code location. * Adds user script to create graphic from PointStat MPR output. * Adds use case gallery image and updates observation dataset info in documentation. * Fixes path to config file include in documentation. * Adds new user script for plotting to the documentation file. * Updates list of expected output files to include UserScript outpout and PB2NC output. * Changes to eliminate code smells. * Finalizes forecast dataset description and scientific objective. * Adds info on additional Python packages required for Python embedding and UserScripts. * Removes extra #. * Removes uneccesary imports in user script. * Adds more clarity that METcalcpy is a required Python package for Python embedding. * Moves the setting of MET_PYTHON_EXE into the configuration file, as the user should ensure the version of Python in their environment at the time of running the use case meets the requirements for both Python embedding and Python UserScripts. * Removes MET_PYTHON_EXE since it will not be needed if a user has compiled MET against a version of Python that has the requirements for Python embedding. MET_PYTHON_EXE should be set in the users' local conf file, and as described in the documentation, simply be set to python3. * Adds updated note for users in the Running METplus section that provides guidance on how to ensure the correct Python is used for the various components (Python embedding, Python UserScripts). * prevent exception when trying to compare a masked (nan) value to a number * turn off use case * added workflow dispatch to workflow to allow rebuild of Docker images periodically to fix any security vulnerabilities in upstream images * added catch for log10(0) (#3043) Co-authored-by: Natalie Babij <nbabij@seneca.rap.ucar.edu> * Feature #2577 develop - WRF Hurricane Matthew use case (#3036) * Feature #2577 WRF Hurricane Matthew use case (#3015) * per #2577, add use case files * turn off use case that shouldn't be on * fix path to data and scripts * fix path to plotting scripts * added scripts to create wrf_plot conda env that contains the METplus Analysis requirements and the wrf package. change use case to use that env and obtain METdataio and METplotpy * fix name of wrf python package * override input and output directories in WRF plotting script * clean up script re: SonarQube * renaming variables for SonarQube * turn off 100-m wind speed plot because zlev files are not included in the input data * added missing closing single quote * add version of package that was obtained for reproducibility * refactor WRF plotting script to split logic into functions to reduce cognitive complexity * try creating output directory before running reformatter * METdataio reformatter does not accept environment variables in its config file, so attempting to change directory to output directory and find input data relative to that directory * change approach to reference plot directory * add WRF and MADIS file types to quick search keywords and start doc for new use case * fix formatting for output file lists and correct info about output directories (use case path are added for automated tests, but are relative to {OUTPUT_BASE} if run by a user) * write log file to logs directory and rename it to include .log extension * add more information for use case doc * turn off use case for PR * refactor map functions script to reduce cognitive complexity for SonarQube * update scientific objective with input from @jaredalee * update use case to reference environment variables in reformatted config file instead of using relative paths and changing directories since changes needed to support env vars in the reformatted are available in develop now * test that changes to use case did not break anything * Feature 2533 ismn use case (#3051) * Initial config file for this use case. * Adds ismn as an acceptable format for ASCI2NC_INPUT_FORMAT. * Latest param file changes. * Documentation file for this use case. * Updates documentation URL in config file. * Updated conf file that produces matched pairs for soil moisture from PointStat. * Updates to documentation for ISMN use case. * Adds UserScripts for plotting. * Latest conf file settings for use case. * Use case gallery image. * Adds summary using NEAREST for point_stat matched pairs. * Adds path to gallery image for use case. * Changes use case template to have the sphinx gallery line separate from the previous section to avoid having the actual sphinx_gallery_path appear in the rendered HTML documentation. Still need to fix use cases that don't have this change. * Per discussion with @georgemccabe and experience using the StatAnalysis Wrapper for the ISMN use case, moving two stat-analysis wrapper config items that were listed as required to the optional section since it worked without setting them. Presumably they may be required based on how a user is running stat-analysis or the stat-analysis wrapper, but they are not always required. * Per discussion with @georgemccabe, adds additional details for adding a new use case instructing the user to return the run setting to false in the use_case_groups.json file prior to merging the PR, which is the desired default setting for all use cases. * Minor change to use case documentation. * Initial work for adding ISMN to Verification Datasets guide. * Adds documentation for the ISMN dataset in the Verification Dataset Guide. * Adds VxDataISMN to use case documentation to link the use case to the verification datasets guide. * Adds new use case to testing suite after adding sample data. * Removes mention of RAL FTP site for sending sample data for a new use case, since that is not allowed anymore and generically refers to the GitHub Discussion describing the new process using Google Drive with a link to the Discussion. * Forgot a space. * Update to MET 12.0.3 bugfix version (#3059) * Update to MET 12.0.3 bugfix version. * Switch from using "secrets.DOCKER_USERNAME/secrets.DOCKER_PASSWORD" to using "dtcenter/secrets.DOCKER_TOKEN". * Feature #3054 develop scan_for_CVEs (#3055) * Per #3054, enhance release-docker-images.yml to support CVE scanning, scheduled weekly runs for one or more versions, and update the logic for pushing X.Y-latest tags to Docker Hub. * Per #3054, update the Release Guide with instructions about updating the GHA release workflows. * Per #3054, fix workflow typo. * Per #3054, update release guide formatting. * Per #3054, add CMD_LOGFILE logic to the time_command function. * Per #3054, refine release guide instructions. * Per #3054, refine release guide instructions. * Per #3054, update the versions in the release-docker-images.yml workflow for version 6.1.0. * Per #3054, update to plotly 6.1.1 and kaleido 1.0.0 versions. * Per #3054, revert changes to python package versions since those do not actually impact the GHA testing environment. * Per #3054, updating plotly version to 6.1.1 and python-kaledio version to 1.0.0 in the documentation and internal scripts. Hoping that will enable the 4 failing METplus use cases to succeed. * Replace metplotpy 3.0 with 3.0.1 under the METplus 6.0 release (#3060) * Replace metviewer 6.0.0 with 6.0.1 (#3065) * updates that were made to main_v6.1 to handle chrome dependency for METplotpy use cases * rotate authorship as was done in main_v6.1 branch * fix typo * update version of METexpress for 6.1 Coordinated Release * add look up for upcoming 6.2 coordinated release * Update the release guide with by adding a |downloadURL| variable since MET/METviewer/METexpress store downloads in a separate website location than the other components. (#3074) * update instructions to create a new project board to use templates (#3075) * update instructions to create a new project board to use templates * improve instructions for updating Zenodo * Apply suggestions from code review Co-authored-by: John Halley Gotway <johnhg@ucar.edu> --------- Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * develop docs v6.2.0 release timeline (#3083) * Update the development timeline for METplus version 6.2.0 and fix a minor error in the Release Guide insturctions for MET. * Removing accidental new file. * Feature #3068 DataIngest wrapper (#3084) * clean up * per #3068, add utility function to download files and added unit tests to confirm it works as expected * return from function is uncompress extension is not present * remove unnecessary close() calls because with handles closing of file handles * Log error and return from init function if c_dict was not initialized properly. This allows create_c_dict function to return None in the child class wrappers (not parent classes) * per #3068, create basic components of DataIngest wrapper * Add argument to getraw function to skip replacement of double slash with single slash. The replacement was added to prevent errors with paths that contain double slashes in MET config files, but this causes issues with URLs. * clean up to remove unnecessary variable * Rename DATA_INGEST_<n>_URL/LOCAL_PATH to INPUT/OUTPUT_TEMPLATE to be consistent with other wrappers. Fix incorrect has_option config function name. Improve logging so it is clear that a file is currently being downloaded. Added skip-if-output-exists logic * added basic use case for new wrapper * Handle gz extension as gzip file. Refactor new functions to reduce cognitive complexity * add option to auto decompress files before writing to disk * add use case documentation * add new use case to run in automated test suite * added config options for INPUT/OUTPUT_DIR to be consistent with other wrappers -- added workaround for getdir function to prevent replacement of double slash in URLs * fixed typo * added documentation for config variables for new wrapper * turn off new use case before creating PR * remove test that takes a very long time to download file * removed deprecated config variable * fixed bug in enhancement to config getdir to properly handle skip of double slash handling * improve logging for readability * change lead sequence in example because it is not used * Per #3068, add unit tests for new DataIngest wrapper * fix typo * refactor automated test scripts to reduce duplication in logic to run commands * improve logging so it is more clear that a diff test failed because it is likely that the truth data does not exist yet (e.g. for new use cases) and exit 2 to match return code for a use case that contains actual diffs in the output * Update the METplus-6.1 version of MET from 12.1.0 to 12.1.1 after the first bugfix release. (#3093) * Per #3087, add support for setting message_type_group_map in PB2NC and EnsembleStat wrappers (#3092) * update climo stdev variable name after input data was updated by @j-opatz * change sonar host url to env to view url for quality gate scan results * Fix FSTDEV typo in statistics_list.rst * Update release-notes.rst for the METplus-6.2 Release Schedule * Feature #3028 RMWAnalysis configs (#3094) * Per #3028, added support for many new rmw_analysis MET config options. Refactor logic to read METplus configs and handle MET config env vars to loop over names and handle them consistently. Reordered variables to match new order in RMWAnalysis_default config file * fix missing line break that caused warnings in doc build * change input URLs to obtain files from DTC web server because MADIS data changed since the truth data was created --------- Co-authored-by: MET Tools Test Account <met_test@seneca.rap.ucar.edu> * Feature #3069 Generate run times using templates (#3089) * removed unused imports * removed unused import * Remove loop_over_times_and_call function because time looping should be handled in RuntimeFreq wrapper. Report an error if a wrapper does not properly inherit from RuntimeFreq or implement its own version of run_all_times * update unit tests because CommandBuilder run_all_times now reports an error * added a fixture to create empty files for testing file I/O logic in the unit tests * cleanup to reduce cognitive complexity, remove unnecessary variable, and clean up whitespace for readability * Per #3069, create get_files_and_time_info function to reuse logic to find files using a directory and template for file window logic and new logic to generate list of times to run based on existing files on disk. Add logic to time_generator to support TIME_GENERATOR_INPUT_DIR/TEMPLATE to generate init or valid times to process. Add unit tests to ensure that correct, sorted list of init/valid times are generated. Add helper function for unit tests to ensure that the expected keys for each time dictionary are present and call helper function from tests that use time_generator to generate run times from beg/end/increment and time list. Added stub to implement similar logic to extract available forecast leads for each init/valid time * Per #3069, use TIME_GENERATOR_INPUT_DIR/TEMPLATE to get list of forecast leads that correspond to the current init or valid time being processed. Updated tests to include this functionality * clean up docstring comments * Per #3069, add tests to ensure that logic to allow multiple time generator dir/templates and get the intersection of the run times works as expected before implementing it * Per #3069, enhance time generator dir/template logic to support multiple templates and get the intersection of the run times * add test for a single directory with multiple templates and add helper functions to reduce duplication * whitespace * add function to intelligently split dir/template values so that the dir will contain the path without any template tags and the template will contain any filename template tags so users can include filename template tags in _INPUT_DIR variables and file window and time generator parsing will still work as expected. Added ample tests to ensure correct behavior * correct comments * update test to include full list of existing wrappers * clean up tests and add test for case that could occur in containers and breaks dir/template split logic * implement fix that properly splits dir/template when a template is used in the root (/) directory * check if either TIME_GENERATOR_INPUT_DIR or _TEMPLATE are set to execute logic to get runtimes based on file templates since logic to properly split the path into dir/template prevents the requirement of using both config variables * added helper function to reduce cognitive complexity and make code more readable * reduce duplication by creating variable to store test data templates that are referenced in multiple tests * Per #3069, add support for time generator templates that only include valid time (obs data) and properly filter init/lead times accordingly. * Per #3069, add documentation to describe how to use new functionality with examples * Per #3069, improve logging to be more clear which times were found from templates and when no forecast leads were found for a given init/valid time * Added more debug logging regarding files/times found from searching. Removed function that was only called once * update use case docs to use latest template content * per #3069, add new use case to demonstrate the use of the new functionality to generate run times based on file templates * allow Example wrapper to support multiple inputs, because why not * added quick search keyword for times from template functionality * update use case docs to match template * Filter analysis files in first pass only if looping by valid time. Clean up comments to be more clear/accurate * create helper function to reduce cognitive complexity * Per #3069, enhance template-based time generation to support {instance} and {custom} template tags. * reduce duplication * fix formatting that breaks in py 3.10 * Log number of times each command was run from each wrapper. Rename function to match its new purpose * refactor to reduce cognitive complexity * remove trailing semicolon from executable name in log * per PR feedback from @CPKalb, add missing METplus config variables that are ignored when using time generator input template * include instance name when logging report of successful/failed runs * group new use cases * hotfix for #3069 -- add support for using now/today template tags in the time generator template and allow TIME_GENERATOR_INPUT_DIR to be unset in favor of setting the entire template in TIME_GENERATOR_INPUT_TEMPLATE * SonarQube clean up for v6.2 (#3108) * remove doxygen files because they aren't used * remove commented code * fix formatting of use case docs that get flagged as SQ issues incorrectly * SQ: fix issues with Dockerfiles * SQ: replace unused vars with underscore * SQ: reduce cognitive complexity * SQ: rename CommandBuilder variable isOK to is_ok * SQ: various issues resolved * fix incorrect variable used in read_filters function * use env var value instead of hard-coded extension * SQ: various clean up to use case scripts to resolve SonarQube issues * fix format variable used for time series * remove check for success of calls to add_met_config_dict_list because it is not necessary * return False instead of None to be consistent with the other return values in the function * remove test that is no longer needed after removing status check * Feature 3041 appendix a release notes (#3101) * Per #3041, updating appendix A * Per #3041, updating release notes to reflect need to update Appendix A * update link to MET Python Embedding reqs to use latest instead of feature branch * Update docs/Release_Guide/release_steps/update_appendix_a_python.rst Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * Update docs/Release_Guide/release_steps/update_appendix_a_python.rst Co-authored-by: John Halley Gotway <johnhg@ucar.edu> --------- Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * Feature #3069 part 2 - requests instead of urllib (#3117) * update unit tests to download from another URL that is more reliable * Per #3069, refactor to use requests packages instead of urllib to increase download speed * add scripts to create new conda docker env that contains the requests package needed to run the DataIngest wrapper * install requests package for unit tests * error in init step if requests package is not installed to prevent failure when running DataIngest wrapper * add requests to workflow used to create conda envs in docker * added version number for package * use requests env for DataIngest use case * replace mix of spaces and tabs with just spaces * change long line to multiple * document info about requirements for python package * install requests package for sonarqube GHA workflow that runs unit tests * Feature #3033 ascii2nc -inputrx (#3114) * remove doxygen files because they aren't used * remove commented code * fix formatting of use case docs that get flagged as SQ issues incorrectly * SQ: fix issues with Dockerfiles * SQ: replace unused vars with underscore * SQ: reduce cognitive complexity * SQ: rename CommandBuilder variable isOK to is_ok * SQ: various issues resolved * fix incorrect variable used in read_filters function * use env var value instead of hard-coded extension * SQ: various clean up to use case scripts to resolve SonarQube issues * fix format variable used for time series * remove check for success of calls to add_met_config_dict_list because it is not necessary * return False instead of None to be consistent with the other return values in the function * remove test that is no longer needed after removing status check * Per #3033, allow ascii2nc wrapper to write a file list file to handle use case that has a large number of input files * remove LOG_LEVEL that is not typically set in use case conf files * per #3033, add documentation for new config variable * add quotes around -inputrx argument * Per #3033, allow ascii2nc input template to be a directory to be searched. Do not write file list file if any of the inputs are a directory or python script. Do not create file list files for other ReformatPoint wrappers pb2nc, madis2nc, and point2grid * add new config to basic use case * Per #3033, update ISMN use case to use -inputrx and directories. Also clean up PROCESS_LIST to make it more readable * Per #3033, add unit tests to ensure command line arguments are set properly * fix syntax error using python 3.10 * Feature #3110 orography (#3113) * remove doxygen files because they aren't used * remove commented code * fix formatting of use case docs that get flagged as SQ issues incorrectly * SQ: fix issues with Dockerfiles * SQ: replace unused vars with underscore * SQ: reduce cognitive complexity * SQ: rename CommandBuilder variable isOK to is_ok * SQ: various issues resolved * fix incorrect variable used in read_filters function * use env var value instead of hard-coded extension * SQ: various clean up to use case scripts to resolve SonarQube issues * fix format variable used for time series * remove check for success of calls to add_met_config_dict_list because it is not necessary * return False instead of None to be consistent with the other return values in the function * remove test that is no longer needed after removing status check * Per #3110, add support for land_mask and topo_mask dictionaries in EnsembleStat wrapper * Per #3110, add support for lapse_rate_correction and msl_agl_conversion dictionaries for PointStat and EnsembleStat * clean up method doc * change check/error to an assert because it is meant to prevent a developer from calling the method with an incorrect argument * remove redundant call to parse_var_list and clean up formatting * add app name to call to parse_var_list so that FCST/OBS_STAT_ANALYSIS_VAR<n> variables are also supported * update test to reflect change in log message * Per #3310, add structure to support setting variables at a per-field basis instead of relying on *_VAR<n>_OPTIONS that require correct MET formatting. This structure provides the groundwork needed to complete issue #2357 * clean up formatting of field options * remove unused import * correct method doc * refactor to reduce cognitive complexity * Per #3110, add support for setting per field land_mask.flag, topo_mask.flag, topo_mask.use_obs_threhs, topo_mask.interp_fcst_thresh, lapse_rate_correction.apply_to, lapse_rate_correction.value, msl_agl_conversion.apply_to, msl_agl_conversion.apply_from, msl_agl_conversion.thresh, and msl_agl_conversion.msl_to_agl for both PointStat and EnsembleStat wrappers * add support for APP_NAME_FCST/OBS_* variables, e.g. GRID_STAT_FCST_INPUT_TEMPLATE, to start moving away from FCST/OBS_APP_NAME variables, e.g. FCST_GRID_STAT_INPUT_TEMPLATE -- The latter will still be supported and a warning will be output if both formats are used in a configuration * fix conflict resolution * Per #3110, added documentation for new config variables to set new configs per obs.field * change . to _ for METplus config names in helper script * Per #3110, add support for setting the interp dictionary within the topo_mask dictionary * enhance helper tool to support dictionaries that contain dictionaries * updated usage statement to include example of dictionary within dictionary * Per #3110, add documentation and tests for new config variables for topo_mask.interp dictionary in PointStat and EnsembleStat * Feature dtcenter/METplus-Internal#63 CVE scan workflows (#3119) * Per dtcenter/METplus-Internal#63, update workflows to use metplus-action-free-disk-space and metplus-action-scan-docker-images * Add info about custom METplus actions * do not cancel other scans if one fails * remove handling of logs directory because build logs are not saved and scan logs are handled by custom action * create unique names for artifacts for each version scanned * remove explicit fail-on-high in case we change the default to turn it on, change scan log artifact to use underscore instead of . to be consistent with other artifacts * change headers for custom METplus actions to be more descriptive * Apply suggestions from code review Co-authored-by: John Halley Gotway <johnhg@ucar.edu> --------- Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * Update the METplus versions after the MET v12.0.4 bugfix release and tweak the release guide instructions and development timeline. (#3121) * move output files to artifact instead of copy to prevent disk space issues * start writing release notes * update workflows to create output data artifacts from the output directory instead of copying the files to the artifact directory -- this prevents disk space issues when copying a large amount of output files which duplicates each file * correct MET Docker requirement for the METplus rc1 release * Feature #3122 v6.2.0-rc1 (#3123) * add more release notes * update the rc1 date * update the DockerHub 6.2.0-rc1 images and dates * change Credit to CREDIT * remove AWS existing build info because we no longer provide those images * update info about creating truth data for new main_vX.Y-ref branch * update version for next dev cycle * free disk space to prevent failure in docker image build/scan * Feature dtcenter/metplus-internal#64 develop (#3126) * Feature #3098 main_v6.2 diff_logic (#3128) * Per #3098, modify the diff logic slightly to handle headerless .tcst files and METdataio data files with the version in the second column. * Fix bug introduced with changes in this PR that cause an error if the content of the first line does not contain more than 1 space-separated items, e.g. file list files * Clean up regex: Use \d instead of [0-9] as recommended by SonarQube. Use \. for period character instead of . which is a wildcard character in regex. Support capital and lowercase V at start of version number in case version is formatted with a lower case in the future. Add documentation for method. * Use math.isclose instead of comparing floats. Initialize idx for print statement in case values_list is empty * Remove header argument because it isn't actually used. Add helper function to check and remove 'file_list' first line for comparison of equivalent file list files that do and do not contain the first line that has the string 'file_list' * fix bug where 4D list does not flatten to 2D properly with existing logic * Per #3098, added unit tests for diffing METdataio reformatter output and TC-Stat output (no header, different version number) * rename constant values used in tests to be capitalized * Fix requests/auth.py:36: DeprecationWarning: Non-string usernames will no longer be supported in Requests 3.0.0. Please convert the object you've passed in (None) to a string or bytes object in the near future to avoid problems * replace duplicate lines * Per #3098, add back logic to log the header value for columns that have a header value instead of just using the column index --------- Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * change URL to download data for DataIngest test to use DTC server to be more reliable * Updated METexpress component versions (#3131) * Feature #3112 RAL use cases (#3120) * Adding new RAL use case files * Updating documentation and some use case files * Testing * Running out of names * Fixing file name * Fixing File name again * Fixing old path * Updating documentation * Moving use cases to medium_range area * Adding more files * Adding documentation * Trying to fix documentation error * Fixing Documentation take 2 * Updating Valid_beg * short output paths by removing use case category and name * fix input data location to use data from PointStat use case * turn on new use cases to test * Update PointStat_fcstCredit_GFS_obsGDAS_6hrRealtime.py Fixing naming inconsistency * fix path to use case confs * Update PointStat_fcstCredit_GFS_obsGDAS_6hrRealtime.py Fixed another typo * clean up use case docs * turn off 1 use case because it ran without issue * change env to use for use case to use one that contains requests python package * turn off all use cases for PR * Fixing a directory typo * Changing medium_range description to start at 5 days * Fixing outdated path * add GHA annotations to help review testing workflow run * change i.e. to e.g. * fix formatting on lists of output files * fix/update list of quick search keywords * Fixed incorrect label in yaml file * Per #3112, add new use cases to the list of use cases that use the requests package * Per #3112, rename Credit to CREDIT throughout and spell out the unfamiliar acronym in the use case documentation. * Per #3112, update the thumbnail image. --------- Co-authored-by: Christina Kalb <kalb@seneca.rap.ucar.edu> Co-authored-by: Christina Kalb <kalb@ucar.edu> Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * Feature dtcenter/met#3243 release guide (#3124) * Update METplus Release Guide with lessons learned from the MET release. * Update branch handling instructions. * update METbaseimage version numbers for latest bugfix releases * update METplotpy bugfix version 3.0.2 for 6.0 * print x/y of pixel with difference * revert incorrect capitalization * Feature #3129 develop - resolve new SonarQube reliability (#3133) * resolve new SonarQube reliability findings * use numpy.isclose instead of math.isclose * Feature #3125 METplotpy release guide re: version numbers (#3137) * Per #3125, update METplotpy release guide steps about updating version number * correct URL for download page - metviewer and generic metplus urls were swapped * Bugfix #3141 develop - INIT_SEQ no matches (#3142) * Per #3141, add unit test to replicate bug * add missing pytest marks * Per #3141, return empty list instead of [0] when INIT_SEQ logic does not find any matching leads within min/max bounds * clean up formatting * SQ: reduce cognitive complexity * Hard-code the latest official release version so it is not assumed based on list of versions in the table. Added next development release to table so that METbaseimage 3.5.0 will be built in GHA using v3.5 label in workflow. Update Release Guide to note that latest version number should be updated in the component version file upon official release * Per #3103, resolving RTD warning * update base image version for 6.0 to 3.3.6 * update base image version for 6.1 and 6.2 to 3.4.6 * update base image version for 7.0 to 3.5.1 * skip diff for plots generated by script in use case because these plots have been sporadically reporting differeneces in a few pixels that are not noticeable with a visual review. These plots are not generated from any METplus tool, so being alerted of differences in these files does not alert us of any issues with the METplus logic that we are focusing on testing * Update the 6.0.2 component version to 6.0.3 (#3145) * Feature 3103 noaa install (#3107) * Per #3103, starting on instructions for installation on NOAA machines * Per #3103, updated to add MSU-HPC information * Per #3103, added conda environment installation information * Per #3103, adjust formatting and start on MET Installation * Per #3103, update formatting and finish MET installation * Per #3103, finish MET installation section and fix formatting. * Per #3103, added METplus, METdataio, METcalcpy,and METplotpy and modified formatting * Per #3103, resolve warning * Per #3103, adding in Installation Process section and correcting typos * Per #3103, correcting references to METplus with the appropriate METplus component. * Per #3103, correct typo * Per #3103, remove commas * Per #3103, added tables for METplus Analysis tools modulefile locations * Per #3103, made more generic with X.Y.Z instances * Per #3103, resolving error. * Per #3103, fixing formattin. * Per 3103, adding clarity for file modification * Per #3103, resolving RTD warning * Per 3103, updating instructions to add wget commands for sample data. * Per 3103, updating instructions to add wget commands for sample data. * Per #3103, combined sample data and data installation location sections. * fix incorrect heading in env var values and update helper script to generate correct heading for future additions * increase header for env vars from to ^^^^ and replace heading 'Environment variables in wrapped MET config' with bold text to avoid unnecessary extra depth, especially in PDF output * hotfix: continue to the next iteration of the loop if the time input is invalid to avoid referencing None values * Feature #3146 SonarQube issues/duplication for 7.0.0-beta1 (#3147) * reduce duplicate code by consolidating scripts into single script to be called with an argument * run modified use case to test that changes did not change results, ci-run-diff * turn off use case after confirming that change did not change results * add helper function to reduce cognitive complexity * Resolve SonarQube issues regarding Define a constant instead of duplicating this literal 'remove_quotes, uppercase' X times by creating a new term called constant which can be used to handle MET configs that need to be uppercase and no quotation marks, e.g. constants found in MET/data/config/ConfigConstants * update use case documentation to reflect merge of 2 python embedding scripts into 1 * add GEN_ENS_PROD_N_MEMBERS to User's Guide * update info about GenEnsProd control member and valid thresh config variables * Per #3103, resolving RTD warning * Per #3146, refactor to reduce cognitive complexity of function to below 15 * Apply suggestions from code review from @jprestop Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> --------- Co-authored-by: jprestop <jpresto@ucar.edu> * Revert "increase header for env vars from to ^^^^ and replace heading 'Environment variables in wrapped MET config' with bold text to avoid unnecessary extra depth, especially in PDF output" This reverts commit 08c74d26c744b7a047e8e4ad127c7725757c2405. * Doc-only change to the METplus Release Guide to update directions about the release docker images workflows. * Doc-only update to the Release Guide to correct the formatting of the tar_files.met-base-vX.Y.tgz file names. We had a dot where we needed a dash. * Doc-only change to correct the indentation for the Zenodo Instructions dropdowns. * fix typo of variable name * update latest coordinated release version to 6.2 * update METexpress version to latest bugfix release * Consolidate steps to announce release to only be included in Coordinated steps (not METplus official steps). Update RAL-IT request to install software to use correct location /nrit/ral (formerly /usr/local) * Copied existing_builds_template.rst to existing_builds.rst and changed X.Y to 7.0 * Ai documentation update (#3160) * Adding AIWP as a feature to search * Update GridStat_fcstCREDIT_GFS_obsGFS_6hrRealtime.py * adding AIWP to Point-Stat use case * Update GridStat_fcstCREDIT_GFS_obsGFS_6hrRealtime.py * Update PointStat_fcstCREDIT_GFS_obsGDAS_6hrRealtime.py * Update quicksearch.rst * Update PointStat_fcstCREDIT_GFS_obsGDAS_6hrRealtime.py * Update GridStat_fcstCREDIT_GFS_obsGFS_6hrRealtime.py * Update GridStat_fcstCREDIT_GFS_obsGFS_6hrRealtime.py * Update PointStat_fcstCREDIT_GFS_obsGDAS_6hrRealtime.py * move links to various guides to top of Release Guide * move heading with ToC to fix broken left-hand menu * Bugfix #3100 time in field info (#3115) * Per #3100, if var/field information differs for files gathered for the same run but with different time information (like running ONCE_PER_INIT_OR_VALID), then instead of failing an assert, replace the differing var item with a wildcard. This can occur when time information is specified in the field level, because each time will contain a different level value. The values stored in the var_list for all files appears to only be used to set output information like output_prefix, so it is more accurate to store a wildcard here so it is clear that there is not a single level value used to gather these data. * Per #3100, add unit test to recreate assertion failure * fix link to info about how to obtain Gempak utility * fix logic to handle differing var/field info * Per #3100, add unit test for case where time info is used in data field * Per #3100, move logic in SeriesAnalysis wrapper so it can be used by GridDiag and potentially other wrappers. This logic handles formatting field info to properly substitute time information for each input. Also set input_time_info for each input data read so that this information can be read elsewhere as needed -- this info differs from the time_info that is already stored because that value is specific to the output time and can contain a wildcard character to note multiple times, e.g. multiple leads or init times * Feature #3157 RegridDataPlane verification grid quotes (#3159) * Per #3157, update glossary entry for RegridDataPlane verification grid to note that the value can be a named grid or a grid specification string, not just a path to a file * Per #3157, add unit tests to demonstrate failure wrt verification grid values that already include quotation marks. * Per #3157, add call to remove_quotes to properly handle when user puts quotes around the input value * remove unused import * Add 'reporting: DTC Benchmarking' label to common_labels.txt. * Delete index.html files post wget data retrieval Remove unnecessary index.html files after data download. * rotate/update author list * Bugfix #3169 quotes around field name (#3170) * Per #3169, add tests to demonstrate bug described in the issue * clean up unit tests to reduce duplication * Per #3169, add properly handle quotes around field name * Per #3169, add more tests that expose another situation that does not properly handle quotes that should be addressed * Per #3169, properly handle quotes for field name when input is GRIB PDS * add a test that exposes a crash when GRIB PDS is configured but no thresholds are provided * Raise ValueError exception if misconfiguration where no thresholds are specified for PROB data in GRIB PDS instead of crashing with cryptic error message. Enhance test to check for the exception * update commit of coveralls GHA to fix permission failure * Feature #3171 conda documentation (#3172) * update commit of coveralls GHA to fix permission failure * fix typo in commit hash * rearrange badges and add conda and coverage badges * Per #3171, add information about conda install * change dateutil requirement to >= version because exact version is not actually required * add link and clean up wording * Per PR feedback from @jprestop, add information about creating a conda environemnt (if desired) and activating an environment before installing METplus. Also add supported platform and version information * rename example conda env name as requested by @jprestop * Bugfix #2983 develop - StatAnalysis sorted leads and model skip (#3179) * clean up tests to use common functionality to test wrapper runs and add expected command to compare * Per #2983, add tests to demonstrate bug * Per #2983, modified existing tests to ensure that _beg and _end values are computed properly when time values are listed out of order, while preserving the order that the items were specified by the user * Per #2983, add sort_list=False to calls to get_met_time_list when the order of the items specified by the user should be preserved. Also added sort_list=True, which is the default value, to all calls to get_met_time_list and get_delta_list so it is more clear that the lists returned by those calls are sorted * Add test to demonstrate bug with multiple grouped models as described in discussion #2982, fix bug by removing whitespace around model values that are formatted with quotation marks and a space after each comma * Add support for setting column, weight, ss_index_name, and ss_index_vld_thresh in MET config file -- these were set using config overrides in discussion #2982. Also added missing test for hss_ec_value * cleanup to reduce cognitive complexity and resolve linter complaints * added and modified tests to increase code coverage * change return value of function to prevent crash when wrapper is misconfigured -- a new unit test to ensure that the wrapper reports an error uncovered that it crashes instead of gracefully reporting the misconfiguration * cleanup formatting and update incorrect comments * add tests to increase code coverage and ensure correct results for other configurations * Fix diff_util_netcdf_count_masked (#3182) * Update diff_util.py to flag differences in NetCDF files based on the number of grid points with masked bad data values. * Update make_nc logic to define the fill_value attribute for the temp variable and add a test to confirm differing number of valid data. * Feature #3146 SonarQube / OTTI fixes - weather regime use cases (#3181) * remove unused variable * resolve 'Bad use of null-like value' issue * resolve 'Copy-paste error' OTTI issue and make script more efficient by reducing the number of file reads * remove function that is not used by the use cases * remove unused functions, reduce duplicate code, and reduce cognitive complexity * refactor to reduce cognitive complexity and implement better error checking to prevent null access * return z500 detrend value passed to function if EOF not computed instead of None, updated comment * use env var to easily change conda location in commands to match your local location * Fixing output directories * Per #3146, SQ: refactor to reduce cognitive complexity --------- Co-authored-by: Christina Kalb <kalb@ucar.edu> * hotfix: add verification dataset search keyword for GDAS prepbufr data * hotfix: fix link to search for use cases that use GDAS prepbufr data * Updated the development timeline for the 7.0.0 release * Do not fail security scan if critical CVEs are found. * Update the existing builds template * Updating existing builds page * Fixing italics problem* * Added Coming Soon label for Ursa * Update schedule as metplus-bot so that user receives weekly emails * Feature 2963 fire wx use cases (#3164) * Adding use case image * Adding MvMODE FireWx use case * Adding Fire Weather use cases * Updating use case image * Adding new use cases to the use case groups * Updating python embedding file * Updating documentation * Updating documentation and MTD use case * Adding HRRR use case * Adding documentation for Point-Stat use case * Updating use case groups, etc * Fixing some bad documentation links * Changing m/s to mph * Fixing weird link * Fixing file names * Adding some use case updates * Updating MvMODE case to use multiple thresholds * Updating images * Adding plotting yaml files * Fixing missing se…

* fix skip of any python except 3.10 and change env var name to prevent conflict with reserved env var used by METplus to set parm base * remove dependencies that don't need to be listed because they are made available by other dependencies * add MET license file * remove GS Fonts due to licensing issues and it is optional * prevent warning for numpy because it is actually used * add back libopenblas for linux only * Feature 2746 sfs use cases (#2954) * Adding Soil Moisture averaging case * Adding a new use case to compare soil moisture of SFS-GSL vs ERA5. The use case uses a python embedding file to read the SFS-GSL files. * Adding documentation to the soil moisture use case to compare SFS-GSL against ERA5 * Adding a thumbnail for the use case * Adding updated soil moisture use case * Updating title line * Adding Use Case image * Updating docs * Trying to fix use case image * Updating documentation * Updating documentation * Fixing documenation * Updating documentation * Updating directory * Adding use case data * Adding the s2s_soil_moisture directory * Updating documentation location * Adding Documentation README * Adding config override for actions * Fixing typo * Testing Config overrides * Testing new timeframe * Changing CI overrides * Renamed files and moved them to s2s_soil_moisture directory * Updated file to change ERA to ERA5Land * Moved file to the correct directory and updated * Changed file name * Fixed directory name * Fixed directory name * Fixed directory name * Added test for the soil moisture use case * Adding another use case * Setting tests to false for pull requests * Fixed image file name * Testing sonar cube change * Turning a test back on to check a change * Update sfs_gsl_model_wrapper.py * Testing a sonar cube update * Updated documentation as per new template * Updated documentation as per new template * Gallery path update * Update use_case_groups.json * Turning tests off for PR * Removed boilerplate language * Changing log level --------- Co-authored-by: Mrinal Biswas <biswas@ucar.edu> Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> * add example of using the allow_empty extra arg keyword * change minimum supported version of python to 3.10.4 because the wrappers use logic that requires that version and we are unable to run the unit tests for earlier versions in GitHub Actions * fix tests that check python versions * update function that checks if a value is a number to properly handle numpy masked values (MaskedConstant) * fix incorrect label * Feature 2904 tif fuse case (#2987) * first pass over doc, basic files. Needs config update and finalize doc * add rioxarray conda environment * update out-of-date info * added image, updated docs, attempt to resolve compile issue * removing space for error * updated remaining files, adjusted docs * fix incorrect naming for another use case that had errors in use case doc rendering * fix typos and formatting * fix path to thumbnail image * add py_embed keyword to tell automated test suite to set MET_PYTHON_EXE to the version of python in the rioxarray conda env * fix SonarQube complaint "Fix this attribute access on a value that can be 'None'." despite the function cannot return None because it will exit the script if an invalid input is provided --------- Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Hotfix to Release Guide instructions for MET to fix a minor formatting issue. * change version of sphinx-rtd-theme to a version that is available on conda-forge * Feature #2962 RMWAnalysis wrapper (#2985) * per #2962, begin implementation of RMWAnalysis wrapper * start basic use case conf * read BOTH_VAR<n> variables into data field * change pycharm python interpreter * ignore file generated by building docs locally * update use case to read input data * Per #2962, add documentation, unit tests, and basic use case for RMWAnalysis wrapper * per #2962, added thumbnail for use case documentation * fix incorrect formatting * per feedback from @JohnHalleyGotway, change 5 config variables to support lists of values instead of a single string * Bugfix #2890 develop - prevent incorrect error for skipped time (#2988) * per #2890, refactor Example wrapper to support all EXAMPLE_RUNTIME_FREQ settings to expand its use and reproduce the bug to be addressed * add docs for setting runtime freq for Example wrapper * prevent crash checking an init or valid time that is a wildcard * clean up function docs * Per #2890, add more checks for skipping times so wrappers don't incorrectly report an error if an init/valid time is skipped. Create function to handle skip checks, logging run time, and adding instance/custom to templates to reduce duplicate code. Change skip logging from DEBUG to INFO so it is more clear what is being skipped. Rename run_at_time function to be more clear of its purpose * update metplus_dev environment to include the documentation requirements * Per #2890, added unit tests for example wrapper to increase code coverage on new code. Improved error reporting if incorrect time configs are set * Feature #2984 v6.1.0-rc1 (#2991) * add release notes for rc1 * update DockerHub rc1 releases * fix indentation * Per #2984, updating title of issue 2863 --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * change deprecated maintainer to label * prevent GHA 'System.IO.IOException: No space left on device' error by removing data from original location as well * remove docker image after pushing to DockerHub to free up disk space * add new workflow to only run and update the truth data for use case groups that had differences in the last non-ref branch run. runs all use case groups if there is no previous non-ref run to handle first main_vX.Y-ref run before main_vX.Y has been created. also supports workflow dispatch to force update of all truth data as needed * remove problematic run-name logic because PR merge that will trigger the run has the information we need * remove logic to update truth data from testing workflow since it is now handled in update_truth_data.yml * rename job so workflow looks more similar to testing * rename workflow so it is clear that it updates the *-ref branch instead of actually updating the truth data on DockerHub * add release note for missing bugfix issue * push fix when no latest run for main_v6.1 branch exists * do not run anything in testing workflow for -ref branches * update version for development towards 6.2.0-rc1 * Feature 2932 use case amdar pbl (#2989) * Enhanced AMDAR PBLH METplus use case * Fixes title underline length error. * Adds missing .csv for airport filename in the conf file. * Reverts addition of .csv suffix since the Python embedding script adds it, and adjusts Python embedding script locations to use PARM_BASE instead of INPUT_BASE. * Changes the NumPy where statement to use the string nan instead of the NumPy nan constant to match the data type of the tailNumber column at this point. The string nan is used also on the subsequent line for valid_tails. --------- Co-authored-by: Daniel Adriaansen <dadriaan@ucar.edu> * Feature #2998 develop - updating truth data instructions (#3001) * per #2998, update instructions for updating truth data to note that the latest develop branch run needs to finish running. also cleaned up some out-of-date info * add note to delete the temporary branch generated for updating develop-ref * enhance pytest fixture so it will work with wrappers that don't have class variable WRAPPER_ENV_VAR_KEYS * clean up tests * Feature 2953 installation files (#3004) * Per #2953, update Orion file for Python 3.12 * Per #2953, updating Existing Builds page for installation on Orion * Per #2953, updating Casper file for Python 3.12 * Per #2953, adding script for METplus-6.1 using Python 3.12 * Per #2953, updated installation information for Casper * Per #2953, updated compiler for Orion * Per #2953 updating compiler in Existing Builds for Orion * Per #2953, added qhull needed for atlas on hera * Per #2953, remove qhull as it was actually causing problems on hera * Per #2953, updating hera file for Python 3.12 * Per #2953, updating with hera information * Per #2953, updating jet file for Python 3.12 * Per * Per #2953, adding internal/scripts/installation to paths-ignore * Per #2953, updating existing builds page. * Bugfix #2980 develop - PCPCombine 3 year accum bug (#3005) * per #2980, add test to recreate bug when building a 3 year accumulation using 1 year accums * Per #2980, fix bug in determining accumulation seconds used to find last valid time to check for an accumulation -- The search time minus the output accumulation is used as the reference time to compute the number of seconds of the input accumulation. The input accumulation is subtracted from this time to determine the number of seconds, which depending on the time can cross over a leap year when the actual accumulation calculations do not. This fix adds the input accumulation to the 'first' time to ensure that it is correctly calculated * Feature #3008 Support lead months/years in template substitution (#3009) * clean up code formatting * Per #3008, add support for months and years in {lead} filename template tags. Added tests to ensure correct behavior occurs. Rename format_hms function to be more clear of what it actually does now * reduce SonarQube findings by reducing cognitive complexity. Clean up documentation * Feature 2857 support (#3012) * Per #2857, updated User Support section and also modified case of other headers * Per #2857, resolved duplicate explicit target name warning. * Per 2857, added information about the team drive where data is uploaded to and labelling for organization or institution. * add swpc_metpy env to GHA workflow to create conda envs * pull changes from metplus-feedstock * reduce required python version to match metplus/PYTHON_VERSION_MIN * add met_tool_wrapper use case to existing group * Feature 2999 improve diffs for met tests (#3019) * added some temp code to check for .log files being diffed * removed checks for .log files / reverted * added some more detail to output, including number files * changed calc_transport.log to calc_transport.out and updated ref in docs * skip .log files * changes to diff_text_lines * added temp check for files without extensions * added temp check for files without extensions / fixed * don't compare files without extensions, e.g. file_list in MET tests * forgot to hit save after deleting a couple lines before the last commit... * improved handling of nc files * enhanced numeric diffs for nc files * implemented logic for setting tiny values to zero as pper MET feature 3134 * implemented diff checking on sig figs * slight tweak to order of equality checks * added debug line * more debugging * fixed _round_sig_figs * fixed bug in _set_zero * another fix * reinstated rounding precision for diffs after trying only sig fig rouding * bumped SIG_FIG for diffs to 7 * fix to _round_sig_figs * simplified _is_number() * some tidying * refactor/cleanup to remove code smells reported by SonarQube * more cleanup * added unit tests for skip file extensions logic and new equal comparison functions * add comments to describe each list of file extensions --------- Co-authored-by: Natalie Babij <nbabij@seneca.rap.ucar.edu> Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * add config variables for setting file_type in the climo_mean/stdev dictionaries (#3027) * Feature 2921 develop sphinx macports (#3025) * Per #2921, adding documentation for setting up a sphinx environment using MacPorts * Per #2921, adding context to explain why a developer may need to run these commands. * Feature #2575 TCI use case - generic model (#2768) * Generalizes METplus config file entries relevant to the FCST dataset used to compute TCI. * per #2575, rename variables, scripts, etc. to avoid using model-specific wording to encourage users to modify this use case to use other models * cleanup * merged (old) develop and resolved conflicts * fixed incorrect merge * update use case doc file to more closely match use case documentation template * replace bad quotation mark character and fix indentation * fix formatting * more formatting changes * fix formatting to match template for METplus Workflow section and moved use case specific config variable descriptions inside dropdown for python scripts * Removes errant include of Python embedding script, tidies up glossary RST, and adds special note under the dataset section that if a user has GRIB or other data without a time dimension, they need to somehow create a data object or file with a time dimension in order to use the fcst_tci Python embedding script since it assumes there is a time dimension on the forecast data. --------- Co-authored-by: Daniel Adriaansen <dadriaan@ucar.edu> * Feature #2980 refactor PCPCombine file search (#3017) * per #2980, refactor PCPCombine wrapper to find input files like other RuntimeFreq wrappers to be consistent. Clean up logic to remove duplicate code. Update pytests to match changes to wrapper * fix typo in name of temporary var list * clean up logic for SonarQube * handle if temp var (field) list is None or an empty list * fixed bug where USER_DEFINED method was reporting an error when it shouldn't be because it doesn't gather any files * refactored logic into function to reduce duplicate code * move call to set_environment_variables right before call to app so environment variables are not printed if the command will not be run * update ExtractTiles error count if regrid_data_plane has any errors so entire run fails from error * add unit test to capture bug where ExtractTiles is failing * fix bug from previous commit by settings output template/dir from FCST/OBS * update test to run just FCST and just OBS instead of always both * change input template config to legacy supported OBS_REGRID_DATA_PLANE_TEMPLATE (instead of OBS_REGRID_DATA_PLANE_INPUT_TEMPLATE) to reproduce bug in RegridDataPlane_multi_field_one_file use case * clean up formatting complaints * reduce cognitive complexity of function. use utility function instead of calling regex logic explicitly. make returns consistent * fix bug in RegridDataPlane -- this wrapper supports setting a generic FCST/OBS_REGRID_DATA_PLANE_TEMPLATE that can be used as both _INPUT_TEMPLATE and _OUTPUT_TEMPLATE. This is used in one of the met_tool_wrapper examples but was not supported by RuntimeFreq's get_input_templates, so I modified the function to look for generic TEMPLATE if INPUT_TEMPLATE was not set * reset level to be used in the output filename template substitution back to the output accumulation aka lookback seconds * Only require input template to be set for run methods other than USER_DEFINED and SUM. * add unit test for USER_DEFINED run method * refactor to reduce duplicate code * add pytest fixtures to reduce duplicate code in unit tests * improve/expand SUPPRESS_WARNINGS logic * add argument to skip logging when input files cannot be found and just return True/False * clean up formatting * refactor RegridDataPlane wrapper to use RuntimeFreq input file finding logic and remove functions that are no longer needed * update unit tests to match changes to wrapper * fix ExtractTiles to not report errors when files are not found (previously reported warnings) * reduce cognitive complexity * split copyable environment variable commands onto their own lines so it is easier to read and share with others for debugging purposes * refactor ExtractTiles wrapper to find files using RuntimeFreq logic since it calls RegridDataPlane to create the tiles. Update unit tests to reflect changes and ensure that failures are captured properly * clean up formatting * clean up formatting * Feature 3014 sfs updates (#3020) * Adding graphic * Updating python embedding script location * Updating environment * Updating environment * Removing bad path * Adding CI override * Updating documentation * Updating documenttion again * Take 3 * Another documentation update * Updating use case * Updating the order of processes * Updating documentation * Cleaning up unused variables * Updating documentation * Adding climatology. Testing will fail b/c haven't uploaded data * Adding Series Analysis climatology and updating documentation * Another doc update * Fixing valid_beg * fixing time * Fixing documentation typo * Fixed another typo * Fixing log verbosity and block size * Updating some code * Fixing typo * Testing removal of file_type * Removing file type * Setting tests to false for pull requests * minor grammar, spelling updates * Fixing increment * Fixing Log level * Update GridStat_fcstSFSGSL_obsERA5Land_SoilMoisture.conf * editing plot title --------- Co-authored-by: Christina Kalb <kalb@seneca.rap.ucar.edu> Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Feature 2723 use case sep climates (#3030) * added config and docs. Added use case to testing files. Still needs img, testing * updated config with file_type, use case img * Small grammar change --------- Co-authored-by: Christina Kalb <kalb@seneca.rap.ucar.edu> * Feature 2390 ctp hum index (#2981) * Initial Python embedding script for computing CTP from GDAS prepbufr radiosondes (obs). * Cleanup of extra debugging and development blocks. * Initial Python embedding script for computing HI from GDAS prepbufr radiosondes (obs). * Minor changes to print statements. * Major cleanup. * Removes check for highest pressure since that could vary. Check was moved to function in METcalcpy. * Conf file for this use case. * Python embedding script for FCST for CTP. * Changes for UserScript and GenVxMask. * Changes for function signature to take a station_id argument. * Python embedding script to compute Humidity Index from UFS/HR1 forecast data. * Script to create a lat/lon text file for GenVxMask to read and create a gridded mask file to use, based on radiosonde station locations in a GDAS prepBUFR file. * Updates to account for new station_id argument by calc_ctp function in METcalcpy. * Major changes to workflow using multiprocessing for computing CTP from UFS/HR1 forecast data. * Removes erroneous code from script that this script was copied from. * Removes unused code from development and print statements from development. * Removes print statements used during development. * Removes unused code from development and print statements from development. * Updated use case configuration file including config sections for each metric, GenVxMask options, and UserScript options. * Turns down verbosity for PointStat. * Adds print statement that the Python embedding script is starting for log purposes. * Adds print statement to log when it is starting to each Python embedding script, and also removes the interp argument from calc_humidity_index for obs to use the default value in METcalcpy. * Adds debug message for printing the station being processed, with the default turned off. * Major updates to documentation file for use case. * Updates to the UserScript to allow a user to control which sites are included in the mask file for GenVxMask. * Cleanup of use case config gile to remove unused items. * Switches to F strings, wraps some print statements in DEBUG, and switches to NumPy max/min. * Removes unused functions from forecast Python embedding scripts. * Switches to F strings and wraps some print statements in DEBUG flag. * Switches to F strings and adds script name to print statements. * Adds space around print statements to aid log file inspection. * Adds DEBUG level as a command line argument and wraps last remaining print statement in DEBUG flag. This can now be controlled via the METplus config file. * Adds controlling DEBUG level for observation Python embedding scripts. * Updates documentation URL in config file. * Changes subsequent processing steps to read from OUTPUT_DIR, adjusts for subset of global grid to demonstrate functionality, and adjusts input data paths to be relative to INPUT_BASE and the use case category structure for GHA. * Updates for sample data and running the use case in GHA. * Typo in the JSON file. * Changes the MET_PYTHON_DIR to the installed location rather than the source code location. * Adds user script to create graphic from PointStat MPR output. * Adds use case gallery image and updates observation dataset info in documentation. * Fixes path to config file include in documentation. * Adds new user script for plotting to the documentation file. * Updates list of expected output files to include UserScript outpout and PB2NC output. * Changes to eliminate code smells. * Finalizes forecast dataset description and scientific objective. * Adds info on additional Python packages required for Python embedding and UserScripts. * Removes extra #. * Removes uneccesary imports in user script. * Adds more clarity that METcalcpy is a required Python package for Python embedding. * Moves the setting of MET_PYTHON_EXE into the configuration file, as the user should ensure the version of Python in their environment at the time of running the use case meets the requirements for both Python embedding and Python UserScripts. * Removes MET_PYTHON_EXE since it will not be needed if a user has compiled MET against a version of Python that has the requirements for Python embedding. MET_PYTHON_EXE should be set in the users' local conf file, and as described in the documentation, simply be set to python3. * Adds updated note for users in the Running METplus section that provides guidance on how to ensure the correct Python is used for the various components (Python embedding, Python UserScripts). * prevent exception when trying to compare a masked (nan) value to a number * turn off use case * added workflow dispatch to workflow to allow rebuild of Docker images periodically to fix any security vulnerabilities in upstream images * added catch for log10(0) (#3043) Co-authored-by: Natalie Babij <nbabij@seneca.rap.ucar.edu> * Feature #2577 develop - WRF Hurricane Matthew use case (#3036) * Feature #2577 WRF Hurricane Matthew use case (#3015) * per #2577, add use case files * turn off use case that shouldn't be on * fix path to data and scripts * fix path to plotting scripts * added scripts to create wrf_plot conda env that contains the METplus Analysis requirements and the wrf package. change use case to use that env and obtain METdataio and METplotpy * fix name of wrf python package * override input and output directories in WRF plotting script * clean up script re: SonarQube * renaming variables for SonarQube * turn off 100-m wind speed plot because zlev files are not included in the input data * added missing closing single quote * add version of package that was obtained for reproducibility * refactor WRF plotting script to split logic into functions to reduce cognitive complexity * try creating output directory before running reformatter * METdataio reformatter does not accept environment variables in its config file, so attempting to change directory to output directory and find input data relative to that directory * change approach to reference plot directory * add WRF and MADIS file types to quick search keywords and start doc for new use case * fix formatting for output file lists and correct info about output directories (use case path are added for automated tests, but are relative to {OUTPUT_BASE} if run by a user) * write log file to logs directory and rename it to include .log extension * add more information for use case doc * turn off use case for PR * refactor map functions script to reduce cognitive complexity for SonarQube * update scientific objective with input from @jaredalee * update use case to reference environment variables in reformatted config file instead of using relative paths and changing directories since changes needed to support env vars in the reformatted are available in develop now * test that changes to use case did not break anything * Feature 2533 ismn use case (#3051) * Initial config file for this use case. * Adds ismn as an acceptable format for ASCI2NC_INPUT_FORMAT. * Latest param file changes. * Documentation file for this use case. * Updates documentation URL in config file. * Updated conf file that produces matched pairs for soil moisture from PointStat. * Updates to documentation for ISMN use case. * Adds UserScripts for plotting. * Latest conf file settings for use case. * Use case gallery image. * Adds summary using NEAREST for point_stat matched pairs. * Adds path to gallery image for use case. * Changes use case template to have the sphinx gallery line separate from the previous section to avoid having the actual sphinx_gallery_path appear in the rendered HTML documentation. Still need to fix use cases that don't have this change. * Per discussion with @georgemccabe and experience using the StatAnalysis Wrapper for the ISMN use case, moving two stat-analysis wrapper config items that were listed as required to the optional section since it worked without setting them. Presumably they may be required based on how a user is running stat-analysis or the stat-analysis wrapper, but they are not always required. * Per discussion with @georgemccabe, adds additional details for adding a new use case instructing the user to return the run setting to false in the use_case_groups.json file prior to merging the PR, which is the desired default setting for all use cases. * Minor change to use case documentation. * Initial work for adding ISMN to Verification Datasets guide. * Adds documentation for the ISMN dataset in the Verification Dataset Guide. * Adds VxDataISMN to use case documentation to link the use case to the verification datasets guide. * Adds new use case to testing suite after adding sample data. * Removes mention of RAL FTP site for sending sample data for a new use case, since that is not allowed anymore and generically refers to the GitHub Discussion describing the new process using Google Drive with a link to the Discussion. * Forgot a space. * Update to MET 12.0.3 bugfix version (#3059) * Update to MET 12.0.3 bugfix version. * Switch from using "secrets.DOCKER_USERNAME/secrets.DOCKER_PASSWORD" to using "dtcenter/secrets.DOCKER_TOKEN". * Feature #3054 develop scan_for_CVEs (#3055) * Per #3054, enhance release-docker-images.yml to support CVE scanning, scheduled weekly runs for one or more versions, and update the logic for pushing X.Y-latest tags to Docker Hub. * Per #3054, update the Release Guide with instructions about updating the GHA release workflows. * Per #3054, fix workflow typo. * Per #3054, update release guide formatting. * Per #3054, add CMD_LOGFILE logic to the time_command function. * Per #3054, refine release guide instructions. * Per #3054, refine release guide instructions. * Per #3054, update the versions in the release-docker-images.yml workflow for version 6.1.0. * Per #3054, update to plotly 6.1.1 and kaleido 1.0.0 versions. * Per #3054, revert changes to python package versions since those do not actually impact the GHA testing environment. * Per #3054, updating plotly version to 6.1.1 and python-kaledio version to 1.0.0 in the documentation and internal scripts. Hoping that will enable the 4 failing METplus use cases to succeed. * Replace metplotpy 3.0 with 3.0.1 under the METplus 6.0 release (#3060) * Replace metviewer 6.0.0 with 6.0.1 (#3065) * updates that were made to main_v6.1 to handle chrome dependency for METplotpy use cases * rotate authorship as was done in main_v6.1 branch * fix typo * update version of METexpress for 6.1 Coordinated Release * add look up for upcoming 6.2 coordinated release * Update the release guide with by adding a |downloadURL| variable since MET/METviewer/METexpress store downloads in a separate website location than the other components. (#3074) * update instructions to create a new project board to use templates (#3075) * update instructions to create a new project board to use templates * improve instructions for updating Zenodo * Apply suggestions from code review Co-authored-by: John Halley Gotway <johnhg@ucar.edu> --------- Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * develop docs v6.2.0 release timeline (#3083) * Update the development timeline for METplus version 6.2.0 and fix a minor error in the Release Guide insturctions for MET. * Removing accidental new file. * Feature #3068 DataIngest wrapper (#3084) * clean up * per #3068, add utility function to download files and added unit tests to confirm it works as expected * return from function is uncompress extension is not present * remove unnecessary close() calls because with handles closing of file handles * Log error and return from init function if c_dict was not initialized properly. This allows create_c_dict function to return None in the child class wrappers (not parent classes) * per #3068, create basic components of DataIngest wrapper * Add argument to getraw function to skip replacement of double slash with single slash. The replacement was added to prevent errors with paths that contain double slashes in MET config files, but this causes issues with URLs. * clean up to remove unnecessary variable * Rename DATA_INGEST_<n>_URL/LOCAL_PATH to INPUT/OUTPUT_TEMPLATE to be consistent with other wrappers. Fix incorrect has_option config function name. Improve logging so it is clear that a file is currently being downloaded. Added skip-if-output-exists logic * added basic use case for new wrapper * Handle gz extension as gzip file. Refactor new functions to reduce cognitive complexity * add option to auto decompress files before writing to disk * add use case documentation * add new use case to run in automated test suite * added config options for INPUT/OUTPUT_DIR to be consistent with other wrappers -- added workaround for getdir function to prevent replacement of double slash in URLs * fixed typo * added documentation for config variables for new wrapper * turn off new use case before creating PR * remove test that takes a very long time to download file * removed deprecated config variable * fixed bug in enhancement to config getdir to properly handle skip of double slash handling * improve logging for readability * change lead sequence in example because it is not used * Per #3068, add unit tests for new DataIngest wrapper * fix typo * refactor automated test scripts to reduce duplication in logic to run commands * improve logging so it is more clear that a diff test failed because it is likely that the truth data does not exist yet (e.g. for new use cases) and exit 2 to match return code for a use case that contains actual diffs in the output * Update the METplus-6.1 version of MET from 12.1.0 to 12.1.1 after the first bugfix release. (#3093) * Per #3087, add support for setting message_type_group_map in PB2NC and EnsembleStat wrappers (#3092) * update climo stdev variable name after input data was updated by @j-opatz * change sonar host url to env to view url for quality gate scan results * Fix FSTDEV typo in statistics_list.rst * Update release-notes.rst for the METplus-6.2 Release Schedule * Feature #3028 RMWAnalysis configs (#3094) * Per #3028, added support for many new rmw_analysis MET config options. Refactor logic to read METplus configs and handle MET config env vars to loop over names and handle them consistently. Reordered variables to match new order in RMWAnalysis_default config file * fix missing line break that caused warnings in doc build * change input URLs to obtain files from DTC web server because MADIS data changed since the truth data was created --------- Co-authored-by: MET Tools Test Account <met_test@seneca.rap.ucar.edu> * Feature #3069 Generate run times using templates (#3089) * removed unused imports * removed unused import * Remove loop_over_times_and_call function because time looping should be handled in RuntimeFreq wrapper. Report an error if a wrapper does not properly inherit from RuntimeFreq or implement its own version of run_all_times * update unit tests because CommandBuilder run_all_times now reports an error * added a fixture to create empty files for testing file I/O logic in the unit tests * cleanup to reduce cognitive complexity, remove unnecessary variable, and clean up whitespace for readability * Per #3069, create get_files_and_time_info function to reuse logic to find files using a directory and template for file window logic and new logic to generate list of times to run based on existing files on disk. Add logic to time_generator to support TIME_GENERATOR_INPUT_DIR/TEMPLATE to generate init or valid times to process. Add unit tests to ensure that correct, sorted list of init/valid times are generated. Add helper function for unit tests to ensure that the expected keys for each time dictionary are present and call helper function from tests that use time_generator to generate run times from beg/end/increment and time list. Added stub to implement similar logic to extract available forecast leads for each init/valid time * Per #3069, use TIME_GENERATOR_INPUT_DIR/TEMPLATE to get list of forecast leads that correspond to the current init or valid time being processed. Updated tests to include this functionality * clean up docstring comments * Per #3069, add tests to ensure that logic to allow multiple time generator dir/templates and get the intersection of the run times works as expected before implementing it * Per #3069, enhance time generator dir/template logic to support multiple templates and get the intersection of the run times * add test for a single directory with multiple templates and add helper functions to reduce duplication * whitespace * add function to intelligently split dir/template values so that the dir will contain the path without any template tags and the template will contain any filename template tags so users can include filename template tags in _INPUT_DIR variables and file window and time generator parsing will still work as expected. Added ample tests to ensure correct behavior * correct comments * update test to include full list of existing wrappers * clean up tests and add test for case that could occur in containers and breaks dir/template split logic * implement fix that properly splits dir/template when a template is used in the root (/) directory * check if either TIME_GENERATOR_INPUT_DIR or _TEMPLATE are set to execute logic to get runtimes based on file templates since logic to properly split the path into dir/template prevents the requirement of using both config variables * added helper function to reduce cognitive complexity and make code more readable * reduce duplication by creating variable to store test data templates that are referenced in multiple tests * Per #3069, add support for time generator templates that only include valid time (obs data) and properly filter init/lead times accordingly. * Per #3069, add documentation to describe how to use new functionality with examples * Per #3069, improve logging to be more clear which times were found from templates and when no forecast leads were found for a given init/valid time * Added more debug logging regarding files/times found from searching. Removed function that was only called once * update use case docs to use latest template content * per #3069, add new use case to demonstrate the use of the new functionality to generate run times based on file templates * allow Example wrapper to support multiple inputs, because why not * added quick search keyword for times from template functionality * update use case docs to match template * Filter analysis files in first pass only if looping by valid time. Clean up comments to be more clear/accurate * create helper function to reduce cognitive complexity * Per #3069, enhance template-based time generation to support {instance} and {custom} template tags. * reduce duplication * fix formatting that breaks in py 3.10 * Log number of times each command was run from each wrapper. Rename function to match its new purpose * refactor to reduce cognitive complexity * remove trailing semicolon from executable name in log * per PR feedback from @CPKalb, add missing METplus config variables that are ignored when using time generator input template * include instance name when logging report of successful/failed runs * group new use cases * hotfix for #3069 -- add support for using now/today template tags in the time generator template and allow TIME_GENERATOR_INPUT_DIR to be unset in favor of setting the entire template in TIME_GENERATOR_INPUT_TEMPLATE * SonarQube clean up for v6.2 (#3108) * remove doxygen files because they aren't used * remove commented code * fix formatting of use case docs that get flagged as SQ issues incorrectly * SQ: fix issues with Dockerfiles * SQ: replace unused vars with underscore * SQ: reduce cognitive complexity * SQ: rename CommandBuilder variable isOK to is_ok * SQ: various issues resolved * fix incorrect variable used in read_filters function * use env var value instead of hard-coded extension * SQ: various clean up to use case scripts to resolve SonarQube issues * fix format variable used for time series * remove check for success of calls to add_met_config_dict_list because it is not necessary * return False instead of None to be consistent with the other return values in the function * remove test that is no longer needed after removing status check * Feature 3041 appendix a release notes (#3101) * Per #3041, updating appendix A * Per #3041, updating release notes to reflect need to update Appendix A * update link to MET Python Embedding reqs to use latest instead of feature branch * Update docs/Release_Guide/release_steps/update_appendix_a_python.rst Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * Update docs/Release_Guide/release_steps/update_appendix_a_python.rst Co-authored-by: John Halley Gotway <johnhg@ucar.edu> --------- Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * Feature #3069 part 2 - requests instead of urllib (#3117) * update unit tests to download from another URL that is more reliable * Per #3069, refactor to use requests packages instead of urllib to increase download speed * add scripts to create new conda docker env that contains the requests package needed to run the DataIngest wrapper * install requests package for unit tests * error in init step if requests package is not installed to prevent failure when running DataIngest wrapper * add requests to workflow used to create conda envs in docker * added version number for package * use requests env for DataIngest use case * replace mix of spaces and tabs with just spaces * change long line to multiple * document info about requirements for python package * install requests package for sonarqube GHA workflow that runs unit tests * Feature #3033 ascii2nc -inputrx (#3114) * remove doxygen files because they aren't used * remove commented code * fix formatting of use case docs that get flagged as SQ issues incorrectly * SQ: fix issues with Dockerfiles * SQ: replace unused vars with underscore * SQ: reduce cognitive complexity * SQ: rename CommandBuilder variable isOK to is_ok * SQ: various issues resolved * fix incorrect variable used in read_filters function * use env var value instead of hard-coded extension * SQ: various clean up to use case scripts to resolve SonarQube issues * fix format variable used for time series * remove check for success of calls to add_met_config_dict_list because it is not necessary * return False instead of None to be consistent with the other return values in the function * remove test that is no longer needed after removing status check * Per #3033, allow ascii2nc wrapper to write a file list file to handle use case that has a large number of input files * remove LOG_LEVEL that is not typically set in use case conf files * per #3033, add documentation for new config variable * add quotes around -inputrx argument * Per #3033, allow ascii2nc input template to be a directory to be searched. Do not write file list file if any of the inputs are a directory or python script. Do not create file list files for other ReformatPoint wrappers pb2nc, madis2nc, and point2grid * add new config to basic use case * Per #3033, update ISMN use case to use -inputrx and directories. Also clean up PROCESS_LIST to make it more readable * Per #3033, add unit tests to ensure command line arguments are set properly * fix syntax error using python 3.10 * Feature #3110 orography (#3113) * remove doxygen files because they aren't used * remove commented code * fix formatting of use case docs that get flagged as SQ issues incorrectly * SQ: fix issues with Dockerfiles * SQ: replace unused vars with underscore * SQ: reduce cognitive complexity * SQ: rename CommandBuilder variable isOK to is_ok * SQ: various issues resolved * fix incorrect variable used in read_filters function * use env var value instead of hard-coded extension * SQ: various clean up to use case scripts to resolve SonarQube issues * fix format variable used for time series * remove check for success of calls to add_met_config_dict_list because it is not necessary * return False instead of None to be consistent with the other return values in the function * remove test that is no longer needed after removing status check * Per #3110, add support for land_mask and topo_mask dictionaries in EnsembleStat wrapper * Per #3110, add support for lapse_rate_correction and msl_agl_conversion dictionaries for PointStat and EnsembleStat * clean up method doc * change check/error to an assert because it is meant to prevent a developer from calling the method with an incorrect argument * remove redundant call to parse_var_list and clean up formatting * add app name to call to parse_var_list so that FCST/OBS_STAT_ANALYSIS_VAR<n> variables are also supported * update test to reflect change in log message * Per #3310, add structure to support setting variables at a per-field basis instead of relying on *_VAR<n>_OPTIONS that require correct MET formatting. This structure provides the groundwork needed to complete issue #2357 * clean up formatting of field options * remove unused import * correct method doc * refactor to reduce cognitive complexity * Per #3110, add support for setting per field land_mask.flag, topo_mask.flag, topo_mask.use_obs_threhs, topo_mask.interp_fcst_thresh, lapse_rate_correction.apply_to, lapse_rate_correction.value, msl_agl_conversion.apply_to, msl_agl_conversion.apply_from, msl_agl_conversion.thresh, and msl_agl_conversion.msl_to_agl for both PointStat and EnsembleStat wrappers * add support for APP_NAME_FCST/OBS_* variables, e.g. GRID_STAT_FCST_INPUT_TEMPLATE, to start moving away from FCST/OBS_APP_NAME variables, e.g. FCST_GRID_STAT_INPUT_TEMPLATE -- The latter will still be supported and a warning will be output if both formats are used in a configuration * fix conflict resolution * Per #3110, added documentation for new config variables to set new configs per obs.field * change . to _ for METplus config names in helper script * Per #3110, add support for setting the interp dictionary within the topo_mask dictionary * enhance helper tool to support dictionaries that contain dictionaries * updated usage statement to include example of dictionary within dictionary * Per #3110, add documentation and tests for new config variables for topo_mask.interp dictionary in PointStat and EnsembleStat * Feature dtcenter/METplus-Internal#63 CVE scan workflows (#3119) * Per dtcenter/METplus-Internal#63, update workflows to use metplus-action-free-disk-space and metplus-action-scan-docker-images * Add info about custom METplus actions * do not cancel other scans if one fails * remove handling of logs directory because build logs are not saved and scan logs are handled by custom action * create unique names for artifacts for each version scanned * remove explicit fail-on-high in case we change the default to turn it on, change scan log artifact to use underscore instead of . to be consistent with other artifacts * change headers for custom METplus actions to be more descriptive * Apply suggestions from code review Co-authored-by: John Halley Gotway <johnhg@ucar.edu> --------- Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * Update the METplus versions after the MET v12.0.4 bugfix release and tweak the release guide instructions and development timeline. (#3121) * move output files to artifact instead of copy to prevent disk space issues * start writing release notes * update workflows to create output data artifacts from the output directory instead of copying the files to the artifact directory -- this prevents disk space issues when copying a large amount of output files which duplicates each file * correct MET Docker requirement for the METplus rc1 release * Feature #3122 v6.2.0-rc1 (#3123) * add more release notes * update the rc1 date * update the DockerHub 6.2.0-rc1 images and dates * change Credit to CREDIT * remove AWS existing build info because we no longer provide those images * update info about creating truth data for new main_vX.Y-ref branch * update version for next dev cycle * free disk space to prevent failure in docker image build/scan * Feature dtcenter/metplus-internal#64 develop (#3126) * Feature #3098 main_v6.2 diff_logic (#3128) * Per #3098, modify the diff logic slightly to handle headerless .tcst files and METdataio data files with the version in the second column. * Fix bug introduced with changes in this PR that cause an error if the content of the first line does not contain more than 1 space-separated items, e.g. file list files * Clean up regex: Use \d instead of [0-9] as recommended by SonarQube. Use \. for period character instead of . which is a wildcard character in regex. Support capital and lowercase V at start of version number in case version is formatted with a lower case in the future. Add documentation for method. * Use math.isclose instead of comparing floats. Initialize idx for print statement in case values_list is empty * Remove header argument because it isn't actually used. Add helper function to check and remove 'file_list' first line for comparison of equivalent file list files that do and do not contain the first line that has the string 'file_list' * fix bug where 4D list does not flatten to 2D properly with existing logic * Per #3098, added unit tests for diffing METdataio reformatter output and TC-Stat output (no header, different version number) * rename constant values used in tests to be capitalized * Fix requests/auth.py:36: DeprecationWarning: Non-string usernames will no longer be supported in Requests 3.0.0. Please convert the object you've passed in (None) to a string or bytes object in the near future to avoid problems * replace duplicate lines * Per #3098, add back logic to log the header value for columns that have a header value instead of just using the column index --------- Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * change URL to download data for DataIngest test to use DTC server to be more reliable * Updated METexpress component versions (#3131) * Feature #3112 RAL use cases (#3120) * Adding new RAL use case files * Updating documentation and some use case files * Testing * Running out of names * Fixing file name * Fixing File name again * Fixing old path * Updating documentation * Moving use cases to medium_range area * Adding more files * Adding documentation * Trying to fix documentation error * Fixing Documentation take 2 * Updating Valid_beg * short output paths by removing use case category and name * fix input data location to use data from PointStat use case * turn on new use cases to test * Update PointStat_fcstCredit_GFS_obsGDAS_6hrRealtime.py Fixing naming inconsistency * fix path to use case confs * Update PointStat_fcstCredit_GFS_obsGDAS_6hrRealtime.py Fixed another typo * clean up use case docs * turn off 1 use case because it ran without issue * change env to use for use case to use one that contains requests python package * turn off all use cases for PR * Fixing a directory typo * Changing medium_range description to start at 5 days * Fixing outdated path * add GHA annotations to help review testing workflow run * change i.e. to e.g. * fix formatting on lists of output files * fix/update list of quick search keywords * Fixed incorrect label in yaml file * Per #3112, add new use cases to the list of use cases that use the requests package * Per #3112, rename Credit to CREDIT throughout and spell out the unfamiliar acronym in the use case documentation. * Per #3112, update the thumbnail image. --------- Co-authored-by: Christina Kalb <kalb@seneca.rap.ucar.edu> Co-authored-by: Christina Kalb <kalb@ucar.edu> Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * Feature dtcenter/met#3243 release guide (#3124) * Update METplus Release Guide with lessons learned from the MET release. * Update branch handling instructions. * update METbaseimage version numbers for latest bugfix releases * update METplotpy bugfix version 3.0.2 for 6.0 * print x/y of pixel with difference * revert incorrect capitalization * Feature #3129 develop - resolve new SonarQube reliability (#3133) * resolve new SonarQube reliability findings * use numpy.isclose instead of math.isclose * Feature #3125 METplotpy release guide re: version numbers (#3137) * Per #3125, update METplotpy release guide steps about updating version number * correct URL for download page - metviewer and generic metplus urls were swapped * Bugfix #3141 develop - INIT_SEQ no matches (#3142) * Per #3141, add unit test to replicate bug * add missing pytest marks * Per #3141, return empty list instead of [0] when INIT_SEQ logic does not find any matching leads within min/max bounds * clean up formatting * SQ: reduce cognitive complexity * Hard-code the latest official release version so it is not assumed based on list of versions in the table. Added next development release to table so that METbaseimage 3.5.0 will be built in GHA using v3.5 label in workflow. Update Release Guide to note that latest version number should be updated in the component version file upon official release * Per #3103, resolving RTD warning * update base image version for 6.0 to 3.3.6 * update base image version for 6.1 and 6.2 to 3.4.6 * update base image version for 7.0 to 3.5.1 * skip diff for plots generated by script in use case because these plots have been sporadically reporting differeneces in a few pixels that are not noticeable with a visual review. These plots are not generated from any METplus tool, so being alerted of differences in these files does not alert us of any issues with the METplus logic that we are focusing on testing * Update the 6.0.2 component version to 6.0.3 (#3145) * Feature 3103 noaa install (#3107) * Per #3103, starting on instructions for installation on NOAA machines * Per #3103, updated to add MSU-HPC information * Per #3103, added conda environment installation information * Per #3103, adjust formatting and start on MET Installation * Per #3103, update formatting and finish MET installation * Per #3103, finish MET installation section and fix formatting. * Per #3103, added METplus, METdataio, METcalcpy,and METplotpy and modified formatting * Per #3103, resolve warning * Per #3103, adding in Installation Process section and correcting typos * Per #3103, correcting references to METplus with the appropriate METplus component. * Per #3103, correct typo * Per #3103, remove commas * Per #3103, added tables for METplus Analysis tools modulefile locations * Per #3103, made more generic with X.Y.Z instances * Per #3103, resolving error. * Per #3103, fixing formattin. * Per 3103, adding clarity for file modification * Per #3103, resolving RTD warning * Per 3103, updating instructions to add wget commands for sample data. * Per 3103, updating instructions to add wget commands for sample data. * Per #3103, combined sample data and data installation location sections. * fix incorrect heading in env var values and update helper script to generate correct heading for future additions * increase header for env vars from to ^^^^ and replace heading 'Environment variables in wrapped MET config' with bold text to avoid unnecessary extra depth, especially in PDF output * hotfix: continue to the next iteration of the loop if the time input is invalid to avoid referencing None values * Feature #3146 SonarQube issues/duplication for 7.0.0-beta1 (#3147) * reduce duplicate code by consolidating scripts into single script to be called with an argument * run modified use case to test that changes did not change results, ci-run-diff * turn off use case after confirming that change did not change results * add helper function to reduce cognitive complexity * Resolve SonarQube issues regarding Define a constant instead of duplicating this literal 'remove_quotes, uppercase' X times by creating a new term called constant which can be used to handle MET configs that need to be uppercase and no quotation marks, e.g. constants found in MET/data/config/ConfigConstants * update use case documentation to reflect merge of 2 python embedding scripts into 1 * add GEN_ENS_PROD_N_MEMBERS to User's Guide * update info about GenEnsProd control member and valid thresh config variables * Per #3103, resolving RTD warning * Per #3146, refactor to reduce cognitive complexity of function to below 15 * Apply suggestions from code review from @jprestop Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> --------- Co-authored-by: jprestop <jpresto@ucar.edu> * Revert "increase header for env vars from to ^^^^ and replace heading 'Environment variables in wrapped MET config' with bold text to avoid unnecessary extra depth, especially in PDF output" This reverts commit 08c74d26c744b7a047e8e4ad127c7725757c2405. * Doc-only change to the METplus Release Guide to update directions about the release docker images workflows. * Doc-only update to the Release Guide to correct the formatting of the tar_files.met-base-vX.Y.tgz file names. We had a dot where we needed a dash. * Doc-only change to correct the indentation for the Zenodo Instructions dropdowns. * fix typo of variable name * update latest coordinated release version to 6.2 * update METexpress version to latest bugfix release * Consolidate steps to announce release to only be included in Coordinated steps (not METplus official steps). Update RAL-IT request to install software to use correct location /nrit/ral (formerly /usr/local) * Copied existing_builds_template.rst to existing_builds.rst and changed X.Y to 7.0 * Ai documentation update (#3160) * Adding AIWP as a feature to search * Update GridStat_fcstCREDIT_GFS_obsGFS_6hrRealtime.py * adding AIWP to Point-Stat use case * Update GridStat_fcstCREDIT_GFS_obsGFS_6hrRealtime.py * Update PointStat_fcstCREDIT_GFS_obsGDAS_6hrRealtime.py * Update quicksearch.rst * Update PointStat_fcstCREDIT_GFS_obsGDAS_6hrRealtime.py * Update GridStat_fcstCREDIT_GFS_obsGFS_6hrRealtime.py * Update GridStat_fcstCREDIT_GFS_obsGFS_6hrRealtime.py * Update PointStat_fcstCREDIT_GFS_obsGDAS_6hrRealtime.py * move links to various guides to top of Release Guide * move heading with ToC to fix broken left-hand menu * Bugfix #3100 time in field info (#3115) * Per #3100, if var/field information differs for files gathered for the same run but with different time information (like running ONCE_PER_INIT_OR_VALID), then instead of failing an assert, replace the differing var item with a wildcard. This can occur when time information is specified in the field level, because each time will contain a different level value. The values stored in the var_list for all files appears to only be used to set output information like output_prefix, so it is more accurate to store a wildcard here so it is clear that there is not a single level value used to gather these data. * Per #3100, add unit test to recreate assertion failure * fix link to info about how to obtain Gempak utility * fix logic to handle differing var/field info * Per #3100, add unit test for case where time info is used in data field * Per #3100, move logic in SeriesAnalysis wrapper so it can be used by GridDiag and potentially other wrappers. This logic handles formatting field info to properly substitute time information for each input. Also set input_time_info for each input data read so that this information can be read elsewhere as needed -- this info differs from the time_info that is already stored because that value is specific to the output time and can contain a wildcard character to note multiple times, e.g. multiple leads or init times * Feature #3157 RegridDataPlane verification grid quotes (#3159) * Per #3157, update glossary entry for RegridDataPlane verification grid to note that the value can be a named grid or a grid specification string, not just a path to a file * Per #3157, add unit tests to demonstrate failure wrt verification grid values that already include quotation marks. * Per #3157, add call to remove_quotes to properly handle when user puts quotes around the input value * remove unused import * Add 'reporting: DTC Benchmarking' label to common_labels.txt. * Delete index.html files post wget data retrieval Remove unnecessary index.html files after data download. * rotate/update author list * Bugfix #3169 quotes around field name (#3170) * Per #3169, add tests to demonstrate bug described in the issue * clean up unit tests to reduce duplication * Per #3169, add properly handle quotes around field name * Per #3169, add more tests that expose another situation that does not properly handle quotes that should be addressed * Per #3169, properly handle quotes for field name when input is GRIB PDS * add a test that exposes a crash when GRIB PDS is configured but no thresholds are provided * Raise ValueError exception if misconfiguration where no thresholds are specified for PROB data in GRIB PDS instead of crashing with cryptic error message. Enhance test to check for the exception * update commit of coveralls GHA to fix permission failure * Feature #3171 conda documentation (#3172) * update commit of coveralls GHA to fix permission failure * fix typo in commit hash * rearrange badges and add conda and coverage badges * Per #3171, add information about conda install * change dateutil requirement to >= version because exact version is not actually required * add link and clean up wording * Per PR feedback from @jprestop, add information about creating a conda environemnt (if desired) and activating an environment before installing METplus. Also add supported platform and version information * rename example conda env name as requested by @jprestop * Bugfix #2983 develop - StatAnalysis sorted leads and model skip (#3179) * clean up tests to use common functionality to test wrapper runs and add expected command to compare * Per #2983, add tests to demonstrate bug * Per #2983, modified existing tests to ensure that _beg and _end values are computed properly when time values are listed out of order, while preserving the order that the items were specified by the user * Per #2983, add sort_list=False to calls to get_met_time_list when the order of the items specified by the user should be preserved. Also added sort_list=True, which is the default value, to all calls to get_met_time_list and get_delta_list so it is more clear that the lists returned by those calls are sorted * Add test to demonstrate bug with multiple grouped models as described in discussion #2982, fix bug by removing whitespace around model values that are formatted with quotation marks and a space after each comma * Add support for setting column, weight, ss_index_name, and ss_index_vld_thresh in MET config file -- these were set using config overrides in discussion #2982. Also added missing test for hss_ec_value * cleanup to reduce cognitive complexity and resolve linter complaints * added and modified tests to increase code coverage * change return value of function to prevent crash when wrapper is misconfigured -- a new unit test to ensure that the wrapper reports an error uncovered that it crashes instead of gracefully reporting the misconfiguration * cleanup formatting and update incorrect comments * add tests to increase code coverage and ensure correct results for other configurations * Fix diff_util_netcdf_count_masked (#3182) * Update diff_util.py to flag differences in NetCDF files based on the number of grid points with masked bad data values. * Update make_nc logic to define the fill_value attribute for the temp variable and add a test to confirm differing number of valid data. * Feature #3146 SonarQube / OTTI fixes - weather regime use cases (#3181) * remove unused variable * resolve 'Bad use of null-like value' issue * resolve 'Copy-paste error' OTTI issue and make script more efficient by reducing the number of file reads * remove function that is not used by the use cases * remove unused functions, reduce duplicate code, and reduce cognitive complexity * refactor to reduce cognitive complexity and implement better error checking to prevent null access * return z500 detrend value passed to function if EOF not computed instead of None, updated comment * use env var to easily change conda location in commands to match your local location * Fixing output directories * Per #3146, SQ: refactor to reduce cognitive complexity --------- Co-authored-by: Christina Kalb <kalb@ucar.edu> * hotfix: add verification dataset search keyword for GDAS prepbufr data * hotfix: fix link to search for use cases that use GDAS prepbufr data * Updated the development timeline for the 7.0.0 release * Do not fail security scan if critical CVEs are found. * Update the existing builds template * Updating existing builds page * Fixing italics problem* * Added Coming Soon label for Ursa * Update schedule as metplus-bot so that user receives weekly emails * Feature 2963 fire wx use cases (#3164) * Adding use case image * Adding MvMODE FireWx use case * Adding Fire Weather use cases * Updating use case image * Adding new use cases to the use case groups * Updating python embedding file * Updating documentation * Updating documentation and MTD use case * Adding HRRR use case * Adding documentation for Point-Stat use case * Updating use case groups, etc * Fixing some bad documentation links * Changing m/s to mph * Fixing weird link * Fixing file names * Adding some use case updates * Updating MvMODE case to use multiple thresholds * Updating images * Adding plotting yaml files * Fixing missing semicolon * Updating documentation * Fixing indent * Turning on one of the tests * MTD use case testing * Testing PointStat use case * Testing MvMODE use case * Removing CI override because it's behaving oddly * Updating MvMODE case to run all steps * Reducing automated testing for fire2 * Adding py_embed to use case group * Documentation corrections and updates * Fixing list error * Another documentation update * Documentation updates * Shortening testing for Point-Stat case * Changes * Updating contributors guide * Updating contributors guide * Updates * Updating a path to be more consistent * removing commented out line * Turning tests off for PR * SonarQube Sucks * Updating path * Fixed incorrect threshold * updated language, spelling --------- Co-authored-by: Christina Kalb <kalb@seneca.rap.ucar.edu> Co-authored-by: j-opatz <jopatz@ucar.edu> * update METviewer version after v6.2.1 bugfix release * update base image versions * update METbaseimage version * remove support for v6.0 and add v6.2 * hotfix: combine fire use cases 0-3 into single run in automated tests * hotfix: fix SQ complaint about unreachable code * add note for how to handle adding new data for multiple use cases at once * Updated METexpress version (#3198) * Update metbaseimage version to 3.5.3 and 3.4.8 * Extract logic that handles MTD file list file naming based on var name or python embedding from check_for_python_embedding function because the function did multiple things and its purpose was confusing. The check_for_python_embedding function now only sets the file_type if the input is a python embedding script, the MET tool supports multiple python inputs, and the file_type was not explicitly specified by the user. The prevents an error because some tools require …

Christina Kalb and others added 30 commits May 9, 2025 22:56

Adding use case image

c4cb2fa

Merge branch 'develop' into feature_2963_fire_wx_use_cases

de6d51b

Adding MvMODE FireWx use case

934ccf4

Adding Fire Weather use cases

0a03edf

Updating use case image

e5d03cf

Adding new use cases to the use case groups

780e75d

Updating python embedding file

33b017b

Updating documentation

35548de

Updating documentation and MTD use case

bac6726

Merge branch 'develop' into feature_2963_fire_wx_use_cases

64f9ee5

Adding HRRR use case

edc36ed

Merge branch 'develop' into feature_2963_fire_wx_use_cases

020e227

Adding documentation for Point-Stat use case

61f55de

Updating use case groups, etc

17f989d

Fixing some bad documentation links

e35cdc6

Changing m/s to mph

905269b

Fixing weird link

398e459

Fixing file names

7dfc9a3

Adding some use case updates

099ee95

Updating MvMODE case to use multiple thresholds

ef7f529

Updating images

250a652

Adding plotting yaml files

ba5a147

Fixing missing semicolon

4d60f71

Updating documentation

742ef22

Fixing indent

742688c

Turning on one of the tests

ff924ed

MTD use case testing

158a469

Testing PointStat use case

26d4140

Testing MvMODE use case

1f0ee3f

Removing CI override because it's behaving oddly

732c2ce

CPKalb and others added 12 commits November 12, 2025 10:08

Adding py_embed to use case group

7532421

Documentation corrections and updates

ac31101

Fixing list error

00293a0

Another documentation update

e3c3831

Documentation updates

eda0d66

Shortening testing for Point-Stat case

4d39cc9

Changes

eb74e35

Updating contributors guide

4cf134a

Updating contributors guide

408252d

Updates

c7998f9

Updating a path to be more consistent

29b4d06

removing commented out line

38f62e1

CPKalb linked an issue Nov 26, 2025 that may be closed by this pull request

New Use Case: Fire Weather Use cases #2963

Closed

26 tasks

Christina Kalb added 2 commits November 26, 2025 22:29

Turning tests off for PR

2b8c5de

SonarQube Sucks

20a3a70

CPKalb requested a review from j-opatz November 26, 2025 22:40

CPKalb added this to METplus-13.0 Development Nov 26, 2025

github-project-automation Bot moved this to 🩺 Needs Triage in METplus-13.0 Development Nov 26, 2025

CPKalb moved this from 🩺 Needs Triage to 🔎 In review in METplus-13.0 Development Nov 26, 2025

CPKalb added this to the METplus-7.0.0 milestone Nov 26, 2025

Christina Kalb and others added 3 commits November 26, 2025 22:53

Updating path

50665a1

Fixed incorrect threshold

d8a49f5

updated language, spelling

499a732

j-opatz approved these changes Jan 12, 2026

View reviewed changes

CPKalb merged commit 64b9ca3 into develop Jan 15, 2026
77 of 83 checks passed

github-project-automation Bot moved this from 🔎 In review to 🏁 Done in METplus-13.0 Development Jan 15, 2026

metplus-bot added a commit that referenced this pull request Jan 17, 2026

added entry to update truth change log: develop #3164

8d408a0

Navigation Menu

Uh oh!

Feature 2963 fire wx use cases - #3164

Feature 2963 fire wx use cases#3164
CPKalb merged 49 commits into
developfrom
feature_2963_fire_wx_use_cases

CPKalb commented Nov 26, 2025 •

edited

Loading

Uh oh!

j-opatz left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

CPKalb commented Nov 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Testing

Pull Request Checklist

Uh oh!

j-opatz left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

CPKalb commented Nov 26, 2025 •

edited

Loading