[develop] feat: additional MM evaluation packages (#1336)

* Adds additional evaluation packages (ISH/MET, AQS PM, and AQS VOC) to the MELODIES MONET post-processing.
* Metatask created for prep as there is heavier data preparation tasks in ISH/MET and AQS PM. Breaking them into separate tasks.

---------

Co-authored-by: Michael Lueken <63728921+MichaelLueken@users.noreply.github.com>

Author: benkozi

doc/UsersGuide/BuildingRunningTesting/AQM.rst

@@ -386,7 +386,7 @@ Once data is appropriately staged, the use case workflow configuration file may
    * - ``cpl_aqm_parm.USE_FIX_AQM_S3_DATA_STAGE``
      - Defaults to false. Set to true if fixed data was downloaded to the stage directory.
 
-MM (MM) Evaluation
+MELODIES MONET (MM) Evaluation
 ================================
 
 SRW-AQM provides an optional task group leveraging `MELODIES MONET <https://melodies-monet.readthedocs.io/en/stable/>`__ for model evaluation.
@@ -400,95 +400,12 @@ SRW-AQM provides an optional task group leveraging `MELODIES MONET <https://melo
 How to Run the MM Evaluation
 ------------------------------
 
-To run the evaluation suite, a user will need to follow these steps. Depending on the MM packages, tasks, and forecast window duration, the jobs can be computationally demanding. Per the usual HPC recommendations, start small and scale as needed.
-
-Install the ``aqm-eval`` Anaconda environment
-+++++++++++++++++++++++++++++++++++++++++++++++
-
-#. Clone the ``AQM-Eval`` repository: ``git clone -b main https://github.com/NOAA-EPIC/AQM-Eval.git``.
-#. Activate the ``srw_app`` Anaconda environment.
-#. Create the ``aqm-eval`` environment: ``cd AQM-Eval && conda create -f environment.yml && conda run -n aqm-eval pip install .``
-
-Stage Observational Datasets
-++++++++++++++++++++++++++++++
-
-For SRW-AQM use cases, the `AQM-Eval Data Sync utility <https://github.com/NOAA-EPIC/AQM-Eval?tab=readme-ov-file#installation>`__ utility is recommended to quickly stage all required observations.
-
-.. code-block:: console
-
-    $ conda run -n aqm-eval aqm-data-sync observations --dst-dir <path to download location>
-
-MM also offers utility to download and prepare observational datasets. See `Downloading Observations <https://melodies-monet.readthedocs.io/en/stable/getting_started/downloading_obs.html>`__ for more information.
-
-Check Availability of ICs and LBCs
-++++++++++++++++++++++++++++++++++++
-
-For the evaulation to work properly, at least a 48-hour forecast is required. It may be necessary to download additional IC and LBC datasets. See :ref:`acquire-use-case-data`.
-
-Enable the MM Workflow Task Group
-+++++++++++++++++++++++++++++++++++
-
-Under ``workflow.tasksgroups`` in the experiment configuration, add or uncomment ``- parm/wflow/aqm_post_melodies_monet.yaml``.
-
-Configure Paths to Observational Datasets
-+++++++++++++++++++++++++++++++++++++++++++
-
-For AirNow, set ``task_mm_prep.MM_OBS_AIRNOW_FN_TEMPLATE`` to the appropriate path. Wildcards may be used it the experiment's forecast window extends beyond a month.
-
-Configure "Scorecard" Base Model (Optional)
-+++++++++++++++++++++++++++++++++++++++++++++
-
-MM provides a set of `scorecard evaluations <https://melodies-monet.readthedocs.io/en/stable/users_guide/supported_plots.html#id8>`__ used to compare a base and evaluation model run. Setting ``task_mm_prep.MM_BASE_MODEL_EXPT_DIR`` to a different SRW experiment root directory will enable scorecard plotting.
-
-Transfer MM Output (Optional)
-+++++++++++++++++++++++++++++++
-
-Unless overridden by ``task_mm_prep.MM_OUTPUT_DIR``, the MM evaluation output is written to ``${EXPT_DIR}/mm_output``. Users may wish to transfer the PNG and CSV output to their local machine for simplified review.
-
-Select MM Evaluation Packages (Optional)
-+++++++++++++++++++++++++++++++++++++++++
-
-*COMING SOON! Currently, only the "chemistry" evaluation package is supported.*
-
-Select MM Evaluation Tasks (Optional)
-+++++++++++++++++++++++++++++++++++++++
-
-Default MM evaluation tasks may be overridden using ``task_mm_run.MM_EVAL_TASKS_SINGLE_MODEL`` or ``task_mm_run.MM_EVAL_TASKS_MULTI_MODEL``. The latter is used in the case a base model is provided for intercomparison.
-
-Users are encouraged to consult MM `Supported Plots <https://melodies-monet.readthedocs.io/en/stable/users_guide/supported_plots.html>`__ and `Supported Statistsics <https://melodies-monet.readthedocs.io/en/stable/users_guide/supported_stats.html>`__ documentation for information on the plots and statistics genenerated by the MM tasks.
-
-Overview of the MM Evaluation Task Group
-------------------------------------------
-
-The MM evaluation task group consists of an ``mm_prep`` task followed by a number of MM task-specific jobs (metatasks) ``mm_run_<MM package>_<MM task>``. The ``mm_run_<MM package>_save_paired`` task must run before other metatasks.
+.. note::
 
-As the forecast windows increases in time duration, users are encouraged to tune the ``task_mm_run.execution.walltime`` configuration parameter. Please reach out to SRW support with questions on walltime and node tasking. The default configurations are not expected to handle all use cases and customization may be required.
+    The AQS PM and AQS VOCs evaluations do not run on Derecho. Set the packages' ``active`` flag to ``false`` (i.e., ``melodies_monet_parm.aqm.packages.aqs_pm.active: false``) when using Derecho.
 
-MM Configuration Variables
-++++++++++++++++++++++++++++
+To run the evaluation suite, a user will need to follow the steps described in the [aqm-mm-eval](https://github.com/NOAA-EPIC/AQM-Eval/wiki/aqm%E2%80%90mm%E2%80%90eval) wiki. An example MM evaluation configuration tuned to the AEROMMA use case is provided at ``ush/aqm-use-cases/config.aqm.AEROMMA.yaml``. High-level steps required to run an evaluation using the AEROMMA use case data:
 
-.. list-table:: MM Configuration Variables
-   :widths: 20 20 50
-   :header-rows: 1
-
-   * - Configuration Variable
-     - Default
-     - Description
-   * - ``task_mm_prep.MM_OUTPUT_DIR``
-     - ``null``
-     - Output directory for MM-generated model evaluation plots and statistics. If ``null``, defaults to ``${EXPTDIR}/mm_output``.
-   * - ``task_mm_prep.MM_EVAL_PACKAGES``
-     - ``[chem]``
-     - Evaluation packages to initialize and run.
-   * - ``task_mm_prep.MM_BASE_MODEL_EXPT_DIR``
-     - ``null``
-     - If set to another SRW experiment path, MM will generate "scorecards" for model inter-comparison.
-   * - ``task_mm_prep.MM_OBS_AIRNOW_FN_TEMPLATE``
-     - ``null``
-     - Path, optionally with wildcards, selecting the AirNow observation files used by MM. For example: ``/staged/obs/Observations/AirNow/AirNow_2023*.nc``
-   * - ``task_mm_prep.MM_EVAL_TASKS_SINGLE_MODEL``
-     - ``["timeseries", "taylor", "spatial_bias", "spatial_overlay", "boxplot", "multi_boxplot", "csi", "stats"]``
-     - MM evaluation tasks to run for a single model.
-   * - ``task_mm_prep.MM_EVAL_TASKS_MULTI_MODEL``
-     - ``["timeseries", "taylor", "spatial_bias", "spatial_overlay", "boxplot", "multi_boxplot", "csi", "stats", "scorecard_rmse", "scorecard_ioa", "scorecard_nmb", "scorecard_nme"]``
-     - MM evaluation tasks to run for two models (i.e., model intercomparison).
+1. Download observational data using ``aqm-data-sync``: https://github.com/NOAA-EPIC/AQM-Eval/wiki/aqm%E2%80%90mm%E2%80%90eval#stage-observational-data
+2. Configure observational datasets: https://github.com/NOAA-EPIC/AQM-Eval/wiki/aqm%E2%80%90mm%E2%80%90eval#observational-datasets
+3. Activate the evaluation task group by setting ``melodies_monet_parm.aqm.active: true`` and uncommenting ``workflow.taskgroups[parm/wflow/aqm_post_melodies_monet.yaml]``.

doc/conf.py

@@ -132,6 +132,8 @@
                             r"develop-20250321/.*",
                             r"installation",
                             r"grids",
+                            "stage-observational-data",
+                            "observational-datasets"
                             ]
 
 linkcheck_allowed_redirects = {r"https://github\.com/ufs-community/ufs-srweather-app/wiki/.*": 

jobs/JSRW_AQM_MELODIES_MONET_CONCAT_STATS

@@ -0,0 +1,80 @@
+#!/usr/bin/env bash
+
+
+#
+#-----------------------------------------------------------------------
+#
+# Source the variable definitions file and the bash utility functions.
+#
+#-----------------------------------------------------------------------
+#
+. $USHdir/source_util_funcs.sh
+sections=(
+  user
+  nco
+  workflow
+)
+for sect in ${sections[*]} ; do
+  source_yaml ${GLOBAL_VAR_DEFNS_FP} ${sect}
+done
+. $USHdir/job_preamble.sh
+
+#
+#-----------------------------------------------------------------------
+#
+# Save current shell options (in a global array).  Then set new options
+# for this script/function.
+#
+#-----------------------------------------------------------------------
+#
+{ save_shell_opts; . $USHdir/preamble.sh; } > /dev/null 2>&1
+#
+#-----------------------------------------------------------------------
+#
+# Get the full path to the file in which this script/function is located 
+# (scrfunc_fp), the name of that file (scrfunc_fn), and the directory in
+# which the file is located (scrfunc_dir).
+#
+#-----------------------------------------------------------------------
+#
+scrfunc_fp=$( $READLINK -f "${BASH_SOURCE[0]}" )
+scrfunc_fn=$( basename "${scrfunc_fp}" )
+scrfunc_dir=$( dirname "${scrfunc_fp}" )
+#
+#-----------------------------------------------------------------------
+#
+# Print message indicating entry into script.
+#
+#-----------------------------------------------------------------------
+#
+print_info_msg "
+========================================================================
+Entering script:  \"${scrfunc_fn}\"
+In directory:     \"${scrfunc_dir}\"
+
+This is the J-job script for MELODIES MONET evaluation/verification
+========================================================================"
+#
+#-----------------------------------------------------------------------
+#
+# Call the ex-script for this J-job and pass to it the necessary variables.
+#
+#-----------------------------------------------------------------------
+#
+aqm-mm-eval concat-stats --root-dir "${MM_OUTPUT_DIR}" --out-path "${MM_OUTPUT_DIR}"/concat-stats.csv
+#
+#-----------------------------------------------------------------------
+#
+# Run job postamble.
+#
+#-----------------------------------------------------------------------
+#
+job_postamble
+#
+#-----------------------------------------------------------------------
+#
+# Restore the shell options saved at the beginning of this script/function.
+#
+#-----------------------------------------------------------------------
+#
+{ restore_shell_opts; } > /dev/null 2>&1

jobs/JSRW_AQM_MELODIES_MONET_PREP

@@ -61,7 +61,7 @@ This is the J-job script for AQM MELODIES MONET PREP
 #
 #-----------------------------------------------------------------------
 #
-aqm-mm-eval srw-init --expt-dir ${COMOUT}
+aqm-mm-eval srw-init --expt-dir ${COMOUT} --package ${MM_EVAL_PACKAGE}
 #
 #-----------------------------------------------------------------------
 #

jobs/JSRW_AQM_MELODIES_MONET_RUN

@@ -61,7 +61,7 @@ This is the J-job script for MELODIES MONET evaluation/verification
 #
 #-----------------------------------------------------------------------
 #
-aqm-mm-eval srw-run --expt-dir ${COMOUT} --package-selector ${MM_EVAL_PACKAGE} --task-selector ${MM_EVAL_TASK}
+aqm-mm-eval srw-run --expt-dir ${COMOUT} --package ${MM_EVAL_PACKAGE} --task ${MM_EVAL_TASK}
 #
 #-----------------------------------------------------------------------
 #

modulefiles/tasks/derecho/mm_concat_stats.local.lua

@@ -0,0 +1,2 @@
+unload("python_srw")
+load("python_srw_aqm_eval")

modulefiles/tasks/gaeac6/aqm_ics.local.lua

@@ -1,2 +1,11 @@
 unload("python_srw")
+
+unload("esmf/8.8.0")
+unload("python/3.11.7")
+unload("python-venv/1.0")
+unload("py-mpi4py/4.0.1")
+unload("py-numpy/1.26.4")
+unload("py-pip/23.1.2")
+unload("py-pyyaml/6.0.2")
+
 load("python_srw_aqm")

modulefiles/tasks/gaeac6/fire_emission.local.lua

@@ -1,2 +1,11 @@
 unload("python_srw")
+
+unload("esmf/8.8.0")
+unload("python/3.11.7")
+unload("python-venv/1.0")
+unload("py-mpi4py/4.0.1")
+unload("py-numpy/1.26.4")
+unload("py-pip/23.1.2")
+unload("py-pyyaml/6.0.2")
+
 load("python_srw_aqm")

modulefiles/tasks/gaeac6/mm_concat_stats.local.lua

@@ -0,0 +1,2 @@
+unload("python_srw")
+load("python_srw_aqm_eval")

modulefiles/tasks/gaeac6/nexus_emission.local.lua

@@ -1,4 +1,13 @@
 unload("python_srw")
+
+unload("esmf/8.8.0")
+unload("python/3.11.7")
+unload("python-venv/1.0")
+unload("py-mpi4py/4.0.1")
+unload("py-numpy/1.26.4")
+unload("py-pip/23.1.2")
+unload("py-pyyaml/6.0.2")
+
 load("python_srw_aqm")
 
 setenv("FI_CXI_RX_MATCH_MODE", "hybrid")