Incorrect/insufficient MRMS observation file dependencies for GridStat

[mkavulich]

Expected behavior

All WE2E tests

Current behavior

Depending on the timing of when observation retrieval tasks complete, GridStat tasks can fail because not all observations have been retrieved. Here is an example from the vx-det_multicyc_last-obs-00z_ncep-hrrr WE2E test, which is the only one I've seen occur consistently: the run_MET_GridStat_vx_REFC_mem000 and run_MET_GridStat_vx_RETOP_mem000 tasks for the final cycle only fail due to missing observations.

================================================================================================================================
202404302100    run_MET_PcpCombine_APCP01h_obs_CCPA                    16749315           SUCCEEDED                   0         1          13.0
202404302100    run_MET_PcpCombine_APCP03h_obs_CCPA                    16749316           SUCCEEDED                   0         1          10.0
202404302100    check_post_output_mem000                    16747836           SUCCEEDED                   0         1          10.0
202404302100    run_MET_PcpCombine_APCP01h_fcst_mem000                    16748244           SUCCEEDED                   0         1          45.0
202404302100    run_MET_PcpCombine_APCP03h_fcst_mem000                    16748245           SUCCEEDED                   0         1          42.0
202404302100    run_MET_GridStat_vx_APCP01h_mem000                    16749490           SUCCEEDED                   0         1         186.0
202404302100    run_MET_GridStat_vx_APCP03h_mem000                    16749491           SUCCEEDED                   0         1          85.0
202404302100    run_MET_GridStat_vx_REFC_mem000                    16749220                DEAD                   1         1           5.0
202404302100    run_MET_GridStat_vx_RETOP_mem000                    16749221                DEAD                   1         1           5.0
202404302100    run_MET_PointStat_vx_SFC_mem000                    16749992           SUCCEEDED                   0         1          45.0
202404302100    run_MET_PointStat_vx_UPA_mem000                    16749993           SUCCEEDED                   0         1         106.0
================================================================================================================================
202405010000            get_obs_mrms                    16747837           SUCCEEDED                   0         1        1414.0
202405010000            get_obs_ndas                    16747838           SUCCEEDED                   0         1        1060.0
202405010000    run_MET_Pb2nc_obs_NDAS                    16749944           SUCCEEDED                   0         1          17.0

Running rocotocheck to observe the dependencies shows that the mrms_available dependency returned true, but this dependency (find /scratch3/BMC/dtc/dtc-vx-workflow_testing/feature/regression_tests/output.827a3e8/vx-det_multicyc_last-obs-00z_ncep-hrrr/wflow_flag_files -name "get_obs_mrms_*_complete.txt" -exec basename {} \; 2>/dev/null | grep -oE '[0-9]{8}' | awk -F/ -v s=20240429 -v e=20240430 '($NF >= s && $NF <= e)' | test $(wc -l) -ge 2) is only checking for the files from 20240429 and 20240430, even though 20240501 is also needed.

Re-running these tasks manually after the observation retrieval tasks all complete results in success, so this is simply a dependency problem, not a functional problem.

Machines affected

Likely all

Steps To Reproduce

Run WE2E test vx-det_multicyc_last-obs-00z_ncep-hrrr; this is easiest to replicate if run with a number of other tasks retrieving observations at the same time. You may observe a failure like the one described above depending on which observation retrieval tasks finish first.

Detailed Description of Fix

Unclear, I need to dig into how these dependencies are defined to figure out the changes needed.

Additional Information

The issue may have something to do with the month rollover, since the only test I've observed this problem on is vx-det_multicyc_last-obs-00z_ncep-hrrr, which is the only one that has a forecast running from one month to the next and depends on MRMS obs.