Feature #2858 produtil - remove sqlite3 dependency

[georgemccabe]

Fixes failures that are preventing dtcenter/MET#3435 from being merged.

Note: This PR should NOT close issue #2858, as the code base still contains other components of produtil.

Change Summary

• Update METplusConfig class to include config handling logic found in produtil.config, such as locking (to be thread safe), error checking, etc.
• Remove produtil.config and other related files to remove sqlite3 dependency that was causing failures running in Docker containers after recent updates to the METbaseimage
• Update use case tests logic to use version of python built in METbaseimage for use cases that don't need a special python instance to call run_metplus.py to ensure that Docker containers we provide work as expected
• Add unit test to ensure nocheck config getter functions behave as expected -- they should not set a default value back to the config that would be available in the metplus_final conf file
• Update run_metplus exit codes so that running with help argument, e.g. --help does NOT return in a non-zero (failure) value. Also adjusted failure exit codes to be able to discern failure (no args vs. failure in run vs. exception thrown in python code)

Pull Request Testing

• Describe testing already performed for these changes:
  

Ran GHA testing workflow using MET image that was broken (with sqlite3 errors) to confirm that removal of produtil.config resolves failures

Ran GHA testing workflow on branch without sqlite3 fixes with changes to automated tests to use the Python instance built in METbaseimage to confirm that more tests will properly fail if there is something wrong with the base image Python. Previously, only the GFDL-Tracker and GempakToCF use cases used this Python instance. These tools are not being actively supported, so the removal of these use cases could potentially hide issues with the Docker image without this enhancement.

• Recommend testing for the reviewer(s) to perform, including the location of input datasets, and any additional instructions:
  

Review code changes and perform any additional testing you think is necessary

• Do these changes include sufficient documentation updates, ensuring that no errors or warnings exist in the build of the documentation? [Yes]

• Do these changes include sufficient testing updates? [Yes]

• Will this PR result in changes to the test suite? [No]
  
  If yes, describe the new output and/or changes to the existing output:
  

• Do these changes introduce new SonarQube findings? [No]
  
  If yes, please describe:

• Please complete this pull request review by 9/4/2026.
  

Pull Request Checklist

See the METplus Workflow for details.

• Add any new Python packages to the METplus Components Python Requirements table.
• For any new datasets, an entry to the METplus Verification Datasets Guide.
• Review the source issue metadata (required labels, projects, and milestone).
• Complete the PR definition above.
• Ensure the PR title matches the feature or bugfix branch name.
• Define the PR metadata, as permissions allow.
  Select: Reviewer(s) and Development issue
  Select: Milestone as the version that will include these changes
  Select: Coordinated METplus-X.Y Support project for bugfix releases or METplus-Wrappers-X.Y.Z Development project for official releases
• After submitting the PR, select the ⚙️ icon in the Development section of the right hand sidebar. Search for the issue that this PR will close and select it, if it is not already selected.
• After the PR is approved, merge your changes. If permissions do not allow this, request that the reviewer do the merge.
• Close the linked issue and delete your feature or bugfix branch from GitHub.

[coveralls]

Coverage Report for CI Build 33792534175

Coverage decreased (-0.08%) to 91.992%

Details

• Coverage decreased (-0.08%) from the base build.
• Patch coverage: 19 uncovered changes across 1 file (133 of 152 lines covered, 87.5%).
• No coverage regressions found.

Uncovered Changes

File	Changed	Covered	%
metplus/util/config_metplus.py	152	133	87.5%

Coverage Regressions

No coverage regressions found.

---

Coverage Stats

Relevant Lines:	10577
Covered Lines:	9730
Line Coverage:	91.99%
Coverage Strength:	1.84 hits per line

---

💛 - Coveralls

[Copilot]

🟡 Changes recommended

A Docker automation path can generate invalid commands when python3 is not discoverable via shutil.which, and there is also a small but user-visible log message typo to fix.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR removes produtil.config (and its sqlite3-backed datastore components) to eliminate sqlite3-related failures in Docker/METbaseimage environments, while preserving needed configuration behavior inside METplus via an updated native METplusConfig. It also updates CLI exit codes and test automation so CI/use-case execution better reflects real container behavior.

Changes:

• Replaced inheritance from produtil.config.ProdConfig with a METplus-native METplusConfig implementation (locking, substitution, default handling, nocheck getters).
• Removed produtil modules that introduced sqlite3 dependency (produtil.config, produtil.datastore, produtil.numerics).
• Updated run_metplus.py exit codes and adjusted unit/use-case tests and GHA helper logic to align with the container’s Python behavior.

File summaries

File	Description
ush/run_metplus.py	Adjusts CLI exit codes for no-args/help/error/exception cases.
metplus/util/config_metplus.py	Reworks METplusConfig to be METplus-native with locking and updated getter/substitution behavior.
metplus/produtil/config.py	Removed to drop sqlite3-linked produtil configuration implementation.
metplus/produtil/datastore.py	Removed sqlite3 datastore implementation.
metplus/produtil/numerics.py	Removed unused produtil numerics utilities associated with removed config/datastore stack.
internal/tests/pytests/util/config_metplus/test_config_metplus.py	Adds coverage for nocheck getter behavior.
internal/tests/pytests/run_metplus/test_run_metplus.py	Updates expected exit codes for help/no-args/invalid args scenarios.
.github/jobs/get_use_case_commands.py	Updates Docker automation environment setup to prefer base-image Python where appropriate and refactors component-install setup.

Review details

• Files reviewed: 8/8 changed files
• Comments generated: 2
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[JohnHalleyGotway]

I approve of these change.

I note that...

• all GHA tests pass
• 3 unused files are removed from produtil
• exit codes in run_metplus.py are updated as described. I tested the branch locally to confirm the behavior.
• the user's guide makes no mention of codes returned by the run_metplus.py script

That information could be added (perhaps near this Configuration chapter), but isn't strictly required for this PR.

• Lastly as I sanity check, I ran run_metplus.py in Docker with develop and feature_2558_produtil_sqlite3 to confirm...

The problem:

> docker run -it --rm dtcenter/metplus-dev:develop /metplus/METplus/ush/run_metplus.py
...
ModuleNotFoundError: No module named '_sqlite3'

And the fix:

> docker run -it --rm dtcenter/metplus-dev:feature_2858_produtil_sqlite3-pull_request /metplus/METplus/ush/run_metplus.py
Running METplus v13.0.0-rc1-dev
Usage: run_metplus.py arg1 arg2 arg3
    -h|--help               Display this usage statement

Arguments:
/path/to/parmfile.conf -- Specify custom configuration file to use
section.option=value -- override conf options on the command line

[JohnHalleyGotway]

As discussed, I'm proceeding with merging this PR.