Fix tests failing when installation process tests are run via cdds account

[Ed (mo-gill)]

Fixes #1038

[Ed (mo-gill)]

Commit f0aafc8 resolves:
tests/test_misc/test_system_information.py::test_system_information_valid - assert 1 == 0

It was caused by the discrepancy in CDDS_ENV_COMMAND in setup_env_for_devel:

# Pass a command which would set up a cdds conda environment on SPICE
export CDDS_ENV_COMMAND="source $CDDS_DIR/setup_env_for_devel"

vs the way its set in the installation process:
conda env config vars set CDDS_ENV_COMMAND="conda activate $HOME/conda_environments/cdds-X.Y.Z"

[Ed (mo-gill)]

Commit 235ba3d fixes all the errors involving configparser e.g.:
configparser.InterpolationSyntaxError: '$' must be followed by '$' or '{', found: '$CDDS_DIR/cdds/cdds/tests/test_validate/data/variables_file.txt'

Turned out it was because at the point of running the tests in the installation, $CDDS_DIR isn't set. I think usually it's set in setup_env_for_devel which isn't run as part of the installation process. The addition of the setting of that variable via a pytest fixture will hopefully avoid that in future and it does fix those errors.

[Ed (mo-gill)]

Commit b0ea0a0 fixed these errors caused by hardcoded paths.

FAILED checkout_for_fixing_tests/CDDS/cdds/cdds/tests/test_extract/test_validate.py::TestValidate::test_calculate_file_frequency - RuntimeError: Request config file "cdds/cdds/tests/test_extract/data/test_request_minimal.cfg" not found. Please check the command line arguments and try again.
FAILED checkout_for_fixing_tests/CDDS/cdds/cdds/tests/test_extract/test_validate.py::TestValidate::test_configure_mapping_for_each_variable - RuntimeError: Request config file "cdds/cdds/tests/test_extract/data/test_request_minimal.cfg" not found. Please check the command line arguments and try again.
FAILED checkout_for_fixing_tests/CDDS/cdds/cdds/tests/test_extract/test_validate.py::TestValidate::test_process_pp_streamtype - RuntimeError: Request config file "cdds/cdds/tests/test_extract/data/test_request_minimal.cfg" not found. Please check the command line arguments and try again.

and

FAILED checkout_for_fixing_tests/CDDS/cdds/cdds/tests/test_extract/test_validate.py::TestValidate::test_calculate_file_frequency - AttributeError: The "variable_list_file" does not exist.
FAILED checkout_for_fixing_tests/CDDS/cdds/cdds/tests/test_extract/test_validate.py::TestValidate::test_configure_mapping_for_each_variable - AttributeError: The "variable_list_file" does not exist.
FAILED checkout_for_fixing_tests/CDDS/cdds/cdds/tests/test_extract/test_validate.py::TestValidate::test_process_pp_streamtype - AttributeError: The "variable_list_file" does not exist.

[Ed (mo-gill)]

Commit 39d843a fixed the hardcoded paths so that were causing these failures:

FAILED checkout_for_fixing_tests/CDDS/cdds/cdds/tests/test_misc/test_stream_mappings.py::TestStreamMappings::test_read_variables_file - FileNotFoundError: [Errno 2] No such file or directory: 'cdds/cdds/tests/test_misc/data/inputs/minimal_variables.txt'
FAILED checkout_for_fixing_tests/CDDS/cdds/cdds/tests/test_misc/test_stream_mappings.py::TestStreamMappings::test_save_mappings - FileNotFoundError: [Errno 2] No such file or directory: 'cdds/cdds/tests/test_misc/data/outputs/minimal_variables_final_output.txt'

and

checkout_for_fixing_tests/CDDS/cdds/cdds/tests/test_sim_review/test_process_critical_errors.py::TestCheckIssuesInCmor::test_check_issues_for_variable - FileNotFoundError: [Errno 2] No such file or directory: 'cdds/cdds/tests/test_deprecated/test_transfer/data/test_cmor.log.gz'
FAILED checkout_for_fixing_tests/CDDS/cdds/cdds/tests/test_sim_review/test_process_critical_errors.py::TestCheckIssuesInCmor::test_check_issues_in_cmor_axis - FileNotFoundError: [Errno 2] No such file or directory: 'cdds/cdds/tests/test_deprecated/test_transfer/data/test_cmor.log.gz'
FAILED checkout_for_fixing_tests/CDDS/cdds/cdds/tests/test_sim_review/test_process_critical_errors.py::TestCheckIssuesInCmor::test_check_issues_in_cmor_variable - FileNotFoundError: [Errno 2] No such file or directory: 'cdds/cdds/tests/test_deprecated/test_transfer/data/test_cmor.log.gz'
FAILED checkout_for_fixing_tests/CDDS/cdds/cdds/tests/test_sim_review/test_process_critical_errors.py::TestCheckIssuesInCmor::test_check_issues_in_cmor_write - FileNotFoundError: [Errno 2] No such file or directory: 'cdds/cdds/tests/test_deprecated/test_transfer/data/test_cmor.log.gz'
FAILED checkout_for_fixing_tests/CDDS/cdds/cdds/tests/test_sim_review/test_process_critical_errors.py::TestCheckIssuesInCmor::test_check_issues_in_cmor_zfactor - FileNotFoundError: [Errno 2] No such file or directory: 'cdds/cdds/tests/test_deprecated/test_transfer/data/test_cmor.log.gz'

[Ed (mo-gill)]

More on autouse here, as i can't see it used elsewhere in CDDS. Seems like a handy way to ensure it's always used by tests rather than having to explicitly request it.

[Ed (mo-gill)]

These tests that are supposed to be run during installation now pass:

echo "# Executing tests for cdds:"
pytest -s $SRCDIR/cdds --doctest-modules -m 'not slow and not integration and not rabbitMQ and not data_request'
pytest -s $SRCDIR/cdds -m slow
pytest -s $SRCDIR/cdds -m integration
pytest -s $SRCDIR/cdds -m data_request
echo "# Executing tests for mip_convert:"
pytest -s $SRCDIR/mip_convert --doctest-modules -m 'not slow and not mappings and not superslow'
pytest -s $SRCDIR/mip_convert -m mappings
pytest -s $SRCDIR/mip_convert -m slow

If anyone wants to test it i've left two dirs in the cdds home:

checkout_for_fixing_tests has the same changes that i applied in this branch when i was iterating. The environment that uses it is called fix_the_tests_editable

checkout_for_comparison is just a checkout of current state of main in case i needed to do a meld.

I'll clean all that out once this has been merged.

I've also run the unit tests on my own space and everything is passing there as well.

Copilot was used during development.