Add validation config, goldens, counters, and existing statvar mcf for statistics_poland

[abhishekjaisw]

Description

This PR configures validation, golden verification sets, and output counters for the statistics_poland statvar import.

Changes Included:

1. Manifest Configuration (statvar_imports/statistics_poland/manifest.json):

   • Added --output_counters=counters/StatisticsPoland_output_counters.csv to the processor script invocation.
   • Added --existing_statvar_mcf=gs://unresolved_mcf/scripts/statvar/stat_vars.mcf to reuse existing resolved StatVars.
   • Added "counters/*.csv" and "golden_data/*.csv" to source_files so outputs and goldens are preserved and uploaded to GCS.
   • Included "stat_var_mcf": "StatisticsPoland_output_stat_vars.mcf" to provide definitions for new StatVars and avoid missing references.
   • Linked "validation_config_file": "validation_config.json".
   • Enabled invoke_import_validation, invoke_import_tool, invoke_differ_tool, and set "use_autopush_dc_api": false in config_override.
   • Configured resource limits (8 CPU, 32 GB RAM, 100 GB Disk).

2. Golden Files & Validation Config:

   • Created statvar_imports/statistics_poland/validation_config.json with GOLDENS_CHECK and DELETED_RECORDS_PERCENT rules (threshold: 0.1).
   • Generated golden_data/golden_summary_report.csv and golden_data/golden_observations.csv from verified prod runs.

3. Metadata & Script Updates:

   • Added recon_property,"variableMeasured,measurementMethod" in StatisticsPoland_metadata.csv to filter reconciliation to relevant variables and avoid excessive DC API queries on numeric observations.
   • Updated download_input_data.py to address pandas groupby transpose compatibility and removed /tmp fallback on PermissionError.
   • Updated README.md documentation with new CLI flags.

Testing:

• Cloud Batch Dev Run: Successfully executed on Dev (datcom-infosys-dev).

• GCS Output Directory: GCS Run 2026_08_07T05_12_44_959410_07_00

• Validation Output (validation_output.csv):

ValidationName	Status	Details
check_deleted_records_percent	PASSED	{"deleted_records_count": 0, "previous_obs_count": 40392, "percent": 0.0, "threshold": 0.1}
check_empty_import	PASSED	{"num_nodes": 42317, "num_rows": 42228}
check_missing_refs_count	PASSED	{"missing_refs_count": 0}
check_lint_error_count	PASSED	{"lint_error_count": 0}
check_goldens_summary_report	PASSED	{"validate-goldens-sets": 1, "validate-goldens-inputs": 108, "validate-goldens-expected": 108, "validate-goldens-input-matched": 108, "validate-goldens-matched": 108}
check_goldens_observations	PASSED	{"validate-goldens-sets": 1, "validate-goldens-inputs": 42228, "validate-goldens-expected": 17, "validate-goldens-input-matched": 42228, "validate-goldens-matched": 17}

• Differ Summary (differ_summary.json):

  • Observations: added_obs_count: 1,836 (new data for 2025), deleted_obs_count: 0.
  • Schema: added_schema_count: 24 — all 24 StatVars (e.g. Count_Person_16To19Years, Count_Person_20To24Years_Female, etc.) already exist in the Data Commons Knowledge Graph. They are flagged as added only relative to the March baseline MCF files which previously lacked stat_var_mcf.

• Formatted and validated with ./run_tests.sh -l.

[gemini-code-assist]

Code Review

This pull request updates the Poland Census import configuration, adds validation configurations with golden data, and introduces a recon_lookup_api flag in the schema reconciler. The review feedback highlights that changing the default value of recon_lookup_api to False globally will break other imports, so it should remain True by default. Additionally, the reviewer advises against silently falling back to /tmp/source_files on permission errors in the download script, and notes that counters/.csv and golden_data/.csv should not be included in the source_files list in manifest.json.

[saanikaaa]

Ran full Cloud Batch execution successfully on Dev (datcom-infosys-dev). - pls attach validation_output in PR description

[abhishekjaisw]

@saanikaaa I have updated the PR description with the Cloud Batch GCS run link and the complete validation_output.csv table showing all checks passed.