HistoParser is a reproducible, patient-aware pathomics workflow. This repository
(BMGLab/Pathomics) hosts the package and the full, leakage-resistant analysis behind:
G. Şimşek, Y. Kaymaz. Nuclear Morphology Tracks Molecular Class Unevenly Across Four Cancers. Submitted to Current Genomics, 2026.
- Paper & reproducibility — the manuscript, figures, tables and the scripts that
regenerate them from the analysis outputs:
docs/manuscript_pathomics_multicohort/ - Workflow package —
src/histoparser/
The workflow was first developed for glioblastoma (histopathology, genomics and MRI radiomics) and is extended here to a supervised, patient-aware test of whether nuclear morphometry from routine H&E predicts molecular class across four TCGA cohorts (renal, glioma, breast, lung).
Legacy note. An earlier version of this project lives at
BMGLab/HistoParser; this repository supersedes it for the pathomics work and the paper. The original upstream text is preserved under legacy_upstream/README_upstream.md.
Read this before citing any number from this repository: reports/audit/REPOSITORY_AUDIT.md. Several published results do not survive correction, and one input file is joined incorrectly.
| Path | Contents |
|---|---|
src/histoparser/ |
The hardened, patient-aware package |
legacy_upstream/ |
Verbatim copy of the published upstream code, unmodified |
Phase1…Phase4/, Additional_Experiments/ |
The original working analysis, preserved as-is |
configs/ |
Every scientific parameter; nothing that changes a result is hard-coded |
benchmarks/preregistration.yaml |
Primary endpoints, frozen before any comparative result |
reports/audit/ |
Repository, data, ID and claims audit |
reports/tables/, reports/metrics/ |
Machine-readable results — the source of truth |
reports/manuscript/ |
Methods/results/limitations generated from those files |
tests/ |
Regression tests for the specific defects that were found |
Two named modes run throughout:
legacy_reproduction— reproduces the original scripts' behaviour, limitations included. Results are exploratory and, where selection leakage occurs, optimistic.hardened_primary— leakage-resistant, deterministic, patient-aware. Used for primary reporting.
git clone <this repository>
cd HistoParser
python -m venv .venv && source .venv/bin/activate
pip install -e . # analysis stack only — no slide readingOptional extras:
pip install -e '.[viz]' # UMAP + seaborn for figures
pip install -e '.[dev]' # pytest, ruff, mypy
pip install -e '.[wsi]' # whole-slide reading; needs the OpenSlide system libraryOpenSlide is a system library and must be installed first:
brew install openslide # macOS
sudo apt-get install openslide-tools # Ubuntu/DebianIf import openslide fails with OSError: cannot load library 'libopenslide', the Python
package is installed but the system library is not.
No slide images, MRI volumes or patient-level exports are committed here. Stage them
yourself and point configs/*.yaml at them.
data/
upstream_pathology_features/output_density/ # per-patch nuclear feature CSVs (gitignored)
7316UP-109/
patch_24576_55296.png_features.csv
7316UP-109_mean_features.csv
...
wsi/ # optional: .ndpi/.svs slides (gitignored)
manifests/ # small, committed: checksums and ID maps
annotations/ # small, committed: expert annotations
The per-patch feature tables come from the upstream repository. Because that repository is 1.3 GB, clone it without blobs:
git clone --filter=blob:none --no-checkout https://github.com/BMGLab/HistoParser.git upstream
cd upstream
git sparse-checkout init --no-cone
git sparse-checkout set '/*' '!/slides/' '!/output/'
git checkout main
cp -R output_density ../data/upstream_pathology_features/A plain git clone of that repository does not complete in reasonable time.
See data/README.md for what may and may not be committed.
Runs every stage end to end with small resampling counts. Results are not publishable — it verifies that the pipeline executes, not that the numbers are final.
make smokeEquivalent to:
histoparser audit --config configs/smoke.yaml
histoparser cluster --config configs/smoke.yaml
histoparser associate --config configs/smoke.yaml
histoparser model-radiomics --config configs/smoke.yamlOutputs land in reports/smoke/.
make legacy # reproduce the published pipeline and compare against the manuscript
make hardened # the primary, leakage-resistant analysis
make report # regenerate the manuscript-facing documents from the metrics filesOr one stage at a time:
histoparser audit --config configs/hardened_primary.yaml
histoparser cluster --config configs/hardened_primary.yaml
histoparser associate --config configs/hardened_primary.yaml
histoparser model-radiomics --config configs/hardened_primary.yaml
histoparser report --config configs/hardened_primary.yamlEvery run writes a provenance record to reports/runs/<run_id>.json containing the git
SHA, config hash, seeds, input checksums, package versions and hardware.
| Config | Purpose |
|---|---|
configs/hardened_primary.yaml |
Primary analysis; robust aggregation, in-fold selection |
configs/hardened_legacyagg.yaml |
Hardened analysis on the legacy 72-feature matrix, so the effect of the aggregation change can be isolated |
configs/legacy_reproduction.yaml |
The upstream recipe, limitations included |
configs/smoke.yaml |
Fast end-to-end check |
A cohort sourced end-to-end from the NCI Genomic Data Commons, open-access tier, run on an HPC host (BioLab) rather than here. See reports/manuscript/LUNG_COHORT_ANALYSIS_PLAN.md for the biological questions and reports/audit/MISSING_INPUTS_LUNG.md for what is not staged yet.
make setup-hpc # system OpenSlide + the wsi/acquisition extras
make lung-manifest # GDC metadata query only; no large download
make lung-download # opt-in; --allow-large-downloads under the hood
make lung-sample lung-extract # WORKERS defaults to every detected core (e.g. 112)
make lung-hardened # audit, cluster, model-genomicsconfigs/cohorts/lung_luad_lusc/ holds this cohort's acquisition.yaml, smoke.yaml,
hardened_primary.yaml and preregistration.yaml, mirroring the GBM configs' shapes. No
existing GBM config or result is touched by any of this.
| File | Contents |
|---|---|
reports/tables/sample_flow.tsv |
Counts at every stage with exclusion reasons |
reports/tables/slide_inventory.tsv |
Per-slide patches, nuclei, protocol deviations |
reports/tables/radiomics_duplication_audit.tsv |
How much of the radiomics table is replicated feature vectors |
reports/tables/cluster_k_scan.tsv |
Internal indices for every candidate k |
reports/tables/cluster_stability_by_k.tsv |
Patient-level bootstrap Jaccard per cluster |
reports/tables/association_results.tsv |
Every tested variable with effect size, CI, raw p and BH q |
reports/tables/radiomics_lopo_results.tsv |
LOPO metrics, both selection scopes, both targets |
reports/tables/radiomics_feature_selection_frequency.tsv |
How often each feature was selected across folds |
reports/metrics/*.json |
Structured summaries — what the manuscript documents are generated from |
make testThe suite targets the defects that were actually found: patient leakage, global vs in-fold
selection, patient-level bootstrap width, undefined metrics returning NaN rather than 0,
detection of duplicated feature vectors and conflicting labels, and the misaligned genomic
join. It includes the canonical regression test that shuffling patient labels collapses
out-of-sample performance toward chance.
- Microns-per-pixel is unknown for every slide. No whole-slide image survives; nuclear size and shape features are in level-0 pixels and are not demonstrably comparable across slides.
- The published patch sample is unseeded and cannot be regenerated.
- The genomic annotations in
cluster_genetic.csvare joined to the wrong slides for 42 of 52 annotated rows. This repository repairs the join; the published associations were computed before the repair. - The radiomics table has 18 independent observations, not 38. 20 of 38 rows replicate another row's MRI feature vector under a different label.
- Tissue-mask and nuclear-segmentation accuracy cannot be assessed without expert annotations, which do not exist. Comparisons against other automated tools measure agreement, not accuracy.
- The cohort is 72.5% glioblastoma; the remainder is metastasis, astrocytoma and meningioma.
Full detail: reports/manuscript/LIMITATIONS.md and reports/audit/MISSING_INPUTS.md.
None yet. The upstream repository has no licence, which means all rights are reserved and nobody may legally reuse the code. This is a decision for the authors; see reports/audit/LICENSE_DECISION.md for the options and the dependency constraints.
HistomicsTK, OpenSlide, scikit-learn, SciPy, NumPy, pandas, matplotlib, UMAP, and CaPTk (used under its own academic terms) for the MRI radiomic features.