StyleShift-VDR

Code for studying presentation sensitivity and style bias in visual document retrievers.

Goal

Visual document retrievers (ColPali, ColQwen and similar models) search over page images instead of extracted text. This project asks whether these models care about how a page looks, not only what it says:

1. Invariance: if only the visual style of a page changes, do retrieval scores and rankings change?
2. Style bias: in a corpus that mixes styles, are some styles retrieved more often than their share of the corpus, especially among non-relevant results?
3. Factors: which visual factor (font, size and density, polarity, contrast, colour, layout, decoration) drives the effect, controlling for OCR readability?
4. Training-free fixes: do per-style score normalisation or fusion with OCR + BM25 reduce the bias?

Method overview

1. Load text retrieval datasets (BEIR SciFact and NFCorpus, test splits).
2. Render every document as an HTML page in 10 visual styles at a fixed page size (1024 × 1366 px) and take a screenshot. The words are byte-identical in every style; a test checks this.
3. Check readability by running OCR on each page and computing the character error rate against the source text.
4. Score every query against every (document, style) page with each retriever. Each page is embedded once per model, and only the score matrices are saved.
5. Run experiments from the saved scores: single-style and mixed-style retrieval, a swap test, factor analysis and training-free mitigation.
6. Report nDCG@10, Recall@10, Kendall tau, exposure ratio per style, and paired bootstrap confidence intervals.

Styles are defined in configs/styles.yaml, models (with pinned revisions) in configs/models.yaml, and datasets in configs/datasets.yaml.

Repository layout

configs/     styles, models and datasets (YAML)
src/         pipeline code: data, rendering, OCR check, encoding, scoring, experiments, metrics
tests/       unit and smoke tests
notebooks/   thin Kaggle/Colab notebooks that call src/
data/        generated data (not in git; regenerated by scripts)
results/     generated results (not in git)

Install

git clone <repo-url>
cd StyleShift-VDR
python -m venv .venv
source .venv/bin/activate        # Windows: .venv\Scripts\activate
pip install -r requirements.txt
playwright install chromium      # browser used for rendering
pre-commit install               # optional, for contributors

Model encoding needs a GPU. The GPU steps are designed to run on free Kaggle or Colab T4 sessions, and they are resumable: work is written in shards and finished shards are skipped.

Copy .env.example to .env and fill in your own tokens if needed. Never commit .env.

How to run

The pipeline is being built step by step. Today you can run the tests:

pytest

Commands for rendering, scoring and experiments will be documented here as each stage is added.

Results

Coming soon.

License

Code is released under the MIT License. Datasets and models keep their own licences; see the model cards linked from configs/models.yaml.