Issue #10 - Anchor paper analysis & domain shift framing (Shift-Lab)

[pegahani]

Owner: @amir-fakhimi, @amir-malekhosseini, @helena-

Goal
Establish a shared and precise understanding of domain, shift, and generalization assumptions, and analyze how these assumptions translate (or fail to translate) to LLM-based decision pipelines.

This issue is about conceptual alignment, not summarization.

In classical domain adaptation literature, a shift typically refers to a change in the data distribution between training and deployment domains.

However, in the SHIFT-Lab project we are interested in a broader class of perturbations affecting decision systems. In particular, shifts may occur at different levels:

**Data shift **: changes in the input distribution

Constraint shift: changes in system constraints (budgets, thresholds, rules)

Objective / preference shift changes in the optimization objective or reward

The goal of this issue is to understand:

1. What formal assumptions classical domain adaptation theory makes.
2. Which of these assumptions break in LLM settings.
3. Whether modern engineering responses (DAPT, LoRA, RAG, ICL) should be interpreted as adaptation mechanisms or selection mechanisms.
4. Whether some phenomena described as "domain shift" may actually correspond to constraint or objective shifts.

Anchor paper (mandatory for everyone)

Ben-David et al. (2010) — A Theory of Learning from Different Domains

This paper defines the formal notion of domain, shift, and generalization bounds.
All other papers should be read through this lens.

Calderon et al. (2024) -- Measuring the Robustness of NLP Models to Domain Shifts

Additional paper assignments (role-based)

Helena - Theory & assumptions

• Johansson et al. (2019) — Support and Invertibility in Domain-Invariant Representations

• Sugiyama & Kawanabe (2012) — Machine Learning in Non-Stationary Environments (selected chapters)

Focus:

• formal assumptions required for domain generalization
• identifiability and support overlap
• impossibility results
• which assumptions break for large language models.

Amir Mohammad Fakhimi - Empirical framing & evaluation

• Gururangan et al. (2020) — Don’t Stop Pretraining (DAPT)

• Min et al. (2022) — Rethinking In-Context Learning

Focus:

• how "domain shift" is operationalized in practice
• what current benchmarks actually measure
• what kinds of shifts remain invisible in typical evaluations.

Amir Malekhosseini - Adaptation mechanisms

Finn et al. (2017) — Model-Agnostic Meta-Learning (MAML)

Dou et al. (2019) — Domain Generalization via Model-Agnostic Learning

(skim / positioning) Hu et al. (2022) LoRA or Lewis et al. (2020) RAG

Focus:

• what kind of shift each method implicitly assumes
• when adaptation corresponds to learning new parameters
• when it corresponds to selection or retrieval mechanisms.

Expected output
Each student writes one short note (½–1 page) and posts it in the issue comments.

The note must answer:

1. What is the formal problem being solved?
2. What is assumed known vs unknown?
3. One assumption from Ben-David (2010) that is violated in LLMs
4. One engineering response (DAPT / LoRA / RAG / ICL) to this violation
5. Could this violation be interpreted as:
   • a data shift, or
   • a constraint / objective shift?
6. One open question this raises for LLM adaptation

No slides. No code.

Definition of done

• Anchor paper read by all
• One note per student posted
• At least one explicit link between theory assumption and modern adaptation method
• At least one limitation or impossibility clearly articulated
• At least one discussion point related to structural shifts beyond data distribution

Estimated effort
~1 week (4–6 hours)