Skip to content

project-tips: expand the data-leakage line into a concrete checklist - #712

Open
ipezygj wants to merge 1 commit into
DataTalksClub:mainfrom
ipezygj:leakage-checklist-project-tips
Open

project-tips: expand the data-leakage line into a concrete checklist#712
ipezygj wants to merge 1 commit into
DataTalksClub:mainfrom
ipezygj:leakage-checklist-project-tips

Conversation

@ipezygj

@ipezygj ipezygj commented Aug 9, 2026

Copy link
Copy Markdown

What

projects/project-tips.md already tells students to "avoid data leakage — don't include target-derived features", but gives no way to actually check for it. This expands that one line into a short, checkable list:

  • the would-I-know-this-at-prediction-time question,
  • the single-feature probe (train on each feature alone; near-perfect from one feature = leak),
  • "a suspiciously high score is a symptom, not a success",
  • a worked example from a dataset capstone projects genuinely use: DataCo late-delivery prediction scores ~97% when the shipping-time columns are included — but the label is computed from those columns, so the model restates the definition. With only order-time features the same models get ~69%. Students hit this exact trap (e.g. several past cohort capstones use this dataset), and public works keep republishing the leaked figure — 28 of 65 in a census of papers and notebooks. The linked audit has reproduction scripts, so it doubles as an example of what a good project README can look like when a number needs defending.

Framing follows the course's tone: the lower, honest number is the one worth putting in the README — which is also what the rubric's "clear problem description" asks for.

(Also mentioned in a mail to Alexey on Aug 8 — this is the PR that mail promised.)

The tips already said 'avoid data leakage' but gave no way to check for
it. Adds the prediction-time question, the single-feature probe, and a
worked example from a dataset capstone projects actually use (DataCo
late delivery: ~97% with the shipping-time columns the label is computed
from, ~69% with only order-time features).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant