Ground ML platform and career pages

Author: alexeygrigorev

_articles/freelance-data-engineer.md

@@ -9,61 +9,162 @@ related:
   - Power Analysis
 ---
 
-## Definition and Scope
+A/A testing is an experiment-platform validation technique. A team splits
+traffic into two or more groups and shows every group the same product
+experience. Then it checks whether assignment, exposure logging, metrics, and
+analysis behave as if nothing changed.
 
-A/A testing splits traffic into groups that receive the same experience. The
-purpose isn't to test a product change. It's to check whether randomization,
-assignment logging, tracking, and analysis behave as expected before a team
-trusts A/B results.
+In the DataTalks.Club archive, A/A testing sits between
+[event tracking]({{ '/wiki/event-tracking/' | relative_url }}),
+[product analytics]({{ '/wiki/product-analytics/' | relative_url }}), and
+[A/B testing]({{ '/wiki/a-b-testing/' | relative_url }}). It doesn't answer
+whether a feature works. It answers whether the experiment system is trustworthy
+enough to test a feature.
 
-In the archive, A/A testing is a platform trust practice between
-instrumentation and experimentation. If the system can't produce balanced groups
-and comparable metrics when nothing changed, teams shouldn't use it to judge
-real product changes.
+[Jakob Graff]({{ '/people/jakobgraff/' | relative_url }}) gives the clearest
+definition in
+[Product Analytics and A/B Testing at 27:52]({{ '/podcasts/ab-testing-and-product-experimentation/' | relative_url }}).
+He describes an A/A test as a traffic split where both groups see the exact same
+thing. A planned 50/50 split might become 60/40. One identical group might also
+appear to convert far better than the other. In either case, the team should
+look at randomization, tracking, and analysis before trusting later experiments.
 
+## Common Definition
 
-## Recurring Archive Themes
+A common archive definition is: A/A testing validates the experiment system
+under a no-treatment condition. The expected outcome isn't perfect equality.
+Users still differ, and random noise remains. The system should produce balanced
+assignment, comparable metrics, and explainable variation.
 
-A/A tests reveal assignment problems. Jakob Graff warns that a planned 50/50
-split becoming 60/40 means the traffic splitter or triggering logic needs
-investigation before any A/B conclusion is credible.
+Jakob's discussion of experimentation infrastructure gives the practical version
+of that definition.
 
-A/A tests also reveal metric and tracking problems. If both groups see the same
-experience but conversion rates diverge beyond plausible noise, the team should
-look for instrumentation bugs and biased defaults. Missing assignment events or
-analysis mistakes may also explain the gap.
+At 24:44 in
+[Product Analytics and A/B Testing]({{ '/podcasts/ab-testing-and-product-experimentation/' | relative_url }}),
+Jakob says the traffic splitter must randomize on the right unit. Teams may use
+a user ID, session ID, or cookie. At 27:52, he says teams should track the app's
+call to the splitter. They should also track whether the app receives a sensible
+assignment. Bad connection handling can bias the test if offline users all fall
+into the same default group.
 
-The practice applies to third-party tools too. The archive notes that external
-experimentation platforms can still produce untrusted splits, so teams should
-validate them instead of assuming the dashboard is correct.
+## Guest Emphasis
 
-A/A intuition helps explain p-values. Later in the same episode, Jakob asks how
-likely it would be to see the observed uplift in an A/A test, where no
-treatment exists.
+The archive has one direct A/A-testing discussion, so the disagreement is mostly
+about emphasis across adjacent experiment topics.
 
-## Episode Evidence
+Jakob treats A/A testing as a trust-building step for
+[experimentation]({{ '/wiki/experimentation/' | relative_url }}) platforms. In
+[Product Analytics and A/B Testing at 30:05]({{ '/podcasts/ab-testing-and-product-experimentation/' | relative_url }}),
+he says teams should run simple tests first. Teams should use a single decision
+metric and avoid strange product logic that makes assignment hard to track.
 
-These episodes give the strongest evidence:
+Other podcast discussions focus more on what happens after the platform is
+trusted. [Rishabh Bhargava]({{ '/people/rishabhbhargava/' | relative_url }})
+connects A/B tests with production ML rollout and shadow mode in
+[From Analytics to Production ML at 28:42]({{ '/podcasts/production-ml-mlops-and-data-team-building/' | relative_url }}).
 
-- [Product Analytics and A/B Testing](https://datatalks.club/podcast.html),
-  24:44-27:52, explains traffic splitters and assignment tracking. It also
-  covers offline users defaulting into one group and monitoring around
-  experimentation infrastructure. Source:
-  `../datatalksclub.github.io/_podcast/ab-testing-and-product-experimentation.md`.
-- [Product Analytics and A/B Testing](https://datatalks.club/podcast.html),
-  27:52-32:39, defines A/A testing as showing the same experience to both
-  groups, then checking split balance and metric comparability.
-- [Product Analytics and A/B Testing](https://datatalks.club/podcast.html),
-  47:44-50:12, uses A/A test logic to explain p-value intuition to product
-  stakeholders without leading with formal hypothesis-test language.
+[Aleksander Molak]({{ '/people/aleksandermolak/' | relative_url }}) places
+randomized experiments inside a broader
+[causal inference]({{ '/wiki/causal-inference/' | relative_url }}) toolkit in
+[Causal Inference for Machine Learning at 26:16]({{ '/podcasts/causal-inference-for-machine-learning/' | relative_url }}).
+Those episodes assume the experiment result can be interpreted. Jakob's A/A
+point comes earlier in the chain. First prove that the measurement and
+assignment system can produce a sane null result.
+
+## Experiment Validation
+
+A/A testing validates the mechanics inside an A/B test:
+
+- the traffic splitter returns stable assignments
+- the product applies the assignment consistently
+- the exposure event is logged at the right moment
+- the analysis reads the same assignment that the product used
+- the primary metric behaves similarly across identical groups
+
+Jakob's
+[27:52 discussion]({{ '/podcasts/ab-testing-and-product-experimentation/' | relative_url }})
+is especially practical because it includes failure modes. If an app defaults
+offline users into Group A, the control group is no longer comparable to the
+treatment group. If an external platform promises a 50/50 split but returns
+55/45, the dashboard result needs investigation before the product team acts on
+it.
+
+The same idea applies to third-party and in-house systems. At
+[23:54]({{ '/podcasts/ab-testing-and-product-experimentation/' | relative_url }}),
+Jakob compares external tools with building a traffic splitter with engineers.
+At 30:05, he says the first real experiment should stay simple. A/A testing is
+part of that same sequence: remove platform uncertainty before adding product
+uncertainty.
+
+## Metrics
+
+A/A tests are useful only when the team understands the metric it's checking.
+
+Jakob distinguishes noisy metrics from stable ones in
+[Product Analytics and A/B Testing at 33:23]({{ '/podcasts/ab-testing-and-product-experimentation/' | relative_url }}).
+Revenue per install can jump around. Click-through rate may be easier to
+interpret. An A/A test that looks different across groups may reveal a bug. It
+may also reveal that the metric is too noisy for a short experiment.
+
+This is why A/A testing connects to [Metrics]({{ '/wiki/metrics/' | relative_url }}).
+The primary metric should match the rollout decision, and supporting metrics
+should help diagnose what moved. In Jakob's framing, teams should choose one
+decision metric before the test. If they look at too many metrics after the
+fact, they make it easier to mistake noise for a finding.
+
+## Instrumentation
+
+A/A testing is also an instrumentation check. The product needs to log who was
+assigned, when the assignment happened, and whether the user actually saw the
+experience being tested. Those events should follow the same rules the team uses
+in its [tracking plan]({{ '/wiki/tracking-plans/' | relative_url }}).
+
+Jakob explains the instrumentation risk in
+[Product Analytics and A/B Testing at 24:44]({{ '/podcasts/ab-testing-and-product-experimentation/' | relative_url }}).
+Teams may randomize by user, session, or cookie. The analysis must use the same
+unit. If product code assigns by session but analytics aggregates by user, an
+apparently balanced A/A test can still hide ambiguous exposure logic.
+
+## Power Analysis
+
+A/A testing doesn't replace [power analysis]({{ '/wiki/power-analysis/' | relative_url }})
+because the two checks answer different questions. A/A testing checks whether
+the platform behaves under no treatment. Power analysis checks whether a planned
+experiment has enough observations to detect the effect size the team cares
+about.
+
+In
+[Product Analytics and A/B Testing at 37:44]({{ '/podcasts/ab-testing-and-product-experimentation/' | relative_url }}),
+Jakob explains how teams can estimate test duration. They use metric
+distribution, expected impact, and daily traffic. A short A/A test can catch
+obvious assignment failures. It can't prove that every future A/B test has
+enough sample size for a small product effect.
+
+## Product Analytics
+
+For [product analytics]({{ '/wiki/product-analytics/' | relative_url }}), A/A
+testing protects decision quality. Product managers and analysts will eventually
+ask whether a treatment improved conversion, retention, or revenue. They may
+also ask whether it improved engagement.
+If the no-treatment system already creates unexplained differences, those later
+answers are weak.
+
+Jakob also uses A/A testing as an explanation tool. At
+[47:44 in the same episode]({{ '/podcasts/ab-testing-and-product-experimentation/' | relative_url }}),
+he explains p-values through a no-treatment comparison. The practical question
+is how likely the observed uplift would be in an A/A test where both groups saw
+the same thing. That framing helps stakeholders understand significance without
+starting from formal hypothesis-test language.
 
 ## Related Pages
 
-Useful adjacent pages:
+Use these adjacent pages to place A/A testing in the broader experiment stack:
 
 - [A/B Testing]({{ '/wiki/a-b-testing/' | relative_url }})
 - [Experimentation]({{ '/wiki/experimentation/' | relative_url }})
 - [Metrics]({{ '/wiki/metrics/' | relative_url }})
 - [Power Analysis]({{ '/wiki/power-analysis/' | relative_url }})
 - [Evaluation]({{ '/wiki/evaluation/' | relative_url }})
 - [Tracking Plans]({{ '/wiki/tracking-plans/' | relative_url }})
+- [Event Tracking]({{ '/wiki/event-tracking/' | relative_url }})
+- [Product Analytics]({{ '/wiki/product-analytics/' | relative_url }})