feat(v4): HAD post-fit aggregate() + mode inference; per-level bootstrap-gate convergence (2(b) PR-4, M-027/M-139) (#751)

Author: igerber

CHANGELOG.md

@@ -8,6 +8,53 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ## [Unreleased]
 
 ### Added
+- **HeterogeneousAdoptionDiD post-fit `aggregate()` + panel-shape mode
+  inference, and the per-level bootstrap-gate convergence** (v4 program 2(b)
+  PR-4; ledger rows [M-027] + new row [M-139], amendments to [M-020]..[M-023],
+  [M-092], [M-093]). `HeterogeneousAdoptionDiD.fit(aggregate=)` and
+  `did_had_pretest_workflow(aggregate=)` are deprecated (`FutureWarning`;
+  removed in 4.0). Unlike every prior adopter the param was a MODE SELECTOR
+  over mutually exclusive panel shapes, so the successor is inference: both
+  surfaces now select the mode from the panel (two distinct periods -> the
+  overall single-period WAS estimator / pretest battery; more -> the
+  event-study ones), and a plain multi-period `fit()` - which previously
+  raised the two-period shape error - now just works (no numbers change on
+  any previously-working call; supplied legacy values warn and run the
+  legacy routing unchanged, invalid values still raise). Post-fit
+  `aggregate()` lands on both results classes as PURE VIEWS with per-class
+  supported subsets and NO retained kit (results unpickled from any release
+  aggregate identically): `aggregate('simple')` on overall fits is a one-row
+  bit-exact relay whose `target` carries the estimand label
+  (`WAS`/`WAS_d_lower`), with `n = n_obs` contributing units and
+  provenance-exact df; `aggregate('event_study')` on event-study fits is a
+  `build_event_study_surface` passthrough (cband fields included). The
+  `_from_had` adapter's `n_kind` is corrected `"obs"` -> `"units"`
+  (`n_obs_per_horizon` counts units); `EventStudyResults` gains an
+  `estimand` provenance field plus a per-row `estimand` column in the
+  pinned `to_dataframe()` schema (`"att"` for every ATT producer, the
+  WAS-family label for HAD, relayed from `target_parameter`) so the
+  container heading, `to_dict()` serialization, and detached frames never
+  mislabel the estimand; and `AggregationResult.summary()` now
+  renders the target column + neutral `estimate` heading whenever the single
+  distinct target is not `"att"` (sized to the longest label) - previously a
+  WAS or dCDH estimand row rendered under a hard-coded `ATT` heading;
+  uniform-`att` and ContinuousDiD att/acrt output are byte-stable. HAD
+  event-study containers stay outside `compute_honest_did` /
+  `compute_pretrends_power` - DEFERRED, not by-design: the coefficients ARE
+  reference-normalized against the F-1 anchor, but the anchor row is omitted
+  (identically zero and the WAS is not identified there) and no joint
+  cross-horizon covariance exists; both terminal TypeErrors say so.
+  **Per-level bootstrap-gate convergence:** CallawaySantAnna, EfficientDiD,
+  ImputationDiD and TwoStageDiD previously failed closed on EVERY
+  `aggregate()` level of a bootstrapped fit; their `aggregate('simple')` is
+  a bit-exact relay of the stored overall quintet (percentile se/p/CI beside
+  the finite `safe_inference` t), so it now dispatches before the bootstrap
+  gate and stays available - with a NaN df column, because no df governs
+  percentile inference - while the recompute levels keep failing closed
+  (this supersedes the uniform-conservatism decision recorded with
+  [M-021]..[M-023]; the normative per-level rule lives in v4-design
+  section 6). The `HADPretestReport.aggregate` FIELD survives as output
+  metadata recording which battery ran; only the params die.
 - **ContinuousDiD post-fit `aggregate()` - a MIXED view/recompute adopter**
   (v4 program 2(b) PR-3c; ledger row [M-025]). `ContinuousDiD.fit(aggregate=)`
   is deprecated (`FutureWarning`; removed in 4.0, and the no-underscore
@@ -85,9 +132,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
     copy of the working frame (O(n_obs); replicate designs additionally
     retain the O(n_obs x R) replicate matrix). A `store_kit` opt-out is
     tracked in DEFERRED.md.
-  - Bootstrapped fits fail closed for ALL aggregate() levels including
-    'simple' (uniform CS/EfficientDiD parity; replay wiring is tracked
-    in TODO.md); a fit whose bootstrap FAILED aggregates normally.
+  - Bootstrapped fits: `aggregate('simple')` relays the stored quintet
+    verbatim with a NaN df column, while the recompute levels fail
+    closed (replay wiring is tracked in TODO.md; the per-level policy
+    converged with [M-027] later in the same cycle); a fit whose
+    bootstrap FAILED aggregates normally.
     TwoStage post-fit event-study containers reproduce the M-092
     contract exactly (joint GMM vcov + index + df on analytical fits;
     vcov=None with the replayed df on replicate fits).
@@ -142,11 +191,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
     successor fails closed on unknown types (behavior improvement).
     `EfficientDiD`'s balance rule is the anchor-horizon rule - the
     same rule CallawaySantAnna uses.
-  - Bootstrapped fits (`n_bootstrap > 0`) fail closed on `aggregate()`
-    at every level (CS parity); the deprecated fit-time aggregation
-    remains the supported bootstrapped route, and its group rows now
-    clear the analytical `df_used` provenance under the bootstrap
-    override. Exact post-fit bootstrap replay is a tracked TODO row.
+  - Bootstrapped fits (`n_bootstrap > 0`): `aggregate('simple')` relays
+    the stored quintet verbatim with a NaN df column, while the
+    recompute levels fail closed (the per-level policy converged with
+    [M-027] later in the same cycle); the deprecated fit-time
+    aggregation remains the supported bootstrapped route for those
+    levels, and its group rows still clear the analytical `df_used`
+    provenance under the bootstrap override. Exact post-fit bootstrap
+    replay is a tracked TODO row.
   - The aggregation methods moved verbatim to the new
     `diff_diff/efficient_did_aggregation.py` (importable by both the
     estimator and the results module); `efficient_did.py` drops below
@@ -286,8 +338,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
     delta/no-scalar-count) and a `df` column resolved from the ACTUAL
     inference path (the `L_max>=2` delta keeps its analytical survey df
     even under bootstrap, per the REGISTRY delta-SE note). Because
-    nothing is recomputed, bootstrap fits are permitted - unlike
-    CallawaySantAnna's kit-based `aggregate()`, which stays fail-closed.
+    nothing is recomputed, bootstrap fits are permitted (since [M-027]
+    the kit-based adopters' `'simple'` relays are permitted on
+    bootstrapped fits too - only their recompute levels stay closed).
   - Consumer hardening that landed with the wave: `plot_event_study`
     renders MULTI-reference containers via row-aligned hollow anchors
     (never dropped, never filled estimates; explicit `reference_period=`
@@ -760,9 +813,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   and forms no cross-cohort mass — reporting one would be fabricated.
   Fail-closed elsewhere too: `aggregate("calendar")` raises (CS has no
   calendar aggregator; the DEFERRED row stands), a non-`None` `weights`
-  raises, and `aggregate()` on a **bootstrapped** fit raises rather than
-  substituting analytical inference for percentile-bootstrap statistics —
-  bootstrap replay is tracked in `TODO.md`.
+  raises, and the recompute levels of `aggregate()` on a **bootstrapped**
+  fit raise rather than substituting analytical inference for
+  percentile-bootstrap statistics (`aggregate('simple')` relays the stored
+  bootstrap inference with a NaN df column since [M-027]'s per-level
+  convergence) — bootstrap replay is tracked in `TODO.md`.
   `AggregationResult.df` is per-row provenance: the degrees of freedom that
   actually produced that row's stored p-value and interval, read from the
   carrier the fit used (`survey_metadata.df_survey` on explicit survey

METHODOLOGY_REVIEW.md

@@ -856,10 +856,10 @@ These three are feature deferrals (paper-supported extensions that the library h
 
 **Deviations from the paper / from R / library extensions:**
 1. **Equal-weighting on the continuous path** (paper does not prescribe a unit-weighting scheme; library uses per-unit `w_g = 1` matching `_nprobust_port.lprobust`'s default, NOT cell-size weights). Locked in `tests/test_methodology_had.py::TestHADDeviations::test_equal_weighting_is_per_row_not_per_dose_cell` (probes the deviation via selective low-dose-region replication on a nonlinear DGP: per-row equal weighting predicts the att shifts; cell-size weighting predicts invariance).
-2. **Sup-t bootstrap gating** — runs only when `aggregate="event_study"` AND `(weights= or survey_design= supplied)` AND `cband=True`. Unweighted event-study bit-exactly preserves pre-Phase 4.5 B output. Locked in `TestHADDeviations::test_sup_t_bootstrap_skipped_*`.
+2. **Sup-t bootstrap gating** — runs only in the event-study mode (multi-period panels; the deprecated `aggregate=` override maps to it, M-027) AND `(weights= or survey_design= supplied)` AND `cband=True`. Unweighted event-study bit-exactly preserves pre-Phase 4.5 B output. Locked in `TestHADDeviations::test_sup_t_bootstrap_skipped_*`.
 3. **Pierce-Schott Figure 2 replication waived** — R parity at `atol=1e-8` is a stronger anchor; paper Section 5.2 self-acknowledges NP estimators are too noisy on LBD-restricted PNTR data. See REGISTRY Deviations § "Pierce-Schott (2016) Figure 2 replication harness deferred" for the full scope-caveat statement.
 4. **Table 1 coverage-rate reproduction waived** — same R-parity-is-stronger rationale; R parity locks point estimate + SE + CI bounds bit-exactly, coverage-rate MC would re-verify the CCF asymptotic coverage already pinned. Paper Table 1 (89% / 93% / 95% under-coverage at G=100 / 500 / 2500) documents the asymptotic gap that BOTH R and Python inherit.
-5. **Staggered-timing fail-closed `ValueError`** at `diff_diff/had.py:1511` (paper prescribes "Warn"; library raises). Library extension toward stricter safety — `UserWarning` would let the silent-misuse bug class through. Locked in `TestHADDeviations::test_staggered_timing_fail_closed_value_error`.
+5. **Staggered-timing fail-closed `ValueError`** at `diff_diff/had.py:1624` (paper prescribes "Warn"; library raises). Library extension toward stricter safety — `UserWarning` would let the silent-misuse bug class through. Locked in `TestHADDeviations::test_staggered_timing_fail_closed_value_error`.
 6. **Eq. 18 linear-trend-detrended joint Stute SHIPPED** (PR #389) and R-parity-locked against `DIDHAD::did_had(..., trends_lin=TRUE)` v2.0.0 in `tests/test_did_had_parity.py` (3 DGPs × 5 method combos at `atol=1e-8`). The `tests/test_methodology_had.py::TestHADJointStute` walkthrough deliberately covers only the un-detrended mean-independence and linearity variants (no coverage duplication with the R-parity surface). The Pierce-Schott (2016) NUMERICAL replication against the published p=0.51 anchor on the LBD-restricted PNTR panel is what's waived (Deviations Note #3).
 
 **Outstanding Concerns:**
@@ -1468,7 +1468,7 @@ Goldens at `benchmarks/data/r_conleyreg_conley_golden.json`; generator at `bench
 - **Survey + non-HC1 vcov** — HC2 / HC2-BM / classical **explicitly raise `NotImplementedError`** under `survey_design=` (a fail-closed guard: the survey TSL / replicate-refit variance would otherwise silently discard the requested sandwich family): `stacked_did.py:428`, `sun_abraham.py:751`, `wooldridge.py:702`, `twfe.py:252`.
 - **Survey + user `cluster=`** — **explicitly raises** (a fail-closed guard, not a silent drop): the survey TSL / replicate-refit variance would otherwise ignore `cluster=`, so the combination is rejected at construction: `efficient_did.py:518`, `staggered.py:1719`, `imputation.py:314`, `two_stage.py:1439`, `triple_diff.py:674`.
 - **SyntheticControl** — no survey support yet (`synthetic_control.py:335`).
-- **HAD-specific** — `trends_lin=True` + survey (`had.py:3067`); QUG pretest + survey (extreme-order statistic not smooth in the empirical CDF, `had_pretests.py:1455`); `lonely_psu='adjust'` + singleton strata on the sup-t / Stute bootstrap.
+- **HAD-specific** — `trends_lin=True` + survey (`had.py:3192`); QUG pretest + survey (extreme-order statistic not smooth in the empirical CDF, `had_pretests.py:1455`); `lonely_psu='adjust'` + singleton strata on the sup-t / Stute bootstrap.
 - **HonestDiD M>0 smoothness** survey FLCI uses asymptotic normal only (`df_survey=0`→NaN); tracked in the HonestDiD section.
 
 ---