feat(migration): internal migration-readiness endpoint (content + site search) (#36360)

[fabrizzio-dotCMS]

Problem

Support/QA had no single place to answer "is it safe to change the OpenSearch migration phase right now, and if not, what do I fix?" before promoting. QA Round 2 on #36360 showed the failure mode: promoting a phase while an index's ES and OpenSearch copies were out of sync led to silent empty/partial results, and the downgrade case (Phase-3-only content invisible after a rollback) was undocumented.

This adds an internal, read-only migration-readiness endpoint that condenses that status with actionable recommendations, and retires the role-gated .os reveal in the index portlets in favor of it.

What this adds

GET /api/v1/index/migration/readiness — internal, read-only, advisory. Never mutates anything; the fix is always the operator re-running the crawl/reindex, which self-heals through the existing write-path gates (#36797 read half, #36825 write half).

• Not public. @Hidden (absent from the OpenAPI / API-playground schema) and gated to CMS administrators or members of the migration support role (OS_MIGRATION_INDEX_VISIBILITY_ROLE_KEY, default os_migration_qa); anyone else gets a 403.
• Covers both mirrored families — versioned content indices (working/live) and Site Search indices. content is a keyed object (WORKING / LIVE — a fixed pair); siteSearch is a list (an open set). Each entry is a per-index ES↔OS diff: {indexName, es:{exists,docCount,physicalName}, os:{exists,docCount,physicalName}, verdict, recommendation}, verdict IN_SYNC / MISSING_COUNTERPART / COUNT_DRIFT. physicalName is the full name as stored on each server (cluster-prefixed; .os-tagged on OpenSearch), and the report carries the top-level clusterId.
• Overall verdict: current phase + read/write engines + a dualWrite flag; safeToAdvance (toward OpenSearch-only) and safeToRollback (downgrade) with outOfSyncCount, a human summary, and per-index blockers. Field order: clusterId, phase, content, siteSearch, verdict.
• Bare model response — returns the readiness object directly (no ResponseEntityView envelope), serialized by Jackson from typed records.
• Stateless, exact counts. Every field derived at request time. The Site Search half uses SiteSearchAPI.documentCount; the content half reads each engine leaf's getIndicesStats() (index _stats primaries.docs.count) — never a search total (capped at 10,000, which would hide drift on large indices). Both reconcilers query the two engine leaves directly (not the phase-aware router), so both sides show in every phase.
• safeToRollback needs no history: a downgrade routes reads back to Elasticsearch, so it is unsafe when any index's ES copy is behind its OpenSearch counterpart — derivable from the same snapshot, nothing persisted.

Teardown — index portlet visibility is now phase-only (reverts I-4). MigrationIndexVisibility no longer reveals .os by role: .os indices are hidden in Phases 0/1/2 and shown in Phase 3, for everyone. The role key is retained only to gate this endpoint, which is now the single source of truth for migration/QA.

Layers

• com.dotcms.content.index.migration: MirrorStatus (shared per-index diff), SiteSearchMirrorReconciler, ContentIndexMirrorReconciler, MigrationReadinessService (composition + verdict), MigrationReadiness (report DTO). Pure, no mutation.
• com.dotcms.rest.api.v1.index.MigrationReadinessResource: the JAX-RS resource + role gate.

Tests — 31 unit green

• MigrationReadinessServiceTest (16): advance/rollback verdicts across phases 0–3, missing-counterpart and >10k count-drift blocking advance, content-keyed-by-slot / siteSearch-as-list, clusterId, driftPercent, the phase-aware outOfSyncCount (Phase 0 expected-missing excluded, other mismatches still counted), and rollback fail-safe on unmeasurable counts.
• ContentIndexMirrorReconcilerTest (5): in-sync, missing OS counterpart, count drift, null/absent slots, physical names.
• MigrationReadinessResourceTest (5): role gate — admin/role-member allowed, non-admin-without-role denied, null user denied, access-lookup failure fails closed.
• MigrationIndexVisibilityTest (5): rewritten to the phase-only contract.

Remaining test (flagging): a container integration test that drives HTTP 403/200 through the JAX-RS stack is not included yet — the gate decision and both reconcilers are unit-covered. Happy to add it if you'd like it before merge.

Notes

• Stacked on fix(sitesearch): reconcile ES/OS index mirrors on the write path (#36360) #36825 (issue-36360-sitesearch-mirror-reconciliation) because it consumes SiteSearchAPI.documentCount from that PR. Based against that branch so the diff shows only the readiness changes; will retarget to main once fix(sitesearch): reconcile ES/OS index mirrors on the write path (#36360) #36825 merges.
• Docs: new "Migration-readiness endpoint" subsection in docs/backend/OPENSEARCH_MIGRATION.md.

🤖 Generated with Claude Code

This PR fixes: #36360

[claude]

Claude finished @fabrizzio-dotCMS's task in 11m 34s —— View job

---

Code Review — migration-readiness endpoint

Reviewed the readiness layers (MigrationReadinessService, MirrorStatus, both reconcilers, MigrationReadinessResource), the visibility teardown, and the tests. Verified against the three prior findings addressed in e6ae14ce.

New Issues

• 🟡 Medium: dotCMS/src/main/java/com/dotcms/content/index/migration/ContentIndexMirrorReconciler.java:66-67 — the ES/OS stats reads are unguarded, and the failure modes are asymmetric. osImpl.getIndicesStats() (OSIndexAPIImpl:475-478) catches everything and returns an empty map, so a down OpenSearch degrades to "OS copy missing". But esImpl.getIndicesStats() (ESIndexAPI:133-165) has no try/catch: it does ((Map)jsonMap.get("indices")).forEach(...) with no null check, so an unreachable/erroring Elasticsearch throws (NPE or the underlying request exception) straight through statuses() → evaluate() → the resource, which returns the model directly with no catch → HTTP 500. This reconciler deliberately made index-name loading fail-safe (loadIndiciesQuietly, line 116-121) but not the count read next to it — so the one endpoint an operator reaches for while a cluster is unhealthy can hard-fail on exactly that condition instead of reporting a partial diff. The Site Search half has the same shape: documentCount is internally guarded (returns -1), but esImpl.listIndices() / existsOnAllWriteEngines() in SiteSearchMirrorReconciler:53,63 are not. Assumption: ES getIndicesStats()/listIndices() throw when the ES cluster is unreachable. What to verify: whether that's the real behavior in your deployment — if so, wrap each stats/list read in a Try-with-warn (mirroring loadIndiciesQuietly and the OS side) so a single-engine outage yields a degraded report rather than a 500. Fix this →

Resolved

• ✅ SiteSearchMirrorReconciler — canEvaluate() dead code / phantom evaluable field removed; no remaining references in main or test.
• ✅ MigrationReadinessService.java:63-65,152-159 — outOfSyncCount is now phase-aware via needsAttentionIn(phase, status); Phase 0 excludes the expected not-yet-built OS counterpart while still counting an orphan OS copy or a two-copy drift. Covered by phase0_missingOsCounterparts_notCountedAsOutOfSync and phase0_unexpectedMismatch_stillCountedAsOutOfSync.
• ✅ MigrationReadinessService.java:169-173 — safeToRollback no longer greens on an unmeasurable count: blocksRollback treats docCount() < 0 on either engine as unsafe before the numeric compare, so es=100 / os=-1 no longer resolves through 100 < -1. Covered by unknownOsCount_notSafeToRollback / unknownEsCount_notSafeToRollback.

Notes (non-blocking, not flagged)

• The role gate correctly fails closed (isMigrationSupportUser returns false on any thrown lookup), requires admin and role, and @Hidden keeps it off the OpenAPI schema — matches the PR's stated contract.
• IndexResourceHelper.indexStatsList() / ESIndexResource dropping the User parameter is consistent with the phase-only visibility teardown; MigrationIndexVisibility.filter no longer consults a user.
• The Phase 0 leftover-index summary note assumes "left over on OpenSearch", but a two-copy count-drift row (also counted) would render the same wording slightly imprecisely — cosmetic only.
• As the PR body already flags, there's no container-level integration test driving 403/200 through the JAX-RS stack; the gate and reconcilers are unit-covered.

No blocking issues. The one Medium is the error-path asymmetry above — worth confirming before merge given this endpoint's role during migration troubleshooting.

· issue-36360-migration-readiness-endpoint

[mergify]

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

• Queue this pull request

[fabrizzio-dotCMS]

Thanks — all three findings were real. Addressed in e6ae14c.

1. canEvaluate() dead code / phantom evaluable field — removed. Confirmed no callers anywhere in main or test, and there is no evaluable field on MigrationReadiness/PhaseInfo. The phase interpretation the method encoded already lives in MigrationReadinessService (the per-phase advance/rollback branches), so the method was a duplicate API surface with no wiring. Dropped it and fixed the stale PR-body sentence, which now describes the dualWrite flag that actually ships.

2. Phase 0 outOfSyncCount contradicting the summary — now phase-aware. outOfSyncCount is computed through a new needsAttentionIn(phase, status) instead of the raw needsAttention(): in Phase 0 an index whose ES copy exists but whose OpenSearch counterpart does not is the expected state (counterparts are built during dual-write), so it no longer counts. The exclusion is deliberately narrow — an OpenSearch copy with no ES source, or a drift between two copies that both exist, is still unexpected before the migration starts and stays counted. To avoid re-creating the same contradiction in that rarer case, the Phase 0 "nothing to reconcile yet" summary now appends a note naming the leftover indices when the residual count is non-zero.

3. safeToRollback false green on an unmeasurable count — fixed. The check moved into blocksRollback(status), which treats docCount() < 0 on either engine as unsafe before any numeric comparison, so es=100 / os=-1 no longer resolves through 100 < -1 to a green. Your assumption about the sentinel was right for the Site Search half; for the content half I traced it: those counts come from getIndicesStats() and are existence-gated (absent key → exists=false, count 0), so -1 does not currently arise there — the guard is uniform anyway so the two halves cannot diverge later. The ES-side -1 was already fail-safe by accident (-1 < any real count) but is now explicit, which also covers es=-1 / os=-1.

Tests: 31 unit green (MigrationReadinessServiceTest 12 → 16), covering Phase 0 expected-missing excluded, Phase 0 unexpected mismatch still counted, and unknown OS / unknown ES counts blocking rollback. docs/backend/OPENSEARCH_MIGRATION.md updated for both contract changes.