feat: replace Elasticsearch with PostgreSQL full-text search (PPT-2644, 2/2)

[camreeves]

Depends on the models search_vector migration (models branch PPT-2644-pg-full-text-search, migration 20260810100500000) being deployed; builds against models 9.x as-is.
(2026-08-10: rebased onto current master — zero conflicts with the PPT-2000/2032 consent work and #444 — and OPENAPI_DOC.yml regenerated to pick up the new admin-consent flow endpoint master had added without regenerating.)

What

Every index route (24 controllers) now queries PostgreSQL directly instead of Elasticsearch:

• q is translated by Utils::TextSearch into a bound-parameter to_tsquery('simple', …) of AND-ed prefix tokens against the generated search_vector columns. The sanitizer never raises and degrades ES-era syntax (field prefixes, +|-"()*~, boolean words) into plain terms — pinned by spec vectors including Backoffice's tags:(+level AND +building) string.
• Filters became SQL WHERE clauses with the same semantics they effectively had (multi-zone AND-containment, only-when-true booleans, etc.).
• ES parent-child became EXISTS joins: modules↔driver (search modules by driver name — instantly consistent on driver rename, unlike the stale-copy ES join docs), system-triggers↔trigger, systems↔trigger-instances, and assets↔asset-type (previously a commented-out TODO, now implemented).
• Deterministic ordering everywhere (name, id default) — ES "relevance" order was already discarded by the id-rehydration path, and intra-page order was PG-arbitrary.
• /reindex + /backfill are deprecated 200 no-ops (Backoffice's admin card calls them); search_ingest left the /cluster/versions fan-out; the search-ingest + direct neuroplastic deps are gone (neuroplastic remains transitively via models until the models major drops it).
• Exact totals in X-Total-Count, and the Link header off-by-one that silently skipped one row per page (also affecting the 7 pre-existing SQL routes) is fixed. ref is accepted-and-ignored; fields likewise (it never was a projection).
• Spec harness: elastic + search-ingest containers gone; ~40 refresh_elastic/sleep/poll sites now synchronous asserts.
• OPENAPI_DOC.yml regenerated — zero ES references.

Deliberate behavior fixes (each with a pinning spec)

1. GET /systems?trigger_id= — never returned anything (ES query was unsatisfiable) → real join.
2. GET /systems?email= — no-op alone; widened the group/subsystem authorization scope when combined → strict AND filter.
3. Scoped (non-admin) GET /modules returned [] whenever the user could access >1 module → IN().
4. Alerts multi-dashboard scoping matched nothing for authorities with >1 dashboard → IN().
5. alerts?enabled=false, asset_categories?hidden=false were silently ignored → filter both values.
6. GET /systems/:id/triggers?q= was silently ignored → matches parent trigger text.

New coverage for previously untested surface: authentications (×3, no spec file existed), domains index, schema, webrtc /rooms, short_url index, and the systems filter matrix (capacity/bookable/features/public/signage/zone_id/email/trigger_id).

Deliberate divergences

• Secrets/encrypted content are no longer searchable (ES indexed password_digest, api-key secrets, unencrypted settings_string, …).
• Multi-word q is AND-of-prefixes (ES OR-ed terms; autocomplete UIs already intersect client-side).
• fields=name^2 boost syntax is ignored.
• asciifolding (café→cafe) dropped ('simple' config, matching the guests precedent).

Test evidence (2026-08-10, local, post-rebase)

Full containerised suite (incl. core + build_service containers): 876 examples, 0 failures, 1 error, 1 pending. The pending is the pre-existing Edges::ConnectionManager skip. The single error is short_url_spec:6 — a pre-existing models defect, not this branch: Shortener#short_id allows only 63 distinct ids per wall-clock second ((now << 6) + rand(63)), so two same-second creates collide at ~1/63; standalone re-runs green. Worth its own one-line follow-up in models. All 4 specs that need core/build_service (websocket exec/debug, public_events driver call, build_monitor, root cluster-versions) passed.

⚠ Before merge

• spec/migration/Dockerfile pinned to the models branch — reverted in 41867ad; the lock now tracks released models 9.107.1 (migration fix(edge/connection): refactor how connections are handled #324 + Shortener fix fix(webhook): resolve 404 on friendly webhook URI #325, both merged).

Rollout: merge models migration → this → one release with ES still deployed-but-unread (rollback = previous image) → then infra removal (branches prepared: local, k8s-helm, PlaceOS meta, init, user-interfaces e2e, backoffice, ts-client, docs).

Full plan + research: https://gist.github.com/camreeves/4b3cb2cffd2544e894ec34dcdae799a6

CI flake ledger (pre-existing, not from this PR)

For reviewers triaging CI: the suite runs with randomized order, and these master specs are order/timing/randomness-sensitive independent of this change — each was diagnosed during this PR's verification (details in the branch history):

• api_key_spec "rejects expired API keys" — 1s expiry vs 1.5s sleep, thin margin on a loaded runner.
• websocket/session_spec "debug"/"ignore" — core round-trip race (test(websocket): gate the bind/debug/ignore specs on the response arriving #444 reduced but did not eliminate it).
• uploads_spec:363 — creates a default Storage where some orders leave one behind (authority_id need to be unique).
• short_url_spec Shortener collision — fix(webhook): resolve 404 on friendly webhook URI #325 merged; the lock update in 41867ad resolves this one.
• playlist_spec/user_spec (models repo) — documented order/lock flakes, reproduce on master.

Two genuinely new flaky specs introduced by this PR were found and fixed in-branch (Generator.module rolls a random driver role; specs asserting scoped/filtered visibility now pin a non-logic role).

[greptile-apps]

Greptile Summary

The PR replaces Elasticsearch-backed index searches with PostgreSQL full-text queries and SQL relationship filters, while retaining deprecated indexing endpoints as compatibility no-ops.

• Adds shared text-query sanitization and SQL pagination helpers.
• Converts controller filtering, authorization scoping, and relationship searches to PostgreSQL queries.
• Removes direct Elasticsearch/search-ingest dependencies and updates the test harness.
• Regenerates the OpenAPI document for the revised search and pagination contracts.

Confidence Score: 5/5

The PR appears safe to merge because no blocking failure remains within the eligible follow-up-review scope.

No blocking failure remains.

Important Files Changed

Filename	Overview
src/placeos-rest-api/utilities/text-search.cr	Introduces bounded Unicode tokenization and bound-parameter prefix tsquery construction.
src/placeos-rest-api/controllers/application.cr	Adds shared PostgreSQL search pagination and corrects the next-page offset calculation.
src/placeos-rest-api/controllers/systems.cr	Converts system search and filter composition to SQL, including relationship-based trigger filtering.
src/placeos-rest-api/controllers/modules.cr	Converts module search and scoped access filtering to SQL with driver relationship matching.
src/placeos-rest-api/controllers/assets.cr	Converts asset search and filters to SQL and adds asset-type relationship matching.
src/placeos-rest-api/controllers/root.cr	Retains reindex and backfill routes as deprecated compatibility no-ops.
shard.yml	Removes direct search-ingest and neuroplastic dependencies after the PostgreSQL search migration.
OPENAPI_DOC.yml	Regenerates API documentation for PostgreSQL search semantics, offset pagination, and deprecated indexing routes.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
    Client[API index request] --> Params[Parse q, filters, limit, offset]
    Params --> Sanitize[Sanitize q into prefix tokens]
    Sanitize --> TSQuery[Bound to_tsquery simple]
    Params --> Scope[Apply authorization and SQL filters]
    TSQuery --> Query[PostgreSQL model query]
    Scope --> Query
    Query --> Exists[Optional relationship EXISTS joins]
    Exists --> Page[Deterministic ordering and pagination]
    Page --> Response[JSON results plus count and Link headers]

Loading

_{Reviews (4): Last reviewed commit: "chore: track released placeos-models 9.1..." | Re-trigger Greptile}