Launch gateway hardening: probes, retries, sidecar CPU, DB pools, rate limiting

[lorenzo-norcini-scale]

Consolidated hardening from the 2026-08-13 gateway incident (per-tenant rate limiting tracked in MLI-8236, sync-forwarder timeout in MLI-8206). Supersedes #864, #865, #866 and #867.

Changes

Readiness (app + chart). healthcheck is now async def, so the /readyz probe no longer queues behind blocked threadpool requests; the probe gets an explicit timeoutSeconds (default 5, gateway.readinessProbeTimeoutSeconds). This removes the mechanism that ejected saturated-but-healthy pods (11/60 Ready during the incident). No livenessProbe is added: restarting a saturated pod discards in-flight work.

Async-task polls off the shared threadpool (app). GET /v1/async-tasks/{task_id} does a blocking result-backend read (S3 on AWS) and previously ran in the shared anyio threadpool, so a growing backlog's poll load starved every other sync route. The handler is now async and dispatches the read through a dedicated CapacityLimiter(40).

Retry policy (chart). VirtualService retries no longer include 503 (retryOn: connect-failure,unavailable,502,504); retrying the overload signal roughly 4x-ed offered load at saturation. Policy configurable under gateway.retries; perTryTimeout unset by default because the route carries streaming requests.

Sidecar CPU request (chart). sidecar.istio.io/proxyCPU: 250m on gateway pods. Without it, sidecars starved on CPU-packed nodes during scale-out and new pods never became Ready.

Lazy DB engines (app). DBManager built five engines per worker process eagerly (sync/async x RW/RO + NullPool); the gateway uses only the async pair. Engines are now built per kind on first use, cutting idle Postgres pools per pod by more than half and making scale-out stop storming the reader's connection limit.

Per-pod proxy rate limits (chart, default off). Values-gated EnvoyFilter installing local_ratelimit on the gateway sidecars: per-pod token buckets driven by a rateLimits.routes list (overflow 429s at the proxy without consuming a gateway worker) plus a throttledTenants list that clamps named callers on the async-task routes across both accepted Authorization forms, replacing hand-authored fault-injection during incidents.

Per-user rate limiting (app, default off). Redis-backed fixed-window limiter as a per-route FastAPI dependency (user_rate_limit(route_class), composed with authentication), using the existing aioredis pool with a cached client; counter increment and TTL are one atomic Lua eval; 429 + Retry-After. Configured via user_rate_limits in the service config with a log-only rollout mode. Fails open on any Redis error or a check exceeding 100ms; outage and log-only warnings are sampled.

Sync forwarder timeout (from #864). Explicit, configurable timeout on the sync forwarder.

Docs. Async-task polling guidance using the guide's existing tenacity idiom (jittered exponential backoff, bounded outstanding set).

Known limitation

Per-user limits key on the resolved user_id. While prod authentication falls back to FakeAuthenticationRepository (plugins package unimportable in the internal image), any string authenticates, so a caller could rotate credentials past user-keyed limits; the identity-blind per-pod buckets are the backstop. Fixing the plugins packaging is a separate security work item in the internal repo.

Verification

• tests/unit: all passing (one pre-existing test requires the WORKSPACE env var). Parametrized unit tests for the limiter: enforce vs log-only, unconfigured routes, fail-open on error and on timeout.
• Chart rendered with default and override values: probe timeout, retry block, proxyCPU annotation, and EnvoyFilter (route buckets, tenant clamp path-scoped, rendered Basic prefix matches the incident-derived value) all emit as expected; EnvoyFilter absent when disabled.
• black / ruff / isort / mypy pass on all touched files.
• Reviewed by 4 parallel cleanup agents plus 3 rounds of external review; all findings applied or explicitly deferred (security item above).

🤖 Generated with Claude Code

Greptile Summary

The PR hardens gateway behavior under saturation by adjusting probes and retries, isolating async-task polling, lazily creating database engines, and adding proxy and Redis-backed rate limiting.

• Adds configurable gateway readiness, sidecar CPU, retry, and local-rate-limit policies.
• Adds per-user async-task limits and a dedicated thread limiter for result polling.
• Lazily initializes database session families and introduces configurable forwarding timeouts.
• Documents bounded async-task polling with exponential backoff.

Confidence Score: 0/5

The PR does not appear safe to merge because proxy rate limiting remains ineffective and forgeable, async database pools are not disposed correctly, and Celery forwarding can still block indefinitely.

The current Envoy filter neither enables its virtual-host descriptor actions nor authenticates tenant header prefixes before consuming shared buckets; database credential refresh calls asynchronous disposal without awaiting it; and the Celery requests path still omits the newly configured timeout.

Files Needing Attention: charts/model-engine/templates/istio-ratelimit-envoyfilter.yaml, model-engine/model_engine_server/db/base.py, model-engine/model_engine_server/inference/forwarding/forwarding.py

Important Files Changed

Filename	Overview
charts/model-engine/templates/istio-ratelimit-envoyfilter.yaml	Adds optional per-pod route and tenant token buckets to gateway sidecars.
model-engine/model_engine_server/db/base.py	Replaces eager construction of five database engine families with synchronized lazy initialization and credential-expiry refresh.
model-engine/model_engine_server/inference/forwarding/forwarding.py	Adds a validated configurable timeout to the forwarder configuration and asynchronous HTTP transport.
model-engine/model_engine_server/api/rate_limits.py	Adds an authenticated Redis-backed fixed-window limiter with log-only, fail-open, timeout, and circuit-breaker behavior.
model-engine/model_engine_server/api/tasks_v1.py	Applies per-user limits to async-task routes and moves blocking result retrieval onto a dedicated capacity-limited thread pool.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  Client[Client] --> Envoy[Gateway sidecar]
  Envoy --> API[FastAPI gateway]
  API --> Redis[(Redis rate-limit counter)]
  API --> PollLimiter[Dedicated poll thread limiter]
  PollLimiter --> ResultBackend[(Async result backend)]
  API --> DBManager[Lazy DBManager]
  DBManager --> PostgreSQL[(PostgreSQL)]
  API --> Forwarder[Inference forwarder]
  Forwarder --> Model[User model service]

Loading

_{Reviews (4): Last reviewed commit: "fix(chart+gateway): Envoy 1.23 compatibi..." | Re-trigger Greptile}

Context used (4)

• Knowledge Base — Model-engine API and HTTP surface
• Knowledge Base — Database persistence for model-engine state
• Knowledge Base — Model-engine deployment and startup
• Knowledge Base — Inference forwarding across HTTP and Celery

[greptile-apps]

Virtual-host descriptors remain disabled

When proxy rate limiting is enabled, the route and tenant actions are attached to the virtual host, but the local rate-limit filter does not enable vh_rate_limits; requests therefore use the effectively unlimited default bucket instead of the configured descriptors, so async-task shedding and tenant clamps do not return the intended 429 responses.

Suggested change

	always_consume_default_token_bucket: false
	always_consume_default_token_bucket: false
	vh_rate_limits: true

Knowledge Base Used: Model-engine deployment and startup

Prompt To Fix With AI

This is a comment left during a code review.
Path: charts/model-engine/templates/istio-ratelimit-envoyfilter.yaml
Line: 44

Comment:
**Virtual-host descriptors remain disabled**

When proxy rate limiting is enabled, the route and tenant actions are attached to the virtual host, but the local rate-limit filter does not enable `vh_rate_limits`; requests therefore use the effectively unlimited default bucket instead of the configured descriptors, so async-task shedding and tenant clamps do not return the intended 429 responses.

```suggestion
            always_consume_default_token_bucket: false
            vh_rate_limits: true
```

**Knowledge Base Used:** [Model-engine deployment and startup](https://app.greptile.com/scale-ai/-/custom-context/knowledge-base/scaleapi/llm-engine/-/docs/deployment-and-startup.md)

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[greptile-apps]

Async engine disposal is unawaited

When an Azure database credential approaches expiry after an async engine has been cached, this synchronous loop calls the coroutine-based AsyncEngine.dispose() without awaiting it, leaving the old async pool and connections undisposed while replacement pools are created.

Knowledge Base Used: Database persistence for model-engine state

Prompt To Fix With AI

This is a comment left during a code review.
Path: model-engine/model_engine_server/db/base.py
Line: 255-256

Comment:
**Async engine disposal is unawaited**

When an Azure database credential approaches expiry after an async engine has been cached, this synchronous loop calls the coroutine-based `AsyncEngine.dispose()` without awaiting it, leaving the old async pool and connections undisposed while replacement pools are created.

**Knowledge Base Used:** [Database persistence for model-engine state](https://app.greptile.com/scale-ai/-/custom-context/knowledge-base/scaleapi/llm-engine/-/docs/database-persistence.md)

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[greptile-apps]

Celery forwarding remains unbounded

When the user service accepts a Celery-forwarded request but never returns a response, Forwarder.__call__ ignores the configured timeout_seconds value and waits indefinitely in requests.post, causing workers to remain occupied until available worker capacity is exhausted.

Knowledge Base Used: Inference forwarding across HTTP and Celery

Prompt To Fix With AI

This is a comment left during a code review.
Path: model-engine/model_engine_server/inference/forwarding/forwarding.py
Line: 178

Comment:
**Celery forwarding remains unbounded**

When the user service accepts a Celery-forwarded request but never returns a response, `Forwarder.__call__` ignores the configured `timeout_seconds` value and waits indefinitely in `requests.post`, causing workers to remain occupied until available worker capacity is exhausted.

**Knowledge Base Used:** [Inference forwarding across HTTP and Celery](https://app.greptile.com/scale-ai/-/custom-context/knowledge-base/scaleapi/llm-engine/-/docs/inference-forwarding.md)

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.