ui-server: persistent gRPC connection to frontend has no keepalive, silently goes stale after backend pod recycle

[itavy]

Describe the bug

server/rpc/rpc.go's Dial() creates a single, long-lived *grpc.ClientConn for the lifetime of the ui-server process (this is intentional - see #2149, which moved away from connection-per-request for performance). However, that persistent connection is created with grpc.WithConnectParams (governing reconnect backoff after a failure is already detected) but no grpc.WithKeepaliveParams. Without keepalive pings, the client has no active way to detect a half-dead connection - e.g. if the specific backend Temporal frontend server it originally connected to becomes unreachable (a pod restart/rolling deploy behind a Kubernetes Service, or any environment where the frontend process can change independently of ui-server), the client can be left holding a connection that looks open but never proxies further requests to a live backend.

Every backend-proxied API (/api/v1/namespaces, /api/v1/cluster-info, etc.) then returns 503 immediately (fast, not a timeout - consistent with a connection-level failure rather than a hung RPC), and this persists indefinitely: nothing in ui-server proactively closes and re-dials the stale connection. Only a full process restart (forcing a new grpc.Dial() call) recovers it. ui-server's own /healthz only checks the process itself, not backend gRPC connectivity, so this isn't caught by a typical Kubernetes liveness probe either.

To Reproduce

This is an infrastructure-level repro, not a UI click-path:

1. Run ui-server pointed at a Temporal frontend running behind a Kubernetes Service, in an environment where frontend pods can be replaced independently of the ui-server pod (e.g. a normal rolling deploy of the Temporal server Deployment).
2. Leave ui-server running. At some point, cause the specific frontend pod/connection it's using to be replaced (a rolling restart of the frontend Deployment is sufficient - this doesn't require anything exotic).
3. Without restarting ui-server, call any endpoint that proxies to the Temporal gRPC API, e.g. GET /api/v1/namespaces or GET /api/v1/cluster-info.
4. Observe a 503 returned quickly (single-digit milliseconds), even though the Temporal cluster itself is healthy and reachable (verified independently via a fresh CLI connection to the same frontend).
5. Restart the ui-server pod/process (no other change). The same requests immediately start succeeding again.

Expected behavior

The persistent gRPC connection should be resilient to the backend server it's talking to becoming unreachable - either via grpc.WithKeepaliveParams(keepalive.ClientParameters{...}) so gRPC actively probes the connection and triggers the existing reconnect/backoff logic on failure, or via some other mechanism that avoids requiring a full ui-server process restart to recover.

Screenshots

N/A - backend/API-level issue, not a rendering bug.

Desktop (please complete the following information):

• N/A (server-side gRPC client issue, not browser-specific)

Additional context

• ui-server version: temporalio/ui:2.52.0
• Relevant code: server/rpc/rpc.go's Dial() - grpc.ConnectParams is configured, grpc.WithKeepaliveParams is not.
• Confirmed the mTLS client-cert side of things is not the cause: server/rpc/tls.go's certLoader.GetClientCertificate already correctly reloads the client cert from disk per-handshake, so this isn't a stale-cert issue.
• Reproduced with a long-running ui-server instance whose backend Temporal frontend Deployment had been rolled several times since ui-server last started - every namespace/cluster-info call failed with 503 until ui-server itself was restarted, at which point everything worked immediately with no other change.