Skip to content

fix(cluster): SSH 暂断时保留原 VM 推理会话 - #1168

Merged
dofastted merged 3 commits into
mainfrom
fix/cluster-vm-session-continuity
Nov 17, 2026
Merged

dofastted merged 3 commits into
mainfrom
fix/cluster-vm-session-continuity

Conversation

@dofastted

@dofastted dofastted commented Nov 16, 2026 •

Copy link
Copy Markdown
Owner

改动

  • 将已配置的 TLS 直连健康探测与 SSH 管理链路解耦。SSH 暂断时,已建立的 TLS 流继续;调度用槽位内核健康结果确认原 VM 可用,不以 gate 可达代替内核健康。
  • 两条链路都不可用时,在原调度预算内等待并保留会话绑定,超时返回 503 cluster_slot_reconnecting。不修改手动停调度和维护状态。
  • 仅在业务字节尚未转发时回退到同一 VM 的 SSH;可能已投递的请求不自动重放或换号,返回 502 cluster_delivery_unconfirmed 或 SSE 错误事件,不合成成功结束。

本地验证

  • 22 个改动的 JS 文件通过 Biome,git diff --check 通过。
  • 定向回归 577 通过、1 失败;SSH 暂断保流、原 VM 恢复和断流不重放的新增测试通过。失败的原生 handoff 用例在无此补丁的主线同样复现。
  • 后端核心集 262 通过、1 失败:api-channel-routing 审查计数断言不稳定,单文件复核 35/35,同文件在主线也出现审查计数断言失败;不将整套核心集标为通过。
  • 未部署,未进行真实集群断链验证。TLS 直连需按 docs/PANEL_API.md 启用。

Note

High Risk
Changes cluster slot routing, pool scheduling, inference transport, and error semantics on critical request paths; misbehavior could strand sessions, drop in-flight work, or mis-report delivery.

Overview
Decouples TLS direct-link health from the SSH management path and changes how cluster slots stay schedulable during link blips.

The control plane now probes kin-slot-gate every 5s (independent of SSH). direct_link.state uses fresh probe results instead of SSH-backed checks. slotDataPath / vmNodeDataPath classify each VM as ssh, direct, or offline: during SSH backoff/connecting, an existing relay plus a healthy gate can keep the VM on the direct TLS path instead of treating the node as offline.

Scheduling and transport wait on the sticky/home VM within the existing budget when both paths are down (503 cluster_slot_reconnecting), prefer the home VM when direct is viable, and require real worker/kernel health on the direct path—not gate ping alone. SocketRelay waits for gate acceptance before forwarding bytes and can safely fall back to SSH only before payload crosses the gate.

No silent replay: possibly-delivered cluster requests surface 502 cluster_delivery_unconfirmed; failover, kernel-router retries, stream truncation, and SSE handling stop without account failover or max_tokens fake completion. Slot diagnosis and local proxy probing are adjusted so SSH-only checks don’t falsely fail egress/container state when direct data path is up.

Reviewed by Cursor Bugbot for commit 8b93c6e. Configure here.

(cherry picked from commit 55c7ea84e9d1e405aeb9b932456f32c4bf36fdc8)
@cursor

cursor Bot commented Nov 16, 2026

Copy link
Copy Markdown
Contributor

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: 4912a271-a66b-4271-b419-a3165ba12e15)

60cec60 起 live CRAG health 会在推理 POST 前校验 license,
该用例未准备 license,被 license_unconfigured 403 挡在上游之前。
@cursor

cursor Bot commented Nov 17, 2026

Copy link
Copy Markdown
Contributor

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: 719bf75d-ee23-40bc-8901-de210d5e8e40)

@dofastted
dofastted merged commit 49871a8 into main Nov 17, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant