Skip to content

Kubernetes: run the jump pod with Guaranteed QoS - #4451

Merged
un-def merged 1 commit into
masterfrom
issue_4407_k8s_jump_pod_resources
Nov 3, 2026
Merged

un-def merged 1 commit into
masterfrom
issue_4407_k8s_jump_pod_resources

Conversation

@un-def

@un-def un-def commented Nov 2, 2026

Copy link
Copy Markdown
Collaborator

The jump pod requested no resources, so it was BestEffort, the first pod the kubelet evicts under node pressure. Since #4447, its ReplicaSet recreates it, but every eviction still drops all SSH sessions through the jump pod.

  • The jump pod container now requests and limits CPU and memory equally, which gives it the Guaranteed QoS class. The defaults, 500m and 1Gi, come from measuring the jump pod image: ~9 MiB plus ~1.6 MiB per tunneled SSH session, and 130-180 MiB/s of tunneled traffic at 500m.
  • The new proxy_jump.cpu and proxy_jump.memory properties override the defaults. Values must be Kubernetes quantities of at least 100m and 32Mi.
  • The container also requests 64Mi of ephemeral storage, without a limit. QoS doesn't cover disk pressure, under which the kubelet first evicts pods whose disk usage exceeds their ephemeral-storage request.
  • A jump pod that can't be scheduled, e.g., due to insufficient resources, is now logged as a warning with the scheduler's reason. Before, the only symptom was runs hitting the provisioning timeout.

Notes:

  • New values don't apply to an existing jump pod. The docs describe how to apply them with kubectl.
  • A legacy jump pod that the ReplicaSet adopts on upgrade stays BestEffort until it's recreated.

Fixes: #4407

The jump pod requested no resources, so it was BestEffort, the first
pod the kubelet evicts under node pressure. Since #4447, its ReplicaSet
recreates it, but every eviction still drops all SSH sessions through
the jump pod.

* The jump pod container now requests and limits CPU and memory
  equally, which gives it the Guaranteed QoS class. The defaults, 500m
  and 1Gi, come from measuring the jump pod image: ~9 MiB plus ~1.6 MiB
  per tunneled SSH session, and 130-180 MiB/s of tunneled traffic at
  500m.
* The new proxy_jump.cpu and proxy_jump.memory properties override the
  defaults. Values must be Kubernetes quantities of at least 100m and
  32Mi.
* The container also requests 64Mi of ephemeral storage, without a
  limit. QoS doesn't cover disk pressure, under which the kubelet first
  evicts pods whose disk usage exceeds their ephemeral-storage request.
* A jump pod that can't be scheduled, e.g., due to insufficient
  resources, is now logged as a warning with the scheduler's reason.
  Before, the only symptom was runs hitting the provisioning timeout.

Notes:

* New values don't apply to an existing jump pod. The docs describe
  how to apply them with kubectl.
* A legacy jump pod that the ReplicaSet adopts on upgrade stays
  BestEffort until it's recreated.

Fixes: #4407
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@un-def
un-def merged commit cc8b950 into master Nov 3, 2026
29 checks passed
@un-def
un-def deleted the issue_4407_k8s_jump_pod_resources branch November 3, 2026 07:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Kubernetes runs keep failing after the jump pod is evicted

1 participant