** Work in Progress**: This is an experimental exploration into using reinforcement learning to optimize Kubernetes node resource allocation. The goal is to improve latency for critical workloads through dynamic CPU weight adjustments.
A Kubernetes node resource controller that attempts to dynamically optimize CPU allocation (cpu.weight) across workloads using reinforcement learning (contextual bandit). This is a research/exploration project - it's not production-ready and the approaches are still being tested and refined. The controller runs as a node-agent and tries to balance competing workload requirements: maintaining low latency for critical services while handling batch processing.
This project is exploring a closed-loop control system that:
- Discovers and classifies workloads running on a Kubernetes node by reading labels
- Collects metrics including CPU usage from cgroups, latency from app endpoints, and scheduling data
- Uses reinforcement learning (contextual bandit) to make decisions about CPU weight adjustments
- Has a safety layer to validate actions before applying them to prevent obvious violations
- Attempts to learn by collecting rewards based on observed system performance changes
Targets workload environments with multiple competing classes:
- Latency-critical services (e.g., APIs, real-time processing) - trying to keep low latency
- Batch workloads (e.g., data processing, ML jobs) - less latency-sensitive
- Default services (e.g., monitoring, logging) - best effort
Note: Whether this actually improves things or just maintains status quo is still being investigated.
The controller runs a continuous control loop every 5 seconds, following this sequence:
┌─────────────────────────────────────────────────────────────────────┐
│ Control Loop (Every 5 seconds) │
└─────────────────────────────────────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────┐
│ 1. Discover Pods on Node │
│ - Watch Kubernetes API │
│ - Filter by node name │
│ - Extract labels & annotations │
└──────────────────┬───────────────────────────┘
│
▼
┌──────────────────────────────────────────────┐
│ 2. Classify Workloads │
│ - Read pod labels │
│ - Group by: latency-critical, batch, │
│ default │
└──────────────────┬───────────────────────────┘
│
▼
┌──────────────────────────────────────────────┐
│ 3. Collect Metrics │
│ - CPU usage per class (from cgroups) │
│ - P95/P99 latency per class │
│ (scrape from app /metrics) │
│ - Current cpu.weight per class │
│ - Runqueue length (contention proxy) │
└──────────────────┬───────────────────────────┘
│
▼
┌──────────────────────────────────────────────┐
│ 4. Build State Vector │
│ - Aggregate metrics │
│ - Normalize values │
│ - Include previous action │
└──────────────────┬───────────────────────────┘
│
▼
┌──────────────────────────────────────────────┐
│ 5. Query RL Optimizer (Contextual Bandit) │
│ - Send State │
│ - Receive Action │
│ - Timeout: 100ms │
└──────────────────┬───────────────────────────┘
│
▼
┌──────────────────────────────────────────────┐
│ 6. Validate Action (Safety Layer) │
│ - Check weight bounds (100-10000) │
│ - Check rate limits │
│ - Validate delta from current │
└──────────────────┬───────────────────────────┘
│
▼
┌────────────────┐
│ Valid Action? │
└────────┬───────┘
│
┌────────────┴────────────┐
│ │
Yes No
│ │
▼ ▼
┌──────────────────────┐ ┌──────────────────┐
│ 7. Execute Action │ │ 7. Skip Action │
│ - Update cgroups │ │ - Log reason │
│ - Log action │ └──────────────────┘
└──────────┬───────────┘
│
▼
┌──────────────────────────────────────────────┐
│ 8. Wait & Collect New State (2 seconds) │
│ - Allow metrics to reflect change │
│ - Collect post-action state │
└──────────┬───────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────┐
│ 9. Calculate Reward │
│ - Latency improvement │
│ - SLA violations penalty │
│ - Oscillation penalty │
└──────────┬───────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────┐
│ 10. Send Feedback to RL (Learn) │
│ - Previous state │
│ - Action taken │
│ - Reward received │
│ - New state │
└──────────┬───────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────┐
│ 11. Check for Degradation │
│ - Compare P99 with snapshot │
│ - If >20% increase → Rollback │
└──────────┬───────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────┐
│ 12. Update Metrics │
│ - Export to Prometheus │
│ - Update action counters │
└──────────┬───────────────────────────────────┘
│
▼
[Wait 5 seconds]
│
└─────── Loop ────────┘
The controller collects metrics from multiple sources in step 3:
- Reads from
/sys/fs/cgroup/hierarchy per workload class - Aggregates CPU time across all pods in each class
- Calculates delta from previous collection interval for real-time usage
- Important: You must verify the correct cgroupv2 paths on your system (see Verifying cgroupv2 Paths)
- Scrapes Prometheus-format metrics from each pod's metrics endpoint
- Extracts P95 and P99 latency percentiles
- Aggregates per workload class
- Used for SLA validation and reward calculation
- Reads runqueue length from
/proc/schedstator similar sources - Serves as a proxy for CPU scheduling contention
- Helps detect when the system is under-provisioned
- Reads current
cpu.weightvalues from cgroups - Tracks the last action taken for state continuity
- Used to validate and bound the next action
- Docker installed on your machine
- KinD (Kubernetes in Docker) cluster running
- kubectl configured to access your cluster
- Go 1.21+ (if building locally)
kind create cluster --name multi-node-cluster --config kind_config.yamlIf you don't have a kind_config.yaml, create one:
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: multi-node-cluster
nodes:
- role: control-plane
- role: worker
- role: workerdocker build -t rl-node-controller:latest .kind load docker-image rl-node-controller:latest --name multi-node-clusterCreate a DaemonSet to run the controller on each node:
kubectl apply -f - <<EOF
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: rl-node-controller
namespace: kube-system
spec:
selector:
matchLabels:
app: rl-node-controller
template:
metadata:
labels:
app: rl-node-controller
spec:
hostNetwork: true
hostPID: true
nodeSelector: {}
serviceAccountName: rl-node-controller
containers:
- name: controller
image: rl-node-controller:latest
imagePullPolicy: IfNotPresent
env:
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
volumeMounts:
- name: cgroup
mountPath: /sys/fs/cgroup
- name: host-proc
mountPath: /proc
securityContext:
privileged: true
resources:
requests:
cpu: 100m
memory: 128Mi
limits:
cpu: 500m
memory: 512Mi
volumes:
- name: cgroup
hostPath:
path: /sys/fs/cgroup
- name: host-proc
hostPath:
path: /proc
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: rl-node-controller
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: rl-node-controller
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["nodes"]
verbs: ["get", "list"]
- apiGroups: [""]
resources: ["namespaces"]
verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: rl-node-controller
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: rl-node-controller
subjects:
- kind: ServiceAccount
name: rl-node-controller
namespace: kube-system
EOFDeploy workloads with proper classifications:
# Latency-critical service
kubectl apply -f test-workload/latency-critical-pod.yaml
kubectl apply -f test-workload/latency-critical-service.yaml
# Batch workload
kubectl apply -f test-workload/batch-pod.yaml
kubectl apply -f test-workload/batch-service.yaml
# Load generator
kubectl apply -f test-workload/load-generator.yaml# Follow controller logs from a specific node
kubectl logs -n kube-system -l app=rl-node-controller -f
# View metrics collected
kubectl port-forward -n kube-system daemonset/rl-node-controller 9090:9090
# Then visit http://localhost:9090/metrics# SSH into your KinD node or run this inside a pod
# Check if cgroupv2 is mounted
mount | grep cgroup
# Verify the structure
ls -la /sys/fs/cgroup/
# For a specific pod, find its cgroup path
kubectl get pod <pod-name> -o jsonpath='{.metadata.uid}'
# Then look in /sys/fs/cgroup/pod<uid>/
# Read CPU metrics
cat /sys/fs/cgroup/pod.slice/pod-<uid>.slice/cpu.statEdit cgroup/cgroupManager.go to ensure it uses the correct paths for:
- CPU stat files:
cpu.stat - Memory files:
memory.current - Weight configuration:
cpu.weight
Common paths:
- Container runtime managed:
/sys/fs/cgroup/pod.slice/pod-{uid}.slice/ - Kubelet default:
/sys/fs/cgroup/kubepods/ - Systemd managed:
/sys/fs/cgroup/system.slice/
Test your configuration:
# Inside the controller pod or on the node
ls -la /sys/fs/cgroup/pod.slice/
cat /sys/fs/cgroup/pod.slice/*/cpu.statThe Dockerfile builds a multi-stage image:
# Build stage
FROM golang:1.21-alpine AS builder
WORKDIR /build
COPY . .
RUN go mod download
RUN CGO_ENABLED=0 GOOS=linux go build -o controller .
# Runtime stage
FROM alpine:latest
RUN apk --no-cache add ca-certificates curl
COPY --from=builder /build/controller /usr/local/bin/
CMD ["controller"]Build manually:
# Standard build
docker build -t rl-node-controller:latest .
# With build args
docker build \
--build-arg GO_VERSION=1.21 \
-t rl-node-controller:v1.0 .
# Debug build (with logs)
docker build --progress=plain -t rl-node-controller:debug .# Deploy controller DaemonSet
kubectl apply -f controller-deployment.yaml
# Check controller status
kubectl get daemonset -n kube-system rl-node-controller
kubectl describe daemonset -n kube-system rl-node-controller
# View logs
kubectl logs -n kube-system -l app=rl-node-controller
kubectl logs -n kube-system -l app=rl-node-controller -p # previous logs
kubectl logs -n kube-system -l app=rl-node-controller --tail=100 -f
# View on specific node
kubectl logs -n kube-system -l app=rl-node-controller --field-selector spec.nodeName=<node-name>
# Execute command in controller pod
kubectl exec -it -n kube-system <pod-name> -- sh# Deploy all test workloads
kubectl apply -f test-workload/
# Check pod status
kubectl get pods -L node-controller-workload-class,workload-type
kubectl describe pod <pod-name>
# View pod metrics (requires metrics-server)
kubectl top pod
# Port-forward to services
kubectl port-forward svc/latency-critical-service 8080:80
kubectl port-forward svc/batch-service 8081:80# Check events
kubectl get events --sort-by='.lastTimestamp'
# Verify RBAC permissions
kubectl auth can-i get pods --as=system:serviceaccount:kube-system:rl-node-controller
# Check node resources
kubectl describe node <node-name>
kubectl top node
# Exec into pod and check cgroups
kubectl exec -it <pod-name> -- cat /proc/self/cgroup
# Access metrics endpoint
kubectl exec -it <pod-name> -- curl localhost:8080/metrics- controller/: Main control loop and orchestration
- metrics/: Metrics collection from cgroupv2 and application endpoints
- cgroup/: cgroupv2 interface and manipulation
- pods/: Kubernetes pod discovery and classification
- rl-client/: Contextual bandit implementation and decision-making
- safety/: Safety constraints and rollback mechanisms
- config/: Configuration management
# Run tests (if any exist)
go test ./...
# Build locally
go build -o bin/controller
# Run with default config
./bin/controllerEdit config/kind_config.yaml to experiment with:
- Control loop interval
- Workload classifications
- SLA thresholds
- Safety constraint parameters
- Does the RL agent actually make better decisions or just cause oscillations?
- How sensitive is the system to the reward function design?
- Does cgroupv2 path detection work reliably across different environments?
- Is 5 second control loop interval appropriate?
- Are the metric aggregations capturing the right signals?
- How much does the safety layer actually help vs. preventing beneficial actions?
Disclaimer: This is experimental work still in progress. Not recommended for production. Use for research, testing, and learning purposes only.