From 98aabeda518c28267d43b4da5d62e07636467ee7 Mon Sep 17 00:00:00 2001 From: Said Boudjelda Date: Tue, 20 Oct 2026 02:07:14 +0200 Subject: [PATCH 1/4] fix(chaos): envOverrides may not pick the Litmus namespace or pods The namespace guard checks a step's targetNamespace, but a Litmus step's envOverrides went to the ChaosEngine as given, after the env Kates sets. The chaos-runner (3.28) sets CHAOS_NAMESPACE and TARGETS and then lets the engine's env overwrite them; litmus-go looks TARGET_PODS up in the namespace of TARGETS, or else CHAOS_NAMESPACE; pod-network-partition cuts off APP_LABEL's pods in APP_NAMESPACE. Every experiment runs as litmus-admin, which the kates-chaos chart lets delete pods and evict in every namespace by default (rbac.nodeChaos.enabled). So an override such as APP_NAMESPACE=kube-system, or CHAOS_NAMESPACE with TARGET_PODS, aimed a step past both the namespace check and the broker count. FaultLimits already refuses the overrides TOTAL_CHAOS_DURATION and RAMP_TIME; it now refuses CHAOS_NAMESPACE, TARGETS, APP_NAMESPACE, APP_LABEL, APP_KIND and TARGET_PODS too. The safety guard applies it just before the namespace check, and the coordinator and the compound orchestrator apply it as well. TARGET_NODE and NODE_LABEL, which pick the node a NODE_DRAIN drains, stay allowed; the guard checks that node. The guard test for APP_NAMESPACE=kube-system fails on the old guard. The book, the disruption guide and the examples README name the refused variables. No caveat ref moves. --- cli/examples/README.md | 2 +- docs/book/07-chaos-practice.md | 5 ++++- kates/docs/disruption-guide.md | 2 +- .../com/bmscomp/kates/chaos/FaultLimits.java | 20 +++++++++++++++++- .../bmscomp/kates/chaos/FaultLimitsTest.java | 13 ++++++++++++ .../disruption/DisruptionSafetyGuardTest.java | 21 +++++++++++++++++++ 6 files changed, 59 insertions(+), 4 deletions(-) diff --git a/cli/examples/README.md b/cli/examples/README.md index 32b2ab1d..5ea5c948 100644 --- a/cli/examples/README.md +++ b/cli/examples/README.md @@ -156,7 +156,7 @@ that sets a field its run doesn't read, or that the API would refuse. | `networkLatencyMs` | int | Added latency ms (`NETWORK_LATENCY`) | | `targetTopic` | string | A disruption plan aims a fault at the leader of this topic's `targetPartition`. A resilience run doesn't, but a `DNS_ERROR` on `litmus-crd` takes it as `TARGET_HOSTNAMES` | | `targetPartition` | int | Partition of `targetTopic`, in a disruption plan only | -| `envOverrides` | map | Env vars for the Litmus experiment (`litmus-crd` only). For a `NODE_DRAIN`, `TARGET_NODE` or `NODE_LABEL` picks the node instead of the pod `targetLabel` picks | +| `envOverrides` | map | Env vars for the Litmus experiment (`litmus-crd` only). For a `NODE_DRAIN`, `TARGET_NODE` or `NODE_LABEL` picks the node instead of the pod `targetLabel` picks. None may pick the experiment's namespace or pods (`CHAOS_NAMESPACE`, `TARGETS`, `APP_NAMESPACE`, `APP_LABEL`, `APP_KIND`, `TARGET_PODS`) | ### `probes` field reference diff --git a/docs/book/07-chaos-practice.md b/docs/book/07-chaos-practice.md index 58e5e2cf..816060c4 100644 --- a/docs/book/07-chaos-practice.md +++ b/docs/book/07-chaos-practice.md @@ -188,6 +188,8 @@ A fault's `targetLabel` is a Kubernetes label selector, in the syntax `kubectl - A fault acts in its `targetNamespace`, `kafka` when the step leaves it out. Kates refuses a step in any namespace but two: the Kafka cluster's, `kates.chaos.kafka.namespace` (`kafka` by default), and the one the Kates API runs in (`kates` on a default install). The `kates` chart's ClusterRole lets the Kates API delete and patch pods in every namespace, so this refusal is what keeps a pod fault off the rest of the cluster, such as the CoreDNS pods in `kube-system`. A `NODE_DRAIN` still evicts every pod on its node, whatever its namespace. +On `litmus-crd`, `envOverrides` may not set the Litmus variables that pick the experiment's namespace or pods: `CHAOS_NAMESPACE`, `TARGETS`, `APP_NAMESPACE`, `APP_LABEL`, `APP_KIND` and `TARGET_PODS`. Litmus applies an override after the namespace and pods Kates sets, and by default the `kates-chaos` chart lets `litmus-admin` delete pods in every namespace (`rbac.nodeChaos.enabled`), so an override could aim a step past both the namespace check and the broker count. Kates refuses such a step before any fault. `TARGET_NODE` and `NODE_LABEL` still pick the node a `NODE_DRAIN` drains. + A pod-level fault hits one pod unless told otherwise. The first of these that applies decides which pods: 1. `targetPod`, when set. @@ -563,6 +565,7 @@ A refused plan starts nothing. Over the API it returns `422` with status `REJECT | `Step '…': NODE_DRAIN with targetAll picks pods on N nodes (…)` | Narrow `targetLabel` to the pods of one node, or name the node in `envOverrides.TARGET_NODE` | | `Step '…': NODE_DRAIN: …, the node the Kates API runs on, and draining it would evict the Kates API …` | Aim the drain at a pod on another node, or name another node in `envOverrides.TARGET_NODE` | | `Step '…':`, then a fault parameter that `is below` its floor or `is above the limit of` its ceiling | Bring the parameter into its range, or raise the limit (see [Fault Parameter Limits](#fault-parameter-limits)) | +| `Step '…': envOverrides.… would aim the Litmus experiment at a namespace or pods the safety guard doesn't check; …` | Drop the override, and pick the pods with `targetNamespace`, `targetPod`, `targetLabel` or `targetBrokerId` | | `Step '…': targetNamespace … is neither …, the Kafka cluster's namespace (kates.chaos.kafka.namespace), nor the Kates API's own, …` | Aim the step at the Kafka cluster's namespace or the Kates API's own. No setting adds another | | `Plan would affect N brokers but maxAffectedBrokers=M` | Narrow the selectors, or raise the limit if you mean it | | `Plan would affect ALL N brokers — cluster would lose availability` | Narrow the selectors: Kates never runs a plan that hits every broker | @@ -606,7 +609,7 @@ Every number in a fault spec has a range, and Kates refuses a fault with one out Kates checks every parameter whatever the fault's type, so a value no provider reads has to be in range too. A field you leave out gets its default, which is always in range. -Two rules go beyond the table. First, a fault that ends only when its duration does needs a `chaosDurationSec` of at least 1: `NETWORK_PARTITION`, `NETWORK_LATENCY`, `CPU_STRESS`, `MEMORY_STRESS`, `IO_STRESS`, `DNS_ERROR`, `DISK_FILL` and `NODE_DRAIN`. The others may set `0`, because a deleted pod comes back by itself, and a `ROLLING_RESTART` or `SCALE_DOWN` with `0` doesn't wait. Second, `envOverrides` may not set `TOTAL_CHAOS_DURATION` or `RAMP_TIME`, which would run a Litmus experiment past the limits. +Two rules go beyond the table. First, a fault that ends only when its duration does needs a `chaosDurationSec` of at least 1: `NETWORK_PARTITION`, `NETWORK_LATENCY`, `CPU_STRESS`, `MEMORY_STRESS`, `IO_STRESS`, `DNS_ERROR`, `DISK_FILL` and `NODE_DRAIN`. The others may set `0`, because a deleted pod comes back by itself, and a `ROLLING_RESTART` or `SCALE_DOWN` with `0` doesn't wait. Second, `envOverrides` may not set `TOTAL_CHAOS_DURATION` or `RAMP_TIME`, which would run a Litmus experiment past the limits, nor the variables that pick its namespace or pods (see [Targeting Pods](#targeting-pods)). The duration ceiling matters most on `kubernetes`, where a CPU or IO stress runs in an ephemeral container that Kubernetes can't remove: its duration is all that stops it. A `ROLLING_RESTART`'s `chaosDurationSec` is its wait for the Cluster Operator to roll every pod, so on a cluster whose roll takes over an hour, raise `kates.chaos.limits.max-duration-sec`. diff --git a/kates/docs/disruption-guide.md b/kates/docs/disruption-guide.md index df3dc1de..32e6f1a9 100644 --- a/kates/docs/disruption-guide.md +++ b/kates/docs/disruption-guide.md @@ -361,7 +361,7 @@ This prevents a common mistake: using a broad label selector like `strimzi.io/co A typo in a number does as much damage as one in a selector: a stress meant for 60 seconds that runs for 60,000, or `cpuCores: 0`, which Litmus reads as every core. So every fault's parameters are checked against the `kates.chaos.limits.*` ceilings before anything is injected, whichever way the fault comes in: a plan, a playbook, a template, a schedule, a compound run or a resilience run. A plan with a fault outside them is rejected like any other unsafe plan; a compound or resilience run gets a `400` naming each parameter. -With the defaults, `chaosDurationSec` runs from 0 to 3,600 seconds, `delayBeforeSec` from 0 to 600, `gracePeriodSec` from 0 to 300, `networkLatencyMs` from 1 to 30,000, `fillPercentage` from 1 to 100, `cpuCores` and `ioWorkers` from 1 to 64, and `memoryMb` from 1 to 32,768. The [configuration reference](overview.md#chaos-limits) names the setting for each ceiling. A fault undone when its duration ends needs a `chaosDurationSec` of at least 1: `NETWORK_PARTITION`, `NETWORK_LATENCY`, `CPU_STRESS`, `MEMORY_STRESS`, `IO_STRESS`, `DNS_ERROR`, `DISK_FILL` and `NODE_DRAIN`. `envOverrides` may not set `TOTAL_CHAOS_DURATION` or `RAMP_TIME`, which would run a Litmus experiment past the limits. +With the defaults, `chaosDurationSec` runs from 0 to 3,600 seconds, `delayBeforeSec` from 0 to 600, `gracePeriodSec` from 0 to 300, `networkLatencyMs` from 1 to 30,000, `fillPercentage` from 1 to 100, `cpuCores` and `ioWorkers` from 1 to 64, and `memoryMb` from 1 to 32,768. The [configuration reference](overview.md#chaos-limits) names the setting for each ceiling. A fault undone when its duration ends needs a `chaosDurationSec` of at least 1: `NETWORK_PARTITION`, `NETWORK_LATENCY`, `CPU_STRESS`, `MEMORY_STRESS`, `IO_STRESS`, `DNS_ERROR`, `DISK_FILL` and `NODE_DRAIN`. `envOverrides` may not set `TOTAL_CHAOS_DURATION` or `RAMP_TIME`, which would run a Litmus experiment past the limits, nor `CHAOS_NAMESPACE`, `TARGETS`, `APP_NAMESPACE`, `APP_LABEL`, `APP_KIND` or `TARGET_PODS`, which would aim it at a namespace or pods the safety guard doesn't check. ### RBAC Permission Verification diff --git a/kates/src/main/java/com/bmscomp/kates/chaos/FaultLimits.java b/kates/src/main/java/com/bmscomp/kates/chaos/FaultLimits.java index 39f38c28..a73a31a9 100644 --- a/kates/src/main/java/com/bmscomp/kates/chaos/FaultLimits.java +++ b/kates/src/main/java/com/bmscomp/kates/chaos/FaultLimits.java @@ -54,15 +54,33 @@ public class FaultLimits { DisruptionType.DISK_FILL, DisruptionType.NODE_DRAIN); + private static final String UNCHECKED_TARGET = "would aim the Litmus experiment at a namespace or pods the safety" + + " guard doesn't check; set targetNamespace, targetPod, targetLabel or targetBrokerId instead"; + /** * Litmus settings an override would take past the limits. Kates sets * TOTAL_CHAOS_DURATION from chaosDurationSec, and an override comes after * it; RAMP_TIME makes Litmus wait before the fault and after it, longer * than Kates waits for the result. + * + *

The rest pick the experiment's namespace or pods (litmus-go and + * chaos-runner 3.28), past the namespaces and the blast radius the safety + * guard checks, and by the kates-chaos chart's defaults litmus-admin may + * delete pods in every namespace. The + * chaos-runner applies an override after the CHAOS_NAMESPACE and TARGETS + * it sets itself; pod experiments look up TARGET_PODS in either, and + * pod-network-partition cuts off APP_LABEL's pods in APP_NAMESPACE. + * TARGET_NODE and NODE_LABEL stay: the guard checks the node they drain. */ private static final Map UNBOUNDED_LITMUS_ENV = Map.of( "RAMP_TIME", "would delay the Litmus fault, with no limit, past the time Kates waits for it", - "TOTAL_CHAOS_DURATION", "would set the fault's duration past its limit; set chaosDurationSec instead"); + "TOTAL_CHAOS_DURATION", "would set the fault's duration past its limit; set chaosDurationSec instead", + "CHAOS_NAMESPACE", UNCHECKED_TARGET, + "TARGETS", UNCHECKED_TARGET, + "APP_NAMESPACE", UNCHECKED_TARGET, + "APP_LABEL", UNCHECKED_TARGET, + "APP_KIND", UNCHECKED_TARGET, + "TARGET_PODS", UNCHECKED_TARGET); // Each default is written twice: in the annotation, for the application, // and as the field's value, for a FaultLimits made with new, as tests make it. diff --git a/kates/src/test/java/com/bmscomp/kates/chaos/FaultLimitsTest.java b/kates/src/test/java/com/bmscomp/kates/chaos/FaultLimitsTest.java index 5cf347af..cba5c704 100644 --- a/kates/src/test/java/com/bmscomp/kates/chaos/FaultLimitsTest.java +++ b/kates/src/test/java/com/bmscomp/kates/chaos/FaultLimitsTest.java @@ -117,6 +117,19 @@ void anOverrideMayNotSetTheLitmusDurationOrRampTime() throws Exception { List.copyOf(limits.violations(spec).keySet())); } + /** What litmus-go 3.28 and its chaos-runner read to pick the experiment's namespace or pods. */ + @ParameterizedTest + @CsvSource({"CHAOS_NAMESPACE", "TARGETS", "APP_NAMESPACE", "APP_LABEL", "APP_KIND", "TARGET_PODS"}) + void anOverrideMayNotPickTheLitmusNamespaceOrPods(String key) { + FaultSpec spec = FaultSpec.builder("probe") + .envOverrides(Map.of(key, "kube-system", "TARGET_NODE", "gamma", "NODE_LABEL", "drainable=true")) + .build(); + + assertEquals( + List.of("envOverrides." + key), + List.copyOf(limits.violations(spec).keySet())); + } + @Test void checkRefusesNamingEveryParameterInFieldOrder() throws Exception { FaultSpec spec = posted("{\"experimentName\":\"runaway\",\"disruptionType\":\"CPU_STRESS\"," diff --git a/kates/src/test/java/com/bmscomp/kates/disruption/DisruptionSafetyGuardTest.java b/kates/src/test/java/com/bmscomp/kates/disruption/DisruptionSafetyGuardTest.java index 11630f0c..02a236b8 100644 --- a/kates/src/test/java/com/bmscomp/kates/disruption/DisruptionSafetyGuardTest.java +++ b/kates/src/test/java/com/bmscomp/kates/disruption/DisruptionSafetyGuardTest.java @@ -325,6 +325,27 @@ void aFaultInTheKatesNamespaceOrInNoneIsAllowed() { "a step without a namespace is in the Kafka namespace"); } + /** + * Litmus applies an override after the namespace and pods Kates sets, and + * litmus-admin may delete pods in every namespace, so this passed and the + * experiment ran in kube-system. + */ + @Test + void aLitmusOverrideThatPicksTheNamespaceIsRefused() { + createZonedBrokers(); + FaultSpec dns = FaultSpec.builder("dns") + .targetBrokerId(0) + .disruptionType(DisruptionType.POD_KILL) + .envOverrides(Map.of("APP_NAMESPACE", "kube-system")) + .build(); + + assertEquals( + List.of("Step 'step-0': envOverrides.APP_NAMESPACE would aim the Litmus experiment at a namespace or" + + " pods the safety guard doesn't check; set targetNamespace, targetPod, targetLabel or" + + " targetBrokerId instead"), + guard.validatePlan(plan(1, dns)).errors()); + } + private static FaultSpec rollingRestart(String selector, int chaosDurationSec) { return FaultSpec.builder("roll") .targetLabel(selector) From 32675610da5de18d9ac0bf5259c58e5c63542ae5 Mon Sep 17 00:00:00 2001 From: Said Boudjelda Date: Tue, 20 Oct 2026 02:32:00 +0200 Subject: [PATCH 2/4] fix(chaos): envOverrides may set only TARGET_NODE and NODE_LABEL Litmus starts helper pods for its stress, network, DNS and disk-fill experiments, and they run privileged as root with the host's PIDs. An override picked their image (LIB_IMAGE), their service account (CHAOS_SERVICE_ACCOUNT) and a host path they mount (SOCKET_PATH), and litmus-go 3.28 pastes many values into a shell in them: CHAOS_TYPE and MATCH_SCHEME unquoted, SPOOF_MAP and TARGET_HOSTNAMES in single quotes (pod-dns-error), CPU_LOAD and VOLUME_MOUNT_PATH (stress), and NETWORK_INTERFACE and the ports (network-chaos). So nearly any override could run a command as root on a node. Instead of refusing the keys known to be unsafe, FaultLimits now refuses every override except TARGET_NODE and NODE_LABEL, which pick the node a NODE_DRAIN drains; the guard checks that node, and Litmus uses both through the API, not a shell. The refusal covers the keys the previous commit listed, TOTAL_CHAOS_DURATION and RAMP_TIME, and any a later Litmus adds. Cluster settings such as CONTAINER_RUNTIME and SOCKET_PATH go in the kates-chaos chart's experiments.definitions[].env. targetTopic reached the same shell: on litmus-crd a DNS_ERROR passes it as TARGET_HOSTNAMES, and nothing checked it. It now has to be a Kafka topic name, 1 to 249 letters, digits, '.', '_' or '-'; a hostname such as krafter-kafka-bootstrap.kafka.svc still fits. The guard test now refuses APP_NAMESPACE and LIB_IMAGE together, and FaultLimitsTest covers 13 keys and the topic rule; they fail on the previous commit. The book, the disruption guide and the examples README say what envOverrides and targetTopic may hold. --- cli/examples/README.md | 4 +- docs/book/07-chaos-practice.md | 7 ++- kates/docs/disruption-guide.md | 4 +- .../com/bmscomp/kates/chaos/FaultLimits.java | 57 +++++++++--------- .../bmscomp/kates/chaos/FaultLimitsTest.java | 58 +++++++++++++++++-- .../disruption/DisruptionSafetyGuardTest.java | 15 +++-- 6 files changed, 97 insertions(+), 48 deletions(-) diff --git a/cli/examples/README.md b/cli/examples/README.md index 5ea5c948..b55251ac 100644 --- a/cli/examples/README.md +++ b/cli/examples/README.md @@ -154,9 +154,9 @@ that sets a field its run doesn't read, or that the API would refuse. | `fillPercentage` | int | Disk fill % (`IO_STRESS`) | | `ioWorkers` | int | Parallel I/O workers (`IO_STRESS`) | | `networkLatencyMs` | int | Added latency ms (`NETWORK_LATENCY`) | -| `targetTopic` | string | A disruption plan aims a fault at the leader of this topic's `targetPartition`. A resilience run doesn't, but a `DNS_ERROR` on `litmus-crd` takes it as `TARGET_HOSTNAMES` | +| `targetTopic` | string | A disruption plan aims a fault at the leader of this topic's `targetPartition`. A resilience run doesn't, but a `DNS_ERROR` on `litmus-crd` takes it as `TARGET_HOSTNAMES`. It has to be a Kafka topic name: letters, digits, `.`, `_` or `-` | | `targetPartition` | int | Partition of `targetTopic`, in a disruption plan only | -| `envOverrides` | map | Env vars for the Litmus experiment (`litmus-crd` only). For a `NODE_DRAIN`, `TARGET_NODE` or `NODE_LABEL` picks the node instead of the pod `targetLabel` picks. None may pick the experiment's namespace or pods (`CHAOS_NAMESPACE`, `TARGETS`, `APP_NAMESPACE`, `APP_LABEL`, `APP_KIND`, `TARGET_PODS`) | +| `envOverrides` | map | Env vars for the Litmus experiment (`litmus-crd` only). Only `TARGET_NODE` or `NODE_LABEL`, which pick the node a `NODE_DRAIN` drains instead of the pod `targetLabel` picks; any other variable is refused | ### `probes` field reference diff --git a/docs/book/07-chaos-practice.md b/docs/book/07-chaos-practice.md index 816060c4..82bd2d32 100644 --- a/docs/book/07-chaos-practice.md +++ b/docs/book/07-chaos-practice.md @@ -188,7 +188,7 @@ A fault's `targetLabel` is a Kubernetes label selector, in the syntax `kubectl - A fault acts in its `targetNamespace`, `kafka` when the step leaves it out. Kates refuses a step in any namespace but two: the Kafka cluster's, `kates.chaos.kafka.namespace` (`kafka` by default), and the one the Kates API runs in (`kates` on a default install). The `kates` chart's ClusterRole lets the Kates API delete and patch pods in every namespace, so this refusal is what keeps a pod fault off the rest of the cluster, such as the CoreDNS pods in `kube-system`. A `NODE_DRAIN` still evicts every pod on its node, whatever its namespace. -On `litmus-crd`, `envOverrides` may not set the Litmus variables that pick the experiment's namespace or pods: `CHAOS_NAMESPACE`, `TARGETS`, `APP_NAMESPACE`, `APP_LABEL`, `APP_KIND` and `TARGET_PODS`. Litmus applies an override after the namespace and pods Kates sets, and by default the `kates-chaos` chart lets `litmus-admin` delete pods in every namespace (`rbac.nodeChaos.enabled`), so an override could aim a step past both the namespace check and the broker count. Kates refuses such a step before any fault. `TARGET_NODE` and `NODE_LABEL` still pick the node a `NODE_DRAIN` drains. +On `litmus-crd`, `envOverrides` may set only `TARGET_NODE` and `NODE_LABEL`, which pick the node a `NODE_DRAIN` drains, and Kates refuses a step that sets any other variable before any fault. Litmus applies an override after the env Kates sets, so variables such as `CHAOS_NAMESPACE`, `TARGETS` or `TARGET_PODS` would aim the experiment past both the namespace check and the broker count, and by default the `kates-chaos` chart lets `litmus-admin` delete pods in every namespace (`rbac.nodeChaos.enabled`). Others reach the helper pods Litmus starts for stress, network and DNS faults, which run privileged as root: `LIB_IMAGE` is their image, `CHAOS_SERVICE_ACCOUNT` their service account, and the helpers paste values such as `CHAOS_TYPE` or `NETWORK_INTERFACE` into a shell command. Settings of the cluster, such as `CONTAINER_RUNTIME` and `SOCKET_PATH` on a cluster that doesn't run containerd, go in the `kates-chaos` chart's `experiments.definitions[].env` instead. A pod-level fault hits one pod unless told otherwise. The first of these that applies decides which pods: @@ -565,7 +565,8 @@ A refused plan starts nothing. Over the API it returns `422` with status `REJECT | `Step '…': NODE_DRAIN with targetAll picks pods on N nodes (…)` | Narrow `targetLabel` to the pods of one node, or name the node in `envOverrides.TARGET_NODE` | | `Step '…': NODE_DRAIN: …, the node the Kates API runs on, and draining it would evict the Kates API …` | Aim the drain at a pod on another node, or name another node in `envOverrides.TARGET_NODE` | | `Step '…':`, then a fault parameter that `is below` its floor or `is above the limit of` its ceiling | Bring the parameter into its range, or raise the limit (see [Fault Parameter Limits](#fault-parameter-limits)) | -| `Step '…': envOverrides.… would aim the Litmus experiment at a namespace or pods the safety guard doesn't check; …` | Drop the override, and pick the pods with `targetNamespace`, `targetPod`, `targetLabel` or `targetBrokerId` | +| `Step '…': envOverrides.… isn't one of the two envOverrides may set, TARGET_NODE and NODE_LABEL, …` | Drop the override and use the fault's own fields, such as `targetNamespace`, `targetLabel` or `chaosDurationSec`; set a cluster setting in the `kates-chaos` chart (see [Targeting Pods](#targeting-pods)) | +| `Step '…': targetTopic isn't a Kafka topic name: …` | Name the topic, or a `DNS_ERROR`'s host, with letters, digits, `.`, `_` or `-` only | | `Step '…': targetNamespace … is neither …, the Kafka cluster's namespace (kates.chaos.kafka.namespace), nor the Kates API's own, …` | Aim the step at the Kafka cluster's namespace or the Kates API's own. No setting adds another | | `Plan would affect N brokers but maxAffectedBrokers=M` | Narrow the selectors, or raise the limit if you mean it | | `Plan would affect ALL N brokers — cluster would lose availability` | Narrow the selectors: Kates never runs a plan that hits every broker | @@ -609,7 +610,7 @@ Every number in a fault spec has a range, and Kates refuses a fault with one out Kates checks every parameter whatever the fault's type, so a value no provider reads has to be in range too. A field you leave out gets its default, which is always in range. -Two rules go beyond the table. First, a fault that ends only when its duration does needs a `chaosDurationSec` of at least 1: `NETWORK_PARTITION`, `NETWORK_LATENCY`, `CPU_STRESS`, `MEMORY_STRESS`, `IO_STRESS`, `DNS_ERROR`, `DISK_FILL` and `NODE_DRAIN`. The others may set `0`, because a deleted pod comes back by itself, and a `ROLLING_RESTART` or `SCALE_DOWN` with `0` doesn't wait. Second, `envOverrides` may not set `TOTAL_CHAOS_DURATION` or `RAMP_TIME`, which would run a Litmus experiment past the limits, nor the variables that pick its namespace or pods (see [Targeting Pods](#targeting-pods)). +Three rules go beyond the table. First, a fault that ends only when its duration does needs a `chaosDurationSec` of at least 1: `NETWORK_PARTITION`, `NETWORK_LATENCY`, `CPU_STRESS`, `MEMORY_STRESS`, `IO_STRESS`, `DNS_ERROR`, `DISK_FILL` and `NODE_DRAIN`. The others may set `0`, because a deleted pod comes back by itself, and a `ROLLING_RESTART` or `SCALE_DOWN` with `0` doesn't wait. Second, `envOverrides` may set only `TARGET_NODE` and `NODE_LABEL` (see [Targeting Pods](#targeting-pods)), so not `TOTAL_CHAOS_DURATION` or `RAMP_TIME`, which would run a Litmus experiment past the limits. Third, a `targetTopic` has to be a Kafka topic name, 1 to 249 letters, digits, `.`, `_` or `-`: on `litmus-crd`, a `DNS_ERROR` passes it to a shell command as `TARGET_HOSTNAMES`. The duration ceiling matters most on `kubernetes`, where a CPU or IO stress runs in an ephemeral container that Kubernetes can't remove: its duration is all that stops it. A `ROLLING_RESTART`'s `chaosDurationSec` is its wait for the Cluster Operator to roll every pod, so on a cluster whose roll takes over an hour, raise `kates.chaos.limits.max-duration-sec`. diff --git a/kates/docs/disruption-guide.md b/kates/docs/disruption-guide.md index 32e6f1a9..8ba38c32 100644 --- a/kates/docs/disruption-guide.md +++ b/kates/docs/disruption-guide.md @@ -154,7 +154,7 @@ The `FaultSpec` is deliberately backend-agnostic. Whether you are using Litmus C | `networkLatencyMs` | `int` | `100` | Delay in milliseconds (NETWORK_LATENCY only) | | `fillPercentage` | `int` | `80` | Disk fill target (DISK_FILL only) | | `cpuCores` | `int` | `1` | CPU cores to stress (CPU_STRESS only) | -| `envOverrides` | `Map` | `{}` | Additional env vars for the chaos engine | +| `envOverrides` | `Map` | `{}` | `TARGET_NODE` or `NODE_LABEL` for a Litmus `NODE_DRAIN`; any other variable is refused | The defaults apply to a `faultSpec` posted as JSON as well as to a playbook step: a field the JSON leaves out gets the value in this table, and a field it sets, `0` included, keeps that value. So an omitted `targetBrokerId` means one random matching pod, and `"targetBrokerId": 0` means the broker pod whose name ends in `-0`. The same holds for each entry of `probes`. @@ -361,7 +361,7 @@ This prevents a common mistake: using a broad label selector like `strimzi.io/co A typo in a number does as much damage as one in a selector: a stress meant for 60 seconds that runs for 60,000, or `cpuCores: 0`, which Litmus reads as every core. So every fault's parameters are checked against the `kates.chaos.limits.*` ceilings before anything is injected, whichever way the fault comes in: a plan, a playbook, a template, a schedule, a compound run or a resilience run. A plan with a fault outside them is rejected like any other unsafe plan; a compound or resilience run gets a `400` naming each parameter. -With the defaults, `chaosDurationSec` runs from 0 to 3,600 seconds, `delayBeforeSec` from 0 to 600, `gracePeriodSec` from 0 to 300, `networkLatencyMs` from 1 to 30,000, `fillPercentage` from 1 to 100, `cpuCores` and `ioWorkers` from 1 to 64, and `memoryMb` from 1 to 32,768. The [configuration reference](overview.md#chaos-limits) names the setting for each ceiling. A fault undone when its duration ends needs a `chaosDurationSec` of at least 1: `NETWORK_PARTITION`, `NETWORK_LATENCY`, `CPU_STRESS`, `MEMORY_STRESS`, `IO_STRESS`, `DNS_ERROR`, `DISK_FILL` and `NODE_DRAIN`. `envOverrides` may not set `TOTAL_CHAOS_DURATION` or `RAMP_TIME`, which would run a Litmus experiment past the limits, nor `CHAOS_NAMESPACE`, `TARGETS`, `APP_NAMESPACE`, `APP_LABEL`, `APP_KIND` or `TARGET_PODS`, which would aim it at a namespace or pods the safety guard doesn't check. +With the defaults, `chaosDurationSec` runs from 0 to 3,600 seconds, `delayBeforeSec` from 0 to 600, `gracePeriodSec` from 0 to 300, `networkLatencyMs` from 1 to 30,000, `fillPercentage` from 1 to 100, `cpuCores` and `ioWorkers` from 1 to 64, and `memoryMb` from 1 to 32,768. The [configuration reference](overview.md#chaos-limits) names the setting for each ceiling. A fault undone when its duration ends needs a `chaosDurationSec` of at least 1: `NETWORK_PARTITION`, `NETWORK_LATENCY`, `CPU_STRESS`, `MEMORY_STRESS`, `IO_STRESS`, `DNS_ERROR`, `DISK_FILL` and `NODE_DRAIN`. `envOverrides` may set only `TARGET_NODE` and `NODE_LABEL`, which pick the node a `NODE_DRAIN` drains: Litmus applies an override after the env Kates sets, and its helper pods run privileged as root and paste some values into a shell. A `targetTopic` has to be a Kafka topic name, 1 to 249 letters, digits, `.`, `_` or `-`. ### RBAC Permission Verification diff --git a/kates/src/main/java/com/bmscomp/kates/chaos/FaultLimits.java b/kates/src/main/java/com/bmscomp/kates/chaos/FaultLimits.java index a73a31a9..b194aabd 100644 --- a/kates/src/main/java/com/bmscomp/kates/chaos/FaultLimits.java +++ b/kates/src/main/java/com/bmscomp/kates/chaos/FaultLimits.java @@ -4,6 +4,7 @@ import java.util.LinkedHashMap; import java.util.Map; import java.util.Set; +import java.util.regex.Pattern; import java.util.stream.Collectors; import jakarta.enterprise.context.ApplicationScoped; @@ -54,33 +55,23 @@ public class FaultLimits { DisruptionType.DISK_FILL, DisruptionType.NODE_DRAIN); - private static final String UNCHECKED_TARGET = "would aim the Litmus experiment at a namespace or pods the safety" - + " guard doesn't check; set targetNamespace, targetPod, targetLabel or targetBrokerId instead"; - /** - * Litmus settings an override would take past the limits. Kates sets - * TOTAL_CHAOS_DURATION from chaosDurationSec, and an override comes after - * it; RAMP_TIME makes Litmus wait before the fault and after it, longer - * than Kates waits for the result. - * - *

The rest pick the experiment's namespace or pods (litmus-go and - * chaos-runner 3.28), past the namespaces and the blast radius the safety - * guard checks, and by the kates-chaos chart's defaults litmus-admin may - * delete pods in every namespace. The - * chaos-runner applies an override after the CHAOS_NAMESPACE and TARGETS - * it sets itself; pod experiments look up TARGET_PODS in either, and - * pod-network-partition cuts off APP_LABEL's pods in APP_NAMESPACE. - * TARGET_NODE and NODE_LABEL stay: the guard checks the node they drain. + * The only Litmus variables an override may set: they pick the node a + * NODE_DRAIN drains, and the safety guard checks that node. Each of the + * rest takes a fault past what Kates checks (litmus-go and chaos-runner + * 3.28). The chaos-runner applies an override after the env it and Kates + * set: TOTAL_CHAOS_DURATION, RAMP_TIME, and the namespace and pods + * (CHAOS_NAMESPACE, TARGETS, TARGET_PODS). The helper pods + * run privileged as root with the host's PIDs: LIB_IMAGE is their image, + * CHAOS_SERVICE_ACCOUNT their account, SOCKET_PATH a host path they mount, + * and they paste values such as CHAOS_TYPE, CPU_LOAD or NETWORK_INTERFACE + * into a shell. Settings of the cluster, such as CONTAINER_RUNTIME, belong + * in the kates-chaos chart's experiment definitions. */ - private static final Map UNBOUNDED_LITMUS_ENV = Map.of( - "RAMP_TIME", "would delay the Litmus fault, with no limit, past the time Kates waits for it", - "TOTAL_CHAOS_DURATION", "would set the fault's duration past its limit; set chaosDurationSec instead", - "CHAOS_NAMESPACE", UNCHECKED_TARGET, - "TARGETS", UNCHECKED_TARGET, - "APP_NAMESPACE", UNCHECKED_TARGET, - "APP_LABEL", UNCHECKED_TARGET, - "APP_KIND", UNCHECKED_TARGET, - "TARGET_PODS", UNCHECKED_TARGET); + private static final Set LITMUS_ENV_OVERRIDES = Set.of("TARGET_NODE", "NODE_LABEL"); + + /** A legal Kafka topic name; on litmus-crd a DNS_ERROR's goes into a shell as TARGET_HOSTNAMES. */ + private static final Pattern TOPIC_NAME = Pattern.compile("[a-zA-Z0-9._-]{1,249}"); // Each default is written twice: in the annotation, for the application, // and as the field's value, for a FaultLimits made with new, as tests make it. @@ -127,10 +118,13 @@ public Map violations(FaultSpec spec) { } range(found, "delayBeforeSec", spec.delayBeforeSec(), 0, maxDelaySec, MAX_DELAY_SEC); if (spec.envOverrides() != null) { - UNBOUNDED_LITMUS_ENV.entrySet().stream() - .filter(e -> spec.envOverrides().containsKey(e.getKey())) - .sorted(Map.Entry.comparingByKey()) - .forEach(e -> found.put("envOverrides." + e.getKey(), e.getValue())); + spec.envOverrides().keySet().stream() + .filter(key -> !LITMUS_ENV_OVERRIDES.contains(key)) + .sorted() + .forEach(key -> found.put( + "envOverrides." + key, + "isn't one of the two envOverrides may set, TARGET_NODE and NODE_LABEL, which pick the" + + " node a NODE_DRAIN drains; set the fault's own fields instead")); } range(found, "networkLatencyMs", spec.networkLatencyMs(), 1, maxNetworkLatencyMs, MAX_NETWORK_LATENCY_MS); range(found, "fillPercentage", spec.fillPercentage(), 1, maxFillPercentage, MAX_FILL_PERCENTAGE); @@ -138,6 +132,11 @@ public Map violations(FaultSpec spec) { range(found, "memoryMb", spec.memoryMb(), 1, maxMemoryMb, MAX_MEMORY_MB); range(found, "ioWorkers", spec.ioWorkers(), 1, maxIoWorkers, MAX_IO_WORKERS); range(found, "gracePeriodSec", spec.gracePeriodSec(), 0, maxGracePeriodSec, MAX_GRACE_PERIOD_SEC); + if (spec.targetTopic() != null + && !spec.targetTopic().isEmpty() + && !TOPIC_NAME.matcher(spec.targetTopic()).matches()) { + found.put("targetTopic", "isn't a Kafka topic name: 1 to 249 letters, digits, '.', '_' or '-'"); + } return found; } diff --git a/kates/src/test/java/com/bmscomp/kates/chaos/FaultLimitsTest.java b/kates/src/test/java/com/bmscomp/kates/chaos/FaultLimitsTest.java index cba5c704..5dee7944 100644 --- a/kates/src/test/java/com/bmscomp/kates/chaos/FaultLimitsTest.java +++ b/kates/src/test/java/com/bmscomp/kates/chaos/FaultLimitsTest.java @@ -105,22 +105,43 @@ void aFaultThatEndsByItselfMayHaveNoDuration(DisruptionType type) { .build())); } - /** Litmus applies an override after the duration Kates sets, so it would undo the limit. */ @Test - void anOverrideMayNotSetTheLitmusDurationOrRampTime() throws Exception { + void refusedOverridesAreNamedInKeyOrder() throws Exception { FaultSpec spec = posted("{\"experimentName\":\"probe\",\"disruptionType\":\"NETWORK_LATENCY\"," + "\"envOverrides\":{\"TOTAL_CHAOS_DURATION\":\"86400\",\"RAMP_TIME\":\"3600\"," + "\"NETWORK_INTERFACE\":\"eth0\"}}"); assertEquals( - List.of("envOverrides.RAMP_TIME", "envOverrides.TOTAL_CHAOS_DURATION"), + List.of( + "envOverrides.NETWORK_INTERFACE", + "envOverrides.RAMP_TIME", + "envOverrides.TOTAL_CHAOS_DURATION"), List.copyOf(limits.violations(spec).keySet())); } - /** What litmus-go 3.28 and its chaos-runner read to pick the experiment's namespace or pods. */ + /** + * Litmus applies an override after the env Kates sets: its duration, the + * namespace and pods (litmus-go and chaos-runner 3.28). Others reach its + * helper pods, which run privileged as root: their image, their service + * account, and values the helpers paste into a shell. + */ @ParameterizedTest - @CsvSource({"CHAOS_NAMESPACE", "TARGETS", "APP_NAMESPACE", "APP_LABEL", "APP_KIND", "TARGET_PODS"}) - void anOverrideMayNotPickTheLitmusNamespaceOrPods(String key) { + @CsvSource({ + "TOTAL_CHAOS_DURATION", + "RAMP_TIME", + "CHAOS_NAMESPACE", + "TARGETS", + "APP_NAMESPACE", + "APP_LABEL", + "APP_KIND", + "TARGET_PODS", + "LIB_IMAGE", + "CHAOS_SERVICE_ACCOUNT", + "SOCKET_PATH", + "CHAOS_TYPE", + "NETWORK_INTERFACE" + }) + void anOverrideMaySetOnlyTheNodeADrainTakes(String key) { FaultSpec spec = FaultSpec.builder("probe") .envOverrides(Map.of(key, "kube-system", "TARGET_NODE", "gamma", "NODE_LABEL", "drainable=true")) .build(); @@ -130,6 +151,31 @@ void anOverrideMayNotPickTheLitmusNamespaceOrPods(String key) { List.copyOf(limits.violations(spec).keySet())); } + /** On litmus-crd, a DNS_ERROR's targetTopic goes into a shell command in Litmus's root helper pod. */ + @Test + void aTargetTopicMustBeAKafkaTopicName() { + FaultSpec.Builder dns = FaultSpec.builder("dns").disruptionType(DisruptionType.DNS_ERROR); + + assertEquals( + List.of("targetTopic"), + List.copyOf( + limits.violations(dns.targetTopic("x'; reboot -f; '").build()) + .keySet())); + assertEquals( + List.of("targetTopic"), + List.copyOf(limits.violations(dns.targetTopic("a".repeat(250)).build()) + .keySet())); + assertEquals( + Map.of(), + limits.violations(dns.targetTopic("__consumer_offsets").build())); + assertEquals( + Map.of(), + limits.violations( + dns.targetTopic("krafter-kafka-bootstrap.kafka.svc").build())); + assertEquals(Map.of(), limits.violations(dns.targetTopic("").build())); + assertEquals(Map.of(), limits.violations(dns.targetTopic(null).build())); + } + @Test void checkRefusesNamingEveryParameterInFieldOrder() throws Exception { FaultSpec spec = posted("{\"experimentName\":\"runaway\",\"disruptionType\":\"CPU_STRESS\"," diff --git a/kates/src/test/java/com/bmscomp/kates/disruption/DisruptionSafetyGuardTest.java b/kates/src/test/java/com/bmscomp/kates/disruption/DisruptionSafetyGuardTest.java index 02a236b8..462959ca 100644 --- a/kates/src/test/java/com/bmscomp/kates/disruption/DisruptionSafetyGuardTest.java +++ b/kates/src/test/java/com/bmscomp/kates/disruption/DisruptionSafetyGuardTest.java @@ -328,21 +328,24 @@ void aFaultInTheKatesNamespaceOrInNoneIsAllowed() { /** * Litmus applies an override after the namespace and pods Kates sets, and * litmus-admin may delete pods in every namespace, so this passed and the - * experiment ran in kube-system. + * experiment ran in kube-system. LIB_IMAGE picks the image of Litmus's + * helper pods, which run privileged as root. */ @Test - void aLitmusOverrideThatPicksTheNamespaceIsRefused() { + void aLitmusOverrideOtherThanTheDrainNodeIsRefused() { createZonedBrokers(); FaultSpec dns = FaultSpec.builder("dns") .targetBrokerId(0) .disruptionType(DisruptionType.POD_KILL) - .envOverrides(Map.of("APP_NAMESPACE", "kube-system")) + .envOverrides(Map.of("APP_NAMESPACE", "kube-system", "LIB_IMAGE", "example.com/any:1")) .build(); + String allowed = " isn't one of the two envOverrides may set, TARGET_NODE and NODE_LABEL, which pick the" + + " node a NODE_DRAIN drains; set the fault's own fields instead"; assertEquals( - List.of("Step 'step-0': envOverrides.APP_NAMESPACE would aim the Litmus experiment at a namespace or" - + " pods the safety guard doesn't check; set targetNamespace, targetPod, targetLabel or" - + " targetBrokerId instead"), + List.of( + "Step 'step-0': envOverrides.APP_NAMESPACE" + allowed, + "Step 'step-0': envOverrides.LIB_IMAGE" + allowed), guard.validatePlan(plan(1, dns)).errors()); } From 085bf6af7614e2199c5b09466bf891ba6fe4e0e6 Mon Sep 17 00:00:00 2001 From: Said Boudjelda Date: Tue, 20 Oct 2026 02:42:42 +0200 Subject: [PATCH 3/4] fix(chaos): NETWORK_PARTITION on litmus-crd is a Kates NetworkPolicy On the default litmus-crd provider a NETWORK_PARTITION ran the Litmus pod-network-partition experiment, which, from the litmus-go 3.28 source, never cut anything off: it fails with "provide the appLabel" when TARGETS is empty, and Kates builds the ChaosEngine with no appinfo, so the runner sets none. It also ignores TARGET_PODS, lists its targets again from TARGETS, and selects the pods for its NetworkPolicy by one label, so it could not aim at the one pod Kates picks anyway. The split-brain and consumer-isolation playbooks and the resilience-network-partition example relied on it. Not reproduced on a cluster. The Litmus provider now hands NETWORK_PARTITION to the kubernetes provider, as it does POD_DELETE, ROLLING_RESTART and SCALE_DOWN: a deny-all NetworkPolicy per target pod, matched by all of the pod's labels, removed when chaosDurationSec ends or the partition fails. The NETWORK_PARTITION entry leaves EXPERIMENT_MAP, which no longer reaches it. The kates chart (0.11.5) grants create, delete and deletecollection on networkpolicies in the default ClusterRole, out from behind rbac.directChaos, since both backends now need them; directChaos keeps statefulsets/scale and pods/ephemeralcontainers. The chart README, version tables, book (provider table, LitmusChaos integration, split-brain, dry-run RBAC, rollback), architecture chapter, playbook catalog, disruption guide and troubleshooting say so, and the two MCP caveats that list what litmus-crd hands the kubernetes provider name it too. Caveat and comment refs into LitmusChaosProvider.java follow its 5 new lines. LitmusChaosProviderTest checks that broker 2 gets a Kates NetworkPolicy for the duration and no ChaosEngine; it fails on the old routing. --- README.md | 2 +- charts/kates/Chart.yaml | 9 ++++-- charts/kates/README.md | 5 ++-- charts/kates/templates/rbac.yaml | 12 ++++---- charts/kates/values.yaml | 4 +-- cli/cmd/mcp_caveats.go | 2 +- cli/cmd/mcp_chaos_preview.go | 16 +++++----- cli/cmd/mcp_tools_chaos.go | 13 +++++---- docs/book/02-architecture.md | 6 ++-- docs/book/07-chaos-practice.md | 18 ++++++------ docs/book/appendix-d-versions.md | 2 +- kates/docs/disruption-guide.md | 2 +- kates/docs/playbook-catalog.md | 2 +- kates/docs/troubleshooting.md | 2 +- .../kates/chaos/LitmusChaosProvider.java | 13 ++++++--- .../kates/chaos/LitmusChaosProviderTest.java | 29 +++++++++++++++++++ 16 files changed, 88 insertions(+), 49 deletions(-) diff --git a/README.md b/README.md index 3cba2097..fbe8732a 100644 --- a/README.md +++ b/README.md @@ -213,7 +213,7 @@ When you install charts with Helm directly: | [`kafka-ui`](charts/kafka-ui/) | 0.3.0 | v1.5.0 | A Helm chart for deploying Kafka UI (Kafbat) with Strimzi SCRAM-SHA-512 authentication | | [`kates-chaos`](charts/kates-chaos/) | 2.2.1 | 3.28.0 | Kates Chaos Engineering — wraps the LitmusChaos execution plane (operator, exporter, CRDs) with Kafka-specific RBAC, experiment/engine templating, and monitoring | | [`kates-platform`](charts/kates-platform/) | 0.7.0 | 1.0.0 | Umbrella chart for the full Kates platform — Kafka, Kates, and supporting infrastructure | -| [`kates`](charts/kates/) | 0.11.4 | 1.26.0 | Kates — Kafka Advanced Testing & Engineering Suite | +| [`kates`](charts/kates/) | 0.11.5 | 1.26.0 | Kates — Kafka Advanced Testing & Engineering Suite | | [`legacy-kafka`](charts/legacy-kafka/) | 0.2.0 | 3.9.1 | A deliberately old Kafka (2.x on ZooKeeper, or 3.x on KRaft) to migrate FROM — the replication source that the pinned Strimzi release cannot deploy | | [`mirror-maker2`](charts/mirror-maker2/) | 0.11.5 | 4.3.1 | Strimzi KafkaMirrorMaker2 — cross-cluster replication, DR, and cross-version migration (2.x/3.x → 4.x) | | [`monitoring`](charts/monitoring/) | 1.6.0 | 82.4.3 | Kates Monitoring — wraps kube-prometheus-stack with Kates-specific dashboards and configuration | diff --git a/charts/kates/Chart.yaml b/charts/kates/Chart.yaml index b99d9417..022b3e91 100644 --- a/charts/kates/Chart.yaml +++ b/charts/kates/Chart.yaml @@ -2,9 +2,10 @@ apiVersion: v2 name: kates description: Kates — Kafka Advanced Testing & Engineering Suite type: application -# Patch: appVersion and the image tag move to 1.26.0, the tag the v1.26.0 -# release publishes. Templates are otherwise unchanged. -version: 0.11.4 +# Patch: the default ClusterRole grants create, delete and deletecollection +# on networkpolicies, which only rbac.directChaos granted before. +# appVersion is unchanged. +version: 0.11.5 # The non-blocking health endpoints this chart's probes now point at come from # a BUILD-TIME property (quarkus.messaging.health.enabled=false), so they only # exist in the image built from this branch. Keep appVersion and @@ -28,6 +29,8 @@ annotations: artifacthub.io/category: integration-delivery artifacthub.io/license: Apache-2.0 artifacthub.io/changes: | + - kind: fixed + description: "The default ClusterRole grants create, delete and deletecollection on networkpolicies, which only rbac.directChaos granted before. On the default litmus-crd provider, NETWORK_PARTITION ran the Litmus pod-network-partition experiment, which fails before cutting anything off without the appinfo Kates leaves out; the Kates API now applies its own deny-all NetworkPolicy to each target pod on both chaos providers, as the kubernetes provider did, and removes it when the duration ends. rbac.directChaos still grants statefulsets/scale and pods/ephemeralcontainers" - kind: changed description: "appVersion 1.26.0 and the image tag move to 1.26.0. The backend changes since 1.25.1: the cluster health check reports a KRaft quorum it cannot describe instead of dropping it (#403); the security audit and pentest read each listener's protocol (#379); the tuning report endpoint is gone (#386); run retention keeps the run a baseline names (#378); a status poll writes only over the status it read (#377); each INTEGRITY run reads in a consumer group of its own (#376); compound runs and resilience faults go through the safety guard and the disruption lease (#375); orphan recovery undoes faults left from before the API started (#374); and the native image starts with a named API keys file (#371). Templates are otherwise unchanged." - kind: added diff --git a/charts/kates/README.md b/charts/kates/README.md index 256e934d..e9272ead 100644 --- a/charts/kates/README.md +++ b/charts/kates/README.md @@ -111,15 +111,14 @@ Upgrading from 0.10.7 or earlier: the ClusterRole there granted `get`, `list` an ### Chaos permissions -With the default chaos provider, `litmus-crd`, Kates only creates and watches `ChaosEngine`s: Litmus runs every experiment as its own `litmus-admin` service account, and the default ClusterRole covers that. The direct Kubernetes backend (`kubernetes`, or `hybrid` on a cluster without Litmus CRDs) injects the faults itself, so it needs write access the default role does not grant. `rbac.directChaos=true` adds it: +With the default chaos provider, `litmus-crd`, Kates creates and watches `ChaosEngine`s, and Litmus runs every experiment as its own `litmus-admin` service account; the default ClusterRole covers that, and the faults Kates injects itself on both backends. The direct Kubernetes backend (`kubernetes`, or `hybrid` on a cluster without Litmus CRDs) injects the faults itself, so it needs write access the default role does not grant. `rbac.directChaos=true` adds it: | Resource | Verbs | Used by | |----------|-------|---------| -| `networking.k8s.io` `networkpolicies` | `create`, `delete`, `deletecollection` | `NETWORK_PARTITION` creates a deny-all policy; cleanup and rollback delete by label, startup orphan recovery by name | | `apps` `statefulsets/scale` | `get`, `update` | On a Kafka that Strimzi does not manage: `SCALE_DOWN`, its rollback and orphan recovery set replicas | | `pods/ephemeralcontainers` | `update` | `CPU_STRESS` and `IO_STRESS` add a stress container to the target pod | -`ROLLING_RESTART` needs none of these, on either backend, and neither does `SCALE_DOWN` on a Strimzi cluster. On a Strimzi cluster `ROLLING_RESTART` annotates the Kafka pods for the Strimzi Cluster Operator to roll, and on a plain StatefulSet, such as `charts/legacy-kafka`, it restarts the StatefulSet. `SCALE_DOWN` lowers `spec.replicas` of a KafkaNodePool. Both run on either backend, so the default role grants `patch` on pods, on StatefulSets and on `kafkanodepools`. On a Kafka that Strimzi does not manage, `SCALE_DOWN` also patches the StatefulSet, to record its original replica count. +`NETWORK_PARTITION` needs none of these, on either backend: Kates creates its deny-all NetworkPolicy itself, and cleanup, rollback and startup orphan recovery delete it, so the default role grants `create`, `delete` and `deletecollection` on `networkpolicies`. `ROLLING_RESTART` needs none of them either, and neither does `SCALE_DOWN` on a Strimzi cluster. On a Strimzi cluster `ROLLING_RESTART` annotates the Kafka pods for the Strimzi Cluster Operator to roll, and on a plain StatefulSet, such as `charts/legacy-kafka`, it restarts the StatefulSet. `SCALE_DOWN` lowers `spec.replicas` of a KafkaNodePool. Both run on either backend, so the default role grants `patch` on pods, on StatefulSets and on `kafkanodepools`. On a Kafka that Strimzi does not manage, `SCALE_DOWN` also patches the StatefulSet, to record its original replica count. These rules are cluster-wide, like the rest of the role, and `pods/ephemeralcontainers` lets Kates start a container in any pod. Leave them off unless you use the direct backend. It is selected with `chaos.provider`: diff --git a/charts/kates/templates/rbac.yaml b/charts/kates/templates/rbac.yaml index 41d57835..fdb3e5d9 100644 --- a/charts/kates/templates/rbac.yaml +++ b/charts/kates/templates/rbac.yaml @@ -51,20 +51,22 @@ rules: - apiGroups: ["core.strimzi.io"] resources: ["strimzipodsets"] verbs: ["get", "list"] + # NETWORK_PARTITION is a NetworkPolicy Kates applies itself on both chaos + # backends, so it is not behind rbac.directChaos. Cleanup, rollback and + # orphan recovery delete Kates NetworkPolicies by label: deletecollection. + - apiGroups: ["networking.k8s.io"] + resources: ["networkpolicies"] + verbs: ["create", "delete", "deletecollection"] {{- if .Values.rbac.directChaos }} {{- /* The direct Kubernetes chaos backend (kates.chaos.provider=kubernetes, or hybrid on a cluster without Litmus) makes these writes itself. Litmus runs every experiment as its own litmus-admin service account, so a Litmus - install needs none of them. Cleanup and rollback delete NetworkPolicies by - label, which is deletecollection; on a Kafka that Strimzi does not manage, + install needs none of them. On a Kafka that Strimzi does not manage, SCALE_DOWN sets replicas through the StatefulSet's scale subresource (its PATCH on the StatefulSet is granted above, for ROLLING_RESTART); CPU_STRESS and IO_STRESS add a container through pods/ephemeralcontainers. */}} - - apiGroups: ["networking.k8s.io"] - resources: ["networkpolicies"] - verbs: ["create", "delete", "deletecollection"] - apiGroups: ["apps"] resources: ["statefulsets/scale"] verbs: ["get", "update"] diff --git a/charts/kates/values.yaml b/charts/kates/values.yaml index f7897a49..ea7769e2 100644 --- a/charts/kates/values.yaml +++ b/charts/kates/values.yaml @@ -58,8 +58,8 @@ rbac: # -- Create ClusterRole and ClusterRoleBinding create: true # -- Grant the writes the direct Kubernetes chaos backend makes itself - # (KATES_CHAOS_PROVIDER=kubernetes, or hybrid without Litmus): NetworkPolicies, - # StatefulSet scaling, and pod ephemeral containers. Litmus runs its + # (KATES_CHAOS_PROVIDER=kubernetes, or hybrid without Litmus): StatefulSet + # scaling and pod ephemeral containers. Litmus runs its # experiments as litmus-admin and needs none of this. directChaos: false # -- Namespaces where the INTEGRATION_CDC test may read its database's diff --git a/cli/cmd/mcp_caveats.go b/cli/cmd/mcp_caveats.go index 1a1e56bd..a765ca8a 100644 --- a/cli/cmd/mcp_caveats.go +++ b/cli/cmd/mcp_caveats.go @@ -221,7 +221,7 @@ var mcpCaveatsCore = []mcpCaveat{ Refs: []string{ mcpAnchoredRef("kates/src/main/resources/application.properties:274-275", "# Chaos coordination (noop | kubernetes | litmus-crd | hybrid)", "kates.chaos.provider=litmus-crd"), - mcpAnchoredRef(mcpJava+"chaos/LitmusChaosProvider.java:57,102", + mcpAnchoredRef(mcpJava+"chaos/LitmusChaosProvider.java:62,107", "int resultPollIntervalMs = 5_000;", "Instant start = Instant.now();"), }, }, diff --git a/cli/cmd/mcp_chaos_preview.go b/cli/cmd/mcp_chaos_preview.go index b4d3d6c5..17797eee 100644 --- a/cli/cmd/mcp_chaos_preview.go +++ b/cli/cmd/mcp_chaos_preview.go @@ -86,9 +86,9 @@ var mcpAdHocBounds = map[string]mcpAdHocBound{ // field descriptions say so: the kubernetes provider sizes IO_STRESS by // ioWorkers alone (KubernetesChaosProvider.java:525-538), while litmus-crd // sizes it by fillPercentage as well as ioWorkers -// (LitmusChaosProvider.java:314-318). Both honour gracePeriodSec: litmus-crd +// (LitmusChaosProvider.java:319-323). Both honour gracePeriodSec: litmus-crd // hands a POD_DELETE to the kubernetes provider, which deletes the pod with it -// (LitmusChaosProvider.java:42-54,81-83; KubernetesChaosProvider.java:205-213). +// (LitmusChaosProvider.java:41-59,86-88; KubernetesChaosProvider.java:205-213). // fillPercentage stays out of ad-hoc plans with DISK_FILL, the one type it is // meant for. var mcpAdHocTypeParam = map[string]string{ @@ -344,7 +344,7 @@ func mcpCheckAdHocTargets(at string, s *mcpAdHocStep) error { if targets > 1 { return mcpInvalidArgument(at+" sets more than one of targetBrokerId, targetPod and targetTopic; give at most one.", "") } - // LitmusChaosProvider.java:319-323 makes a DNS_ERROR step's topic the + // LitmusChaosProvider.java:324-328 makes a DNS_ERROR step's topic the // hostname whose lookups fail, so the fault would break only a hostname // named after the topic. if s.DisruptionType == "DNS_ERROR" && s.TargetTopic != "" { @@ -761,11 +761,11 @@ func mcpTargetingOf(fs *mcpFaultSpecView) string { // reading of plan §5.2 than its text ("no experimentName in the resolved // plan"). Every playbook sets experimentName on every step // (kates/src/main/resources/playbooks/*.yaml), so the literal rule would flag -// them all, while the Litmus provider maps every type but three to its own -// experiment and routes those three, POD_DELETE, ROLLING_RESTART and -// SCALE_DOWN, to the kubernetes provider: experimentName picks the experiment -// only for a step without a type -// (LitmusChaosProvider.java:29-54,81-83,228-255), and otherwise only names +// them all, while the Litmus provider maps every type but four to its own +// experiment and routes those four, POD_DELETE, NETWORK_PARTITION, +// ROLLING_RESTART and SCALE_DOWN, to the kubernetes provider: experimentName +// picks the experiment only for a step without a type +// (LitmusChaosProvider.java:29-59,86-88,233-260), and otherwise only names // the fault. The plan's wording is a follow-up. func mcpAgentLimitFindings(v *mcpPlanView) []mcpAgentLimitFinding { findings := []mcpAgentLimitFinding{} diff --git a/cli/cmd/mcp_tools_chaos.go b/cli/cmd/mcp_tools_chaos.go index 5ddaea03..7c724a75 100644 --- a/cli/cmd/mcp_tools_chaos.go +++ b/cli/cmd/mcp_tools_chaos.go @@ -127,14 +127,14 @@ var mcpCaveatsChaos = []mcpCaveat{ "create NetworkPolicies or scale node pools, and only for the fault types that need one of those; every " + "other type, and any check that fails with an error, counts as permitted, and a denied check is a step " + "warning, never an error that makes wouldSucceed false. On the default litmus-crd provider Litmus " + - "injects every fault except POD_DELETE, ROLLING_RESTART and SCALE_DOWN as its own litmus-admin service " + - "account, so for those the check does not show whether the fault can run.", + "injects every fault except POD_DELETE, NETWORK_PARTITION, ROLLING_RESTART and SCALE_DOWN as its own " + + "litmus-admin service account, so for those the check does not show whether the fault can run.", Refs: []string{ mcpAnchoredRef(mcpJava+"disruption/DisruptionSafetyGuard.java:275-278,772-841", "boolean canExecute = checkRbacPermissions(spec);", `"Insufficient RBAC permissions for "`, "boolean checkRbacPermissions(FaultSpec spec)", `"RBAC check failed, assuming permitted"`), - mcpAnchoredRef(mcpJava+"chaos/LitmusChaosProvider.java:42-54,81-83,265", - "EnumSet.of(DisruptionType.ROLLING_RESTART, DisruptionType.SCALE_DOWN, DisruptionType.POD_DELETE)", + mcpAnchoredRef(mcpJava+"chaos/LitmusChaosProvider.java:41-59,86-88,270", + "ON_KUBERNETES_API = EnumSet.of(", "DisruptionType.POD_DELETE,", "DisruptionType.NETWORK_PARTITION);", "return kubernetes.triggerFault(spec, onInject);", `engineSpec.chaosServiceAccount = "litmus-admin";`), mcpAnchoredRef("kates/src/main/resources/application.properties:274-275", "# Chaos coordination (noop | kubernetes | litmus-crd | hybrid)", "kates.chaos.provider=litmus-crd"), @@ -228,7 +228,8 @@ var mcpCaveatsChaos = []mcpCaveat{ "count from the moment the chaos provider reported the fault going in: after the step's delayBeforeSec, " + "before the provider changed the cluster. On the default litmus-crd provider that moment comes before it " + "picks the pods and creates the ChaosEngine, so the times include however long Litmus took to start the " + - "experiment. The kubernetes provider, which litmus-crd hands POD_DELETE, ROLLING_RESTART and SCALE_DOWN, " + + "experiment. The kubernetes provider, which litmus-crd hands POD_DELETE, NETWORK_PARTITION, ROLLING_RESTART " + + "and SCALE_DOWN, " + "reports before it looks up what to hit, so a fault that then finds nothing to hit fails with recovery " + "times all the same. Time to first ready is the first Ready event from any watched Kafka pod after that " + "moment, which need not be a pod the fault hit. A step whose fault never went in, a Skipped one or one that " + @@ -248,7 +249,7 @@ var mcpCaveatsChaos = []mcpCaveat{ mcpAnchoredRef(mcpJava+"disruption/KafkaIntelligenceService.java:125-127,198-217,241-246,310-324", "public void markDisruptionStart(Instant at)", "private Duration computeTimeToFullIsr()", "this.baselineLag = timeline.getLast().totalLag();", "private Duration computeTimeToLagRecovery(long baseLag)"), - mcpAnchoredRef(mcpJava+"chaos/LitmusChaosProvider.java:81-93,102-121", + mcpAnchoredRef(mcpJava+"chaos/LitmusChaosProvider.java:86-98,107-126", "return kubernetes.triggerFault(spec, onInject);", "CompletableFuture.delayedExecutor(spec.delayBeforeSec(), TimeUnit.SECONDS, executor.get());", "long startNanos = System.nanoTime();", "buildChaosEngine(spec, engineName, experimentName)", diff --git a/docs/book/02-architecture.md b/docs/book/02-architecture.md index f351d5cb..e2399a58 100644 --- a/docs/book/02-architecture.md +++ b/docs/book/02-architecture.md @@ -42,7 +42,7 @@ flowchart LR PROM -->|"scrapes"| KAFKA ``` -Read it from the left. The CLI calls the Kates API over REST, with the URL and API key of your [CLI context](appendix-a-glossary.md#gl-cli-context). For the commands that install and reach the stack, such as `kates deploy` and `kates ports`, it runs `kubectl` and `helm` against your current Kubernetes context instead. The [Commands](10-cli-reference.md#commands) table in CLI Reference says which command takes which path. In the cluster, the Kates API talks to [`krafter`](appendix-a-glossary.md#gl-krafter), the Kafka cluster under test, as an ordinary Kafka client, and keeps each run in PostgreSQL. On the default [chaos provider](appendix-a-glossary.md#gl-chaos-provider), it creates [LitmusChaos](appendix-a-glossary.md#gl-litmuschaos) resources for every fault except `POD_DELETE`, `ROLLING_RESTART` and `SCALE_DOWN`, which it injects itself through the Kubernetes API. [Prometheus](appendix-a-glossary.md#gl-prometheus) scrapes the [brokers](appendix-a-glossary.md#gl-broker) and the Kates API, and the Kates API reads Prometheus back for a disruption plan's broker metrics. +Read it from the left. The CLI calls the Kates API over REST, with the URL and API key of your [CLI context](appendix-a-glossary.md#gl-cli-context). For the commands that install and reach the stack, such as `kates deploy` and `kates ports`, it runs `kubectl` and `helm` against your current Kubernetes context instead. The [Commands](10-cli-reference.md#commands) table in CLI Reference says which command takes which path. In the cluster, the Kates API talks to [`krafter`](appendix-a-glossary.md#gl-krafter), the Kafka cluster under test, as an ordinary Kafka client, and keeps each run in PostgreSQL. On the default [chaos provider](appendix-a-glossary.md#gl-chaos-provider), it creates [LitmusChaos](appendix-a-glossary.md#gl-litmuschaos) resources for every fault except `POD_DELETE`, `NETWORK_PARTITION`, `ROLLING_RESTART` and `SCALE_DOWN`, which it injects itself through the Kubernetes API. [Prometheus](appendix-a-glossary.md#gl-prometheus) scrapes the [brokers](appendix-a-glossary.md#gl-broker) and the Kates API, and the Kates API reads Prometheus back for a disruption plan's broker metrics. Each part in the table runs in one place. The namespaces are those of the default [isolated topology](appendix-a-glossary.md#gl-isolated-topology); with `--topology single`, every part below that runs in the cluster, except Trogdor, shares `kates-stack`, as [Single-Namespace vs Multi-Namespace](12-deployment.md#single-namespace-vs-multi-namespace) describes. @@ -175,7 +175,7 @@ The diagram shows the parts behind a disruption plan. Look at the Providers grou ```mermaid %%| label: fig-architecture-disruption-engine %%| fig-cap: "A disruption plan's orchestrator checks the plan, reads Kafka's state, and hands every fault to the one chaos provider that kates.chaos.provider selects." -%%| fig-alt: "Flowchart in four groups. Control: the playbook catalog feeds the disruption orchestrator, which calls the safety guard. Intelligence: the Kafka intelligence service provides ISR tracking, consumer lag and leader resolution. Providers: the setting kates.chaos.provider selects the Litmus provider by default, the Kubernetes provider, or the hybrid provider, which picks one of the two; the Litmus provider sends POD_DELETE, ROLLING_RESTART and SCALE_DOWN to the Kubernetes provider. Reporting: the orchestrator writes a disruption report, graded by the SLA grader and fed by the Prometheus metrics capture." +%%| fig-alt: "Flowchart in four groups. Control: the playbook catalog feeds the disruption orchestrator, which calls the safety guard. Intelligence: the Kafka intelligence service provides ISR tracking, consumer lag and leader resolution. Providers: the setting kates.chaos.provider selects the Litmus provider by default, the Kubernetes provider, or the hybrid provider, which picks one of the two; the Litmus provider sends POD_DELETE, NETWORK_PARTITION, ROLLING_RESTART and SCALE_DOWN to the Kubernetes provider. Reporting: the orchestrator writes a disruption report, graded by the SLA grader and fed by the Prometheus metrics capture." graph LR subgraph Control DO[DisruptionOrchestrator] @@ -216,7 +216,7 @@ graph LR CP -->|hybrid| HCP HCP -.->|picks one| LCP HCP -.->|picks one| KCP - LCP -->|"POD_DELETE,
ROLLING_RESTART,
SCALE_DOWN"| KCP + LCP -->|"POD_DELETE,
NETWORK_PARTITION,
ROLLING_RESTART,
SCALE_DOWN"| KCP DR --> SG DR --> PMC ``` diff --git a/docs/book/07-chaos-practice.md b/docs/book/07-chaos-practice.md index 82bd2d32..590184da 100644 --- a/docs/book/07-chaos-practice.md +++ b/docs/book/07-chaos-practice.md @@ -101,14 +101,14 @@ The table compares the four values. Check the Default column first: a stock inst | Provider | Default | Needs | Fault types it runs | `kates` chart RBAC | |----------|---------|-------|---------------------|--------------------| -| `litmus-crd` | Yes | LitmusChaos, with each type's experiment installed | Every type through a Litmus experiment, except `POD_DELETE`, `ROLLING_RESTART` and `SCALE_DOWN`, which use the Kubernetes API | The default role | -| `kubernetes` | No | Nothing beyond Kates | `POD_KILL`, `POD_DELETE`, `LEADER_ELECTION`, `NETWORK_PARTITION`, `CPU_STRESS`, `IO_STRESS`, `ROLLING_RESTART`, `SCALE_DOWN` | `rbac.directChaos=true` for `NETWORK_PARTITION`, `CPU_STRESS` and `IO_STRESS` | +| `litmus-crd` | Yes | LitmusChaos, with each type's experiment installed | Every type through a Litmus experiment, except `POD_DELETE`, `NETWORK_PARTITION`, `ROLLING_RESTART` and `SCALE_DOWN`, which use the Kubernetes API | The default role | +| `kubernetes` | No | Nothing beyond Kates | `POD_KILL`, `POD_DELETE`, `LEADER_ELECTION`, `NETWORK_PARTITION`, `CPU_STRESS`, `IO_STRESS`, `ROLLING_RESTART`, `SCALE_DOWN` | `rbac.directChaos=true` for `CPU_STRESS` and `IO_STRESS` | | `hybrid` | No | Nothing: it uses LitmusChaos when the Litmus [CRDs](appendix-a-glossary.md#gl-crd) exist | Those of the provider it picks | As for the provider it picks | | `noop` | No | Nothing | None: every fault comes back `Skipped` | None | So `NETWORK_LATENCY`, `MEMORY_STRESS`, `DNS_ERROR`, `DISK_FILL` and `NODE_DRAIN` need LitmusChaos, and [The Hybrid Provider](#the-hybrid-provider) shows how `hybrid` picks. -On a stock install you already have LitmusChaos: `kates deploy`, and so `make all`, installs the `kates-chaos` chart unless you pass `--with-chaos=false`. The chart installs seven experiments in its target namespaces, `kafka` by default: `pod-delete`, `pod-cpu-hog`, `pod-memory-hog`, `pod-network-partition`, `pod-io-stress`, `pod-dns-error` and `node-drain`. It installs no `disk-fill` and no `pod-network-latency`, so `DISK_FILL` and `NETWORK_LATENCY` steps fail until you add those experiments yourself. +On a stock install you already have LitmusChaos: `kates deploy`, and so `make all`, installs the `kates-chaos` chart unless you pass `--with-chaos=false`. The chart installs seven experiments in its target namespaces, `kafka` by default: `pod-delete`, `pod-cpu-hog`, `pod-memory-hog`, `pod-network-partition`, `pod-io-stress`, `pod-dns-error` and `node-drain`; Kates doesn't use `pod-network-partition`. It installs no `disk-fill` and no `pod-network-latency`, so `DISK_FILL` and `NETWORK_LATENCY` steps fail until you add those experiments yourself. When Kates picks the provider, it checks that the provider can work, and falls back to `noop` when it can't or when the setting names no provider. For `litmus-crd` the check is only that Kates can list ChaosEngines: it proves the CRDs exist, not that the Litmus [operator](appendix-a-glossary.md#gl-operator) runs. A fallback is easy to miss. The Kates API logs it once, at ERROR, with the words `falling back to noop`, and every fault after that comes back `Skipped`, so a plan ends `PARTIAL` and a resilience run `CHAOS_FAILED`. A provider that works logs `Chaos provider:` and its name instead. @@ -124,7 +124,7 @@ rbac: directChaos: true ``` -Leave `rbac.directChaos` off with `litmus-crd`. Litmus runs its experiments as its own `litmus-admin` service account, and the extra rules are cluster-wide: they let Kates create [NetworkPolicies](appendix-a-glossary.md#gl-networkpolicy) in any namespace and start a container in any pod. +Leave `rbac.directChaos` off with `litmus-crd`. Litmus runs its experiments as its own `litmus-admin` service account, and the extra rules are cluster-wide: they let Kates scale any StatefulSet and start a container in any pod. Kates doesn't check a type against the provider before the fault. `kates disruption types` lists every type whatever the provider, and neither the safety guard nor the dry run looks, so an unsupported type fails only when its step reaches the fault. On `kubernetes`, the step's `Failure` line in `kates disruption status` reads `DisruptionType not supported by kubernetes provider`. On `litmus-crd`, Kates creates the ChaosEngine even when its experiment isn't installed, and waits up to `chaosDurationSec` plus two minutes for a result. @@ -132,7 +132,7 @@ Each Litmus fault leaves a ChaosEngine named `kates-- ### Direct Kubernetes API -The `kubernetes` provider implements these disruptions against the Kubernetes API directly. It needs no extra tooling, but `NETWORK_PARTITION`, `CPU_STRESS` and `IO_STRESS` need `rbac.directChaos=true` on the `kates` chart: +The `kubernetes` provider implements these disruptions against the Kubernetes API directly. It needs no extra tooling, but `CPU_STRESS` and `IO_STRESS` need `rbac.directChaos=true` on the `kates` chart: | Type | Implementation | Effect | |------|---------------|--------| @@ -149,7 +149,7 @@ The provider injects each fault once, and doesn't try it again when a call fails ### LitmusChaos Integration -On the default `litmus-crd` provider, Kates maps every disruption type but three to a Litmus experiment (for example, `POD_KILL` and `LEADER_ELECTION` both map to `pod-delete`). The exceptions are `ROLLING_RESTART`, because no Litmus experiment does a rolling restart, `SCALE_DOWN`, because `pod-delete` kills a broker that its [StrimziPodSet](appendix-a-glossary.md#gl-strimzipodset) brings straight back, and `POD_DELETE`, because `pod-delete` never deletes a pod with `gracePeriodSec`. It deletes the pod at once with the `FORCE=true` that pods of a StrimziPodSet need, and with the pod's own grace period otherwise. The `litmus-crd` provider hands all three to the `kubernetes` provider, so they run the same way on both. Five types are only available through Litmus: +On the default `litmus-crd` provider, Kates maps every disruption type but four to a Litmus experiment (for example, `POD_KILL` and `LEADER_ELECTION` both map to `pod-delete`). The exceptions are `NETWORK_PARTITION`, because `pod-network-partition` ignores the pods Kates picks and fails without an `appinfo` label, which Kates can't give it for one pod, `ROLLING_RESTART`, because no Litmus experiment does a rolling restart, `SCALE_DOWN`, because `pod-delete` kills a broker that its [StrimziPodSet](appendix-a-glossary.md#gl-strimzipodset) brings straight back, and `POD_DELETE`, because `pod-delete` never deletes a pod with `gracePeriodSec`. It deletes the pod at once with the `FORCE=true` that pods of a StrimziPodSet need, and with the pod's own grace period otherwise. The `litmus-crd` provider hands all four to the `kubernetes` provider, so they run the same way on both. Five types are only available through Litmus: | Type | Litmus Experiment | Effect | |------|-------------------|--------| @@ -290,7 +290,7 @@ steps: ### split-brain -Aims a `NETWORK_PARTITION` fault at node 0 to test cluster consensus under split-brain conditions [@bailis2014network; @alquraan2018analysis]. For 60 seconds, the `kubernetes` chaos provider adds a NetworkPolicy with no allow rules to the pod; the `litmus-crd` provider runs the Litmus `pod-network-partition` experiment instead. The playbook does not look up the active controller: `targetBrokerId: 0` picks the broker whose pod name ends in `-0`, which is the active controller only if node 0 also has the controller role and leads the metadata [quorum](appendix-a-glossary.md#gl-quorum) at the time. `targetBrokerId` never picks a dedicated controller; to aim the fault at one, name its pod in `targetPod`. +Aims a `NETWORK_PARTITION` fault at node 0 to test cluster consensus under split-brain conditions [@bailis2014network; @alquraan2018analysis]. For 60 seconds, Kates adds a NetworkPolicy with no allow rules to the pod, on either chaos provider. The playbook does not look up the active controller: `targetBrokerId: 0` picks the broker whose pod name ends in `-0`, which is the active controller only if node 0 also has the controller role and leads the metadata [quorum](appendix-a-glossary.md#gl-quorum) at the time. `targetBrokerId` never picks a dedicated controller; to aim the fault at one, name its pod in `targetPod`. ```mermaid graph LR @@ -525,7 +525,7 @@ The dry run answers UNSAFE when the guard would refuse to run the playbook, and kates disruption playbook run leader-cascade --dry-run && kates disruption playbook run leader-cascade ``` -For `POD_KILL`, `POD_DELETE`, `LEADER_ELECTION`, `NETWORK_PARTITION`, `NETWORK_LATENCY`, `ROLLING_RESTART` and `SCALE_DOWN` steps, the dry run also asks the Kubernetes API whether the Kates service account may make the step's change. It asks for `delete` on pods for the three pod kills, `create` on NetworkPolicies for the two network types, `patch` on pods for `ROLLING_RESTART`, and the node pool or StatefulSet writes a `SCALE_DOWN` makes. A missing permission is a step warning, which doesn't make the dry run UNSAFE. Other disruption types get no RBAC check, and a check that cannot run counts as permitted. On `litmus-crd`, Litmus injects every type except `POD_DELETE`, `ROLLING_RESTART` and `SCALE_DOWN` as its own service account, so a missing permission for those types says nothing about whether the step can run. +For `POD_KILL`, `POD_DELETE`, `LEADER_ELECTION`, `NETWORK_PARTITION`, `NETWORK_LATENCY`, `ROLLING_RESTART` and `SCALE_DOWN` steps, the dry run also asks the Kubernetes API whether the Kates service account may make the step's change. It asks for `delete` on pods for the three pod kills, `create` on NetworkPolicies for the two network types, `patch` on pods for `ROLLING_RESTART`, and the node pool or StatefulSet writes a `SCALE_DOWN` makes. A missing permission is a step warning, which doesn't make the dry run UNSAFE. Other disruption types get no RBAC check, and a check that cannot run counts as permitted. On `litmus-crd`, Litmus injects every type except `POD_DELETE`, `NETWORK_PARTITION`, `ROLLING_RESTART` and `SCALE_DOWN` as its own service account, so a missing permission for those types says nothing about whether the step can run. The preview describes the cluster at that moment. A leader can move before its step starts, and the step kills whichever broker leads the partition when it starts. @@ -627,7 +627,7 @@ What rollback does depends on the fault. The table lists what it undoes, and wha | Fault | What rollback does | What ends the fault otherwise | |-------|--------------------|-------------------------------| -| `NETWORK_PARTITION` | Deletes every NetworkPolicy labeled `managed-by=kates` in the step's namespace, the label the `kubernetes` provider puts on its policies | On `kubernetes`, Kates deletes its policies when `chaosDurationSec` ends, or as soon as the partition fails; Litmus gets `chaosDurationSec` as its duration | +| `NETWORK_PARTITION` | Deletes every NetworkPolicy labeled `managed-by=kates` in the step's namespace, the label Kates puts on its policies | Kates deletes its policies when `chaosDurationSec` ends, or as soon as the partition fails | | `SCALE_DOWN` | Gives every KafkaNodePool and StatefulSet in the step's namespace that carries `kates.io/original-replicas` its replicas back | Nothing: the broker stays removed until rollback, or until orphan recovery when the Kates API restarts | | Every other type | Nothing, although the step report still says it rolled back | The StrimziPodSet recreates a deleted pod; a stress container on `kubernetes` stops after `chaosDurationSec`; Litmus gets `chaosDurationSec` as its duration | diff --git a/docs/book/appendix-d-versions.md b/docs/book/appendix-d-versions.md index 033560a0..9af63df6 100644 --- a/docs/book/appendix-d-versions.md +++ b/docs/book/appendix-d-versions.md @@ -22,7 +22,7 @@ Every version the platform pins, in one place. This table is generated from the | `kafka-ui` chart | 0.3.0 (app v1.5.0) | `charts/kafka-ui/Chart.yaml` | | `kates-chaos` chart | 2.2.1 (app 3.28.0) | `charts/kates-chaos/Chart.yaml` | | `kates-platform` chart | 0.7.0 (app 1.0.0) | `charts/kates-platform/Chart.yaml` | -| `kates` chart | 0.11.4 (app 1.26.0) | `charts/kates/Chart.yaml` | +| `kates` chart | 0.11.5 (app 1.26.0) | `charts/kates/Chart.yaml` | | `legacy-kafka` chart | 0.2.0 (app 3.9.1) | `charts/legacy-kafka/Chart.yaml` | | `mirror-maker2` chart | 0.11.5 (app 4.3.1) | `charts/mirror-maker2/Chart.yaml` | | `monitoring` chart | 1.6.0 (app 82.4.3) | `charts/monitoring/Chart.yaml` | diff --git a/kates/docs/disruption-guide.md b/kates/docs/disruption-guide.md index 8ba38c32..88093ff6 100644 --- a/kates/docs/disruption-guide.md +++ b/kates/docs/disruption-guide.md @@ -68,7 +68,7 @@ The difference between `POD_KILL` and `POD_DELETE` is the difference between "yo A network partition is the most insidious failure mode in distributed systems. The broker is still running — it still has all its data, its processes are healthy, its disk is fine — but it cannot communicate with the rest of the cluster. From the partitioned broker's perspective, the rest of the cluster has disappeared. From the cluster's perspective, the partitioned broker has disappeared. Both sides continue operating independently, which is exactly the definition of a split brain. -Kates implements network partitions by creating firewall rules (via Litmus ChaosEngine CRDs) that drop all traffic to and from the target pod. This means the partitioned broker can still receive Kubernetes health probes (if configured separately), but cannot communicate with other brokers or clients. +Kates implements a network partition as a NetworkPolicy with no allow rules on the target pod, on either chaos provider, so it drops all traffic to and from that pod. This means the partitioned broker can still receive Kubernetes health probes (if configured separately), but cannot communicate with other brokers or clients. The consequences of a network partition depend on which broker is partitioned: diff --git a/kates/docs/playbook-catalog.md b/kates/docs/playbook-catalog.md index 05e72052..2f3b36e3 100644 --- a/kates/docs/playbook-catalog.md +++ b/kates/docs/playbook-catalog.md @@ -179,7 +179,7 @@ steps: The playbook does not look up the active controller. `targetBrokerId: 0` aims the fault at the broker whose pod name ends in `-0` among the pods labelled `strimzi.io/component-type=kafka`. `targetBrokerId` picks among brokers only, so it never picks a dedicated KRaft controller (`strimzi.io/broker-role=false`): if node 0 is one, the step falls back to the first broker, and the dry run warns about it. Node 0 is the active controller only if it also has the controller role and leads the metadata quorum when the step runs; `kafka-metadata-quorum.sh --bootstrap-server :9092 describe --status` prints the quorum's `LeaderId`, so check it before you run the playbook. -With the direct Kubernetes backend, the partition is a NetworkPolicy that denies all ingress and egress for that pod, held for `chaosDurationSec: 60` and then deleted; the Litmus backend runs `pod-network-partition` instead. To isolate a different broker, post the same step to `POST /api/disruptions` with `targetBrokerId` changed. To isolate a dedicated controller, name its pod in `targetPod` instead, such as `krafter-controllers-alpha-3` on the default cluster. Built-in playbooks take no parameters. +On either chaos backend, the partition is a NetworkPolicy that denies all ingress and egress for that pod, held for `chaosDurationSec: 60` and then deleted. To isolate a different broker, post the same step to `POST /api/disruptions` with `targetBrokerId` changed. To isolate a dedicated controller, name its pod in `targetPod` instead, such as `krafter-controllers-alpha-3` on the default cluster. Built-in playbooks take no parameters. ### What to Look For diff --git a/kates/docs/troubleshooting.md b/kates/docs/troubleshooting.md index 87bd4fd1..2fcff521 100644 --- a/kates/docs/troubleshooting.md +++ b/kates/docs/troubleshooting.md @@ -138,7 +138,7 @@ Or use leader-aware targeting, which always resolves to exactly one pod: **Root cause:** This typically indicates a problem with the chaos provider backend: -1. **Litmus is not installed.** If the `HybridChaosProvider` selected the Litmus path for a complex fault (network partition, CPU stress), but Litmus CRDs are not installed in the cluster, the CRD creation fails and the provider hangs waiting for a response. +1. **Litmus is not installed.** If the `HybridChaosProvider` selected the Litmus path for a fault only Litmus runs (memory stress, DNS error), but Litmus CRDs are not installed in the cluster, the CRD creation fails and the provider hangs waiting for a response. 2. **RBAC permissions are missing.** Even though the safety guard checks permissions, there are edge cases where the permission check passes but the actual operation fails (e.g., cluster-scoped vs. namespace-scoped permissions). diff --git a/kates/src/main/java/com/bmscomp/kates/chaos/LitmusChaosProvider.java b/kates/src/main/java/com/bmscomp/kates/chaos/LitmusChaosProvider.java index 16533f82..4e24f97a 100644 --- a/kates/src/main/java/com/bmscomp/kates/chaos/LitmusChaosProvider.java +++ b/kates/src/main/java/com/bmscomp/kates/chaos/LitmusChaosProvider.java @@ -33,7 +33,6 @@ public class LitmusChaosProvider implements ChaosProvider { Map.entry(DisruptionType.IO_STRESS, "pod-io-stress"), Map.entry(DisruptionType.DNS_ERROR, "pod-dns-error"), Map.entry(DisruptionType.DISK_FILL, "disk-fill"), - Map.entry(DisruptionType.NETWORK_PARTITION, "pod-network-partition"), Map.entry(DisruptionType.NETWORK_LATENCY, "pod-network-latency"), Map.entry(DisruptionType.NODE_DRAIN, "node-drain"), // Leader election is triggered by deleting the current partition leader pod @@ -48,10 +47,16 @@ public class LitmusChaosProvider implements ChaosProvider { * does it take a grace period: it deletes with none under FORCE=true, * which pods of a StrimziPodSet need (see buildChaosEngine), and with the * pod's own otherwise. A POD_DELETE is deleted with gracePeriodSec, so the - * broker gets the controlled shutdown the fault is meant to test. + * broker gets the controlled shutdown the fault is meant to test. Nor does + * pod-network-partition (litmus-go 3.28) cut off the pods Kates picks: it + * ignores TARGET_PODS, selects by the label in TARGETS, which comes from + * the appinfo Kates leaves out, and without it fails before any change. */ - private static final Set ON_KUBERNETES_API = - EnumSet.of(DisruptionType.ROLLING_RESTART, DisruptionType.SCALE_DOWN, DisruptionType.POD_DELETE); + private static final Set ON_KUBERNETES_API = EnumSet.of( + DisruptionType.ROLLING_RESTART, + DisruptionType.SCALE_DOWN, + DisruptionType.POD_DELETE, + DisruptionType.NETWORK_PARTITION); /** How often the ChaosResult is read until it has a verdict. */ int resultPollIntervalMs = 5_000; diff --git a/kates/src/test/java/com/bmscomp/kates/chaos/LitmusChaosProviderTest.java b/kates/src/test/java/com/bmscomp/kates/chaos/LitmusChaosProviderTest.java index d29009a2..222b2129 100644 --- a/kates/src/test/java/com/bmscomp/kates/chaos/LitmusChaosProviderTest.java +++ b/kates/src/test/java/com/bmscomp/kates/chaos/LitmusChaosProviderTest.java @@ -676,6 +676,35 @@ void podDeleteDeletesThePodWithItsGracePeriod() throws Exception { .isEmpty()); } + @Test + void networkPartitionCutsOffThePodWithAKatesNetworkPolicy() throws Exception { + // Used to run pod-network-partition, which in litmus-go 3.28 needs + // TARGETS from an appinfo Kates leaves out: it failed with "provide the + // appLabel" before cutting anything off, and it ignores TARGET_PODS. + FaultSpec spec = FaultSpec.builder("split") + .targetBrokerId(2) + .disruptionType(DisruptionType.NETWORK_PARTITION) + .chaosDurationSec(1) + .build(); + + CompletableFuture fault = provider.triggerFault(spec); + var policy = client.network().networkPolicies().inNamespace("kafka").withName("kates-netpol-krafter-brokers-2"); + long deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(5); + while (policy.get() == null && !fault.isDone() && System.nanoTime() < deadline) { + Thread.sleep(20); + } + assertNotNull(policy.get(), "broker 2 was never cut off"); + ChaosOutcome outcome = fault.get(5, TimeUnit.SECONDS); + + assertTrue(outcome.isPass(), outcome.failureReason()); + assertNull(policy.get(), "the partition outlasted chaosDurationSec"); + assertTrue(client.resources(ChaosEngine.class) + .inNamespace("kafka") + .list() + .getItems() + .isEmpty()); + } + @ParameterizedTest @EnumSource( value = DisruptionType.class, From a77f33950acf05200c95dca0deee20f476c11e20 Mon Sep 17 00:00:00 2001 From: Said Boudjelda Date: Tue, 20 Oct 2026 07:04:16 +0200 Subject: [PATCH 4/4] fix(chaos): a fault may carry no cmdProbe, and Litmus gets no probe On litmus-crd every cmdProbe in a fault's own probes (a plan step, a resilience run's chaosSpec, a compound fault) went into the ChaosEngine as a Continuous probe with no source. litmus-go 3.28 runs such a probe inline, with /bin/sh -c in the experiment pod (pkg/probe/cmdprobe.go), every 5 seconds, as litmus-admin, which the kates-chaos chart by default lets delete pods and evict in every namespace and patch nodes. So a key with chaos:run could delete pods in kube-system through a probe, past the namespace check and the envOverrides allowlist. FaultLimits now refuses a cmdProbe in a fault's probes, the type a probe without one gets, on every path that applies the limits: the safety guard, the coordinator and the compound orchestrator. The Litmus provider no longer builds probes at all; it skipped every other type already, so nothing reaches Litmus. Kates evaluates probes itself only in a resilience run, with sh -c in a broker pod: a cmdProbe goes in its top-level probes, which this leaves alone. Nothing shipped puts a probe in a fault; the resilience examples and scenarios use kafkaProbe and k8sProbe at the top level. FaultLimitsTest refuses a probe running kubectl, and LitmusChaosProvider Test checks that no probe reaches the ChaosEngine; both fail on the old code. The book's limits and refusal table, the disruption guide, the overview and the examples README say where a cmdProbe may go. --- cli/examples/README.md | 2 ++ docs/book/07-chaos-practice.md | 3 +- kates/docs/disruption-guide.md | 2 +- kates/docs/overview.md | 2 +- .../com/bmscomp/kates/chaos/FaultLimits.java | 11 ++++++ .../kates/chaos/LitmusChaosProvider.java | 35 ++----------------- .../bmscomp/kates/chaos/FaultLimitsTest.java | 15 ++++++++ .../kates/chaos/LitmusChaosProviderTest.java | 24 +++---------- 8 files changed, 40 insertions(+), 54 deletions(-) diff --git a/cli/examples/README.md b/cli/examples/README.md index b55251ac..e6a3d9e5 100644 --- a/cli/examples/README.md +++ b/cli/examples/README.md @@ -162,6 +162,8 @@ that sets a field its run doesn't read, or that the API would refuse. Every probe runs before the fault and after it, until all pass or `maxRecoveryWaitSec` is up; a `Continuous` one runs while the fault lasts too. +A `cmdProbe` belongs here, not in `chaosSpec.probes`: Kates refuses a fault that +carries one, since on `litmus-crd` Litmus ran it as `litmus-admin`. | YAML key | Type | Notes | |---|---|---| diff --git a/docs/book/07-chaos-practice.md b/docs/book/07-chaos-practice.md index 590184da..a149f0bf 100644 --- a/docs/book/07-chaos-practice.md +++ b/docs/book/07-chaos-practice.md @@ -566,6 +566,7 @@ A refused plan starts nothing. Over the API it returns `422` with status `REJECT | `Step '…': NODE_DRAIN: …, the node the Kates API runs on, and draining it would evict the Kates API …` | Aim the drain at a pod on another node, or name another node in `envOverrides.TARGET_NODE` | | `Step '…':`, then a fault parameter that `is below` its floor or `is above the limit of` its ceiling | Bring the parameter into its range, or raise the limit (see [Fault Parameter Limits](#fault-parameter-limits)) | | `Step '…': envOverrides.… isn't one of the two envOverrides may set, TARGET_NODE and NODE_LABEL, …` | Drop the override and use the fault's own fields, such as `targetNamespace`, `targetLabel` or `chaosDurationSec`; set a cluster setting in the `kates-chaos` chart (see [Targeting Pods](#targeting-pods)) | +| `Step '…': probes[N] is a cmdProbe, which a fault may not carry; …` | Drop the probe, or move it to a resilience run's top-level `probes` | | `Step '…': targetTopic isn't a Kafka topic name: …` | Name the topic, or a `DNS_ERROR`'s host, with letters, digits, `.`, `_` or `-` only | | `Step '…': targetNamespace … is neither …, the Kafka cluster's namespace (kates.chaos.kafka.namespace), nor the Kates API's own, …` | Aim the step at the Kafka cluster's namespace or the Kates API's own. No setting adds another | | `Plan would affect N brokers but maxAffectedBrokers=M` | Narrow the selectors, or raise the limit if you mean it | @@ -610,7 +611,7 @@ Every number in a fault spec has a range, and Kates refuses a fault with one out Kates checks every parameter whatever the fault's type, so a value no provider reads has to be in range too. A field you leave out gets its default, which is always in range. -Three rules go beyond the table. First, a fault that ends only when its duration does needs a `chaosDurationSec` of at least 1: `NETWORK_PARTITION`, `NETWORK_LATENCY`, `CPU_STRESS`, `MEMORY_STRESS`, `IO_STRESS`, `DNS_ERROR`, `DISK_FILL` and `NODE_DRAIN`. The others may set `0`, because a deleted pod comes back by itself, and a `ROLLING_RESTART` or `SCALE_DOWN` with `0` doesn't wait. Second, `envOverrides` may set only `TARGET_NODE` and `NODE_LABEL` (see [Targeting Pods](#targeting-pods)), so not `TOTAL_CHAOS_DURATION` or `RAMP_TIME`, which would run a Litmus experiment past the limits. Third, a `targetTopic` has to be a Kafka topic name, 1 to 249 letters, digits, `.`, `_` or `-`: on `litmus-crd`, a `DNS_ERROR` passes it to a shell command as `TARGET_HOSTNAMES`. +Four rules go beyond the table. First, a fault that ends only when its duration does needs a `chaosDurationSec` of at least 1: `NETWORK_PARTITION`, `NETWORK_LATENCY`, `CPU_STRESS`, `MEMORY_STRESS`, `IO_STRESS`, `DNS_ERROR`, `DISK_FILL` and `NODE_DRAIN`. The others may set `0`, because a deleted pod comes back by itself, and a `ROLLING_RESTART` or `SCALE_DOWN` with `0` doesn't wait. Second, `envOverrides` may set only `TARGET_NODE` and `NODE_LABEL` (see [Targeting Pods](#targeting-pods)), so not `TOTAL_CHAOS_DURATION` or `RAMP_TIME`, which would run a Litmus experiment past the limits. Third, a `targetTopic` has to be a Kafka topic name, 1 to 249 letters, digits, `.`, `_` or `-`: on `litmus-crd`, a `DNS_ERROR` passes it to a shell command as `TARGET_HOSTNAMES`. Fourth, a fault's own `probes` may hold no `cmdProbe`, the type a probe without one gets: on `litmus-crd`, Litmus ran it with `sh -c` in its experiment pod as `litmus-admin`, which may delete pods in every namespace. A resilience run's top-level `probes` may, since Kates runs those in a broker pod, and Kates sends Litmus no probe at all. The duration ceiling matters most on `kubernetes`, where a CPU or IO stress runs in an ephemeral container that Kubernetes can't remove: its duration is all that stops it. A `ROLLING_RESTART`'s `chaosDurationSec` is its wait for the Cluster Operator to roll every pod, so on a cluster whose roll takes over an hour, raise `kates.chaos.limits.max-duration-sec`. diff --git a/kates/docs/disruption-guide.md b/kates/docs/disruption-guide.md index 88093ff6..1d6ffb90 100644 --- a/kates/docs/disruption-guide.md +++ b/kates/docs/disruption-guide.md @@ -361,7 +361,7 @@ This prevents a common mistake: using a broad label selector like `strimzi.io/co A typo in a number does as much damage as one in a selector: a stress meant for 60 seconds that runs for 60,000, or `cpuCores: 0`, which Litmus reads as every core. So every fault's parameters are checked against the `kates.chaos.limits.*` ceilings before anything is injected, whichever way the fault comes in: a plan, a playbook, a template, a schedule, a compound run or a resilience run. A plan with a fault outside them is rejected like any other unsafe plan; a compound or resilience run gets a `400` naming each parameter. -With the defaults, `chaosDurationSec` runs from 0 to 3,600 seconds, `delayBeforeSec` from 0 to 600, `gracePeriodSec` from 0 to 300, `networkLatencyMs` from 1 to 30,000, `fillPercentage` from 1 to 100, `cpuCores` and `ioWorkers` from 1 to 64, and `memoryMb` from 1 to 32,768. The [configuration reference](overview.md#chaos-limits) names the setting for each ceiling. A fault undone when its duration ends needs a `chaosDurationSec` of at least 1: `NETWORK_PARTITION`, `NETWORK_LATENCY`, `CPU_STRESS`, `MEMORY_STRESS`, `IO_STRESS`, `DNS_ERROR`, `DISK_FILL` and `NODE_DRAIN`. `envOverrides` may set only `TARGET_NODE` and `NODE_LABEL`, which pick the node a `NODE_DRAIN` drains: Litmus applies an override after the env Kates sets, and its helper pods run privileged as root and paste some values into a shell. A `targetTopic` has to be a Kafka topic name, 1 to 249 letters, digits, `.`, `_` or `-`. +With the defaults, `chaosDurationSec` runs from 0 to 3,600 seconds, `delayBeforeSec` from 0 to 600, `gracePeriodSec` from 0 to 300, `networkLatencyMs` from 1 to 30,000, `fillPercentage` from 1 to 100, `cpuCores` and `ioWorkers` from 1 to 64, and `memoryMb` from 1 to 32,768. The [configuration reference](overview.md#chaos-limits) names the setting for each ceiling. A fault undone when its duration ends needs a `chaosDurationSec` of at least 1: `NETWORK_PARTITION`, `NETWORK_LATENCY`, `CPU_STRESS`, `MEMORY_STRESS`, `IO_STRESS`, `DNS_ERROR`, `DISK_FILL` and `NODE_DRAIN`. `envOverrides` may set only `TARGET_NODE` and `NODE_LABEL`, which pick the node a `NODE_DRAIN` drains: Litmus applies an override after the env Kates sets, and its helper pods run privileged as root and paste some values into a shell. A `targetTopic` has to be a Kafka topic name, 1 to 249 letters, digits, `.`, `_` or `-`. A fault's own `probes` may hold no `cmdProbe`, which Litmus ran as `litmus-admin`; a resilience run's top-level `probes` may. ### RBAC Permission Verification diff --git a/kates/docs/overview.md b/kates/docs/overview.md index 65427375..84bc5fe3 100644 --- a/kates/docs/overview.md +++ b/kates/docs/overview.md @@ -117,7 +117,7 @@ Kates uses a pluggable backend architecture for both performance testing and cha | Provider | Description | When to Use | |----------|-------------|-------------| -| `litmus-crd` | Creates Litmus ChaosEngine custom resources in Kubernetes. Litmus handles the actual fault injection lifecycle, including experiment pods, probes, and cleanup. | Full chaos engineering with Litmus installed | +| `litmus-crd` | Creates Litmus ChaosEngine custom resources in Kubernetes. Litmus handles the actual fault injection lifecycle, including experiment pods and cleanup; Kates sends it no probes. | Full chaos engineering with Litmus installed | | `kubernetes` | Uses the Kubernetes API directly (e.g., `kubectl delete pod`, scale deployments, apply network policies). No external chaos framework required. | Lightweight chaos, no CRD dependencies | | `noop` | Simulates fault injection without actually doing anything. Returns synthetic outcomes for testing the orchestration logic. | Unit testing, dry-runs | | `hybrid` | Routes faults to the best available provider automatically. Prefers Litmus for supported experiment types, falls back to direct Kubernetes API for others. | Production with partial Litmus coverage | diff --git a/kates/src/main/java/com/bmscomp/kates/chaos/FaultLimits.java b/kates/src/main/java/com/bmscomp/kates/chaos/FaultLimits.java index b194aabd..c5634db0 100644 --- a/kates/src/main/java/com/bmscomp/kates/chaos/FaultLimits.java +++ b/kates/src/main/java/com/bmscomp/kates/chaos/FaultLimits.java @@ -137,6 +137,17 @@ public Map violations(FaultSpec spec) { && !TOPIC_NAME.matcher(spec.targetTopic()).matches()) { found.put("targetTopic", "isn't a Kafka topic name: 1 to 249 letters, digits, '.', '_' or '-'"); } + // On litmus-crd a fault's cmdProbe ran with sh -c in Litmus's experiment + // pod, as litmus-admin, which may delete pods in every namespace. + for (int i = 0; spec.probes() != null && i < spec.probes().size(); i++) { + ProbeSpec probe = spec.probes().get(i); + if (probe != null && "cmdProbe".equals(probe.type())) { + found.put( + "probes[" + i + "]", + "is a cmdProbe, which a fault may not carry; give it to a resilience run's top-level probes," + + " which Kates runs in a broker pod"); + } + } return found; } diff --git a/kates/src/main/java/com/bmscomp/kates/chaos/LitmusChaosProvider.java b/kates/src/main/java/com/bmscomp/kates/chaos/LitmusChaosProvider.java index 4e24f97a..d82e659f 100644 --- a/kates/src/main/java/com/bmscomp/kates/chaos/LitmusChaosProvider.java +++ b/kates/src/main/java/com/bmscomp/kates/chaos/LitmusChaosProvider.java @@ -363,38 +363,9 @@ ChaosEngine buildChaosEngine(FaultSpec spec, String engineName, String experimen ChaosEngineSpec.ExperimentSpec expSpec = new ChaosEngineSpec.ExperimentSpec(); expSpec.components = components; - if (spec.probes() != null && !spec.probes().isEmpty()) { - List litmusProbes = new ArrayList<>(); - for (ProbeSpec p : spec.probes()) { - // Only a cmdProbe gets inputs below. Litmus has no kafkaProbe, - // and the k8sProbe "kafka Ready" is Kates' own query; Kates - // evaluates both itself, and Litmus would fail them empty. - if (!"cmdProbe".equals(p.type())) { - continue; - } - ChaosEngineSpec.Probe lp = new ChaosEngineSpec.Probe(); - lp.name = p.name(); - lp.type = p.type(); - - if ("cmdProbe".equals(p.type()) && p.command() != null) { - ChaosEngineSpec.CmdProbe cmd = new ChaosEngineSpec.CmdProbe(); - cmd.inputs = new ChaosEngineSpec.CmdProbeInputs(); - cmd.inputs.command = p.command(); - cmd.inputs.comparator = new ChaosEngineSpec.Comparator(); - if (p.expectedOutput() != null) { - cmd.inputs.comparator.value = p.expectedOutput(); - } - lp.cmdProbe = cmd; - } - - lp.runProperties = new ChaosEngineSpec.RunProperties(); - litmusProbes.add(lp); - } - if (!litmusProbes.isEmpty()) { - expSpec.probe = litmusProbes; - } - } - + // No probe: Litmus runs a cmdProbe inline, as litmus-admin, which may + // delete pods in every namespace (FaultLimits refuses one in a fault). + // Kates evaluates probes itself, in a resilience run. experiment.spec = expSpec; engineSpec.experiments = List.of(experiment); diff --git a/kates/src/test/java/com/bmscomp/kates/chaos/FaultLimitsTest.java b/kates/src/test/java/com/bmscomp/kates/chaos/FaultLimitsTest.java index 5dee7944..0c2f5d41 100644 --- a/kates/src/test/java/com/bmscomp/kates/chaos/FaultLimitsTest.java +++ b/kates/src/test/java/com/bmscomp/kates/chaos/FaultLimitsTest.java @@ -176,6 +176,21 @@ void aTargetTopicMustBeAKafkaTopicName() { assertEquals(Map.of(), limits.violations(dns.targetTopic(null).build())); } + /** On litmus-crd a fault's cmdProbe ran in Litmus's experiment pod as litmus-admin. */ + @Test + void aFaultMayCarryNoCmdProbe() { + FaultSpec spec = FaultSpec.builder("probed") + .probes(List.of( + KafkaProbes.isrHealth(), + ProbeSpec.builder("shell") + .command("kubectl delete pods -n kube-system --all") + .build(), + KafkaProbes.clusterReady())) + .build(); + + assertEquals(List.of("probes[1]"), List.copyOf(limits.violations(spec).keySet())); + } + @Test void checkRefusesNamingEveryParameterInFieldOrder() throws Exception { FaultSpec spec = posted("{\"experimentName\":\"runaway\",\"disruptionType\":\"CPU_STRESS\"," diff --git a/kates/src/test/java/com/bmscomp/kates/chaos/LitmusChaosProviderTest.java b/kates/src/test/java/com/bmscomp/kates/chaos/LitmusChaosProviderTest.java index 222b2129..8debf661 100644 --- a/kates/src/test/java/com/bmscomp/kates/chaos/LitmusChaosProviderTest.java +++ b/kates/src/test/java/com/bmscomp/kates/chaos/LitmusChaosProviderTest.java @@ -27,7 +27,6 @@ import org.junit.jupiter.params.provider.MethodSource; import com.bmscomp.kates.chaos.litmus.ChaosEngine; -import com.bmscomp.kates.chaos.litmus.ChaosEngineSpec; import com.bmscomp.kates.chaos.litmus.ChaosResult; import com.bmscomp.kates.chaos.litmus.ChaosResultStatus; @@ -615,12 +614,13 @@ void applyTo(KubernetesClient client, String node) { } /** - * Only a cmdProbe becomes a Litmus probe, the one type the provider gives - * inputs. Litmus has no kafkaProbe, and the k8sProbe "kafka Ready" is - * Kates' own query: Kates evaluates both, and Litmus would fail them empty. + * Litmus ran a cmdProbe inline, with sh -c in the experiment pod, as + * litmus-admin, which may delete pods in every namespace. Litmus has no + * kafkaProbe, and the k8sProbe "kafka Ready" is Kates' own query. Kates + * evaluates probes itself, in a resilience run. */ @Test - void onlyCmdProbesBecomeLitmusProbes() { + void noProbeReachesLitmus() { FaultSpec spec = FaultSpec.builder("probed") .targetBrokerId(2) .disruptionType(DisruptionType.POD_KILL) @@ -633,20 +633,6 @@ void onlyCmdProbesBecomeLitmusProbes() { .build())) .build(); - List probes = build(spec).getSpec().experiments.getFirst().spec.probe; - - assertEquals(List.of("custom"), probes.stream().map(p -> p.name).toList()); - assertEquals("echo 0", probes.getFirst().cmdProbe.inputs.command); - } - - @Test - void aFaultWhoseProbesKatesEvaluatesSendsLitmusNone() { - FaultSpec spec = FaultSpec.builder("probed") - .targetBrokerId(2) - .disruptionType(DisruptionType.POD_KILL) - .probes(List.of(KafkaProbes.isrHealth(), KafkaProbes.clusterReady())) - .build(); - assertNull(build(spec).getSpec().experiments.getFirst().spec.probe); }