Repository navigation
fix(chaos): close the Litmus paths around the namespace guard - #429
Merged
Merged
Conversation
The namespace guard checks a step's targetNamespace, but a Litmus step's envOverrides went to the ChaosEngine as given, after the env Kates sets. The chaos-runner (3.28) sets CHAOS_NAMESPACE and TARGETS and then lets the engine's env overwrite them; litmus-go looks TARGET_PODS up in the namespace of TARGETS, or else CHAOS_NAMESPACE; pod-network-partition cuts off APP_LABEL's pods in APP_NAMESPACE. Every experiment runs as litmus-admin, which the kates-chaos chart lets delete pods and evict in every namespace by default (rbac.nodeChaos.enabled). So an override such as APP_NAMESPACE=kube-system, or CHAOS_NAMESPACE with TARGET_PODS, aimed a step past both the namespace check and the broker count. FaultLimits already refuses the overrides TOTAL_CHAOS_DURATION and RAMP_TIME; it now refuses CHAOS_NAMESPACE, TARGETS, APP_NAMESPACE, APP_LABEL, APP_KIND and TARGET_PODS too. The safety guard applies it just before the namespace check, and the coordinator and the compound orchestrator apply it as well. TARGET_NODE and NODE_LABEL, which pick the node a NODE_DRAIN drains, stay allowed; the guard checks that node. The guard test for APP_NAMESPACE=kube-system fails on the old guard. The book, the disruption guide and the examples README name the refused variables. No caveat ref moves.
Litmus starts helper pods for its stress, network, DNS and disk-fill experiments, and they run privileged as root with the host's PIDs. An override picked their image (LIB_IMAGE), their service account (CHAOS_SERVICE_ACCOUNT) and a host path they mount (SOCKET_PATH), and litmus-go 3.28 pastes many values into a shell in them: CHAOS_TYPE and MATCH_SCHEME unquoted, SPOOF_MAP and TARGET_HOSTNAMES in single quotes (pod-dns-error), CPU_LOAD and VOLUME_MOUNT_PATH (stress), and NETWORK_INTERFACE and the ports (network-chaos). So nearly any override could run a command as root on a node. Instead of refusing the keys known to be unsafe, FaultLimits now refuses every override except TARGET_NODE and NODE_LABEL, which pick the node a NODE_DRAIN drains; the guard checks that node, and Litmus uses both through the API, not a shell. The refusal covers the keys the previous commit listed, TOTAL_CHAOS_DURATION and RAMP_TIME, and any a later Litmus adds. Cluster settings such as CONTAINER_RUNTIME and SOCKET_PATH go in the kates-chaos chart's experiments.definitions[].env. targetTopic reached the same shell: on litmus-crd a DNS_ERROR passes it as TARGET_HOSTNAMES, and nothing checked it. It now has to be a Kafka topic name, 1 to 249 letters, digits, '.', '_' or '-'; a hostname such as krafter-kafka-bootstrap.kafka.svc still fits. The guard test now refuses APP_NAMESPACE and LIB_IMAGE together, and FaultLimitsTest covers 13 keys and the topic rule; they fail on the previous commit. The book, the disruption guide and the examples README say what envOverrides and targetTopic may hold.
On the default litmus-crd provider a NETWORK_PARTITION ran the Litmus pod-network-partition experiment, which, from the litmus-go 3.28 source, never cut anything off: it fails with "provide the appLabel" when TARGETS is empty, and Kates builds the ChaosEngine with no appinfo, so the runner sets none. It also ignores TARGET_PODS, lists its targets again from TARGETS, and selects the pods for its NetworkPolicy by one label, so it could not aim at the one pod Kates picks anyway. The split-brain and consumer-isolation playbooks and the resilience-network-partition example relied on it. Not reproduced on a cluster. The Litmus provider now hands NETWORK_PARTITION to the kubernetes provider, as it does POD_DELETE, ROLLING_RESTART and SCALE_DOWN: a deny-all NetworkPolicy per target pod, matched by all of the pod's labels, removed when chaosDurationSec ends or the partition fails. The NETWORK_PARTITION entry leaves EXPERIMENT_MAP, which no longer reaches it. The kates chart (0.11.5) grants create, delete and deletecollection on networkpolicies in the default ClusterRole, out from behind rbac.directChaos, since both backends now need them; directChaos keeps statefulsets/scale and pods/ephemeralcontainers. The chart README, version tables, book (provider table, LitmusChaos integration, split-brain, dry-run RBAC, rollback), architecture chapter, playbook catalog, disruption guide and troubleshooting say so, and the two MCP caveats that list what litmus-crd hands the kubernetes provider name it too. Caveat and comment refs into LitmusChaosProvider.java follow its 5 new lines. LitmusChaosProviderTest checks that broker 2 gets a Kates NetworkPolicy for the duration and no ChaosEngine; it fails on the old routing.
On litmus-crd every cmdProbe in a fault's own probes (a plan step, a resilience run's chaosSpec, a compound fault) went into the ChaosEngine as a Continuous probe with no source. litmus-go 3.28 runs such a probe inline, with /bin/sh -c in the experiment pod (pkg/probe/cmdprobe.go), every 5 seconds, as litmus-admin, which the kates-chaos chart by default lets delete pods and evict in every namespace and patch nodes. So a key with chaos:run could delete pods in kube-system through a probe, past the namespace check and the envOverrides allowlist. FaultLimits now refuses a cmdProbe in a fault's probes, the type a probe without one gets, on every path that applies the limits: the safety guard, the coordinator and the compound orchestrator. The Litmus provider no longer builds probes at all; it skipped every other type already, so nothing reaches Litmus. Kates evaluates probes itself only in a resilience run, with sh -c in a broker pod: a cmdProbe goes in its top-level probes, which this leaves alone. Nothing shipped puts a probe in a fault; the resilience examples and scenarios use kafkaProbe and k8sProbe at the top level. FaultLimitsTest refuses a probe running kubectl, and LitmusChaosProvider Test checks that no probe reaches the ChaosEngine; both fail on the old code. The book's limits and refusal table, the disruption guide, the overview and the examples README say where a cmdProbe may go.
bmscomp
force-pushed
the
fix/litmus-cmdprobe
branch
from
October 20, 2026 09:21
6899470 to
a77f339
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follows #427, which limits a chaos step's
targetNamespaceto the Kafka namespace or the Kates API's own. A review of #427 found three ways around that check on the defaultlitmus-crdprovider, all through what Kates hands Litmus. Every Litmus experiment runs aslitmus-admin, which thekates-chaoschart lets delete pods, evict and patch nodes in every namespace by default (rbac.nodeChaos.enabled). Reading litmus-go and chaos-runner 3.28 for those turned up a fourth problem:NETWORK_PARTITIONnever worked on Litmus.Commits
CHAOS_NAMESPACEandTARGETSfirst and then lets the ChaosEngine's env overwrite them. litmus-go looksTARGET_PODSup in either namespace. SoenvOverridescould aim a step atkube-system, past both the namespace check and the broker count.TARGET_NODEandNODE_LABEL(3267561). This replaces commit 1's denylist with an allowlist. Litmus's stress, network, DNS and disk-fill helper pods run privileged as root with hostPID.LIB_IMAGEsets their image andCHAOS_SERVICE_ACCOUNTtheir service account. litmus-go also pastes many env values intobash -cthere (CHAOS_TYPE,MATCH_SCHEME,SPOOF_MAP,CPU_LOAD,NETWORK_INTERFACE…), so nearly any override was root on a node.targetTopicreached the same shell asTARGET_HOSTNAMESon aDNS_ERROR. It now has to be a Kafka topic name; a hostname likekrafter-kafka-bootstrap.kafka.svcstill fits.pod-network-partitionfails with "provide the appLabel" whenTARGETSis empty, and Kates sends no appinfo. It also ignoresTARGET_PODSand selects its pods by one label. Sosplit-brain,consumer-isolationand the resilience example never cut anything off. Litmus now hands the type to the kubernetes provider, as it already doesPOD_DELETE. Thekateschart (0.11.5) moves NetworkPolicy create/delete/deletecollection out from behindrbac.directChaos.cmdProbebecame an inline Litmus probe, and Litmus runs those with/bin/sh -cin the experiment pod aslitmus-admin. A resilience run's top-levelprobes, which Kates runs in a broker pod, are unchanged. Nothing shipped puts a probe in a fault.All refusals go through
FaultLimits, which the safety guard, the coordinator and the compound orchestrator apply. Each change has a test that fails on the code before it. The book (07, 02, versions appendix), the disruption guide, the playbook catalog, the overview, the troubleshooting page, the examples README and the chart README match. The two MCP caveats that list whatlitmus-crdhands the kubernetes provider nameNETWORK_PARTITION, and their refs followLitmusChaosProvider.java.Upgrade notes
envOverrideskey other thanTARGET_NODE/NODE_LABEL, or acmdProbeinside a fault, is now refused (422for a plan,400for a resilience run). Cluster settings such asCONTAINER_RUNTIME/SOCKET_PATHgo in thekates-chaoschart'sexperiments.definitions[].env.katesClusterRole now lets the API create and delete NetworkPolicies cluster-wide; fix(chaos): a step may target only the Kafka namespace or the Kates API's own #427's namespace check limits where it does.Testing
./mvnw test(KUBECONFIG pointed at a nonexistent file, gRPC codegen skipped): 1,650 tests, 0 failures, 1 skippedgo vet ./...andgo test ./...incli, includingTestMCPCaveat*helm lint charts/kates, plushelm templatewithrbac.directChaosfalse and trueNot tested on a cluster. The
pod-network-partitionfailure and the helper-pod shell use come from reading the 3.28 sources, not from a reproduction.