Skip to content

fix(chaos): close the Litmus paths around the namespace guard - #429

Merged
bmscomp merged 4 commits into
mainfrom
fix/litmus-cmdprobe
Oct 20, 2026
Merged

bmscomp merged 4 commits into
mainfrom
fix/litmus-cmdprobe

Conversation

@bmscomp

@bmscomp bmscomp commented Oct 20, 2026 •

Copy link
Copy Markdown
Owner

Follows #427, which limits a chaos step's targetNamespace to the Kafka namespace or the Kates API's own. A review of #427 found three ways around that check on the default litmus-crd provider, all through what Kates hands Litmus. Every Litmus experiment runs as litmus-admin, which the kates-chaos chart lets delete pods, evict and patch nodes in every namespace by default (rbac.nodeChaos.enabled). Reading litmus-go and chaos-runner 3.28 for those turned up a fourth problem: NETWORK_PARTITION never worked on Litmus.

Commits

  1. envOverrides may not pick the Litmus namespace or pods (98aabed). chaos-runner sets CHAOS_NAMESPACE and TARGETS first and then lets the ChaosEngine's env overwrite them. litmus-go looks TARGET_PODS up in either namespace. So envOverrides could aim a step at kube-system, past both the namespace check and the broker count.
  2. envOverrides may set only TARGET_NODE and NODE_LABEL (3267561). This replaces commit 1's denylist with an allowlist. Litmus's stress, network, DNS and disk-fill helper pods run privileged as root with hostPID. LIB_IMAGE sets their image and CHAOS_SERVICE_ACCOUNT their service account. litmus-go also pastes many env values into bash -c there (CHAOS_TYPE, MATCH_SCHEME, SPOOF_MAP, CPU_LOAD, NETWORK_INTERFACE…), so nearly any override was root on a node. targetTopic reached the same shell as TARGET_HOSTNAMES on a DNS_ERROR. It now has to be a Kafka topic name; a hostname like krafter-kafka-bootstrap.kafka.svc still fits.
  3. NETWORK_PARTITION on litmus-crd is a Kates NetworkPolicy (085bf6a). pod-network-partition fails with "provide the appLabel" when TARGETS is empty, and Kates sends no appinfo. It also ignores TARGET_PODS and selects its pods by one label. So split-brain, consumer-isolation and the resilience example never cut anything off. Litmus now hands the type to the kubernetes provider, as it already does POD_DELETE. The kates chart (0.11.5) moves NetworkPolicy create/delete/deletecollection out from behind rbac.directChaos.
  4. A fault may carry no cmdProbe, and Litmus gets no probe (a77f339). A fault's cmdProbe became an inline Litmus probe, and Litmus runs those with /bin/sh -c in the experiment pod as litmus-admin. A resilience run's top-level probes, which Kates runs in a broker pod, are unchanged. Nothing shipped puts a probe in a fault.

All refusals go through FaultLimits, which the safety guard, the coordinator and the compound orchestrator apply. Each change has a test that fails on the code before it. The book (07, 02, versions appendix), the disruption guide, the playbook catalog, the overview, the troubleshooting page, the examples README and the chart README match. The two MCP caveats that list what litmus-crd hands the kubernetes provider name NETWORK_PARTITION, and their refs follow LitmusChaosProvider.java.

Upgrade notes

  • A plan, schedule or resilience file that sets any envOverrides key other than TARGET_NODE/NODE_LABEL, or a cmdProbe inside a fault, is now refused (422 for a plan, 400 for a resilience run). Cluster settings such as CONTAINER_RUNTIME/SOCKET_PATH go in the kates-chaos chart's experiments.definitions[].env.
  • The default kates ClusterRole now lets the API create and delete NetworkPolicies cluster-wide; fix(chaos): a step may target only the Kafka namespace or the Kates API's own #427's namespace check limits where it does.

Testing

  • ./mvnw test (KUBECONFIG pointed at a nonexistent file, gRPC codegen skipped): 1,650 tests, 0 failures, 1 skipped
  • Spotless check on JDK 25, go vet ./... and go test ./... in cli, including TestMCPCaveat*
  • helm lint charts/kates, plus helm template with rbac.directChaos false and true

Not tested on a cluster. The pod-network-partition failure and the helper-pod shell use come from reading the 3.28 sources, not from a reproduction.

The namespace guard checks a step's targetNamespace, but a Litmus step's
envOverrides went to the ChaosEngine as given, after the env Kates sets.
The chaos-runner (3.28) sets CHAOS_NAMESPACE and TARGETS and then lets
the engine's env overwrite them; litmus-go looks TARGET_PODS up in the
namespace of TARGETS, or else CHAOS_NAMESPACE; pod-network-partition
cuts off APP_LABEL's pods in APP_NAMESPACE. Every experiment runs as
litmus-admin, which the kates-chaos chart lets delete pods and evict
in every namespace by default (rbac.nodeChaos.enabled). So an override
such as APP_NAMESPACE=kube-system, or CHAOS_NAMESPACE with TARGET_PODS,
aimed a step past both the namespace check and the broker count.

FaultLimits already refuses the overrides TOTAL_CHAOS_DURATION and
RAMP_TIME; it now refuses CHAOS_NAMESPACE, TARGETS, APP_NAMESPACE,
APP_LABEL, APP_KIND and TARGET_PODS too. The safety guard applies it
just before the namespace check, and the coordinator and the compound
orchestrator apply it as well. TARGET_NODE and NODE_LABEL, which pick
the node a NODE_DRAIN drains, stay allowed; the guard checks that node.

The guard test for APP_NAMESPACE=kube-system fails on the old guard. The
book, the disruption guide and the examples README name the refused
variables. No caveat ref moves.
Litmus starts helper pods for its stress, network, DNS and disk-fill
experiments, and they run privileged as root with the host's PIDs. An
override picked their image (LIB_IMAGE), their service account
(CHAOS_SERVICE_ACCOUNT) and a host path they mount (SOCKET_PATH), and
litmus-go 3.28 pastes many values into a shell in them: CHAOS_TYPE and
MATCH_SCHEME unquoted, SPOOF_MAP and TARGET_HOSTNAMES in single quotes
(pod-dns-error), CPU_LOAD and VOLUME_MOUNT_PATH (stress), and
NETWORK_INTERFACE and the ports (network-chaos). So nearly any override
could run a command as root on a node.

Instead of refusing the keys known to be unsafe, FaultLimits now refuses
every override except TARGET_NODE and NODE_LABEL, which pick the node a
NODE_DRAIN drains; the guard checks that node, and Litmus uses both
through the API, not a shell. The refusal covers the keys the previous
commit listed, TOTAL_CHAOS_DURATION and RAMP_TIME, and any a later
Litmus adds. Cluster settings such as CONTAINER_RUNTIME and SOCKET_PATH
go in the kates-chaos chart's experiments.definitions[].env.

targetTopic reached the same shell: on litmus-crd a DNS_ERROR passes it
as TARGET_HOSTNAMES, and nothing checked it. It now has to be a Kafka
topic name, 1 to 249 letters, digits, '.', '_' or '-'; a hostname such
as krafter-kafka-bootstrap.kafka.svc still fits.

The guard test now refuses APP_NAMESPACE and LIB_IMAGE together, and
FaultLimitsTest covers 13 keys and the topic rule; they fail on the
previous commit. The book, the disruption guide and the examples README
say what envOverrides and targetTopic may hold.
On the default litmus-crd provider a NETWORK_PARTITION ran the Litmus
pod-network-partition experiment, which, from the litmus-go 3.28 source,
never cut anything off: it fails with "provide the appLabel" when
TARGETS is empty, and Kates builds the ChaosEngine with no appinfo, so
the runner sets none. It also ignores TARGET_PODS, lists its targets
again from TARGETS, and selects the pods for its NetworkPolicy by one
label, so it could not aim at the one pod Kates picks anyway. The
split-brain and consumer-isolation playbooks and the
resilience-network-partition example relied on it. Not reproduced on a
cluster.

The Litmus provider now hands NETWORK_PARTITION to the kubernetes
provider, as it does POD_DELETE, ROLLING_RESTART and SCALE_DOWN: a
deny-all NetworkPolicy per target pod, matched by all of the pod's
labels, removed when chaosDurationSec ends or the partition fails. The
NETWORK_PARTITION entry leaves EXPERIMENT_MAP, which no longer reaches
it.

The kates chart (0.11.5) grants create, delete and deletecollection on
networkpolicies in the default ClusterRole, out from behind
rbac.directChaos, since both backends now need them; directChaos keeps
statefulsets/scale and pods/ephemeralcontainers. The chart README,
version tables, book (provider table, LitmusChaos integration,
split-brain, dry-run RBAC, rollback), architecture chapter, playbook
catalog, disruption guide and troubleshooting say so, and the two MCP
caveats that list what litmus-crd hands the kubernetes provider name it
too. Caveat and comment refs into LitmusChaosProvider.java follow its
5 new lines.

LitmusChaosProviderTest checks that broker 2 gets a Kates NetworkPolicy
for the duration and no ChaosEngine; it fails on the old routing.
On litmus-crd every cmdProbe in a fault's own probes (a plan step, a
resilience run's chaosSpec, a compound fault) went into the ChaosEngine
as a Continuous probe with no source. litmus-go 3.28 runs such a probe
inline, with /bin/sh -c in the experiment pod (pkg/probe/cmdprobe.go),
every 5 seconds, as litmus-admin, which the kates-chaos chart by default
lets delete pods and evict in every namespace and patch nodes. So a key
with chaos:run could delete pods in kube-system through a probe, past
the namespace check and the envOverrides allowlist.

FaultLimits now refuses a cmdProbe in a fault's probes, the type a probe
without one gets, on every path that applies the limits: the safety
guard, the coordinator and the compound orchestrator. The Litmus
provider no longer builds probes at all; it skipped every other type
already, so nothing reaches Litmus. Kates evaluates probes itself only
in a resilience run, with sh -c in a broker pod: a cmdProbe goes in its
top-level probes, which this leaves alone. Nothing shipped puts a probe
in a fault; the resilience examples and scenarios use kafkaProbe and
k8sProbe at the top level.

FaultLimitsTest refuses a probe running kubectl, and LitmusChaosProvider
Test checks that no probe reaches the ChaosEngine; both fail on the old
code. The book's limits and refusal table, the disruption guide, the
overview and the examples README say where a cmdProbe may go.
@bmscomp
bmscomp changed the base branch from fix/chaos-target-namespace to main October 20, 2026 09:21
@bmscomp
bmscomp force-pushed the fix/litmus-cmdprobe branch from 6899470 to a77f339 Compare October 20, 2026 09:21
@bmscomp
bmscomp merged commit 7b4ed28 into main Oct 20, 2026
64 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant