Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -213,7 +213,7 @@ When you install charts with Helm directly:
| [`kafka-ui`](charts/kafka-ui/) | 0.3.0 | v1.5.0 | A Helm chart for deploying Kafka UI (Kafbat) with Strimzi SCRAM-SHA-512 authentication |
| [`kates-chaos`](charts/kates-chaos/) | 2.2.1 | 3.28.0 | Kates Chaos Engineering — wraps the LitmusChaos execution plane (operator, exporter, CRDs) with Kafka-specific RBAC, experiment/engine templating, and monitoring |
| [`kates-platform`](charts/kates-platform/) | 0.7.0 | 1.0.0 | Umbrella chart for the full Kates platform — Kafka, Kates, and supporting infrastructure |
| [`kates`](charts/kates/) | 0.11.4 | 1.26.0 | Kates — Kafka Advanced Testing & Engineering Suite |
| [`kates`](charts/kates/) | 0.11.5 | 1.26.0 | Kates — Kafka Advanced Testing & Engineering Suite |
| [`legacy-kafka`](charts/legacy-kafka/) | 0.2.0 | 3.9.1 | A deliberately old Kafka (2.x on ZooKeeper, or 3.x on KRaft) to migrate FROM — the replication source that the pinned Strimzi release cannot deploy |
| [`mirror-maker2`](charts/mirror-maker2/) | 0.11.5 | 4.3.1 | Strimzi KafkaMirrorMaker2 — cross-cluster replication, DR, and cross-version migration (2.x/3.x → 4.x) |
| [`monitoring`](charts/monitoring/) | 1.6.0 | 82.4.3 | Kates Monitoring — wraps kube-prometheus-stack with Kates-specific dashboards and configuration |
Expand Down
9 changes: 6 additions & 3 deletions charts/kates/Chart.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,10 @@ apiVersion: v2
name: kates
description: Kates — Kafka Advanced Testing & Engineering Suite
type: application
# Patch: appVersion and the image tag move to 1.26.0, the tag the v1.26.0
# release publishes. Templates are otherwise unchanged.
version: 0.11.4
# Patch: the default ClusterRole grants create, delete and deletecollection
# on networkpolicies, which only rbac.directChaos granted before.
# appVersion is unchanged.
version: 0.11.5
# The non-blocking health endpoints this chart's probes now point at come from
# a BUILD-TIME property (quarkus.messaging.health.enabled=false), so they only
# exist in the image built from this branch. Keep appVersion and
Expand All @@ -28,6 +29,8 @@ annotations:
artifacthub.io/category: integration-delivery
artifacthub.io/license: Apache-2.0
artifacthub.io/changes: |
- kind: fixed
description: "The default ClusterRole grants create, delete and deletecollection on networkpolicies, which only rbac.directChaos granted before. On the default litmus-crd provider, NETWORK_PARTITION ran the Litmus pod-network-partition experiment, which fails before cutting anything off without the appinfo Kates leaves out; the Kates API now applies its own deny-all NetworkPolicy to each target pod on both chaos providers, as the kubernetes provider did, and removes it when the duration ends. rbac.directChaos still grants statefulsets/scale and pods/ephemeralcontainers"
- kind: changed
description: "appVersion 1.26.0 and the image tag move to 1.26.0. The backend changes since 1.25.1: the cluster health check reports a KRaft quorum it cannot describe instead of dropping it (#403); the security audit and pentest read each listener's protocol (#379); the tuning report endpoint is gone (#386); run retention keeps the run a baseline names (#378); a status poll writes only over the status it read (#377); each INTEGRITY run reads in a consumer group of its own (#376); compound runs and resilience faults go through the safety guard and the disruption lease (#375); orphan recovery undoes faults left from before the API started (#374); and the native image starts with a named API keys file (#371). Templates are otherwise unchanged."
- kind: added
Expand Down
5 changes: 2 additions & 3 deletions charts/kates/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -111,15 +111,14 @@ Upgrading from 0.10.7 or earlier: the ClusterRole there granted `get`, `list` an

### Chaos permissions

With the default chaos provider, `litmus-crd`, Kates only creates and watches `ChaosEngine`s: Litmus runs every experiment as its own `litmus-admin` service account, and the default ClusterRole covers that. The direct Kubernetes backend (`kubernetes`, or `hybrid` on a cluster without Litmus CRDs) injects the faults itself, so it needs write access the default role does not grant. `rbac.directChaos=true` adds it:
With the default chaos provider, `litmus-crd`, Kates creates and watches `ChaosEngine`s, and Litmus runs every experiment as its own `litmus-admin` service account; the default ClusterRole covers that, and the faults Kates injects itself on both backends. The direct Kubernetes backend (`kubernetes`, or `hybrid` on a cluster without Litmus CRDs) injects the faults itself, so it needs write access the default role does not grant. `rbac.directChaos=true` adds it:

| Resource | Verbs | Used by |
|----------|-------|---------|
| `networking.k8s.io` `networkpolicies` | `create`, `delete`, `deletecollection` | `NETWORK_PARTITION` creates a deny-all policy; cleanup and rollback delete by label, startup orphan recovery by name |
| `apps` `statefulsets/scale` | `get`, `update` | On a Kafka that Strimzi does not manage: `SCALE_DOWN`, its rollback and orphan recovery set replicas |
| `pods/ephemeralcontainers` | `update` | `CPU_STRESS` and `IO_STRESS` add a stress container to the target pod |

`ROLLING_RESTART` needs none of these, on either backend, and neither does `SCALE_DOWN` on a Strimzi cluster. On a Strimzi cluster `ROLLING_RESTART` annotates the Kafka pods for the Strimzi Cluster Operator to roll, and on a plain StatefulSet, such as `charts/legacy-kafka`, it restarts the StatefulSet. `SCALE_DOWN` lowers `spec.replicas` of a KafkaNodePool. Both run on either backend, so the default role grants `patch` on pods, on StatefulSets and on `kafkanodepools`. On a Kafka that Strimzi does not manage, `SCALE_DOWN` also patches the StatefulSet, to record its original replica count.
`NETWORK_PARTITION` needs none of these, on either backend: Kates creates its deny-all NetworkPolicy itself, and cleanup, rollback and startup orphan recovery delete it, so the default role grants `create`, `delete` and `deletecollection` on `networkpolicies`. `ROLLING_RESTART` needs none of them either, and neither does `SCALE_DOWN` on a Strimzi cluster. On a Strimzi cluster `ROLLING_RESTART` annotates the Kafka pods for the Strimzi Cluster Operator to roll, and on a plain StatefulSet, such as `charts/legacy-kafka`, it restarts the StatefulSet. `SCALE_DOWN` lowers `spec.replicas` of a KafkaNodePool. Both run on either backend, so the default role grants `patch` on pods, on StatefulSets and on `kafkanodepools`. On a Kafka that Strimzi does not manage, `SCALE_DOWN` also patches the StatefulSet, to record its original replica count.

These rules are cluster-wide, like the rest of the role, and `pods/ephemeralcontainers` lets Kates start a container in any pod. Leave them off unless you use the direct backend. It is selected with `chaos.provider`:

Expand Down
12 changes: 7 additions & 5 deletions charts/kates/templates/rbac.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -51,20 +51,22 @@ rules:
- apiGroups: ["core.strimzi.io"]
resources: ["strimzipodsets"]
verbs: ["get", "list"]
# NETWORK_PARTITION is a NetworkPolicy Kates applies itself on both chaos
# backends, so it is not behind rbac.directChaos. Cleanup, rollback and
# orphan recovery delete Kates NetworkPolicies by label: deletecollection.
- apiGroups: ["networking.k8s.io"]
resources: ["networkpolicies"]
verbs: ["create", "delete", "deletecollection"]
{{- if .Values.rbac.directChaos }}
{{- /*
The direct Kubernetes chaos backend (kates.chaos.provider=kubernetes, or
hybrid on a cluster without Litmus) makes these writes itself. Litmus runs
every experiment as its own litmus-admin service account, so a Litmus
install needs none of them. Cleanup and rollback delete NetworkPolicies by
label, which is deletecollection; on a Kafka that Strimzi does not manage,
install needs none of them. On a Kafka that Strimzi does not manage,
SCALE_DOWN sets replicas through the StatefulSet's scale subresource (its
PATCH on the StatefulSet is granted above, for ROLLING_RESTART); CPU_STRESS
and IO_STRESS add a container through pods/ephemeralcontainers.
*/}}
- apiGroups: ["networking.k8s.io"]
resources: ["networkpolicies"]
verbs: ["create", "delete", "deletecollection"]
- apiGroups: ["apps"]
resources: ["statefulsets/scale"]
verbs: ["get", "update"]
Expand Down
4 changes: 2 additions & 2 deletions charts/kates/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -58,8 +58,8 @@ rbac:
# -- Create ClusterRole and ClusterRoleBinding
create: true
# -- Grant the writes the direct Kubernetes chaos backend makes itself
# (KATES_CHAOS_PROVIDER=kubernetes, or hybrid without Litmus): NetworkPolicies,
# StatefulSet scaling, and pod ephemeral containers. Litmus runs its
# (KATES_CHAOS_PROVIDER=kubernetes, or hybrid without Litmus): StatefulSet
# scaling and pod ephemeral containers. Litmus runs its
# experiments as litmus-admin and needs none of this.
directChaos: false
# -- Namespaces where the INTEGRATION_CDC test may read its database's
Expand Down
2 changes: 1 addition & 1 deletion cli/cmd/mcp_caveats.go
Original file line number Diff line number Diff line change
Expand Up @@ -221,7 +221,7 @@ var mcpCaveatsCore = []mcpCaveat{
Refs: []string{
mcpAnchoredRef("kates/src/main/resources/application.properties:274-275",
"# Chaos coordination (noop | kubernetes | litmus-crd | hybrid)", "kates.chaos.provider=litmus-crd"),
mcpAnchoredRef(mcpJava+"chaos/LitmusChaosProvider.java:57,102",
mcpAnchoredRef(mcpJava+"chaos/LitmusChaosProvider.java:62,107",
"int resultPollIntervalMs = 5_000;", "Instant start = Instant.now();"),
},
},
Expand Down
16 changes: 8 additions & 8 deletions cli/cmd/mcp_chaos_preview.go
Original file line number Diff line number Diff line change
Expand Up @@ -86,9 +86,9 @@ var mcpAdHocBounds = map[string]mcpAdHocBound{
// field descriptions say so: the kubernetes provider sizes IO_STRESS by
// ioWorkers alone (KubernetesChaosProvider.java:525-538), while litmus-crd
// sizes it by fillPercentage as well as ioWorkers
// (LitmusChaosProvider.java:314-318). Both honour gracePeriodSec: litmus-crd
// (LitmusChaosProvider.java:319-323). Both honour gracePeriodSec: litmus-crd
// hands a POD_DELETE to the kubernetes provider, which deletes the pod with it
// (LitmusChaosProvider.java:42-54,81-83; KubernetesChaosProvider.java:205-213).
// (LitmusChaosProvider.java:41-59,86-88; KubernetesChaosProvider.java:205-213).
// fillPercentage stays out of ad-hoc plans with DISK_FILL, the one type it is
// meant for.
var mcpAdHocTypeParam = map[string]string{
Expand Down Expand Up @@ -344,7 +344,7 @@ func mcpCheckAdHocTargets(at string, s *mcpAdHocStep) error {
if targets > 1 {
return mcpInvalidArgument(at+" sets more than one of targetBrokerId, targetPod and targetTopic; give at most one.", "")
}
// LitmusChaosProvider.java:319-323 makes a DNS_ERROR step's topic the
// LitmusChaosProvider.java:324-328 makes a DNS_ERROR step's topic the
// hostname whose lookups fail, so the fault would break only a hostname
// named after the topic.
if s.DisruptionType == "DNS_ERROR" && s.TargetTopic != "" {
Expand Down Expand Up @@ -761,11 +761,11 @@ func mcpTargetingOf(fs *mcpFaultSpecView) string {
// reading of plan §5.2 than its text ("no experimentName in the resolved
// plan"). Every playbook sets experimentName on every step
// (kates/src/main/resources/playbooks/*.yaml), so the literal rule would flag
// them all, while the Litmus provider maps every type but three to its own
// experiment and routes those three, POD_DELETE, ROLLING_RESTART and
// SCALE_DOWN, to the kubernetes provider: experimentName picks the experiment
// only for a step without a type
// (LitmusChaosProvider.java:29-54,81-83,228-255), and otherwise only names
// them all, while the Litmus provider maps every type but four to its own
// experiment and routes those four, POD_DELETE, NETWORK_PARTITION,
// ROLLING_RESTART and SCALE_DOWN, to the kubernetes provider: experimentName
// picks the experiment only for a step without a type
// (LitmusChaosProvider.java:29-59,86-88,233-260), and otherwise only names
// the fault. The plan's wording is a follow-up.
func mcpAgentLimitFindings(v *mcpPlanView) []mcpAgentLimitFinding {
findings := []mcpAgentLimitFinding{}
Expand Down
13 changes: 7 additions & 6 deletions cli/cmd/mcp_tools_chaos.go
Original file line number Diff line number Diff line change
Expand Up @@ -127,14 +127,14 @@ var mcpCaveatsChaos = []mcpCaveat{
"create NetworkPolicies or scale node pools, and only for the fault types that need one of those; every " +
"other type, and any check that fails with an error, counts as permitted, and a denied check is a step " +
"warning, never an error that makes wouldSucceed false. On the default litmus-crd provider Litmus " +
"injects every fault except POD_DELETE, ROLLING_RESTART and SCALE_DOWN as its own litmus-admin service " +
"account, so for those the check does not show whether the fault can run.",
"injects every fault except POD_DELETE, NETWORK_PARTITION, ROLLING_RESTART and SCALE_DOWN as its own " +
"litmus-admin service account, so for those the check does not show whether the fault can run.",
Refs: []string{
mcpAnchoredRef(mcpJava+"disruption/DisruptionSafetyGuard.java:275-278,772-841",
"boolean canExecute = checkRbacPermissions(spec);", `"Insufficient RBAC permissions for "`,
"boolean checkRbacPermissions(FaultSpec spec)", `"RBAC check failed, assuming permitted"`),
mcpAnchoredRef(mcpJava+"chaos/LitmusChaosProvider.java:42-54,81-83,265",
"EnumSet.of(DisruptionType.ROLLING_RESTART, DisruptionType.SCALE_DOWN, DisruptionType.POD_DELETE)",
mcpAnchoredRef(mcpJava+"chaos/LitmusChaosProvider.java:41-59,86-88,270",
"ON_KUBERNETES_API = EnumSet.of(", "DisruptionType.POD_DELETE,", "DisruptionType.NETWORK_PARTITION);",
"return kubernetes.triggerFault(spec, onInject);", `engineSpec.chaosServiceAccount = "litmus-admin";`),
mcpAnchoredRef("kates/src/main/resources/application.properties:274-275",
"# Chaos coordination (noop | kubernetes | litmus-crd | hybrid)", "kates.chaos.provider=litmus-crd"),
Expand Down Expand Up @@ -228,7 +228,8 @@ var mcpCaveatsChaos = []mcpCaveat{
"count from the moment the chaos provider reported the fault going in: after the step's delayBeforeSec, " +
"before the provider changed the cluster. On the default litmus-crd provider that moment comes before it " +
"picks the pods and creates the ChaosEngine, so the times include however long Litmus took to start the " +
"experiment. The kubernetes provider, which litmus-crd hands POD_DELETE, ROLLING_RESTART and SCALE_DOWN, " +
"experiment. The kubernetes provider, which litmus-crd hands POD_DELETE, NETWORK_PARTITION, ROLLING_RESTART " +
"and SCALE_DOWN, " +
"reports before it looks up what to hit, so a fault that then finds nothing to hit fails with recovery " +
"times all the same. Time to first ready is the first Ready event from any watched Kafka pod after that " +
"moment, which need not be a pod the fault hit. A step whose fault never went in, a Skipped one or one that " +
Expand All @@ -248,7 +249,7 @@ var mcpCaveatsChaos = []mcpCaveat{
mcpAnchoredRef(mcpJava+"disruption/KafkaIntelligenceService.java:125-127,198-217,241-246,310-324",
"public void markDisruptionStart(Instant at)", "private Duration computeTimeToFullIsr()",
"this.baselineLag = timeline.getLast().totalLag();", "private Duration computeTimeToLagRecovery(long baseLag)"),
mcpAnchoredRef(mcpJava+"chaos/LitmusChaosProvider.java:81-93,102-121",
mcpAnchoredRef(mcpJava+"chaos/LitmusChaosProvider.java:86-98,107-126",
"return kubernetes.triggerFault(spec, onInject);",
"CompletableFuture.delayedExecutor(spec.delayBeforeSec(), TimeUnit.SECONDS, executor.get());",
"long startNanos = System.nanoTime();", "buildChaosEngine(spec, engineName, experimentName)",
Expand Down
6 changes: 4 additions & 2 deletions cli/examples/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -154,14 +154,16 @@ that sets a field its run doesn't read, or that the API would refuse.
| `fillPercentage` | int | Disk fill % (`IO_STRESS`) |
| `ioWorkers` | int | Parallel I/O workers (`IO_STRESS`) |
| `networkLatencyMs` | int | Added latency ms (`NETWORK_LATENCY`) |
| `targetTopic` | string | A disruption plan aims a fault at the leader of this topic's `targetPartition`. A resilience run doesn't, but a `DNS_ERROR` on `litmus-crd` takes it as `TARGET_HOSTNAMES` |
| `targetTopic` | string | A disruption plan aims a fault at the leader of this topic's `targetPartition`. A resilience run doesn't, but a `DNS_ERROR` on `litmus-crd` takes it as `TARGET_HOSTNAMES`. It has to be a Kafka topic name: letters, digits, `.`, `_` or `-` |
| `targetPartition` | int | Partition of `targetTopic`, in a disruption plan only |
| `envOverrides` | map | Env vars for the Litmus experiment (`litmus-crd` only). For a `NODE_DRAIN`, `TARGET_NODE` or `NODE_LABEL` picks the node instead of the pod `targetLabel` picks |
| `envOverrides` | map | Env vars for the Litmus experiment (`litmus-crd` only). Only `TARGET_NODE` or `NODE_LABEL`, which pick the node a `NODE_DRAIN` drains instead of the pod `targetLabel` picks; any other variable is refused |

### `probes` field reference

Every probe runs before the fault and after it, until all pass or
`maxRecoveryWaitSec` is up; a `Continuous` one runs while the fault lasts too.
A `cmdProbe` belongs here, not in `chaosSpec.probes`: Kates refuses a fault that
carries one, since on `litmus-crd` Litmus ran it as `litmus-admin`.

| YAML key | Type | Notes |
|---|---|---|
Expand Down
Loading
Loading