From 7c7978278e30c7d2048bd123ade0d52779c50555 Mon Sep 17 00:00:00 2001 From: Steven Sklar Date: Tue, 13 Oct 2026 16:13:51 -0400 Subject: [PATCH 1/3] docs(operator): singleton has no default PDB; recommend node auto-repair Co-Authored-By: Claude Opus 5.5 --- .../configuration.md | 11 ++++---- .../known-limitations.md | 16 +++++++---- .../operations/database.md | 28 +++++++++++-------- 3 files changed, 33 insertions(+), 22 deletions(-) diff --git a/documentation/enterprise-kubernetes-operator/configuration.md b/documentation/enterprise-kubernetes-operator/configuration.md index 4778f9cc2..0133c84ec 100644 --- a/documentation/enterprise-kubernetes-operator/configuration.md +++ b/documentation/enterprise-kubernetes-operator/configuration.md @@ -227,12 +227,13 @@ worker-node IAM can provide ambient ECR pull access; IRSA cannot. `spec.scheduling` supports node selectors, affinity, tolerations, topology spread, priority class, and an operator-managed PodDisruptionBudget (PDB). -Without an override, pods get soft hostname anti-affinity, soft zone spread, and -a PDB with `minAvailable: 1` for one instance or `instances-1` for HA. +Without an override, pods get soft hostname anti-affinity and soft zone spread. +A replicated cluster also gets a PDB with `minAvailable: instances-1`. -The single-instance default intentionally blocks voluntary eviction. Before a -node drain, scale out, lower `spec.scheduling.podDisruptionBudget.minAvailable`, -or disable the PDB after accepting the availability risk. Follow the +A one-instance cluster gets no PDB by default, because a budget over a single +pod would block every node drain. A drain therefore restarts the database. Set +`spec.scheduling.podDisruptionBudget.enabled: true` to block voluntary eviction +instead. Follow the [node maintenance runbook](/docs/enterprise-kubernetes-operator/operations/database/#node-maintenance-and-disruption-budgets). Custom affinity or topology spread replaces the corresponding default rather than merging with it. diff --git a/documentation/enterprise-kubernetes-operator/known-limitations.md b/documentation/enterprise-kubernetes-operator/known-limitations.md index a35fc35d8..3acfe5794 100644 --- a/documentation/enterprise-kubernetes-operator/known-limitations.md +++ b/documentation/enterprise-kubernetes-operator/known-limitations.md @@ -16,15 +16,19 @@ operational planning. A one-instance cluster has no replica to promote. On an unreachable node, the operator does not create a second pod while the old writer might still run. -Recovery waits for safe node/volume recovery or human action. Use store-backed -replication before an incident; see +Recovery waits for safe node/volume recovery or human action. Turn on your +platform's node auto-repair so that an unhealthy node is replaced without +waiting for a person; see +[Node maintenance](/docs/enterprise-kubernetes-operator/operations/database/#node-maintenance-and-disruption-budgets). +Use store-backed replication before an incident; see [Scale out](/docs/enterprise-kubernetes-operator/operations/database/#scale-out). -### Singleton drain is blocked by default +### Draining a singleton's node restarts the database -The default PodDisruptionBudget has `minAvailable: 1`, so voluntary drain of the -singleton waits indefinitely. Before maintenance, scale out, lower -`minAvailable`, or disable the PDB after accepting downtime. See +A one-instance cluster has no PodDisruptionBudget by default. A voluntary drain +evicts the only pod, and the operator recreates it on another node in the same +zone. Expect a short outage on every node drain or node upgrade. To avoid it, +scale out before maintenance. See [Node maintenance](/docs/enterprise-kubernetes-operator/operations/database/#node-maintenance-and-disruption-budgets). ### Failover is never automatic diff --git a/documentation/enterprise-kubernetes-operator/operations/database.md b/documentation/enterprise-kubernetes-operator/operations/database.md index fbc53c6fb..a65ad421c 100644 --- a/documentation/enterprise-kubernetes-operator/operations/database.md +++ b/documentation/enterprise-kubernetes-operator/operations/database.md @@ -405,27 +405,27 @@ into a new cluster. ## Node maintenance and disruption budgets -The default PodDisruptionBudget (PDB) uses `minAvailable: 1` for a singleton and -`instances-1` for a replicated cluster. A singleton therefore blocks voluntary -eviction and can make `kubectl drain` wait indefinitely. +A replicated cluster gets a default PodDisruptionBudget (PDB) with +`minAvailable: instances-1`, which permits one voluntary disruption at a time. -Before planned maintenance, choose one safe option: +A singleton gets no PDB by default. `kubectl drain` and managed node upgrades +evict its only pod, and the operator recreates it on another node in the same +zone as its volume. The database is down until the new pod is ready. To avoid +that outage, scale out and wait for a healthy, current replica before planned +maintenance. -1. scale out and wait for a healthy, current replica; -2. lower `spec.scheduling.podDisruptionBudget.minAvailable`; or -3. set `spec.scheduling.podDisruptionBudget.enabled: false` only after accepting - database downtime. +To make a singleton block voluntary eviction instead, enable the PDB. A drain of +its node then waits until you lower `minAvailable` or disable the PDB again: ```yaml spec: scheduling: podDisruptionBudget: enabled: true - minAvailable: 0 ``` -Restore the normal PDB after maintenance and verify the current-generation -writer contract: `Available=True/PrimaryReady`, `Progressing=False/Settled`, and +After maintenance, verify the current-generation writer contract: +`Available=True/PrimaryReady`, `Progressing=False/Settled`, and `WriteHealthy=True/Healthy`. A PDB protects only voluntary disruption. On node loss, the operator reports @@ -435,6 +435,12 @@ the node/volume or follow the explicit [promotion and failover](/docs/enterprise-kubernetes-operator/high-availability/#promotion-and-failover) procedure. +A node that is `NotReady` while its machine keeps running leaves the pod stuck +until the machine is repaired or replaced. Turn on node auto-repair so this +happens without a person: AKS repairs unhealthy nodes automatically, and on EKS +you enable node auto repair on the managed node group. Once the node is gone, +the operator recreates the pod on a healthy node. + ## Delete a database cluster Deleting a `QuestDBCluster` removes operator-owned pods, Services, ConfigMaps, From e429923c5132e65bc5f9e86e00e7a5f79f65d4d8 Mon Sep 17 00:00:00 2001 From: Steven Sklar Date: Fri, 16 Oct 2026 10:58:15 -0400 Subject: [PATCH 2/3] docs(operator): node repair may reboot rather than replace Co-Authored-By: Claude Opus 5.5 --- .../enterprise-kubernetes-operator/operations/database.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/documentation/enterprise-kubernetes-operator/operations/database.md b/documentation/enterprise-kubernetes-operator/operations/database.md index a65ad421c..04edd95cf 100644 --- a/documentation/enterprise-kubernetes-operator/operations/database.md +++ b/documentation/enterprise-kubernetes-operator/operations/database.md @@ -438,8 +438,8 @@ procedure. A node that is `NotReady` while its machine keeps running leaves the pod stuck until the machine is repaired or replaced. Turn on node auto-repair so this happens without a person: AKS repairs unhealthy nodes automatically, and on EKS -you enable node auto repair on the managed node group. Once the node is gone, -the operator recreates the pod on a healthy node. +you enable node auto repair on the managed node group. Once the node is +repaired or replaced, the pod runs again. ## Delete a database cluster From a76d4ad5e5ba26adc66abe20176a5b4e72cc5ccd Mon Sep 17 00:00:00 2001 From: Steven Sklar Date: Wed, 16 Sep 2026 20:51:09 -0400 Subject: [PATCH 3/3] docs(operator): add safe node-pool maintenance runbook --- .../operations/database.md | 171 ++++++++++++++++++ 1 file changed, 171 insertions(+) diff --git a/documentation/enterprise-kubernetes-operator/operations/database.md b/documentation/enterprise-kubernetes-operator/operations/database.md index 04edd95cf..82aafec69 100644 --- a/documentation/enterprise-kubernetes-operator/operations/database.md +++ b/documentation/enterprise-kubernetes-operator/operations/database.md @@ -414,6 +414,177 @@ zone as its volume. The database is down until the new pod is ready. To avoid that outage, scale out and wait for a healthy, current replica before planned maintenance. +### Rotate a node pool without reducing replica capacity + +Use a temporary surge replica for planned node-pool replacement. Kubernetes node +maintenance remains external to the operator: your infrastructure tooling +cordons, drains, powers off, and deletes Nodes; the operator manages only the +QuestDB Pods and PVCs. + +This procedure preserves each existing instance identity and PVC. After a Pod is +evicted, the operator recreates the same instance on an eligible node and +reattaches its PVC. The temporary highest-serial replica is removed when you +restore the original instance count. + +Before starting: + +- use a replicated cluster with a completed backup seed, as required for + [scale out](#scale-out); +- require current generation, `Available=True/PrimaryReady`, + `Progressing=False/Settled`, `WriteHealthy=True/Healthy`, and + `ReplicationHealthy=True`; +- provide replacement capacity in every zone required by the existing PVCs, + matching the cluster's node selectors, affinity, and tolerations; +- finish any ordinary rollout, promotion, or cold-manager handoff; and +- record the original instance count, Pod UIDs, roles, Nodes, and PVCs. + +```sh +ORIGINAL_INSTANCES="$(kubectl get questdbcluster -n \ + -o jsonpath='{.spec.instances}')" +[ "$ORIGINAL_INSTANCES" -ge 2 ] + +kubectl get questdbcluster -n -o wide +kubectl get pods -n -l questdb.io/cluster= \ + -L questdb.io/instance,questdb.io/role,questdb.io/ro-ready -o wide +kubectl get pvc -n -l questdb.io/cluster= -o wide +``` + +#### 1. Cordon retiring nodes + +Cordon every retiring node before creating the surge replica. Cordon does not +move existing Pods, but it prevents the scheduler from placing the surge or any +replacement Pod on those nodes. + +```sh +kubectl cordon +kubectl get nodes +``` + +By default, QuestDB Pods have preferred hostname anti-affinity with weight 100 +and a soft zone spread with `maxSkew: 1`. These rules prefer different hosts and +balanced zones but do not guarantee either under capacity pressure. An explicit +`spec.scheduling.affinity` fully replaces the default hostname anti-affinity; an +explicit `topologySpreadConstraints` list replaces the default zone spread. +Configure any required hard placement rules before this maintenance and let the +resulting Pod rollout settle first. + +#### 2. Add and verify one surge replica + +Increase the instance count by one. One surge is sufficient when nodes are +drained sequentially, which the default PDB requires. + +```sh +SURGE_INSTANCES=$((ORIGINAL_INSTANCES + 1)) +kubectl patch questdbcluster -n --type merge \ + -p "{\"spec\":{\"instances\":${SURGE_INSTANCES}}}" + +GENERATION="$(kubectl get questdbcluster -n \ + -o jsonpath='{.metadata.generation}')" +for _ in $(seq 1 120); do + OBSERVED="$(kubectl get questdbcluster -n \ + -o jsonpath='{.status.observedGeneration}')" + READY="$(kubectl get questdbcluster -n \ + -o jsonpath='{.status.readyInstances}')" + [ "$OBSERVED" = "$GENERATION" ] && \ + [ "$READY" = "$SURGE_INSTANCES" ] && break + sleep 10 +done +[ "$OBSERVED" = "$GENERATION" ] +[ "$READY" = "$SURGE_INSTANCES" ] + +kubectl get questdbcluster -n \ + -o jsonpath='{range .status.conditions[?(@.type=="ReplicationHealthy")]}{.status}{" "}{.reason}{" "}{.message}{"\n"}{end}{range .status.replication.replicas[*]}{.instance}{" caughtUp="}{.caughtUp}{" caughtUpNow="}{.caughtUpNow}{" lagTxns="}{.lagTxns}{"\n"}{end}' +kubectl get pods -n -l questdb.io/cluster= \ + -L questdb.io/instance,questdb.io/role,questdb.io/ro-ready -o wide +``` + +Proceed only when `ReplicationHealthy=True`, every expected instance is Ready, +and the new replica has a current `caughtUpNow=true` observation and +`questdb.io/ro-ready=true`. A latched `caughtUp=true` by itself is not current +freshness. Confirm that the surge Pod is not on a retiring node. + +If the surge remains Pending, do not drain anything. Add compatible capacity or +restore `spec.instances` to `ORIGINAL_INSTANCES` and uncordon the nodes to +abort. + +#### 3. Handle a primary on a retiring node + +Draining the primary recreates the same primary identity elsewhere and briefly +interrupts writes. For a planned handoff instead, first re-home an original +replica onto a non-retiring node, then create a +[`QuestDBPromotion`](/docs/enterprise-kubernetes-operator/high-availability/#promotion-and-failover) +targeting that replica and wait for it to complete. Drain the primary's node +last. Do not promote the temporary highest-serial surge replica: keeping it a +replica makes the final scale-in remove the temporary instance rather than a +long-lived one. Structured cold storage may require moving its manager first; +follow [Move the cold-storage manager](#move-the-cold-storage-manager). + +Do not use an Emergency promotion for routine maintenance. It explicitly accepts +loss of WAL that the old primary has not uploaded. + +#### 4. Drain one node at a time + +Drain one retiring node through your cloud's managed-node workflow or with +`kubectl drain`. The QuestDB eviction consumes the default PDB's one allowed +voluntary disruption, so wait for full recovery before draining the next node. + +```sh +kubectl drain --ignore-daemonsets +``` + +Do not bypass the PDB or force-delete a QuestDB Pod to accelerate a healthy +planned drain. After the eviction, the operator waits for the old Pod object to +be gone, then recreates that same instance and PVC. The cordon keeps it off +every retiring node. + +Before proceeding to the next node, require all of the following again: + +- `status.observedGeneration` equals `metadata.generation`; +- `status.readyInstances` equals `SURGE_INSTANCES`; +- `ReplicationHealthy=True` and every replica is currently healthy; +- the recreated Pod has a new UID and runs on a non-retiring node; and +- the ordinary writer-health contract is satisfied. + +If recovery stalls, leave the surge in place and stop the maintenance workflow. +Inspect Pod scheduling events, PVC/PV topology, volume attachment, Node health, +and cluster conditions. Recover the node or add same-zone capacity; do not move +on to another drain while the cluster is below the surge count. + +Repeat this step sequentially for each retiring node that hosts a QuestDB Pod. +Nodes with no QuestDB Pod do not require another database surge or recovery +cycle. + +#### 5. Return to the steady instance count + +After every QuestDB Pod runs on a non-retiring node and the cluster is healthy, +restore the original count. Scale-in removes the temporary highest-serial +replica. + +```sh +kubectl patch questdbcluster -n --type merge \ + -p "{\"spec\":{\"instances\":${ORIGINAL_INSTANCES}}}" +``` + +Wait for current generation, `readyInstances=ORIGINAL_INSTANCES`, settled +conditions, and the full writer and replication health contracts. Inventory Pods +and PVCs afterward. With `pvcRetentionPolicy: Retain`, the temporary replica's +PVC remains for deliberate inspection or cleanup; with `Delete`, the operator +removes it during scale-in. + +The retiring Nodes can now remain cordoned and be removed by the external +node-pool workflow. If maintenance is cancelled instead, uncordon only Nodes +that remain valid destinations. + +It is safe to pause with the surge replica running. Restoring the steady count +is cleanup, not a prerequisite for database availability. + +### Other maintenance choices + +For a replicated cluster, when a surge replica is not appropriate, lower +`spec.scheduling.podDisruptionBudget.minAvailable`, or set +`spec.scheduling.podDisruptionBudget.enabled: false` only after explicitly +accepting the resulting database downtime. + To make a singleton block voluntary eviction instead, enable the PDB. A drain of its node then waits until you lower `minAvailable` or disable the PDB again: