Skip to content

docs(operator): add safe node-pool maintenance runbook - #557

Open
sklarsa wants to merge 7 commits into
mainfrom
docs/operator-node-pool-maintenance
Open

sklarsa wants to merge 7 commits into
mainfrom
docs/operator-node-pool-maintenance

Conversation

@sklarsa

@sklarsa sklarsa commented Sep 17, 2026

Copy link
Copy Markdown
Contributor

What

Documents a complete add-before-remove node-pool maintenance workflow for replicated QuestDB clusters:

  • cordon retiring Nodes before adding capacity;
  • add and validate one surge replica;
  • handle a primary through planned promotion;
  • drain one Node at a time under the default PDB;
  • wait for the original Pod/PVC identity to re-home and recover; and
  • restore the steady instance count and handle the temporary PVC.

It also documents the default hostname/zone placement behavior, scheduling overrides, same-zone capacity requirements, and safe stop/recovery points.

Why

The existing instances API already supports safe node-pool rotation through scale out → drain → scale in. A proposed spec.rotation.retire API in questdb/questdb-enterprise-operator#346 had no controller behavior and duplicated this workflow, so that PR was closed in favor of documenting the supported procedure.

Verification

  • yarn build
  • git diff --check

@github-actions

github-actions Bot commented Sep 17, 2026 •

Copy link
Copy Markdown

🚀 Build success!

Latest successful preview: https://preview-557--questdb-documentation.netlify.app/docs/

Commit SHA: cc4d1c7

📦 Build generates a preview & updates the link on each commit.

@sklarsa
sklarsa force-pushed the docs/operator-node-pool-maintenance branch from b6830e2 to a76d4ad Compare October 16, 2026 15:24
@sklarsa
sklarsa changed the base branch from main to docs/operator-singleton-pdb October 16, 2026 15:24
@sklarsa
sklarsa marked this pull request as ready for review October 16, 2026 17:14
javier
javier previously approved these changes Oct 19, 2026
javier added a commit that referenced this pull request Oct 19, 2026
…air (#581)

**Draft: hold until the operator release that contains
questdb/questdb-enterprise-operator#448.** Until then the published
behaviour is still the old one.

Operator change: a one-instance cluster no longer gets a
PodDisruptionBudget by default, so a node drain or node upgrade restarts
the database instead of hanging.

- `known-limitations.md`: "Singleton drain is blocked by default"
becomes "Draining a singleton's node restarts the database".
- `configuration.md`: the default PDB applies to replicated clusters
only; how to opt a singleton back in.
- `operations/database.md`: the node maintenance section is rewritten
for the new default.
- Adds a note recommending node auto-repair for a node that is
`NotReady` while its machine keeps running (the other half of
questdb/questdb-enterprise-operator#64).

Before merging:
- `reference/api.md` is not touched; it is regenerated from the operator
at release and will pick up the new field text then.
- This overlaps #557, which edits the same opening of the node
maintenance section in `operations/database.md`. Whichever merges second
needs a rebase.
- The auto-repair sentence says AKS repairs unhealthy nodes
automatically and that EKS needs node auto repair enabled on the managed
node group. Worth a check against current cloud docs.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: javier ramírez <javier@questdb.io>
Base automatically changed from docs/operator-singleton-pdb to main October 19, 2026 08:11
@javier
javier dismissed their stale review October 19, 2026 08:11

The base branch was changed.

sklarsa and others added 2 commits October 19, 2026 10:37
…ol-maintenance

# Conflicts:
#	documentation/enterprise-kubernetes-operator/operations/database.md
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants