Skip to content

Commit b09e3d2

Browse files
Adds node rebuild section to add/remove replica
- Using only attach with the tables UUID is usually enough Co-authored-by: Diego Nieto <dnieto@altinity.com>
1 parent c45f665 commit b09e3d2

1 file changed

Lines changed: 38 additions & 2 deletions

File tree

content/en/altinity-kb-setup-and-maintenance/altinity-kb-data-migration/add_remove_replica.md

Lines changed: 38 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,13 @@
11
---
2-
title: "Add/Remove a new replica to a ClickHouse® cluster"
2+
title: "Add/Remove/Rebuild a ClickHouse® replica"
33
linkTitle: "add_remove_replica"
44
description: >
5-
How to add/remove a new ClickHouse replica manually and using `clickhouse-backup`
5+
How to add/remove a new ClickHouse replica manually and using `clickhouse-backup`, and how to rebuild a replica that lost local storage
66
keywords:
77
- clickhouse replica
88
- clickhouse add replica
99
- clickhouse remove replica
10+
- clickhouse rebuild node
1011
---
1112

1213
## ADD nodes/replicas to a ClickHouse® cluster
@@ -272,3 +273,38 @@ FROM system.replicas
272273
```
273274

274275
- Delete the replica in the cluster configuration: `remote_servers.xml` and shutdown the node/replica removed.
276+
277+
## REBUILD a node that lost local storage
278+
279+
Before touching schema, make sure the new node's server config (macros, ZooKeeper/Keeper connection, `storage_configuration`) matches the dead node's.
280+
281+
A dead replica's registration under `/clickhouse/tables/.../replicas/<name>` survives in Keeper untouched, since Keeper is a separate service the dead node never had access to. That's what makes rebuild work at all: re-attaching under the *same replica name* reconnects to that leftover state and replication fetches whatever is missing.
282+
283+
If a durable/cold-tier disk did survive the node's death (only "warm"/local storage was lost), the part directories on that surviving disk are named after the table's UUID. Just run this against any **currently healthy** replica to get ready-to-run `ATTACH` statements (same UUIDs) for the rebuilt node:
284+
285+
```sql
286+
SELECT
287+
replaceRegexpOne(replaceOne(concat(create_table_query, ';'), '(', 'ON CLUSTER \'{cluster}\' ('), 'CREATE (TABLE|DICTIONARY|VIEW|LIVE VIEW|WINDOW VIEW)', 'ATTACH \\1 IF NOT EXISTS')
288+
FROM
289+
system.tables
290+
WHERE engine != 'MaterializedView' and
291+
database NOT IN ('system', 'information_schema', 'INFORMATION_SCHEMA') AND
292+
create_table_query != '' AND
293+
name NOT LIKE '.inner.%%' AND
294+
name NOT LIKE '.inner_id.%%'
295+
INTO OUTFILE '/tmp/schema.sql' AND STDOUT
296+
FORMAT TSVRaw
297+
SETTINGS show_table_uuid_in_table_create_query_if_not_nil=1;
298+
```
299+
300+
{{% alert title="Warning" color="warning" %}}
301+
Only reuse a UUID this way if you're certain that replica's ZK registration was never dropped and the surviving disk truly belongs to this table
302+
{{% /alert %}}
303+
304+
Parts still present under that UUID on the surviving disk load in place (checksum-verified, no network transfer); only the parts that lived on the lost disk get fetched from the healthy replica.
305+
306+
Two things this depends on:
307+
308+
309+
- **`ATTACH` only reconnects, it never creates ZK bookkeeping.** If the replica's ZK path was already dropped (or never existed, a genuinely new node), the same `ATTACH` command leaves the table permanently readonly (`No metadata in ZooKeeper for .../replicas/<name>: table will stay in readonly mode`) no error, no self-healing. Use `CREATE TABLE` for a node that never held this replica before; that path does create the ZK registration.
310+
- **Rebuilt/fetched parts land on the first storage-policy volume**, not wherever they originally lived, the storage policy itself has no memory of prior tiering, and `move_factor`-based moves only trigger under disk-space pressure, not on rebuild. If the table has a TTL `TO VOLUME`/`TO DISK` move rule, it re-sorts these parts back to cold automatically in the background (no `MOVE PARTITION` needed) once TTL is next checked. Without a TTL rule, you have to move them manually.

0 commit comments

Comments
 (0)