|
1 | 1 | --- |
2 | | -title: "Add/Remove a new replica to a ClickHouse® cluster" |
| 2 | +title: "Add/Remove/Rebuild a ClickHouse® replica" |
3 | 3 | linkTitle: "add_remove_replica" |
4 | 4 | description: > |
5 | | - How to add/remove a new ClickHouse replica manually and using `clickhouse-backup` |
| 5 | + How to add/remove a new ClickHouse replica manually and using `clickhouse-backup`, and how to rebuild a replica that lost local storage |
6 | 6 | keywords: |
7 | 7 | - clickhouse replica |
8 | 8 | - clickhouse add replica |
9 | 9 | - clickhouse remove replica |
| 10 | + - clickhouse rebuild node |
10 | 11 | --- |
11 | 12 |
|
12 | 13 | ## ADD nodes/replicas to a ClickHouse® cluster |
@@ -272,3 +273,38 @@ FROM system.replicas |
272 | 273 | ``` |
273 | 274 |
|
274 | 275 | - Delete the replica in the cluster configuration: `remote_servers.xml` and shutdown the node/replica removed. |
| 276 | + |
| 277 | +## REBUILD a node that lost local storage |
| 278 | + |
| 279 | +Before touching schema, make sure the new node's server config (macros, ZooKeeper/Keeper connection, `storage_configuration`) matches the dead node's. |
| 280 | + |
| 281 | +A dead replica's registration under `/clickhouse/tables/.../replicas/<name>` survives in Keeper untouched, since Keeper is a separate service the dead node never had access to. That's what makes rebuild work at all: re-attaching under the *same replica name* reconnects to that leftover state and replication fetches whatever is missing. |
| 282 | + |
| 283 | +If a durable/cold-tier disk did survive the node's death (only "warm"/local storage was lost), the part directories on that surviving disk are named after the table's UUID. Just run this against any **currently healthy** replica to get ready-to-run `ATTACH` statements (same UUIDs) for the rebuilt node: |
| 284 | + |
| 285 | +```sql |
| 286 | +SELECT |
| 287 | + replaceRegexpOne(replaceOne(concat(create_table_query, ';'), '(', 'ON CLUSTER \'{cluster}\' ('), 'CREATE (TABLE|DICTIONARY|VIEW|LIVE VIEW|WINDOW VIEW)', 'ATTACH \\1 IF NOT EXISTS') |
| 288 | +FROM |
| 289 | + system.tables |
| 290 | +WHERE engine != 'MaterializedView' and |
| 291 | + database NOT IN ('system', 'information_schema', 'INFORMATION_SCHEMA') AND |
| 292 | + create_table_query != '' AND |
| 293 | + name NOT LIKE '.inner.%%' AND |
| 294 | + name NOT LIKE '.inner_id.%%' |
| 295 | +INTO OUTFILE '/tmp/schema.sql' AND STDOUT |
| 296 | +FORMAT TSVRaw |
| 297 | +SETTINGS show_table_uuid_in_table_create_query_if_not_nil=1; |
| 298 | +``` |
| 299 | + |
| 300 | +{{% alert title="Warning" color="warning" %}} |
| 301 | +Only reuse a UUID this way if you're certain that replica's ZK registration was never dropped and the surviving disk truly belongs to this table |
| 302 | +{{% /alert %}} |
| 303 | + |
| 304 | +Parts still present under that UUID on the surviving disk load in place (checksum-verified, no network transfer); only the parts that lived on the lost disk get fetched from the healthy replica. |
| 305 | + |
| 306 | +Two things this depends on: |
| 307 | + |
| 308 | + |
| 309 | +- **`ATTACH` only reconnects, it never creates ZK bookkeeping.** If the replica's ZK path was already dropped (or never existed, a genuinely new node), the same `ATTACH` command leaves the table permanently readonly (`No metadata in ZooKeeper for .../replicas/<name>: table will stay in readonly mode`) no error, no self-healing. Use `CREATE TABLE` for a node that never held this replica before; that path does create the ZK registration. |
| 310 | +- **Rebuilt/fetched parts land on the first storage-policy volume**, not wherever they originally lived, the storage policy itself has no memory of prior tiering, and `move_factor`-based moves only trigger under disk-space pressure, not on rebuild. If the table has a TTL `TO VOLUME`/`TO DISK` move rule, it re-sorts these parts back to cold automatically in the background (no `MOVE PARTITION` needed) once TTL is next checked. Without a TTL rule, you have to move them manually. |
0 commit comments