fix: a nullable UNIQUE key stalls the entire replication stream

[minguyen9988]

[minguyen9988]

Summary

A MySQL table with a UNIQUE key but no PRIMARY KEY whose key column is nullable produces a CREATE TABLE that ClickHouse rejects with Code: 44 ILLEGAL_COLUMN. Because DDL is retried indefinitely, that failure stalls the entire replication stream — not just the offending table.

Root cause

#1394 added allow_nullable_key=1 gated on isKeylessTable — the all-columns fallback. But #1389/#1394 also derive a sorting key from the UNIQUE key when no PRIMARY KEY exists, and MySQL permits any number of NULLs in a UNIQUE index. That path emitted ORDER BY (a) with no SETTINGS allow_nullable_key=1.

A PRIMARY KEY never needs the setting (MySQL forces those columns NOT NULL), so the fix keys on whether the sorting key was derived by the parser rather than on isKeylessTable.

Measured — before

MySQL 8.0.36 ROW/FULL/GTID -> ClickHouse 24.8.14.10547:

table          mysql   ch   note
n_uk               5    1   nullable UNIQUE, no PK -- CREATE failed
u_notnull          5    0   created before the stall, never populated
u_nullable         5    -   table never created
u_composite        3    -   table never created
u_pk_control       3    -   table never created  <- ordinary PK table

30 retries later the stream had made no further progress. The last three tables have no nullable key at all — they were lost purely because they sat behind the failing DDL.

Measured — after

table          mysql   ch   sorting_key    result
f_uk               4    4   a              PASS  (nullable UNIQUE, no PK)
f_uk_notnull       5    5   a              PASS
f_uk_comp          3    3   a, b           PASS  (composite, nullable member)
f_pk               2    2   a              PASS  (no allow_nullable_key emitted)
f_keyless          2    2   a, b           PASS
f_alembic          1    1   version_num    PASS
f_behind           2    2   id             PASS  (stream not stalled)

Code: 44 errors: 0

Content verified, not just counts: f_uk survives an UPDATE and a DELETE as 1:a,2:bX,3:c,4:d, matching MySQL exactly.

Why existing tests missed it

Every UNIQUE-key case in CreateTableUniqueKeySortKeyTest declares its column NOT NULL. The DDL is only rejected once a nullable column reaches the key. Three tests added: testNullableUniqueKeyEmitsAllowNullableKey, testPartiallyNullableCompositeUniqueKeyEmitsAllowNullableKey, and testPrimaryKeyDoesNotEmitAllowNullableKey (guards the other direction).

Failing-first proven: on the unmodified parent 33ff7077, the suite runs 8 / 2 failures — precisely the two new nullable cases. With the fix, 8 / 0.

Test results

• CreateTableUniqueKeySortKeyTest — 8 run / 0 failures
• CreateTableNoKeySortKeyTest — 7 run / 0 failures
• MySqlDDLParserListenerImplTest#testAutoCreateTable fails identically on the unmodified parent 33ff7077 — pre-existing, unrelated.

Rest of the image validated clean

Everything else in the published image passed end-to-end: BIT b8/b16/b24/b64 exact vs MySQL HEX(), Decimal(30,15) exact, bulk multi-row UPDATE/DELETE (#1391), pre-ALTER ADD COLUMN NULL-fill (#1389), the rename drain (#1395 — 90 rows, 0 corruption, INT->BIGINT preserving 2^63-1).

[minguyen9988]

Correction — second commit added (afbef19c)

My first commit fixed the Code: 44 stall by emitting allow_nullable_key for a nullable UNIQUE-key sorting key. That made the CREATE TABLE succeed but traded a loud stall for silent data loss, and an adversarial review caught it. Verified live before acting on it:

table            mysql   ch   contents
z_nulluk             4    2   ch has keyed,third -- 'first','second' LOST
z_nulluk_comp        3    2   partially-nullable composite, same loss

MySQL does not treat NULLs as equal for uniqueness, so a nullable UNIQUE index permits any number of NULL-keyed rows — it is not a row identity. ClickHouse does compare NULLs as equal in a sorting key, so adopting such a key collapses those distinct source rows.

So a UNIQUE key is now adopted only when every one of its columns is NOT NULL. Otherwise the table has no usable declared identity and falls through to the all-columns fallback, which matches MySQL's own semantics: rows are distinguished by value. A warning names the table and the reason.

The allow_nullable_key gate is now keyed on whether the emitted key can actually be nullable — only the all-columns fallback can. A PRIMARY KEY is NOT NULL by MySQL's rule, and a UNIQUE key is now adopted only when NOT NULL, so neither gets the setting. testNotNullCompositeUniqueKeyRemainsSortKey guards that, and it caught the over-correction.

Final end-to-end, fresh stack, fixed jar

table            mysql   ch   sorting_key    result
q_nulluk             3    3   a, v           PASS  (nullable UNIQUE)
q_nulluk_comp        3    3   a, b, v        PASS  (partially nullable)
q_nnuk               4    4   a              PASS  (NOT NULL UNIQUE kept)
q_nnuk_comp          3    3   a, b           PASS
q_pk                 2    2   a              PASS
q_keyless            2    2   a, v           PASS
q_alembic            1    1   version_num    PASS
q_behind             2    2   id             PASS  (stream not stalled)

Code: 44 errors: 0

Content verified, not just counts: q_nulluk holds first,keyed,second_upd on both sides after an UPDATE and a DELETE that each targeted a NULL-keyed row. The three NULL-keyed rows are preserved and individually addressable — the property the first commit lost.

Tests: CreateTableUniqueKeySortKeyTest 9 run / 0 failures, CreateTableNoKeySortKeyTest 7 run / 0 failures.