Add design doc for declared schemas on materialized views

[antiguru]

Motivation

This design document outlines a solution to a long-standing problem where materialized view schemas drift across optimizer versions. Currently, schemas are re-derived from the query on every boot, making them dependent on optimizer behavior for nullability and key inference. This causes cascading issues through dependent views, sinks, and replacements.

This design is a prerequisite for the replacement materialized views feature and enables users to have schema contracts on materialized views.

Description

The document proposes extending CREATE MATERIALIZED VIEW syntax to accept optional column definitions with types, nullability, and key constraints, similar to CREATE TABLE:

CREATE MATERIALIZED VIEW order_totals (
    customer_id int8 NOT NULL,
    region text,
    total numeric(38, 2) NOT NULL,
    PRIMARY KEY (customer_id)
) IN CLUSTER c AS
    SELECT customer_id, any_value(region), sum(amount)
    FROM orders GROUP BY customer_id;

Key design decisions:

• Declared schemas are optimizer-independent: The RelationDesc comes directly from the column definitions, not from inference.
• Types are cast with assignment semantics: Query output is cast to declared types using the same rules as INSERT INTO.
• Nullability is explicit: NOT NULL columns are added to non-null assertions checked per row, matching ASSERT NOT NULL behavior.
• Keys must be provable: Declared PRIMARY KEY and UNIQUE constraints are validated at creation time against keys the optimizer can infer from the query. This prevents silent correctness issues downstream.
• Replacements inherit schemas: A replacement without column definitions for a target with a declared schema inherits that schema, enabling drop-in replacements.
• Backward compatible: Materialized views without column definitions keep current behavior (inferred schema).

The document includes implementation sketches for parser, planner, optimizer, catalog, and sequencer changes, all within the SQL layer. No changes to the durable catalog format, persist, or compute protocol are needed.

A Phase 2 follow-up could freeze inferred schemas at creation time for all materialized views, removing the implicit schema entirely.

Verification

This is a design document only. No code changes or tests are included. The document is self-contained and ready for review and discussion before implementation begins.

https://claude.ai/code/session_01NrWZDcjDrqsHdkVGSrhuUj

[frankmcsherry]

Some thoughts. Like the idea generally, but would like more eyes on it!

[frankmcsherry]

Feels like a moment to potentially put nullability and keys in a second tier of "type", as long as these are established through semantic rather than syntactic analysis. Don't have to, but if we lock them in there's the problem of the optimizer "regressing" and failing to uphold those properties, or at least failing to infer them completely, leaving us confused about whether there is a violation or not (e.g. fix to query plan, or miss in query analysis).

[antiguru]

Added this framing in 3b7b80f: names and scalar types are structural (the query is cast to them), nullability and keys are a second tier of semantic properties. To avoid the ambiguity you describe, analysis never decides after creation whether a declared property holds. NOT NULL is checked against the data at runtime, and keys are the user's assertion, not compared against later optimizers' analysis.

Leaving the second tier inferred entirely is now an alternative in the doc. The catch is that nullability and keys are part of the RelationDesc persist stores as the shard schema, so leaving them inferred keeps the drift this sets out to fix, unless we also separate them from the persist schema identity.

---

Generated by Claude Code

[frankmcsherry]

This statement seems false, in that it certainly is a property of the optimizer version. I agree with the text "a key it proved once is taken to be a property of the query and its inputs' schemas", but worth being clear. I don't believe we are as certain of the key inference analysis as the text hopes.

[antiguru]

Agreed, reworded to "taken to be a property of the query and its inputs' schemas", and the doc now says the inference is an analysis of the optimizer version that runs it and can be wrong (3b7b80f).

---

Generated by Claude Code

[frankmcsherry]

The language in the paragraph above using concepts like "proof" and "sound" are .. optimistic at best. I recommend we aim for language that admits the possibility of error, and frames it that way. For example, this matches the "unenforced" language SQL uses elsewhere. We could choose to filter nulls into a DLQ for example, making it "enforced" but .. still a worry. The primary keys are unenforced and unvalidated, both.

[antiguru]

Agreed. Keys are now described as unenforced and unvalidated against the data. The creation-time check is a guard against obvious mistakes, not a proof, and the soft re-check on boot is gone, since a miss there is exactly the violation-or-analysis-gap ambiguity. NOT NULL stays enforced (an error today), and erroring vs. a dead-letter queue is an open question now (3b7b80f).

---

Generated by Claude Code

[antiguru]

Updated the design in 2e957bb after looking harder at whether a key confirmed in version i still holds in version i'.

It does not, even when key inference is correct in both versions. Inference is relative to the query's semantics and the inputs' keys in the version that runs it, and both change across upgrades: uniqueness-preserving functions (text_to_name/varchar(n) in #36653, "char"/bytea in #36663, array lower bounds in #37393), planner fixes such as #37506, and builtin keys such as mz_dataflow_global_ids in #36135. A frozen key would also outlive fixes to the analysis that confirmed it, which today's re-inferred keys pick up on the next upgrade. Nullability does not have this problem, because a declared NOT NULL is checked on every row at runtime.

The design now freezes only names, scalar types and nullability. Declared keys are expected keys that every plan checks against the running version's inferred keys. Creation still fails on a key the analysis does not confirm. On boot or EXPLAIN REPLAN, an unconfirmed key is dropped from the RelationDesc instead of trusted, the materialized view keeps running, and the key is recorded in a new mz_internal.mz_materialized_view_unconfirmed_keys relation so the user learns about it. The system does not try to tell a false key from weaker analysis, which is undecidable in general; it only refuses to trust what the running version cannot confirm. Frozen keys moved to Alternatives, and upsert sinks keyed on a dropped key and recording a key's derivation are new open questions.

---

Generated by Claude Code

[mgree]

The core plan makes sense to me. Would be good to think through how this touches durable LIR, but we have a moment to figure it out.

[mgree]

If we're failing on bad keys, we might want to fail (or at least warn) on fallible casts. If I write a query that can produce int8s and I cast to int2... it seems very likely that I will have a bad time (or we've somehow overwidened a computation).

[antiguru]

Agreed. The design now uses implicit casts only (e10f6f8), so int8 to int2, float8 to numeric, and numeric to numeric(p, s) have to be written in the query. Two implicit casts can still fail or round, text to varchar(n)/char(n) (length) and int8/numeric to float8, and the doc says so.

Posted by Claude Code.

[mgree]

This seems like we can end up with some weird round-trip casts: the first schema definition says int2 (which induces a cast from int8), but then the user realizes that they actually need int8... which we'll cast up from int2. Is that what we want?

(At a certain point, you need to just blue/green the thing, but I don't know exactly where that point is.)

[antiguru]

With implicit-only casts, a replacement whose query produces int8 for an inherited int2 column now fails to plan instead of narrowing. Going from int2 to int8 changes the declared type, which is a schema change. Replacements must match the target schema, so that case is out of scope and is where blue/green applies.

Posted by Claude Code.

[mgree]

Why make them name the schema at all, if there's only one thing they can name?

[antiguru]

The explicit definitions are the opt-in. If purification filled them in from an implicit target, every replacement of an implicit MV without a column list would silently get a declared schema. That would turn inferred NOT NULL into assertions and inferred keys into expected keys, which is the Phase 2 decision the doc defers. To avoid retyping, EXPLAIN ... WITH (schema) on the target prints the definitions to paste. Added both points to the doc.

Posted by Claude Code.

[mgree]

This isn't the behavior described above... don't replacements without a schema inherit the schema?

My understanding of that section was:

1. if new has no schema (column list or no), inherit any pre-existing schema
2. if new has a schema, either (a) it must be castable from the pre-existing schema or (b) there must not be a pre-existing schema.

Or am I misunderstanding? (I don't know exactly what "purification" is...)

[antiguru]

You read the replacements section correctly. The sequencer step was stale, and I've fixed it. Purification is the step before planning that rewrites the statement using catalog state. It copies the target's column definitions into a replacement that has no column list, so the stored create_sql spells out the schema. The sequencer then validates the result against the target. Its rejection of a replacement without a schema for a target with one is only a backstop for statements that skipped purification. One correction to (2): for an implicit target, the declared schema must equal the target's current desc. Castable is not enough.

Posted by Claude Code.

[mgree]

If we want this kind of enforcement, well... well, we can expect that there will likely be an index on the fields that are keys, so perhaps we can check assertions when building those arrangements (rather than building a custom arrangement just for the assertion)?

[antiguru]

Good idea, added under runtime-enforced keys. The catch is that an index runs in its own dataflow, possibly on another cluster. Dependents that read the MV from persist rely on the key whether or not that index exists. So an index-based check could report a violation but not prevent wrong results downstream.

Posted by Claude Code.

[mgree]

I might be in favor of implicit-only---if we're introducing casts with lots of computational content/failure modes, then the user should be opting in more explicitly than writing a type declaration elsewhere.

[antiguru]

Adopted implicit-only. Assignment casts moved to Alternatives. Switching to them later would accept strictly more statements, so we can relax this without breaking existing ones.

Posted by Claude Code.

[mgree]

Again, my gut says: allow implicit casts, everything else should be made explicit.

[antiguru]

Done, see the reply above. The remaining open question is narrower: whether inherited schemas should also reject widening casts.

Posted by Claude Code.

[mgree]

Performance already depends on the optimizer? We should skip it if we can.

[antiguru]

I'd keep the assertion. Skipping it makes correctness depend on nullability inference, not just performance. A null that reaches a column persist thinks is non-nullable panics the encoder (DatumEncoder::push) instead of raising a MustNotBeNull error. That is exactly the risk implicit MVs carry today. The check only looks at the asserted datums of rows the sink encodes anyway, though I have not measured its cost. Moved this to Alternatives with that reasoning.

Posted by Claude Code.

[mgree]

Reflecting this somehow might be good for tracking---ours (how many people are using this?) and users (how far are we converting our MVs to schemas).

[antiguru]

Agreed, and the doc now records both uses. The proposal is an mz_internal relation while the feature is behind the flag, and a column on mz_materialized_views once it is stable. The prototype doesn't do this yet.

Posted by Claude Code.

[antiguru]

@mgree thanks. e10f6f8 addresses the review:

• Casts to declared types are implicit-only. Narrowing casts and numeric rescaling must be written in the query, and assignment casts are now an alternative.
• Fixed the stale sequencer step that contradicted schema inheritance.
• Added an "Interaction with pinned LIR" section. Declared names, types and nullability give a pinned plan a fixed output contract. Whether a pinned plan should also pin its confirmed keys is left to the pinned LIR design.
• Documented why the NOT NULL assertion is never skipped, and how index arrangements could check keys.
• Made catalog exposure of declared schemas an open question about the form only. Whether to expose it is settled: yes.

Posted by Claude Code.

[antiguru]

On handling a mismatch after an upgrade, where the new optimizer derives a different type than the declared or stored schema: rendering has to continue and objects can't be dropped. A notice is easy to miss. Pinning LIR avoids the question, and it would also keep correctness fixes such as #39779 from reaching existing objects.

An alternative is to freeze the interface, re-optimize the implementation, and adapt the new plan to the stored desc at the boundary. Then no mismatch fails at boot.

• Narrower nullability (stored nullable, new NOT NULL, the expr: Correct over-conservative properties of record_get and map_build_from_record_list #39814 map_build case): keep the stored type, nothing to do.
• Wider nullability (stored NOT NULL, new nullable): add a non-null assertion. It fires only if the old binary claimed NOT NULL for data that held NULLs, which is already a correctness bug, and then it produces per-row errors in the error collection rather than failing to render.
• Keys: don't freeze them for the optimizer, because a false key gives wrong results and losing one only costs optimizations. For upsert sink KEYs, treat a key that no longer validates as NOT ENFORCED on re-plan, as the doc proposes.
• Scalar types: assignment cast to the stored type, with failing rows as errors. Anonymous record types remain the exception.

One gap: the Kafka sink ignores its input's error collection (_err_collection in src/storage/src/sink/kafka.rs). An assertion that fires in an MV would show up on SELECT, but rows would silently go missing from a Kafka sink downstream. For sinks, the check would need to live in the sink and stall it.

The same boundary applies to sinks on undeclared MVs and views, which this design doesn't cover. SQL-805 tracks that.

Posted by Claude Code.

[antiguru]

A correction to my comment above, after a Slack discussion. Frank pointed out that keys and nullability are semantic properties: they are beliefs the optimizer holds and that can change between versions. Column types are syntactic and can be a hard contract. My comment proposed freezing inferred nullability into the stored schema, which turns a belief into a contract. That is the same mistake behind the incidents.

• Tables and sources (SQL-759): persist matches the full RelationDesc, keys included, so a changed belief crashes boot.
• map_build (expr: Correct over-conservative properties of record_get and map_build_from_record_list #39814): a sink derived Avro nullability from inference.
• Upsert sinks (expr: Don't mark lossy time and timestamp casts uniqueness-preserving #39779): the declared KEY was re-validated against inferred keys.

Reframed, the stored schema holds only declared or syntactic information. Inferred keys and nullability are per-version beliefs that the optimizer uses but nothing downstream treats as a contract. Only declared contracts are enforced:

• A declared NOT NULL becomes a non-null assertion.
• A declared sink KEY becomes NOT ENFORCED when the optimizer can no longer confirm it.
• External schemas such as Avro default to nullable unless a column is declared NOT NULL.

With nothing inferred written down, an upgrade has nothing to mismatch. The open part is still runtime enforcement on sinks that ignore their error collection, such as Kafka.

Posted by Claude Code.