From e21396aa63c8efc9ed05add888212ad9efc7e964 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 6 Oct 2026 18:14:18 +0000 Subject: [PATCH 1/7] design: declared schemas for materialized views Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01NrWZDcjDrqsHdkVGSrhuUj --- ...61006_materialized_view_declared_schema.md | 256 ++++++++++++++++++ 1 file changed, 256 insertions(+) create mode 100644 doc/developer/design/20261006_materialized_view_declared_schema.md diff --git a/doc/developer/design/20261006_materialized_view_declared_schema.md b/doc/developer/design/20261006_materialized_view_declared_schema.md new file mode 100644 index 0000000000000..8e2c579808233 --- /dev/null +++ b/doc/developer/design/20261006_materialized_view_declared_schema.md @@ -0,0 +1,256 @@ +# Declared schemas for materialized views + +Associated: +- [Replacement materialized views](./20251111_replacement_materialized_views.md) + +## The Problem + +A materialized view's schema is not stored anywhere. +The catalog stores only the `create_sql`, and every time it is planned (creation, every boot, every replacement) we re-derive the `RelationDesc` from the query: + +* Column names and scalar types come from the HIR type of the query (`HirRelationExpr::top_level_typ`). +* Nullability and keys are back-ported from the *locally optimized* MIR (`infer_sql_type_for_catalog` in `src/adapter/src/coord.rs`, called from `CatalogState::parse_item` in `src/adapter/src/catalog/state.rs` and from `optimize::materialized_view::Optimizer`). +* `ASSERT NOT NULL` columns are forced to non-nullable on top of that. + +The schema is therefore a function of the SQL text, the schemas of the inputs, *and the optimizer version*. +Any change to nullability or key inference in `mz_transform` changes the schema of existing materialized views on the next upgrade. +The code already contains several workarounds for this: + +* `StorageController::evolve_nullability_for_bootstrap` re-registers each materialized view's shard schema on boot because "across versions of Materialize the nullability of columns can change based on updates to our optimizer". +* The persist columnar encoder marks every Arrow field nullable for the same reason (`RowColumnarEncoder::finish` in `src/repr/src/row/encode.rs`). + +The drift leaks into everything that consumes the schema: + +* Dependent views and materialized views are planned against the materialized view's `RelationDesc`, so drift cascades through the dependency graph. +* Sinks derive their output schemas from the same `RelationDesc` on every boot. + A nullability change alters the schema we publish downstream. +* Upsert sink `KEY (...)` validation in `plan_create_sink` checks the requested key against the *inferred* keys of the materialized view. + That check runs whenever the sink's `create_sql` is re-planned, so an optimizer that stops inferring a key can make an existing sink fail to plan. +* Replacement materialized views require the replacement's `RelationDesc` to equal the target's (`RelationDesc::diff` in the `CreateMaterializedView` sequencer), including nullability and keys. + A replacement whose query is semantically compatible but whose inferred nullability or keys differ is rejected, and the user has no way to state which schema they mean. +* Tooling that wants schema contracts (e.g. the dbt adapter's model contracts) has to emulate them with `ASSERT NOT NULL` and a post-hoc column comparison. + +## Success Criteria + +* A user can state the full schema of a materialized view: column names, types, nullability, and keys. +* A declared schema is independent of the optimizer. + Re-planning the same `create_sql` with a different optimizer yields the same `RelationDesc`. +* Declared constraints are never trusted without justification. + Results stay correct if a declared constraint does not hold for the data. +* Replacements are validated against the declared schema, and can be written to fit a given schema. +* No change to the durable catalog format, persist, or the compute protocol. + +## Out of Scope + +* Schema evolution (`ALTER MATERIALIZED VIEW ... ADD COLUMN`, or replacements with a superset schema). + A declared schema is the prerequisite for it, but it is a separate design. +* Runtime enforcement of uniqueness. +* `DEFAULT`, `CHECK`, `FOREIGN KEY`, and `COLLATE` on materialized view columns. +* Declared schemas for non-materialized views and indexes. + +## Solution Proposal + +### Syntax + +Extend the existing optional column list of `CREATE MATERIALIZED VIEW` to accept column definitions and table constraints, using the grammar `CREATE TABLE` already parses: + +```sql +CREATE [OR REPLACE] [REPLACEMENT] MATERIALIZED VIEW [IF NOT EXISTS] name + [ ( column_name [, ...] ) + | ( { column_name data_type [ NULL | NOT NULL | PRIMARY KEY | UNIQUE ] ... + | [ CONSTRAINT name ] { PRIMARY KEY | UNIQUE } ( column_name [, ...] ) + } [, ...] ) + ] + [FOR target] + [IN CLUSTER cluster] + [WITH (...)] + AS query; +``` + +For example: + +```sql +CREATE MATERIALIZED VIEW order_totals ( + customer_id int8 NOT NULL, + region text, + total numeric(38, 2) NOT NULL, + PRIMARY KEY (customer_id) +) IN CLUSTER c AS + SELECT customer_id, any_value(region), sum(amount) + FROM orders GROUP BY customer_id; +``` + +The bare-name form keeps its current meaning (rename only, schema inferred). +The two forms are mutually exclusive: if any column has a type, every column must have one. +The parser distinguishes them with two tokens of lookahead after `(`: an identifier followed by `,` or `)` starts the bare-name form, anything else starts the definition form. + +We keep the object type `MATERIALIZED VIEW` and do not introduce `CREATE MATERIALIZED TABLE`, see [Alternatives](#create-materialized-table). + +### Semantics + +A materialized view with column definitions has a *declared schema*. +Its `RelationDesc` is exactly the declared one: names, scalar types, nullability, and keys come from the column definitions, and nothing inferred by the optimizer is merged in. +Without column definitions, behavior is unchanged. + +**Arity and names.** +The number of declared columns must equal the query's arity. +Columns map by position, and declared names replace the query's names, as the bare-name form does today. + +**Types.** +The query output is cast to the declared types with assignment casts (`CastContext::Assignment`), the same rule `INSERT INTO t SELECT ...` applies. +The cast is planned into the HIR expression with the existing `query::cast_relation`, so it is part of the dataflow and appears in `EXPLAIN`. +A column that cannot be assignment-cast is a planning error naming the column and both types. +A cast that fails at runtime (e.g. overflow on `int8` to `int2`) produces an error in the materialized view, like any other evaluation error. + +**Nullability.** +A column without `NOT NULL` is nullable, as in `CREATE TABLE`. +This is deliberately not "whatever the optimizer infers", since that would reintroduce the dependency we are removing. +Every `NOT NULL` column (including `PRIMARY KEY` columns) is added to the existing `non_null_assertions`, which the compute sink checks per row and turns into a `MustNotBeNull` error (`src/compute/src/render/sinks.rs`). +The assertion is required for correctness, not just for error reporting. +The persist encoder panics on a null in a non-nullable column (`DatumEncoder::push`), so a declared `NOT NULL` must never reach it unchecked. +Declaring `NOT NULL` is therefore the column-definition spelling of `ASSERT NOT NULL`. +Using `WITH (ASSERT NOT NULL ...)` together with column definitions is rejected to avoid two ways to say the same thing in one statement. + +**Keys.** +Keys in a `RelationDesc` are consumed by the optimizer of every dependent query (e.g. to elide `Distinct` and `Reduce`). +An incorrect key produces wrong results, which is why `CREATE TABLE` keys are gated behind `unsafe_enable_table_keys`. +We therefore do not trust declared keys blindly. +At creation, each declared `PRIMARY KEY` or `UNIQUE` key must be a superset of some key the optimizer infers for the (cast) query, otherwise creation fails with an error listing the keys we could prove. +This mirrors the existing upsert sink `KEY` validation. + +The check runs only when the statement is first executed (in the sequencer, next to the replacement schema check), not when `create_sql` is re-planned on boot. +This is sound: MIR key inference is conservative, so a key it proved once is a property of the query and its inputs' schemas, not of the optimizer version. +A later optimizer that fails to re-derive the key does not make the key false. +The inputs' keys are themselves either declared (and proven), inferred (and sound), or source keys we already trust today. + +A `UNIQUE` constraint on nullable columns without `NULLS NOT DISTINCT` is an error. +(`plan_create_table` treats it as "not a key" instead, which would make a declared key silently disappear.) +`PRIMARY KEY` implies `NOT NULL` on its columns. + +Note that assignment casts can drop keys from the inferred set when the cast function does not report `preserves_uniqueness`. +The user then sees the key-proof error and can adjust the query or the declared type. + +**Types that cannot be spelled.** +The type grammar has no anonymous record type (`ResolvedDataType` has no such variant). +A query producing an anonymous record column can still get a declared schema by declaring a named composite type with `CREATE TYPE ... AS (...)`, since record-to-record casts are implicit. +The named type becomes a dependency of the materialized view, as with tables. + +### Replacements + +With a declared schema, a replacement is validated against the target's declared `RelationDesc` with the same `RelationDesc::diff` check as today. +The difference is that both sides are now under the user's control, so a mismatch in nullability or keys is something the user wrote, not something the optimizer decided. + +A replacement without column definitions for a target *with* a declared schema inherits the target's schema. +To keep accidental casts out of what is meant to be a drop-in replacement, inheritance requires the query's scalar types to match the target's exactly (no casts). +Nullability and keys are handled as if declared: `NOT NULL` becomes an assertion and keys must be provable. +`MaterializedView::apply_replacement` already takes the column list from the replacement's statement, it has to carry the inherited definitions over so the target's `create_sql` remains self-describing. + +A replacement *with* column definitions for a target *without* one is allowed if the declared desc equals the target's current desc. +This is the migration path from an implicit to a declared schema for an existing materialized view. + +### Implementation sketch + +1. **AST and parser** (`mz-sql-parser`). + Replace `CreateMaterializedViewStatement::columns: Vec` with an enum, roughly `Names(Vec)` and `Definitions { columns: Vec>, constraints: Vec> }`. + Reuse `Parser::parse_columns` for the definition form. + Update `AstDisplay` and the parser datadriven tests. +2. **Planner** (`src/sql/src/plan/statement/ddl.rs`). + `plan_create_table` and `plan_source_export_desc` contain near-identical code turning `ColumnDef`s and `TableConstraint`s into a `SqlRelationType`. + Factor that into one helper that also reports which options it accepts, and call it from `plan_create_materialized_view`. + Apply `cast_relation` to the planned query, add `NOT NULL` columns to `non_null_assertions`, and add `declared_desc: Option` to `plan::MaterializedView`. + The type-name references resolve through the normal name resolution, so dependencies on custom types are recorded in `resolved_ids`. +3. **Optimizer** (`src/adapter/src/optimize/materialized_view.rs`). + Pass the declared desc to `Optimizer::new`. + In the global stage, use it as the sink's `from_desc` and `value_desc` when present, otherwise derive it as today. + Soft-assert that arity and scalar types agree with the local plan's type. +4. **Catalog** (`CatalogState::parse_item`). + Apply the same rule: `desc = declared_desc` if present, otherwise `infer_sql_type_for_catalog` plus assertions. + This is the single point that makes the schema optimizer-independent on boot. +5. **Sequencer** (`src/adapter/src/coord/sequencer/inner/create_materialized_view.rs`). + Add the key proof next to the replacement check, using the keys of the local MIR plan's type. + Implement schema inheritance for replacements there, since it needs the target's desc. +6. **Feature flag** `enable_materialized_view_column_definitions`, off in production and on in the test and CI configuration. +7. **Docs** for `CREATE MATERIALIZED VIEW`, and a note in the dbt adapter that contracts can be expressed directly. + +Nothing changes in the durable catalog format (the schema lives in `create_sql`), persist, or compute. +`evolve_nullability_for_bootstrap` becomes a no-op for materialized views with declared schemas, because their desc no longer changes between versions. + +### Feasibility + +The change is small to medium and mostly in the SQL layer. +Every runtime mechanism it needs already exists: assignment casts for `INSERT`, column-definition parsing for `CREATE TABLE`, non-null assertions for `ASSERT NOT NULL`, key validation for upsert sinks, and desc comparison for replacements. +The main risks are: + +* **Parser ambiguity** between the two column-list forms. + Two-token lookahead resolves it, and roundtrip tests cover it. +* **Cast semantics surprising users**, e.g. assignment casts to `varchar(n)` or `numeric(p, s)`. + This is the same behavior as `INSERT`, see open questions. +* **Key proofs failing for reasonable queries** because MIR key inference is incomplete (e.g. through casts or `UNION ALL` of disjoint inputs). + Users can drop the key from the declaration. Runtime enforcement could be added later for those cases. + +### Phase 2: freezing inferred schemas + +Declared schemas fix the problem only for materialized views that opt in. +Two follow-ups would remove the implicit schema entirely: + +* **Freeze at creation.** + When a materialized view is created without column definitions, write the inferred schema into `create_sql`, the same way the selected `AS OF` is written back today. + Inferred keys were proven by the optimizer at creation, so they satisfy the key rule. + Inferred `NOT NULL` becomes a runtime assertion, which also turns a nullability-inference bug from a persist encoder panic into a query error. +* **Migrate existing materialized views.** + Rewrite their `create_sql` with the schema their shard currently has. + Persist stores the complete `RelationDesc`, including nullability and keys, as the shard's schema (`encode_schema` for `SourceData` in `src/storage-types/src/sources.rs`), so the migration can use the last schema the system committed to rather than the new optimizer's inference. + Materialized views with columns of anonymous record type cannot be migrated and would keep an implicit schema. + +Both change what `SHOW CREATE MATERIALIZED VIEW` prints for users who never asked for it, so they deserve their own decision. + +## Minimal Viable Prototype + +* Parser, planner, optimizer and catalog changes for names, types, `NULL`/`NOT NULL`, behind the feature flag. +* Key declarations with the creation-time proof. +* Replacement validation against declared schemas and schema inheritance. +* Tests: + * sqllogictest for parsing, casts, error messages, `SHOW CREATE`, key proof failures, and the `MustNotBeNull` runtime error. + * testdrive for an upsert sink keyed on a declared primary key. + * A platform check that a materialized view with a declared schema survives upgrade and restart with an unchanged `RelationDesc`. + +## Alternatives + +### `CREATE MATERIALIZED TABLE` + +A new statement would signal "this object has a fixed schema", but: + +* It would be a new object type, or an alias for one, with the cost the replacement design already rejected for replacements as first-class items: duplicated planning and sequencing, and monitoring (`mz_materialized_views`, `EXPLAIN`, replacements) that does not apply out of the box. +* "Table" suggests the object accepts writes (`INSERT`, `UPDATE`), which it does not. +* The only difference to a materialized view is whether the schema is spelled out, and an optional column list expresses that without a new noun. + +### Persist the inferred desc separately in the durable catalog + +Freezing the inferred desc in a new catalog field fixes drift without any syntax. +It does not let users or replacements *state* the schema, and it creates a second source of truth next to `create_sql` that the two would have to agree with. +Phase 2 achieves the same freezing through `create_sql`. + +### Unenforced keys + +Accepting declared keys without proof (like `KEY (...) NOT ENFORCED` on sinks) is simpler, but the optimizer consumes `RelationDesc` keys, so a wrong declaration silently produces wrong results downstream. +Keys that are informational only would need a place in `RelationDesc` that the optimizer ignores, which does not exist today. + +### Runtime-enforced keys + +Checking uniqueness in the dataflow means maintaining a count per key, i.e. an arrangement the size of the materialized view. +It removes the incompleteness of the key proof but adds a large and surprising cost. +It can be added later as an opt-in for keys the optimizer cannot prove. + +### Schema as a `WITH` option + +`WITH (SCHEMA (...))` or similar would avoid touching the column list, but it splits naming (column list) from typing (option) and departs from the `CREATE TABLE` grammar users and tools already know. + +## Open questions + +* Assignment casts, or implicit casts only? + Implicit-only is stricter and avoids silent truncation-style surprises, but forces users to write casts in the query for common cases such as `numeric` scale. +* Should replacement schema inheritance allow casts after all? +* Should `NOT NULL` skip the runtime assertion when the optimizer proves non-nullability? + The assertion is a per-row datum scan in the sink, and skipping it would make performance, though not correctness, depend on the optimizer. +* Should `mz_materialized_views` expose whether the schema is declared? +* Do we want Phase 2, and if so, should freezing at creation be the default or opt-in? From 1ddfc6d5a949ddaa0450aeb2bc505a89bd205f4e Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 6 Oct 2026 18:34:39 +0000 Subject: [PATCH 2/7] design: note key syntax precedent Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01NrWZDcjDrqsHdkVGSrhuUj --- .../design/20261006_materialized_view_declared_schema.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/doc/developer/design/20261006_materialized_view_declared_schema.md b/doc/developer/design/20261006_materialized_view_declared_schema.md index 8e2c579808233..b20db367915b7 100644 --- a/doc/developer/design/20261006_materialized_view_declared_schema.md +++ b/doc/developer/design/20261006_materialized_view_declared_schema.md @@ -241,6 +241,14 @@ Checking uniqueness in the dataflow means maintaining a count per key, i.e. an a It removes the incompleteness of the key proof but adds a large and surprising cost. It can be added later as an opt-in for keys the optimizer cannot prove. +### Sink-style `KEY (...)` instead of `PRIMARY KEY` / `UNIQUE` + +Materialize has two precedents for declaring keys. +User-facing, `CREATE SINK ... KEY (...) [NOT ENFORCED]` validates a requested key against the inferred keys, which is the same check this design applies. +In column-definition lists, `PRIMARY KEY` and `UNIQUE` are what `CREATE TABLE` parses and what purification writes into the `create_sql` of tables created from Postgres, MySQL and SQL Server sources. +Since the declared schema is a column-definition list, we follow the second precedent and reuse its grammar and planning code. +A sink-style `KEY (...)` clause would also invite `NOT ENFORCED`, which we reject for materialized views (see [Unenforced keys](#unenforced-keys)). + ### Schema as a `WITH` option `WITH (SCHEMA (...))` or similar would avoid touching the column list, but it splits naming (column list) from typing (option) and departs from the `CREATE TABLE` grammar users and tools already know. From c794e9b55b68569ded9a27ed0d64037b6d72cf5a Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 6 Oct 2026 21:21:36 +0000 Subject: [PATCH 3/7] design: address review of declared materialized view schemas Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01NrWZDcjDrqsHdkVGSrhuUj --- ...61006_materialized_view_declared_schema.md | 65 ++++++++++++++----- 1 file changed, 48 insertions(+), 17 deletions(-) diff --git a/doc/developer/design/20261006_materialized_view_declared_schema.md b/doc/developer/design/20261006_materialized_view_declared_schema.md index b20db367915b7..1d01e1d59d8c1 100644 --- a/doc/developer/design/20261006_materialized_view_declared_schema.md +++ b/doc/developer/design/20261006_materialized_view_declared_schema.md @@ -16,8 +16,8 @@ The schema is therefore a function of the SQL text, the schemas of the inputs, * Any change to nullability or key inference in `mz_transform` changes the schema of existing materialized views on the next upgrade. The code already contains several workarounds for this: -* `StorageController::evolve_nullability_for_bootstrap` re-registers each materialized view's shard schema on boot because "across versions of Materialize the nullability of columns can change based on updates to our optimizer". -* The persist columnar encoder marks every Arrow field nullable for the same reason (`RowColumnarEncoder::finish` in `src/repr/src/row/encode.rs`). +* On boot, the coordinator calls `StorageController::evolve_nullability_for_bootstrap` to re-register each materialized view's shard schema, because "across versions of Materialize the nullability of columns can change based on updates to our optimizer" (the comment at the call site in `src/adapter/src/coord.rs`). +* The persist columnar encoder marks every Arrow field nullable for the same reason (`RowColumnarEncoder::finish` in `src/repr/src/row/encode.rs`, which cites [database-issues#2488](https://github.com/MaterializeInc/database-issues/issues/2488)). The drift leaks into everything that consumes the schema: @@ -92,6 +92,10 @@ A materialized view with column definitions has a *declared schema*. Its `RelationDesc` is exactly the declared one: names, scalar types, nullability, and keys come from the column definitions, and nothing inferred by the optimizer is merged in. Without column definitions, behavior is unchanged. +Opting in has a cost: a key or non-null fact the optimizer infers today but the user does not declare is gone. +Dependent queries lose the optimizations it enabled (e.g. `Distinct` or `Reduce` elision), and an upsert sink `KEY (...)` on an undeclared key fails to plan. +Users who want to keep them have to declare them. + **Arity and names.** The number of declared columns must equal the query's arity. Columns map by position, and declared names replace the query's names, as the bare-name form does today. @@ -109,7 +113,7 @@ Every `NOT NULL` column (including `PRIMARY KEY` columns) is added to the existi The assertion is required for correctness, not just for error reporting. The persist encoder panics on a null in a non-nullable column (`DatumEncoder::push`), so a declared `NOT NULL` must never reach it unchecked. Declaring `NOT NULL` is therefore the column-definition spelling of `ASSERT NOT NULL`. -Using `WITH (ASSERT NOT NULL ...)` together with column definitions is rejected to avoid two ways to say the same thing in one statement. +Using `WITH (ASSERT NOT NULL ...)` together with a declared schema, written out or inherited by a replacement, is rejected to avoid two ways to say the same thing in one statement. **Keys.** Keys in a `RelationDesc` are consumed by the optimizer of every dependent query (e.g. to elide `Distinct` and `Reduce`). @@ -118,14 +122,18 @@ We therefore do not trust declared keys blindly. At creation, each declared `PRIMARY KEY` or `UNIQUE` key must be a superset of some key the optimizer infers for the (cast) query, otherwise creation fails with an error listing the keys we could prove. This mirrors the existing upsert sink `KEY` validation. -The check runs only when the statement is first executed (in the sequencer, next to the replacement schema check), not when `create_sql` is re-planned on boot. -This is sound: MIR key inference is conservative, so a key it proved once is a property of the query and its inputs' schemas, not of the optimizer version. +The check runs when the statement is first optimized, for `CREATE` and `EXPLAIN CREATE` alike, but not when `create_sql` is re-planned on boot or by `EXPLAIN REPLAN`. +This relies on MIR key inference being correct in the version that runs the proof: a key it proved once is a property of the query and its inputs' schemas, not of the optimizer version. A later optimizer that fails to re-derive the key does not make the key false. The inputs' keys are themselves either declared (and proven), inferred (and sound), or source keys we already trust today. +The flip side is that a key-inference bug at creation time is frozen into `create_sql`, where today a fixed optimizer would stop inferring the bad key on the next upgrade. +Re-running the proof on boot as a soft check that only logs is a cheap way to detect that. A `UNIQUE` constraint on nullable columns without `NULLS NOT DISTINCT` is an error. -(`plan_create_table` treats it as "not a key" instead, which would make a declared key silently disappear.) -`PRIMARY KEY` implies `NOT NULL` on its columns. +This includes column-level `UNIQUE`, which has no `NULLS NOT DISTINCT` spelling and therefore requires `NOT NULL`. +(`plan_create_table` stops processing constraints at the first such one instead, which drops it and every later key, so a declared key would silently disappear.) +`PRIMARY KEY` implies `NOT NULL` on its columns, and declaring a primary key column `NULL` is an error. +`RelationDesc` stores key columns sorted, so the declared column order of a key is not preserved, as for tables. Note that assignment casts can drop keys from the inferred set when the cast function does not report `preserves_uniqueness`. The user then sees the key-proof error and can adjust the query or the declared type. @@ -140,10 +148,15 @@ The named type becomes a dependency of the materialized view, as with tables. With a declared schema, a replacement is validated against the target's declared `RelationDesc` with the same `RelationDesc::diff` check as today. The difference is that both sides are now under the user's control, so a mismatch in nullability or keys is something the user wrote, not something the optimizer decided. -A replacement without column definitions for a target *with* a declared schema inherits the target's schema. -To keep accidental casts out of what is meant to be a drop-in replacement, inheritance requires the query's scalar types to match the target's exactly (no casts). -Nullability and keys are handled as if declared: `NOT NULL` becomes an assertion and keys must be provable. -`MaterializedView::apply_replacement` already takes the column list from the replacement's statement, it has to carry the inherited definitions over so the target's `create_sql` remains self-describing. +A replacement without a column list for a target *with* a declared schema inherits the target's schema. +Purification copies the target's column definitions into the replacement's statement, so inheriting is shorthand for writing them out: the query is cast to them with assignment casts, `NOT NULL` becomes an assertion, keys must be provable, and the feature flag applies. +The replacement's `create_sql` then states its schema. +Re-planning it on boot does not depend on the target, custom types in the definitions are recorded as dependencies, and `MaterializedView::apply_replacement` keeps taking the column list from the replacement's statement. + +Requiring the query's types to match the target's exactly, without casts, would keep accidental casts out of a drop-in replacement. +It needs to know at planning time whether the definitions were inherited, which the stored statement cannot say, and checking exact types when re-planning would make an upgrade that changes a query's inferred type fail on boot. + +A replacement with a bare-name list for a target with a declared schema is rejected, because renaming conflicts with inheriting the names. A replacement *with* column definitions for a target *without* one is allowed if the declared desc equals the target's current desc. This is the migration path from an implicit to a declared schema for an existing materialized view. @@ -156,7 +169,8 @@ This is the migration path from an implicit to a declared schema for an existing Update `AstDisplay` and the parser datadriven tests. 2. **Planner** (`src/sql/src/plan/statement/ddl.rs`). `plan_create_table` and `plan_source_export_desc` contain near-identical code turning `ColumnDef`s and `TableConstraint`s into a `SqlRelationType`. - Factor that into one helper that also reports which options it accepts, and call it from `plan_create_materialized_view`. + Materialized views need different semantics in that code (nullable `UNIQUE` is an error rather than dropped, keys are not gated behind `unsafe_enable_table_keys`, and only `NULL`, `NOT NULL`, `PRIMARY KEY` and `UNIQUE` are accepted), so `plan_create_materialized_view` gets its own helper. + Sharing one helper with options is possible, but it would change `CREATE TABLE` behavior (the dropped-keys issue above), which is a separate decision. Apply `cast_relation` to the planned query, add `NOT NULL` columns to `non_null_assertions`, and add `declared_desc: Option` to `plan::MaterializedView`. The type-name references resolve through the normal name resolution, so dependencies on custom types are recorded in `resolved_ids`. 3. **Optimizer** (`src/adapter/src/optimize/materialized_view.rs`). @@ -167,13 +181,24 @@ This is the migration path from an implicit to a declared schema for an existing Apply the same rule: `desc = declared_desc` if present, otherwise `infer_sql_type_for_catalog` plus assertions. This is the single point that makes the schema optimizer-independent on boot. 5. **Sequencer** (`src/adapter/src/coord/sequencer/inner/create_materialized_view.rs`). - Add the key proof next to the replacement check, using the keys of the local MIR plan's type. - Implement schema inheritance for replacements there, since it needs the target's desc. + Run the key proof on the keys of the local MIR plan's type, in the optimize stage so that `EXPLAIN CREATE` rejects what `CREATE` rejects. + Reject a replacement without a declared schema for a target with one. + Schema inheritance for replacements lives in purification (`src/sql/src/pure.rs`), which can read the target from the catalog and add to the statement's resolved ids. 6. **Feature flag** `enable_materialized_view_column_definitions`, off in production and on in the test and CI configuration. -7. **Docs** for `CREATE MATERIALIZED VIEW`, and a note in the dbt adapter that contracts can be expressed directly. +7. **mz-deploy.** Its typecheck catalog uses the declared desc when present. + Unit tests run a materialized view as a temporary view, which cannot declare a schema, so a declared materialized view is lowered to casts to the declared types. + Declared `NOT NULL` and keys are not checked in unit tests. +8. **Docs** for `CREATE MATERIALIZED VIEW`, and a note in the dbt adapter that contracts can be expressed directly. Nothing changes in the durable catalog format (the schema lives in `create_sql`), persist, or compute. -`evolve_nullability_for_bootstrap` becomes a no-op for materialized views with declared schemas, because their desc no longer changes between versions. +`evolve_nullability_for_bootstrap` still runs for materialized views with declared schemas, but has no effect, because their desc no longer changes between versions. + +### `EXPLAIN ... WITH (schema)` + +To make declared schemas easy to write, `EXPLAIN` gets a `schema` option that prints the schema of a dataflow's exports as column definitions. +It applies to materialized views and subscribes, in the text format. +Keys over nullable columns print as `UNIQUE NULLS NOT DISTINCT`, and every key prints as `UNIQUE`, so the output can be pasted into a declaration but does not say which key was declared as `PRIMARY KEY`. +The option is not behind the feature flag, because it is useful for inspecting inferred schemas too. ### Feasibility @@ -186,6 +211,7 @@ The main risks are: * **Cast semantics surprising users**, e.g. assignment casts to `varchar(n)` or `numeric(p, s)`. This is the same behavior as `INSERT`, see open questions. * **Key proofs failing for reasonable queries** because MIR key inference is incomplete (e.g. through casts or `UNION ALL` of disjoint inputs). + In particular, `Map` carries a key over to a new column only if exactly one of its expressions preserves uniqueness, and `cast_relation` puts all casts into one `Map`, so declaring two columns with widening casts loses every key through them. Users can drop the key from the declaration. Runtime enforcement could be added later for those cases. ### Phase 2: freezing inferred schemas @@ -199,6 +225,7 @@ Two follow-ups would remove the implicit schema entirely: Inferred `NOT NULL` becomes a runtime assertion, which also turns a nullability-inference bug from a persist encoder panic into a query error. * **Migrate existing materialized views.** Rewrite their `create_sql` with the schema their shard currently has. + `evolve_nullability_for_bootstrap` overwrites the shard schema with the newly inferred desc on every boot, so the migration has to read the shard's latest schema before that call, in the same boot. Persist stores the complete `RelationDesc`, including nullability and keys, as the shard's schema (`encode_schema` for `SourceData` in `src/storage-types/src/sources.rs`), so the migration can use the last schema the system committed to rather than the new optimizer's inference. Materialized views with columns of anonymous record type cannot be migrated and would keep an implicit schema. @@ -257,8 +284,12 @@ A sink-style `KEY (...)` clause would also invite `NOT ENFORCED`, which we rejec * Assignment casts, or implicit casts only? Implicit-only is stricter and avoids silent truncation-style surprises, but forces users to write casts in the query for common cases such as `numeric` scale. -* Should replacement schema inheritance allow casts after all? + The prototype uses assignment casts. +* Should replacement schema inheritance reject casts? + The prototype allows them, see [Replacements](#replacements). * Should `NOT NULL` skip the runtime assertion when the optimizer proves non-nullability? The assertion is a per-row datum scan in the sink, and skipping it would make performance, though not correctness, depend on the optimizer. + The prototype never skips it. * Should `mz_materialized_views` expose whether the schema is declared? + The prototype does not. * Do we want Phase 2, and if so, should freezing at creation be the default or opt-in? From 3b7b80ffe9517cab805c07dca3a4bcf2f5f18ef1 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 7 Oct 2026 20:24:47 +0200 Subject: [PATCH 4/7] design: frame declared keys as unenforced Describe nullability and keys as a second tier of the declared schema: `NOT NULL` is enforced at runtime, keys are unenforced and unvalidated, and the creation-time key check is a guard against mistakes rather than a proof. Add keeping nullability and keys inferred as an alternative, and dead-letter routing of NOT NULL violations as an open question. Co-Authored-By: Claude Opus 5.5 --- ...61006_materialized_view_declared_schema.md | 56 ++++++++++++------- 1 file changed, 36 insertions(+), 20 deletions(-) diff --git a/doc/developer/design/20261006_materialized_view_declared_schema.md b/doc/developer/design/20261006_materialized_view_declared_schema.md index 1d01e1d59d8c1..647d114cade83 100644 --- a/doc/developer/design/20261006_materialized_view_declared_schema.md +++ b/doc/developer/design/20261006_materialized_view_declared_schema.md @@ -35,8 +35,9 @@ The drift leaks into everything that consumes the schema: * A user can state the full schema of a materialized view: column names, types, nullability, and keys. * A declared schema is independent of the optimizer. Re-planning the same `create_sql` with a different optimizer yields the same `RelationDesc`. -* Declared constraints are never trusted without justification. - Results stay correct if a declared constraint does not hold for the data. +* A declared `NOT NULL` is enforced: a violation surfaces as an error, independent of the optimizer. +* A declared key is an unenforced constraint, as in SQL systems that accept `NOT ENFORCED` keys. + Materialize rejects keys its analysis of the query does not support at creation, but it does not validate them against the data, so a wrong key gives wrong results. * Replacements are validated against the declared schema, and can be written to fit a given schema. * No change to the durable catalog format, persist, or the compute protocol. @@ -92,6 +93,12 @@ A materialized view with column definitions has a *declared schema*. Its `RelationDesc` is exactly the declared one: names, scalar types, nullability, and keys come from the column definitions, and nothing inferred by the optimizer is merged in. Without column definitions, behavior is unchanged. +The declared schema has two tiers. +Names and scalar types are structural: the query is cast to them, so they hold by construction. +Nullability and keys are semantic properties of the data, which the optimizer can only estimate by analysis, and that analysis can be incomplete or wrong in a given version. +Declaring them fixes what the system assumes, but analysis never decides after creation whether they hold: `NOT NULL` is checked against the data at runtime, and keys are the user's assertion. +This avoids the ambiguity of comparing a declared property against a later optimizer's analysis, where a disagreement could be a violation in the data or a miss in the analysis. + Opting in has a cost: a key or non-null fact the optimizer infers today but the user does not declare is gone. Dependent queries lose the optimizations it enabled (e.g. `Distinct` or `Reduce` elision), and an upsert sink `KEY (...)` on an undeclared key fails to plan. Users who want to keep them have to declare them. @@ -118,16 +125,15 @@ Using `WITH (ASSERT NOT NULL ...)` together with a declared schema, written out **Keys.** Keys in a `RelationDesc` are consumed by the optimizer of every dependent query (e.g. to elide `Distinct` and `Reduce`). An incorrect key produces wrong results, which is why `CREATE TABLE` keys are gated behind `unsafe_enable_table_keys`. -We therefore do not trust declared keys blindly. -At creation, each declared `PRIMARY KEY` or `UNIQUE` key must be a superset of some key the optimizer infers for the (cast) query, otherwise creation fails with an error listing the keys we could prove. +Declared keys are unenforced and unvalidated against the data, so they are a promise by the user. +To catch obvious mistakes, creation requires each declared `PRIMARY KEY` or `UNIQUE` key to contain some key the optimizer infers for the (cast) query, otherwise it fails with an error listing the inferred keys. This mirrors the existing upsert sink `KEY` validation. +The check is a guard, not a guarantee: MIR key inference is an analysis of the optimizer version that runs it, and it can be wrong. The check runs when the statement is first optimized, for `CREATE` and `EXPLAIN CREATE` alike, but not when `create_sql` is re-planned on boot or by `EXPLAIN REPLAN`. -This relies on MIR key inference being correct in the version that runs the proof: a key it proved once is a property of the query and its inputs' schemas, not of the optimizer version. -A later optimizer that fails to re-derive the key does not make the key false. -The inputs' keys are themselves either declared (and proven), inferred (and sound), or source keys we already trust today. +After creation, a key the check accepted is taken to be a property of the query and its inputs' schemas. +A later optimizer that fails to infer the key does not reject the materialized view, and it could not tell a wrong key from a gap in its own analysis anyway. The flip side is that a key-inference bug at creation time is frozen into `create_sql`, where today a fixed optimizer would stop inferring the bad key on the next upgrade. -Re-running the proof on boot as a soft check that only logs is a cheap way to detect that. A `UNIQUE` constraint on nullable columns without `NULLS NOT DISTINCT` is an error. This includes column-level `UNIQUE`, which has no `NULLS NOT DISTINCT` spelling and therefore requires `NOT NULL`. @@ -136,7 +142,7 @@ This includes column-level `UNIQUE`, which has no `NULLS NOT DISTINCT` spelling `RelationDesc` stores key columns sorted, so the declared column order of a key is not preserved, as for tables. Note that assignment casts can drop keys from the inferred set when the cast function does not report `preserves_uniqueness`. -The user then sees the key-proof error and can adjust the query or the declared type. +The user then sees the key check error and can adjust the query or the declared type. **Types that cannot be spelled.** The type grammar has no anonymous record type (`ResolvedDataType` has no such variant). @@ -149,7 +155,7 @@ With a declared schema, a replacement is validated against the target's declared The difference is that both sides are now under the user's control, so a mismatch in nullability or keys is something the user wrote, not something the optimizer decided. A replacement without a column list for a target *with* a declared schema inherits the target's schema. -Purification copies the target's column definitions into the replacement's statement, so inheriting is shorthand for writing them out: the query is cast to them with assignment casts, `NOT NULL` becomes an assertion, keys must be provable, and the feature flag applies. +Purification copies the target's column definitions into the replacement's statement, so inheriting is shorthand for writing them out: the query is cast to them with assignment casts, `NOT NULL` becomes an assertion, keys go through the creation-time check, and the feature flag applies. The replacement's `create_sql` then states its schema. Re-planning it on boot does not depend on the target, custom types in the definitions are recorded as dependencies, and `MaterializedView::apply_replacement` keeps taking the column list from the replacement's statement. @@ -181,7 +187,7 @@ This is the migration path from an implicit to a declared schema for an existing Apply the same rule: `desc = declared_desc` if present, otherwise `infer_sql_type_for_catalog` plus assertions. This is the single point that makes the schema optimizer-independent on boot. 5. **Sequencer** (`src/adapter/src/coord/sequencer/inner/create_materialized_view.rs`). - Run the key proof on the keys of the local MIR plan's type, in the optimize stage so that `EXPLAIN CREATE` rejects what `CREATE` rejects. + Run the key check against the keys of the local MIR plan's type, in the optimize stage so that `EXPLAIN CREATE` rejects what `CREATE` rejects. Reject a replacement without a declared schema for a target with one. Schema inheritance for replacements lives in purification (`src/sql/src/pure.rs`), which can read the target from the catalog and add to the statement's resolved ids. 6. **Feature flag** `enable_materialized_view_column_definitions`, off in production and on in the test and CI configuration. @@ -210,7 +216,7 @@ The main risks are: Two-token lookahead resolves it, and roundtrip tests cover it. * **Cast semantics surprising users**, e.g. assignment casts to `varchar(n)` or `numeric(p, s)`. This is the same behavior as `INSERT`, see open questions. -* **Key proofs failing for reasonable queries** because MIR key inference is incomplete (e.g. through casts or `UNION ALL` of disjoint inputs). +* **Key checks failing for reasonable queries** because MIR key inference is incomplete (e.g. through casts or `UNION ALL` of disjoint inputs). In particular, `Map` carries a key over to a new column only if exactly one of its expressions preserves uniqueness, and `cast_relation` puts all casts into one `Map`, so declaring two columns with widening casts loses every key through them. Users can drop the key from the declaration. Runtime enforcement could be added later for those cases. @@ -221,7 +227,7 @@ Two follow-ups would remove the implicit schema entirely: * **Freeze at creation.** When a materialized view is created without column definitions, write the inferred schema into `create_sql`, the same way the selected `AS OF` is written back today. - Inferred keys were proven by the optimizer at creation, so they satisfy the key rule. + Inferred keys pass the key check by construction, with the same caveat that the inference can be wrong. Inferred `NOT NULL` becomes a runtime assertion, which also turns a nullability-inference bug from a persist encoder panic into a query error. * **Migrate existing materialized views.** Rewrite their `create_sql` with the schema their shard currently has. @@ -234,10 +240,10 @@ Both change what `SHOW CREATE MATERIALIZED VIEW` prints for users who never aske ## Minimal Viable Prototype * Parser, planner, optimizer and catalog changes for names, types, `NULL`/`NOT NULL`, behind the feature flag. -* Key declarations with the creation-time proof. +* Key declarations with the creation-time check. * Replacement validation against declared schemas and schema inheritance. * Tests: - * sqllogictest for parsing, casts, error messages, `SHOW CREATE`, key proof failures, and the `MustNotBeNull` runtime error. + * sqllogictest for parsing, casts, error messages, `SHOW CREATE`, key check failures, and the `MustNotBeNull` runtime error. * testdrive for an upsert sink keyed on a declared primary key. * A platform check that a materialized view with a declared schema survives upgrade and restart with an unchanged `RelationDesc`. @@ -257,16 +263,23 @@ Freezing the inferred desc in a new catalog field fixes drift without any syntax It does not let users or replacements *state* the schema, and it creates a second source of truth next to `create_sql` that the two would have to agree with. Phase 2 achieves the same freezing through `create_sql`. -### Unenforced keys +### Keys without the creation-time check -Accepting declared keys without proof (like `KEY (...) NOT ENFORCED` on sinks) is simpler, but the optimizer consumes `RelationDesc` keys, so a wrong declaration silently produces wrong results downstream. +Accepting declared keys without any check (like `KEY (...) NOT ENFORCED` on sinks) is simpler. +The optimizer consumes `RelationDesc` keys, so a mistyped declaration silently produces wrong results downstream, and the check catches the common case of a key the query plainly does not have. Keys that are informational only would need a place in `RelationDesc` that the optimizer ignores, which does not exist today. +### Nullability and keys stay inferred + +Only names and scalar types could be declarable, with nullability and keys left to analysis as a second tier of the type. +That avoids freezing an analysis result, but it keeps the drift this design sets out to remove: nullability and keys are part of the `RelationDesc` that persist stores as the shard's schema, and that sinks, dependent views and replacements consume. +Separating them from the persist schema identity is possible, but it is a larger change to persist and the controller. + ### Runtime-enforced keys Checking uniqueness in the dataflow means maintaining a count per key, i.e. an arrangement the size of the materialized view. -It removes the incompleteness of the key proof but adds a large and surprising cost. -It can be added later as an opt-in for keys the optimizer cannot prove. +It turns declared keys into enforced constraints, but adds a large and surprising cost. +It can be added later as an opt-in, also for keys the optimizer cannot infer. ### Sink-style `KEY (...)` instead of `PRIMARY KEY` / `UNIQUE` @@ -287,9 +300,12 @@ A sink-style `KEY (...)` clause would also invite `NOT ENFORCED`, which we rejec The prototype uses assignment casts. * Should replacement schema inheritance reject casts? The prototype allows them, see [Replacements](#replacements). -* Should `NOT NULL` skip the runtime assertion when the optimizer proves non-nullability? +* Should `NOT NULL` skip the runtime assertion when the optimizer infers non-nullability? The assertion is a per-row datum scan in the sink, and skipping it would make performance, though not correctness, depend on the optimizer. The prototype never skips it. +* Should a `NOT NULL` violation error the materialized view, or route the row to a dead-letter queue? + An error blocks reads at the affected times. A dead-letter queue keeps the materialized view readable, but drops data silently unless someone watches the queue. + The prototype errors. * Should `mz_materialized_views` expose whether the schema is declared? The prototype does not. * Do we want Phase 2, and if so, should freezing at creation be the default or opt-in? From 2e957bb7226bfc2334b0c1d243dbb3e2ba6944af Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 7 Oct 2026 20:53:16 +0200 Subject: [PATCH 5/7] design: re-check declared keys on every plan A key confirmed by one version's analysis can stop holding in a later version without any optimizer bug, through changed function or query semantics or changed input keys. Freeze only names, types and the enforced nullability. Treat declared keys as expected keys that every plan checks against the running version's inferred keys: creation fails on an unconfirmed key, and a re-plan drops it and records it in mz_internal.mz_materialized_view_unconfirmed_keys. Co-Authored-By: Claude Opus 5.5 --- ...61006_materialized_view_declared_schema.md | 99 ++++++++++++------- 1 file changed, 64 insertions(+), 35 deletions(-) diff --git a/doc/developer/design/20261006_materialized_view_declared_schema.md b/doc/developer/design/20261006_materialized_view_declared_schema.md index 647d114cade83..40b4c4fccc348 100644 --- a/doc/developer/design/20261006_materialized_view_declared_schema.md +++ b/doc/developer/design/20261006_materialized_view_declared_schema.md @@ -33,11 +33,11 @@ The drift leaks into everything that consumes the schema: ## Success Criteria * A user can state the full schema of a materialized view: column names, types, nullability, and keys. -* A declared schema is independent of the optimizer. - Re-planning the same `create_sql` with a different optimizer yields the same `RelationDesc`. +* Declared names, types and nullability are independent of the optimizer. + Re-planning the same `create_sql` with a different optimizer yields the same names, types and nullability. * A declared `NOT NULL` is enforced: a violation surfaces as an error, independent of the optimizer. -* A declared key is an unenforced constraint, as in SQL systems that accept `NOT ENFORCED` keys. - Materialize rejects keys its analysis of the query does not support at creation, but it does not validate them against the data, so a wrong key gives wrong results. +* A declared key is trusted only while the running version's analysis confirms it. + A key that a later version cannot confirm stops being trusted, and Materialize reports that to the user. * Replacements are validated against the declared schema, and can be written to fit a given schema. * No change to the durable catalog format, persist, or the compute protocol. @@ -90,14 +90,15 @@ We keep the object type `MATERIALIZED VIEW` and do not introduce `CREATE MATERIA ### Semantics A materialized view with column definitions has a *declared schema*. -Its `RelationDesc` is exactly the declared one: names, scalar types, nullability, and keys come from the column definitions, and nothing inferred by the optimizer is merged in. +Its names, scalar types and nullability are exactly the declared ones, and nothing inferred by the optimizer is merged in. +Its keys are the declared keys that the running version confirms, see [Keys](#keys). Without column definitions, behavior is unchanged. -The declared schema has two tiers. +The parts of a declared schema differ in what makes them hold. Names and scalar types are structural: the query is cast to them, so they hold by construction. -Nullability and keys are semantic properties of the data, which the optimizer can only estimate by analysis, and that analysis can be incomplete or wrong in a given version. -Declaring them fixes what the system assumes, but analysis never decides after creation whether they hold: `NOT NULL` is checked against the data at runtime, and keys are the user's assertion. -This avoids the ambiguity of comparing a declared property against a later optimizer's analysis, where a disagreement could be a violation in the data or a miss in the analysis. +Nullability and keys are semantic properties of the query's output, which are undecidable in general (the query language includes negation, aggregation and `WITH MUTUALLY RECURSIVE`), so the optimizer can only approximate them by analysis. +A declared `NOT NULL` does not rely on analysis, because every row is checked at runtime, so it can be frozen like names and types. +A declared key cannot be checked against the data without a uniqueness arrangement, so it relies on analysis, and analysis differs between versions. Opting in has a cost: a key or non-null fact the optimizer infers today but the user does not declare is gone. Dependent queries lose the optimizations it enabled (e.g. `Distinct` or `Reduce` elision), and an upsert sink `KEY (...)` on an undeclared key fails to plan. @@ -123,17 +124,33 @@ Declaring `NOT NULL` is therefore the column-definition spelling of `ASSERT NOT Using `WITH (ASSERT NOT NULL ...)` together with a declared schema, written out or inherited by a replacement, is rejected to avoid two ways to say the same thing in one statement. **Keys.** -Keys in a `RelationDesc` are consumed by the optimizer of every dependent query (e.g. to elide `Distinct` and `Reduce`). -An incorrect key produces wrong results, which is why `CREATE TABLE` keys are gated behind `unsafe_enable_table_keys`. -Declared keys are unenforced and unvalidated against the data, so they are a promise by the user. -To catch obvious mistakes, creation requires each declared `PRIMARY KEY` or `UNIQUE` key to contain some key the optimizer infers for the (cast) query, otherwise it fails with an error listing the inferred keys. -This mirrors the existing upsert sink `KEY` validation. -The check is a guard, not a guarantee: MIR key inference is an analysis of the optimizer version that runs it, and it can be wrong. - -The check runs when the statement is first optimized, for `CREATE` and `EXPLAIN CREATE` alike, but not when `create_sql` is re-planned on boot or by `EXPLAIN REPLAN`. -After creation, a key the check accepted is taken to be a property of the query and its inputs' schemas. -A later optimizer that fails to infer the key does not reject the materialized view, and it could not tell a wrong key from a gap in its own analysis anyway. -The flip side is that a key-inference bug at creation time is frozen into `create_sql`, where today a fixed optimizer would stop inferring the bad key on the next upgrade. +Keys in a `RelationDesc` are consumed by the optimizer of every dependent query and by sinks, and an incorrect key produces wrong results: +`ReduceElision` drops a `DISTINCT` or `GROUP BY` on a key, `RedundantJoin` and `SemijoinIdempotence` drop joins, and an upsert sink keyed on a false key silently keeps one row per key. +This is why `CREATE TABLE` keys are gated behind `unsafe_enable_table_keys`. + +A key that version i confirms does not stay valid in version i', even if key inference is correct in both versions. +Inference is relative to the query's semantics and the inputs' declared keys in the version that runs it, and both change between versions: + +* Keys derived through scalar functions rely on those functions being injective (`preserves_uniqueness` on the function, carried through `Map` in `MirRelationExpr::keys_with_input_keys`). + The annotation covers many casts to text, and a change to such a function's output format can break injectivity. + It has also been wrong: `text_to_name` and `varchar(n)` truncation (#36653), `text` to `"char"` and `bytea` (#36663), and array casts ignoring lower bounds (#37393) each had to be corrected. +* Planner fixes change what a query computes, for example the decorrelation fix for correlated CTEs (#37506). +* Input keys change: builtin relation descs are code, and #36135 changed the key of `mz_dataflow_global_ids` from `(id)` to `(id, global_id)` after the wrong key let the optimizer drop a `DISTINCT`. + Keys of upstream views and materialized views are re-inferred per version and inherit the same issues. + +Keys derived structurally, from `Reduce` or `Distinct` grouping, `TopK` with limit 1, joins on keyed equivalences and filters on literals, do not depend on any function's behavior, but they still depend on the query's semantics and the input keys. +Freezing a declared key into the `RelationDesc` would keep a key alive after the semantics or input keys that justified it changed, and after a fix to the analysis that confirmed it, which today's re-inferred keys pick up on the next upgrade. + +Declared keys are therefore *expected* keys, which every plan of the materialized view checks against the keys the running version infers for the (cast) query: + +* At creation, for `CREATE` and `EXPLAIN CREATE` alike, a declared `PRIMARY KEY` or `UNIQUE` key that does not contain an inferred key is an error that lists the inferred keys. + This mirrors the existing upsert sink `KEY` validation. +* When `create_sql` is re-planned, on boot or by `EXPLAIN REPLAN`, the `RelationDesc` keeps the declared keys that the running version confirms and drops the rest. + The materialized view keeps running, and the dropped key is recorded in a new `mz_internal.mz_materialized_view_unconfirmed_keys` relation (materialized view, key columns, and the version that could not confirm it), so the user learns that the system no longer trusts it. + +A dropped key does not tell whether the key is false or the analysis got weaker. +The system cannot decide that, and the design does not try to: it only refuses to trust what the running version cannot confirm, and says so. +Dropping a key changes the materialized view's `RelationDesc` on boot, which today's inferred keys already do: `evolve_nullability_for_bootstrap` re-registers the shard schema with the re-planned desc. A `UNIQUE` constraint on nullable columns without `NULLS NOT DISTINCT` is an error. This includes column-level `UNIQUE`, which has no `NULLS NOT DISTINCT` spelling and therefore requires `NOT NULL`. @@ -151,8 +168,8 @@ The named type becomes a dependency of the materialized view, as with tables. ### Replacements -With a declared schema, a replacement is validated against the target's declared `RelationDesc` with the same `RelationDesc::diff` check as today. -The difference is that both sides are now under the user's control, so a mismatch in nullability or keys is something the user wrote, not something the optimizer decided. +With a declared schema, a replacement is validated against the target's declared schema: names, types and nullability with the same `RelationDesc::diff` check as today, and keys by comparing the declared keys rather than the confirmed ones. +The difference is that both sides are now under the user's control, so a mismatch is something the user wrote, not something the optimizer decided. A replacement without a column list for a target *with* a declared schema inherits the target's schema. Purification copies the target's column definitions into the replacement's statement, so inheriting is shorthand for writing them out: the query is cast to them with assignment casts, `NOT NULL` becomes an assertion, keys go through the creation-time check, and the feature flag applies. @@ -185,9 +202,11 @@ This is the migration path from an implicit to a declared schema for an existing Soft-assert that arity and scalar types agree with the local plan's type. 4. **Catalog** (`CatalogState::parse_item`). Apply the same rule: `desc = declared_desc` if present, otherwise `infer_sql_type_for_catalog` plus assertions. - This is the single point that makes the schema optimizer-independent on boot. + For a declared desc, keep only the declared keys that the inferred type confirms, and record the others for `mz_materialized_view_unconfirmed_keys`. + This is the single point that makes names, types and nullability optimizer-independent on boot. 5. **Sequencer** (`src/adapter/src/coord/sequencer/inner/create_materialized_view.rs`). Run the key check against the keys of the local MIR plan's type, in the optimize stage so that `EXPLAIN CREATE` rejects what `CREATE` rejects. + A re-plan (`EXPLAIN REPLAN`) drops unconfirmed keys instead, as on boot. Reject a replacement without a declared schema for a target with one. Schema inheritance for replacements lives in purification (`src/sql/src/pure.rs`), which can read the target from the catalog and add to the statement's resolved ids. 6. **Feature flag** `enable_materialized_view_column_definitions`, off in production and on in the test and CI configuration. @@ -197,7 +216,7 @@ This is the migration path from an implicit to a declared schema for an existing 8. **Docs** for `CREATE MATERIALIZED VIEW`, and a note in the dbt adapter that contracts can be expressed directly. Nothing changes in the durable catalog format (the schema lives in `create_sql`), persist, or compute. -`evolve_nullability_for_bootstrap` still runs for materialized views with declared schemas, but has no effect, because their desc no longer changes between versions. +`evolve_nullability_for_bootstrap` still runs for materialized views with declared schemas, and only has an effect when a version drops an unconfirmed key. ### `EXPLAIN ... WITH (schema)` @@ -216,7 +235,7 @@ The main risks are: Two-token lookahead resolves it, and roundtrip tests cover it. * **Cast semantics surprising users**, e.g. assignment casts to `varchar(n)` or `numeric(p, s)`. This is the same behavior as `INSERT`, see open questions. -* **Key checks failing for reasonable queries** because MIR key inference is incomplete (e.g. through casts or `UNION ALL` of disjoint inputs). +* **Key checks failing for reasonable queries**, at creation or after an upgrade, because MIR key inference is incomplete (e.g. through casts or `UNION ALL` of disjoint inputs). In particular, `Map` carries a key over to a new column only if exactly one of its expressions preserves uniqueness, and `cast_relation` puts all casts into one `Map`, so declaring two columns with widening casts loses every key through them. Users can drop the key from the declaration. Runtime enforcement could be added later for those cases. @@ -227,7 +246,7 @@ Two follow-ups would remove the implicit schema entirely: * **Freeze at creation.** When a materialized view is created without column definitions, write the inferred schema into `create_sql`, the same way the selected `AS OF` is written back today. - Inferred keys pass the key check by construction, with the same caveat that the inference can be wrong. + Inferred keys become expected keys and are re-checked on every plan like declared ones. Inferred `NOT NULL` becomes a runtime assertion, which also turns a nullability-inference bug from a persist encoder panic into a query error. * **Migrate existing materialized views.** Rewrite their `create_sql` with the schema their shard currently has. @@ -240,12 +259,12 @@ Both change what `SHOW CREATE MATERIALIZED VIEW` prints for users who never aske ## Minimal Viable Prototype * Parser, planner, optimizer and catalog changes for names, types, `NULL`/`NOT NULL`, behind the feature flag. -* Key declarations with the creation-time check. +* Key declarations with the check on every plan and `mz_materialized_view_unconfirmed_keys`. * Replacement validation against declared schemas and schema inheritance. * Tests: * sqllogictest for parsing, casts, error messages, `SHOW CREATE`, key check failures, and the `MustNotBeNull` runtime error. * testdrive for an upsert sink keyed on a declared primary key. - * A platform check that a materialized view with a declared schema survives upgrade and restart with an unchanged `RelationDesc`. + * A platform check that a materialized view with a declared schema survives upgrade and restart with unchanged names, types and nullability, and with its keys confirmed. ## Alternatives @@ -263,22 +282,27 @@ Freezing the inferred desc in a new catalog field fixes drift without any syntax It does not let users or replacements *state* the schema, and it creates a second source of truth next to `create_sql` that the two would have to agree with. Phase 2 achieves the same freezing through `create_sql`. -### Keys without the creation-time check +### Frozen keys + +Declared keys could be checked only at creation and then trusted forever, which makes the whole `RelationDesc` stable across versions. +As [Keys](#keys) shows, a key confirmed at creation can become false without any optimizer bug, through changed function or query semantics or changed input keys, and a frozen key would also outlive fixes to the analysis that confirmed it. +The system could not tell the user, because it would never look again. + +### Keys without any check Accepting declared keys without any check (like `KEY (...) NOT ENFORCED` on sinks) is simpler. The optimizer consumes `RelationDesc` keys, so a mistyped declaration silently produces wrong results downstream, and the check catches the common case of a key the query plainly does not have. Keys that are informational only would need a place in `RelationDesc` that the optimizer ignores, which does not exist today. -### Nullability and keys stay inferred +### Nullability stays inferred -Only names and scalar types could be declarable, with nullability and keys left to analysis as a second tier of the type. -That avoids freezing an analysis result, but it keeps the drift this design sets out to remove: nullability and keys are part of the `RelationDesc` that persist stores as the shard's schema, and that sinks, dependent views and replacements consume. -Separating them from the persist schema identity is possible, but it is a larger change to persist and the controller. +Only names and scalar types could be declarable, with nullability left to analysis like keys. +Unlike keys, a declared `NOT NULL` is enforced on every row, so freezing it does not rely on analysis, and leaving it inferred would keep the drift this design sets out to remove: nullability is part of the `RelationDesc` that persist stores as the shard's schema, and that sinks, dependent views and replacements consume. ### Runtime-enforced keys Checking uniqueness in the dataflow means maintaining a count per key, i.e. an arrangement the size of the materialized view. -It turns declared keys into enforced constraints, but adds a large and surprising cost. +It turns declared keys into enforced constraints that could be frozen like `NOT NULL`, but adds a large and surprising cost. It can be added later as an opt-in, also for keys the optimizer cannot infer. ### Sink-style `KEY (...)` instead of `PRIMARY KEY` / `UNIQUE` @@ -287,7 +311,7 @@ Materialize has two precedents for declaring keys. User-facing, `CREATE SINK ... KEY (...) [NOT ENFORCED]` validates a requested key against the inferred keys, which is the same check this design applies. In column-definition lists, `PRIMARY KEY` and `UNIQUE` are what `CREATE TABLE` parses and what purification writes into the `create_sql` of tables created from Postgres, MySQL and SQL Server sources. Since the declared schema is a column-definition list, we follow the second precedent and reuse its grammar and planning code. -A sink-style `KEY (...)` clause would also invite `NOT ENFORCED`, which we reject for materialized views (see [Unenforced keys](#unenforced-keys)). +A sink-style `KEY (...)` clause would also invite `NOT ENFORCED`, which we reject for materialized views (see [Keys without any check](#keys-without-any-check)). ### Schema as a `WITH` option @@ -308,4 +332,9 @@ A sink-style `KEY (...)` clause would also invite `NOT ENFORCED`, which we rejec The prototype errors. * Should `mz_materialized_views` expose whether the schema is declared? The prototype does not. +* What happens to an upsert sink keyed on a key that a new version drops? + Its `create_sql` fails to re-plan on boot, as it does today when an inferred key disappears. + Planning such a sink could keep running with a notice instead, like `KEY (...) NOT ENFORCED`. +* Should a dropped key say why it was dropped? + Recording at creation how a key was derived (only structural steps, through which uniqueness-preserving functions, from which input keys) would let a later version tell an analysis gap apart from a changed function or a missing input key. * Do we want Phase 2, and if so, should freezing at creation be the default or opt-in? From 1d9944c34d7136604717be18d02189b188cb7a41 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 7 Oct 2026 20:57:40 +0200 Subject: [PATCH 6/7] design: keep upsert sinks on dropped keys planning A re-plan error at catalog load panics, so an upsert sink keyed on a key that a new version drops would crash-loop boot. Treat such a key as NOT ENFORCED when re-planning persisted SQL, and keep the error for new CREATE SINK statements. Co-Authored-By: Claude Opus 5.5 --- .../design/20261006_materialized_view_declared_schema.md | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/doc/developer/design/20261006_materialized_view_declared_schema.md b/doc/developer/design/20261006_materialized_view_declared_schema.md index 40b4c4fccc348..db1555cccda7d 100644 --- a/doc/developer/design/20261006_materialized_view_declared_schema.md +++ b/doc/developer/design/20261006_materialized_view_declared_schema.md @@ -152,6 +152,12 @@ A dropped key does not tell whether the key is false or the analysis got weaker. The system cannot decide that, and the design does not try to: it only refuses to trust what the running version cannot confirm, and says so. Dropping a key changes the materialized view's `RelationDesc` on boot, which today's inferred keys already do: `evolve_nullability_for_bootstrap` re-registers the shard schema with the re-planned desc. +A dropped key must not stop dependents from re-planning, because a re-plan error at catalog load panics (`invalid persisted SQL` in `src/adapter/src/catalog/apply.rs`). +An upsert sink checks its `KEY (...)` against the keys of its input on every plan (`plan_sink` in `src/sql/src/plan/statement/ddl.rs`), so a sink keyed on a dropped key would crash-loop the next boot. +When re-planning persisted SQL, which is planned without a `PlanContext`, an upsert key that is no longer a key of the input is therefore treated as `NOT ENFORCED`: the sink keeps running and the planner adds a notice. +New `CREATE SINK` statements keep the error. +Today's inferred keys can disappear on upgrade in the same way, so this also closes an existing crash. + A `UNIQUE` constraint on nullable columns without `NULLS NOT DISTINCT` is an error. This includes column-level `UNIQUE`, which has no `NULLS NOT DISTINCT` spelling and therefore requires `NOT NULL`. (`plan_create_table` stops processing constraints at the first such one instead, which drops it and every later key, so a declared key would silently disappear.) @@ -332,9 +338,6 @@ A sink-style `KEY (...)` clause would also invite `NOT ENFORCED`, which we rejec The prototype errors. * Should `mz_materialized_views` expose whether the schema is declared? The prototype does not. -* What happens to an upsert sink keyed on a key that a new version drops? - Its `create_sql` fails to re-plan on boot, as it does today when an inferred key disappears. - Planning such a sink could keep running with a notice instead, like `KEY (...) NOT ENFORCED`. * Should a dropped key say why it was dropped? Recording at creation how a key was derived (only structural steps, through which uniqueness-preserving functions, from which input keys) would let a later version tell an analysis gap apart from a changed function or a missing input key. * Do we want Phase 2, and if so, should freezing at creation be the default or opt-in? From e10f6f8eedba24d404c22051a65d6b1f698615a1 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 7 Oct 2026 21:12:30 +0200 Subject: [PATCH 7/7] design: implicit casts, pinned LIR, and review follow-ups Cast query output to declared types with implicit casts only, so narrowing casts are written in the query. Note how declared schemas relate to pinned LIR, how index arrangements could check keys, why the NOT NULL assertion is never skipped, and fix the stale sequencer step. Co-Authored-By: Claude Opus 5.5 --- ...61006_materialized_view_declared_schema.md | 68 +++++++++++++------ 1 file changed, 48 insertions(+), 20 deletions(-) diff --git a/doc/developer/design/20261006_materialized_view_declared_schema.md b/doc/developer/design/20261006_materialized_view_declared_schema.md index db1555cccda7d..2254a48bc181d 100644 --- a/doc/developer/design/20261006_materialized_view_declared_schema.md +++ b/doc/developer/design/20261006_materialized_view_declared_schema.md @@ -77,7 +77,7 @@ CREATE MATERIALIZED VIEW order_totals ( total numeric(38, 2) NOT NULL, PRIMARY KEY (customer_id) ) IN CLUSTER c AS - SELECT customer_id, any_value(region), sum(amount) + SELECT customer_id, any_value(region), sum(amount)::numeric(38, 2) FROM orders GROUP BY customer_id; ``` @@ -109,10 +109,14 @@ The number of declared columns must equal the query's arity. Columns map by position, and declared names replace the query's names, as the bare-name form does today. **Types.** -The query output is cast to the declared types with assignment casts (`CastContext::Assignment`), the same rule `INSERT INTO t SELECT ...` applies. +The query output is cast to the declared types with implicit casts (`CastContext::Implicit`), the casts Materialize inserts without being asked, for example `int4` to `int8` or `numeric` to `float8`. The cast is planned into the HIR expression with the existing `query::cast_relation`, so it is part of the dataflow and appears in `EXPLAIN`. -A column that cannot be assignment-cast is a planning error naming the column and both types. -A cast that fails at runtime (e.g. overflow on `int8` to `int2`) produces an error in the materialized view, like any other evaluation error. +A column that cannot be implicitly cast is a planning error naming the column and both types, and the user writes the cast in the query. +Implicit casts widen, so the narrowing casts that fail at runtime, `int8` to `int2` or `float8` to `numeric`, and the rounding cast from `numeric` to `numeric(p, s)` all have to be written in the query. +Two implicit casts can still fail or lose precision: `text` to `varchar(n)` or `char(n)` errors on values longer than `n`, and `int8` or `numeric` to `float8` rounds. +A cast that fails at runtime produces an error in the materialized view, like any other evaluation error. +Assignment casts, which `INSERT INTO t SELECT ...` applies, would accept the narrowing casts silently. +A type declared once is easy to forget when the query changes, so a narrowing cast should be visible where the query is written, see [Assignment casts](#assignment-casts). **Nullability.** A column without `NOT NULL` is nullable, as in `CREATE TABLE`. @@ -164,7 +168,7 @@ This includes column-level `UNIQUE`, which has no `NULLS NOT DISTINCT` spelling `PRIMARY KEY` implies `NOT NULL` on its columns, and declaring a primary key column `NULL` is an error. `RelationDesc` stores key columns sorted, so the declared column order of a key is not preserved, as for tables. -Note that assignment casts can drop keys from the inferred set when the cast function does not report `preserves_uniqueness`. +Note that casts can drop keys from the inferred set when the cast function does not report `preserves_uniqueness`. The user then sees the key check error and can adjust the query or the declared type. **Types that cannot be spelled.** @@ -178,17 +182,21 @@ With a declared schema, a replacement is validated against the target's declared The difference is that both sides are now under the user's control, so a mismatch is something the user wrote, not something the optimizer decided. A replacement without a column list for a target *with* a declared schema inherits the target's schema. -Purification copies the target's column definitions into the replacement's statement, so inheriting is shorthand for writing them out: the query is cast to them with assignment casts, `NOT NULL` becomes an assertion, keys go through the creation-time check, and the feature flag applies. +Purification copies the target's column definitions into the replacement's statement, so inheriting is shorthand for writing them out: the query is cast to them with implicit casts, `NOT NULL` becomes an assertion, keys go through the creation-time check, and the feature flag applies. +Because casts are implicit, a replacement whose query produces `int8` for an inherited `int2` column fails to plan instead of narrowing silently. +Changing a declared type is a schema change, which replacements do not support (see [Out of Scope](#out-of-scope)). The replacement's `create_sql` then states its schema. Re-planning it on boot does not depend on the target, custom types in the definitions are recorded as dependencies, and `MaterializedView::apply_replacement` keeps taking the column list from the replacement's statement. -Requiring the query's types to match the target's exactly, without casts, would keep accidental casts out of a drop-in replacement. +Requiring the query's types to match the target's exactly, without casts, would also keep widening casts out of a drop-in replacement. It needs to know at planning time whether the definitions were inherited, which the stored statement cannot say, and checking exact types when re-planning would make an upgrade that changes a query's inferred type fail on boot. A replacement with a bare-name list for a target with a declared schema is rejected, because renaming conflicts with inheriting the names. A replacement *with* column definitions for a target *without* one is allowed if the declared desc equals the target's current desc. This is the migration path from an implicit to a declared schema for an existing materialized view. +`EXPLAIN ... WITH (schema)` on the target prints the definitions to paste. +Purification could fill them in instead, but a replacement without a column list for an implicit target then opts into a declared schema without the user asking, which is the decision [Phase 2](#phase-2-freezing-inferred-schemas) defers. ### Implementation sketch @@ -213,8 +221,9 @@ This is the migration path from an implicit to a declared schema for an existing 5. **Sequencer** (`src/adapter/src/coord/sequencer/inner/create_materialized_view.rs`). Run the key check against the keys of the local MIR plan's type, in the optimize stage so that `EXPLAIN CREATE` rejects what `CREATE` rejects. A re-plan (`EXPLAIN REPLAN`) drops unconfirmed keys instead, as on boot. - Reject a replacement without a declared schema for a target with one. + Validate a replacement's declared schema against its target's. Schema inheritance for replacements lives in purification (`src/sql/src/pure.rs`), which can read the target from the catalog and add to the statement's resolved ids. + The sequencer still rejects a replacement without a declared schema for a target with one, which only a statement that skipped purification can reach. 6. **Feature flag** `enable_materialized_view_column_definitions`, off in production and on in the test and CI configuration. 7. **mz-deploy.** Its typecheck catalog uses the declared desc when present. Unit tests run a materialized view as a temporary view, which cannot declare a schema, so a declared materialized view is lowered to casts to the declared types. @@ -234,13 +243,13 @@ The option is not behind the feature flag, because it is useful for inspecting i ### Feasibility The change is small to medium and mostly in the SQL layer. -Every runtime mechanism it needs already exists: assignment casts for `INSERT`, column-definition parsing for `CREATE TABLE`, non-null assertions for `ASSERT NOT NULL`, key validation for upsert sinks, and desc comparison for replacements. +Every runtime mechanism it needs already exists: relation casts for `INSERT`, column-definition parsing for `CREATE TABLE`, non-null assertions for `ASSERT NOT NULL`, key validation for upsert sinks, and desc comparison for replacements. The main risks are: * **Parser ambiguity** between the two column-list forms. Two-token lookahead resolves it, and roundtrip tests cover it. -* **Cast semantics surprising users**, e.g. assignment casts to `varchar(n)` or `numeric(p, s)`. - This is the same behavior as `INSERT`, see open questions. +* **Implicit casts being too strict**, e.g. `sum(amount)` declared as `numeric(38, 2)` needs a cast in the query. + The planning error names the column and both types. * **Key checks failing for reasonable queries**, at creation or after an upgrade, because MIR key inference is incomplete (e.g. through casts or `UNION ALL` of disjoint inputs). In particular, `Map` carries a key over to a new column only if exactly one of its expressions preserves uniqueness, and `cast_relation` puts all casts into one `Map`, so declaring two columns with widening casts loses every key through them. Users can drop the key from the declaration. Runtime enforcement could be added later for those cases. @@ -309,8 +318,17 @@ Unlike keys, a declared `NOT NULL` is enforced on every row, so freezing it does Checking uniqueness in the dataflow means maintaining a count per key, i.e. an arrangement the size of the materialized view. It turns declared keys into enforced constraints that could be frozen like `NOT NULL`, but adds a large and surprising cost. +An index on exactly the key columns already holds that arrangement, so the check could ride on it instead of building a second one. +The index runs in its own dataflow, possibly on another cluster, and dependents that read the materialized view from persist rely on the key whether or not the index exists, so a violation it finds could be reported but not prevented. It can be added later as an opt-in, also for keys the optimizer cannot infer. +### Assignment casts + +Casting with assignment casts, as `INSERT INTO t SELECT ...` does, accepts every cast that implicit casts accept plus the narrowing ones: `int8` to `int2`, `float8` to `numeric`, `numeric` to `numeric(p, s)`. +It saves writing casts in the query, most often for `numeric` scale. +A narrowing cast that fails at runtime errors the materialized view, and with an inherited schema the cast is not written anywhere in the replacement, so the user does not see it unless they read `EXPLAIN`. +Moving from implicit to assignment casts later accepts strictly more statements, so it stays possible without breaking existing ones. + ### Sink-style `KEY (...)` instead of `PRIMARY KEY` / `UNIQUE` Materialize has two precedents for declaring keys. @@ -323,21 +341,31 @@ A sink-style `KEY (...)` clause would also invite `NOT ENFORCED`, which we rejec `WITH (SCHEMA (...))` or similar would avoid touching the column list, but it splits naming (column list) from typing (option) and departs from the `CREATE TABLE` grammar users and tools already know. +### Skipping the `NOT NULL` assertion on inferred columns + +The sink could skip the assertion for columns the optimizer infers to be non-nullable, as performance already depends on the optimizer elsewhere. +Correctness would then depend on nullability inference being right: a null that reaches a non-nullable column panics the persist encoder (`DatumEncoder::push`) instead of producing a `MustNotBeNull` error. +Implicit materialized views carry that risk today, and declared schemas are meant to remove it. +The assertion inspects the asserted datums of each row the sink writes, which the sink encodes anyway, and we have not measured its cost. + +## Interaction with pinned LIR + +[Pinned LIR](./20260826_pinned_lir.md) stores a materialized view's plan instead of re-optimizing it on boot. +A declared schema gives the pinned plan a fixed output contract: names, types and nullability no longer change when the optimizer does, so a pinned plan stays type-compatible with the catalog desc across versions. +Keys are less settled: this design re-checks declared keys against the running version's analysis of the query, while a pinned plan was produced by an older version. +Whether a pinned plan should also pin the keys confirmed when it was planned, and how a changed input key invalidates them, belongs to the pinned LIR design. + ## Open questions -* Assignment casts, or implicit casts only? - Implicit-only is stricter and avoids silent truncation-style surprises, but forces users to write casts in the query for common cases such as `numeric` scale. - The prototype uses assignment casts. -* Should replacement schema inheritance reject casts? +* Should replacement schema inheritance reject widening casts? The prototype allows them, see [Replacements](#replacements). -* Should `NOT NULL` skip the runtime assertion when the optimizer infers non-nullability? - The assertion is a per-row datum scan in the sink, and skipping it would make performance, though not correctness, depend on the optimizer. - The prototype never skips it. * Should a `NOT NULL` violation error the materialized view, or route the row to a dead-letter queue? An error blocks reads at the affected times. A dead-letter queue keeps the materialized view readable, but drops data silently unless someone watches the queue. The prototype errors. -* Should `mz_materialized_views` expose whether the schema is declared? - The prototype does not. +* How should the catalog expose whether a schema is declared? + It is useful to us for adoption and to users for tracking a migration to declared schemas. + An `mz_internal` relation fits while the feature is behind a flag, and a column of `mz_materialized_views` once it is stable. + The prototype does not expose it. * Should a dropped key say why it was dropped? Recording at creation how a key was derived (only structural steps, through which uniqueness-preserving functions, from which input keys) would let a later version tell an analysis gap apart from a changed function or a missing input key. * Do we want Phase 2, and if so, should freezing at creation be the default or opt-in?