[val-materialize]

Adds a new Export data > Patterns section with one page, Transactional downstream updates using Kafka. The page shows how a consumer loading a Kafka sink into another store (Snowflake example) can expose only timestamps the sink has fully committed, by gating on the sink's progress frontier and joining data and progress records via the materialize-sink-id header.

Where each fact comes from

Fact	Source
Every message carries materialize-timestamp (value is the timestamp as a decimal string)	src/storage/src/sink/kafka.rs (KafkaTxProducer::send, time.to_string())
materialize-sink-id header value is the sink's GlobalId (for example u123), emitted when kafka_sink_emit_sink_id_header is on	src/storage/src/sink/kafka.rs (sink_id_header: ... sink_id.to_string()); src/storage-types/src/dyncfgs.rs
Progress record key is mz-sink-{sink_id}	src/storage-client/src/sink.rs (ProgressKey::new)
Progress record value is JSON {"frontier":[<ts>],"version":<n>}; frontier is the exclusive upper of committed times	src/storage/src/sink/kafka.rs (ProgressRecord, serialize_frontier, commit_transaction, progress_record_migration test)
Progress record is written in the same Kafka transaction as the data	src/storage/src/sink/kafka.rs (commit_transaction)
A transaction and progress record are committed on every frontier advance, even with no data	src/storage/src/sink/kafka.rs (sink loop, Event::Progress branch)
Default progress topic name _materialize-progress-{env}-{connection}, shared by all sinks on a connection; name exposed in mz_kafka_connections.sink_progress_topic	src/storage-types/src/connections.rs (progress_topic); doc/user/content/sql/create-sink/kafka.md; doc/user/content/sql/system-catalog/mz_catalog.md
Consumers need isolation.level=read_committed; exactly-once details	doc/user/content/sql/create-sink/kafka.md (Exactly-once processing)
materialize- header prefix is reserved	src/storage/src/sink/kafka.rs; doc/user/content/sql/create-sink/kafka.md
Snowflake connector stores payload in RECORD_CONTENT, headers in RECORD_METADATA:headers, and the key in RECORD_METADATA:key only with StringConverter	Snowflake Kafka connector overview

Notes for reviewers

• GA status: the materialize-sink-id header and the HEADERS option are GA in v26.48 (SS-517 - Enable Kafka sink headers feature and sink-id header by default #39908 turns both flags on by default), so the page has no private preview caveat. The Kafka sink docs drop their preview callouts in doc: mark the materialize-sink-id header and HEADERS option as GA #39829, and the v26.48.0 release notes (v26.48.0 release notes #40113) link to this page.
• The Snowflake SQL is marked illustrative. The page names no connector settings except key.converter, which the Snowflake docs confirm.

Part of SS-504.

🤖 Generated with Claude Code

[linear-code]

SS-504

EDU-107

[maheshwarip]

@val-materialize lmk when this is ready for review!

[val-materialize]

Open question for reviewers: the read_committed caveat

The Considerations section says downstream consumers must read with isolation.level=read_committed, linking to the existing end-to-end exactly-once section. This wasn't in the original outline. I added it because the progress-frontier gate doesn't protect against reading uncommitted data on its own. If a sink transaction aborts (for example, on a restart), its data messages can still be loaded by a consumer that reads uncommitted records. After the sink resumes, Materialize writes those changes again in a new transaction, and a later progress record covers their timestamps. The view would then expose both copies.

What I'm unsure about:

• Whether the Snowflake Kafka connector reads committed data by default. The standard Kafka consumer default is read_uncommitted, but I haven't confirmed how the Snowflake connector, or Kafka Connect in general, configures its consumer, or which setting overrides it. That's why the page describes the requirement in prose and doesn't name a connector or Connect setting.
• Whether this belongs on this page. The requirement applies to any consumer of a Kafka sink, not just this pattern. The alternative is to drop the bullet and rely on the link to exactly-once processing.

If someone can confirm how the Snowflake connector handles isolation level, I'll either add the exact setting or simplify the bullet.

[maheshwarip]

Rephrase as: When you load messages from Kafka, into another data store

[maheshwarip]

hmm I didn't understand this part! Is this the query I run in in Snowflake? I think I'd be more verbose with this tutorial; let's explain all the steps from:

1. Creating a sink in Materialize to Kafka
2. Creating a transactional update from Kafka => Snowflake (or other destination)