Skip to content

Read the type catalog on the stream's own connection before start - #155

Merged
isdaniel merged 5 commits into
isdaniel:mainfrom
LucaCappelletti94:upstream/stream-type-catalog
Nov 5, 2026
Merged

isdaniel merged 5 commits into
isdaniel:mainfrom
LucaCappelletti94:upstream/stream-type-catalog

Conversation

@LucaCappelletti94

Copy link
Copy Markdown
Contributor

Summary

A stream started with with_binary(true) sends a column in binary only when its type has a send function, and as text otherwise. For the built-in types with hand-assigned OIDs that is fixed, but for extension types, enums, domains and composites only pg_type knows, and a consumer had no way to read it on the connection the stream decodes from. A second connection can land on another cluster when the conninfo lists several hosts, the same hazard #141 removed for the startup metadata.

type_catalog() reads every pg_type row before start, like slot_confirmed_flush_lsn(), and returns each type's OID, schema, name and typtype, whether it has a send function, and its element, domain base, range subtype and composite relation. Those are the facts tokio-postgres reads to resolve a type, so a consumer can decode domains, arrays, ranges and composites from this one read. Types created after it are missing, and the stream does not read the catalog again on reconnect.

Related issues

None.

Type of change

  • Bug fix
  • New feature
  • Performance improvement
  • Refactor / internal cleanup
  • Documentation
  • CI / build

Checklist

  • make before-git-push passes locally (check, build, format, audit, test, doc-check)
  • Tests added/updated for the change (coverage stays ≥ 90%)
  • Both backends build (rustls-tls and libpq)
  • Docs / README updated if public API changed

Performance notes

n/a, the read runs once before start and the hot path is untouched.

@codspeed

codspeed Bot commented Nov 2, 2026 •

Copy link
Copy Markdown

Merging this PR will not alter performance

✅ 50 untouched benchmarks


Comparing LucaCappelletti94:upstream/stream-type-catalog (f308a0b) with main (c4da9fc)

Open in CodSpeed

@codecov

codecov Bot commented Nov 2, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 96.46%. Comparing base (c4da9fc) to head (f308a0b).

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #155      +/-   ##
==========================================
- Coverage   96.48%   96.46%   -0.02%     
==========================================
  Files          36       36              
  Lines       29314    29438     +124     
==========================================
+ Hits        28283    28398     +115     
- Misses       1031     1040       +9     
Flag Coverage Δ
integration 64.81% <85.48%> (+0.29%) ⬆️
unit 93.43% <99.19%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The catalog query can invalidate an exported snapshot, lacks metadata needed to decode several PostgreSQL types, and includes a test that assumes the bootstrap role is named postgres.

5 open issues
What changed in this PR

Adds a pre-start type catalog read on the replication stream’s own connection, extending the startup metadata API without risking a second connection reaching a different cluster.

Changes:

  • Introduces TypeInfo and type_catalog() for type identities, binary-send capability, and related type OIDs.
  • Re-exports TypeInfo from the crate root.
  • Adds parsing tests and a live PostgreSQL test comparing catalog metadata with streamed column formats.
File Description
src/​stream.rs Adds the catalog query, public metadata API, row parsing, and unit tests.
src/​lib.rs Exposes TypeInfo at the crate root.
integration-tests/​startup_metadata.rs Tests catalog metadata and binary column formats on a live stream.

Review effort: Balanced

Comment thread integration-tests/startup_metadata.rs Resolved
Comment thread src/stream.rs Resolved
Comment thread src/stream.rs Resolved
Comment thread src/stream.rs Resolved
Comment thread src/stream.rs Resolved

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues are blocking the pre-start type-catalog API change.

No open issues 🎉

5 resolved since last review

Review effort: Balanced

@isdaniel

isdaniel commented Nov 5, 2026

Copy link
Copy Markdown
Owner
  1. "binary exactly when its type sends binary" is not true for containers (stream.rs:336, :1049). pgoutput checks only the column type's own send function; array_send/record_send/range send then need the element/subtype/field send function, and fail without it.

    CREATE EXTENSION isn;
    CREATE TABLE t (id int PRIMARY KEY, a isbn[]);
    CREATE PUBLICATION p FOR TABLE t;
    -- binary stream: type_catalog() reports _isbn sends_binary = true
    INSERT INTO t VALUES (1, ARRAY['0-7381-1230-5'::isbn]);
    -- walsender: ERROR: no binary output function available for type isbn
    base head
    arrays, composites, ranges/multiranges over a send-less type, domain over such an array same server error same error, catalog says true

Not only aclitem: isbn, ean13, seg, query_int are send-less. A domain over a send-less type is correctly false. Suggested wording: "pgoutput picks binary from the column type's own send function. For arrays, domains, ranges and composites, a send-less element, subtype or field type makes the walsender fail." Also say that element/base_type/range_subtype can be followed to the leaf, but composite fields cannot be seen (relation() is only an OID).

  1. type_catalog() docs: add "requires replication=database" (as slot_confirmed_flush_lsn says), and that on the manual-export path it must run before ensure_replication_slot(), since the guard releases only on start() or the managed hand-back. Measured: an enum created in that window is missing from the catalog and delivered as Binary.

@LucaCappelletti94

Copy link
Copy Markdown
Contributor Author

Both are in the docs now.

@isdaniel
isdaniel merged commit c846274 into isdaniel:main Nov 5, 2026
25 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants