diff --git a/.changeset/verify-schema.md b/.changeset/verify-schema.md new file mode 100644 index 0000000..3b1ff03 --- /dev/null +++ b/.changeset/verify-schema.md @@ -0,0 +1,5 @@ +--- +"@maple-dev/effect-orm": minor +--- + +`effect-orm verify --schema` compares the table definitions with a live database, with no migrations folder and no ledger. It exits 3 on drift and prints one line per difference (`--json` for machines), so CI fails on a missing column, a type or nullability change, or a missing index. It reads the config's `schema` and `driver` and runs the same comparison as `verify`, for ClickHouse and Postgres; a SQLite or D1 config is refused, since there is no SQLite catalog reader yet. In code: `verifySchema(config, cwd)` from `/kit`, and `Migrate.verifySnapshot(snapshot)` to compare a database with any snapshot. diff --git a/design/migrations.md b/design/migrations.md index f8f05b1..a59c94d 100644 --- a/design/migrations.md +++ b/design/migrations.md @@ -348,6 +348,10 @@ CREATE TABLE IF NOT EXISTS _effect_orm_migration_steps ( `Migrate.verify` introspects (`system.tables`, `system.columns`, `system.data_skipping_indices`, `create_table_query` through `formatQuery()`) and diffs the live schema against the snapshot **of the last applied migration**, not the code's HEAD. That reports drift and partial applies honestly. +`verify --schema` (`Kit.verifySchema`) is the other question: do the definitions at HEAD match +this database? It builds a snapshot from the schema modules and runs the same comparison +(`Migrate.verifySnapshot`), with no folder and no ledger, for a database another tool migrates +and for a CI check after deploy. ### 4.6 CLI @@ -359,7 +363,7 @@ One bin, `effect-orm` (next to `ch-bench`): | `check` | no | DAG validity, recomputed ids, name/DAG order, commutativity, backfill convergence lint, shipped-migration immutability against a git base (`--base origin/main`) | | `migrate` | yes | Runs `Migrate.run` with a `SqlClient` layer exported from the config file | | `status` / `plan` | yes | Pending migrations and their rendered steps, no execution | -| `verify` | yes | 4.5 | +| `verify [--schema]` | yes | 4.5; `--schema` compares with the definitions, not the ledger | | `push` | yes | Dev only: introspect, diff, apply, no files | | `pull [--init]` | yes | Introspect into `defineTable` source; `--init` baselines | diff --git a/docs/migrations.md b/docs/migrations.md index f661350..9401f60 100644 --- a/docs/migrations.md +++ b/docs/migrations.md @@ -8,7 +8,7 @@ into migrations: | Entry | Runs where | What it does | | ---------------------------------- | ------------------ | ----------------------------------------------------------------------------- | | `@maple-dev/effect-orm/schema` | anywhere, pure | reads `table` / `materializedView` values: DDL rendering, snapshots, the diff | -| `@maple-dev/effect-orm/kit` | Node or Bun | `generate` and `check` over a migrations folder; the `effect-orm` command | +| `@maple-dev/effect-orm/kit` | Node or Bun | `generate` and `check` over a migrations folder, `verifySchema`; the `effect-orm` command | | `@maple-dev/effect-orm/migrate` | anywhere Effect runs | applies migrations through a driver you provide, `status`, `verify` | Both ClickHouse and Postgres. The model follows drizzle-kit (a committed snapshot per migration, @@ -255,6 +255,52 @@ both sides (`formatQuery`, `defaultValueOfTypeName`). It checks tables, engine f columns, skipping indexes, and view targets and bodies; it does not check TTL, codecs, settings, or comments yet. `effect-orm verify` exits 3 when it finds drift. +### Checking the definitions against a database + +`verify` needs the migrations folder and the ledger. To check the table definitions against a +database that another tool migrates, or in CI after a deploy, compare it with the definitions +directly: + +```sh +effect-orm verify --schema # exit 0: no drift; exit 3: drift, one line per difference +effect-orm verify --schema --json # {"drift":[{"entity":"users.email","problem":"not_null",...}]} +``` + +```text +Drift against the schema modules: + users.email: type (expected text, actual character varying(10)) + users.name: missing + users.users_email_unq: missing +``` + +It reads the config's `schema` modules and `driver`, and nothing in `out`. The comparison is the +one `verify` makes, with the same checks and the same limits, against a snapshot of the +definitions as they stand instead of the last applied one. So a database that is behind reports +the missing parts as drift; use plain `verify` where the ledger exists. The ledger tables are +never reported. ClickHouse and Postgres only: effect-orm has no SQLite catalog reader yet, so +with `dialect: "sqlite"` (SQLite or D1) it fails with a `KitError` (`unsupported`) before it +connects. In code, `verifySchema(config, cwd)` from `/kit` does the same, and +`Migrate.verifySnapshot(snapshot)` compares the database with any snapshot (a SQLite snapshot +fails with `MigrateSourceError`): + +```ts title="migrations-verify-schema.ts" +import { Effect } from "effect" +import * as PG from "@maple-dev/effect-orm/postgres" +import * as Migrate from "@maple-dev/effect-orm/migrate" +import * as S from "@maple-dev/effect-orm/schema" + +const Users = PG.table("users", { + columns: { id: PG.text, email: PG.text }, + primaryKey: ["id"], +}) + +export const drift = Effect.gen(function* () { + const entities = yield* S.pgEntitiesOf([Users]) + const snapshot = yield* S.makeSnapshot(entities, [S.ORIGIN_ID], "postgres") + return yield* Migrate.verifySnapshot(snapshot) +}) +``` + ### Running one step at a time `run` applies everything in one process. An orchestrator that runs each statement as its own diff --git a/docs/reference.md b/docs/reference.md index fe8f4eb..5c5788f 100644 --- a/docs/reference.md +++ b/docs/reference.md @@ -511,12 +511,12 @@ placeholders, double-quoted identifiers. | `ClickHouseStatement`, `parseStatement`, `renderStatement`, `withSettings`, `withFormat` | `/sql` | | `ClickHouseStatementFromString`, `splitTerminalClauses`, `maskLiteralsAndComments` | `/sql` | | Schema tooling: `isSchemaObject`, `makeSnapshot`, `renderSchema`, `diffSchemas`, `diffPgSchemas`, `fromDrizzleSnapshot`, snapshot and entity schemas (`PgEnumEntity`, `PgUniqueEntity`, …), `Hint`; for SQLite, `sqliteEntitiesOf`, `renderSqliteSchema`, `renderSqliteCreateTable`, `renderSqliteCreateIndex`, `diffSqliteSchemas`, `fromDrizzleSqliteSnapshot`, `SqliteMigrationOp`, `renderSqliteOp`, `renderSqliteMigrationSql`, `labelOfSqlite`, `rebuildName`, the `SqliteSnapshot` schema and the `SqliteTableEntity`, `SqliteColumnEntity`, `SqliteIndexEntity`, `SqliteUniqueEntity` schemas | `/schema` | -| `run`, `status`, `verify`, `baseline`, `MigrationDriver` and the migrate errors | `/migrate` | +| `run`, `status`, `verify`, `verifySnapshot`, `baseline`, `MigrationDriver` and the migrate errors | `/migrate` | | `Database`, `run`, `sql`, `query`, `execute`, `transaction`, `requireTransaction`, `afterCommit`, `retryContention`, `Transaction` | `/database` | | `DatabaseError`, `TransactionCommitFailed`, `TransactionRollbackFailed` and the other transaction errors | `/database` | | `withQuerySummary`, `QuerySummary` — the `db.query.summary` of the statements an effect runs, for spans | `/database` | | `ClickhouseHttpClient`, `layer`, `make`, `ClickhouseHttpClientConfig` — a ClickHouse `SqlClient` over `fetch`, for Workers | `/clickhouse-http` | -| `defineConfig`, `generate`, `check`, `pull`, `loadSchema`, `readMigrations`, `analyze`, `KitError` | `/kit` | +| `defineConfig`, `generate`, `check`, `pull`, `verifySchema`, `loadSchema`, `readMigrations`, `analyze`, `KitError` | `/kit` | | `defineSuite`, `query`, `caseFromCompiled`, `runSuite`, `compareRuns`, `compareBudgets` | `/benchmark` | | `Suite`, `RunOutput`, `BenchmarkError` and benchmark contracts | `/benchmark` | | `makeHttpClient`, `makeHttpTransport`, `httpConfigFromEnv` | `/benchmark/http` | diff --git a/src/kit.ts b/src/kit.ts index 5ffef5f..c72aa0c 100644 --- a/src/kit.ts +++ b/src/kit.ts @@ -17,4 +17,5 @@ export { type KitConfig, } from "./kit/generate" export { pull, type PullOptions, type PullResult } from "./kit/pull" +export { verifySchema, type VerifySchemaResult } from "./kit/verify-schema" export { runCli } from "./kit/cli" diff --git a/src/kit/__fixtures__/pglite-driver.ts b/src/kit/__fixtures__/pglite-driver.ts new file mode 100644 index 0000000..5035035 --- /dev/null +++ b/src/kit/__fixtures__/pglite-driver.ts @@ -0,0 +1,15 @@ +// A config `driver` for CLI tests: a fresh in-memory PGlite database that +// holds what `statements` create. + +import { PgliteClient } from "@effect/sql-pglite" +import { Effect, Layer } from "effect" +import * as SqlClient from "effect/sql/SqlClient" +import * as Migrate from "../../migrate" + +export const pgliteDriver = (statements: ReadonlyArray) => + Layer.effectDiscard( + Effect.gen(function* () { + const sql = yield* SqlClient.SqlClient + for (const statement of statements) yield* sql.unsafe(statement) + }), + ).pipe(Layer.provideMerge(Migrate.layerSqlClient()), Layer.provide(PgliteClient.layer({}))) diff --git a/src/kit/cli.ts b/src/kit/cli.ts index 3b7cdec..c8db778 100644 --- a/src/kit/cli.ts +++ b/src/kit/cli.ts @@ -8,6 +8,7 @@ import * as Migrate from "../migrate" import { Hints, type Hint } from "../schema/diff" import { check, generate, KitError, readMigrations, type KitConfig } from "./generate" import { pull } from "./pull" +import { verifySchema } from "./verify-schema" const help = `effect-orm: schema migrations for ClickHouse and Postgres, and generated SQLite migrations @@ -21,7 +22,8 @@ const help = `effect-orm: schema migrations for ClickHouse and Postgres, and gen check Validate the migrations folder (snapshot chain, branch conflicts) migrate [--strict] Apply pending migrations (needs config.driver; not SQLite yet) status Applied, pending, partial, or changed, per migration - verify Compare the database with the last applied snapshot + verify [--schema] Compare the database with the last applied snapshot; with --schema, with + the schema modules, without reading the migrations folder or the ledger baseline Record and every one before it as applied, without running them resolve Record what a step left "started" did, after checking the database: --ran | --not-ran --ran skips it on the next migrate, --not-ran runs it @@ -43,6 +45,7 @@ const options = { out: { type: "string" }, tables: { type: "string" }, force: { type: "boolean" }, + schema: { type: "boolean" }, ran: { type: "boolean" }, "not-ran": { type: "boolean" }, json: { type: "boolean" }, @@ -77,6 +80,9 @@ const promptConfirm = (hint: Hint): Effect.Effect => return /^y(es)?$/i.test(answer.trim()) }) +const describeDrift = (d: Migrate.Drift): string => + ` ${d.entity}: ${d.problem}${d.expected !== undefined || d.actual !== undefined ? ` (expected ${d.expected ?? "-"}, actual ${d.actual ?? "-"})` : ""}` + const withDriver = (config: KitConfig, body: Effect.Effect) => config.driver === undefined ? Effect.fail(new KitError({ code: "config", message: "this command needs a database: set `driver` in the config" })) @@ -197,6 +203,13 @@ const program = (args: ReadonlyArray, print: (line: string) => void) => case "migrate": case "status": case "verify": { + if (command === "verify" && parsed.values.schema === true) { + const result = yield* verifySchema(config, cwd) + out(result, () => + result.drift.length === 0 ? ["No drift against the schema modules."] : ["Drift against the schema modules:", ...result.drift.map(describeDrift)], + ) + return result.drift.length === 0 ? 0 : 3 + } const migrations = yield* readMigrations(resolve(cwd, config.out)) const render = config.render ?? {} const dialect = config.dialect !== undefined ? { dialect: config.dialect } : undefined @@ -216,7 +229,7 @@ const program = (args: ReadonlyArray, print: (line: string) => void) => ? ["No applied migration with a snapshot to compare against."] : result.drift.length === 0 ? [`No drift against ${result.against}.`] - : [`Drift against ${result.against}:`, ...result.drift.map((d) => ` ${d.entity}: ${d.problem}${d.expected !== undefined || d.actual !== undefined ? ` (expected ${d.expected ?? "-"}, actual ${d.actual ?? "-"})` : ""}`)], + : [`Drift against ${result.against}:`, ...result.drift.map(describeDrift)], ) return result.drift.length === 0 ? 0 : 3 } diff --git a/src/kit/generate.ts b/src/kit/generate.ts index c1d6f8e..d68224c 100644 --- a/src/kit/generate.ts +++ b/src/kit/generate.ts @@ -90,6 +90,29 @@ export const loadSchema = (config: KitConfig, cwd: string): Effect.Effect, KitError> => + Effect.gen(function* () { + const objects = yield* loadSchema(config, cwd) + const dialect = config.dialect ?? "clickhouse" + const entities: Effect.Effect, SchemaDefinitionError> = + dialect === "postgres" + ? pgEntitiesOf(objects) + : dialect === "sqlite" + ? sqliteEntitiesOf(objects as ReadonlyArray>) + : entitiesOf(objects) + return yield* entities.pipe( + Effect.mapError( + (error) => + new KitError({ + code: "config", + message: "the schema definitions have problems", + details: error.problems.map((problem) => `${problem.object}: ${problem.message}`), + }), + ), + ) + }) + /** Read `//...` with node:fs. */ export const readMigrations = (out: string): Effect.Effect, KitError> => Effect.gen(function* () { @@ -219,23 +242,7 @@ export const generate = (config: KitConfig, cwd: string, options: GenerateOption snapshotEntities = imported.entities prevIds = [ORIGIN_ID] } else { - const objects = yield* loadSchema(config, cwd) - const entities: Effect.Effect, SchemaDefinitionError> = - dialect === "postgres" - ? pgEntitiesOf(objects) - : dialect === "sqlite" - ? sqliteEntitiesOf(objects as ReadonlyArray>) - : entitiesOf(objects) - const next = yield* entities.pipe( - Effect.mapError( - (error) => - new KitError({ - code: "config", - message: "the schema definitions have problems", - details: error.problems.map((problem) => `${problem.object}: ${problem.message}`), - }), - ), - ) + const next = yield* schemaEntities(config, cwd) snapshotEntities = next if (options.baseline === "schema") prevIds = [ORIGIN_ID] else if (options.custom === true) snapshotEntities = analysis.base diff --git a/src/kit/verify-schema.test.ts b/src/kit/verify-schema.test.ts new file mode 100644 index 0000000..ba44b1c --- /dev/null +++ b/src/kit/verify-schema.test.ts @@ -0,0 +1,142 @@ +// `verifySchema`: the schema modules against a live database, with no +// migrations folder and no ledger. + +import { mkdtempSync, rmSync, writeFileSync } from "node:fs" +import { tmpdir } from "node:os" +import { join, resolve } from "node:path" +import { PgliteClient } from "@effect/sql-pglite" +import { afterEach, beforeEach, describe, expect, it } from "@effect/vitest" +import { Effect, Layer } from "effect" +import * as SqlClient from "effect/sql/SqlClient" +import { runCli, verifySchema, type KitConfig } from "../kit" +import * as Migrate from "../migrate" +import { makeSnapshot, ORIGIN_ID } from "../schema" + +const src = resolve(import.meta.dirname, "..") + +const schemaModule = ` +import * as PG from "${src}/postgres" + +export const Users = PG.table("users", { + columns: { + id: PG.text, + email: PG.text, + name: PG.nullable(PG.text), + }, + primaryKey: ["id"], + indexes: [PG.uniqueIndex("users_email_unq", ["email"])], +}) +` + +const created = [ + `CREATE TABLE users (id text NOT NULL, email text NOT NULL, name text, CONSTRAINT users_pkey PRIMARY KEY (id))`, + `CREATE UNIQUE INDEX users_email_unq ON users (email)`, +] + +describe("verifySchema", () => { + let dir = "" + beforeEach(() => { + dir = mkdtempSync(join(tmpdir(), "effect-orm-verify-schema-")) + writeFileSync(join(dir, "schema.ts"), schemaModule) + }) + afterEach(() => rmSync(dir, { recursive: true, force: true })) + + const withDb = (body: (config: KitConfig) => Effect.Effect) => + Effect.gen(function* () { + const context = yield* Layer.build(Migrate.layerSqlClient().pipe(Layer.provideMerge(PgliteClient.layer({})))) + // The same database for the test body and for the config's driver. + const config: KitConfig = { dialect: "postgres", schema: "./schema.ts", out: "./migrations", driver: Layer.succeedContext(context) } + return yield* body(config).pipe(Effect.provideContext(context)) + }).pipe(Effect.scoped) + + it.effect("reads clean when the database matches the definitions, with no migrations folder", () => + withDb((config) => + Effect.gen(function* () { + const sql = yield* SqlClient.SqlClient + for (const statement of created) yield* sql.unsafe(statement) + expect(yield* verifySchema(config, dir)).toEqual({ drift: [] }) + }), + ), + ) + + it.effect("reports a missing column, a type change, a nullability change and a missing index", () => + withDb((config) => + Effect.gen(function* () { + const sql = yield* SqlClient.SqlClient + yield* sql.unsafe(`CREATE TABLE users (id text NOT NULL, email varchar(10), CONSTRAINT users_pkey PRIMARY KEY (id))`) + const { drift } = yield* verifySchema(config, dir) + expect(drift.map((d) => `${d.problem} ${d.entity}`).sort()).toEqual([ + "missing users.name", + "missing users.users_email_unq", + "not_null users.email", + "type users.email", + ]) + expect(drift.find((d) => d.problem === "type")).toMatchObject({ expected: "text", actual: "character varying(10)" }) + }), + ), + ) + + it.effect("refuses a SQLite config before building its driver, with no catalog reader to compare with", () => + Effect.gen(function* () { + // A driver that would fail to build: the refusal must come first. + const config: KitConfig = { dialect: "sqlite", schema: "./schema.ts", out: "./migrations", driver: Layer.effectContext(Effect.die("built")) } + const error = yield* Effect.flip(verifySchema(config, dir)) + expect(error).toMatchObject({ _tag: "@maple-dev/effect-orm/KitError", code: "unsupported" }) + expect(error.message).toMatch(/SQLite or D1/) + }), + ) + + it.effect("verifySnapshot refuses a SQLite snapshot rather than comparing it as ClickHouse", () => + Effect.gen(function* () { + const snapshot = yield* makeSnapshot([], [ORIGIN_ID], "sqlite") + const error = yield* Effect.flip(Migrate.verifySnapshot(snapshot).pipe(Effect.provide(Migrate.layerSqlClient()), Effect.provide(PgliteClient.layer({})))) + expect(error).toBeInstanceOf(Migrate.MigrateSourceError) + expect(error.message).toMatch(/SQLite databases cannot be verified/) + }), + ) +}) + +describe("effect-orm verify --schema", () => { + let dir = "" + let lines: Array = [] + const cli = (...args: Array) => { + const cwd = process.cwd() + process.chdir(dir) + return runCli(args, (line) => lines.push(line)).finally(() => process.chdir(cwd)) + } + /** A config whose driver is a fresh database holding what `statements` create. */ + const config = (name: string, statements: ReadonlyArray) => + writeFileSync( + join(dir, name), + `import { pgliteDriver } from "${src}/kit/__fixtures__/pglite-driver" +export default { dialect: "postgres", schema: "./schema.ts", out: "./migrations", driver: pgliteDriver(${JSON.stringify(statements)}) } +`, + ) + + beforeEach(() => { + dir = mkdtempSync(join(tmpdir(), "effect-orm-verify-schema-cli-")) + lines = [] + writeFileSync(join(dir, "schema.ts"), schemaModule) + }) + afterEach(() => rmSync(dir, { recursive: true, force: true })) + + it("exits 0 when the database matches, with no migrations folder", async () => { + config("clean.config.ts", created) + expect(await cli("verify", "--schema", "--config", "clean.config.ts")).toBe(0) + expect(lines).toEqual(["No drift against the schema modules."]) + }) + + it("exits 3 and prints one line per difference", async () => { + config("drift.config.ts", [`CREATE TABLE users (id text NOT NULL, email text, CONSTRAINT users_pkey PRIMARY KEY (id))`]) + expect(await cli("verify", "--schema", "--config", "drift.config.ts")).toBe(3) + expect(lines).toEqual([ + "Drift against the schema modules:", + " users.email: not_null (expected NOT NULL, actual NULL)", + " users.name: missing", + " users.users_email_unq: missing", + ]) + lines = [] + expect(await cli("verify", "--schema", "--json", "--config", "drift.config.ts")).toBe(3) + expect(JSON.parse(lines[0]!).drift).toHaveLength(3) + }) +}) diff --git a/src/kit/verify-schema.ts b/src/kit/verify-schema.ts new file mode 100644 index 0000000..816349f --- /dev/null +++ b/src/kit/verify-schema.ts @@ -0,0 +1,53 @@ +// `effect-orm verify --schema`: the schema modules against a live database. +// +// `verify` compares the database with the snapshot of the last applied +// migration, so it needs the migrations folder and the ledger. This compares +// it with the table definitions as they stand: for a database that another +// tool migrates, or none, and for a CI check that the code and the database +// agree. The comparison itself is `verifySnapshot`, the one `verify` uses. + +import { Effect, Layer } from "effect" +import { verifySnapshot, type Drift, type VerifyOptions } from "../migrate/verify" +import type { MigrateSourceError, MigrateSqlError } from "../migrate/errors" +import { ORIGIN_ID } from "../schema/entities" +import { makeSnapshot } from "../schema/snapshot" +import { KitError, schemaEntities, type KitConfig } from "./generate" + +export interface VerifySchemaResult { + readonly drift: ReadonlyArray +} + +/** + * Compare the database behind `config.driver` with the definitions in + * `config.schema`. Reads no migrations folder and no ledger; the ledger tables + * are never reported. + */ +export const verifySchema = ( + config: KitConfig, + cwd: string, + options: Pick = {}, +): Effect.Effect => + Effect.gen(function* () { + const driver = config.driver + if (driver === undefined) { + return yield* new KitError({ code: "config", message: "this command needs a database: set `driver` in the config" }) + } + // `generate` writes SQLite DDL, but nothing reads a SQLite catalog back yet. + if (config.dialect === "sqlite") { + return yield* new KitError({ + code: "unsupported", + message: "verify --schema cannot check a SQLite or D1 database yet: effect-orm has no SQLite catalog reader to compare the definitions with", + }) + } + const entities = yield* schemaEntities(config, cwd) + const snapshot = yield* makeSnapshot(entities, [ORIGIN_ID], config.dialect ?? "clickhouse") + // Built apart, so a driver that cannot start fails as a KitError and the + // comparison keeps its own typed errors. + const context = yield* Layer.build(driver).pipe( + Effect.mapError( + (cause) => new KitError({ code: "config", message: `Cannot build the config's driver: ${cause instanceof Error ? cause.message : String(cause)}` }), + ), + ) + const drift = yield* verifySnapshot(snapshot, options, "the schema modules").pipe(Effect.provideContext(context)) + return { drift } + }).pipe(Effect.scoped) diff --git a/src/migrate.ts b/src/migrate.ts index 1ca8f24..b7e1b90 100644 --- a/src/migrate.ts +++ b/src/migrate.ts @@ -49,4 +49,4 @@ export { type ResolveStepOptions, type RunOptions, } from "./migrate/run" -export { verify, type Drift, type VerifyOptions, type VerifyResult } from "./migrate/verify" +export { verify, verifySnapshot, type Drift, type VerifyOptions, type VerifyResult } from "./migrate/verify" diff --git a/src/migrate/verify.ts b/src/migrate/verify.ts index 1ee74d0..f40b7ff 100644 --- a/src/migrate/verify.ts +++ b/src/migrate/verify.ts @@ -1,4 +1,5 @@ -// Drift: the live schema against the snapshot of the last applied migration. +// Drift: the live schema against the snapshot of the last applied migration +// (`verify`), or against any snapshot (`verifySnapshot`). // // Not against the code's latest schema: a database three migrations behind is // behind, not drifted. Expressions and types are normalized by the server @@ -12,10 +13,10 @@ import { Effect } from "effect" import { quoteClickHouseString } from "../sql/sql-fragment" -import type { SchemaDialect, ClickHouseSnapshot, ColumnEntity, IndexEntity, MaterializedViewEntity, TableEntity } from "../schema/entities" +import type { SchemaDialect, ClickHouseSnapshot, ColumnEntity, IndexEntity, MaterializedViewEntity, Snapshot, TableEntity } from "../schema/entities" import { MigrationDriver } from "./driver" import { readClickHouseCatalog, splitEngineFull, stripDatabase, type CatalogRow } from "./introspect" -import type { MigrateSourceError, MigrateSqlError } from "./errors" +import { MigrateSourceError, type MigrateSqlError } from "./errors" import { LEDGER_TABLES, readApplied } from "./ledger" import { readPgApplied } from "./pg-ledger" import { verifyPg, type PgVerifyOptions } from "./pg-verify" @@ -123,11 +124,33 @@ export const verify = ( const against = [...migrations].reverse().find((m) => appliedNames.has(m.name) && m.snapshot !== undefined) const snapshot = against?.snapshot if (against === undefined || snapshot === undefined) return { against: undefined, drift: [] } - if (snapshot.dialect === "postgres") return { against: against.name, drift: yield* verifyPg(snapshot, against.name, options) } if (snapshot.dialect === "sqlite") return yield* refuseSqlite(migrations, "sqlite").pipe(Effect.as({ against: undefined, drift: [] })) - return { against: against.name, drift: yield* verifyClickHouse(snapshot) } + return { against: against.name, drift: yield* verifySnapshot(snapshot, options, against.name) } }) +/** + * Compare the database with any snapshot, with no migrations folder and no + * ledger: the one `makeSnapshot` builds from the table definitions, say. The + * dialect is the snapshot's. `verify` is this, with the snapshot of the last + * applied migration. `label` names the snapshot in a Postgres error. + */ +export const verifySnapshot = ( + snapshot: Snapshot, + options: PgVerifyOptions = {}, + label = "snapshot", +): Effect.Effect, MigrateSqlError | MigrateSourceError, MigrationDriver> => + snapshot.dialect === "postgres" + ? verifyPg(snapshot, label, options) + : snapshot.dialect === "sqlite" + ? // No SQLite catalog reader yet: refused rather than compared as ClickHouse. + Effect.fail( + new MigrateSourceError({ + migration: label, + message: "SQLite databases cannot be verified by effect-orm yet; there is no SQLite catalog reader to compare the definitions with", + }), + ) + : verifyClickHouse(snapshot) + const verifyClickHouse = (snapshot: ClickHouseSnapshot): Effect.Effect, MigrateSqlError, MigrationDriver> => Effect.gen(function* () { const driver = yield* MigrationDriver diff --git a/tests/migrate.clickhouse.test.ts b/tests/migrate.clickhouse.test.ts index 9796bc1..4ffdfe6 100644 --- a/tests/migrate.clickhouse.test.ts +++ b/tests/migrate.clickhouse.test.ts @@ -315,5 +315,27 @@ describe("migrate", () => { ), ) }) + + it("compares a snapshot of the definitions with a database that has no ledger", async () => { + await Effect.runPromise( + withDatabase( + Effect.gen(function* () { + // Created by hand, as another tool would: no migrations folder, no ledger. + const entities = yield* S.entitiesOf([Events, Totals, TotalsMv]) + const sql = yield* ClickhouseClient.ClickhouseClient + for (const statement of S.renderSchema(entities)) yield* sql.asCommand(sql.unsafe(statement)) + const snapshot = yield* S.makeSnapshot(entities, [S.ORIGIN_ID]) + expect(yield* Migrate.verifySnapshot(snapshot)).toEqual([]) + + yield* sql.asCommand(sql.unsafe("ALTER TABLE events MODIFY COLUMN Name Nullable(String)")) + yield* sql.asCommand(sql.unsafe("ALTER TABLE events DROP INDEX idx_name")) + expect(yield* Migrate.verifySnapshot(snapshot)).toEqual([ + { entity: "events.Name", problem: "type", expected: "String", actual: "Nullable(String)" }, + { entity: "events.idx_name", problem: "missing" }, + ]) + }), + ), + ) + }) }) })