feat(kit): verify --schema checks the table definitions against a live database

.changeset/verify-schema.md

@@ -0,0 +1,5 @@
+---
+"@maple-dev/effect-orm": minor
+---
+
+`effect-orm verify --schema` compares the table definitions with a live database, with no migrations folder and no ledger. It exits 3 on drift and prints one line per difference (`--json` for machines), so CI fails on a missing column, a type or nullability change, or a missing index. It reads the config's `schema` and `driver` and runs the same comparison as `verify`, for ClickHouse and Postgres; a SQLite or D1 config is refused, since there is no SQLite catalog reader yet. In code: `verifySchema(config, cwd)` from `/kit`, and `Migrate.verifySnapshot(snapshot)` to compare a database with any snapshot.

design/migrations.md

@@ -348,6 +348,10 @@ CREATE TABLE IF NOT EXISTS _effect_orm_migration_steps (
 `Migrate.verify` introspects (`system.tables`, `system.columns`, `system.data_skipping_indices`,
 `create_table_query` through `formatQuery()`) and diffs the live schema against the snapshot **of
 the last applied migration**, not the code's HEAD. That reports drift and partial applies honestly.
+`verify --schema` (`Kit.verifySchema`) is the other question: do the definitions at HEAD match
+this database? It builds a snapshot from the schema modules and runs the same comparison
+(`Migrate.verifySnapshot`), with no folder and no ledger, for a database another tool migrates
+and for a CI check after deploy.
 
 ### 4.6 CLI
 
@@ -359,7 +363,7 @@ One bin, `effect-orm` (next to `ch-bench`):
 | `check`                  | no             | DAG validity, recomputed ids, name/DAG order, commutativity, backfill convergence lint, shipped-migration immutability against a git base (`--base origin/main`) |
 | `migrate`                | yes            | Runs `Migrate.run` with a `SqlClient` layer exported from the config file |
 | `status` / `plan`        | yes            | Pending migrations and their rendered steps, no execution  |
-| `verify`                 | yes            | 4.5                                                         |
+| `verify [--schema]`      | yes            | 4.5; `--schema` compares with the definitions, not the ledger |
 | `push`                   | yes            | Dev only: introspect, diff, apply, no files                |
 | `pull [--init]`          | yes            | Introspect into `defineTable` source; `--init` baselines   |
 

docs/migrations.md

@@ -8,7 +8,7 @@ into migrations:
 | Entry                              | Runs where         | What it does                                                                  |
 | ---------------------------------- | ------------------ | ----------------------------------------------------------------------------- |
 | `@maple-dev/effect-orm/schema`     | anywhere, pure     | reads `table` / `materializedView` values: DDL rendering, snapshots, the diff |
-| `@maple-dev/effect-orm/kit`        | Node or Bun        | `generate` and `check` over a migrations folder; the `effect-orm` command     |
+| `@maple-dev/effect-orm/kit`        | Node or Bun        | `generate` and `check` over a migrations folder, `verifySchema`; the `effect-orm` command |
 | `@maple-dev/effect-orm/migrate`    | anywhere Effect runs | applies migrations through a driver you provide, `status`, `verify`        |
 
 Both ClickHouse and Postgres. The model follows drizzle-kit (a committed snapshot per migration,
@@ -255,6 +255,52 @@ both sides (`formatQuery`, `defaultValueOfTypeName`). It checks tables, engine f
 columns, skipping indexes, and view targets and bodies; it does not check TTL, codecs, settings,
 or comments yet. `effect-orm verify` exits 3 when it finds drift.
 
+### Checking the definitions against a database
+
+`verify` needs the migrations folder and the ledger. To check the table definitions against a
+database that another tool migrates, or in CI after a deploy, compare it with the definitions
+directly:
+
+```sh
+effect-orm verify --schema          # exit 0: no drift; exit 3: drift, one line per difference
+effect-orm verify --schema --json   # {"drift":[{"entity":"users.email","problem":"not_null",...}]}
+```
+
+```text
+Drift against the schema modules:
+  users.email: type (expected text, actual character varying(10))
+  users.name: missing
+  users.users_email_unq: missing
+```
+
+It reads the config's `schema` modules and `driver`, and nothing in `out`. The comparison is the
+one `verify` makes, with the same checks and the same limits, against a snapshot of the
+definitions as they stand instead of the last applied one. So a database that is behind reports
+the missing parts as drift; use plain `verify` where the ledger exists. The ledger tables are
+never reported. ClickHouse and Postgres only: effect-orm has no SQLite catalog reader yet, so
+with `dialect: "sqlite"` (SQLite or D1) it fails with a `KitError` (`unsupported`) before it
+connects. In code, `verifySchema(config, cwd)` from `/kit` does the same, and
+`Migrate.verifySnapshot(snapshot)` compares the database with any snapshot (a SQLite snapshot
+fails with `MigrateSourceError`):
+
+```ts title="migrations-verify-schema.ts"
+import { Effect } from "effect"
+import * as PG from "@maple-dev/effect-orm/postgres"
+import * as Migrate from "@maple-dev/effect-orm/migrate"
+import * as S from "@maple-dev/effect-orm/schema"
+
+const Users = PG.table("users", {
+	columns: { id: PG.text, email: PG.text },
+	primaryKey: ["id"],
+})
+
+export const drift = Effect.gen(function* () {
+	const entities = yield* S.pgEntitiesOf([Users])
+	const snapshot = yield* S.makeSnapshot(entities, [S.ORIGIN_ID], "postgres")
+	return yield* Migrate.verifySnapshot(snapshot)
+})
+```
+
 ### Running one step at a time
 
 `run` applies everything in one process. An orchestrator that runs each statement as its own

docs/reference.md

@@ -511,12 +511,12 @@ placeholders, double-quoted identifiers.
 | `ClickHouseStatement`, `parseStatement`, `renderStatement`, `withSettings`, `withFormat`         | `/sql`            |
 | `ClickHouseStatementFromString`, `splitTerminalClauses`, `maskLiteralsAndComments`               | `/sql`            |
 | Schema tooling: `isSchemaObject`, `makeSnapshot`, `renderSchema`, `diffSchemas`, `diffPgSchemas`, `fromDrizzleSnapshot`, snapshot and entity schemas (`PgEnumEntity`, `PgUniqueEntity`, …), `Hint`; for SQLite, `sqliteEntitiesOf`, `renderSqliteSchema`, `renderSqliteCreateTable`, `renderSqliteCreateIndex`, `diffSqliteSchemas`, `fromDrizzleSqliteSnapshot`, `SqliteMigrationOp`, `renderSqliteOp`, `renderSqliteMigrationSql`, `labelOfSqlite`, `rebuildName`, the `SqliteSnapshot` schema and the `SqliteTableEntity`, `SqliteColumnEntity`, `SqliteIndexEntity`, `SqliteUniqueEntity` schemas | `/schema`         |
-| `run`, `status`, `verify`, `baseline`, `MigrationDriver` and the migrate errors                  | `/migrate`        |
+| `run`, `status`, `verify`, `verifySnapshot`, `baseline`, `MigrationDriver` and the migrate errors | `/migrate`        |
 | `Database`, `run`, `sql`, `query`, `execute`, `transaction`, `requireTransaction`, `afterCommit`, `retryContention`, `Transaction` | `/database` |
 | `DatabaseError`, `TransactionCommitFailed`, `TransactionRollbackFailed` and the other transaction errors | `/database` |
 | `withQuerySummary`, `QuerySummary` — the `db.query.summary` of the statements an effect runs, for spans | `/database` |
 | `ClickhouseHttpClient`, `layer`, `make`, `ClickhouseHttpClientConfig` — a ClickHouse `SqlClient` over `fetch`, for Workers | `/clickhouse-http` |
-| `defineConfig`, `generate`, `check`, `pull`, `loadSchema`, `readMigrations`, `analyze`, `KitError` | `/kit`            |
+| `defineConfig`, `generate`, `check`, `pull`, `verifySchema`, `loadSchema`, `readMigrations`, `analyze`, `KitError` | `/kit`            |
 | `defineSuite`, `query`, `caseFromCompiled`, `runSuite`, `compareRuns`, `compareBudgets`          | `/benchmark`      |
 | `Suite`, `RunOutput`, `BenchmarkError` and benchmark contracts                                   | `/benchmark`      |
 | `makeHttpClient`, `makeHttpTransport`, `httpConfigFromEnv`                                       | `/benchmark/http` |

src/kit.ts

@@ -17,4 +17,5 @@ export {
 	type KitConfig,
 } from "./kit/generate"
 export { pull, type PullOptions, type PullResult } from "./kit/pull"
+export { verifySchema, type VerifySchemaResult } from "./kit/verify-schema"
 export { runCli } from "./kit/cli"

src/kit/__fixtures__/pglite-driver.ts

@@ -0,0 +1,15 @@
+// A config `driver` for CLI tests: a fresh in-memory PGlite database that
+// holds what `statements` create.
+
+import { PgliteClient } from "@effect/sql-pglite"
+import { Effect, Layer } from "effect"
+import * as SqlClient from "effect/sql/SqlClient"
+import * as Migrate from "../../migrate"
+
+export const pgliteDriver = (statements: ReadonlyArray<string>) =>
+	Layer.effectDiscard(
+		Effect.gen(function* () {
+			const sql = yield* SqlClient.SqlClient
+			for (const statement of statements) yield* sql.unsafe(statement)
+		}),
+	).pipe(Layer.provideMerge(Migrate.layerSqlClient()), Layer.provide(PgliteClient.layer({})))

src/kit/cli.ts

@@ -8,6 +8,7 @@ import * as Migrate from "../migrate"
 import { Hints, type Hint } from "../schema/diff"
 import { check, generate, KitError, readMigrations, type KitConfig } from "./generate"
 import { pull } from "./pull"
+import { verifySchema } from "./verify-schema"
 
 const help = `effect-orm: schema migrations for ClickHouse and Postgres, and generated SQLite migrations
 
@@ -21,7 +22,8 @@ const help = `effect-orm: schema migrations for ClickHouse and Postgres, and gen
   check                            Validate the migrations folder (snapshot chain, branch conflicts)
   migrate [--strict]               Apply pending migrations (needs config.driver; not SQLite yet)
   status                           Applied, pending, partial, or changed, per migration
-  verify                           Compare the database with the last applied snapshot
+  verify [--schema]                Compare the database with the last applied snapshot; with --schema, with
+                                   the schema modules, without reading the migrations folder or the ledger
   baseline <migration>             Record <migration> and every one before it as applied, without running them
   resolve <migration> <step>       Record what a step left "started" did, after checking the database:
           --ran | --not-ran        --ran skips it on the next migrate, --not-ran runs it
@@ -43,6 +45,7 @@ const options = {
 	out: { type: "string" },
 	tables: { type: "string" },
 	force: { type: "boolean" },
+	schema: { type: "boolean" },
 	ran: { type: "boolean" },
 	"not-ran": { type: "boolean" },
 	json: { type: "boolean" },
@@ -77,6 +80,9 @@ const promptConfirm = (hint: Hint): Effect.Effect<boolean> =>
 		return /^y(es)?$/i.test(answer.trim())
 	})
 
+const describeDrift = (d: Migrate.Drift): string =>
+	`  ${d.entity}: ${d.problem}${d.expected !== undefined || d.actual !== undefined ? ` (expected ${d.expected ?? "-"}, actual ${d.actual ?? "-"})` : ""}`
+
 const withDriver = <A, E>(config: KitConfig, body: Effect.Effect<A, E, Migrate.MigrationDriver>) =>
 	config.driver === undefined
 		? Effect.fail(new KitError({ code: "config", message: "this command needs a database: set `driver` in the config" }))
@@ -197,6 +203,13 @@ const program = (args: ReadonlyArray<string>, print: (line: string) => void) =>
 			case "migrate":
 			case "status":
 			case "verify": {
+				if (command === "verify" && parsed.values.schema === true) {
+					const result = yield* verifySchema(config, cwd)
+					out(result, () =>
+						result.drift.length === 0 ? ["No drift against the schema modules."] : ["Drift against the schema modules:", ...result.drift.map(describeDrift)],
+					)
+					return result.drift.length === 0 ? 0 : 3
+				}
 				const migrations = yield* readMigrations(resolve(cwd, config.out))
 				const render = config.render ?? {}
 				const dialect = config.dialect !== undefined ? { dialect: config.dialect } : undefined
@@ -216,7 +229,7 @@ const program = (args: ReadonlyArray<string>, print: (line: string) => void) =>
 						? ["No applied migration with a snapshot to compare against."]
 						: result.drift.length === 0
 							? [`No drift against ${result.against}.`]
-							: [`Drift against ${result.against}:`, ...result.drift.map((d) => `  ${d.entity}: ${d.problem}${d.expected !== undefined || d.actual !== undefined ? ` (expected ${d.expected ?? "-"}, actual ${d.actual ?? "-"})` : ""}`)],
+							: [`Drift against ${result.against}:`, ...result.drift.map(describeDrift)],
 				)
 				return result.drift.length === 0 ? 0 : 3
 			}

src/kit/generate.ts

@@ -90,6 +90,29 @@ export const loadSchema = (config: KitConfig, cwd: string): Effect.Effect<Readon
 		return mine
 	})
 
+/** The entities the schema modules define, for the config's dialect. */
+export const schemaEntities = (config: KitConfig, cwd: string): Effect.Effect<ReadonlyArray<AnySchemaEntity>, KitError> =>
+	Effect.gen(function* () {
+		const objects = yield* loadSchema(config, cwd)
+		const dialect = config.dialect ?? "clickhouse"
+		const entities: Effect.Effect<ReadonlyArray<AnySchemaEntity>, SchemaDefinitionError> =
+			dialect === "postgres"
+				? pgEntitiesOf(objects)
+				: dialect === "sqlite"
+					? sqliteEntitiesOf(objects as ReadonlyArray<SqliteSchemaTable<string, any>>)
+					: entitiesOf(objects)
+		return yield* entities.pipe(
+			Effect.mapError(
+				(error) =>
+					new KitError({
+						code: "config",
+						message: "the schema definitions have problems",
+						details: error.problems.map((problem) => `${problem.object}: ${problem.message}`),
+					}),
+			),
+		)
+	})
+
 /** Read `<out>/<name>/...` with node:fs. */
 export const readMigrations = (out: string): Effect.Effect<ReadonlyArray<LoadedMigration>, KitError> =>
 	Effect.gen(function* () {
@@ -219,23 +242,7 @@ export const generate = (config: KitConfig, cwd: string, options: GenerateOption
 			snapshotEntities = imported.entities
 			prevIds = [ORIGIN_ID]
 		} else {
-			const objects = yield* loadSchema(config, cwd)
-			const entities: Effect.Effect<ReadonlyArray<AnySchemaEntity>, SchemaDefinitionError> =
-				dialect === "postgres"
-					? pgEntitiesOf(objects)
-					: dialect === "sqlite"
-						? sqliteEntitiesOf(objects as ReadonlyArray<SqliteSchemaTable<string, any>>)
-						: entitiesOf(objects)
-			const next = yield* entities.pipe(
-				Effect.mapError(
-					(error) =>
-						new KitError({
-							code: "config",
-							message: "the schema definitions have problems",
-							details: error.problems.map((problem) => `${problem.object}: ${problem.message}`),
-						}),
-				),
-			)
+			const next = yield* schemaEntities(config, cwd)
 			snapshotEntities = next
 			if (options.baseline === "schema") prevIds = [ORIGIN_ID]
 			else if (options.custom === true) snapshotEntities = analysis.base