Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/dialect-in-context.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"@maple-dev/effect-orm": minor
---

`CurrentDialect`, a `Context.Reference`, sets the dialect a `compile` writes for when no `options.dialect` is given: `CH.compile(query, params).pipe(Effect.provideService(CH.CurrentDialect, postgresDialect))`. The order is `options.dialect`, then `CurrentDialect`, then the entry's dialect. A compile's state (its dialect, syntax, bound values and failures) is now kept in the context of the fiber that runs it, not in module state, so compiles for two dialects in concurrent fibers stay independent.
5 changes: 5 additions & 0 deletions .changeset/type-level-dialect.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"@maple-dev/effect-orm": minor
---

A query that cannot compile for its target is now a type error, not only a failure at compile time. A built-in function of one dialect (`CH.countIf`, `PG.countIf`, `SQLite.count`, ...) marks its result with `ClickHouseOnly`, `PostgresOnly` or `SqliteOnly` (SQLite and D1 share one set), and `compile`, `compileUnion` and `run` refuse a query with a function of another dialect (`wrongDialect`), or with functions of two (`mixedDialects`). Row locks (`forUpdate`, `forShare`, ...) are refused the same way for ClickHouse, SQLite and D1 (`unsupportedClause`), as only Postgres writes them. The target is `options.dialect`, else the entry's dialect; `fromSqlClient` now returns a `DatabaseApi<"postgres">`, `DatabaseApi<"clickhouse">` or `DatabaseApi<"sqlite">` whose `run` checks it. A column ref now names its source (`SourceEntry`), and a query refuses a column of a source that is not in its FROM or JOIN (`sourceNotInQuery`); a correlated subquery can use the outer query's accessor. The runtime checks stay. Also, `orderBy(($) => [...])` now carries the params of every sort key, not only the first.
15 changes: 12 additions & 3 deletions design/dialects.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,10 @@ What we deliberately do not copy:
every `Str` fragment, column literal, and inline param. `compile` installs the dialect for
the length of the compile (`withDialect`, the same save/restore pattern as
`withSubqueryCompiler`), because literals are written inside the query callbacks, which
have no dialect argument. The `__PARAM_` safety is now enforced rather than assumed: a
have no dialect argument. The installed values live in the compiling fiber's context
(`src/sql/fiber-local.ts`), not in module state, so two compiles in two fibers cannot see
each other's dialect. The dialect is `CurrentDialect`, a `Context.Reference` a caller can
also provide. The `__PARAM_` safety is now enforced rather than assumed: a
literal that contains the marker fails with `InvalidLiteral`, so a dialect with weaker
escaping cannot turn a value into a placeholder.
3. **Identifier quoting (done).** Column refs are `Ident` fragments carrying their qualifier
Expand Down Expand Up @@ -82,7 +85,13 @@ What we deliberately do not copy:
- The package is still named for ClickHouse and the root entry still exports the ClickHouse
function catalog. Renaming, or a `core` entry without ClickHouse functions, is a packaging
decision for a later release.
- The ClickHouse function catalog writes ClickHouse SQL whatever the dialect. Functions could
refuse to compile under another dialect instead of producing SQL that fails at the server.
- The ClickHouse function catalog writes ClickHouse SQL whatever the dialect. A built-in
function now refuses to compile under another dialect, and is a type error there too: its
result carries a `DialectEntry` beside its params, which `compile` and
`DatabaseApi<Functions>.run` check against the target. The `Database` in context has no
dialect in its type, so it checks only that a query does not mix two. A `CompiledQuery` does
not carry its dialect in the type yet. A clause only some dialects write rides the same
channel as a `ClauseEntry` (the row locks, `RowLock`, Postgres only). `format` and
`distinctOn` could use it too; they are refused only at compile time for now.
- MySQL or SQLite would need `?` placeholders (`reuse: false` already exists) and their own
types and functions; nothing in the core should need to change.
2 changes: 1 addition & 1 deletion docs/database.md
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,7 @@ Calling `withdraw(1, 30)` outside `transfer` does not compile: `requireTransacti

| Export | What it is |
| --- | --- |
| `fromSqlClient(sql, options)` | A `DatabaseApi` over a client. `options.dialect` is required; `options.requireTenantScope` refuses unscoped queries ([Tenant scoping](./tenant-scoping.md#enforcing-it)) |
| `fromSqlClient(sql, options)` | A `DatabaseApi` over a client. `options.dialect` is required, and types it: with `postgresDialect` it is a `DatabaseApi<"postgres">`, whose `run` refuses ClickHouse and SQLite functions in the type (`d1Dialect` and `sqliteDialect` give a `DatabaseApi<"sqlite">`, which also refuses row locks) ([Checked in the type](./params-and-compilation.md#checked-in-the-type)); `options.requireTenantScope` refuses unscoped queries ([Tenant scoping](./tenant-scoping.md#enforcing-it)) |
| `layerSqlClient(options)` | A `Database` layer over the `SqlClient` in context |
| `Database` | The service |
| `run(query, params?)` | Compile a query for the database's dialect, run it, decode its rows |
Expand Down
5 changes: 5 additions & 0 deletions docs/joins-and-subqueries.md
Original file line number Diff line number Diff line change
Expand Up @@ -119,6 +119,11 @@ CH.from(Services, "s")
.where(($) => [$.OrgId.eq("org_123"), CH.exists(inner)])
```

Or build the inner query inside the outer callback and use the outer accessor: `e.Name.eq($.Name)`
writes `s.Name`, and the type checks that the outer query has a source `s` (see
[Checked in the type](./params-and-compilation.md#checked-in-the-type)). A string `outerRef` is
not checked.

All three also accept a pre-compiled SQL string, for the rare case where that is all you have.

> **A subquery never scopes its outer query.** `WHERE x IN (SELECT y FROM t WHERE OrgId = 'a')`
Expand Down
95 changes: 94 additions & 1 deletion docs/params-and-compilation.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,79 @@ says so: `defineFn`, `defineCondFn` and `compileTypedFnCall` from the extending
hand-written `makeExpr` does not. A param the type does not see is still checked when
compiling, as below.

## Checked in the type

The same channel carries three more facts, and `compile`, `compileUnion` and `run` check them
before the query runs.

**The dialect of a function.** A built-in function of one dialect marks its result:
`CH.countIf` with `ClickHouseOnly`, `PG.countIf` with `PostgresOnly`, `SQLite.count` with
`SqliteOnly` (SQLite and D1 share the `"sqlite"` set). A query that uses one is a type error
where it is compiled or run for another dialect:

```ts
const slow = CH.from(Events).select(($) => ({ n: CH.countIf($.DurationMs.gt(500)) }))

CH.compile(slow) // ok: ClickHouse is the /clickhouse default
PG.compile(slow) // type error: wrongDialect "this query uses ClickHouse functions; it cannot compile for the postgres dialect"
PG.compile(slow, {}, { dialect: CH.clickhouseDialect }) // ok: options.dialect decides
```

The target is `options.dialect` when it is given, else the entry's dialect. A `Database` built
with `fromSqlClient(client, { dialect: postgresDialect })` is a `DatabaseApi<"postgres">`, and
its `run` checks the same way. A dialect typed as a plain `Dialect` (a custom one, or the
`Database` in context) is not checked against one set, but a query with functions of both
dialects is refused everywhere (`mixedDialects`): no dialect can compile it. `/sqlite`'s
`compile` checks against `"sqlite"`, and so does a `DatabaseApi<"sqlite">` from
`fromSqlClient(client, { dialect: d1Dialect })`. Portable functions
(`nullIf`, `/clickhouse`'s `lower`) and the shared operators add nothing. `CurrentDialect` is
read at run time, so the type cannot see it: under a `CurrentDialect` of another dialect, pass
`options.dialect` too. The compile-time `QueryBuilderDefect` stays, for a caller past the types.

**A clause only some dialects write.** A row lock (`forUpdate`, `forNoKeyUpdate`, `forShare`,
`forKeyShare`) adds a `RowLock` (`ClauseEntry<"FOR UPDATE", "postgres">`): only Postgres has row
locks, so compiling or running the query for ClickHouse, SQLite or D1 is a type error
(`unsupportedClause "FOR UPDATE has no meaning for the sqlite dialect"`). On SQLite and D1, claim a
row with a conditional update instead (`update(…).set(…).where(($) => [$.status.eq("pending")]).returning(…)`),
or keep the writer single (a Durable Object). As with functions, a dialect typed as a plain `Dialect`
is not checked, and the compile still refuses the lock at run time.

**The source of a column.** A column ref names its source: the FROM alias, else the table
name, or the join alias. A query provides its own sources, so a column of a source it does not
have is a type error where it is compiled or run. That is a column of another query's accessor,
kept outside the callback it came from:

```ts
let services!: CH.JoinedColumnAccessor<(typeof Services)["columns"], {}, "s">
CH.from(Services, "s").select(($) => ((services = $), { id: $.Id }))

const leaked = CH.from(Events).select("Id").where(() => [services.Team.eq("core")])
CH.compile(leaked) // type error: sourceNotInQuery "a column of `s` is used, but no FROM or JOIN of the query is named `s`"
```

A subquery may read a source of the query it is nested in. The inner query carries the outer
source on, and the outer query provides it, so a correlated subquery can use the outer
accessor in place of a string `outerRef`:

```ts
CH.from(Services, "s")
.select(($) => ({ team: $.Team }))
.where(($) => [
$.OrgId.eq("org_123"),
CH.exists(
CH.from(Events)
.select((e) => ({ n: e.Name }))
.where((e) => [e.OrgId.eq("org_123"), e.Name.eq($.Name)]),
),
])
// ... WHERE EXISTS (SELECT events.Name AS n FROM events WHERE ... AND events.Name = s.Name)
```

A source whose name the type does not know (a `Table<string, …>`, a non-literal alias) is not
checked. Writes are not checked: their accessors do not name a source.

_(Backed by `src/ch/dialect.test-d.ts`, `src/ch/sources.test-d.ts` and `docs/params-and-compilation.md > A correlated subquery reads the outer accessor`.)_

## What each kind accepts

The declared kind is checked when the value arrives, so a value of the wrong shape is a
Expand Down Expand Up @@ -177,7 +250,7 @@ CH.compileUnsafe(query, params, options?) // CompiledQuery, throws
| `options.rowSchema` | Effect `Schema` used by `decodeRows` / `decodeFirstRow` |
| `options.skipFormat` | Omit a trailing `FORMAT` clause (used internally for subqueries) |
| `options.deferParams` | Leave placeholders unresolved, for SQL spliced into an outer compile |
| `options.dialect` | Overrides the entry's dialect (`clickhouseDialect` from `/clickhouse`, `postgresDialect` from `/postgres`) |
| `options.dialect` | Overrides `CurrentDialect` and the entry's dialect (`clickhouseDialect` from `/clickhouse`, `postgresDialect` from `/postgres`, `sqliteDialect` from `/sqlite`) |

Each dialect entry exports its own `compile`, defaulting to that dialect. Unions use
`compileUnion(union, params)`.
Expand Down Expand Up @@ -261,6 +334,26 @@ than `1`/`0`.
| `clauses.parenthesizeUnionBranches` | Whether each `UNION ALL` branch is wrapped in parentheses |
| `paramCodecs` | Per-kind codec overrides for `param.*` |

### The dialect in Effect context

`CurrentDialect` is a `Context.Reference`. Provide it, and every `compile` in that Effect writes
for that dialect when no `options.dialect` is given. Each fiber has its own value, so compiles
for two dialects can run at the same time in one program:

```ts
const compiled = yield* CH.compile(query, { orgId: "org_1" }).pipe(
Effect.provideService(CH.CurrentDialect, numbered),
)
// compiled.sql: ... WHERE OrgId = $1
// compiled.parameters: ["org_1"]
```

The order is: `options.dialect`, then `CurrentDialect`, then the entry's dialect. The rest of a
compile's state (its syntax, bound values and failures) is kept in the fiber's context too, not in
module state.

_(Backed by `docs/params-and-compilation.md > The dialect can come from Effect context`.)_

Params are resolved by rewriting placeholders in the finished SQL, so a dialect's literals must
never spell the param marker `__PARAM_`. ClickHouse writes it as `\x5F_PARAM_`; a literal that
does contain the marker fails the compile with `InvalidLiteral` rather than being rewritten.
Expand Down
9 changes: 5 additions & 4 deletions docs/postgres.md
Original file line number Diff line number Diff line change
Expand Up @@ -209,10 +209,11 @@ document shape, or to name the bad row in its own error); every other read stays
The shared operators (`eq`, `in_`, `like`, `ilike`, `and`, `or`, `not`, arithmetic, `lit`) work
unchanged. `/postgres` exports only functions Postgres has, plus `nullIf`, which renders the
same on both. The ClickHouse catalog on `/clickhouse` (`quantile`, `toStartOfInterval`, its
`count()`, …) writes ClickHouse SQL, so a query that uses one and is compiled for Postgres is a
`QueryBuilderDefect` naming the function; the Postgres functions above fail the same way on
ClickHouse. A custom `Dialect` opts in with `functions: "clickhouse"` or `"postgres"`; without
it, nothing is checked.
`count()`, …) writes ClickHouse SQL, so a query that uses one is a type error where it is
compiled or run for Postgres, and the Postgres functions above are one on ClickHouse (see
[Checked in the type](./params-and-compilation.md#checked-in-the-type)). Past the types, the
compile still fails with a `QueryBuilderDefect` naming the function. A custom `Dialect` opts in
with `functions: "clickhouse"` or `"postgres"`; without it, nothing is checked.

## Known differences

Expand Down
35 changes: 24 additions & 11 deletions docs/reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ Tables come from the dialect's `table`: [`CH.table`](#tables-and-ddl) or
| `limit(n)` / `offset(n)` | Rounded before emission |
| `format(fmt)` | `"JSON"` \| `"JSONEachRow"` |
| `distinct()` / `distinctOn(...aliases)` | `SELECT DISTINCT` / `SELECT DISTINCT ON (…)` |
| `forUpdate` / `forNoKeyUpdate` / `forShare` / `forKeyShare` | Postgres row locks; options `LockOptions` (`skipLocked`, `noWait`, `of`) |
| `forUpdate` / `forNoKeyUpdate` / `forShare` / `forKeyShare` | Postgres row locks; options `LockOptions` (`skipLocked`, `noWait`, `of`). Refused in the type when compiled or run for another dialect |
| `innerJoin` / `leftJoin` / `crossJoin` | `(table, alias, on?)` |
| `innerJoinQuery` / `leftJoinQuery` / `crossJoinQuery` | `(query, alias, on?)` |
| `withCTE(name, query)` / `withCTE(name, sql, options?)` | Typed query derives scope; SQL form can declare `options.tenantScope` |
Expand All @@ -88,7 +88,11 @@ to its own dialect; `options.dialect` overrides it.

`Dialect`, `DialectClauses` and `ParamStyle` describe a database: how identifiers and literals
are written, how params reach the server, and which clauses exist. Pass one as
`options.dialect`. `DialectTransactions` (with `IsolationLevel` and `TransactionSettings`) says
`options.dialect`, or provide it as `CurrentDialect`, a `Context.Reference<Dialect | undefined>`
that `compile` reads when no `options.dialect` is given. `Compile<Default>`, `CompileUnsafe<Default>`,
`CompileUnion<Default>` and `CompileUnionUnsafe<Default>` are the types of the four compile
functions, with `Default` the function set of the entry's dialect; their options are
`QueryCompileOptions`, `InsertCompileOptions` and `UnionCompileOptions`. `DialectTransactions` (with `IsolationLevel` and `TransactionSettings`) says
which transactions the database supports; see [Database](./database.md). See [Params and compilation](./params-and-compilation.md#dialects) and
[Postgres](./postgres.md).

Expand Down Expand Up @@ -182,17 +186,24 @@ decoded type of a column), `InferEncoded` (its wire type), `ColumnDefs`, `Nullab
**Inference** — `InferOutput`, `InferQueryOutput`, `InferUnionOutput`, `SelectRowOf`,
`InsertRow`, `InsertRowOf`, `UpdateSet`, `UpdateSetOf`.

**What `compile` checks** — `NeedsSelect<Output>` (the query has a SELECT list) and
`ParamsSatisfied<Params, Given>` (the params object fills every required param). A helper generic
**What `compile` checks** — `NeedsSelect<Output>` (the query has a SELECT list),
`ParamsSatisfied<Params, Given>` (the params object fills every required param),
`DialectSatisfied<Params, Functions>` (no built-in function of another dialect, and not two
dialects in one query) and `SourcesSatisfied<Params>` (no column of a source that is not in the
query's FROM or JOIN). See [Params and compilation](./params-and-compilation.md#checked-in-the-type). A helper generic
over a query's output takes `CHQuery<…, Output> & NeedsSelect<Output>` so it can pass the query on
to `compile`.

**Params a query carries** — `ParamEntry<Name, Value>` (one `param.*` a query needs, in its type),
`OptionalParamEntry<Name, Value>` and `OptionalParams<P>` (the same for a param under `when` /
`whenTrue`, or one whose name is a union, which the params object may leave out) and
`ParamsIn<X>` (the params an expression or argument list carries). You rarely write them, but
`tsc --declaration` names them when a package exports a query with a param or a `defineFn` helper,
so they are public.
`ParamsIn<X>` (the params an expression or argument list carries). Beside the params, the same
channel carries `DialectEntry<Set>` (a built-in function of one dialect: `ClickHouseOnly`,
`PostgresOnly` or `SqliteOnly`; `DialectsIn<P>` reads the sets), `ClauseEntry<Clause, Sets>` (a
clause only some dialects write: `RowLock<Clause>` for the Postgres row locks) and `SourceEntry<Name>` (a column of source
`Name`; `SourceOf<Name>` is the entry of a ref, `BindSources<P, Sources>` removes the ones a query
provides). You rarely write them, but `tsc --declaration` names them when a package exports a
query with a param or a `defineFn` helper, so they are public.

**Everything else** — `Table` (what `from` and the write builders accept; every `table` value is
one), `Expr`, `ColumnRef`, `Condition`, `Comparable` (what a value of a type may be compared
Expand Down Expand Up @@ -406,7 +417,8 @@ Types: `WindowSpec`, `CompiledWindowSpec`, `WindowFrameBound`, `WindowRowsFrame`
### Dialect

`compile` and the other compile functions default to `clickhouseDialect`: params written into the
SQL as ClickHouse literals.
SQL as ClickHouse literals. A `CurrentDialect` in context, or `options.dialect`, overrides
the default.

---

Expand Down Expand Up @@ -497,7 +509,8 @@ unit union), `dateBin(seconds, ts)`, `now()`, `lower(x)`, `upper(x)`, `length(x)
### Dialect

`compile` and the other compile functions default to `postgresDialect`: numbered `$n`
placeholders, double-quoted identifiers.
placeholders, double-quoted identifiers. A `CurrentDialect` in context, or `options.dialect`, overrides
the default.

---

Expand Down Expand Up @@ -535,8 +548,8 @@ conditions).
## SQLite and D1

`@maple-dev/effect-orm/sqlite` exports the shared query builder and the four compile functions.
`sqliteDialect` is the default; `d1Dialect` uses the same SQL with a 100-binding limit and no
interactive transactions. See [SQLite and D1](./sqlite.md) for driver wiring and limits.
`sqliteDialect` is the default; a `CurrentDialect` in context, or `options.dialect`, overrides it.
`d1Dialect` uses the same SQL with a 100-binding limit and no interactive transactions. See [SQLite and D1](./sqlite.md) for driver wiring and limits.

| Export | Purpose |
| --- | --- |
Expand Down
Loading
Loading