Repository navigation
expr: Accept stateful unary and binary functions in the sqlfunc macro - #38941
Conversation
9ef1418 to
abbd17d
Compare
abbd17d to
e9df1b0
Compare
QA LLM Review1. MEDIUM --
|
e9df1b0 to
053b0cf
Compare
|
Confirmed and fixed in 053b0cf. I reproduced it exactly as described: deleting the guard left the test green. The cause is worth recording, because it made an assertion that had been verified go stale. That assertion was live when it was written. Merging the scalar function registry from #38868 into this stack put the attribute text into the expansion through Two more assertions in the same file had the same problem, and they are older than this stack: All three were verified in both directions rather than only after the fix. Posted by Claude Code. |
236f0de to
12e3254
Compare
There was a problem hiding this comment.
Seems fine. My agent has some feedback---the most actionable of which is that a bunch of functions are now convertible into the macro (not just ListLengthMax) and we might as well do it as part of this work imo.
Details
Strong suggestions
- The new &self path for unary and binary never gets compiled against real code. RangeCreate is variadic, and the variadic arity already supported &self. So the new unary and binary receiver path (self_offset in generate.rs:196, the self.method(..) dispatch) is covered only by the unary_self and binary_self insta snapshots. Snapshots check the token output, not whether it type-checks. Converting one stateful unary function would fix this, since the trait change already touched them all (e.g. PadChar or CastInt32ToNumeric). One binary function would help too (e.g. ListLengthMax, which this PR marks with a TODO anyway). The registry diff would then show that the conversion preserves semantics, the same way it does for RangeCreate.
- The new skip_display/sqlname conflict error uses darling::Error::unknown_field. It renders as "Unknown field: sqlname has no effect with skip_display…", which is misleading because sqlname is a known field. The TODO at generate.rs:236 admits this constructor is wrong. This PR also drops the TODO's only stated reason for not fixing it (the snapshot pinning), yet adds a third call site. Use Error::custom for the new check at least, or fix all three now that nothing pins the text.
Nits
- sqlfunc.md breaks the repo's comment and doc rules. The "Shapes the macro does not cover" section says "The ten remaining…", "The nine impl<E: Eval> blocks…" and "which EagerUnaryFunc::call now supplies". CLAUDE.md asks docs to avoid chronology, and the hardcoded counts go stale with every conversion PR. The counts are correct today: I found 10 non-generic and 9 generic LazyUnaryFunc impls. Describe the categories and drop the counts and "now".
- The doc overstates when a struct name is required. sqlfunc.md says the struct name "is required when a &self receiver is present". In fact the macro falls back to the CamelCase of the function name (generate.rs:182). That fallback is why range_create → RangeCreate would work without the explicit RangeCreate. When the names don't match, the user gets an opaque "cannot find type" error instead of a macro error. Either reword the doc, or reject &self without an explicit name at unary and binary arity.
- skip_display = true is accepted without &self. The macro then generates a unit struct, which has no state for a hand-written Display to depend on, so the documented rationale doesn't apply. Consider rejecting it there.
- The list.rs TODO singles out ListLengthMax. About 35 hand-written stateful EagerUnaryFunc impls in this same diff are equally convertible now. Either drop the TODO or keep a single TODO in sqlfunc.md.
| | Effective params | Dispatches to | Notes | | ||
| |---|---|---| | ||
| | 0 | Error | Nullary functions are not supported. | | ||
| | 1 | `EagerUnaryFunc` | Does not support `&RowArena`. | | ||
| | 2 | `EagerBinaryFunc` | Supports `&RowArena`. | | ||
| | 1 | `EagerUnaryFunc` | Supports `&RowArena` and `&self`. | | ||
| | 2 | `EagerBinaryFunc` | Supports `&RowArena` and `&self`. | | ||
| | 3+ | `EagerVariadicFunc` | Supports `&RowArena` and `&self`. | |
There was a problem hiding this comment.
Now that they're uniform, couldn't we just say everyone supports &RowArena and &self?
There was a problem hiding this comment.
Yes. The table drops its notes column, and one sentence below it says every arity accepts a &self receiver and a trailing &RowArena. In 405659c.
Posted by Claude Code.
| ### Struct name | ||
|
|
||
| For variadic functions, the struct name can be specified as the first positional argument. | ||
| The struct name can be specified as the first positional argument, at any arity. |
There was a problem hiding this comment.
| The struct name can be specified as the first positional argument, at any arity. | |
| The struct name can be specified as the first positional argument to the `sqlfunc` macro. |
If it's at any arity, we don't need to mention arity at all.
There was a problem hiding this comment.
Applied, together with the correction from your review that the name is never strictly required: it defaults to the camel-cased function name, and with &self it must match the externally defined struct. In 405659c.
Posted by Claude Code.
|
|
||
| A trailing `&RowArena` parameter gives the function access to temporary storage for allocating return values that borrow from the arena. | ||
| It is excluded from arity detection and from the generated `Input` type. | ||
| The arena is always passed to unary, binary, and variadic `call` implementations (the trait requires it); for functions that don't use it, the parameter is simply unused. |
There was a problem hiding this comment.
Not sure this last sentence is adding anything.
There was a problem hiding this comment.
Agreed, dropped in 405659c.
Posted by Claude Code.
| /// The `list_length_max` implementation. | ||
| /// | ||
| /// We're not deriving `sqlfunc` here because we need to pass in the `max_layer` parameter. | ||
| /// TODO: convertible to `#[sqlfunc]`, which accepts a `&self` receiver. |
There was a problem hiding this comment.
Any reason not to just do this now?
There was a problem hiding this comment.
It's done in #39095, which converts list.rs and removes the TODO. It moved there because the plan put one PR per source file.
Posted by Claude Code.
12e3254 to
57ba070
Compare
57ba070 to
405659c
Compare
|
Thanks. Where each point landed:
Posted by Claude Code. |
405659c to
78897ac
Compare
78897ac to
a3df7ca
Compare
a3df7ca to
0482bcb
Compare
mgree
left a comment
There was a problem hiding this comment.
LGTM, just one clarification about introduces_nulls.
| pub offset: i64, | ||
| } | ||
| #[sqlfunc(ArrayIndex, sqlname = "array_index", introduces_nulls = true)] | ||
| #[sqlfunc(ArrayIndex, sqlname = "array_index")] |
There was a problem hiding this comment.
Just confirming: the drop of introduces_nulls is because we're inferring the correct (true) value based on the return type now?
There was a problem hiding this comment.
Yes. array_index returns Option<T>, and an Option return type sets the introduces_nulls default to true, so the modifier only restated it. The registry snapshot still reports introduces_nulls: true for array_index before and after this PR.
Posted by Claude Code.
`EagerBinaryFunc::call` and `EagerVariadicFunc::call` take an arena, unary did not. That gap blocks the compound-type casts that allocate their output, which have to stay hand-written `LazyUnaryFunc` impls to reach an arena at all. `EagerUnaryFunc::call` now takes `temp_storage: &'a RowArena`, and the blanket `LazyUnaryFunc` impl forwards the arena it already holds. `Shape::takes_arena` answers true for every arity, so `#[sqlfunc]` generates the parameter on unary `call` and forwards it when the annotated function asks for one. A unary function with a trailing `&RowArena` is therefore accepted rather than rejected. The hand-written unary impls take the parameter and ignore it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`#[sqlfunc(StructName, ..)]` on a function with a `&self` receiver now works for
unary and binary, as it already did for variadic. The macro attaches an inherent
method and the trait impl to the existing struct instead of defining a unit
struct.
Both arms previously indexed arguments from position zero, so the receiver reached
`arg_type` and produced `compile_error!("Unsupported argument type")`. Because the
arities share one generator, this is an offset and a call expression rather than
two copies of the variadic logic.
Also inlines `Shape::takes_arena`, which had become a constant-valued method with
a single call site now that every arity takes the arena, and drops the test that
looped over shapes asserting it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
generate's doc comment and the attribute-parsing comment in sqlfunc both described the struct-name argument as exclusive to variadic dispatch. Any arity accepts one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Twenty-four of the hand-written scalar functions format their name from struct
state, for example `extract_{unit}_ts`. Suppressing the generated `Display` lets
them keep the impl they already have, which is simpler than teaching `sqlname` to
evaluate an expression with `self` in scope.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The insta_test_skip_display assert checked for "impl std :: fmt :: Display" with spaces around the double colons, the raw TokenStream::to_string() idiom. But test_sqlfunc formats output through prettyplease, which never spaces double colons, so the assert could never fire and only the snapshot enforced suppression. Match the unspaced form prettyplease actually produces. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`RangeCreate` carries an `elem_type` field, so it needs the stateful shape the macro now accepts: the function takes `&self` and the macro generates the `call` body as a method on the existing struct. Its `Display` impl picks a name from `elem_type` rather than using a fixed string, so `skip_display` keeps the hand-written impl and the macro generates only `FuncName`. That removes the manual `func_name!` entry, which would otherwise collide with the generated impl. The output type is expressed with `output_type_expr` because it reads `elem_type` and ignores the input types, matching the hand-written body. The function returns `Range<Datum<'a>>` rather than packing the range itself, so it needs no arena and `introduces_nulls` derives the `false` the impl declared. The remaining trait methods derive the same values from the signature that they derived from the associated types before, so the generated impl matches. Also corrects two stale claims in the macro documentation: unary functions now accept a `&RowArena` and a `&self` receiver, and `output_type_expr` applies to all three arities. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three documents describe which modifiers apply to which arity: the field docs on `Modifiers`, the macro's rustdoc, and `doc/developer/sqlfunc.md`. They had drifted from the tables in `shape.rs`, which are the actual authority. Corrects `output_type_expr` to all arities, `is_infix_op` to binary and variadic, `is_monotone` to all arities, and adds the three modifiers the rustdoc omitted entirely. Moves the struct name, `&self` receiver, and `&RowArena` sections out from under the variadic heading, since all three apply at every arity, and states that the generated struct and the generated `Display` are each conditional. Adds the `non_nullable_input_is_nullable` term to the null-handling formula. The formula omitted it, so it could not be used to reason about a function whose parameters do not all accept NULL. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`sqlname` feeds only the generated `Display` impl, so pairing it with `skip_display` silently discards it. Rejecting the combination turns a modifier that does nothing into a compile error, and both modifier references now say so. Records the interlock behind unary's nullability tail. Unary omits the non-nullable-position term the other two arities carry, which is sound only because `propagates_nulls` is not a unary modifier, so unary's value always equals what that term would test. A test already pins that absence, but it did so without saying why, leaving a maintainer who deliberately adds the modifier free to delete the assertion. The assertion now names what depends on it. Corrects three claims the tree no longer supports. `is_infix_op` has a default on every `Eager*Func` trait, so it does not need to be specified. The nine `LazyUnaryFunc` implementations that stay hand-written are the ones generic over `Eval`, not every compound-type cast, and the ten others are convertible now that `EagerUnaryFunc::call` supplies a `RowArena`. `ListLengthMax` gave its `max_layer` field as the reason it could not use the macro, which a `&self` receiver now handles. Replaces the struct-name example, which showed a two-parameter signature that detects as binary under a name saying variadic, with the real `array_fill` signature. Normalizes the test module on `super::` for its own helper. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ntroduces_nulls under output_type_expr `inverse` and `negate` now name the function, and the macro wraps it as `Some(crate::UnaryFunc::from(..))` or `Some(crate::BinaryFunc::from(..))`. Every call site spelled that wrapper out, through `to_unary!` or a `Some(..into())` string, so the attribute now carries only what differs per function. `to_unary!` stays for the hand-written `impl<E: Eval>` casts. `output_type_expr` no longer requires `introduces_nulls`. Left out, it falls back to the trait default, `Output::nullable()`, which is sound: an output type that cannot hold NULL cannot produce one. Call sites whose explicit value equalled that default drop it. The ones that return a `Datum` or an `Option` but yield NULL only for NULL inputs keep `introduces_nulls = false`. The registry confirms that no declared property moved. `RangeCreate` returns `Range<Datum<'a>>` rather than packing the range itself, so it needs no arena and derives the `false` it declared. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…p_display on a unit struct The modifier conflict checks built their errors with `unknown_field`, which renders as "Unknown field: <message>" for fields the macro knows. They now use `Error::custom`, like every other legality error in the crate. `skip_display` exists so that a struct whose name depends on its state can keep a hand-written `Display`. Without a `&self` receiver the macro defines a unit struct, which has no such state, so the combination is now an error. The unary and binary `&self` paths were covered only by insta snapshots, which check the generated tokens but not that they compile. A test in `mz-expr` defines a stateful unary and a stateful binary function and evaluates both through their eager traits. The reference drops the per-row arity notes in favour of one sentence, states that the struct name defaults to the camel-cased function name with or without `&self`, and drops the counts and the chronology from the section on shapes the macro does not cover. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
With every arity accepting a receiver, `Arity` carried the same `has_self` in each variant and `sqlfunc` repeated the `generate` call per arity. `determine_shape` returns the `Shape` directly, and `signature::sql_args` owns the rule that a receiver and a trailing `&RowArena` are not SQL arguments, which `determine_arity` and `variadic_params` each spelled out with their own index arithmetic. The three diagnostics that started with a capital letter are lowercase, like the rest of the crate's. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
`generate` built the output column type and the `introduces_nulls` method in two passes, first from `output_type` and then overriding both from `output_type_expr` and `introduces_nulls`. `output_type_and_introduces_nulls` decides both in one place from the three modifiers and the generic-signature derivation. The generated code is unchanged, as the snapshots show. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The test module had grown to most of `lib.rs`. It moves to `test.rs` under the same module name, so the insta snapshot names stay the same. The three tests without the Miri reason now carry it like the rest, and the two public test helpers get doc comments. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The reference said the generated trait implementations live in the trait modules, that `output.nullable` comes from `introduces_nulls`, and that `output_type` generates an `output_type` method. It had no entry for `is_eliminable_cast`, two examples repeated the function name as `sqlname`, and several lines used em-dashes and arrows as sentence structure. The `output_type_expr` entry also names the bindings the expression can read. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Unary and binary read their parameter types with `arg_type`, which indexed past the receiver by hand and added `'a` only to a top-level reference, so a unary `Option<&str>` parameter produced an `Input` type without a lifetime. Variadic already used `sql_args` with `patch_lifetimes`. `sql_params` now does that for every arity and keeps the positional `a` and `b` names for unary and binary. `generate` derives the receiver itself instead of taking it as a flag, `is_variadic_arg` moves next to the other type checks in `signature.rs`, and the arena parameter of `call` is written inline, since every arity has it. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Modifier checks were split between `reject_inapplicable`, two `skip_display` checks in `generate`, and the `output_type` conflict in the output-type resolution. Each stopped at the first problem and reported it at the whole attribute. `Modifiers::validate` runs them all before generation, reports every error, and attaches each to the modifier it concerns. `Modifiers::get` returns one method-producing modifier, so `override_methods` walks the arity table directly instead of searching a list in another order, and `Modifiers::iter` goes away. `ReturnTy::to_tokens` is renamed to `tokens`, since it does not implement `quote::ToTokens`. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The two arms of `Shape::nullability` computed the same expression and differed only in whether they bound it to `is_null` first. They are one arm now, and the variadic snapshots pick up the binding. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
`derive_output_type_for_generics` classified each generic against the output type to pick one, and `derive_output_type_for_generic` classified the chosen one again, with an `Absent` check its only caller made unreachable. The usage is now passed along. The error for a generic missing from the inputs names the generic instead of always saying `T`. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* The `sqlfunc` rustdoc in `mz-expr-derive` listed every modifier and had drifted: `negate` still took `Some(BinaryFunc::..)` and `output_type_expr` still required `introduces_nulls`. It points at `doc/developer/sqlfunc.md`, which owns that reference. * `sqlfunc.md` put the `test` modifier's snapshots in the macro crate, although they land beside the calling file. It gave a different reason than `Shape::nullability` for the missing unary position term, and it did not describe the output-type derivation for generic signatures. * The `sqlfunc_source` doc called the lifetime-patched parameter types "as written". * `sqlfunc_for_test` was a one-line wrapper with one caller. The binary arena fixture was named `unary_fn`, and the variadic modifier fixture borrowed `Greatest`, which the macro does not cover. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
0482bcb to
97a1efe
Compare
) The blanket `LazyBinaryFunc` impl for `EagerBinaryFunc` fed `InputDatumType::try_from_iter` an iterator built from a closure, which is a new type for every function and every `Eval` implementor. Evaluating both arguments into an array gives every call site one iterator type, so argument unwrapping is instantiated once per input type. Hand-written `EagerBinaryFunc` impls get this too. `cargo llvm-lines -p mz-expr` drops from 1,786,549 to 1,703,578 lines, 4.6% fewer. Results don't change for two arguments, the only count planning produces, and the macro snapshots are untouched. Based on #38941. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
EagerUnaryFunc::callreceives a&RowArena, and the#[sqlfunc]macro accepts a&selfreceiver and a trailing&RowArenaat unary and binary arity, not only at variadic. This unblocks the hand-written functions that need state or an arena. A newskip_displaymodifier keeps a hand-writtenDisplayfor names that depend on state.inverseandnegatetake the function itself, andoutput_type_exprno longer requiresintroduces_nulls.RangeCreateis converted as the first user, and the macro reference indoc/developer/sqlfunc.mdis corrected.The last commits tidy the macro crate. Modifier validation happens in one place and reports every error at its modifier, all arities take their parameters the same way, so a nested reference in a unary or binary parameter gets the trait's lifetime, and the tests move to
test.rs.The registry reports only
range_create'ssqlfunc_signaturemoving. No declared property changed.Based on #38929.
🤖 Generated with Claude Code