[fupelaqu]

fix(sql): a qualified GROUP BY key resolves to its bucket in HAVING

Story IDENT-4.

Summary

BI tools qualify every column: SELECT g.status, COUNT(*) … FROM idx g GROUP BY g.status HAVING g.status = 'a'. Which grouping level a HAVING key condition addresses used to be decided by the
level's DISPLAY name, and a display name is not an identity:

• a qualified spelling, or either mixed spelling (GROUP BY g.status HAVING status = 'a', GROUP BY status HAVING g.status = 'a'), never matched its level, so the condition was refused;
• a self-join's two levels (GROUP BY o.category, p.category) share a name after one resolution, so
  a condition on one correlation filtered both levels, or the wrong one;
• two functions of one column as two keys (GROUP BY YEAR(d), MONTH(d)) share a name, so a
  condition on MONTH(d) also filtered the YEAR level and the result was empty;
• at a venue that resolves the statement only once (a remembered schema miss, a mapping that cannot
  be read), a qualified key's metrics never reached its level and came back NULL.

What changes:

• ONE decision, SingleSearch.bucketFor, recorded when an identifier is resolved: the alias it was
  written with; else the GROUP BY key with the same correlation and the same grouped expression (an
  unqualified name being the main table's column); else the spelling lookup, when it is compatible.
  It reads a positional list of the buckets (keyedBuckets) that never collapses two of them;
  bucketNames is derived from it.
• ONE comparison, Bucket.sameKeyAs (display name, correlation, grouped expression), read by the
  three include sites and by keyBucketOf, which now asks the recorded attachment first. An argument
  of a nameless GROUP BY key (UPPER(b.c), CONCAT(a.c, b.c)) keeps the name match it always had.
• A qualified GROUP BY key's copy is resolved like its level, so its metrics stay on their level
  after one resolution.
• The function chain of a qualified column resolves with the column (the qualified-plain branch of
  GenericIdentifier.update), as in every other branch: a key script reads doc['d'], never
  doc['q.d'].
• The HAVING rules see what the key filter applies: an OR across two levels is keyed by the level's
  GROUP BY rendering, so two functions of one column are two levels; a MATCH is a leaf of every
  rule; a key filter on a level Elasticsearch groups with a date histogram is refused, with the
  WHERE remedy. That last refusal is structural, not a type rule: the level the bridges emit for a
  DATE_TRUNC key, or any key whose function returns a date, is a date histogram, which has no key
  filter, whatever the column's mapping. Bucket.isDateHistogram is the one predicate both bridges
  and the classifier read.
• In a statement the relational engine evaluates (a JOIN, a derived table, a CTE), the OR rule
  compares mechanisms only and the two key-filter rules (a direct comparison only; one union per
  list) do not apply: nothing nests there, no terms filter is built, and the engine evaluates the
  condition after the join.
• A WHERE comparison no longer looks at whether its column is also a GROUP BY key to choose its
  Painless: a temporal comparison stays a date comparison (it took the numeric fallback, a raw > on
  two dates, exactly when the column was grouped).

Before → after

statement (FROM idx g, keyword key)	venue	before	after
GROUP BY g.k HAVING g.k = 'x' (key unaliased)	concrete index, index pattern	refused	answered
GROUP BY g.k HAVING k = 'x' / GROUP BY k HAVING g.k = 'x'	every venue	refused	answered
SELECT g.k AS a … GROUP BY g.k HAVING g.k = 'x'	concrete index, index pattern	answered	answered
same, one resolution (remembered schema miss)	one resolution	metrics NULL	answered
GROUP BY g.k HAVING COUNT(*) > 35	one resolution	its metric never reached the level (the selector reads a missing path)	answered
GROUP BY g.k HAVING MATCH (g.k) AGAINST ('x') OR COUNT(*) > 35	every venue	wrong (intersection) or Elasticsearch error	refused
GROUP BY YEAR(d), MONTH(d) HAVING MONTH(d) = 1	one resolution	empty	answered
GROUP BY YEAR(d), MONTH(d) HAVING YEAR(d) = 2024 AND MONTH(d) = 1	one resolution	refused	answered
GROUP BY YEAR(d), MONTH(d) HAVING YEAR(d) = 2024 OR MONTH(d) = 1	every venue	wrong (the AND) at one resolution, an Elasticsearch error at a mapped index	refused
GROUP BY DATE_TRUNC(d, MONTH) HAVING DATE_TRUNC(d, MONTH) = '2024-01-01'	every venue	condition ignored	refused, WHERE remedy
GROUP BY o.k, p.k HAVING p.k = 'x' (self-join)	in process, both resolutions	refused	the p level only
JOIN: GROUP BY c1, c2 HAVING c1 = 'x' OR c2 = 'y', UPPER(d.k) = 'X', d.k > 'm'	in process	refused	accepted (as v0.23.0)

"Before" is measured (RUN on 8.18.3 and 6.8.23 before this change, or RENDER where stated). "After"
is verified in process (PARSE / RENDER at both resolutions) and at RUN on 8.18.3 and 6.8.23 by
GroupByCompletenessSpec on the final code (concrete index, index pattern, one-resolution venue).

Release notes

1. A qualified GROUP BY key in HAVING now filters, in every spelling: GROUP BY g.status HAVING g.status = 'a', GROUP BY g.status HAVING status = 'a' and GROUP BY status HAVING g.status = 'a' keep exactly the groups GROUP BY status HAVING status = 'a' keeps (they were refused). This
   includes a JOIN (for example HAVING <function>(<aggregate>) silently drops the entire HAVING clause — every group is returned #389's GROUP BY d.dept_name HAVING d.dept_name = 'eng'), which the
   relational engine executes; JOIN execution is verified by the softclient4es-arrow JOIN tests after
   merge. One exception: a key whose function renders its own argument (CAST, TRY_CAST,
   SAFE_CAST, ISNULL, ISNOTNULL) is matched by its spelling, so its two mixed spellings
   stay refused in a statement Elasticsearch groups (in a JOIN, a derived table or a CTE they are
   accepted and left to the relational engine); write it the same way in GROUP BY and HAVING.
2. On a search resolved only once — an index queried before it existed, inside the schema-miss
   window, or an index whose mapping cannot be read — a qualified GROUP BY key's metrics are per
   group again (they came back NULL), and a HAVING over them is answered
   (GROUP BY g.category HAVING COUNT(*) > 35 keeps its two groups — RUN at the one-resolution
   venue on 8.18.3 and 6.8.23). Index patterns are resolved twice since fix(sql): date arithmetic follows the SQL engines; type checks wait for the column types #412 and were already right.
3. Two functions of one column as two keys (GROUP BY YEAR(d), MONTH(d)): a condition on one no
   longer empties the result, and the AND of conditions on both is answered at a one-resolution
   venue (it was refused); at a mapped index a date key fails for the reason recorded below.
4. Newly refused — the answer was wrong, or an Elasticsearch error, before — the bare spelling
   included: an OR across two functions of one column (YEAR(d) = 2024 OR MONTH(d) = 1, which
   Elasticsearch ran as an AND); a key condition on a DATE_TRUNC key or on a key whose function
   returns a date (Elasticsearch groups it with a date histogram, which has no key filter: on a
   DATE_TRUNC key the condition was ignored, and the other keys failed with an Elasticsearch
   error); a MATCH … AGAINST on a key combined by OR with an aggregate, or over
   two keys (MATCH (g, h)), in a statement Elasticsearch groups.
5. Messages: a condition on another table's column of a JOIN now reads "neither a GROUP BY key nor
   an aggregate"; a qualified key condition in a materialized view gets the view's own refusal; the
   "DIFFERENT GROUP BY keys" refusal names the keys as the GROUP BY writes them; a materialized view
   whose HAVING is a MATCH on its grouping key is refused (its transform has no key filter, so
   the MATCH was dropped).
6. A function of a qualified column (DATE_PARSE(q.d, …) and every other registry function) reads
   the resolved field after one resolution: its key script reads doc['d'], not doc['q.d']
   (render-level evidence; no answer is claimed while date keys at a mapped index fail for the
   reason recorded below).
7. A temporal WHERE comparison of a GROUP BY key column no longer depends on the column being
   grouped. With a computed date (WHERE q.d > DATE_ADD(q.e, INTERVAL 1 DAY) … GROUP BY q.d),
   qualified or bare, it is rendered as a date comparison at both resolutions (render-level; the
   bare spelling rendered a raw >, which Painless refuses); at a one-resolution venue both
   spellings keep the one group SQL keeps (RUN on 8.18.3; the rows run on Elasticsearch 7 and up,
   7.17 and 9.0 in CI). A comparison of two date
   columns (WHERE q.d > q.e … GROUP BY q.d) is a date comparison once the mapping is read; at one
   resolution neither column is typed, and it stays a raw >, as on main. No claim is made for
   Elasticsearch 6.8.
8. softclient4es-arrow: JOIN / derived-table / CTE statements with a key condition — stated only once
   the softclient4es-arrow JOIN tests pass against this change after merge.
9. A JOIN grouping one table's column beside a nameless key over another table's same-named column
   (GROUP BY a.category, UPPER(b.category) HAVING UPPER(b.category) = 'X') is accepted, as in
   v0.23.0.
10. In a JOIN, an OR across two GROUP BY keys (GROUP BY c1, c2 HAVING c1 = 'x' OR c2 = 'y'),
    refused on main since the key rules were added, is accepted again, as in v0.23.0: the relational
    engine evaluates it after the join.
11. In a JOIN, the key conditions HAVING <function>(<aggregate>) silently drops the entire HAVING clause — every group is returned #389 refused — a function of the key (UPPER(d.dept_name) = 'ENG'), a range (d.dept_name > 'm'), an AND on one key — are accepted again, as in v0.23.0.

• No downstream rebuild: additive only, no case-class arity change.

Tests

Evidence levels: PARSE / VALIDATE / RENDER (the bytes sent) at one resolution (R1: Parser.apply; a
remembered schema miss, an unreadable mapping) and two (R2: a concrete index, an index pattern since
#412; updateAll for a JOIN); RUN on a real cluster.

In process (plain sbt, Scala 2.13):

suite	tests
sql/test	1,882 passed, 0 failed
softclient4es-sql-bridge/test	286 passed, 0 failed
es6bridge/test	294 passed, 0 failed
core/test	1,261 passed, 0 failed
softclient4es-core-testkit/compile	ok

• QualifiedGroupKeySpec (new): a derived population — 3 FROM shapes (one table, two indices, a
  self-join) × 3 qualifiers × 11 groupings × 3 HAVING spellings × 8 operators = 264 cells, each at
  R1 and R2, against an oracle stated from the SQL alone (168 accepted, 96 refused, 0 wrong), plus
  the same population with the key aliased in the SELECT; the self-join in both directions and
  after a second and a third resolution; GROUP BY <alias> over a qualified column; two functions
  of one column (the condition on one, the AND, the OR); every registry function key (387) in four
  spellings; the wrapping venues (derived table, CTE, WHERE subquery, materialized view); the
  date-histogram rule; MATCH in every rule (including a view and a date key); one-key AND/OR
  mixes in every spelling; nameless JOIN keys; the JOIN rows the relational engine evaluates and
  their single-index controls; UNNEST keys, tableQualifierOf against GenericIdentifier.update
  over a derived name population, window partition keys.
• QualifiedGroupKeyEmissionSpec (new, both bridges): the bytes sent are the bare spelling's at
  both resolutions; the self-join filter on its own level; the metrics on their level after one
  resolution; every registry function key over FROM t q reads doc['d'] and never doc['q.d']
  (774 renders, 648 key scripts); the 40 keys whose function returns a date are emitted as a date
  histogram, and a key condition on each is refused; a temporal WHERE comparison is the one its
  ungrouped twin emits and a date comparison; Bucket.isDateHistogram is true exactly where the
  bridge emits a date histogram. On the Elasticsearch 6.8 bridge, the 6.8 oracle keeps the same keys
  for every spelling, and the 6.8 refusal set is refused by the serializer in every spelling.
• BucketIncludesSpec (the integer-key check in every spelling), SchemaAttachSpec (the body a
  one-resolution venue sends carries the metrics and the key filter), HavingNullAwareSelectorSpec
  (a MATCH over two keys is now refused by the OR rule).

Mutations (one decision each, the file restored by bytes and cmp): 29 decisions, each RED except as stated — the three lookup rules, the
identity (correlation and expression), the qualifier-resolved copies, the mirror of update's
qualifier rule, the three conjuncts of sameKeyAs, the attachment-first keyBucketOf and its name
arm, the level key of the OR rule, the date-histogram arm, the three MATCH arms, the chain
resolution of the qualified branch, the WHERE comparison, the explicit two-correlation dedup, the
relational-engine scoping of the OR rule and of the two key-filter rules, the nameless-key
exception: 28 RED (1 to 22 failing tests each). The "no unqualified fast path" mutation is GREEN by
design (an optimisation; its evidence is the cost table).

Differential (27,234 inputs: the repository corpus, the population and its wrappers, every
registry function key in nine shapes, and every statement the tests write; this tree vs main and
main vs main): no statement without a GROUP BY moved; no render moved (the bytes a materialized
view persists are unchanged for every input); no isScriptField moved; the statements whose two
resolutions disagree went from 1,398 to 3 (none new); main vs main is byte-identical. Every move is
classified — refused → accepted: 367 qualified key conditions (and their wrappers), 29 JOIN key conditions as in
v0.23.0, 1 JOIN OR across two aliased keys, 1 AND of two functions of one column; refused → refused
with another named reason: 170 to "neither a GROUP BY key nor an aggregate", 121 to the date-histogram
refusal, 120 to the materialized view's key refusal, 4 others (a qualified one-key mix now refused as
its bare twin is; a qualified OR across two functions; two mixed spellings of ISNULL / ISNOTNULL
keys, listed below); accepted → refused: listed below; accepted, body changed: 434 a qualified key's
metrics back on their level at one resolution, 24 a function key script reading the resolved field,
24 a qualified MATCH channel present at one resolution, 3 a condition no longer copied onto a
sibling function level, 2 the WHERE date comparison. The relational-engine statements newly refused:
18, each a MATCH over a column no key of its table groups (the relational engine has no MATCH),
plus the two derived-table / CTE bodies above (Elasticsearch groups the body). The WHERE change alone (this tree with and without it): no aggs /
bucket_selector byte moved on either bridge; the four WHERE moves are temporal comparisons of a
grouped column. The differential's second resolution attaches no mapping (outside the population),
so a comparison of two date columns (WHERE q.d > q.e GROUP BY q.d) is byte-identical to main
there, a raw > at both of its resolutions; its date comparison once the mapping is read is
asserted by QualifiedGroupKeyEmissionSpec at the second resolution with a date mapping (RENDER).

Accepted before, refused now (each with what it answered before):

• an OR across two functions of one column, and the same in a derived-table or CTE body: RUN at a
  one-resolution venue answered the AND [(3,3,6)] where SQL means three groups (both majors;
  fixture group_by_one_resolution_dates);
• a MATCH on the key OR an aggregate: RUN answered the intersection {cat_37} where SQL means
  {cat_36, cat_37} (concrete index and pattern, both majors), and an Elasticsearch error for the
  qualified spelling at one resolution (fixture group_by_completeness);
• a MATCH over two keys (GROUP BY g, h HAVING MATCH (g, h) AGAINST ('x'), which
  HavingNullAwareSelectorSpec pinned as accepted): RENDER, the pattern on both nested levels, i.e.
  the AND of the two, where SQL means the OR; now refused by the OR rule, its message quoting the
  MATCH as written;
• a materialized view whose HAVING is a MATCH on its key: source level, the view's transform has
  no key filter, so the MATCH was dropped (not run: the extensions venue);
• a MATCH in a JOIN over a column no key of its table groups: it matched another table's key by
  name; the relational engine has no MATCH;
• a key condition on a date-histogram level (42 inputs): RENDER, a date histogram with no filter for
  every one; RUN, DATE_TRUNC returned all three months (concrete, pattern, one resolution, both
  majors), the others fail with an Elasticsearch error.

Integration legs (Docker, sbt with JDK 17), final code, once each:

leg	tests
es8java/testOnly *JavaClientGroupByCompletenessSpec (8.18.3)	93 passed, 0 failed
es6rest/testOnly *RestHighLevelClientGroupByCompletenessSpec (6.8.23)	93 passed, 0 failed

Every IDENT-4 row passes on both majors against the same oracle — the four spellings × eight
operators at the concrete index, the index pattern and the one-resolution venue, a qualified key's metrics and a HAVING over them,
GROUP BY <select alias> over a qualified column, the WHERE subquery, the alias remedy, the qualified fractional-key check, the Elasticsearch 6 refusal
set, the date-level refusal and its WHERE remedy, the MATCH refusals, the OR across two functions of one column — except the two
one-resolution DATE_ADD rows, which run on Elasticsearch 7 and up only.

Cost

Median µs over 5 rounds, rotated this tree / main / main again (the last pair is the noise floor),
same session, warm JIT; the range is min–max over the rounds. Measured, never asserted.

probe · shape	this tree	main	main again
parse · bare SELECT (the per-round baseline)	60.0 [58.8–62.7]	59.4 [57.5–63.8]	60.5 [58.0–65.1]
parse · GROUP BY a.category HAVING a.category = 'x'	199.4 [193.3–208.1]	204.8 [189.4–210.5] (refused)	196.1 [188.6–223.0] (refused)
parse · GROUP BY category HAVING a.category = 'x'	202.1 [193.6–209.8]	204.7 [190.0–207.9] (refused)	195.4 [190.4–225.0] (refused)
parse · self-join … HAVING p.category = 'x'	398.1 [383.5–409.2]	407.8 [379.4–418.6] (refused)	386.6 [383.1–434.7] (refused)
parse · GROUP BY category HAVING category = 'x' (no qualifier)	195.4 [189.5–203.5]	203.0 [189.4–206.7]	190.9 [188.1–216.3]
parse · GROUP BY + HAVING (existing shape)	616.7 [594.6–621.6]	616.8 [572.0–640.8]	586.6 [577.8–670.1]
emission · 21 shapes	within the floor (largest move −8.5 µs, faster)
resolution probe · 87 shapes	within the floor
resolve (resolveWithSchema) · group by + having	9.8 [8.8–27.3]	8.4 [8.3–33.5]	8.8 [8.3–32.5]
resolve · GROUP BY e.code HAVING e.code = 'x'	3.8 [3.7–4.0]	refused on main	refused on main

Every shape's range overlaps main's, except one emission shape where this tree is faster. The
largest median increases (WHERE heavy: +21 µs parse, +33 µs in the resolution probe) are on a
statement with no GROUP BY, where the new lookup returns at its first test, and they sit inside
main's own min–max.
The qualified shapes are refused on main, so their main column is the cost of a refusal, not of an
answer. Not measured: the row path (response parsing is untouched).

Recorded, not fixed

• A date GROUP BY key at a venue that resolves twice (a concrete index, and since fix(sql): date arithmetic follows the SQL engines; type checks wait for the column types #412 an index
  pattern or a comma list): once the mapping is attached the level is a date_histogram with no
  interval unless the key is one DATE_TRUNC, and Elasticsearch refuses it (8.18: "Invalid interval
  specified"; 6.8: "Zero or negative time interval not supported"). Pre-existing at a concrete index;
  at a pattern it is new since fix(sql): date arithmetic follows the SQL engines; type checks wait for the column types #412 (v0.23.0 resolved a pattern once and sent terms). This change
  does not widen it (an error stays an error). It is fixed by its own change before the 0.24.0 tag;
  Bucket.isDateHistogram is the one predicate it changes. The RUN rows over date keys at the
  two-resolution venues are therefore not exercised here; they move to that change.
• A function key whose function is dropped from the emitted level at one resolution:
  DATE_FORMAT, WEEKDAY, DAYOFWEEK, FORMAT, ISNULL, … over a column are emitted as terms
  on the bare field (per-date groups, HTTP 200). Pre-existing, same change as above.
• A GROUP BY key whose function renders its own argument (CAST, TRY_CAST, SAFE_CAST,
  ISNULL, ISNOTNULL, and an aggregate written as a key; an ARRAY(…) key is refused in every
  spelling, as on main, because it renders as an ARRAY_AGG window): its grouped expression keeps
  the qualifier (CAST(t.d AS VARCHAR)), so its two MIXED spellings
  (GROUP BY CAST(t.d AS VARCHAR) HAVING CAST(d AS VARCHAR) = 'v' and the reverse) are refused,
  as they were, in a statement Elasticsearch groups; in a JOIN, a derived table or a CTE they are
  accepted and left to the relational engine (not run here: the softclient4es-arrow JOIN tests after
  merge); the bare and fully-qualified spellings are answered. Not widened: the test asserts
  that no mixed spelling is ever answered with a channel other than the bare spelling's (48 mixed
  spellings of the 24 such keys over one index, all refused). For ISNULL / ISNOTNULL the refusal now names the
  direct-comparison limit of the key filter, where it read "neither a GROUP BY key nor an aggregate".
• A column-vs-column temporal WHERE at one resolution (WHERE q.d > q.e, no mapping): both
  columns are untyped, so the comparison is a raw >, grouped or not. Pre-existing.
• Elasticsearch 6.8 only: a column-vs-column temporal WHERE at two resolutions is rendered as a
  range on the other column's name ({"range":{"d":{"gt":"e||+1d"}}}); a temporal isAfter at
  one resolution fails with a Joda cast. Pre-existing.
• GROUP BY UPPER(p.category) HAVING UPPER(p.category) = 'X' in a JOIN with no grouping of the
  argument's same-named column: refused as neither key nor aggregate, before and after.
• MatchCriteria.includes sends the MATCH text as a literal include pattern. Pre-existing.
• Earlier records kept: a window PARTITION BY key's include, the JOIN "Ambiguous column" message,
  the unused GroupBy.bucketNames, a venue-dependent column name.
• A HAVING on a window PARTITION BY key that is not a GROUP BY key passes validation and its
  include lands on the window's partition level; a key filter also lands on a partition level of the
  same key (both spellings alike). Pre-existing.
• Select.fieldAliases is keyed by identifierName: after one resolution a self-join's
  o.category AS oc, p.category AS pc share a key, so at the second resolution both levels display
  as pc and a HAVING on oc loses its include. Pre-existing, JOINs only.
• In a JOIN, an OR mixing a key condition and an aggregate (HAVING k = 'a' OR COUNT(*) > 1) is
  still refused although the relational engine could answer it. Pre-existing.
• In a JOIN, GROUP BY YEAR(o.d) HAVING MONTH(o.d) = 1: refused at parse on main; now accepted, and
  the relational engine refuses it ("column must appear in the GROUP BY clause"), as 0.23.0 did.

Binary compatibility

Additive only — SingleSearch gains keyedBuckets, bucketFor, keyIdentityOf,
tableQualifierOf, dateHistogramLevelOf, dateHistogramReason; From gains
mainTableCorrelation; Bucket gains sameKeyAs and isDateHistogram; one new private[query]
class. No constructor or case-class arity changes, no public member removed (javap -p of the three
classes: only synthetic lambda bodies differ) ⇒ no downstream rebuild.

NOT RUN locally

ES 7.17 and 9.0 legs and the Elasticsearch 6.8 Jest client (CI runs them); the softclient4es-arrow
JOIN tests, which need this change published and run after merge.

The R2 date-key rows are not exercised at RUN (a date key at a mapped index fails on main for the
reason recorded above); they move to the change that fixes it.

The website copy of the documentation is not part of this PR.

🤖 Generated with Claude Code