feat(sql): expose the set-operation reader to the relational engine; UNION ALL reads agreeing dates of a non-default format

[fupelaqu]

What

The relational engine (softclient4es-arrow) must answer a set operation with the same verdict, types and values as core. Core's set-operation typing and value reading were private, so arrow re-implemented them and drifted: 18 of 22 values stored in non-default forms differed. Core now exposes them, private[elastic], so both paths use ONE implementation. Arrow #198 is the consumer.

• MultiSearch.columnTree(requests, operators) returns each step of the operator tree with its type per column, plus an untyped flag where DuckDB 1.5.5 gives that step no type. Arrow casts there and leaves the other steps to DuckDB. columnTypes and branchTypes are unchanged.
• SetOperationValues.legReaders(requests, operators, version, unreadable) gives, per leg and column, a function that reads a value as its branch's declared type, the way the cluster's Elasticsearch major indexed it.
  • It reads at every depth, also where the branches agree, with one class per type.
  • A value with several elements fails, as at a converting position.
  • unreadable (Fail by default, or KeepAsStored) decides what happens to a date the reader cannot read.
• Core's own UNION ALL reads an agreeing date position, at every depth, when a branch's field has a non-default mapping format (epoch_second, a custom pattern). The column then holds dates instead of the stored numbers and text.
  • A value the reader cannot read is kept as stored (a zone name, a value with several elements, an array of arrays), so nothing core answered before fails.
  • Ordinary agreeing columns are untouched.

Measured

• No regression vs main (fc3558f), Elasticsearch 8.18.3, 180 statements on 3 routes. Agreeing and converting date columns, at the column and inside objects, over:

  • default, epoch_second, epoch_millis and 9 custom-pattern formats;
  • region, abbreviation and long-name zones;
  • unmeasured letters.

  Every statement main answers still answers. The converted dates equal Elasticsearch's indexed values (270 / 270).

• untyped against DuckDB 1.5.5:

  • 400 random multi-step statements (2,320 step-columns): 0 mismatches;
  • every typed step's type equals DuckDB's;
  • casting at the untyped steps answers the root type (800 / 800).

• columnTypes / branchTypes byte-identical to main: 6,000 random statements, plus the 13,120-statement rule table.

• Reachable from arrow: a test in package app.softnetwork.elastic.arrow compiles and calls the new API.

• Tests:

  • sql/test 1844 / 0 and core/test 1213 / 0;
  • the set-operation spec 15 / 15 on Elasticsearch 8.18.3 and 6.8.23. CI runs every major.

• Cost:

  • ordinary agreeing columns: unchanged per row;
  • setup: +0.2 to 0.4 µs per set-operation statement;
  • a non-default-format agreeing column pays the date reader, about 190 ns per row, and only there.

API (additive, private[elastic])

• MultiSearch.columnTree, MultiSearch.SetOperationNode;
• SetOperationValues.legReaders, SetOperationValues.Unreadable;
• SetOperationValues is now private[elastic].

🤖 Generated with Claude Code