Typed column access with As<> / AsStrict<> - #541
Draft
slabko wants to merge 8 commits into
Draft
Conversation
ColumnArrayT/ColumnNullableT/ColumnTupleT/ColumnMapT::Wrap used to steal the source column's internals (rvalue-ref parameter consumed via std::move), leaving the original unusable. Wrap now shares the source's internals through shared_ptr: the returned typed column references the same underlying data, the original stays fully valid, and mutations are visible through both. All Wrap overloads take their argument by const reference (const <Column>& / const Column& / const ColumnRef&), so std::move is no longer required and lvalues and const sources are accepted. - nullable: wrap the nested column via WrapColumn (recursive share); include utils.h for it - array: share nested data and offsets - tuple: build element columns from the source via TupleFromColumn - map: share the backing array of key/value tuples - update doc-comments to describe the shared, non-destructive semantics - add tests for non-stealing behaviour and lvalue/const acceptance ColumnLowCardinalityT::Wrap is converted in the following commit.
Share the dedup map (unique_items_map_) via shared_ptr so a wrapped LowCardinality column shares dictionary, index and map with its source, keeping them coherent across both holders. Relax Wrap to const& (like the other typed columns) so it no longer steals from the source. Add tests covering non-stealing/shared-coherent semantics and lvalue acceptance.
Now that every wrappable column (including ColumnLowCardinalityT) exposes Wrap(const ColumnRef&), the WrapColumn helper no longer needs a non-const rvalue. Take the column by const reference and forward it without std::move; this also lets ColumnArrayT::Wrap pass col.data_ directly instead of copying it into a temporary ColumnRef.
Each typed column's Wrap now comes in two forms per input type: - Wrap(col, ValidationError* error): returns nullptr and, if error is non-null, fills *error on a type mismatch; it never throws. - Wrap(col): calls the two-argument form and throws the resulting ValidationError when it returns nullptr. The recursive WrapColumn helper mirrors the same pair. All Wrap mismatches now throw ValidationError consistently (previously Wrap(const Column&) and LowCardinality threw std::bad_cast). Add a default ValidationError constructor so an empty error can be created without a placeholder message, and add tests covering both the nullptr+error and throwing paths.
The Wrap-related helpers (SliceVector, HasWrapMethod, WrapColumn) depend only on declarations already provided by column.h, so fold them directly into column.h and drop the separate utils.h header. Since column.h is included everywhere, these utilities are now available without an extra include. Remove the now-redundant #include "utils.h" from the column sources and drop utils.h from the CMake source list and install rules.
Column::As<T>() and Column::AsStrict<T>() now recognise "wrappable" typed columns (those exposing a static Wrap method: ColumnArrayT, ColumnTupleT, ColumnMapT, ColumnNullableT, ColumnLowCardinalityT). For such a T that is not already an exact match, they Wrap the column into a storage-sharing typed view instead of failing: As returns nullptr when neither an exact cast nor a wrap is possible, and AsStrict throws ValidationError. Behaviour for non-wrappable T is unchanged, and an exact cast is always tried first to preserve object identity. The const As() overload deliberately does NOT wrap: wrapping would synthesize a mutable, storage-sharing view from a const column (via const_cast), which is not const-correct. It keeps the plain exact-downcast behaviour. The definitions are moved out-of-line below WrapColumn so the wrapping helpers are in scope. The ColumnLowCardinalityT::Wrap dictionary guard is pinned to a strict dynamic_pointer_cast because the constructor binds the dictionary via a reference dynamic_cast and therefore requires the exact stored type.
CreateColumnByType previously mapped LowCardinality(String)/LowCardinality( FixedString) to the strongly-typed ColumnLowCardinalityT<...>, unlike Array, Nullable, Tuple and Map, which all produce their base column type. Return the base ColumnLowCardinality here as well, so behaviour is consistent; callers can obtain the strongly-typed ColumnLowCardinalityT<...> view on demand via the wrapping Column::As<ColumnLowCardinalityT<...>>(). String and FixedString now share a single CreateColumnFromAst-based construction. The Nullable case keeps its own branch (documented) because it must select the ColumnLowCardinality(shared_ptr<ColumnNullable>) ctor overload that seeds the NULL item at dictionary index 0, and the default case keeps an explicit UnimplementedError for unsupported dictionary types. Add a CreateColumnByType.LowCardinality test and type-name round-trip cases.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TL;DR
Column::As<T>()andColumn::AsStrict<T>()are now smarter for the typed columntemplates (
ColumnArrayT,ColumnTupleT,ColumnMapT,ColumnNullableT,ColumnLowCardinalityT). When the requestedTis one of these and the column is notalready exactly
T, they wrap the column into a strongly-typed, storage-sharing viewinstead of failing.
In practice this means you can take a column straight out of a
Selectresult and ask forthe convenient typed interface directly:
No manual reconstruction, no exact-type juggling.
Background: base columns vs. typed columns
ClickHouse composite types are represented by two layers in this library:
ColumnArrayColumnArrayT<Nested>At(),operator[],Append(container)ColumnNullableColumnNullableT<Nested>std::optional-basedAt()/operator[]ColumnTupleColumnTupleT<Cols...>std::tuple-basedAt()/operator[]ColumnMapColumnMapT<Key, Value>At(), iterationColumnLowCardinalityColumnLowCardinalityT<Dict>At(),operator[],AppendMany()When you
Select, the server-side columns are created by the factory as the basetypes (
ColumnArray,ColumnNullable,ColumnTuple,ColumnMap,ColumnLowCardinality).The
…T<>templates are the ergonomic, type-safe wrappers around them.Previously, to get a typed view you needed the column to already be that exact type, or
you had to build the typed column yourself. Now
As<>/AsStrict<>bridge the gapautomatically.
Behavior
A type is wrappable if it exposes a static
Wrapmethod. That is exactly the fivetemplates above (
ColumnArrayT,ColumnTupleT,ColumnMapT,ColumnNullableT,ColumnLowCardinalityT). Everything else (ColumnString,ColumnUInt64, …) isnon-wrappable and behaves as before.
TTAs<T>()(non-const)nullptron mismatch.nullptron mismatch.AsStrict<T>()ValidationErroron failure.As<T>() constnullptron mismatch.nullptron mismatch.Notes:
T, you get the very sameobject back (identity preserved, no new allocation).
Block::operator[]returns a non-constColumnRefby value, soblock[i]->As<…>()always uses the wrapping (non-const) overload — wrapping works out of the box for
Selectresults.
Wrapping is a storage-sharing view
Wrapping does not copy or move the underlying data. The typed wrapper shares the same
storage (dictionary, offsets, index, nested columns) with the source via
shared_ptr.Mutations through one are visible through the other, and the source stays fully valid.
Examples
LowCardinality
Array
Nullable
Tuple
Map
Nested types
Wrapping recurses, so nested typed columns work too:
AsvsAsStrictAsreturnsnullptron a type mismatch — always check it:AsStrictthrowsclickhouse::ValidationErrorinstead, which is handy when a mismatch is aprogramming error you want to surface immediately:
How it works (brief)
HasWrapMethod<T>, detects whetherTexposes a staticWrap.T,As/AsStricttrydynamic_pointer_cast<T>first, then fall back toWrapColumn<T>(...), which callsT::Wrap(...).T::Wrap(...)builds the typed wrapper by sharing the source column's internal storage(
shared_ptr), recursing into nested columns as needed.Wrap(col, ValidationError*)) used to implementAs, andthrowing one-argument forms used by
AsStrict.Limitations & gotchas
As<T>() constdoes not wrap. The const overload performs an exact downcast only.Wrapping would have to synthesize a mutable, storage-sharing view from a
constcolumn(requiring a
const_cast), which is not const-correct, so it is intentionally disabled.Use a non-const handle if you need the typed view (e.g.
block[i]->As<…>()works, sinceBlock::operator[]yields a non-constColumnRef).Check for
nullptr(or useAsStrict).Assignals a mismatch by returningnullptr; it does not throw for a plain type mismatch.Wrapping allocates. Unless the column already is
T(exact-match fast path), wrappingcreates a new wrapper object (with a distinct identity from the source, though it shares
storage). These methods are not
noexcept: allocation can still throwstd::bad_alloc, andthe throwing forms raise
ValidationErroron a mismatch.LowCardinality(Nullable(String))cannot be wrapped into the doubly-typed form.The factory stores such a column with a base
ColumnNullabledictionary, whileColumnLowCardinalityT<T>binds the dictionary as aT&reference and therefore requiresthe dictionary to be exactly
T. As a resultAs<ColumnLowCardinalityT<ColumnNullableT<ColumnString>>>()returnsnullptr. PlainLowCardinality(String)/LowCardinality(FixedString(N))wrap fine.Numeric
LowCardinality(T)is unsupported by the column factory (its default-itemhandling is string-based).