Skip to content

Typed column access with As<> / AsStrict<> - #541

Draft
slabko wants to merge 8 commits into
masterfrom
implicit-wrap
Draft

Typed column access with As<> / AsStrict<>#541
slabko wants to merge 8 commits into
masterfrom
implicit-wrap

Conversation

@slabko

@slabko slabko commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

TL;DR

Column::As<T>() and Column::AsStrict<T>() are now smarter for the typed column
templates (ColumnArrayT, ColumnTupleT, ColumnMapT, ColumnNullableT,
ColumnLowCardinalityT). When the requested T is one of these and the column is not
already exactly T, they wrap the column into a strongly-typed, storage-sharing view
instead of failing.

In practice this means you can take a column straight out of a Select result and ask for
the convenient typed interface directly:

// block[0] is a base ColumnLowCardinality coming from the server.
if (auto lc = block[0]->As<ColumnLowCardinalityT<ColumnString>>()) {
    for (size_t i = 0; i < lc->Size(); ++i)
        std::cout << (*lc)[i] << "\n";   // typed operator[] -> std::string_view
}

No manual reconstruction, no exact-type juggling.


Background: base columns vs. typed columns

ClickHouse composite types are represented by two layers in this library:

Base column Typed wrapper Adds
ColumnArray ColumnArrayT<Nested> typed At(), operator[], Append(container)
ColumnNullable ColumnNullableT<Nested> std::optional-based At() / operator[]
ColumnTuple ColumnTupleT<Cols...> std::tuple-based At() / operator[]
ColumnMap ColumnMapT<Key, Value> key/value At(), iteration
ColumnLowCardinality ColumnLowCardinalityT<Dict> typed At(), operator[], AppendMany()

When you Select, the server-side columns are created by the factory as the base
types (ColumnArray, ColumnNullable, ColumnTuple, ColumnMap, ColumnLowCardinality).
The …T<> templates are the ergonomic, type-safe wrappers around them.

Previously, to get a typed view you needed the column to already be that exact type, or
you had to build the typed column yourself. Now As<>/AsStrict<> bridge the gap
automatically.


Behavior

A type is wrappable if it exposes a static Wrap method. That is exactly the five
templates above (ColumnArrayT, ColumnTupleT, ColumnMapT, ColumnNullableT,
ColumnLowCardinalityT). Everything else (ColumnString, ColumnUInt64, …) is
non-wrappable and behaves as before.

Call Wrappable T Non-wrappable T
As<T>() (non-const) exact cast first; otherwise wrap. Returns nullptr on mismatch. plain cast; nullptr on mismatch.
AsStrict<T>() exact cast first; otherwise wrap. Throws ValidationError on failure. plain cast; throws on failure.
As<T>() const exact cast only — no wrapping; nullptr on mismatch. plain cast; nullptr on mismatch.

Notes:

  • The exact-cast-first step means that if the column already is T, you get the very same
    object back (identity preserved, no new allocation).
  • Block::operator[] returns a non-const ColumnRef by value, so block[i]->As<…>()
    always uses the wrapping (non-const) overload — wrapping works out of the box for Select
    results.

Wrapping is a storage-sharing view

Wrapping does not copy or move the underlying data. The typed wrapper shares the same
storage (dictionary, offsets, index, nested columns) with the source via shared_ptr.
Mutations through one are visible through the other, and the source stays fully valid.

auto base    = block[0]->As<ColumnArray>();                 // base view
auto typed   = block[0]->As<ColumnArrayT<ColumnUInt64>>();  // typed, shares the same storage
// reading through `typed` reflects exactly what `base` holds

Examples

LowCardinality

client.Select("SELECT lc_col FROM my_table", [](const Block& block) {
    // Server sends a base ColumnLowCardinality; wrap it into the typed view.
    auto lc = block[0]->As<ColumnLowCardinalityT<ColumnString>>();
    if (!lc) return;
    for (size_t i = 0; i < lc->Size(); ++i)
        std::cout << lc->At(i) << "\n";   // std::string_view
});

Array

// Array(UInt64)
auto arr = block[0]->As<ColumnArrayT<ColumnUInt64>>();
for (size_t row = 0; row < arr->Size(); ++row) {
    auto values = arr->At(row);                 // ArrayValueView
    for (size_t j = 0; j < values.size(); ++j)
        std::cout << values[j] << " ";          // uint64_t
    std::cout << "\n";
}

Nullable

// Nullable(String)
auto n = block[0]->As<ColumnNullableT<ColumnString>>();
for (size_t i = 0; i < n->Size(); ++i) {
    std::optional<std::string_view> v = n->At(i);
    std::cout << (v ? std::string(*v) : std::string("<null>")) << "\n";
}

Tuple

// Tuple(UInt64, String)
auto t = block[0]->As<ColumnTupleT<ColumnUInt64, ColumnString>>();
for (size_t i = 0; i < t->Size(); ++i) {
    auto [id, name] = t->At(i);                 // std::tuple<uint64_t, std::string_view>
    std::cout << id << " => " << name << "\n";
}

Map

// Map(String, UInt64)
auto m = block[0]->As<ColumnMapT<ColumnString, ColumnUInt64>>();
for (size_t i = 0; i < m->Size(); ++i) {
    auto row = m->At(i);                         // MapValueView
    for (const auto& [key, value] : row)
        std::cout << key << "=" << value << " ";
    std::cout << "\n";
}

Nested types

Wrapping recurses, so nested typed columns work too:

// Array(LowCardinality(String))
auto arr = block[0]->As<ColumnArrayT<ColumnLowCardinalityT<ColumnString>>>();

As vs AsStrict

As returns nullptr on a type mismatch — always check it:

if (auto typed = column->As<ColumnArrayT<ColumnUInt64>>()) {
    // use typed
} else {
    // wrong type
}

AsStrict throws clickhouse::ValidationError instead, which is handy when a mismatch is a
programming error you want to surface immediately:

auto typed = column->AsStrict<ColumnArrayT<ColumnUInt64>>();   // throws on mismatch

How it works (brief)

  • A compile-time trait, HasWrapMethod<T>, detects whether T exposes a static Wrap.
  • For a wrappable T, As/AsStrict try dynamic_pointer_cast<T> first, then fall back to
    WrapColumn<T>(...), which calls T::Wrap(...).
  • T::Wrap(...) builds the typed wrapper by sharing the source column's internal storage
    (shared_ptr), recursing into nested columns as needed.
  • There are non-throwing cores (Wrap(col, ValidationError*)) used to implement As, and
    throwing one-argument forms used by AsStrict.

Limitations & gotchas

  • As<T>() const does not wrap. The const overload performs an exact downcast only.
    Wrapping would have to synthesize a mutable, storage-sharing view from a const column
    (requiring a const_cast), which is not const-correct, so it is intentionally disabled.
    Use a non-const handle if you need the typed view (e.g. block[i]->As<…>() works, since
    Block::operator[] yields a non-const ColumnRef).

  • Check for nullptr (or use AsStrict). As signals a mismatch by returning
    nullptr; it does not throw for a plain type mismatch.

  • Wrapping allocates. Unless the column already is T (exact-match fast path), wrapping
    creates a new wrapper object (with a distinct identity from the source, though it shares
    storage). These methods are not noexcept: allocation can still throw std::bad_alloc, and
    the throwing forms raise ValidationError on a mismatch.

  • LowCardinality(Nullable(String)) cannot be wrapped into the doubly-typed form.
    The factory stores such a column with a base ColumnNullable dictionary, while
    ColumnLowCardinalityT<T> binds the dictionary as a T& reference and therefore requires
    the dictionary to be exactly T. As a result
    As<ColumnLowCardinalityT<ColumnNullableT<ColumnString>>>() returns nullptr. Plain
    LowCardinality(String) / LowCardinality(FixedString(N)) wrap fine.

  • Numeric LowCardinality(T) is unsupported by the column factory (its default-item
    handling is string-based).

slabko added 8 commits July 28, 2026 18:47
ColumnArrayT/ColumnNullableT/ColumnTupleT/ColumnMapT::Wrap used to steal
the source column's internals (rvalue-ref parameter consumed via
std::move), leaving the original unusable.

Wrap now shares the source's internals through shared_ptr: the returned
typed column references the same underlying data, the original stays
fully valid, and mutations are visible through both. All Wrap overloads
take their argument by const reference (const <Column>& / const Column& /
const ColumnRef&), so std::move is no longer required and lvalues and
const sources are accepted.

- nullable: wrap the nested column via WrapColumn (recursive share);
  include utils.h for it
- array:    share nested data and offsets
- tuple:    build element columns from the source via TupleFromColumn
- map:      share the backing array of key/value tuples
- update doc-comments to describe the shared, non-destructive semantics
- add tests for non-stealing behaviour and lvalue/const acceptance

ColumnLowCardinalityT::Wrap is converted in the following commit.
Share the dedup map (unique_items_map_) via shared_ptr so a wrapped
LowCardinality column shares dictionary, index and map with its source,
keeping them coherent across both holders. Relax Wrap to const& (like the
other typed columns) so it no longer steals from the source. Add tests
covering non-stealing/shared-coherent semantics and lvalue acceptance.
Now that every wrappable column (including ColumnLowCardinalityT) exposes
Wrap(const ColumnRef&), the WrapColumn helper no longer needs a non-const
rvalue. Take the column by const reference and forward it without
std::move; this also lets ColumnArrayT::Wrap pass col.data_ directly
instead of copying it into a temporary ColumnRef.
Each typed column's Wrap now comes in two forms per input type:
- Wrap(col, ValidationError* error): returns nullptr and, if error is
  non-null, fills *error on a type mismatch; it never throws.
- Wrap(col): calls the two-argument form and throws the resulting
  ValidationError when it returns nullptr.

The recursive WrapColumn helper mirrors the same pair. All Wrap
mismatches now throw ValidationError consistently (previously
Wrap(const Column&) and LowCardinality threw std::bad_cast).

Add a default ValidationError constructor so an empty error can be
created without a placeholder message, and add tests covering both the
nullptr+error and throwing paths.
The Wrap-related helpers (SliceVector, HasWrapMethod, WrapColumn) depend
only on declarations already provided by column.h, so fold them directly
into column.h and drop the separate utils.h header. Since column.h is
included everywhere, these utilities are now available without an extra
include.

Remove the now-redundant #include "utils.h" from the column sources and
drop utils.h from the CMake source list and install rules.
Column::As<T>() and Column::AsStrict<T>() now recognise "wrappable" typed
columns (those exposing a static Wrap method: ColumnArrayT, ColumnTupleT,
ColumnMapT, ColumnNullableT, ColumnLowCardinalityT). For such a T that is not
already an exact match, they Wrap the column into a storage-sharing typed view
instead of failing: As returns nullptr when neither an exact cast nor a wrap is
possible, and AsStrict throws ValidationError. Behaviour for non-wrappable T is
unchanged, and an exact cast is always tried first to preserve object identity.

The const As() overload deliberately does NOT wrap: wrapping would synthesize a
mutable, storage-sharing view from a const column (via const_cast), which is not
const-correct. It keeps the plain exact-downcast behaviour.

The definitions are moved out-of-line below WrapColumn so the wrapping helpers
are in scope. The ColumnLowCardinalityT::Wrap dictionary guard is pinned to a
strict dynamic_pointer_cast because the constructor binds the dictionary via a
reference dynamic_cast and therefore requires the exact stored type.
CreateColumnByType previously mapped LowCardinality(String)/LowCardinality(
FixedString) to the strongly-typed ColumnLowCardinalityT<...>, unlike Array,
Nullable, Tuple and Map, which all produce their base column type. Return the
base ColumnLowCardinality here as well, so behaviour is consistent; callers can
obtain the strongly-typed ColumnLowCardinalityT<...> view on demand via the
wrapping Column::As<ColumnLowCardinalityT<...>>().

String and FixedString now share a single CreateColumnFromAst-based
construction. The Nullable case keeps its own branch (documented) because it
must select the ColumnLowCardinality(shared_ptr<ColumnNullable>) ctor overload
that seeds the NULL item at dictionary index 0, and the default case keeps an
explicit UnimplementedError for unsupported dictionary types.

Add a CreateColumnByType.LowCardinality test and type-name round-trip cases.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant