From 195290eb45d2965ca2f80995462884eea3f6ed57 Mon Sep 17 00:00:00 2001 From: Marko Topolnik Date: Mon, 19 Oct 2026 16:11:00 +0200 Subject: [PATCH 1/3] docs(qwp): document the 7.1.0 UUID byte order for C, C++, and Rust --- documentation/changelog.mdx | 3 + documentation/concepts/delivery-semantics.md | 5 + documentation/connect/clients/c-and-cpp.md | 151 ++++++++++++++++-- documentation/connect/clients/rust.md | 86 +++++++++- .../wire-protocols/qwp-ingress-websocket.md | 12 +- .../store-and-forward/concepts.md | 19 ++- .../store-and-forward/when-to-use.md | 8 +- 7 files changed, 258 insertions(+), 26 deletions(-) diff --git a/documentation/changelog.mdx b/documentation/changelog.mdx index 9046996dd..303e4d0cd 100644 --- a/documentation/changelog.mdx +++ b/documentation/changelog.mdx @@ -23,6 +23,9 @@ This page tracks significant updates to the QuestDB documentation. ### Updated +- [C and C++](/docs/connect/clients/c-and-cpp/#binary-columns-uuid-long256-and-opaque-bytes) and [Rust](/docs/connect/clients/rust/#binary-columns-uuid-long256-and-opaque-bytes) clients - Documented the 7.1.0 canonical RFC 4122 UUID byte order at the API boundary, UUID and LONG256 claims for binary Arrow columns through field metadata or per-column overrides, and the silent behavior change for code written against 7.0.0. The C and C++ FetchContent example now pins the 7.1.0 tag +- [Rust client](/docs/connect/clients/rust/#arrow-and-polars-ingestion) - Clarified DataFrame-wide `in_doubt` status versus individual flushes and the risk of duplicating committed rows on retry +- [Store-and-forward](/docs/high-availability/store-and-forward/concepts/#reconnect-and-replay) - Corrected replay guidance: connection-local sequence numbers do not deduplicate writes; table-level deduplication with stable row identity is required to suppress duplicates - [Live views](/docs/concepts/live-views/) - Corrected the [supported window functions](/docs/concepts/live-views/#supported-window-functions) to match QuestDB 10.0.1: the statistical functions, EMA, and VWEMA run only in anchored windows, `ksum` does not support `RANGE` frames, `lag` needs a bounded frame or an anchor, and `VARCHAR`, `UUID`, `LONG256`, `DECIMAL`, `BINARY`, and array partition keys are rejected. Added a [rolling standard deviation](/docs/concepts/live-views/#rolling-standard-deviation) recipe for Bollinger bands, rewrote [Freshness](/docs/concepts/live-views/#freshness) to reflect that interval-filtered reads, aggregates, and `ASOF JOIN` see unflushed rows while `LATEST ON` does not, and added the `UPDATE` and `REBASE WAL` invalidation triggers, base WAL retention, and replica freshness. Documented that an out-of-order base row rewrites view rows already emitted, that `view_status` stays `active` while the view's own WAL is suspended, and what to change in `SHOW CREATE LIVE VIEW` output before replaying it - [CREATE LIVE VIEW](/docs/query/sql/create-live-view/) - Fixed the `OWNED BY` example, which requires a parenthesized query, documented the designated-timestamp projection requirement and the remaining anchored-window rules, and expanded the error reference with verified messages. Added a grammar for anchored windows and [examples](/docs/query/sql/create-live-view/#examples) of a `RANGE` frame with a `WHERE` filter and an anchored session window - [Live view configuration](/docs/configuration/live-views/) - Corrected the descriptions of the in-memory buffer growth budget, partition compaction threshold, flush retry budgets, and refresh memory limit diff --git a/documentation/concepts/delivery-semantics.md b/documentation/concepts/delivery-semantics.md index 597c1910c..9f2c70b84 100644 --- a/documentation/concepts/delivery-semantics.md +++ b/documentation/concepts/delivery-semantics.md @@ -38,6 +38,11 @@ the server confirms a batch, the client reconnects and re-sends. If the server had already committed the batch but the acknowledgement was lost in flight, the second send produces duplicates. +QWP's `wireSeq` cannot suppress this replay: it is assigned by receive order, +exists only for response correlation on one connection, and resets after a +reconnect. Requests carry no persistent message identifier that the server can +use to recognize the same frame on the next connection. + This path applies to every QuestDB client deployment. ### Multi-host failover replay diff --git a/documentation/connect/clients/c-and-cpp.md b/documentation/connect/clients/c-and-cpp.md index 9b82a307c..bca3c57f7 100644 --- a/documentation/connect/clients/c-and-cpp.md +++ b/documentation/connect/clients/c-and-cpp.md @@ -28,7 +28,7 @@ include(FetchContent) FetchContent_Declare( c_questdb_client_proj GIT_REPOSITORY https://github.com/questdb/c-questdb-client.git - GIT_TAG 7.0.0) # or a newer release tag, or a commit SHA + GIT_TAG 7.1.0) # or a newer release tag, or a commit SHA FetchContent_MakeAvailable(c_questdb_client_proj) target_link_libraries(your_target questdb_client) @@ -696,7 +696,9 @@ on_error:; the 2 MiB target; if 8 rows still exceed 4 MiB — which takes very large string, binary, or array values — the flush fails instead of splitting. - **Recovery depends on `in_doubt`, not on the error code.** Check - `line_sender_error_in_doubt` (C++: `e.in_doubt()`). False means the queue + `line_sender_error_in_doubt` (C++: `e.in_doubt()`). This describes the + failed operation's input, not earlier independent flushes or replay from an + application checkpoint. False means the queue never took the frame and the chunk is intact: re-flush it. True means delivery is uncertain, so `wait` for what the queue already holds, and resend the chunk only where the table's dedup keys make duplicate rows harmless. A @@ -776,13 +778,18 @@ See `qwp_sender.h` for the exact signatures. Complete list: | `column_bool` | LSB-first packed bitmap | `BOOLEAN` | | `column_ts` + `qwp_ts_unit` (`_micros` / `_nanos`) | int64 since epoch | `TIMESTAMP` / `TIMESTAMP_NS` | | `column_date` | int64 millis since epoch | `DATE` | -| `column_uuid` | 16 bytes, canonical RFC-4122 order | `UUID` | -| `column_long256` | 32 bytes (4 LE limbs) | `LONG256` | +| `column_uuid` | 16 bytes, canonical RFC 4122 big-endian | `UUID` | +| `column_long256` | 32 bytes (4 LE limbs, low limb first) | `LONG256` | | `column_ipv4` | uint32 | `IPV4` | | `column_str` | Arrow Utf8 offsets + bytes | `VARCHAR` | | `column_binary` | Arrow Binary offsets + bytes | `BINARY` | | `symbol_i8` / `_i16` / `_i32` | dict codes + Utf8 dictionary | `SYMBOL` | +`column_uuid` takes the UUID's canonical RFC 4122 bytes, exactly as they are +written in the textual form, and byte-swaps them into QWP wire order for you. +The row-oriented `line_sender_buffer_column_uuid` is the exception: it takes +the two 64-bit wire halves, `(lo, hi)`. + Designated timestamp (exactly once per chunk, before flush): `at_nanos` / `at_micros` / `at_millis` / `at_seconds` (millis and seconds are widened to micros on the wire). Decimals, geohash, arrays, and the @@ -873,17 +880,137 @@ bool ingest(questdb_db* db, struct ArrowArray* array, caller keeps `schema`. On failure check `array->release != NULL` before invoking it. - Per-column wire-type hints (`qwp_arrow_override`: force - SYMBOL/VARCHAR, IPv4, char, geohash precision) steer encoding without - touching the Arrow schema. + SYMBOL/VARCHAR, IPv4, char, geohash precision, UUID, LONG256) choose the + wire type without touching the Arrow schema. An override wins for its + column over any field metadata the schema carries. - To append Arrow **columns** into a chunk alongside hand-built ones, use `qwp_chunk_append_arrow_column`, or `qwp_arrow_import_new` + `..._append_arrow_import` to import once - and slice across many chunks. + and slice across many chunks. Neither takes an overrides array, so an + IPv4, char, geohash, UUID, or LONG256 column has to carry its claim as + field metadata instead. The SYMBOL choice is still available on the import + path: `qwp_arrow_import_new` takes a `symbol_mode` argument + (`qwp_symbol_mode_auto`, `_symbol`, `_not_symbol`). - Dictionary-encoded string columns map to `SYMBOL` by default; plain Utf8 to `VARCHAR`. `qwp_sender.h` lists every Arrow type the client accepts, and the kinds it rejects (`Struct`, `Map`, `Interval`, ...); a rejected type fails with `line_sender_error_arrow_unsupported_column_kind`. +### Binary columns: UUID, LONG256, and opaque bytes + +Binary Arrow columns land as `BINARY` unless the column *claims* a richer +type. The width of a column claims nothing on its own: a bare +`FixedSizeBinary(16)` is opaque bytes, not a UUID. A claim comes from the +schema or from an override: + +| Claim | Lands as | +| --- | --- | +| `ARROW:extension:name = arrow.uuid` on `FixedSizeBinary(16)` | `UUID` | +| `questdb.column_type = uuid` field metadata | `UUID` | +| `questdb.column_type = long256` field metadata | `LONG256` | +| `qwp_arrow_override_uuid` / `qwp_arrow_override_long256` | `UUID` / `LONG256` | + +UUID bytes are canonical RFC 4122 big-endian and the client byte-swaps them +into QWP wire order; LONG256 bytes are little-endian limbs, low limb first, +and go out verbatim. The `questdb.column_type` claims and the two overrides +also apply to variable-length `Binary` / `LargeBinary` / `BinaryView` +columns, where every non-null value must then be exactly 16 or 32 bytes. The +`arrow.uuid` extension is the exception: the Arrow spec fixes its storage to +`FixedSizeBinary(16)`, so the client rejects the label on any other type. A +claim whose width doesn't match fails with +`line_sender_error_arrow_ingest`. + +:::caution Behaviour change + +Before client 7.1.0 a bare `FixedSizeBinary(16)` or `(32)` column became +`UUID` or `LONG256` on width alone, with no claim needed. It is now `BINARY` +unless the column carries one of the claims above, so a batch that used to +produce a `UUID` column now produces a `BINARY` one and reports no error. + +The UUID byte order at the API boundary changed in the same release. Code +written against an earlier version passed QWP wire-order bytes to +`qwp_chunk_column_uuid` and `qwp_reader_query_bind_uuid`; those values are +now stored with their bytes reversed, also with no error. + +::: + +#### Claiming in the schema + +Both metadata claims are attached to the Arrow `Field`, so you make them +wherever the batch is built. In Arrow C++: + +```cpp +// The standard Arrow extension label. FixedSizeBinary(16) only. +auto trade_id = arrow::field("trade_id", arrow::fixed_size_binary(16)) + ->WithMetadata(arrow::key_value_metadata( + {"ARROW:extension:name"}, {"arrow.uuid"})); + +// The QuestDB claim, also valid on Binary / LargeBinary / BinaryView. +auto order_hash = arrow::field("order_hash", arrow::fixed_size_binary(32)) + ->WithMetadata(arrow::key_value_metadata( + {"questdb.column_type"}, {"long256"})); + +auto batch_schema = arrow::schema({ + arrow::field("ts", arrow::timestamp(arrow::TimeUnit::NANO)), + trade_id, + order_hash}); +``` + +`questdb.column_type = uuid` has the same shape with `uuid` as the value. Use +it in place of `arrow.uuid` when the bytes sit in a variable-length binary +column, which the extension label doesn't allow. + +Export the batch built against that schema through `arrow::ExportRecordBatch` +and flush it exactly as above — the claims travel with it, and the flush call +needs no extra arguments. + +#### Claiming at the call site + +An override claims the type per flush and leaves the schema alone. Fill in a +`qwp_arrow_override` per column and pass the array to any +`flush_arrow_batch*` call, where the example above passes no overrides: + + + + +```cpp +using namespace questdb::ingress::literals; + +const ::qwp_arrow_override overrides[] = { + {"trade_id", sizeof("trade_id") - 1, qwp_arrow_override_uuid, 0}, + {"order_hash", sizeof("order_hash") - 1, qwp_arrow_override_long256, 0}, +}; + +sender.flush_arrow_batch_and_wait( + "trades"_tn, array, schema, "ts"_cn, + overrides, std::size(overrides)); +``` + + + + +```c +const qwp_arrow_override overrides[] = { + {"trade_id", sizeof("trade_id") - 1, qwp_arrow_override_uuid, 0}, + {"order_hash", sizeof("order_hash") - 1, qwp_arrow_override_long256, 0}, +}; + +bool ok = qwp_sender_flush_arrow_batch_at_column_and_wait( + sender, QDB_TABLE_NAME_LITERAL("trades"), array, schema, + QDB_COLUMN_NAME_LITERAL("ts"), + overrides, sizeof(overrides) / sizeof(overrides[0]), + qwpws_ack_level_ok, &err); +``` + + + + +`arg` (the trailing `0`) carries the geohash precision for +`qwp_arrow_override_geohash` and is unused by every other kind. An override +that names a column the batch doesn't have, repeats another override's +column, or carries an unknown kind fails with +`line_sender_error_invalid_api_call`. + ## Querying data Get a reader (QWP/WebSocket only), prepare/execute SQL, then stream batches and @@ -1084,6 +1211,12 @@ For width-independent access, use `column::visit` and mantissa as little-endian two's-complement bytes. Check for null before decoding it. +`qwp_reader_column_data_get_bytes` also serves `UUID` and `LONG256`. It hands +back UUID values as 16 canonical RFC 4122 big-endian bytes — the decoder has +already reversed them out of wire order, so they match what +`qwp_chunk_column_uuid` and `bind_uuid` take — and LONG256 values as 32 +little-endian limb bytes, low limb first, verbatim from the wire. + ### Parameterised queries Prepare then bind: C `qwp_reader_prepare` + `qwp_reader_query_bind_*` + @@ -1103,8 +1236,8 @@ outlive any cursor it produces. The complete bind surface (C | `bind_decimal64` / `bind_decimal128` / `bind_decimal256` | unscaled value + scale | `DECIMAL` | | `bind_geohash` | bits + precision | `GEOHASH` | | `bind_varchar` | UTF-8 string | `VARCHAR` | -| `bind_uuid` | 16 bytes | `UUID` | -| `bind_long256` | 32 bytes | `LONG256` | +| `bind_uuid` | 16 bytes, canonical RFC 4122 big-endian | `UUID` | +| `bind_long256` | 32 bytes (4 LE limbs, low limb first) | `LONG256` | | `bind_binary` | bytes + length | `BINARY` (not yet accepted server-side) | | `bind_ipv4` | uint32, host order | `IPV4` (not yet accepted server-side) | diff --git a/documentation/connect/clients/rust.md b/documentation/connect/clients/rust.md index 20fb11dbb..85d24925c 100644 --- a/documentation/connect/clients/rust.md +++ b/documentation/connect/clients/rust.md @@ -271,6 +271,18 @@ QWP cannot preserve nulls for `BOOLEAN`, `BYTE`, or `SHORT`. An absent value in one of those columns is received as `false` or `0`; use a wider nullable type when the distinction matters. +Everywhere in the API a UUID is 16 canonical RFC 4122 big-endian bytes — the +bytes `uuid::Uuid::as_bytes()` gives you — with one exception. `column_uuid` +on the row buffer takes the two 64-bit halves of the QWP wire encoding, +`(lo, hi)`, and is the only place you have to think about wire order. + +`Chunk::column_uuid` takes one 16-byte array per row, so wrap a single value +rather than splitting it yourself: + +```rust +chunk.column_uuid("trade_id", std::slice::from_ref(u.as_bytes()), None)?; +``` + ## Chunk ingestion {#sending-data-column-major} Use a `Chunk` when values already live in column slices. All columns and the @@ -356,8 +368,8 @@ needs an entry in `data`, which the encoder ignores. | `BOOLEAN` | `column_bool(name, bits, row_count, validity)` | LSB-first bit-packed values | | `TIMESTAMP`, `TIMESTAMP_NS` | `column_ts(name, data, TimestampUnit, validity)` | Epoch `i64` values | | `DATE` | `column_date` | Epoch milliseconds | -| `UUID` | `column_uuid` | `&[[u8; 16]]` in canonical RFC-4122 order | -| `LONG256` | `column_long256` | `&[[u8; 32]]` in little-endian limb order | +| `UUID` | `column_uuid` | `&[[u8; 16]]`, one 16-byte value per row, in canonical RFC 4122 big-endian order — each element is what `uuid::Uuid::as_bytes()` gives you | +| `LONG256` | `column_long256` | `&[[u8; 32]]` in little-endian limb order, least-significant limb first | | `IPv4` | `column_ipv4` | Host-order `u32` values | | `VARCHAR` | `column_str`, `column_str_large` | Arrow Utf8 offsets and bytes | | `BINARY` | `column_binary` | Arrow Binary offsets and bytes | @@ -426,7 +438,7 @@ db.flush_arrow_batch( "trades", &record_batch, None, // server-assigned designated timestamp - &[], // no Arrow column overrides + &[], // per-column wire-type overrides; see below Some(AckLevel::Ok), )?; ``` @@ -445,7 +457,10 @@ use questdb::ingress::{ ColumnName, }; -let overrides: [ArrowColumnOverride<'_>; 0] = []; +let overrides = [ + ArrowColumnOverride::Uuid { column: "trade_id" }, + ArrowColumnOverride::Long256 { column: "order_hash" }, +]; let options = PolarsIngestOptions::new() .max_rows(50_000) .timestamp_column(ColumnName::new("timestamp")?) @@ -455,8 +470,9 @@ let options = PolarsIngestOptions::new() db.flush_polars_dataframe("trades", &dataframe, &options)?; ``` -`max_rows(0)` uses the default batch size. Omitting `timestamp_column` asks the -server to assign timestamps. Omitting `ack_level` uses the pool default. +Pass `&[]` for `overrides` when no column needs one. `max_rows(0)` uses the +default batch size. Omitting `timestamp_column` asks the server to assign +timestamps. Omitting `ack_level` uses the pool default. `flush_polars_dataframe` checkpoints the frame and automatically retries the uncommitted tail after a transient failover. `flush_arrow_batch` returns a @@ -466,6 +482,52 @@ uncertain failure can also duplicate rows. Use [deduplication](/docs/concepts/deduplication/) when duplicates would be harmful. +On a failed `flush_polars_dataframe` call, `err.in_doubt()` covers the whole +DataFrame. It is true if any batch may have been delivered, including batches +confirmed by an earlier checkpoint, even when a later batch fails validation. +Internal retries and connection replacements retain this call-level status. +The flag is conservative: it does not identify a safe row offset for resuming +the load. A false flag does not make a validation error retryable; correct the +input first. Low-level Arrow and chunk flushes retain their current-operation +scope and do not aggregate earlier independent calls. + +### Binary columns: UUID, LONG256, and opaque bytes + +Binary columns land as `BINARY` unless the column claims a richer type, and a +byte width claims nothing on its own: a bare `FixedSizeBinary(16)` is opaque +bytes, not a UUID. A claim comes either from the Arrow schema — the +`ARROW:extension:name = arrow.uuid` extension label on `FixedSizeBinary(16)`, +or `questdb.column_type = uuid` / `= long256` field metadata — or from an +`ArrowColumnOverride::Uuid` / `::Long256` entry as above, which wins over any +metadata on that column. + +Polars has no fixed-size binary dtype, so it needs the override. Its `Binary` +columns export as Arrow `BinaryView`, and every non-null value must then be +exactly 16 or 32 bytes. UUID bytes are canonical RFC 4122 big-endian, and the +client byte-swaps them into wire order for you; LONG256 bytes are +little-endian limbs, low limb first, and are sent unchanged. A value of the +wrong width fails with `ErrorCode::ArrowIngest`. + +Polars `Object` columns are rejected outright. They export as +`FixedSizeBinary(8)` holding in-process handles, which is indistinguishable +from ordinary opaque binary once converted, so the client refuses them rather +than storing meaningless addresses. Cast the column to a supported dtype +first. + +:::caution Behaviour change + +Before client 7.1.0 a bare `FixedSizeBinary(16)` or `(32)` column became +`UUID` or `LONG256` on width alone. It is now `BINARY` unless the column +carries one of the claims above. A batch that used to produce a `UUID` column +now produces a `BINARY` one, with no error. + +The UUID byte order at the API boundary changed in the same release. Code +written against an earlier version passed wire-order bytes to +`Chunk::column_uuid` and `bind_uuid`; those values are now stored reversed, +also with no error. + +::: + ## Querying Borrow a reader, prepare SQL, bind values, execute, and pull typed batches: @@ -537,6 +599,11 @@ Available builders include: - `bind_geohash` - `bind_null` and the typed `bind_null_*` variants +`bind_uuid` takes 16 canonical RFC 4122 big-endian bytes by value, so pass +`*u.as_bytes()` or `u.into_bytes()`. `bind_long256` takes 32 little-endian +limb bytes, low limb first. Both are the byte orders the matching result +columns are read back in. + ### Reading columns `BatchView::column(index)` returns a non-exhaustive `ColumnView`. Match the @@ -549,7 +616,7 @@ variant before reading values: | `Symbol` | `resolve(row) -> Option<&str>` | | `Varchar` | `value(row) -> Option<&str>` | | `Binary` | `value(row) -> Option<&[u8]>` | -| `Uuid`, `Long256` | Fixed-size byte-array reference | +| `Uuid`, `Long256` | Fixed-size byte-array reference: `Uuid` yields 16 canonical RFC 4122 big-endian bytes, `Long256` 32 little-endian limb bytes, low limb first | | `Decimal64`, `Decimal128`, `Decimal256` | Integer value plus the column scale | | `Geohash` | Bits plus precision | | `DoubleArray`, `LongArray` | Per-row shape and element data | @@ -689,6 +756,11 @@ publishes or completes its first frame, so treat `None` from `acked_fsn()` as Recovery turns on `err.in_doubt()`, not on `err.code()` and not on whether the buffer or chunk still holds rows. +For these low-level APIs, the flag describes the failed operation's input. It +does not summarize earlier independent flushes or authorize replaying all data +since an application checkpoint. The DataFrame-level aggregation described +above applies specifically to `flush_polars_dataframe`. + When `in_doubt()` is `false`, the flush failed before the queue took the frame, so the rows never entered the send path and your input is intact: re-flush it. When `in_doubt()` is `true`, delivery is diff --git a/documentation/connect/wire-protocols/qwp-ingress-websocket.md b/documentation/connect/wire-protocols/qwp-ingress-websocket.md index 468047401..1eeee6101 100644 --- a/documentation/connect/wire-protocols/qwp-ingress-websocket.md +++ b/documentation/connect/wire-protocols/qwp-ingress-websocket.md @@ -420,7 +420,7 @@ columns, which keeps every message self-contained for store-and-forward replay. | 9 | `0x09` | SYMBOL | var | Dictionary-encoded string | | 10 | `0x0A` | TIMESTAMP | 8 | Microseconds since Unix epoch | | 11 | `0x0B` | DATE | 8 | Milliseconds since Unix epoch | -| 12 | `0x0C` | UUID | 16 | RFC 4122 UUID | +| 12 | `0x0C` | UUID | 16 | UUID, two LE int64 halves, lo first | | 13 | `0x0D` | LONG256 | 32 | 256-bit integer | | 14 | `0x0E` | GEOHASH | var | Geospatial hash | | 15 | `0x0F` | VARCHAR | var | Length-prefixed UTF-8 | @@ -812,7 +812,15 @@ uncompressed mode. ### UUID 16 bytes per value: 8 bytes for the low 64 bits, then 8 bytes for the high -64 bits, both little-endian. +64 bits, both little-endian. Reverse all 16 bytes and you get the canonical +RFC 4122 order. + +The C, C++, and Rust clients take RFC 4122 bytes in their APIs, so +they reverse each value on the way to the wire and again on the way back. The +one exception is the row-buffer call that takes the two 64-bit halves +directly — `line_sender_buffer_column_uuid(buffer, name, lo, hi, err)` and +its Rust equivalent — which is already in wire order and is passed through +unchanged. ### LONG256 diff --git a/documentation/high-availability/store-and-forward/concepts.md b/documentation/high-availability/store-and-forward/concepts.md index 6acafca19..6591c5bfc 100644 --- a/documentation/high-availability/store-and-forward/concepts.md +++ b/documentation/high-availability/store-and-forward/concepts.md @@ -62,8 +62,9 @@ Two distinct counters track frame identity: - **FSN** (frame-sequence-number) — a monotonic counter assigned when a frame is appended to the substrate. FSN survives reconnects and (in SF mode) restarts. It is the substrate's permanent identifier for a frame. -- **wireSeq** — the per-connection counter the server uses for - deduplication, reset to `0` on every successful WebSocket upgrade. +- **wireSeq** — a per-connection counter used to correlate cumulative QWP + responses with sent frames. It resets to `0` on every successful WebSocket + upgrade. On every (re)connect the relationship is pinned: @@ -82,9 +83,11 @@ Two consequences: - Frames **must** be sent in strict order. The wire format does not serialise `wireSeq` — the server assigns it implicitly from receive order. Reordering breaks the FSN mapping. -- After a reconnect, the server sees the **same payloads** at new - `wireSeq` values. Server-side dedup keys off `messageSequence` inside - the payload, not `wireSeq`, so replay does not produce double-writes. +- After a reconnect, the server sees the **same payloads** at new `wireSeq` + values. QWP requests contain neither `wireSeq` nor a persistent message ID, + so the protocol does not deduplicate those payloads across connections. If + the server committed a frame but its acknowledgement was lost, replay can + write the rows again. ## Trim: how unacked data is reclaimed @@ -151,8 +154,10 @@ On every successful (re)connect: 2. `wireSeq` resets to `0`. 3. The read cursor rewinds to the first un-acked frame on disk (or in memory). -4. Frames stream to the wire in FSN order. The server's dedup window - absorbs any frames that landed before the disconnect. +4. Frames stream to the wire in FSN order. A frame that landed before the + disconnect but was not acknowledged is sent again and can duplicate rows; + use table-level `DEDUP UPSERT KEYS` with stable row identity when this must + be suppressed. 5. New frames appended by the producer during replay are picked up automatically — the I/O loop watches a volatile `publishedFsn` cursor. diff --git a/documentation/high-availability/store-and-forward/when-to-use.md b/documentation/high-availability/store-and-forward/when-to-use.md index 8897ed6a9..8d5683825 100644 --- a/documentation/high-availability/store-and-forward/when-to-use.md +++ b/documentation/high-availability/store-and-forward/when-to-use.md @@ -170,7 +170,13 @@ If you are currently using HTTP or TCP ILP ingest, the comparison is: | Server outage tolerance | Best-effort retry | None | Reconnect loop with multi-minute budget | | Multi-host failover | Yes (HTTP only) | No | Yes | | Cross-region durability ack | No | No | Yes (`request_durable_ack=on`) | -| Cluster-wide ordering | Best-effort | Best-effort | FSN-driven, server-deduplicated | +| Cluster-wide ordering | Best-effort | Best-effort | FSN order within each sender stream, not across senders | + +QWP replay is at-least-once: FSNs track local progress but do not suppress +duplicate writes after a reconnect. Use table-level `DEDUP UPSERT KEYS` +with stable event timestamps and row identity to suppress replayed rows. +See [Delivery semantics](/docs/concepts/delivery-semantics/) for the +[deduplication](/docs/concepts/deduplication/) requirements. The transition is application-transparent — `Sender.fromConfig` accepts a `ws::` or `wss::` connect string and the public builder API is the From caf0fd2588e5f25106d557fcf86b057aac3acc51 Mon Sep 17 00:00:00 2001 From: Marko Topolnik Date: Wed, 21 Oct 2026 15:53:02 +0200 Subject: [PATCH 2/3] Document the 7.2.0 buffer frame cap for C, C++, and Rust --- documentation/changelog.mdx | 3 ++- documentation/connect/clients/c-and-cpp.md | 18 +++++++++++++----- documentation/connect/clients/rust.md | 8 ++++++++ 3 files changed, 23 insertions(+), 6 deletions(-) diff --git a/documentation/changelog.mdx b/documentation/changelog.mdx index 303e4d0cd..0766b379f 100644 --- a/documentation/changelog.mdx +++ b/documentation/changelog.mdx @@ -23,7 +23,8 @@ This page tracks significant updates to the QuestDB documentation. ### Updated -- [C and C++](/docs/connect/clients/c-and-cpp/#binary-columns-uuid-long256-and-opaque-bytes) and [Rust](/docs/connect/clients/rust/#binary-columns-uuid-long256-and-opaque-bytes) clients - Documented the 7.1.0 canonical RFC 4122 UUID byte order at the API boundary, UUID and LONG256 claims for binary Arrow columns through field metadata or per-column overrides, and the silent behavior change for code written against 7.0.0. The C and C++ FetchContent example now pins the 7.1.0 tag +- [C and C++](/docs/connect/clients/c-and-cpp/#binary-columns-uuid-long256-and-opaque-bytes) and [Rust](/docs/connect/clients/rust/#binary-columns-uuid-long256-and-opaque-bytes) clients - Documented the 7.1.0 canonical RFC 4122 UUID byte order at the API boundary, UUID and LONG256 claims for binary Arrow columns through field metadata or per-column overrides, and the silent behavior change for code written against 7.0.0. +- [C and C++](/docs/connect/clients/c-and-cpp/#which-errors-mean-what) and [Rust](/docs/connect/clients/rust/#buffer-ingestion) clients - Documented for 7.2.0 the frame cap a QWP/WebSocket buffer flush must fit, that the buffer size reported by `line_sender_buffer_size` and `Buffer::len()` is at least the encoded frame size, and the `batch_too_large` error an oversized buffer or unsplittable chunk frame returns. The C and C++ FetchContent example now pins the 7.2.0 tag - [Rust client](/docs/connect/clients/rust/#arrow-and-polars-ingestion) - Clarified DataFrame-wide `in_doubt` status versus individual flushes and the risk of duplicating committed rows on retry - [Store-and-forward](/docs/high-availability/store-and-forward/concepts/#reconnect-and-replay) - Corrected replay guidance: connection-local sequence numbers do not deduplicate writes; table-level deduplication with stable row identity is required to suppress duplicates - [Live views](/docs/concepts/live-views/) - Corrected the [supported window functions](/docs/concepts/live-views/#supported-window-functions) to match QuestDB 10.0.1: the statistical functions, EMA, and VWEMA run only in anchored windows, `ksum` does not support `RANGE` frames, `lag` needs a bounded frame or an anchor, and `VARCHAR`, `UUID`, `LONG256`, `DECIMAL`, `BINARY`, and array partition keys are rejected. Added a [rolling standard deviation](/docs/concepts/live-views/#rolling-standard-deviation) recipe for Bollinger bands, rewrote [Freshness](/docs/concepts/live-views/#freshness) to reflect that interval-filtered reads, aggregates, and `ASOF JOIN` see unflushed rows while `LATEST ON` does not, and added the `UPDATE` and `REBASE WAL` invalidation triggers, base WAL retention, and replica freshness. Documented that an out-of-order base row rewrites view rows already emitted, that `view_status` stays `active` while the view's own WAL is suspended, and what to change in `SHOW CREATE LIVE VIEW` output before replaying it diff --git a/documentation/connect/clients/c-and-cpp.md b/documentation/connect/clients/c-and-cpp.md index bca3c57f7..01efa74ab 100644 --- a/documentation/connect/clients/c-and-cpp.md +++ b/documentation/connect/clients/c-and-cpp.md @@ -28,7 +28,7 @@ include(FetchContent) FetchContent_Declare( c_questdb_client_proj GIT_REPOSITORY https://github.com/questdb/c-questdb-client.git - GIT_TAG 7.1.0) # or a newer release tag, or a commit SHA + GIT_TAG 7.2.0) # or a newer release tag, or a commit SHA FetchContent_MakeAvailable(c_questdb_client_proj) target_link_libraries(your_target questdb_client) @@ -465,7 +465,13 @@ your own cadence (row count, byte size via `line_sender_buffer_size`, or a timer) and call `flush` yourself. A good starting cadence is about 1,000 rows or 100 ms, whichever comes first: the auto-flush defaults of the other QWP clients. Flushing every row collapses throughput, and a single flush must -stay under the `sf_max_segment_bytes` payload cap (default 4 MiB). Null columns are +stay under the frame cap: the smallest of `max_buf_size` (default 100 MiB), +the batch size the server advertises, and the `sf_max_segment_bytes` segment +payload capacity (just under 4 MiB by default). `line_sender_buffer_size` +(C++ `buffer.size()`) is at least the size of the frame a flush encodes, so a +buffer flushed before its size reaches the cap fits in one frame. A buffer +over the cap fails with `batch_too_large`, publishes nothing, and keeps its +rows. Null columns are written by **omission**: skip the column for that row. There is no `set_null` call. @@ -1377,9 +1383,10 @@ queue and returns before the server acks. 10 GiB disk mode). On hitting either, `flush` blocks up to `sf_append_deadline_millis` (default 30000), then returns `server_flush_error`. Nothing is dropped or overwritten while blocked. A - buffer whose single payload exceeds the frame cap derived from - `sf_max_segment_bytes` (default 4 MiB) is rejected immediately instead; an oversize - chunk is split across frames rather than rejected. + buffer whose single payload exceeds the frame cap (see + [Sending data: row buffers](#sending-data-row-major)) is rejected + immediately with `batch_too_large` instead; an oversize chunk is split + across frames rather than rejected. ## Concurrency: one borrow per worker {#concurrency-one-borrow-per-worker} @@ -1569,6 +1576,7 @@ Dispatch on `line_sender_error_get_code(err)` (C++ | `failover_retry` | Transient transport failure; frames may be in doubt | Dead — every later call fails | **Drop** the borrow, then re-borrow with `borrow_sender_with_retry(reconnect_max_duration_ms())`. With `sf_dir`, unresolved frames replay automatically. | | `server_rejection` | Server refused the data (schema/type conflict, bad name) | Dead — every later call fails | Plain **return** is safe; the pool retires the connection. Fix the data before re-sending; blind retry re-fails. | | `server_flush_error` | Backpressure deadline hit: queue full for `sf_append_deadline_millis` | Usable | Retry later, shed load, or raise the deadline. Nothing was dropped. See [backpressure](#durability-and-backpressure). | +| `batch_too_large` | A buffer's frame, or a chunk frame that cannot be split further, exceeds the frame cap | Usable | A buffer published nothing and keeps its rows: flush fewer rows at a time, or raise the limit that binds (by default `sf_max_segment_bytes`). A chunk may have queued the frames before the oversized one, so check `in_doubt` before resending it. | | `invalid_api_call` | Borrow still at the pool cap after `acquire_timeout_ms`, pool closed, an operation on a borrow after close, or a durable-level wait without `request_durable_ack=on` | n/a | At-cap: treat as backpressure (see [Sizing the pool](#sizing-the-pool)). Closed pool: stop borrowing. | If you are unsure which case you hit, **return is always safe**: the pool diff --git a/documentation/connect/clients/rust.md b/documentation/connect/clients/rust.md index 85d24925c..f0726b0a4 100644 --- a/documentation/connect/clients/rust.md +++ b/documentation/connect/clients/rust.md @@ -227,6 +227,14 @@ successful local publication and retains its allocated capacity. Use `flush_buffer_and_keep()` only when you need to retain the rows for another destination or operation. +A buffer is published as one frame and is never split, so it must stay under +the frame cap: the smallest of `max_buf_size` (default 100 MiB), the batch size +the server advertises, and the `sf_max_segment_bytes` segment payload capacity +(just under 4 MiB by default). `buffer.len()` is at least the size of the frame +a flush encodes, so a buffer flushed before its length reaches the cap fits in +one frame. A buffer over the cap fails with `ErrorCode::BatchTooLarge`, +publishes nothing, and keeps its rows. + ### Buffer column setters Every setter returns `Result<&mut Buffer>`, so setters can be chained. The From 61c4d7951fe7c6544c84b915f38ad4f7e6e957c7 Mon Sep 17 00:00:00 2001 From: Marko Topolnik Date: Wed, 21 Oct 2026 16:10:14 +0200 Subject: [PATCH 3/3] State replay and in_doubt behavior without arguing with earlier text --- documentation/concepts/delivery-semantics.md | 5 ----- documentation/connect/clients/c-and-cpp.md | 5 ++--- documentation/connect/clients/rust.md | 11 +++++------ .../high-availability/store-and-forward/concepts.md | 6 ++---- .../store-and-forward/when-to-use.md | 6 +++--- 5 files changed, 12 insertions(+), 21 deletions(-) diff --git a/documentation/concepts/delivery-semantics.md b/documentation/concepts/delivery-semantics.md index 9f2c70b84..597c1910c 100644 --- a/documentation/concepts/delivery-semantics.md +++ b/documentation/concepts/delivery-semantics.md @@ -38,11 +38,6 @@ the server confirms a batch, the client reconnects and re-sends. If the server had already committed the batch but the acknowledgement was lost in flight, the second send produces duplicates. -QWP's `wireSeq` cannot suppress this replay: it is assigned by receive order, -exists only for response correlation on one connection, and resets after a -reconnect. Requests carry no persistent message identifier that the server can -use to recognize the same frame on the next connection. - This path applies to every QuestDB client deployment. ### Multi-host failover replay diff --git a/documentation/connect/clients/c-and-cpp.md b/documentation/connect/clients/c-and-cpp.md index 01efa74ab..c1bc47197 100644 --- a/documentation/connect/clients/c-and-cpp.md +++ b/documentation/connect/clients/c-and-cpp.md @@ -702,9 +702,8 @@ on_error:; the 2 MiB target; if 8 rows still exceed 4 MiB — which takes very large string, binary, or array values — the flush fails instead of splitting. - **Recovery depends on `in_doubt`, not on the error code.** Check - `line_sender_error_in_doubt` (C++: `e.in_doubt()`). This describes the - failed operation's input, not earlier independent flushes or replay from an - application checkpoint. False means the queue + `line_sender_error_in_doubt` (C++: `e.in_doubt()`). This describes only the + failed operation's input. False means the queue never took the frame and the chunk is intact: re-flush it. True means delivery is uncertain, so `wait` for what the queue already holds, and resend the chunk only where the table's dedup keys make duplicate rows harmless. A diff --git a/documentation/connect/clients/rust.md b/documentation/connect/clients/rust.md index f0726b0a4..7ba54df9c 100644 --- a/documentation/connect/clients/rust.md +++ b/documentation/connect/clients/rust.md @@ -496,8 +496,8 @@ confirmed by an earlier checkpoint, even when a later batch fails validation. Internal retries and connection replacements retain this call-level status. The flag is conservative: it does not identify a safe row offset for resuming the load. A false flag does not make a validation error retryable; correct the -input first. Low-level Arrow and chunk flushes retain their current-operation -scope and do not aggregate earlier independent calls. +input first. For low-level Arrow and chunk flushes, the flag covers only the +failed call. ### Binary columns: UUID, LONG256, and opaque bytes @@ -764,10 +764,9 @@ publishes or completes its first frame, so treat `None` from `acked_fsn()` as Recovery turns on `err.in_doubt()`, not on `err.code()` and not on whether the buffer or chunk still holds rows. -For these low-level APIs, the flag describes the failed operation's input. It -does not summarize earlier independent flushes or authorize replaying all data -since an application checkpoint. The DataFrame-level aggregation described -above applies specifically to `flush_polars_dataframe`. +For these low-level APIs, the flag describes only the failed operation's +input. The DataFrame-level aggregation described above applies specifically to +`flush_polars_dataframe`. When `in_doubt()` is `false`, the flush failed before the queue took the frame, so the rows never entered the send path and your input is intact: re-flush it. diff --git a/documentation/high-availability/store-and-forward/concepts.md b/documentation/high-availability/store-and-forward/concepts.md index 6591c5bfc..26c0e7ae6 100644 --- a/documentation/high-availability/store-and-forward/concepts.md +++ b/documentation/high-availability/store-and-forward/concepts.md @@ -84,10 +84,8 @@ Two consequences: serialise `wireSeq` — the server assigns it implicitly from receive order. Reordering breaks the FSN mapping. - After a reconnect, the server sees the **same payloads** at new `wireSeq` - values. QWP requests contain neither `wireSeq` nor a persistent message ID, - so the protocol does not deduplicate those payloads across connections. If - the server committed a frame but its acknowledgement was lost, replay can - write the rows again. + values. If the server committed a frame but its acknowledgement was lost, + replay writes those rows again. ## Trim: how unacked data is reclaimed diff --git a/documentation/high-availability/store-and-forward/when-to-use.md b/documentation/high-availability/store-and-forward/when-to-use.md index 8d5683825..da6a84c0e 100644 --- a/documentation/high-availability/store-and-forward/when-to-use.md +++ b/documentation/high-availability/store-and-forward/when-to-use.md @@ -172,9 +172,9 @@ If you are currently using HTTP or TCP ILP ingest, the comparison is: | Cross-region durability ack | No | No | Yes (`request_durable_ack=on`) | | Cluster-wide ordering | Best-effort | Best-effort | FSN order within each sender stream, not across senders | -QWP replay is at-least-once: FSNs track local progress but do not suppress -duplicate writes after a reconnect. Use table-level `DEDUP UPSERT KEYS` -with stable event timestamps and row identity to suppress replayed rows. +QWP replay is at-least-once, so a reconnect can write rows twice. Use +table-level `DEDUP UPSERT KEYS` with stable event timestamps and row identity +to suppress replayed rows. See [Delivery semantics](/docs/concepts/delivery-semantics/) for the [deduplication](/docs/concepts/deduplication/) requirements.