docs(qwp): document canonical RFC 4122 UUID byte order

documentation/changelog.mdx

@@ -61,6 +61,8 @@ This page tracks significant updates to the QuestDB documentation.
 
 ### Updated
 
+- [Python client](/docs/connect/clients/python/) - Documented 5.1.0 QWP-only row types, UUID/LONG256 binary column overrides and byte-order migration, and pandas type-preserving round trips including DATE and read-only type metadata
+- [Python client](/docs/connect/clients/python/#dataframe-ingestion) - Noted that `dataframe()` is not atomic, so retrying a failed call can duplicate rows it already committed
 - [query_activity()](/docs/query/functions/meta/#query_activity) - Documented the `is_wal`, `memory_used`, and `memory_limit` columns
 - [wal_tables()](/docs/query/functions/meta/#wal_tables) - Documented the `errorTag`, `errorMessage`, and `memoryPressure` columns; `errorTag` reads `OUT OF MEMORY` after a WAL apply memory limit breach
 - [SHOW](/docs/query/sql/show/) - `SHOW USERS`, `SHOW GROUPS`, and `SHOW SERVICE ACCOUNTS`, including their filtered forms, gain a trailing `memory_limit` column. Clients that read these results by position need [updating](/docs/security/rbac/#memory-limit-upgrade)

documentation/connect/clients/python.md

@@ -277,6 +277,13 @@ The Python value type selects the QuestDB column type:
 | `TimestampMicros`, `TimestampNanos`, `datetime.datetime` | `TIMESTAMP`, `TIMESTAMP_NS` |
 | `numpy.ndarray` of `float64`, any number of dimensions | `DOUBLE[]`, `DOUBLE[][]`, ... matching the array's shape. QuestDB 9.0.0 or later |
 | `decimal.Decimal` | `DECIMAL`, QuestDB 9.2.0 or later |
+| `uuid.UUID` | `UUID`, QWP only |
+| `ipaddress.IPv4Address` | `IPV4`, QWP only |
+| `bytes`, `bytearray`, `memoryview` | `BINARY`, QWP only |
+| `Char` | `CHAR`, QWP only |
+| `DateMillis` | `DATE`, QWP only |
+| `Long256` | `LONG256`, QWP only |
+| `Geohash` | `GEOHASH`, QWP only |
 | `None` | Column omitted for this row, stored as null |
 
 Nulls are written by omission: skip the key or pass `None`; there is no
@@ -287,10 +294,60 @@ strings in `columns` become `VARCHAR`. `DECIMAL` columns must be created
 ahead of time with `CREATE TABLE ... (price DECIMAL(18, 2), ...)`; the server
 does not auto-create them.
 
-`UUID`, `IPv4`, `GEOHASH`, `LONG256`, `CHAR`, `DATE`, and `BINARY` columns
-have no `row()` value type. Route them through
-[`dataframe()`](#dataframe-ingestion), whose `schema_overrides` covers
-`symbol`, `ipv4`, `char`, and `geohash`, or through a SQL `INSERT` via
+The seven types marked "QWP only" need Python client 5.1.0 or later,
+QuestDB 10 or later, and a `udp`, `ws`, or `wss` connection. On a `tcp`,
+`tcps`, `http`, or `https` sender they raise `QuestDBError`.
+
+Three of them map to a Python type you already have: `uuid.UUID`,
+`ipaddress.IPv4Address`, and any bytes-like value. The other four have no
+obvious Python equivalent, so the client gives you a small wrapper for each:
+
+| Wrapper | Takes |
+| --- | --- |
+| `Char("A")` | a one-character string |
+| `DateMillis(1735689600000)` | milliseconds since the Unix epoch |
+| `Long256(0xdeadbeef)` | an unsigned 256-bit `int` |
+| `Geohash(bits, precision)` | the hash bits and how many bits they use |
+| `Geohash.from_string("u33d8b12")` | the text form, 1 to 12 characters |
+
+A geohash column uses one precision for every row. The first row you write
+fixes it, and if a later row has a different precision, `row()` raises
+`QuestDBError` straight away. The bad row is removed, so the rest of the
+buffer is untouched and you can carry on writing.
+
+```python
+import uuid
+from questdb import Char, DateMillis, Geohash, Long256, TimestampNanos
+
+sender.row(
+    "events",
+    columns={
+        "id": uuid.UUID("123e4567-e89b-12d3-a456-426614174000"),
+        "payload": b"\x00\x01",
+        "grade": Char("A"),
+        "day": DateMillis(1735689600000),
+        "hash": Long256(0xdeadbeef),
+        "loc": Geohash.from_string("u33d8b12"),
+    },
+    at=TimestampNanos.now(),
+)
+```
+
+For these types, QuestDB uses one value to mean `NULL`. If you write that
+value, it reads back as `NULL`:
+
+| Type | Value that reads back as `NULL` |
+| --- | --- |
+| `IPV4` | `0.0.0.0` |
+| `DATE` | `INT64_MIN` |
+| `UUID` | `80000000-0000-0000-8000-000000000000` |
+| `LONG256` | all four 64-bit limbs set to `0x8000000000000000` |
+
+`CHAR` and `BINARY` have no such value: `Char("\x00")` and `b""` are stored
+as themselves, not as `NULL`.
+
+You can also write these types with
+[`dataframe()`](#dataframe-ingestion), or with a SQL `INSERT` through
 [`query()`](#querying).
 
 QWP cannot preserve nulls for `BOOLEAN`, `BYTE`, or `SHORT`. An absent value
@@ -403,6 +460,11 @@ with questdb.connect("ws::addr=localhost:9000;") as db:
     db.dataframe(df, table_name="trades", symbols=["symbol"], at="timestamp")
 ```
 
+A `dataframe()` call is not atomic: batches commit as the call proceeds, so a
+call that fails partway through can leave its first rows in the table. Such
+an error has `in_doubt=True`, and retrying the whole call duplicates those
+rows unless the table uses [deduplication](/docs/concepts/deduplication/).
+
 `df` accepts pandas `DataFrame`, polars `DataFrame` and `LazyFrame`, pyarrow
 `Table`, `RecordBatch`, and `RecordBatchReader`, and any object exposing the
 Arrow C Data Interface:
@@ -430,7 +492,7 @@ Parameters:
 | `symbols` | `"auto"` (default: categorical and dictionary columns become `SYMBOL`), a bool, or a list of column names or indices. |
 | `at` | The designated timestamp column (by name or index), a fixed `TimestampNanos` or `datetime` shared by every row, or `questdb.ServerTimestamp`. |
 | `max_rows_per_batch` | Rows per published batch, default 16384. Sets pipelining granularity, not a safety limit — see below. |
-| `schema_overrides` | Per-column wire-type overrides, e.g. `{"addr": "ipv4", "loc": ("geohash", 20)}`; values are `symbol`, `ipv4`, `char`, or `geohash`. |
+| `schema_overrides` | Per-column type, e.g. `{"addr": "ipv4", "loc": ("geohash", 20)}`. Values are `symbol`, `ipv4`, `char`, `uuid`, `long256`, or `("geohash", bits)` with `bits` from 1 to 60. Beats any Arrow field metadata on the column. Needs a frame where every column is Arrow-backed; otherwise it raises `UnsupportedDataFrameShapeError`. |
 
 `max_rows_per_batch` decides how the frame is cut into published batches,
 and each batch is one unit of encoding, memory, and server-side apply.
@@ -439,9 +501,8 @@ negotiated per-batch byte cap regardless of this setting, and a single row
 is never bounded by it. What it does control:
 
 - Peak client memory: each batch is encoded and held as one frame.
-- Recovery quantum: a commit checkpoint fires every 100 batches, so
-  `max_rows_per_batch × 100` rows is the replay window on a transient
-  failover.
+- Commit frequency: the client commits about every 100 batches, so roughly
+  `max_rows_per_batch × 100` rows are in flight between commits.
 - Per-batch overhead: very small batches pay framing and server-side
   apply costs per batch.
 
@@ -461,6 +522,126 @@ row ingestion. A frame the columnar path cannot express raises
 `UnsupportedDataFrameShapeError` with per-column failures in
 `column_failures`.
 
+### Binary, UUID, and LONG256 columns
+
+Binary columns (`pyarrow.binary()`, `pyarrow.large_binary()`, fixed-size
+`pyarrow.binary(n)`, and polars `Binary`) are written as `BINARY`. The width
+of a column does not decide its type, so a 16-byte column is treated as
+plain bytes, not as a UUID.
+
+<details>
+<summary>Breaking changes when upgrading from Python client 5.0 or earlier</summary>
+
+- 16- and 32-byte binary columns are no longer inferred as UUID or LONG256.
+  Mark them with `schema_overrides`, or use the `arrow.uuid` extension type
+  for UUIDs, as shown below.
+- UUID bytes are now big-endian, the same as `uuid.UUID.bytes`. Remove any
+  byte-swapping from your ingestion and query code, for example replace
+  `value.int.to_bytes(16, "little")` with `value.bytes`. Otherwise UUIDs are
+  stored reversed. Columns of `uuid.UUID` objects and query binds are
+  unaffected.
+
+</details>
+
+For UUIDs, an object column of `uuid.UUID` values needs no extra
+configuration. The client handles the byte order:
+
+```python
+import uuid
+
+df = pd.DataFrame({
+    "trade_id": [uuid.uuid4(), uuid.uuid4()],
+    "price": [2615.54, 65432.10],
+    "timestamp": pd.to_datetime([
+        "2025-01-01T00:00:00Z",
+        "2025-01-01T00:00:01Z",
+    ]),
+})
+
+db.dataframe(df, table_name="trades", at="timestamp")
+```
+
+For columns that are already binary, name the types with
+`schema_overrides`. The input must be fully Arrow-backed. This complete
+example builds a pyarrow table with both binary columns:
+
+```python
+from datetime import datetime, timezone
+import uuid
+
+import pyarrow as pa
+import questdb
+
+trade_uuids = [
+    uuid.UUID("123e4567-e89b-12d3-a456-426614174000"),
+    uuid.UUID("123e4567-e89b-12d3-a456-426614174001"),
+]
+table = pa.table({
+    "trade_id": pa.array([u.bytes for u in trade_uuids], type=pa.binary(16)),
+    "order_hash": pa.array(
+        [n.to_bytes(32, "little") for n in (0xdeadbeef, 0xcafebabe)],
+        type=pa.binary(32),
+    ),
+    "price": pa.array([2615.54, 65432.10], type=pa.float64()),
+    "timestamp": pa.array([
+        datetime(2025, 1, 1, tzinfo=timezone.utc),
+        datetime(2025, 1, 1, 0, 0, 1, tzinfo=timezone.utc),
+    ], type=pa.timestamp("us", "UTC")),
+})
+
+with questdb.connect("ws::addr=localhost:9000;") as db:
+    db.dataframe(
+        table,
+        table_name="trades",
+        at="timestamp",
+        schema_overrides={"trade_id": "uuid", "order_hash": "long256"},
+    )
+```
+
+Each non-null value must be exactly 16 bytes for `uuid` or 32 bytes for
+`long256`. UUID bytes are big-endian, as in `uuid.UUID.bytes`. LONG256 bytes
+are a little-endian 256-bit integer, as in `n.to_bytes(32, "little")`.
+
+The third way is a pyarrow column built with the `arrow.uuid` extension type,
+which marks the column as a UUID. Using `trade_uuids` from the example above:
+
+```python
+import pyarrow as pa
+
+trade_ids = pa.ExtensionArray.from_storage(
+    pa.uuid(),
+    pa.array([u.bytes for u in trade_uuids], type=pa.binary(16)),
+)
+```
+
+This needs pyarrow 18 or later. Setting `ARROW:extension:name` in field
+metadata does not work in a pandas frame, which keeps a column's Arrow type
+but not its field metadata.
+
+:::note
+
+In a frame that mixes Arrow-backed and NumPy columns, `schema_overrides` is
+unavailable, so the client rejects an unlabelled 16- or 32-byte column rather
+than guess its type. To write it as `BINARY`, pass it as an object column of
+`bytes`.
+
+:::
+
+### DATE columns
+
+`row()` writes a `DATE` with the `DateMillis` wrapper. For `dataframe()`,
+use an Arrow column of `pyarrow.timestamp("ms")`, `pyarrow.date32()`, or
+`pyarrow.date64()`. There is no `date` value for `schema_overrides`.
+
+A NumPy `datetime64[ms]` column is written as a microsecond `TIMESTAMP`, and
+its timezone-aware `datetime64[ms, tz]` form is rejected, unless
+`df.attrs["questdb"]` marks the column as `date`.
+
+`DATE` query results write back as `DATE`. `to_arrow()` and
+`to_pandas(dtype_backend="pyarrow")` return a `timestamp("ms", "UTC")`
+column, and plain `to_pandas()` marks the column as `date` in
+`df.attrs["questdb"]`. See [Writing a result back](#writing-a-result-back).
+
 Naive timestamps — DataFrame columns and the scalar `at` alike — are
 interpreted as UTC, matching the numpy `datetime64` convention. Prefer
 timezone-aware values throughout.
@@ -585,12 +766,70 @@ thread that created it; `db.close()` waits for open leases.
 | `SYMBOL` | `Categorical` sharing one dictionary across batches |
 | `VARCHAR` | Strings with `None` for null |
 | `DECIMAL`, `UUID`, `BINARY` | `object` columns of `decimal.Decimal`, `uuid.UUID`, `bytes` |
+| `LONG256` | `object` column of Python `int` — the only type wide enough without pyarrow |
+| `IPV4`, `CHAR` | `uint32`, `uint16` |
+| `DATE` | Timezone-naive `datetime64[ms]` holding UTC instants, `NaT` for null |
+| `GEOHASH` | A signed integer wide enough for the column's precision: `int8` up to 7 bits, `int16` to 15, `int32` to 31, `int64` to 60 |
 
 QuestDB's sentinel values (for example `NaN` doubles and `INT64_MIN` longs)
 are decoded as nulls rather than leaking as magic numbers.
-`to_pandas(dtype_backend="pyarrow")`, `dtype_backend="numpy_nullable"`, or
-a `types_mapper=` callable select pyarrow-backed dtypes instead, matching
-the `pd.read_sql` convention.
+`to_pandas(dtype_backend="pyarrow")` selects Arrow-backed dtypes;
+`dtype_backend="numpy_nullable"` selects pandas nullable dtypes where
+available. A `types_mapper=` callable provides custom Arrow-to-pandas
+dtype mapping.
+
+### Writing a result back
+
+Python client 5.1.0 and later preserves UUID, LONG256, IPV4, CHAR, GEOHASH,
+and DATE types when you read a table into pandas, change values, and write
+it back:
+
+```python
+df = db.query("SELECT * FROM trades").to_pandas()
+df["price"] *= 1.01
+db.dataframe(df, table_name="trades_adjusted", at="timestamp")
+```
+
+A pandas dtype does not always identify the QuestDB type. For example,
+`IPV4` arrives as `uint32`, which would otherwise be written as `LONG`.
+`to_pandas()` records the source types in `df.attrs["questdb"]`, and
+`dataframe()` reads that metadata to preserve them.
+
+This works with every `dtype_backend`. With `"numpy_nullable"`, `UUID` and
+`LONG256` columns hold `bytes` rather than `uuid.UUID` and `int`.
+
+Editing the frame is safe. A column you drop, rename, or convert to another
+type simply loses its record, and the write goes ahead with whatever the
+column now is. `symbols` and `schema_overrides` win over it, so you can
+always state a type yourself. Two types do not survive unchanged: `BYTE` and
+`SHORT` columns come back as `INT`, and `INT` as `LONG`.
+
+If a recorded type cannot apply to the column as it now stands — say the
+column is an unsigned integer and the record says `geohash` — the client
+warns and writes the column as its own type implies, rather than failing.
+
+`df.attrs["questdb"]` is read-only, nested mappings included, and editing it
+in place raises `TypeError`. To change it, assign a new dictionary, unpacking
+the existing `columns` mapping to keep its other entries:
+
+```python
+df.attrs["questdb"] = {
+    "version": 1,
+    "columns": {
+        **df.attrs["questdb"]["columns"],
+        "src_ip": {"kind": "ipv4"},
+        "pos": {"kind": "geohash", "precision_bits": 20},
+        "traded_on": {"kind": "date"},
+    },
+}
+```
+
+For a hand-built frame with no existing metadata, omit the unpacking line.
+`version` is required and must be `1`. `kind` is one of `uuid`, `long256`,
+`ipv4`, `char`, `geohash`, or `date`; `precision_bits` goes with `geohash`
+only. Naming a column that is not in the frame does no harm; it is ignored.
+A dictionary without `version`, or with a version this client does not
+know, is ignored completely.
 
 ### DDL, DML, and cancellation
 
@@ -846,7 +1085,7 @@ All failures raise `QuestDBError` (or a subclass). Inspect:
 | Property | Meaning |
 | --- | --- |
 | `code` | A `QuestDBErrorCode` member; compare by identity, e.g. `err.code is QuestDBErrorCode.Cancelled`. |
-| `in_doubt` | `True` when the failed operation may already have delivered its input; retrying can duplicate rows without deduplication. |
+| `in_doubt` | `True` when the failed ingestion operation may already have delivered its input. For QWP `dataframe()`, this includes earlier batches from the same call, even on a later validation error. Retrying can duplicate rows without deduplication. |
 | `sender_error` | Structured server diagnostic for QWP sender failures, or `None`. |
 
 Codes you will most often dispatch on: