diff --git a/documentation/deployment/aws.md b/documentation/deployment/aws.md index b94f1e64a3..69883859c9 100644 --- a/documentation/deployment/aws.md +++ b/documentation/deployment/aws.md @@ -14,7 +14,8 @@ import CodeBlock from "@theme/CodeBlock" |-----------|-------------|-------| | Instance | `m7i.xlarge` or `r7i.2xlarge` | 4-8 vCPUs, 16-64 GiB RAM | | Storage | `gp3`, 200+ GiB | 16000 IOPS / 1000 MBps | -| File system | `zfs` with `lz4` | Or `ext4` if compression not needed | +| File system | `ext4` or `xfs` | Compression is handled by Parquet, not the file system | +| Compression | Storage policy with `TO PARQUET` | Add `TO REMOTE` to tier history to S3 (Enterprise) | | Ports | 9000, 8812, 9009, 9003 | Restrict to known IPs only | --- @@ -75,13 +76,30 @@ before provisioning storage. **File system:** -Use `zfs` with `lz4` compression to reduce storage costs. If you don't need -compression, `ext4` or `xfs` offer slightly better performance. +Format the data disk as `ext4` or `xfs`. QuestDB compresses data itself, so the +file system does not need to. + +**Compression and tiering:** + +QuestDB compresses data by storing older partitions as +[Parquet](/docs/concepts/parquet/#data-compression), which uses `lz4_raw` +compression by default and stays fully queryable. In QuestDB Enterprise, a +[storage policy](/docs/concepts/storage-policy/) with a `TO PARQUET` stage +converts partitions automatically as they age. In QuestDB open source, convert +them manually with +[`ALTER TABLE CONVERT PARTITION`](/docs/concepts/parquet/#converting-to-parquet). + +For further savings in Enterprise, add a `TO REMOTE` stage so the policy also +tiers Parquet partitions to S3 [cold storage](/docs/concepts/cold-storage/). S3 +usually costs less than EBS, the cluster keeps a single copy that every instance +reads instead of one copy per replica, and history can grow beyond the size of +a single volume. **Unsupported storage:** - **EFS** - Not supported. Network latency is too high for database workloads. -- **S3** - Not supported as primary storage. Use for replication (Enterprise only). +- **S3** - Not supported as primary storage. Use for replication and cold + storage (Enterprise only). ### Networking diff --git a/documentation/deployment/azure.md b/documentation/deployment/azure.md index edd2c8b496..846726fce9 100644 --- a/documentation/deployment/azure.md +++ b/documentation/deployment/azure.md @@ -15,7 +15,8 @@ import CodeBlock from "@theme/CodeBlock" |-----------|-------------|-------| | Instance | `D4s_v5` or `E8s_v5` | 4-8 vCPUs, 16-64 GiB RAM | | Storage | Premium SSD v2, 200+ GiB | 16000 IOPS / 1000 MBps | -| File system | `zfs` with `lz4` | Or `ext4` if compression not needed | +| File system | `ext4` or `xfs` | Compression is handled by Parquet, not the file system | +| Compression | Storage policy with `TO PARQUET` | Add `TO REMOTE` to tier history to Blob Storage (Enterprise) | | Ports | 9000, 8812, 9009, 9003 | Restrict to known IPs only | --- @@ -100,14 +101,32 @@ capacity to get required IOPS. **File system:** -Use `zfs` with `lz4` compression to reduce storage costs. If you don't need -compression, `ext4` or `xfs` offer slightly better performance. +Format the data disk as `ext4` or `xfs`. QuestDB compresses data itself, so the +file system does not need to. + +**Compression and tiering:** + +QuestDB compresses data by storing older partitions as +[Parquet](/docs/concepts/parquet/#data-compression), which uses `lz4_raw` +compression by default and stays fully queryable. In QuestDB Enterprise, a +[storage policy](/docs/concepts/storage-policy/) with a `TO PARQUET` stage +converts partitions automatically as they age. In QuestDB open source, convert +them manually with +[`ALTER TABLE CONVERT PARTITION`](/docs/concepts/parquet/#converting-to-parquet). + +For further savings in Enterprise, add a `TO REMOTE` stage so the policy also +tiers Parquet partitions to Blob Storage +[cold storage](/docs/concepts/cold-storage/). Blob Storage usually costs less +than managed disks, the cluster keeps a single copy that every instance reads +instead of one copy per replica, and history can grow beyond the size of a +single disk. **Unsupported storage:** - **Azure NetApp Files** - Not supported as primary storage (NFS latency too high) - **blobfuse2** - Not supported for database workloads -- **Blob Storage** - Supported for Enterprise replication only, not primary storage +- **Blob Storage** - Supported for Enterprise replication and cold storage only, + not primary storage ### Networking diff --git a/documentation/deployment/compression-zfs.md b/documentation/deployment/compression-zfs.md index 7338c6f2dd..59145709d3 100644 --- a/documentation/deployment/compression-zfs.md +++ b/documentation/deployment/compression-zfs.md @@ -5,6 +5,22 @@ description: How to install ZFS and enable system-level compression for QuestDB. --- +:::warning Use Parquet for compression + +The preferred way to compress QuestDB data is to convert partitions to +[Parquet](/docs/concepts/parquet/#data-compression). QuestDB compresses Parquet +partitions natively and keeps them fully queryable, with no file system +requirements. In QuestDB Enterprise, a +[storage policy](/docs/concepts/storage-policy/) converts partitions +automatically as they age, and can also tier them to object storage. In QuestDB +open source, convert partitions manually with +[`ALTER TABLE CONVERT PARTITION`](/docs/concepts/parquet/#converting-to-parquet). + +ZFS compression works at the file system level, outside the database. This page +remains for existing deployments that already run on ZFS. + +::: + QuestDB can use [ZFS](https://openzfs.org/wiki/Main_Page) for system-level compression, reducing disk usage without application changes. diff --git a/documentation/deployment/docker.md b/documentation/deployment/docker.md index 57563e8af7..012fe52e6a 100644 --- a/documentation/deployment/docker.md +++ b/documentation/deployment/docker.md @@ -218,14 +218,6 @@ available on `http://localhost:4000`: docker run -v "$(pwd):/var/lib/questdb/conf" -p 4000:4000 questdb/questdb ``` -:::note - -If you wish to use ZFS for your QuestDB deployment, with Docker, then you will need to enable ZFS on the host volume that Docker uses. - -Please see the [docker documentation](https://docs.docker.com/storage/storagedriver/zfs-driver/) for more information. - -::: - ### Upgrade QuestDB version It is possible to upgrade your QuestDB instance on Docker when a volume is diff --git a/documentation/deployment/gcp.md b/documentation/deployment/gcp.md index a0ed747b94..9f2784f087 100644 --- a/documentation/deployment/gcp.md +++ b/documentation/deployment/gcp.md @@ -16,7 +16,8 @@ import MinimumHardware from "../../src/components/DRY/_questdb_production_hardwa |-----------|-------------|-------| | Instance | `c3-standard-4` or `c3-highmem-8` | 4-8 vCPUs, 16-64 GiB RAM | | Storage | Hyperdisk Balanced data disk, 300+ GiB | 5000 IOPS / 300 MBps for production | -| File system | `ext4` | Or `zfs` with `lz4` for on-disk compression | +| File system | `ext4` | Compression is handled by Parquet, not the file system | +| Compression | Storage policy with `TO PARQUET` | Add `TO REMOTE` to tier history to GCS (Enterprise) | | Ports | 9000, 8812, 9009, 9003 | Restrict to known IPs only | --- @@ -76,16 +77,31 @@ cannot be used on `C3` machines smaller than `88 vCPUs`. **File system:** -Format the data disk as `ext4`. If you need on-disk compression, use `zfs` with -`lz4` instead, at a small performance cost. See -[ZFS compression](/docs/deployment/compression-zfs/). +Format the data disk as `ext4`. QuestDB compresses data itself, so the file +system does not need to. + +**Compression and tiering:** + +QuestDB compresses data by storing older partitions as +[Parquet](/docs/concepts/parquet/#data-compression), which uses `lz4_raw` +compression by default and stays fully queryable. In QuestDB Enterprise, a +[storage policy](/docs/concepts/storage-policy/) with a `TO PARQUET` stage +converts partitions automatically as they age. In QuestDB open source, convert +them manually with +[`ALTER TABLE CONVERT PARTITION`](/docs/concepts/parquet/#converting-to-parquet). + +For further savings in Enterprise, add a `TO REMOTE` stage so the policy also +tiers Parquet partitions to Google Cloud Storage +[cold storage](/docs/concepts/cold-storage/). GCS usually costs less than +Hyperdisk, the cluster keeps a single copy that every instance reads instead of +one copy per replica, and history can grow beyond the size of a single disk. **Unsupported storage:** - **Filestore** - Not supported as primary storage (NFS latency too high). Use for Enterprise replication only (see [Enterprise on GCP](#enterprise-on-gcp)). - **Google Cloud Storage** - Not supported as primary storage. Use for - Enterprise replication only. + Enterprise replication and cold storage only. ### Networking @@ -282,8 +298,8 @@ gcloud compute ssh questdb-europe-west3 --zone europe-west3-a Run these commands on the VM over SSH. They assume the attached data disk uses the custom device name `questdb-data` and format it as `ext4`, which is the -recommended default. If you need on-disk compression, format the data disk as -`zfs` with `lz4` instead. See [ZFS compression](/docs/deployment/compression-zfs/). +recommended default. To compress data, use Parquet rather than the file system, +as described in [Compression and tiering](#storage). :::warning The `mkfs.ext4` command erases the target device. Run it only on a new blank diff --git a/documentation/deployment/hetzner.md b/documentation/deployment/hetzner.md index 6a5b4d35f6..593a00c3ff 100644 --- a/documentation/deployment/hetzner.md +++ b/documentation/deployment/hetzner.md @@ -5,7 +5,6 @@ description: This document explains what hardware to use, and how to provision QuestDB on Hetzner Cloud with storage volumes and automated backups. --- -import FileSystemChoice from "../../src/components/DRY/_questdb_file_system_choice.mdx" import MinimumHardware from "../../src/components/DRY/_questdb_production_hardware-minimums.mdx" import InterpolateReleaseData from "../../src/components/InterpolateReleaseData" import CodeBlock from "@theme/CodeBlock" diff --git a/documentation/getting-started/capacity-planning.md b/documentation/getting-started/capacity-planning.md index e577dfafb0..02924278f4 100644 --- a/documentation/getting-started/capacity-planning.md +++ b/documentation/getting-started/capacity-planning.md @@ -60,13 +60,7 @@ For optimum performance, utilize the maximum settings: ### Supported filesystems -To enable compression and to match our recommended performance profile, we -recommend using [ZFS file system](https://en.wikipedia.org/wiki/ZFS). - -ZFS is required for system-level compression. - -While ZFS is recommended, QuestDB open source supports the following -filesystems: +QuestDB supports the following filesystems: - APFS - EXT4 @@ -94,10 +88,18 @@ QuestDB database. ### Data compression -To enable data compression, filesystem must be ZFS. - -For instructions on how to do so, see the -[ZFS and compression](/docs/deployment/compression-zfs/) guide. +QuestDB compresses data by converting partitions to +[Parquet](/docs/concepts/parquet/#data-compression), which uses `lz4_raw` +compression by default and keeps partitions fully queryable. In QuestDB +Enterprise, a [storage policy](/docs/concepts/storage-policy/) converts +partitions automatically as they age, and can also tier them to +[cold storage](/docs/concepts/cold-storage/) in object storage. In QuestDB open +source, convert partitions manually with +[`ALTER TABLE CONVERT PARTITION`](/docs/concepts/parquet/#converting-to-parquet). + +Compression does not depend on the file system. File-system compression with +[ZFS](/docs/deployment/compression-zfs/) also works, but Parquet is the +recommended approach. ### Write amplification diff --git a/documentation/getting-started/enterprise-quick-start.md b/documentation/getting-started/enterprise-quick-start.md index 86dbd3e2ef..6718948896 100644 --- a/documentation/getting-started/enterprise-quick-start.md +++ b/documentation/getting-started/enterprise-quick-start.md @@ -29,9 +29,8 @@ inform your own unique choices. [5. Ingest data, Kafka Connect (optional)](#5-ingest-data-kafka-connect-optional)\ [6. Query data, PostgreSQL query](#6-query-data-postgresql-query)\ [7. Setup replication](#7-setup-replication)\ -[8. Enable compression](#8-enable-compression)\ -[9. Configure a storage policy](#9-configure-a-storage-policy)\ -[10. Double-check kernel limits](#10-double-check-kernel-limits)\ +[8. Compress and tier data with storage policies](#8-compress-and-tier-data-with-storage-policies)\ +[9. Double-check kernel limits](#9-double-check-kernel-limits)\ [Next steps](#next-steps)\ [FAQ](#faq) @@ -46,7 +45,6 @@ The following are required prior to following this guide: - Buying through Azure? QuestDB Enterprise is also available via [Azure Marketplace](/docs/deployment/azure/#buy-through-azure-marketplace). - Use of a [supported file system](/docs/getting-started/capacity-planning/#supported-filesystems) - - A [Zettabyte File System (ZFS)](https://openzfs.org/wiki/Main_Page) is recommended to enable compression ## Installation guide @@ -439,50 +437,112 @@ Full instructions can be found within the snapshot provides a starting point for the instance, which will soon catch up with the primary node. -## 8. Enable compression +## 8. Compress and tier data with storage policies -Compression requires the -[Zettabyte File System (ZFS)](https://openzfs.org/wiki/Main_Page). +[Storage policies](/docs/concepts/storage-policy/) automate what happens to +partitions as they age. They are not only for moving data to cold storage: the +`TO PARQUET` stage is also how QuestDB Enterprise compresses data on local +volumes. It converts partitions to +[Parquet](/docs/concepts/parquet/#data-compression), compressed with `lz4_raw` +by default, and removes the native files. Converted partitions stay on local +disk, remain fully queryable, and still accept late writes. -We'll assume Ubuntu, and demonstrate the basics CLI commands which you'd apply -in something like an AWS EC2 to enable ZFS: +A policy has up to four stages, each with its own delay: -```bash title="Ubuntu - Install ZFS" -sudo apt update -sudo apt install zfsutils-linux +- `TO PARQUET` converts the partition to compressed Parquet on local disk. +- `TO REMOTE` uploads a Parquet copy of the partition to object storage. +- `DROP LOCAL` removes the local copy. With `TO REMOTE` set, queries switch to + the remote copy. Without it, the data is deleted. +- `DROP REMOTE` deletes the remote copy. + +Storage policies replace plain TTL in Enterprise, where `ALTER TABLE SET TTL` +with a non-zero value is rejected. + +:::note ZFS compression +File-system compression with ZFS is no longer the recommended way to compress +QuestDB data. It works outside the database, while Parquet compression is +managed by QuestDB per partition. If you already run on ZFS, see +[ZFS compression](/docs/deployment/compression-zfs/). +::: + +### Compress partitions on local disk + +A policy with only a `TO PARQUET` stage compresses older partitions and keeps +all data on local disk. Attach it at table creation: + +```questdb-sql title="Web Console - Create a table with a storage policy" +CREATE TABLE trades ( + ts TIMESTAMP, + symbol SYMBOL, + price DOUBLE +) TIMESTAMP(ts) PARTITION BY DAY + STORAGE POLICY(TO PARQUET 3d); ``` -To enable compression, create a ZPool with compression enabled: +Or attach it to an existing table: -```shell title="Ubuntu - Enable compression" -zpool create -m legacy -o feature@lz4_compress=enabled autoexpand=on -O compression=lz4 -t volume1 questdb-pool sdf +```questdb-sql title="Web Console - Set a storage policy on an existing table" +ALTER TABLE trades SET STORAGE POLICY(TO PARQUET 3 DAYS); ``` -The exact commands depend on which version of ZFS you are running. Use the -[ZFS docs](https://openzfs.github.io/openzfs-docs/man/master/8/zpool-create.8.html) -to customize your ZFS to meet your requirements. +Set the `TO PARQUET` delay beyond the window in which late or out-of-order rows +usually arrive. Rows that land on a Parquet partition cause QuestDB to rewrite +it. To trade CPU for a higher compression ratio, change the codec, for example +to `ZSTD`, as described in +[Parquet data compression](/docs/concepts/parquet/#data-compression). -If you are running QuestDB Enterprise in Kubernetes, QuestDB offers a -[Container Storage Interface](https://github.com/container-storage-interface/spec/blob/master/spec.md) -(CSI) Driver to create ZFS volumes in your cluster. +To also delete partitions after a retention window, add a `DROP LOCAL` stage, +for example `STORAGE POLICY(TO PARQUET 3d, DROP LOCAL 1M)`. -Please contact us for more information to see if your version and distribution -of Kubernetes is supported. +Check active policies via the `storage_policies` system view, and track +conversion with `SHOW PARTITIONS`, where converted partitions have `isParquet` +set to `true` and report their size in `parquetFileSize`: -For more on storage and compression, see [Enable compression with ZFS](/docs/deployment/compression-zfs/). +```questdb-sql title="Web Console - Check policies and conversion progress" +SELECT * FROM storage_policies; +SHOW PARTITIONS FROM trades; +``` -## 9. Configure a storage policy +### Tier partitions to object storage -[Storage policies](/docs/concepts/storage-policy/) are the Enterprise primitive -for automated partition retention. A policy converts older partitions to -Parquet, drops the native binary files, and eventually drops the Parquet files -too — on a schedule you define. This supersedes plain TTL in Enterprise, where -`ALTER TABLE SET TTL` with a non-zero value is rejected. +To keep historical partitions queryable without keeping them on local disk, add +the `TO REMOTE` stage. It uploads a Parquet copy to S3, Google Cloud Storage, +Azure Blob Storage, or a filesystem store, and `DROP LOCAL` then evicts the +local copy while queries keep working: + +```questdb-sql title="Web Console - Tier partitions to object storage" +ALTER TABLE trades SET STORAGE POLICY( + TO PARQUET 7 DAYS, + TO REMOTE 14 DAYS, + DROP LOCAL 30 DAYS +); +``` + +Tiering saves more storage on top of Parquet compression: + +- Object storage usually costs less than the block storage behind local + volumes. +- The cluster keeps a single copy of each remote partition. The primary and + every replica read the same object instead of each holding its own copy. +- History is no longer limited by the size of a single volume. + +`DROP LOCAL` makes a partition read-only, so set it beyond the longest window +in which late rows or corrections can arrive. See +[Immutability after DROP LOCAL](/docs/concepts/cold-storage/#immutability-after-drop-local). + +Tiering requires [cold storage](/docs/concepts/cold-storage/) to be enabled and +an object store prefix to be configured first. See +[Operating cold storage](/docs/operations/cold-storage/) for the setup +procedure. + +For the full concept, including the stage model, timing, and replication +behavior, see [Storage Policy](/docs/concepts/storage-policy/) and +[ALTER TABLE SET STORAGE POLICY](/docs/query/sql/alter-table-set-storage-policy/). ### Migrating from TTL when upgrading from OSS Tables that were created in OSS keep their existing `TTL` setting after you -upgrade to Enterprise — no data is lost at upgrade time. On an existing table, +upgrade to Enterprise, so no data is lost at upgrade time. On an existing table, `ALTER TABLE SET TTL` with a non-zero value is then rejected with: ``` @@ -503,7 +563,7 @@ To move a legacy table from `TTL` to a storage policy: ALTER TABLE trades SET TTL 0; ``` -2. **Attach a storage policy** that reproduces — and ideally extends — the +2. **Attach a storage policy** that reproduces, and ideally extends, the retention the TTL used to provide. A policy lets you keep data in Parquet after you would previously have dropped it, so `DROP LOCAL` is the stage that replaces the old TTL horizon: @@ -517,61 +577,14 @@ To move a legacy table from `TTL` to a storage policy: If you want the policy to behave exactly like the old TTL (delete the partition outright after the same interval, with no Parquet conversion), - use a single-stage policy — for example `STORAGE POLICY(DROP LOCAL 1 MONTH)` + use a single-stage policy, for example `STORAGE POLICY(DROP LOCAL 1 MONTH)` to match `TTL 1 MONTH`. Do this for every table you want to keep managed automatically. Tables without a storage policy retain their data indefinitely once their legacy TTL has been cleared. -### Creating new tables with a storage policy - -Attach a policy at table creation: - -```questdb-sql title="Web Console - Create a table with a storage policy" -CREATE TABLE trades ( - ts TIMESTAMP, - symbol SYMBOL, - price DOUBLE -) TIMESTAMP(ts) PARTITION BY DAY - STORAGE POLICY(TO PARQUET 3d, DROP LOCAL 1M); -``` - -Or attach a policy to an existing table: - -```questdb-sql title="Web Console - Set a storage policy on an existing table" -ALTER TABLE trades SET STORAGE POLICY( - TO PARQUET 3 DAYS, - DROP LOCAL 1 MONTH -); -``` - -Check active policies via the `storage_policies` system view: - -```questdb-sql -SELECT * FROM storage_policies; -``` - -For the full concept, including the stage model, timing, and replication -behavior, see -[Storage Policy](/docs/concepts/storage-policy/) and -[ALTER TABLE SET STORAGE POLICY](/docs/query/sql/alter-table-set-storage-policy/). - -### Tiering partitions to object storage - -The policies above keep everything on local disk. To keep historical partitions queryable without keeping them on local disk, add the `TO REMOTE` stage. It uploads a Parquet copy to S3, Google Cloud Storage, Azure Blob Storage, or a filesystem store, and `DROP LOCAL` then evicts the local copy while queries keep working: - -```questdb-sql title="Web Console - Tier partitions to object storage" -ALTER TABLE trades SET STORAGE POLICY( - TO PARQUET 7 DAYS, - TO REMOTE 14 DAYS, - DROP LOCAL 30 DAYS -); -``` - -This requires [cold storage](/docs/concepts/cold-storage/) to be enabled and an object store prefix to be configured first. See [Operating cold storage](/docs/operations/cold-storage/) for the setup procedure. - -## 10. Double-check kernel limits +## 9. Double-check kernel limits QuestDB works together with your server operating system to achieve maximum performance. Prior to putting your server under heavy loads, consider checking @@ -660,11 +673,15 @@ correctly configured on both the primary and replica nodes. Check the logs for any errors related to replication and ensure there's network connectivity between all involved parties. -**Q: Compression is enabled, but I'm not observing reduced storage usage. What -could be the issue?** - -A: Ensure that the ZFS filesystem is correctly configured with compression -enabled. Note that the actual compression ratio achieved can vary significantly -depending on the nature of your data. Some types of data may not compress well. -Review the ZFS compression statistics to understand the compression level being -achieved. If it seems out of expected range, please contact us. +**Q: I set a storage policy with `TO PARQUET`, but I'm not observing reduced +storage usage. What could be the issue?** + +A: A partition converts only once its entire time range is older than the +`TO PARQUET` delay, and the active partition never converts. Policies are +checked every 5 minutes by default, so allow time for the first run. Use +`SHOW PARTITIONS FROM tableName` to confirm which partitions have `isParquet` +set to `true` and compare `parquetFileSize` with `diskSize`. The compression +ratio varies with the nature of your data, and some data does not compress +well. To trade CPU for a higher ratio, switch the codec to `ZSTD`, as described +in [Parquet data compression](/docs/concepts/parquet/#data-compression). If the +ratio still seems out of the expected range, please contact us. diff --git a/src/components/DRY/_questdb_file_system_choice.mdx b/src/components/DRY/_questdb_file_system_choice.mdx deleted file mode 100644 index 90a8d5740a..0000000000 --- a/src/components/DRY/_questdb_file_system_choice.mdx +++ /dev/null @@ -1,2 +0,0 @@ -For the file system, use `zfs` with `lz4`, as this will compress your data. If compression -is not a concern, then use `ext4` or `xfs` for a little higher performance.