Migration to zarr 3

.github/workflows/tests.yml

@@ -16,7 +16,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        python-version: ["3.10", "3.11", "3.12"]
+        python-version: ["3.12", "3.13", "3.14"]
 
     steps:
     - uses: actions/checkout@v3

.gitignore

@@ -1,5 +1,6 @@
 testdata/
 site/
+examples/outputs/
 
 # Byte-compiled / optimized / DLL files
 __pycache__/
@@ -162,5 +163,8 @@ cython_debug/
 #  option (not recommended) you can uncomment the following to ignore the entire idea folder.
 .idea/
 
+# Visual Studio Code
+.vscode/
+
 # Downloaded assets
 docs/assets/downloaded/

CHANGES.md

@@ -1,5 +1,9 @@
-## Version 0.8.1 (in development)
+## Version 0.9.0 (in development)
 
+* Migrated to use `zarr >=3`. Supports writing Zarr v2 and v3 stores, with
+  Zarr v3 as the recommended format for new configurations. The `zarr_version`
+  setting is deprecated in favor of the now-required `zarr_format` and only
+  supports the legacy value `2`. (#111, #107)
 * Replaced dev tools black and flake8 by isort and ruff. 
 * Reformatted code and fixed linter issues.
 * Added `environment.yml` to setup a conda/mamba development environment. 

README.md

@@ -28,7 +28,10 @@ awesome Python packages [xarray](https://docs.xarray.dev/) and [zarr](https://za
 The objective of `zappend` is enabling geodata scientists and developers to 
 robustly create large data cubes. The tool performs transaction-based dataset 
 appends to existing data cubes in the 
-[Zarr format](https://zarr.readthedocs.io/en/stable/spec/v2.html). If an error 
+[Zarr format](https://zarr.readthedocs.io/en/stable/). New targets use Zarr v3
+when configured with `zarr_format: 3`; Zarr v2 remains available through
+`zarr_format: 2`. The deprecated `zarr_version: 2` setting remains accepted for
+existing Zarr v2 configurations. If an error
 occurs during an append step — typically due to I/O problems or out-of-memory 
 conditions — `zappend` will automatically roll back the operation, ensuring that 
 the existing data cube maintains its structural integrity. The design drivers 

docs/config.md

@@ -76,6 +76,11 @@ Variable metadata.
                 Its value is `null`.
           - Disable chunking in all dimensions.
             Its value is `null`.
+      - `shards`:
+        Zarr v3 shard sizes for each dimension of the variable. Each shard size
+        must be a multiple of the corresponding chunk size. Ignored with a
+        warning for `zarr_format` equal to `2`. The items of the list are positive 
+        integers.
       - `fill_value`:
         Must be one of the following:
           - Type _number_.
@@ -132,11 +137,15 @@ Must be one of the following:
 
 Defaults to `"keep"`.
 
+### `zarr_format`
+
+The Zarr format to be used. Required unless the deprecated `zarr_version: 2`
+setting is supplied. Allowed values are `2` and `3`.
+
 ### `zarr_version`
 
-The Zarr version to be used.
-Its value is `2`.
-Defaults to `2`.
+Deprecated legacy setting for Zarr v2 only. Use `zarr_format` in new
+configurations. Its only allowed value is `2`.
 
 ## Data I/O - Target
 

docs/guide.md

@@ -3,23 +3,33 @@
 After [installation](start.md), you can either use the `zappend` [CLI command](cli.md)
 
 ```shell
-zappend -t output/mycube.zarr inputs/*.nc
+zappend -t output/mycube.zarr -c config.yaml inputs/*.nc
+```
+
+where `config.yaml` contains the required output format:
+
+```yaml
+zarr_format: 3
 ```
 
 or the `zappend` [Python function](api.md)
 
 ```bash
 from zappend.api import zappend
 
-zappend(os.listdir("inputs"), target_dir="output/mycube.zarr")
+zappend(
+    os.listdir("inputs"),
+    target_dir="output/mycube.zarr",
+    zarr_format=3,
+)
 ```
 
 Both invocations will create the Zarr dataset `output/mycube.zarr` by concatenating
 the "slice" datasets provided in the `inputs` directory along their `time` dimension. 
 `target_dir` must specify a directory for the Zarr dataset. (Its parent directory must 
-exist.) Both the CLI command and the Python function can be run without any further 
-configuration provided the paths of the target dataset and the source slice datasets 
-are given. The target dataset path must point to a directory that will contain a Zarr 
+exist.) Both the CLI command and the Python function can be run without any further
+configuration beyond the required `zarr_format` setting. The target dataset path must
+point to a directory that will contain a Zarr
 group to be created and updated. The slice dataset paths may be provided as Zarr as 
 well or in other data formats supported by the [xarray.open_dataset()](https://docs.xarray.dev/en/stable/generated/xarray.open_dataset.html) 
 function. Because we provided no additional configuration, the default append dimension
@@ -31,10 +41,12 @@ one, if their paths are given as URIs prefixed with a filesystem protocol such a
 dedicated configuration settings. More on this is given
 in section [_Data I/O_](#data-io) below.
 
-!!! note "Zarr Format v2"
-    By default, `zappend` uses the [Zarr storage specification 2](https://zarr.readthedocs.io/en/stable/spec/v2.html)
-    and has only been tested with this version. The `zarr_version` setting can be used 
-    to change it, e,g, to `3`, but any other value than `2` is currently unsupported.
+!!! note "Zarr formats"
+    New configurations must set `zarr_format`; use `zarr_format: 3` for the
+    recommended [Zarr storage specification 3](https://zarr.readthedocs.io/en/stable/spec/v3.html),
+    or `zarr_format: 2` for Zarr v2 compatibility. `zarr_version: 2` is a
+    deprecated legacy setting; it cannot select Zarr v3. The target's format is fixed when it is created; use the
+    same setting for all later appends.
 
 The tool takes care of generating the target dataset from slice datasets, but doesn't 
 care how the slice datasets are created. Hence, when using the Python `zappend()` 
@@ -71,6 +83,7 @@ zappend(
     os.listdir("inputs"),
     config=["configs/base.yaml", "configs/mycube.yaml"],
     target_dir="outputs/mycube.zarr",
+    zarr_format=3,
     dry_run=True,
 )
 ```
@@ -336,6 +349,27 @@ using the `chunks` setting in the encoding of each variable.
 The value of `chunks` can also be `null`, which means no chunking is 
 desired and the variable's data array will be persisted as one block. 
 
+For Zarr v3, `shards` may additionally be configured for a variable. A shard
+contains an integer number of regular chunks, reducing file or object counts while
+retaining smaller read/write chunks. Each shard size must be at least its
+corresponding chunk size and an exact multiple of it. Shards are not available
+for Zarr v2; if configured for an unsupported format, they are ignored with a
+warning.
+
+```json
+{
+    "zarr_format": 3,
+    "variables": {
+        "chl": {
+            "encoding": {
+                "chunks": [1, 512, 512],
+                "shards": [8, 2048, 2048]
+            }
+        }
+    }
+}
+```
+
 By default, the chunking of the coordinate variable corresponding to the append 
 dimension will be its dimension size in the first slice dataset. Often, the size 
 will be `1` or another small number. Since `xarray` loads coordinates eagerly 
@@ -788,6 +822,7 @@ If you use the `zappend` function, you can pass the function or class directly:
 zappend(
     ["slice-1.nc", "slice-2.nc", "slice-3.nc"],
     target_dir="target.zarr",
+    zarr_format=3,
     slice_source=MySliceSource,
 )
 ```
@@ -876,6 +911,7 @@ class MySliceSource(SliceSource):
 zappend(
     ["slice-1.nc", "slice-2.nc", "slice-3.nc"],
     target_dir="target.zarr",
+    zarr_format=3,
     slice_source=MySliceSource,
 )
 ```

docs/howdoi.md

@@ -33,6 +33,7 @@ zappend(
     sorted(glob.glob("inputs/*.tif")),
     slice_source=get_dataset_from_geotiff,
     target_dir="output/tif-cube.zarr",
+    zarr_format=3,
 )
 ```
 

docs/index.md

@@ -11,7 +11,10 @@ awesome Python packages [xarray](https://docs.xarray.dev/) and [zarr](https://za
 The objective of `zappend` is enabling geodata scientists and developers to 
 robustly create large data cubes. The tool performs transaction-based dataset 
 appends to existing data cubes in the 
-[Zarr format](https://zarr.readthedocs.io/en/stable/spec/v2.html). If an error 
+[Zarr format](https://zarr.readthedocs.io/en/stable/). New targets use Zarr v3
+when configured with `zarr_format: 3`; Zarr v2 remains available through
+`zarr_format: 2`. The deprecated `zarr_version: 2` setting remains accepted for
+existing Zarr v2 configurations. If an error
 occurs during an append step — typically due to I/O problems or out-of-memory 
 conditions — `zappend` will automatically roll back the operation, ensuring that 
 the existing data cube maintains its structural integrity. The design drivers 

docs/start.md

@@ -58,7 +58,11 @@ Process list of local slice paths:
 ```python
 from zappend.api import zappend
 
-zappend(["slice-1.nc", "slice-2.nc", "slice-3.nc"], target_dir="target.zarr")
+zappend(
+    ["slice-1.nc", "slice-2.nc", "slice-3.nc"],
+    target_dir="target.zarr",
+    zarr_format=3,
+)
 ```
 
 Process list of slices stored in S3 [configuration](config.md) in `config`:
@@ -68,6 +72,7 @@ from zappend.api import zappend
 
 config = {
     "target_dir": "target.zarr",
+    "zarr_format": 3,
     "slice_storage_options": {
         "key": "...",
         "secret": "...",
@@ -100,6 +105,7 @@ zappend(
     ["slice-1.nc", "slice-2.nc", "slice-3.nc"],
     slice_source=get_dataset,
     target_dir="target.zarr",
+    zarr_format=3,
 )
 ```
 

environment.yml

@@ -11,8 +11,8 @@ dependencies:
   - numcodecs
   - numpy
   - pyyaml
-  - xarray <2026.4 # constrain to <2026.4 to retain compatibility with zarr 2.
-  - zarr >=2.18,<3
+  - xarray <2026.4
+  - zarr >=3,<4
   # Dev Dependencies
   - isort
   - pytest
@@ -32,3 +32,6 @@ dependencies:
   - h5netcdf
   - s3fs
   - scipy
+  # Development Dependencies - Demos
+  - jupyterlab
+  - matplotlib

examples/create_inputs.py

@@ -0,0 +1,92 @@
+#  Copyright © 2024-2026 Brockmann Consult and contributors
+#  Permissions are hereby granted under the terms of the MIT License:
+#  https://opensource.org/licenses/MIT.
+
+from pathlib import Path
+
+import numpy as np
+import xarray as xr
+
+INPUT_DIR = Path(__file__).parent / "inputs"
+START_TIME = np.datetime64("2024-01-01", "D")
+
+
+def make_slice(time: np.datetime64, index: int) -> xr.Dataset:
+    """Return one small daily slice with a predictable value pattern."""
+    lon = np.arange(-179.95, 180, 0.1, dtype="float32")
+    lat = np.arange(-89.95, 90, 0.1, dtype="float32")
+    data = np.full((1, len(lat), len(lon)), index, dtype="uint8")
+    return xr.Dataset(
+        data_vars={
+            "temperature": (
+                ("time", "lat", "lon"),
+                data.astype("float32"),
+                {"units": "degC", "long_name": "example sea temperature"},
+            ),
+            "quality_flag": (
+                ("time", "lat", "lon"),
+                data,
+                {"long_name": "example quality flag"},
+            ),
+        },
+        coords={"time": [time], "lat": lat, "lon": lon},
+        attrs={"title": "Zappend example slice"},
+    ).chunk({"time": 1, "lat": 450, "lon": 450})
+
+
+def make_zarr_v2_slices(n_slices: int) -> None:
+    """Create *n_slices* daily Zarr v2 inputs from :func:`make_slice`."""
+    times = START_TIME + np.arange(n_slices) * np.timedelta64(1, "D")
+    directory = INPUT_DIR / "zarr-v2"
+    directory.mkdir(parents=True, exist_ok=True)
+    for index, time in enumerate(times):
+        dataset = make_slice(time, index)
+        dataset.to_zarr(
+            directory / f"slice-{time!s}.zarr",
+            mode="w",
+            zarr_format=2,
+            consolidated=True,
+        )
+
+
+def make_zarr_v3_slices(n_slices: int) -> None:
+    """Create *n_slices* daily, sharded Zarr v3 inputs from :func:`make_slice`."""
+    times = START_TIME + np.arange(n_slices) * np.timedelta64(1, "D")
+    directory = INPUT_DIR / "zarr-v3"
+    directory.mkdir(parents=True, exist_ok=True)
+
+    for index, time in enumerate(times):
+        # Xarray requires Dask write chunks to align with the outer Zarr v3
+        # shard grid. The encoding below still defines 450 by 450 inner
+        # chunks inside each 1800 by 3600 shard.
+        dataset = make_slice(time, index).chunk({"time": 1, "lat": 1800, "lon": 3600})
+        encoding = {
+            name: {
+                "chunks": (1, 450, 450),
+                "shards": (1, 1800, 3600),
+            }
+            for name in dataset.data_vars
+        }
+        dataset.to_zarr(
+            directory / f"slice-{time!s}.zarr",
+            mode="w",
+            encoding=encoding,
+            zarr_format=3,
+            consolidated=False,
+        )
+
+
+def make_netcdf_slices(n_slices: int) -> None:
+    """Create *n_slices* daily NetCDF from :func:`make_slice`."""
+    times = START_TIME + np.arange(n_slices) * np.timedelta64(1, "D")
+    directory = INPUT_DIR / "netcdf"
+    directory.mkdir(parents=True, exist_ok=True)
+
+    for index, time in enumerate(times):
+        dataset = make_slice(time, index)
+        encoding = {name: {"chunksizes": (1, 450, 450)} for name in dataset.data_vars}
+        dataset.to_netcdf(
+            directory / f"slice-{time!s}.nc",
+            engine="h5netcdf",
+            encoding=encoding,
+        )