Skip to content

Commit 34d36b1

Browse files
committed
Merge remote-tracking branch 'origin/develop' into ct/row-fn
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com> # Conflicts: # vortex-array/src/scalar_fn/fns/binary/numeric/checked.rs # vortex-array/src/scalar_fn/fns/binary/numeric/primitive.rs # vortex-array/src/scalar_fn/fns/binary/numeric/tests.rs # vortex-compute/src/lane_kernels/map_into.rs
2 parents dfebd42 + a1057db commit 34d36b1

148 files changed

Lines changed: 6689 additions & 4987 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

Cargo.lock

Lines changed: 139 additions & 4 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

docs/api/python/store/opendal.rst

Lines changed: 62 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,10 @@
1-
==================
2-
OpenDAL (COS, OSS)
3-
==================
1+
===========================
2+
OpenDAL (COS, OSS, GooseFS)
3+
===========================
44

5-
Vortex can read from and write to Tencent Cloud COS and Alibaba Cloud OSS through
6-
`OpenDAL <https://opendal.apache.org/>`_, which provides native service support.
5+
Vortex can read from and write to Tencent Cloud COS, Alibaba Cloud OSS, and Tencent Cloud
6+
GooseFS through `OpenDAL <https://opendal.apache.org/>`_, which provides native service
7+
support.
78

89
These stores are available only when Vortex is built with the ``opendal`` feature
910
(e.g. ``maturin develop --features opendal`` or ``cargo build -p vortex-jni --features opendal``).
@@ -13,7 +14,7 @@ These stores are available only when Vortex is built with the ``opendal`` featur
1314

1415
* - Scheme
1516
- Service
16-
- Endpoint variable
17+
- Endpoint / master variable
1718
- Credential variables
1819
* - ``cos://``
1920
- Tencent Cloud COS
@@ -23,6 +24,10 @@ These stores are available only when Vortex is built with the ``opendal`` featur
2324
- Alibaba Cloud OSS
2425
- ``OSS_ENDPOINT``
2526
- ``ALIBABA_CLOUD_ACCESS_KEY_ID``, ``ALIBABA_CLOUD_ACCESS_KEY_SECRET``
27+
* - ``goosefs://``
28+
- Tencent Cloud GooseFS
29+
- ``GOOSEFS_MASTER_ADDR``
30+
- (optional) ``auth_type`` / ``auth_username`` properties
2631

2732
:class:`vortex.store.CosStore`
2833
==============================
@@ -47,6 +52,31 @@ These stores are available only when Vortex is built with the ``opendal`` featur
4752
:param disable_config_load: When ``True``, disable OpenDAL's automatic config loading
4853
and rely only on the explicit configuration. Defaults to ``False``.
4954

55+
:class:`vortex.store.GoosefsStore`
56+
==================================
57+
58+
.. py:class:: vortex.store.GoosefsStore(master_addr, *, root=None, block_size=None, chunk_size=None, write_type=None, auth_type=None, auth_username=None)
59+
60+
A Tencent Cloud GooseFS object store, backed by OpenDAL. Construct it with explicit
61+
configuration and pass it to
62+
:func:`vortex.io.read_url` / :func:`vortex.io.write` via the ``store=`` argument,
63+
exactly like the built-in S3/Azure/GCS stores.
64+
65+
The class is only available when Vortex is built with the ``opendal`` feature; on
66+
a default build, instantiating it raises :class:`ImportError`.
67+
68+
:param master_addr: GooseFS master address(es). Single master:
69+
``"10.0.0.1:9200"``. HA (comma-separated):
70+
``"10.0.0.1:9200,10.0.0.2:9200,10.0.0.3:9200"``.
71+
:param root: Optional key prefix applied to every operation.
72+
:param block_size: Block size in bytes for new files (default: 64 MiB).
73+
:param chunk_size: Chunk size in bytes for streaming RPCs (default: 1 MiB).
74+
:param write_type: Default write type: ``"must_cache"``, ``"cache_through"``,
75+
``"through"``, or ``"async_through"``.
76+
:param auth_type: Authentication type: ``"nosasl"`` or ``"simple"`` (default:
77+
``"simple"``).
78+
:param auth_username: Authentication username (default: current OS user).
79+
5080
Reading from COS
5181
================
5282

@@ -91,3 +121,29 @@ configuration comes from the environment variables OpenDAL's OSS builder reads
91121
import vortex as vx
92122
93123
a = vx.io.read_url("oss://my-bucket/path/to/dataset.vortex")
124+
125+
Reading from GooseFS
126+
====================
127+
128+
Pass a ``goosefs://`` URL directly. The master address is taken from the URL authority, or
129+
from the ``GOOSEFS_MASTER_ADDR`` environment variable when the authority is empty:
130+
131+
.. code-block:: python
132+
133+
import vortex as vx
134+
135+
a = vx.io.read_url("goosefs://10.0.0.1:9200/path/to/dataset.vortex")
136+
137+
Or configure explicitly with :class:`~vortex.store.GoosefsStore` and pass it to
138+
:func:`vortex.io.read_url` via ``store=``:
139+
140+
.. code-block:: python
141+
142+
from vortex.io import read_url
143+
from vortex.store import GoosefsStore
144+
145+
store = GoosefsStore(master_addr="10.0.0.1:9200")
146+
147+
# When `store=` is supplied, the path is a key within the store, so the scheme and
148+
# master address are not part of the path passed to read_url.
149+
a = read_url("path/to/dataset.vortex", store=store)

docs/conf.py

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -52,10 +52,11 @@
5252

5353
nitpicky = True # ensures all :class:, :obj:, etc. links are valid
5454
nitpick_ignore = [
55-
# `vortex.store.CosStore` is re-exported through the private `vortex.store._cos` module,
56-
# and the `ObjectStore` type alias resolves to the private path. The public class is
57-
# fully documented in `opendal.rst`; the private path is intentionally not.
55+
# `vortex.store.CosStore` / `GoosefsStore` are re-exported through private modules,
56+
# and the `ObjectStore` type alias resolves to those private paths. The public
57+
# classes are fully documented in `opendal.rst`; the private paths are intentionally not.
5858
("py:class", "vortex.store._cos.CosStore"),
59+
("py:class", "vortex.store._goosefs.GoosefsStore"),
5960
]
6061

6162
doctest_global_setup = "import pyarrow; import vortex; import vortex as vx; import random; random.seed(a=0)"

encodings/fastlanes/Cargo.toml

Lines changed: 0 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -58,11 +58,6 @@ name = "compute_between"
5858
harness = false
5959
required-features = ["_test-harness"]
6060

61-
[[bench]]
62-
name = "bit_transpose"
63-
harness = false
64-
required-features = ["_test-harness"]
65-
6661
[[bench]]
6762
name = "bitpack_compare"
6863
harness = false

0 commit comments

Comments
 (0)