rfetch

Fetching between remotes, using Web APIs.

This is a rewrite of Rclone, using modern Web APIs for cross runtimes.

A Rclone wrapper for WebAssembly build is also available.

Storage providers

• Azure Blob Storage, and Azurite, its emulator 📄
• DOI: read the files of a dataset with a DOI, on Zenodo, InvenioRDM and Dataverse 📄
• FTP: the files of an FTP server, also with TLS (FTPS), in Node.js, Deno, Bun and Cloudflare Workers 📄
• Google Drive 📄
• HTTP 📄
• The local filesystem 📄
• Memory 📄
• S3: Amazon S3 and the services with the same API, for example Cloudflare R2 📄
• SFTP: the files of an SSH server, with a client of SSH in JavaScript that runs in Node.js, Deno, Bun and Cloudflare Workers 📄
• WebDAV: Nextcloud, ownCloud, Fastmail Files, rclone serve webdav and other WebDAV servers 📄

Virtual storage providers

These backends adapt or modify other storage providers

• Alias: rename existing remotes 📄
• Archive: read the files in zip files (also docx, xlsx, epub, jar and kmz), in 7z files, and in RAR files in store mode 📄
• Chunker 📄
• Combine: several remotes in one tree of folders 📄
• Compress: compress files with gzip 📄
• Crypt: encrypt files 📄
• Tiles: read the map tiles and the metadata of PMTiles and VersaTiles files, as folders Z/X/Y.mvt. rclone has no such backend.
• Union: the merged contents of several remotes 📄

rfetch as a service

_worker.js serves rfetch as a Worker of Cloudflare, at rfetch.pages.dev. It is stateless: it keeps no configuration and no secret. The path of a request is a remote and a path, as for the command, and the options come in the headers RFETCH_*:

curl -H "RFETCH_CONFIG_R2_TYPE: s3" \
  -H "RFETCH_CONFIG_R2_ENDPOINT: https://ACCOUNT_ID.r2.cloudflarestorage.com" \
  -H "RFETCH_CONFIG_R2_ACCESS_KEY_ID: …" \
  -H "RFETCH_CONFIG_R2_SECRET_ACCESS_KEY: …" \
  -H "Accept: application/json" \
  https://rfetch.pages.dev/r2:bucket/folder/

With Accept: application/json, a GET gives the list of rfetch lsjson, and Depth: N lists N levels. Else a folder is an HTML page, and a file is its bytes, with ranges. The prefix RCLONE_ of rclone also works. The fields of the query are options too. A header or a field of the query with an empty value is not an option: the default of the option stays.

The Worker is also a WebDAV server, as rfetch serve webdav: a WebDAV client can use the URL of a remote, for example https://rfetch.pages.dev/r2:bucket/, with the options in its headers. COPY and MOVE can go from a remote to another remote.

Commands

The commands give the output of rclone, and its exit codes: 3 for a folder that does not exist, 4 for a file that does not exist, and 1 for another error.

• backend
• cat
• check, checksum and cryptcheck: compare the files of two places, or the files and a sum file. The backends give no hash: check compares the sizes, and with --download the bytes.
• config
• copy and copyto: copy the files that the target does not have, or that differ by size or by time. A copy gets the time of its source, when the backend can keep it. In the same remote, the backend copies each file, and the bytes do not go through rfetch.
• copyurl
• cryptdecode
• delete, deletefile, purge, rmdir and rmdirs
• hashsum, md5sum and sha1sum: MD5, SHA-1, SHA-256, SHA-512 and CRC-32. rfetch reads each file to compute its hash.
• listremotes
• ls
• lsd
• lsf
• lsjson
• lsl
• mkdir
• move and moveto: in the same remote, the backend moves each file, or the folder when the target does not exist. Else copy, then delete each source file.
• obscure
• rcat
• reveal
• serve http: serve a remote over HTTP, as rclone serve http. A client only reads.
• serve webdav: serve a remote over WebDAV, as rclone serve webdav. A client reads and writes, unless --read-only. With --user and --pass, a client must log in with them.
• size
• touch: make an empty file, or set the time of a file.
• tree
• version

Runtimes

rfetch runs on Node.js 22.2 or later, on Deno and on Bun 1.4 or later:

node main.js COMMAND ...
deno run -A main.js COMMAND ...
bun main.js COMMAND ...

Bun 1.3 cannot read the list of a local folder: its HTMLRewriter gives the error Failed to pipe stream.

node bench/rclone.js RUNTIME... runs the same commands with rfetch on each runtime and with rclone, and first makes sure that they give the same data and the same lists. Each time is the median of 5 runs of the command as a new process, in ms, thus the start of the runtime is in it. The table is one run on a Linux machine with 4 CPUs (Intel Xeon at 2.10 GHz) and rclone 1.75.1. The times on one machine can change by 10 to 25 % from one run to the next, also for rclone. Run the benchmark on your machine to choose a runtime.

Command, in ms	Node.js 22.22	Node.js 24.21	Node.js 26.12	Deno 2.9.6	Bun 1.4.3	rclone 1.75.1
lsjson of an empty folder	135	130	142	108	48	57
local: cat of 64 MiB	166	187	176	155	90	90
local: lsjson --recursive, 1000 files	267	249	226	212	73	50
crypt: cat of 64 MiB	383	395	416	411	402	228
crypt: rcat of 64 MiB	469	449	429	535	461	443
crypt: lsjson --recursive, 1000 files	387	347	346	347	160	113
chunker: cat of 64 MiB, chunks of 8 MiB	235	209	220	188	113	81
chunker: rcat of 64 MiB, chunks of 8 MiB	377	364	363	474	285	380

• lsjson of an empty folder shows the time of the start.
• crypt and chunker are on top of local files. rclone writes the files that rfetch reads.
• Deno gives the time of a local file to the millisecond. The other runtimes and rclone give it to the nanosecond.
• In Bun and in a Cloudflare Worker, HTMLRewriter is a part of the runtime. In Node.js and in Deno, it is WebAssembly, which loads when a command reads the first list.

Development

• Clone the repository and go to its folder.
• npm install to get the dependencies (Node.js 22.2 or later).
• npm start -- COMMAND ... to run a command.
• Alternatively, npm link to install a rfetch program in the PATH, with a symbolic link to main.js, and use rfetch in place of npm start --.
• npm test to run the tests, with the test runner of Node.js.
• npm run bench to compare some commands with rclone, which must be in the PATH.
• node bench/node.js to measure the time of the backends, and node bench/rclone.js to compare the commands with rclone. See CONTRIBUTING.md for the benchmark in a Worker.