FTP backend, with implicit and explicit TLS

.github/workflows/ci.yml

@@ -2,8 +2,8 @@
 #
 # - npm test: the tests, with the oldest Node.js of package.json and the
 #   newer versions. The job installs rclone, 7-Zip, Go, go-pmtiles,
-#   versatiles and Azurite, thus the tests that compare rfetch with these
-#   tools, or that run on them, run too.
+#   versatiles, Azurite and pyftpdlib, thus the tests that compare rfetch
+#   with these tools, or that run on them, run too.
 # - deno fmt --check: the format of the files.
 # - wasm: the tests of the crate of backend/crypt/secretbox.wasm, and its
 #   build from its source gives the same bytes as the file in the repository.
@@ -119,16 +119,30 @@ jobs:
           "$RUNNER_TEMP/azurite/node_modules/.bin/azurite-blob" --version
           echo "AZURITE=$RUNNER_TEMP/azurite/node_modules/.bin/azurite-blob" >> "$GITHUB_ENV"
 
+      # pyftpdlib is a server of FTP, with explicit and implicit TLS through
+      # pyOpenSSL: the tests of the ftp backend run on it. The versions of
+      # all the packages are fixed.
+      - name: Install pyftpdlib
+        run: |
+          python3 -m venv "$RUNNER_TEMP/pyftpdlib"
+          "$RUNNER_TEMP/pyftpdlib/bin/pip" install --quiet --no-deps \
+            pyftpdlib==2.2.0 pyasynchat==1.0.5 pyasyncore==1.0.5 \
+            pyOpenSSL==26.4.0 cryptography==50.0.2 cffi==2.1.1 \
+            pycparser==3.11 typing_extensions==4.16.0
+          "$RUNNER_TEMP/pyftpdlib/bin/pip" check
+          "$RUNNER_TEMP/pyftpdlib/bin/python" -I -c "from pyftpdlib.handlers import TLS_FTPHandler; import pyftpdlib.servers, pyftpdlib, OpenSSL; print(pyftpdlib.__ver__, OpenSSL.__version__)"
+          echo "PYFTPDLIB_PYTHON=$RUNNER_TEMP/pyftpdlib/bin/python" >> "$GITHUB_ENV"
+
       - run: npm ci
 
-      # A test against rclone, 7-Zip, Go, go-pmtiles, versatiles or Azurite
-      # skips when the tool is not installed.
+      # A test against rclone, 7-Zip, Go, go-pmtiles, versatiles, Azurite or
+      # pyftpdlib skips when the tool is not installed.
       # Here the tools are installed, thus such a skip is an error.
       - name: npm test
         shell: bash
         run: |
           npm test 2>&1 | tee "$RUNNER_TEMP/test.log"
-          if grep -E "((rclone|7-Zip|Go|go-pmtiles|versatiles|Azurite|openssl) is not installed|node:sqlite is not in)" "$RUNNER_TEMP/test.log"; then
+          if grep -E "((rclone|7-Zip|Go|go-pmtiles|versatiles|Azurite|openssl|pyftpdlib) is not installed|pyftpdlib, pyOpenSSL or openssl is not installed|node:sqlite is not in)" "$RUNNER_TEMP/test.log"; then
             echo "::error::A test skipped because a tool is not installed."
             exit 1
           fi

CONTRIBUTING.md

@@ -279,9 +279,9 @@ A backend that uploads a large file in parts, for example s3 and azureblob,
 reads the body with `Parts` of `@rfetch/lib/parts`: each part has the size of
 the option `chunk_size`, and only one part is in memory.
 
-The path of a remote is from its root. For sftp, as in rclone, the root is the
-login folder of the user, and a path that starts with `/` is from the root of
-the server: `fetch.js` gives the same pathname for `sftp:a.txt` and for
+The path of a remote is from its root. For sftp and ftp, as in rclone, the root
+is the login folder of the user, and a path that starts with `/` is from the
+root of the server: `fetch.js` gives the same pathname for `sftp:a.txt` and for
 `sftp:/a.txt`, thus it tells a backend with `RFETCH_TYPE_ROOT` of `env`, for
 example `RFETCH_SFTP_ROOT`, which is `/` for a path that starts with `/`. A
 config file, a variable, a header or the query cannot set it.
@@ -479,6 +479,18 @@ They also compare rfetch with the sftp backend of rclone on the same server, for
 example with keys in the format of OpenSSH that the tests write and rclone
 reads. The tests in `backend/sftp/ssh/ssh_test.js` need no server.
 
+The tests of `ftp` use `rclone serve ftp`, with no TLS, and pyftpdlib, with no
+TLS, with explicit TLS and with implicit TLS, and with MFMT, which sets the time
+of a file. The certificates come from `openssl` for each run. The tests on
+pyftpdlib run when the Python of the variable `PYFTPDLIB_PYTHON`, or `python3`,
+has pyftpdlib and pyOpenSSL: get them with
+`pip install pyftpdlib==2.2.0 pyOpenSSL==26.4.0`, for example in a venv. From
+Python 3.12, pip also installs pyasynchat and pyasyncore for pyftpdlib. The
+tests also compare rfetch with the ftp backend of rclone on the same servers.
+`rclone serve ftp` has no MFMT, and its implicit TLS refuses PBSZ and PROT: the
+ftp backend of rclone and of rfetch cannot use it. The tests in
+`backend/ftp/list_test.js` need no server.
+
 The tests of `azureblob` use Azurite, the emulator of Azure Storage, with its
 data in memory: they run when `azurite-blob` is in the `PATH`, or when the
 variable `AZURITE` is its path. Get it with `npm install azurite@3.37.0`. The
@@ -537,8 +549,43 @@ message of the commit.
 ## Continuous integration
 
 The workflow `.github/workflows/ci.yml` runs for each push and each pull
-request. It installs rclone, 7-Zip, Go, go-pmtiles, versatiles and Azurite (the
-runner has openssl), runs `npm test` with Node.js 22, 24 and 26, and runs
-`deno fmt --check`. A test that needs one of these tools skips when the tool is
-not installed: in the workflow, such a skip is an error. Before a push, run
-`npm test` and `deno fmt --check`.
+request. It installs rclone, 7-Zip, Go, go-pmtiles, versatiles, Azurite and
+pyftpdlib (the runner has openssl), runs `npm test` with Node.js 22, 24 and 26,
+and runs `deno fmt --check`. A test that needs one of these tools skips when the
+tool is not installed: in the workflow, such a skip is an error. Before a push,
+run `npm test` and `deno fmt --check`.
+
+## Problems in other tools
+
+These are problems that the tests of rfetch found in other tools. rfetch has a
+workaround for each of them.
+
+- rclone 1.75.1: the ftp backend loses the last bytes of an upload with explicit
+  TLS 1.3. With pyftpdlib 2.2.0, `rclone copyto` of a file of 1 MiB, 8 MiB or 64
+  MiB gives `corrupted on transfer: sizes differ`. With `disable_tls13=true`,
+  the upload is correct. The cause is probably the close of the data connection
+  while data of TLS 1.3 that the client did not read waits: then the connection
+  resets, and the server loses the bytes that it did not read. rfetch had the
+  same problem. Now it ends the writes, reads the answer 226, and then closes
+  the data connection.
+- rclone 1.75.1: `rclone serve ftp` with `--cert` and `--key` is implicit TLS,
+  and it answers 550 to PBSZ and to PROT. The ftp backend of rclone stops at
+  PBSZ, with `550 Action not taken`. rfetch also stops: with no `PROT P`, the
+  data connections are not encrypted.
+- workerd 1.20260926.1 to 1.20261019.1: `tls.connect({ socket })` on a socket of
+  `node:net` throws
+  `Cannot call releaseLock() on a reader with outstanding
+  read promises` when
+  a read waits on the socket. A socket waits for data all the time after it
+  starts to read. No compatibility date or flag changes this: the internal
+  streams of workerd always throw in `releaseLock()` when a read waits. In
+  workerd, the ftp backend reads the control connection of explicit FTPS with
+  the option `onread` of `net.connect`, and its callback stops the reads after
+  each chunk (`backend/ftp/socket.js`).
+- Deno 2.9.6: a socket of `node:tls` loses its end when the server closes the
+  connection while the socket is paused for some time, for example 200 ms: the
+  events `end` and `close` do not come. Node.js and Bun are correct. The ftp
+  backend ends a download at the answer 226 of the server, and does not pause
+  the data connection of a list. In Deno, `tls.connect({ socket })` also does
+  not start TLS on a socket that reads with `onread`: thus only workerd uses
+  `onread`.

README.md

@@ -12,6 +12,8 @@ A [Rclone wrapper for WebAssembly build](./rclone.js) is also available.
   [:page_facing_up:](https://rclone.org/azureblob/)
 - DOI: read the files of a dataset with a DOI, on Zenodo, InvenioRDM and
   Dataverse [:page_facing_up:](https://rclone.org/doi/)
+- FTP: the files of an FTP server, also with TLS (FTPS), in Node.js, Deno, Bun
+  and Cloudflare Workers [:page_facing_up:](https://rclone.org/ftp/)
 - Google Drive [:page_facing_up:](https://rclone.org/drive/)
 - HTTP [:page_facing_up:](https://rclone.org/http/)
 - The local filesystem [:page_facing_up:](https://rclone.org/local/)