Repository navigation
Conversation
The ftp backend reads and writes the files of an FTP server, as the ftp backend of rclone, also with TLS from the start (implicit FTPS) or after AUTH TLS (explicit FTPS). The client of FTP (backend/ftp): - The passive mode with EPSV, then PASV. The data connections go to the host of the control connection, not to the address in the answer of PASV: a server cannot send the client to another host. - The lists of MLSD and MLST, with the times to the second, and LIST of Unix and of MS-DOS, with no time. MDTM gives the time of a file, and MFMT sets it, or MDTM with writing_mdtm, as vsftpd. - A range is REST and RETR. A download that stops before the end closes its connection. - An upload ends the writes of the data connection, then reads the answer of the server, then closes the data connection. A close before the answer, with data of TLS 1.3 that the client did not read, can reset the connection: then the server loses the last bytes. - In Node.js, Deno and Bun, the connections are of node:net and node:tls. In workerd, they are of cloudflare:sockets, because TLS cannot start on a connection of node:net there. The backend: - The options of rclone. A path is from the login folder, and a path that starts with "/" is from the root of the server, with RFETCH_FTP_ROOT of fetch.js. - MOVE of a file or of a folder is a rename. PATCH sets the time on a server with MFMT. COPY gives 501. - After a PUT, the backend checks the size of the file on the server, as rclone. no_check_upload stops this check. - The names have the encoding of rclone. A CR or an LF in a name always changes, as in rclone: a name cannot add a command of FTP. - A connection does one transfer at a time. In Node.js, Deno and Bun, it waits for the next request for idle_timeout. concurrency limits the connections. In workerd, each request has its own connections. - socks_proxy, http_proxy and ask_password give 501. In workerd, no_check_certificate and disable_tls13 give 501. Other changes: - lib/time.js has parseDuration, for sftp and ftp. - sftp: idle_timeout 0 is no limit, as in rclone. - The shared tests accept 501 and 502 for PATCH, as CONTRIBUTING.md says. - The workspace test accepts cloudflare:sockets in backend/ftp. - CI installs pyftpdlib and pyOpenSSL, with fixed versions. The tests run the shared suite on rclone serve ftp, and on pyftpdlib with no TLS, with explicit TLS and with implicit TLS. They check the certificate, LIST in place of MLSD, a download that stops, and a name with CR and LF. They compare rfetch with the ftp backend of rclone on the same servers: the files, the times and the encoded names. rclone serve ftp has no MFMT, and its implicit TLS refuses PBSZ and PROT, thus the ftp backend of rclone cannot use it. In workerd, with wrangler pages dev, the backend reads and writes files with no TLS, with explicit TLS and with implicit TLS, with the check of the certificate. On this machine, a copy of 64 MB to rclone serve ftp takes 0.35 s up and 0.26 s down; rclone takes 0.40 s and 0.29 s. With TLS on pyftpdlib, rfetch takes about 0.3 s each way. The ftp backend of rclone loses the last bytes of this upload with explicit TLS on pyftpdlib.
From Python 3.12, pyftpdlib needs pyasynchat and pyasyncore, because Python has no asynchat and no asyncore. CI installed the packages with --no-deps and with no such package, thus the servers did not start, and the tests of the ftp backend failed with "No server on the port". - CI installs pyasynchat 1.0.5 and pyasyncore 1.0.5, runs pip check, and imports the handlers and the servers of pyftpdlib. - hasPyftpdlib imports the handlers and the servers: a Python with no asynchat skips the tests with a message, and CI sees this skip. - The server of pyftpdlib takes a free port itself and writes it: no other process can take the port before the server. A server that stops gives an error with its log.
The ftp backend uses only node:net and node:tls now, also in workerd.
It does not use cloudflare:sockets.
In workerd, tls.connect({ socket }) on a socket of node:net throws
"Cannot call releaseLock() on a reader with outstanding read promises"
when a read waits on the socket, and a socket of node:net waits for data
all the time. No compatibility date or flag changes this. Thus in
workerd, the control connection of explicit FTPS reads with the option
onread of net.connect, and its callback stops the reads after each
chunk, as Node.js does when the callback gives false. A read that waits
starts them again. Then no read waits when TLS starts.
Deno 2.9.6 loses the end of a socket of node:tls when the server closes
the connection while the socket is paused: a download of more than
1 MiB with TLS did not end. Now a download ends at the end of the data
connection or at the answer 226 of the server, which comes after the
last byte. The data connection of a list does not pause: the client
reads all the list at once. Deno does not start TLS on a socket that
reads with onread, thus only workerd uses onread.
A new test downloads a file of 4 MiB with implicit and explicit TLS,
with a slow reader.
Checked: the tests in Node.js 22; the backend with no TLS, explicit TLS
and implicit TLS in Node.js 22, 24 and 26, Deno 2.9.6, Bun 1.4.0 and
1.4.3, with files of 1 KB to 32 MiB; and _worker.js in wrangler pages
dev (workerd 1.20260926.1), with the check of the certificate.
CONTRIBUTING.md has a new section with the problems that the tests of
rfetch found in other tools, and the workaround of rfetch for each:
- The ftp backend of rclone 1.75.1 loses the last bytes of an upload
with explicit TLS 1.3 on pyftpdlib, for files of 1 MiB to 64 MiB.
With disable_tls13=true, the upload is correct.
- rclone serve ftp with --cert answers 550 to PBSZ and to PROT.
- tls.connect({ socket }) of workerd throws when a read waits on the
socket.
- Deno 2.9.6 loses the end of a socket of node:tls that ends while it
is paused.
The section on continuous integration names pyftpdlib.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add the ftp backend, as in rclone, with implicit TLS and explicit TLS (AUTH TLS). It uses only node:net and node:tls, and runs in Node.js, Deno, Bun and Cloudflare Workers.
Passive mode with EPSV and PASV. The data connections go only to the host of the control connection.
Lists with MLSD and LIST. MDTM and MFMT for the time of a file. Ranges.
After an upload, the backend checks the size of the file on the server.
Security: a CR or an LF in a name is always encoded, so a name cannot add an FTP command.
Workarounds for bugs in other tools: tls.connect({ socket }) in workerd, and the lost end of a TLS socket in Deno. The new section "Problems in other tools" in CONTRIBUTING.md describes these, and two problems in rclone.
Other changes:
sftp: idle_timeout=0 means no limit, as in rclone.
The shared tests accept 501 for a PATCH with no time support, as the contract says.
parseDuration moves into lib/time.js.
Tests: the shared suite on rclone serve ftp, and on pyftpdlib with no TLS, explicit TLS and implicit TLS. A comparison with rclone on the same servers. CI installs pyftpdlib.