Repository navigation
Azure Blob and DOI backends, and the name encoder of rclone - #37
Merged
Merged
Conversation
azureblob reads and writes the blobs of Azure Blob Storage and of Azurite, its emulator, as the azureblob backend of rclone: - Shared Key with the account and its key, a SAS URL, a connection string, or the emulator. The options of Azure AD give 501. - A file of chunk_size or less goes in one Put Blob. A larger file, or a file of unknown size, goes in blocks with the IDs of rclone. - Folders exist when they have a file. With directory_markers, a PUT puts the markers of rclone (hdi_isfolder=true). - The time of a file is in the metadata mtime, as rclone. COPY is a Copy Blob, and PATCH is a Set Blob Metadata. - Ranges as RFC 9110, also a suffix range and a start at the end of the file: Azurite gives 206 for them. - The names have the encoding of the option encoding, with the default of rclone: Slash, BackSlash, Del, Ctl, RightPeriod and InvalidUtf8. - The key, the SAS and the connection string are not in the links of a list. doi reads the files of a dataset with a DOI, as the doi backend of rclone: Zenodo, InvenioRDM and Dataverse. The provider comes from the host, from the landing page or from the option provider. The backend is read only. lib/encoder.js is the encoder of the names of rclone 1.75.1: the masks, the option encoding, and the standard names and paths. The full table of the tests of rclone (10457 cases) passes. The test file keeps the first case of each mask, and a fast-check test of the round trip. lib/parts.js has Parts of s3, for the uploads in parts of s3 and azureblob. lib/xml.js gives the attributes of an element. The shared suite runs on Azurite with three configurations, and the tests compare rfetch with rclone on the same Azurite: the files, the times and the encoded names in the two directions. CI installs Azurite 3.37.0.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add the azureblob backend for Azure Blob Storage and Azurite:
Sign-in with Shared Key, a SAS URL, a connection string, or the emulator.
Upload in blocks, folder markers, the time in the metadata mtime, COPY with Copy Blob, and ranges.
The key, the SAS and the connection string are not in the links of a list.
Add the doi backend (read only): the files of a dataset with a DOI, on Zenodo, InvenioRDM and Dataverse.
Add lib/encoder.js, the name encoder of rclone 1.75.1. All 10457 cases of the tests of rclone pass.
Tests: the shared suite on Azurite, and a comparison with rclone on the same Azurite. CI installs Azurite 3.37.0.