[paulmueller-agent]

Problem

Windows CI on main (run #773) failed two uploads in test_upload_job.py (test_collection, test_full_upload) with:

S3UploadError: Could not upload or server did not return 'ETag' for upload of <FilePart …>[0] using https://objectstore.hpccloud.mpcdf.mpg.de/circle-dev-… (3 retries), got None and …
ConnectionRefusedError: [WinError 10061] No connection could be made because the target machine actively refused it

macOS and Ubuntu passed all 275 tests in the same run. The object store refused connections from the Windows runner for a short time (10:08–10:15); other uploads in the same job worked. The same kind of failure appeared on Windows in the #117 run.

requests_put_data_and_get_etag() retried failed part uploads 3 times immediately, so a refusal lasting a few seconds became a failed upload. The same applied to the request that completes a multipart upload. Users on flaky connections are affected in the same way.

Changes (dcoraid/api/s3_api.py)

• New helper wait_before_retry(attempt, attempts, retry_wait). It waits retry_wait * 2**attempt seconds, i.e. 5 s then 10 s with the defaults, and never after the last attempt.
• requests_put_data_and_get_etag() gets a new parameter retry_wait (default RETRY_WAIT = 5.0). It waits before retrying after an exception and after an ETag mismatch.
• The multipart completion loop in upload_s3_presigned_multipart() waits as well, after exceptions, ETag mismatches and error responses.
• The number of attempts (retries=3) is unchanged. Successful uploads are not delayed.

Tests

• New offline tests/test_api_s3_retry.py, with a mocked requests.put and time.sleep:
  • the waiting sequence (5, 10, 20, nothing after the last attempt)
  • two refused connections followed by success (waits 5 s and 10 s)
  • an ETag mismatch followed by success
  • all attempts failing raises S3UploadError with the original error
  • no waiting on success
• test_api_s3.py and test_upload_job.py pass locally against dcor-dev (20 passed).

Whether the object store limits connections from GitHub's Windows runners would be a question for MPCDF. This PR only makes DCOR-Aid tolerate short interruptions.

🤖 Generated with Claude Code