[paulmueller-agent]

Fixes invalid BagIt archives for datasets archived shortly after their upload. In the earlier full test run on dcor-dev, this made test_bag_dataset fail intermittently. bagit reported that data/calibration_beads_47_condensed.rtdc "exists on filesystem but is not in the manifest".

Cause

DCOR sets a resource's mimetype and sha256 in background jobs after the upload. The BagIt code used two different criteria for condensed files:

• archive.bag_dataset downloads the condensed file for every resource whose name ends with .rtdc.
• manifest.write_manifest only listed it if the mimetype was already RT-DC.

Before DCOR has processed the resource, the condensed file is therefore in the payload but not in the manifest, and the bag is invalid. On top of that, if sha256 was still missing, the manifest contained the line None data/<name>.

Fix

• Both places now use common.is_dc_resource_dict(), which checks the mimetype or the .rtdc/.dc suffix. DownloadJob uses the same check to decide whether a condensed version exists.
• write_manifest lists a condensed file whenever it exists in the payload, so payload and manifest cannot diverge. It only warns if a DC resource has no condensed file.
• If sha256 is not available yet, the hash is computed from the downloaded file, which DownloadJob already verified via its ETag.

Tests

• New offline test test_bag_dataset_fresh_metadata, with mocked downloads, for both cases: with and without server metadata (mimetype/sha256). The bag is validated with the bagit package. Without the fix, the case without metadata fails.
• All BagIt and dcoraid-archive-circle tests pass locally against dcor-dev (27 passed).

🤖 Generated with Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 90.90909% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 72.38%. Comparing base (5443498) to head (8d8b3ec).
⚠️ Report is 37 commits behind head on main.

Files with missing lines	Patch %	Lines
dcoraid/bagit/manifest.py	88.88%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #113      +/-   ##
==========================================
+ Coverage   71.48%   72.38%   +0.90%     
==========================================
  Files          66       71       +5     
  Lines        5842     5997     +155     
==========================================
+ Hits         4176     4341     +165     
+ Misses       1666     1656      -10     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.