Repository navigation
Conversation
Add one CSV partition per Node of a synthetic data set, a manifest with URLs, sizes and checksums, and the seeded generator that writes them. Tests check that the committed files match the generator. Add CI (lint, test, gate) for the Python code of the repository. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Keep LF endings of the test data, write the manifest with LF, say that the URLs follow main, and test the documented difference between the partitions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Reviewer's GuideData structuresAdd the synthetic-growth partitions and manifest that publish their Node assignments, download URLs, sizes, checksums, and dataset metadata.
ImplementationImplement deterministic generation of the Node partitions and their manifest, including the command-line entry point.
ConfigurationConfigure Python development tooling and CI lint, test, and gate jobs, alongside repository ignore and line-ending rules.
DocumentationDocument the synthetic datasets, their locations and regeneration, and the local checks for contributors.
TestsTest generator determinism, partition differences, output and manifest integrity, CLI behavior, and agreement with committed data.
Lock files and vendored codeLock the development dependency set used by the new Python tooling and CI.
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
uniqueg
marked this pull request as ready for review
November 13, 2026 20:33
Linked issuesChecked at commit 50c6acb #6 demo: provide test resources · 0 of 1 met
|
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
There was a problem hiding this comment.
Hey - I've found 5 issues
Prompt for AI Agents
Please address the comments from this code review:
## Individual Comments
### Comment 1
<location path="scripts/generate_test_data.py" line_range="113" />
<code_context>
+ content = _to_csv(make_rows(node=node))
+ target = out_dir / rel_path
+ target.parent.mkdir(parents=True, exist_ok=True)
+ target.write_bytes(content)
+ files.append(
+ {
</code_context>
<issue_to_address>
**Mixed data generations**
When generation is interrupted or different generator versions write to the same output directory, `write_data_sets` overwrites CSVs one at a time and writes the manifest afterward, leaving old and new partitions paired with a stale manifest if generation stops or overlaps; consumers see checksum mismatches or inconsistent partitions.
Generate the CSVs and manifest in a separate staging directory, then publish the completed generation atomically while preventing concurrent writers from mixing generations.
Also at `scripts/generate_test_data.py:139`.
</issue_to_address>
### Comment 2
<location path="data/manifest.json" line_range="17" />
<code_context>
+ {
+ "node": "node-a",
+ "path": "synthetic-growth/node-a.csv",
+ "url": "https://raw.githubusercontent.com/elixir-cloud-aai/elixir-cloud-demos/main/data/synthetic-growth/node-a.csv",
+ "sha256": "425161d3670348f82b5fa1144dc9528b236c5a02fab4e609d5bbcd02fcff0f0f",
+ "size": 1626
</code_context>
<issue_to_address>
**Manifest checksums stop matching**
When `main` advances between a client fetching the manifest and downloading its listed CSVs, the manifest URLs resolve to files from the newer revision, so checksum validation fails or a client combines partitions from different revisions.
Use immutable revision-specific URLs in the manifest instead of URLs that resolve through `main`.
Also at `data/manifest.json:24`.
</issue_to_address>
### Comment 3
<location path="scripts/generate_test_data.py" line_range="74-76" />
<code_context>
+ {
+ "sample_id": f"{node}-{idx + 1:03d}",
+ "condition": condition,
+ "od600": f"{max(0.01, rng.gauss(od_mean + offset, 0.1)):.3f}",
+ "expression": f"{rng.gauss(expr_mean, 1.5):.3f}",
+ }
+ )
+ return rows
</code_context>
<issue_to_address>
**Regeneration changes across Python versions**
When the generator is run under a supported Python version whose `gauss` implementation or math behavior differs from CI's Python 3.13, `random.Random.gauss` can produce a different sequence if its implementation or math behavior changes, so `make_rows` rewrites the CSV bytes and checksums and `test_committed_data_match_generator` fails under that interpreter.
Use a specified, version-independent sampling algorithm for the generated values, or constrain generation and artifact checks to a pinned Python runtime.
</issue_to_address>
### Comment 4
<location path="scripts/generate_test_data.py" line_range="118" />
<code_context>
+ {
+ "node": node,
+ "path": rel_path,
+ "url": f"{base_url}/{rel_path}",
+ "sha256": hashlib.sha256(content).hexdigest(),
+ "size": len(content),
</code_context>
<issue_to_address>
**Partition URLs have extra slash**
When a caller supplies a slash-terminated `base_url`, `write_data_sets` appends another slash when building each URL, producing an empty path segment; servers that distinguish repeated slashes do not find the file, so consumers cannot download the partition.
Normalize the trailing slash from `base_url` before joining it with `rel_path`.
</issue_to_address>
### Comment 5
<location path="README.md" line_range="29-31" />
<code_context>
| --- | --- | --- |
| [2023-ecp-f2f](demos/2023-ecp-f2f/README.md) | ELIXIR Compute Face-to-Face, Helsinki, Feb 7-8, 2023 | Task execution via the [Task Execution Service (TES) API](https://github.com/ga4gh/task-execution-schemas/) on Kubernetes and HPC; TES-based task distribution with random and distance-based task distribution logic; execution of a CWL workflow via the `cwl-tes` TES backend |
+## Test data
+
+The folder [`data/`](data/README.md) holds small, open and synthetic test
+data sets for the demos, e.g. one partition per Node for federated analytics.
+Its [`manifest.json`](data/manifest.json) lists the URL, size and checksum of
</code_context>
<issue_to_address>
**Demo references are missing**
The change documents test data for the demos and publishes a manifest, but does not reference the data from any demo. Add the data references to the demos that use it.
</issue_to_address>Sourcery assessment
Needs a human reviewer. 5 findings to address first, and the CSV partitions and manifest become public and can be downloaded or copied into demo runs; reverting removes the current main URLs but does not erase the original commit or copies already taken. Because these are synthetic fixtures, incorrect values are bounded and can be replaced or regenerated.
Blocking findings: scripts/generate_test_data.py:113, data/manifest.json:17, scripts/generate_test_data.py:76, scripts/generate_test_data.py:118, README.md:31
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Publishes small, open test data sets for the demos (D4.6 test resources).
data/synthetic-growth/node-a.csvandnode-b.csv: one partition per Node, 50 rows each, CC0. The data are synthetic and made up by a seeded generator. They describe no person, organism or experiment.data/manifest.json: the URL (raw GitHub,main), size and SHA-256 of every file. Demos and TES tasks reference the data through this manifest.scripts/generate_test_data.py: the reproducible generator. The tests fail if the committed files differ from its output.data/README.mdand the root README document the data sets, their locations and how to regenerate them.pyproject.tomlwith uv, ruff, mypy and pytest (100% coverage), and.github/workflows/ci.yml(Lint,Test, gateCI passed). The retired demo2023-ecp-f2fis excluded.The federated analytics demo (#5) uses this data and follows in a separate PR.
Closes #6
Verification
uv run pytest: 8 passed, 100% coverageuv run ruff check .,uv run ruff format --check .anduv run mypy: cleanLint,TestandCI passedgreen (run 43502346076)Review
adversarial-reviewerandquality-reviewer: no high findings. Fixed: line endings pinned with.gitattributesand an LF manifest, a note that the URLs followmain, a test for the documented difference between Nodes,match=on the error test, and one form of the commands in the docs.Declined review findings
LintandTestjobs: this follows the org's CI pattern.make_rows: harmless, because the module only uses known Nodes.Note for the maintainer
CI passedcan be required in the repo's ruleset (maintainer action).🤖 Generated with Claude Code
Summary by Sourcery
Introduce reproducible synthetic test data and the tooling, documentation, and CI needed to validate it.
New Features:
Enhancements:
Build:
CI:
Documentation:
Tests:
Chores: