Store optimization data using blobs by frode-aarstad · Pull Request #13948 · equinor/ert

frode-aarstad · 2026-07-06T12:27:04Z

Issue
Resolves #13901

Approach
Short description of the approach

(Screenshot of new behavior in GUI if applicable)

PR title captures the intent of the changes, and is fitting for release notes.
Added appropriate release note label
Commit history is consistent and clean, in line with the contribution guidelines.
Make sure unit tests pass locally after every commit (git rebase -i main --exec 'just rapid-tests')

When applicable

When there are user facing changes: Updated documentation
New behavior or changes to existing untested code: Ensured that unit tests are added (See Ground Rules).
Large PR: Prepare changes in small commits for more convenient review
Bug fix: Add regression test for the bug
Bug fix: Add backport label to latest release (format: 'backport release-branch-name')

codecov-commenter · 2026-07-06T12:39:43Z

Codecov Report

❌ Patch coverage is 98.24561% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 91.93%. Comparing base (47757c5) to head (dbb6238).
⚠️ Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
src/ert/storage/migration/to38.py	96.66%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #13948      +/-   ##
==========================================
- Coverage   91.93%   91.93%   -0.01%     
==========================================
  Files         482      483       +1     
  Lines       33516    33558      +42     
==========================================
+ Hits        30813    30850      +37     
- Misses       2703     2708       +5

Flag	Coverage Δ
cli-tests	`36.43% <10.52%> (-0.04%)`	⬇️
fuzz	`44.43% <28.07%> (-0.03%)`	⬇️
gui-tests	`58.76% <10.52%> (-0.09%)`	⬇️
performance-and-unit-tests	`80.92% <89.47%> (+0.02%)`	⬆️
test	`46.03% <47.36%> (+0.04%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
src/ert/storage/blob_data.py	`100.00% <100.00%> (ø)`
src/ert/storage/local_ensemble.py	`96.80% <100.00%> (+0.03%)`	⬆️
src/ert/storage/local_storage.py	`92.90% <100.00%> (ø)`
src/ert/storage/migration/to38.py	`96.66% <96.66%> (ø)`

... and 1 file with indirect coverage changes

codspeed-hq · 2026-07-06T13:21:59Z

Merging this PR will not alter performance

✅ 36 untouched benchmarks

_{Comparing frode-aarstad:everest-data-in-blobs (dbb6238) with main (47757c5)}

Copilot

Pull request overview

This PR updates ERT’s local storage format to store Everest optimization “batch” parquet dataframes as ensemble blobs (instead of loose *.parquet files under the ensemble directory), including a migration to move existing batch parquet files into blob storage and tests validating the new behavior (including dark-storage HTTP access).

Changes:

Add a new blob metadata type (EVEREST_BATCH_DATA / EverestBatchData) and persist Everest batch dataframes as blobs via LocalEnsemble.save_batch_dataframes().
Introduce local-storage schema version 36 and a to36 migration that moves legacy batch_*.parquet files into ensembles/<ens>/blobs/.
Extend unit tests to cover blob persistence, migration behavior, and dark-storage /ensembles/{id}/blobs + /blobs/{uri} endpoints for Everest batch data.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
tests/ert/unit_tests/storage/test_local_storage.py	Adds tests that batch dataframes are saved as Everest batch blobs and that missing batch data returns `None`.
tests/ert/unit_tests/storage/migration/test_to36.py	Adds migration tests ensuring batch parquet files are moved into blob storage and non-batch parquet files remain untouched.
tests/ert/unit_tests/dark_storage/test_http_endpoints.py	Adds endpoint tests for listing Everest batch dataframe blobs and fetching parquet bytes via blob URI.
src/ert/storage/migration/to36.py	Implements the v36 migration that moves `batch_*.parquet` into `blobs/` with Everest batch blob metadata.
src/ert/storage/local_storage.py	Bumps local storage version to 36 and wires in the `to36` migration.
src/ert/storage/local_ensemble.py	Writes Everest batch dataframes into blob storage and reads them back from blobs via new helper logic.
src/ert/storage/blob_data.py	Adds the `EVEREST_BATCH_DATA` discriminator and `EverestBatchData` model to the blob metadata union.

Copilot

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

ertomatic · 2026-07-14T11:45:07Z

Screenshots differ from baselines. A baseline update PR has been prepared: equinor/ert-testdata#49

xjules · 2026-08-19T06:41:42Z

+info = "Remove is_improvement flag from ensembles"
+
+
+def migrate(path: Path) -> None:


I guess this migration is not required for this PR.
If it is only cleanup then maybe it might be merged with the other migration.

ertomatic · 2026-08-19T07:33:52Z

Screenshot tests now pass. The baseline update PR equinor/ert-testdata#49 has been closed.

xjules · 2026-08-19T07:39:59Z

+    def _read_batch_dataframe(self, dataframe_name: str) -> pl.DataFrame | None:
+        for meta in self.load_blobs(BlobType.EVEREST_BATCH_DATA):
+            if (
+                isinstance(meta.blob_info, EverestBatchData)


I guess this isinstance validation is there for the sake of mypy otherwise it is redundant

Its only for better typing

xjules

No more comments. Good job @frode-aarstad !
Remember to squash the commits.

frode-aarstad self-assigned this Jul 6, 2026

frode-aarstad requested a review from Copilot July 7, 2026 06:09

Copilot started reviewing on behalf of frode-aarstad July 7, 2026 06:09 View session

Copilot AI reviewed Jul 7, 2026

View reviewed changes

Comment thread src/ert/storage/local_ensemble.py Outdated

Comment thread src/ert/storage/local_ensemble.py Outdated

frode-aarstad force-pushed the everest-data-in-blobs branch from 46e03a3 to bcdee6a Compare July 9, 2026 06:52

frode-aarstad requested a review from Copilot July 9, 2026 07:10

Copilot started reviewing on behalf of frode-aarstad July 9, 2026 07:11 View session

Copilot AI reviewed Jul 9, 2026

View reviewed changes

Comment thread src/ert/storage/migration/to36.py Outdated

xjules reviewed Jul 9, 2026

View reviewed changes

Comment thread src/ert/storage/local_ensemble.py

frode-aarstad mentioned this pull request Jul 9, 2026

Stop writing to storage in everest_run_model #13981

Open

Store optimization data using blobs

6fafe67

frode-aarstad force-pushed the everest-data-in-blobs branch from bcdee6a to 6fafe67 Compare July 14, 2026 11:36

ertomatic mentioned this pull request Jul 14, 2026

Update screenshots for equinor/ert#13948 equinor/ert-testdata#49

Closed

xjules reviewed Aug 17, 2026

View reviewed changes

Comment thread tests/ert/unit_tests/storage/test_local_storage.py Outdated

Change migration version to 38

3fc88a1

xjules reviewed Aug 19, 2026

View reviewed changes

Merge branch 'main' into everest-data-in-blobs

6e8aca7

xjules reviewed Aug 19, 2026

View reviewed changes

Move some tests

dbb6238

frode-aarstad force-pushed the everest-data-in-blobs branch from 9391612 to dbb6238 Compare August 19, 2026 08:03

xjules approved these changes Aug 19, 2026

View reviewed changes

frode-aarstad merged commit a072035 into equinor:main Aug 19, 2026
59 of 67 checks passed

frode-aarstad deleted the everest-data-in-blobs branch August 19, 2026 09:56

Navigation Menu

Uh oh!

Store optimization data using blobs - #13948

Store optimization data using blobs#13948
frode-aarstad merged 4 commits into
equinor:mainfrom
frode-aarstad:everest-data-in-blobs

frode-aarstad commented Jul 6, 2026

Uh oh!

codecov-commenter commented Jul 6, 2026 •

edited

Loading

Uh oh!

codspeed-hq Bot commented Jul 6, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

ertomatic commented Jul 14, 2026

Uh oh!

Uh oh!

xjules Aug 19, 2026

Uh oh!

ertomatic commented Aug 19, 2026

Uh oh!

xjules Aug 19, 2026 •

edited

Loading

Uh oh!

frode-aarstad Aug 19, 2026

Uh oh!

xjules left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

		info = "Remove is_improvement flag from ensembles"


		def migrate(path: Path) -> None:

Uh oh!

Conversation

frode-aarstad commented Jul 6, 2026

When applicable

Uh oh!

codecov-commenter commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

codspeed-hq Bot commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will not alter performance

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

ertomatic commented Jul 14, 2026

Uh oh!

Uh oh!

xjules Aug 19, 2026

Choose a reason for hiding this comment

Uh oh!

ertomatic commented Aug 19, 2026

Uh oh!

xjules Aug 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

frode-aarstad Aug 19, 2026

Choose a reason for hiding this comment

Uh oh!

xjules left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

codecov-commenter commented Jul 6, 2026 •

edited

Loading

codspeed-hq Bot commented Jul 6, 2026 •

edited

Loading

xjules Aug 19, 2026 •

edited

Loading