Map version files lazily in describeVersion, describeObject and getObject

[mohideen]

Map version files lazily in describeVersion, describeObject and getObject

Fixes #153

Summary

describeVersion(), getObject() and describeObject() built details for every file in a version before returning (describeObject() for every version). Looking up one file, checking a path, or reading the head version number therefore cost O(files in the object), and describeObject() cost O(versions × files). See # for details and how Fedora hits it.

This PR returns read-only maps that create a file's details when that path is looked up, using the inventory's existing O(1) per-path lookups. The first full iteration creates every entry once and keeps them, so iterating costs about the same as before.

Changes

• New io.ocfl.core.util.VersionStateMap: a read-only Map<String, V> over one version's state.
  • get/containsKey are O(1).
  • keySet() and size() never create values.
  • The first iteration of entrySet()/values() builds a regular map and keeps it for later iterations and lookups.
• ResponseMapper.mapFileDetails returns a VersionStateMap<FileDetails>. This is what describeVersion() and describeObject() use.
• DefaultOcflStorage.getObjectStreams returns a VersionStateMap<OcflFileRetriever>.
• DefaultOcflRepository.getObject builds its OcflObjectVersionFile map as a VersionStateMap on top of those two maps.

No public method signatures change and no dependencies are added.

Compatibility

Code that only reads the results needs no changes. Behaviour changes downstream code could notice:

• Read-only maps. The maps and collections returned by VersionDetails.getFileMap()/getFiles(), OcflObjectVersion.getFiles() and OcflStorage.getObjectStreams() are now read-only. Code that modifies them (or a custom OcflStorage wrapper that rewrites the stream map) gets UnsupportedOperationException. Workaround: new HashMap<>(versionDetails.getFileMap()).
• New instances before the first iteration. Until a map has been iterated, each getFile(path) returns a new FileDetails/OcflObjectVersionFile. Changes made to a returned FileDetails are not reflected in later lookups. equals() comparisons are unaffected.
• Iteration order (also in Jackson output of fileMap) differs from the previous HashMap order. Neither order was ever guaranteed.
• Errors surface later. Inconsistencies in an inventory, such as a state entry with no manifest entry, now surface when that path is looked up, rather than when describeVersion()/getObject() is called.
• Inventory retained. Results hold a reference to the inventory they were built from (typically already in the inventory cache), rather than their own copies of every file's details.

Because of the read-only change, a minor release (2.3.0) may be more appropriate than a patch release.

Benchmarks

JMH, one object whose every version holds all of its files, default inventory cache enabled. Apple M2 Pro, JDK 26, 1 fork, 3 × 1 s warmup, 5 × 1 s measurement. Time per call, 2.2.3 → this PR (1 version unless noted).

Lookups

Operation	1 file	10 files	100 files	1,000 files	10,000 files
describeVersion(head).getVersionNum()	276 → 150 ns*	3.1 µs → 59 ns	36.9 µs → 57 ns	498 µs → 56 ns	7.0 ms → 54 ns
describeVersion(head).containsFile(path)	318 → 63 ns	3.1 µs → 63 ns	41.3 µs → 64 ns	739 µs → 62 ns	6.5 ms → 61 ns
getObject(head).getFile(path) + read	27.8 → 28.9 µs	34.4 → 26.9 µs	104.7 → 28.6 µs	1.3 ms → 32.2 µs	14.0 ms → 31.9 µs
describeObject(id).getHeadVersionNum()	380 → 118 ns	3.1 µs → 132 ns	36.4 µs → 114 ns	751 µs → 124 ns	6.6 ms → 156 ns
same, 10 versions	3.5 µs → 481 ns	37.5 µs → 459 ns	652 µs → 457 ns	7.6 ms → 467 ns	66.1 ms → 537 ns

* High variance in this run (±243 ns); about 60 ns in every other configuration.

Allocation per lookup drops from 1.0 KB (1 file) up to 8.7 MB (10,000 files) to 216 B, and for describeObject with 10 versions from up to 80.6 MB to 3.1 KB. Reading one file is dominated by disk access, so it is unchanged for very small objects.

Iterating over every file (the case laziness can't help)

Operation	1 file	10 files	100 files	1,000 files	10,000 files
Iterate getFiles() once	318 → 334 ns	3.1 → 3.2 µs	39.8 → 41.8 µs	737 → 491 µs	7.4 → 7.5 ms
Iterate the same getFiles() twice	325 → 352 ns	3.2 → 3.3 µs	37.7 → 34.0 µs	558 → 479 µs	7.3 → 8.1 ms

Iteration is within about ±15% of 2.2.3, with allocation unchanged. The largest relative difference is up to 15% (16–75 ns) slower for objects of 1–10 files.

In Fedora: a 500-page item stored as an archival group (2,003 resources in one OCFL object), Fedora 7 with Postgres 17, ocfl-java 2.2.1 → this PR:

Request	2.2.1	This PR
List pages with descriptions embedded	7.2 s	0.16 s
Walk every resource (4,006 requests)	88 s	4.3 s
Get one page	29 ms	4 ms
Rebuild the index at startup	58 s	5 s

This puts the archival group on a par with the same item stored as separate objects (0.3–0.4 s to list, 4–6 s to walk). In the storage layer alone (fcrepo-storage-ocfl 7.0.0), reading every member's headers and RDF went from 3.0 s to 23 ms, or from 28.0 s to 94 ms for a 1,290-member group. Fedora's full test suite (2,056 tests) passes against this branch.

Benchmark source

package bench;

import io.ocfl.api.OcflOption;
import io.ocfl.api.OcflRepository;
import io.ocfl.api.model.ObjectVersionId;
import io.ocfl.api.model.VersionInfo;
import io.ocfl.api.model.VersionNum;
import io.ocfl.core.OcflRepositoryBuilder;
import io.ocfl.core.extension.storage.layout.config.HashedNTupleLayoutConfig;
import io.ocfl.core.storage.OcflStorageBuilder;
import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.util.concurrent.TimeUnit;
import org.openjdk.jmh.annotations.Benchmark;
import org.openjdk.jmh.annotations.BenchmarkMode;
import org.openjdk.jmh.annotations.Fork;
import org.openjdk.jmh.annotations.Level;
import org.openjdk.jmh.annotations.Measurement;
import org.openjdk.jmh.annotations.Mode;
import org.openjdk.jmh.annotations.OutputTimeUnit;
import org.openjdk.jmh.annotations.Param;
import org.openjdk.jmh.annotations.Scope;
import org.openjdk.jmh.annotations.Setup;
import org.openjdk.jmh.annotations.State;
import org.openjdk.jmh.annotations.TearDown;
import org.openjdk.jmh.annotations.Warmup;
import org.openjdk.jmh.infra.Blackhole;

/**
 * Cost of common read operations on a single OCFL object as its file count and version count grow.
 *
 * <p>The object has {@code files} files, written in v1; each later version overwrites 10 files, so every version's
 * state holds {@code files} paths. Repositories are built once on disk (under -Dbench.dir) and reused, so different
 * ocfl-java versions are measured against identical data. The default in-memory inventory cache is enabled, as it is
 * for any repository built with OcflRepositoryBuilder.
 */
@State(Scope.Benchmark)
@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(TimeUnit.MICROSECONDS)
@Warmup(iterations = 3, time = 1)
@Measurement(iterations = 5, time = 1)
@Fork(1)
public class LargeObjectBenchmark {

    private static final String OBJECT_ID = "o1";
    private static final ObjectVersionId HEAD = ObjectVersionId.head(OBJECT_ID);

    @Param({"1000", "10000"})
    public int files;

    @Param({"1", "10"})
    public int versions;

    private OcflRepository repo;
    private String path;

    @Setup(Level.Trial)
    public void setup() throws IOException {
        var base = Paths.get(System.getProperty("bench.dir", "bench-repos"))
                .resolve("f" + files + "-v" + versions);
        var complete = base.resolve("complete");
        var exists = Files.exists(complete);

        repo = new OcflRepositoryBuilder()
                .defaultLayoutConfig(new HashedNTupleLayoutConfig())
                .storage(OcflStorageBuilder.builder()
                        .fileSystem(Files.createDirectories(base.resolve("repo")))
                        .build())
                .workDir(Files.createDirectories(base.resolve("work")))
                .build();

        if (!exists) {
            repo.updateObject(HEAD, new VersionInfo().setMessage("v1"), updater -> {
                for (int i = 0; i < files; i++) {
                    updater.writeFile(content("file " + i), logicalPath(i));
                }
            });
            for (int v = 2; v <= versions; v++) {
                final int version = v;
                repo.updateObject(HEAD, new VersionInfo().setMessage("v" + v), updater -> {
                    for (int i = 0; i < 10; i++) {
                        int n = (version * 10 + i) % files;
                        updater.writeFile(content("file " + n + " v" + version), logicalPath(n), OcflOption.OVERWRITE);
                    }
                });
            }
            Files.createFile(complete);
        }

        path = logicalPath(files / 2);
    }

    @TearDown(Level.Trial)
    public void tearDown() {
        repo.close();
    }

    /** Only the head version number is needed, e.g. to resolve "latest". */
    @Benchmark
    public VersionNum headVersionNum() {
        return repo.describeVersion(HEAD).getVersionNum();
    }

    /** Does one logical path exist in the head version? */
    @Benchmark
    public boolean containsFile() {
        return repo.describeVersion(HEAD).containsFile(path);
    }

    /** Read one file's content from the head version. */
    @Benchmark
    public byte[] readOneFile() throws IOException {
        try (var stream = repo.getObject(HEAD).getFile(path).getStream()) {
            return stream.readAllBytes();
        }
    }

    /** Object-level metadata only (head version number); maps every version. */
    @Benchmark
    public VersionNum describeObjectHead() {
        return repo.describeObject(OBJECT_ID).getHeadVersionNum();
    }

    /** Iterate every file's details once: the case laziness cannot help. */
    @Benchmark
    public void iterateAllFiles(Blackhole bh) {
        for (var file : repo.describeVersion(HEAD).getFiles()) {
            bh.consume(file.getStorageRelativePath());
        }
    }

    /** Iterate the same VersionDetails twice: the case where laziness costs extra. */
    @Benchmark
    public void iterateAllFilesTwice(Blackhole bh) {
        var details = repo.describeVersion(HEAD);
        for (int pass = 0; pass < 2; pass++) {
            for (var file : details.getFiles()) {
                bh.consume(file.getStorageRelativePath());
            }
        }
    }

    private static String logicalPath(int i) {
        return "dir" + (i % 100) + "/file" + i + ".txt";
    }

    private static ByteArrayInputStream content(String value) {
        return new ByteArrayInputStream(value.getBytes(StandardCharsets.UTF_8));
    }
}

Testing

• New VersionStateMapTest: lookups map only the requested path; keys and size map nothing; repeated iteration maps each path once and returns the same instances; equality with an eagerly built map; modifications rejected.
• ResponseMapperTest: file details for every path, including duplicate content and fixity.
• FileSystemOcflStorageTest: getObjectStreams() creates no retrievers until a path is looked up, then exactly one.
• ./mvnw clean install passes: ocfl-java-api 14, ocfl-java-core 326, ocfl-java-aws 19 and ocfl-java-itest 367 tests (filesystem and S3), 0 failures.