feat(core): add ScenarioFileFormat SPI for alternative file formats

[nkuehnel]

Summary

• Adds a ScenarioFileFormat SPI (ServiceLoader-based) that allows alternative file format providers (e.g. protobuf, parquet) to be discovered at runtime
• Includes ScenarioFileFormatRegistry for extension-based provider lookup
• Routes population read/write through PopulationUtils which checks the SPI registry before falling back to built-in XML readers/writers
• Updates DumpDataAtEndImpl and PlansDumpingImpl to use PopulationUtils.writePopulation() as the central dispatch layer
• Allows us to use new file format readers without linking them in matsim-core
• For now wired through for population reading/writing, can be extended for other scenario files

Motivation

MATSim's I/O is currently hardcoded to XML. For large-scale scenarios (100k+ agents with full routes), XML parsing becomes a bottleneck. This SPI enables alternative formats to be plugged in without modifying core code — providers just need to implement ScenarioFileFormat and register via META-INF/services.

Why ServiceLoader instead of Guice?

• ScenarioUtils.loadScenario() and PopulationUtils.readPopulation()/writePopulation() are static utilities that work without a Controler — there's no injector available at that point
• File format providers are stateless and don't need DI lifecycle management or scoping
• Adding a JAR to the classpath automatically enables the format — no explicit module installation required
• This matches Java's standard plugin pattern (JDBC drivers, charset providers, etc.)

I'd like your opinion on this, happy to adapt.
@mrieser @kainagel @sebhoerl

[nkuehnel]

for context, a protobuf population reader implementation based on this architecture achieves 8-12x read speed-up and could live in the protobuf contrib (not part of this PR)

[jfbischoff]

Nice adaption!
I'm wondering if adding Protocol Buffer or whatever as an alternative to XML directly wouldn't actually be a nice addition to the core, though of course this doesn't stand in the way of your implementation.

[nkuehnel]

I'm wondering if adding Protocol Buffer or whatever as an alternative to XML directly wouldn't actually be a nice addition to the core

Either way would work for me, happy to complete it both ways

[paulheinr]

I am not sure if using another injection framework is helpful. I already find this confusing if SPI is used for binding SimWrapper Dashboards (for the same reason that on the scenario level we don't have the guice injector yet).

IMHO, we should rather offer a guice binding already on the scenario level. Before, I would add new file types to the existing framework and decide based on the file ending which reader/writer is used (as it is done right now).

A side node: For our rust prototype, we have already defined a proto scheme https://github.com/matsim-vsp/parallel_qsim_rust/tree/main/rust_qsim/src/simulation/io/proto/types

I am happy to discuss if there are any adaptations needed and where this should be located. Maybe this discussion is a good motivation to create a separate git repository for schema definitions for proto/parquet/... such that it can be used in other repositories. I have thought about this already but for now, the level of suffering wasn't high enough.

[nkuehnel]

Before, I would add new file types to the existing framework and decide based on the file ending which reader/writer is used (as it is done right now).

This is basically what @jfbischoff said, and would mean linking protobuf to matsim core, do I understand right?

Thanks for the proto scheme reference! Happy to work on a shared version of these, I'd have a few suggestions for further improvement.

So a way forward would be to:

1. have the shared scheme repository
2. link to that in matsim and implement protobuf readers/writers in the core based on file extension

[paulheinr]

This is basically what @jfbischoff said, and would mean linking protobuf to matsim core, do I understand right?

Yes.

So a way forward would be to:

1. have the shared scheme repository
2. link to that in matsim and implement protobuf readers/writers in the core based on file extension

Yes, that's what I was thinking of, so maybe a new repository inside of matsim-org, such as matsim-ontologies or matsim-file-formats, ...? Not sure, if could also put XSDs there.

[mrieser]

I would prefer SPI over Guice. If I need a guice injector just to read a scenario, it would harm the dev-experience in my opinion. Currently, we can essentially do Scenario s = ScenarioUtils.createScenario(ConfigUtils.createConfig());. I think we should be able to keep it this simple, otherwise we might break quite a lot of code.

If I understand the proposal correctly, various formats can be registered automatically using SPI, and when reading/writing, the correct format is chosen based on the file-ending. This is actually the same as Java uses for ImageIO to support various image encodings.

Dependency Injection was originally designed to set one specific implementation for a required functionality. Supporting multiple implementations for something requires workarounds (in MATSim, e.g. to support multiple replanning strategies). If I want to provide a converter, reading one format and writing another format, I would probably have to provide two implementations.

TL;DR:

I would prefer SPI, as it is the standard Java way to do this, and I fear that with Guice existing simple code becomes more complex and convoluted. But feel free to convince me otherwise and show me how a solution with Guice could look like.

[nkuehnel]

TBH I'm also lacking the fantasy of how a good guice integration could look like here.

We could still just link protobuf to matsim core and discriminate file reading explicitly, without any guice/spi? Maybe that is actually what @jfbischoff meant with:

I'm wondering if adding Protocol Buffer or whatever as an alternative to XML directly wouldn't actually be a nice addition to the core

[paulheinr]

We could still just link protobuf to matsim core and discriminate file reading explicitly, without any guice/spi?

Sure, I think this is a fair approach. However, should we use this repo for the file formats? https://github.com/matsim-org/matsim-schemas