FunctionApproximationEnv:step()now wraps a scalar integer action in a 1-D array vianp.atleast_1d()before iterating over action dimensions, fixing anAttributeError/TypeErrorthat occurred when the environment has a single continuous dimension and the agent passes a plainint.
FunctionApproximationBenchmark: action space no longer over-allocates a bin forget_benchmark(dimension=1). The integer hyperparameter'suppernow matches the discrete bin count ([3]→ 3 valid indices), so stepping with a valid index no longer raisesIndexError.FunctionApproximationBenchmark:observation_space_argsderivation no longer assumes a fixed 3-entryinstance_description. The size is now derived fromstate_description, keeping it consistent regardless ofomit_instance_typeor toy-function layout.FunctionApproximationBenchmark: module-levelbenchmark_infois no longer mutated across repeatedget_benchmark()calls. A new_isolate_benchmark_info()helper deep-copies the info and itsstate_descriptionlist at construction time.
- Added regression tests for action-space size, observation-space derivation under
omit_instance_type=True, and isolation of the shared info dict.
- DACBO benchmark documentation now builds without the optional
dacboenvdependencies installed;smacandomegaconfare mocked during Sphinx autodoc.
- GitHub release is now created automatically after a successful PyPI publish.
- DACBO benchmark page is now included in the documentation build (was missing from the toctree).
- Corrected the episode termination description: early termination via reference performance is opt-in (
terminate_after_reference_performance_reached=True), not the default behaviour.
pytestbumped from 8.4.2 to 9.1.1.pre-commitbumped from 3.8.0 to 4.6.0.- Minor dependency updates across the development group.
- Added PyPI publish workflow using OIDC trusted publisher.
Bayesian Optimisation is now a supported DAC benchmark. DACBOBenchmark / DACBOEnv frames BO as a sequential decision problem where the agent controls hyperparameters of the BO loop (e.g. acquisition function parameters) at each iteration. It was previously an external dacboenv package; it is now inlined and no longer requires the carps dependency.
- Built on SMAC3 and IOH directly.
- Defaults to all 24 BBOB 2-D functions as the instance set; custom instance sets can be supplied as paths relative to Hydra's search path.
interaction_frequencyparameter controls how often the agent is queried per BO step.ReferencePerformancecomputes normalised regret baselines and supportsseeds=Noneto average over multiple seeds automatically.- Registers as
"DACBO-v0"in the Gymnasium registry on import (requires optionaldacboextra). - Documentation, README, and an example notebook are included.
gymnasiumupper bound removed;>= 1.0is now fully supported.- Upper-version pins removed across all core dependencies (
numpy,pandas,scikit-learn,scipy,matplotlib,seaborn,configspace,imageio,ioh, etc.). Versions follow PEP 440 lower bounds only. pyarrowadded as a core dependency.- Dependabot enabled for the
uvecosystem to surface dependency drift automatically.
setup.pyremoved. The version is now derived from git tags viasetuptools_scm; the fallback is defined inpyproject.toml.flake.nix/.devenvshell added for a fully-pinned contributor environment without manual toolchain management.pre-commithooks updated to run viauvx.
Theoryenv: corrected action-space size computation and relaxed overly strict key constraints.ConfigSpaceintegration: fixed parsing of discrete action spaces and integer hyperparameter conversion.AbstractEnv: task and instance IDs are no longer overwritten when an explicit test set is passed toreset().DACBOEnv:update_optimizercall restored instep()— was silently dropped during refactor.DACBOEnv: guard against callingmodel.train()on an empty initial design dataset.DACBOEnv: dependency guard, action-space construction, and optional-import handling corrected.- Renderers: replaced deprecated
tostring_rgbwithbuffer_rgba. - Documentation: fixed broken DAC literature cross-link.
examples/smac_agent.pydemonstrates a full SMAC-in-agent configuration.
- SGD benchmark tests sped up and hardened.
FunctionApproximationBenchmark: observation space bounds are now derived correctly from theconfig_spacewhen one is provided, rather than falling back to the default bounds. The 1-D and 2-D sigmoid preset methods also set the correct bounds.
iohadded as a core dependency.- Version pins loosened for
gymnasium,imageio, andconfigspace.
Instances for most benchmarks now have a dataclass Type instead of being a simple list. Additionally, the corresponding datasets are saved as proper csv files with headers now. This should make anything relating to instances much more convenient for the user. The benchmarks in question are:
- CMA-ES
- SGD
- Luby
- ToySGD
- Function Approximation
Instead of having two different versions of CMA-ES, we now have a single environment which covers both step size adaption and algorithm variant selection of CMA-ES (formerly ModCMA). By changing the configuration space, the user can select which hyperparameters to adapt. This change includes a switch to the newer "ioh" package, meaning an increased amount of target functions could be interfaced in principle. Anything outside of BBOB will need to be loaded separately from the "read_instance_set" function of the benchmark, however.
To reduce the number of separate classes to maintain for simple function approximation, the Sigmoid variations and the Geometric environment have been fused into the FunctionApproximation Benchmark. There are several options for functions to approximate, as in Sigmoid it is possible to use a discrete space and you can add importance weights for the dimensions. The "get_benchmark" method will still provide the original Sigmoid configurations. Apart from that, this new environment should provide all functionality of the previous environments, just with a simpler path to getting there.
The original SGD benchmark was complex and error prone, lacking important features like compatibility with torchhub. Therefore this has been re-implemented, mostly based on the existing competition version. The code is simpler now and compatible with a larger selection of models and optimizers.
The FastDownward version compatible with the benchmark cannot be used with any Ubuntu version after 16.x - which is fairly old by now. For reference, it is not possible to even build a container for this version of FastDownward on th GitHub servers. Since there is no option to update the planner version without updating the environment and this is tied to significant domain knowledge, FastDownward will now be deprecated. This means there is no testing, the benchmark will not be updated and it will not be listed as an official benchmark any longer. If someone is familiar enough with FastDownward to facilitate an update, please notify us, we'd love to continue this benchmark!
Most instance sets have been updated due to the instance specification change. The Sigmoid ones have been preserved, the rest has been updated. Sampling options for all benchmarks are included in the "instance_set" directory, however.
The logs have been a bit hard to read & work with. We flattened them by removing timestamps, hopefully making them easier to work with.
The dependencies have been upgraded to their current highest possible version.
Some examples had a lot of extra dependencies. We removed many of these for now - this means less explicit RL examples, but if you want to use an RL library, DACBench will plug in like any other env, so you should read their documentation anyway.
There was a persistent configuration mistake which prevented the instance sets to be included in the PyPI installation - this should now be fixed and all instances sets from the repository come with the PyPI package.
The main change in this version is going from OpenAI's gym to the newer gymnasium version. The outward change is slight, but this interface is now incompatible with version 0.1.0. To adapt to this version, you'll mainly have to replace instances of 'done' for termination with two variables: 'terminated' indication algorithm termination and 'truncated' indicating a timeout. Combined they're equal to the old 'done'. Additonally, the default version of the environments is now available in the gym registry.
We added more options for controlling several hyperparameters at once. Using the PettingZoo API, users can now select which hyperparameters to control and use a typical Multi-Agent RL interface to do it. This should provide more freedom in how to solve the problem of scaling up to multiple hyperparameters.
We updated all our dependencies for this release. Please note that this will likely influence benchmark behaviour, so do not directly compare performance of version 0.2.0 with 0.1.0!
The OneLL benchmark is not the Theory benchmark with a similar goal and setup, but a different base problem. For versioning reasons, we removed ModEA, the same problem should be covered by ModCMA. We also add a toy benchmark for higher dimensional spaces, the Geometric Benchmark.
The documentation is now hosted on GitHub.io instead of Read the Docs for versioning reasons. The old version should still be accessible, however.
New benchmarks include the ModCMA IOHExperimenter version of ModEA, the OneLL EA benchmark and a toy version of controlling SGD.
For added reproducibility, we provide Singularity recipes for each benchmark. This way they can be run in containers.
Search Spaces can now be defined via ConfigSpace and are then automatically converted to gym spaces. This should make it easier to recover whcih actions correspond to which hyperparameters.
Initial Version