Experimenter for Iterative Optimization Heuristics (IOHs), built in C++.
- Documentation: https://iohprofiler.github.io/IOHexperimenter.
- Publication: https://arxiv.org/abs/1810.05281.
- Wiki page: https://iohprofiler.github.io.
- Changelog: CHANGELOG.md. Version 0.4.0 changes parts of the Python and C++ APIs: see the upgrade guide.
IOHexperimenter provides:
- A framework to ease the benchmarking of any iterative optimization heuristic.
- Pseudo-Boolean Optimization (PBO) problem set (25 pseudo-Boolean problems).
- Integration of the well-known Black-box Optimization Benchmarking (BBOB) problem set (24 continuous problems).
- W-model problem sets constructed on OneMax and LeadingOnes.
- Integration of the Tree Decomposition (TD) Mk Landscapes problems.
- Submodular optimization problems, as seen in the GECCO '22 workshop.
- Several benchmark suites from the CEC conference.
- Flexible interface for adding new suites and problems.
- Advanced logging module that takes care of registering the data in a seamless manner.
- Data format is compatible with IOHanalyzer.
- Dynamic BinVal functions (paper)
- Double funnel functions (paper)
Available Problem Suites:
- BBOB (Single Objective Noiseless) (COCO)
- SBOX-COST (COCO)
- StarDiscrepancy
- PBO
- Submodular Graph Problems
- CEC 2013 Special Session and Competition on Niching Methods for Multimodal Function Optimization
- CEC 2022 Special Session and Competition on Single Objective Bound Constrained Numerical Optimization
IOHexperimenter is a C++17 library, built with CMake (3.16 or later); programs link its ioh::ioh target. The installation, a Getting-Started guide and the complete API documentation can be found here. In addition to the documentation, some example projects can be found in the example folder of this repository.
The Python interface is available on PyPI: pip install ioh. It needs Python 3.10 or later; 3.12 or later is recommended. There are wheels for Linux (x86_64, aarch64), Windows (x64, ARM64) and macOS (Intel, arm64), also for the free-threaded Python 3.14t and 3.15t; elsewhere pip builds the package from source, which needs a C++17 compiler. To build it from a clone of this repository (with its submodules), run pip install . (see building and testing).
The data files of the problems are installed with the package: ioh.data_root() shows where they are, and ioh.set_data_root(path) or the environment variable IOH_RESOURCES choose another directory.
A tutorial with python in the form of a jupyter notebook can be found in the example folder of this repository. A Getting-Started guide and the full API documentation can be found here.
If you have any questions, comments or suggestions, please don't hesitate contacting us IOHprofiler@liacs.leidenuniv.nl.
- Jacob de Nobel, Leiden Institute of Advanced Computer Science,
- Furong Ye, Leiden Institute of Advanced Computer Science,
- Diederick Vermetten, Leiden Institute of Advanced Computer Science,
- Hao Wang, Leiden Institute of Advanced Computer Science,
- Carola Doerr, CNRS and Sorbonne University,
- Thomas Bäck, Leiden Institute of Advanced Computer Science,
When using IOHprofiler and parts thereof, please kindly cite this work as
Jacob de Nobel, Furong Ye, Diederick Vermetten, Hao Wang, Carola Doerr and Thomas Bäck, IOHexperimenter: Benchmarking Platform for Iterative Optimization Heuristics, arXiv e-prints:2111.04077, 2021.
@ARTICLE{denobel2024IOHexperimenter,
author={de Nobel, Jacob and Ye, Furong and Vermetten, Diederick and Wang, Hao and Doerr, Carola and Bäck, Thomas},
journal={Evolutionary Computation},
title={IOHexperimenter: Benchmarking Platform for Iterative Optimization Heuristics},
year={2024},
volume={32},
number={3},
pages={205-210},
keywords={Iterative optimization heuristics;benchmarking;algorithm comparison},
doi={10.1162/evco_a_00342}
}