Skip to content

Add the AdversarialAttack problems: black-box adversarial attacks - issue #248 - #249

Open
wz7475 wants to merge 1 commit into
IOHprofiler:masterfrom
wz7475:adversarial-attack-suite
Open

wz7475 wants to merge 1 commit into
IOHprofiler:masterfrom
wz7475:adversarial-attack-suite

Conversation

@wz7475

@wz7475 wz7475 commented Nov 6, 2026 •

Copy link
Copy Markdown

Add the AdversarialAttack problems: black-box adversarial attacks - issue #248

Problem Name(s):

AdversarialAttack -- 12 presets: AdversarialMNIST5001..5006 (MNIST, d=784),
AdversarialCIFAR105011..5016 (CIFAR-10, d=3072). Real-valued, maximized.

Added file(s):

  • include/ioh/problem/adversarial/tensor_file.hpp -- reads the IOHNET01 binary container (weights and image
    banks), dequantizing float16/uint8 to float32.
  • include/ioh/problem/adversarial/network.hpp -- dependency-free feed-forward CNN (conv/relu/maxpool/linear). The
    network and the image bank of a dataset are loaded with ioh::data::file and shared with ioh::data::cached.
  • include/ioh/problem/adversarial/adversarial_attack.hpp -- the problem class, with its 12 presets in a static
    registry_entries().
  • src/registry/adversarial.cpp -- the factory of the class.
  • static/adversarial/{mnist,cifar}_{cnn,images}.ioh and PROVENANCE.txt -- the data files (about 11 MB), added to
    static/manifest.json, src/data_manifest.inc and static.tar.gz.
  • tests/cpp/problem/test_adversarial.cpp, tests/data/adversarial.in, and a Python test in
    tests/python/test_problem.py.
  • Small edits: include/ioh/problem.hpp, include/ioh/suite.hpp (the suite, id 5), include/ioh/common/builtins.hpp,
    src/registry/{families.hpp,single_objective.cpp,suites.cpp}, CMakeLists.txt,
    ioh/src/{common.hpp,problem.cpp,real.cpp,real_suite.cpp}, ioh/__init__.py (ProblemClass.ADVERSARIAL), the
    stubs, the docs, README.md and CHANGELOG.md.
  • The golden fixtures (tests/data/golden, tests/python/golden.py): see below.

Definition(s) of the problem(s):

f(delta) = -log softmax(model(x + delta))[y] - alpha * ||delta||_2      (maximized)

x is a fixed base image from a bundled 100-image bank (selected by instance, 1-indexed,
(instance - 1) mod 100), y its true label, delta the search vector in [-eps, eps]^d. Deterministic and
noiseless. x + delta is not clamped to [0, 1] by default (matches the reference implementation's main
experimental path; clamping creates flat regions that break step-size-adaptive optimizers like CMA-ES).

ids dataset eps alpha
5001-5003 MNIST 0.1 / 0.3 / 1.0 0.0
5004-5006 MNIST 0.1 / 0.3 / 1.0 0.05
5011-5013 CIFAR-10 0.1 / 0.3 / 1.0 0.0
5014-5016 CIFAR-10 0.1 / 0.3 / 1.0 0.05

Description of the implementation:

The class lists its presets in registry_entries(), like the star discrepancy and CEC problems, so listing the
registry loads no data. It is registered in the factory of all real problems, in its own factory and in the suite
AdversarialAttack (suite id 5). The network and the 100-image bank of a dataset are read when the first problem of
that dataset is created, and shared by all its instances. Only the scratch activation buffers belong to a problem
object, so one object is not safe to evaluate from several threads at once (the Python bindings and
test_threaded.cpp evaluate different objects).

The CNN forward pass (plain direct convolution) reimplements the architectures of amheattack/models.py in float.
The weights are stored as float16; this is lossless on the full test sets (MNIST 0.9787, CIFAR-10 0.7173, both +0.0000
against the fp32 checkpoint, see static/adversarial/PROVENANCE.txt).

Regression data: tests/data/adversarial.in stores fid iid seed y instead of the usual fid iid x y, because x
has up to 3072 values. The test rebuilds the perturbation from seed with a 3-line LCG
(s = (1103515245*s + 12345) mod 2^31; x_i = eps*(2*s/2^31 - 1), composite seed seed*1000 + image_index), the same
one as the script that generated the file from the PyTorch models. 180 rows: 5 images x 3 seeds x 12 problems. The
tolerance is mixed (err <= 1e-4 || err/max(1,|y|) <= 5e-3): the plain-loop float32 sums differ slightly from
PyTorch's, more so when the input leaves the [0, 1] range of training (the eps=1.0 problems). All 180 rows pass.

Golden fixtures: tests/data/golden freezes the registries and the behaviour of every problem, so it has to change for
a new family. The change only adds the class ADVERSARIAL, its views, its suite and 25 behaviour records; the existing
records are unchanged, and I kept their last digits (a regeneration on another platform rounds some of them
differently, within the tolerance of the test). The one change to an existing value is in the wrap scenario: a real
problem registered at run time (wrap_problem, register_problem) gets the next free id after 5016, 5017, instead of
1121. If that is unwanted, the alternative is to leave the class out of the factory of all real problems (it would then
only be created through its own factory, ProblemClass.ADVERSARIAL and the suite).

Verified in Python: ioh.get_problem(5001, 1, 1, ioh.ProblemClass.ADVERSARIAL), ioh.suite.AdversarialAttack, the
direct constructor ioh.problem.AdversarialAttack("cifar10", eps=0.3, alpha=0.05), and the whole tests/python suite.

Known limitations / things I'd like reviewer input on:

  1. n_variables is accepted but ignored (the dimension is fixed per dataset), as for the graph problems and CEC2013;
    a hard error is the alternative.
  2. instance selects the base image and therefore changes the entire landscape, unlike most IOH problems where
    instance is an affine transformation of one function.
  3. No known optimum: optimum() is left at its default.
  4. Size: static/ grows by about 11 MB and static.tar.gz from 7.1 MB to 17.8 MB (every wheel ships static/).
    Bundling MNIST only and downloading CIFAR-10, or hosting the data elsewhere, are possible if that is too much.
  5. The gtest binaries crash in BaseTest::SetUp (clutchlog) in my environment, also for the existing tests. I ran the
    C++ tests with an entrypoint with an empty SetUp; please check them in CI.

Additional information (e.g., reference of the problems, additional required packages.):

Reference: https://github.com/wz7475/amhe-attack ("Large-scale Testing Global Optimization Methods with Black-box
Adversarial Attacks"). No additional C++ dependencies.

After merging PR please consider citing the PPSN paper, which introduces this kind of approach

@misc{zarzecki2026largescaletestingglobaloptimization,
      title={Large-scale Testing Global Optimization Methods with Black-box Adversarial Attacks}, 
      author={Wojciech Zarzecki and Jarosław Arabas},
      year={2026},
      eprint={2608.13296},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2608.13296}, 
}

…a real-valued suite

Adds 12 real-valued, maximized problems from amhe-attack
(https://github.com/wz7475/amhe-attack), a many-dimensional benchmark in which
an image classifier is attacked:

    f(delta) = -log softmax(model(x + delta))[y] - alpha * ||delta||_2

x is a fixed, correctly classified base image (one of 100 bundled per dataset,
selected by the instance), y its true label, and delta the search vector in
[-eps, eps]^d. The ids are 5001-5006 (MNIST, 784 variables) and 5011-5016
(CIFAR-10, 3072 variables): three eps (0.1, 0.3, 1.0) with alpha 0 and 0.05.

* include/ioh/problem/adversarial: the problem class, which lists its presets
  in registry_entries() like the star discrepancy and CEC problems, and a
  dependency-free float32 CNN (conv/relu/maxpool/linear) whose weights and image
  banks are read from a small binary container (float16/uint8, dequantized on
  load). The network and the images are data files, found with ioh::data::file
  and loaded once and shared with ioh::data::cached, when the first problem of
  a dataset is created: listing the registry loads no data. float16 storage is
  lossless on the full test sets (MNIST 0.9787, CIFAR-10 0.7173).
* The registry: src/registry/adversarial.cpp, the real single-objective factory,
  and the suite AdversarialAttack (suite id 5).
* Python: ioh.problem.AdversarialAttack (also constructible from a dataset, eps
  and alpha), ioh.suite.AdversarialAttack, ioh.ProblemClass.ADVERSARIAL, stubs
  and docs.
* static/adversarial: the data files (about 11 MB) and their provenance, in the
  manifest (static/manifest.json, src/data_manifest.inc) and in static.tar.gz,
  which grows from 7.1 MB to 17.8 MB.
* Tests: tests/cpp/problem/test_adversarial.cpp compares 180 values (5 images x
  3 perturbations x 12 problems) in tests/data/adversarial.in with the PyTorch
  models; a Python test; the golden fixtures gain the new class, its suite and
  records (the existing records are unchanged). The only change to an existing
  value is the id of a problem registered at run time in the real factory, 1121
  before and 5017 now, because the ids 5001-5016 are taken.

n_variables is ignored (the dimension is fixed per dataset), and the instance
changes the whole landscape instead of an affine transformation of one function.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HWyGm6y4zfNq8FUWymfvuT

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant