Repository navigation
Conversation
…a real-valued suite Adds 12 real-valued, maximized problems from amhe-attack (https://github.com/wz7475/amhe-attack), a many-dimensional benchmark in which an image classifier is attacked: f(delta) = -log softmax(model(x + delta))[y] - alpha * ||delta||_2 x is a fixed, correctly classified base image (one of 100 bundled per dataset, selected by the instance), y its true label, and delta the search vector in [-eps, eps]^d. The ids are 5001-5006 (MNIST, 784 variables) and 5011-5016 (CIFAR-10, 3072 variables): three eps (0.1, 0.3, 1.0) with alpha 0 and 0.05. * include/ioh/problem/adversarial: the problem class, which lists its presets in registry_entries() like the star discrepancy and CEC problems, and a dependency-free float32 CNN (conv/relu/maxpool/linear) whose weights and image banks are read from a small binary container (float16/uint8, dequantized on load). The network and the images are data files, found with ioh::data::file and loaded once and shared with ioh::data::cached, when the first problem of a dataset is created: listing the registry loads no data. float16 storage is lossless on the full test sets (MNIST 0.9787, CIFAR-10 0.7173). * The registry: src/registry/adversarial.cpp, the real single-objective factory, and the suite AdversarialAttack (suite id 5). * Python: ioh.problem.AdversarialAttack (also constructible from a dataset, eps and alpha), ioh.suite.AdversarialAttack, ioh.ProblemClass.ADVERSARIAL, stubs and docs. * static/adversarial: the data files (about 11 MB) and their provenance, in the manifest (static/manifest.json, src/data_manifest.inc) and in static.tar.gz, which grows from 7.1 MB to 17.8 MB. * Tests: tests/cpp/problem/test_adversarial.cpp compares 180 values (5 images x 3 perturbations x 12 problems) in tests/data/adversarial.in with the PyTorch models; a Python test; the golden fixtures gain the new class, its suite and records (the existing records are unchanged). The only change to an existing value is the id of a problem registered at run time in the real factory, 1121 before and 5017 now, because the ids 5001-5016 are taken. n_variables is ignored (the dimension is fixed per dataset), and the instance changes the whole landscape instead of an affine transformation of one function. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HWyGm6y4zfNq8FUWymfvuT
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add the AdversarialAttack problems: black-box adversarial attacks - issue #248
Problem Name(s):
AdversarialAttack-- 12 presets:AdversarialMNIST5001..5006(MNIST, d=784),AdversarialCIFAR105011..5016(CIFAR-10, d=3072). Real-valued, maximized.Added file(s):
include/ioh/problem/adversarial/tensor_file.hpp-- reads theIOHNET01binary container (weights and imagebanks), dequantizing float16/uint8 to float32.
include/ioh/problem/adversarial/network.hpp-- dependency-free feed-forward CNN (conv/relu/maxpool/linear). Thenetwork and the image bank of a dataset are loaded with
ioh::data::fileand shared withioh::data::cached.include/ioh/problem/adversarial/adversarial_attack.hpp-- the problem class, with its 12 presets in a staticregistry_entries().src/registry/adversarial.cpp-- the factory of the class.static/adversarial/{mnist,cifar}_{cnn,images}.iohandPROVENANCE.txt-- the data files (about 11 MB), added tostatic/manifest.json,src/data_manifest.incandstatic.tar.gz.tests/cpp/problem/test_adversarial.cpp,tests/data/adversarial.in, and a Python test intests/python/test_problem.py.include/ioh/problem.hpp,include/ioh/suite.hpp(the suite, id 5),include/ioh/common/builtins.hpp,src/registry/{families.hpp,single_objective.cpp,suites.cpp},CMakeLists.txt,ioh/src/{common.hpp,problem.cpp,real.cpp,real_suite.cpp},ioh/__init__.py(ProblemClass.ADVERSARIAL), thestubs, the docs,
README.mdandCHANGELOG.md.tests/data/golden,tests/python/golden.py): see below.Definition(s) of the problem(s):
xis a fixed base image from a bundled 100-image bank (selected byinstance, 1-indexed,(instance - 1) mod 100),yits true label,deltathe search vector in[-eps, eps]^d. Deterministic andnoiseless.
x + deltais not clamped to[0, 1]by default (matches the reference implementation's mainexperimental path; clamping creates flat regions that break step-size-adaptive optimizers like CMA-ES).
Description of the implementation:
The class lists its presets in
registry_entries(), like the star discrepancy and CEC problems, so listing theregistry loads no data. It is registered in the factory of all real problems, in its own factory and in the suite
AdversarialAttack(suite id 5). The network and the 100-image bank of a dataset are read when the first problem ofthat dataset is created, and shared by all its instances. Only the scratch activation buffers belong to a problem
object, so one object is not safe to evaluate from several threads at once (the Python bindings and
test_threaded.cppevaluate different objects).The CNN forward pass (plain direct convolution) reimplements the architectures of
amheattack/models.pyinfloat.The weights are stored as float16; this is lossless on the full test sets (MNIST 0.9787, CIFAR-10 0.7173, both +0.0000
against the fp32 checkpoint, see
static/adversarial/PROVENANCE.txt).Regression data:
tests/data/adversarial.instoresfid iid seed yinstead of the usualfid iid x y, becausexhas up to 3072 values. The test rebuilds the perturbation from
seedwith a 3-line LCG(
s = (1103515245*s + 12345) mod 2^31; x_i = eps*(2*s/2^31 - 1), composite seedseed*1000 + image_index), the sameone as the script that generated the file from the PyTorch models. 180 rows: 5 images x 3 seeds x 12 problems. The
tolerance is mixed (
err <= 1e-4 || err/max(1,|y|) <= 5e-3): the plain-loop float32 sums differ slightly fromPyTorch's, more so when the input leaves the
[0, 1]range of training (theeps=1.0problems). All 180 rows pass.Golden fixtures:
tests/data/goldenfreezes the registries and the behaviour of every problem, so it has to change fora new family. The change only adds the class
ADVERSARIAL, its views, its suite and 25 behaviour records; the existingrecords are unchanged, and I kept their last digits (a regeneration on another platform rounds some of them
differently, within the tolerance of the test). The one change to an existing value is in the
wrapscenario: a realproblem registered at run time (
wrap_problem,register_problem) gets the next free id after 5016, 5017, instead of1121. If that is unwanted, the alternative is to leave the class out of the factory of all real problems (it would then
only be created through its own factory,
ProblemClass.ADVERSARIALand the suite).Verified in Python:
ioh.get_problem(5001, 1, 1, ioh.ProblemClass.ADVERSARIAL),ioh.suite.AdversarialAttack, thedirect constructor
ioh.problem.AdversarialAttack("cifar10", eps=0.3, alpha=0.05), and the wholetests/pythonsuite.Known limitations / things I'd like reviewer input on:
n_variablesis accepted but ignored (the dimension is fixed per dataset), as for the graph problems and CEC2013;a hard error is the alternative.
instanceselects the base image and therefore changes the entire landscape, unlike most IOH problems whereinstanceis an affine transformation of one function.optimum()is left at its default.static/grows by about 11 MB andstatic.tar.gzfrom 7.1 MB to 17.8 MB (every wheel shipsstatic/).Bundling MNIST only and downloading CIFAR-10, or hosting the data elsewhere, are possible if that is too much.
BaseTest::SetUp(clutchlog) in my environment, also for the existing tests. I ran theC++ tests with an entrypoint with an empty
SetUp; please check them in CI.Additional information (e.g., reference of the problems, additional required packages.):
Reference: https://github.com/wz7475/amhe-attack ("Large-scale Testing Global Optimization Methods with Black-box
Adversarial Attacks"). No additional C++ dependencies.
After merging PR please consider citing the PPSN paper, which introduces this kind of approach