Skip to content

fix(jmeter): keep JMeter master alive on cgroup v2 nodes to fix integration tests - #411

Merged
klemmster merged 2 commits into
masterfrom
fix/jmeter-master-jmx-disable-cgroupv2-integration-tests
Nov 12, 2026
Merged

klemmster merged 2 commits into
masterfrom
fix/jmeter-master-jmx-disable-cgroupv2-integration-tests

Conversation

@klemmster

Copy link
Copy Markdown
Contributor

What

Fixes the JMeter integration test verification that fails in CI:

JMeter Integration Test sent 16 requests, but 60 requests were expected. Test failed.

Seen in https://github.com/hellofresh/kangal/actions/runs/43230416554/job/131257234249?pr=409 (and before that on 2026-07-31). The failure is unrelated to the PRs that hit it — it is environmental.

Root cause

Reproduced deterministically against a kind cluster (kindest/node:v1.21.1, same image as CI) on a cgroup v2 host, which is what GitHub runners now use:

  1. The JMeter master image (hellofresh/kangal-jmeter-master:latest, built 2022, JDK 11 + JMeter 5.5 + log4j2) throws a NullPointerException from jdk.internal.platform.cgroupv2.CgroupV2Subsystem.getInstance when log4j2 re-registers its JMX MBeans. Log4j2 logs it as:

    ERROR Could not reconfigure JMX java.lang.NullPointerException
    
  2. The master container's launcher.sh treats any line containing ERROR in the JMeter output as fatal and exit 1 — even though this line is harmless.

  3. The master job dies ~10s in, while the workers are still ramping (2 workers × 30 threads over 30s ⇒ ~2 req/s ⇒ ~16 requests reach the target before teardown). With BackoffLimit: 1 a second master pod starts, hits the same NPE (and Engine is busy - cannot create JMeter engine on the workers), and fails too.

  4. determineLoadTestPhaseFromJob maps the failed master job to phase finished, so TestIntegrationJMeter passes, t.Cleanup deletes the LoadTest and the workers are torn down mid-ramp.

  5. verify_jmeter_integration_test in ci/integration-tests.sh then counts only 16 of the 60 expected response:200 lines in the busybox logs ⇒ CI fails.

This is why the tests were green until early 2026 (cgroup v1 runners) and red afterwards. Captured master pod logs from the repro:

=== Starting test for /tests/testfile.jmx configuration ===
=== Running JMeter load generator ===
2026-11-12 15:27:37,692 main ERROR Could not reconfigure JMX java.lang.NullPointerException
	at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:694)
	...
	at java.base/jdk.internal.platform.cgroupv2.CgroupV2Subsystem.getInstance(Unknown Source)
=== We got an error while running JMeter, exiting 1 ===

Fix

Set JVM_ARGS=-Dlog4j2.disable.jmx=true on the JMeter master job so log4j2 never touches JMX and the spurious ERROR line is never emitted. Verified empirically that the master image produces zero ERROR lines with this flag.

Validation

  • Reproduced on a local kind cluster (cgroup v2 host): before the fix the load test sends 16/60 requests; after the fix exactly 60/60 land on the busybox target, the master job succeeds, and the load test reaches finished.
  • make test-unit passes; new unit test TestNewJMeterMasterJobEnvVars locks in the env vars.

Follow-ups (out of scope here)

  • The proper long-term fix belongs in hellofresh/kangal-jmeter: launcher.sh's grep "ERROR" kill-switch is far too broad — any benign ERROR-level line can abort a load test. The JDK 11 base image is also the underlying trigger and could do with a rebuild.
  • determineLoadTestPhaseFromJob reports a failed master job as finished; arguably it should be errored. That is a behavior change for existing users, so it deserves its own PR/discussion.

klemmster and others added 2 commits November 12, 2026 16:40
The JMeter master image (hellofresh/kangal-jmeter-master, JDK 11) throws
a NullPointerException from its cgroup v2 metrics detection when log4j2
re-registers its JMX MBeans. The exception is logged at ERROR level, and
the master launcher script exits 1 on any ERROR line in the JMeter
output, killing the load test a few seconds after the workers start
ramping. The job retry hits the same NPE (and 'Engine is busy' on the
workers), so the master job fails and the workers get torn down at
roughly 16 of the expected 60 requests.

This broke the JMeter integration test verification in CI once GitHub
runners moved to cgroup v2 hosts, and is reproducible on any cgroup v2
cluster (kind, most modern clusters).

Disable log4j2 JMX registration via JVM_ARGS on the master job to
suppress the spurious ERROR line. Validated end-to-end against a kind
cluster on a cgroup v2 host: busybox receives exactly 60 of 60 expected
requests, master job succeeds, load test reaches finished.

Co-Authored-By: OpenCode <noreply@opencode.ai>
@klemmster
klemmster marked this pull request as ready for review November 12, 2026 15:57
@klemmster
klemmster merged commit 8dfb272 into master Nov 12, 2026
5 of 6 checks passed
@klemmster
klemmster deleted the fix/jmeter-master-jmx-disable-cgroupv2-integration-tests branch November 12, 2026 16:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants