Repository navigation
fix(jmeter): keep JMeter master alive on cgroup v2 nodes to fix integration tests - #411
Merged
klemmster merged 2 commits intoNov 12, 2026
Conversation
The JMeter master image (hellofresh/kangal-jmeter-master, JDK 11) throws a NullPointerException from its cgroup v2 metrics detection when log4j2 re-registers its JMX MBeans. The exception is logged at ERROR level, and the master launcher script exits 1 on any ERROR line in the JMeter output, killing the load test a few seconds after the workers start ramping. The job retry hits the same NPE (and 'Engine is busy' on the workers), so the master job fails and the workers get torn down at roughly 16 of the expected 60 requests. This broke the JMeter integration test verification in CI once GitHub runners moved to cgroup v2 hosts, and is reproducible on any cgroup v2 cluster (kind, most modern clusters). Disable log4j2 JMX registration via JVM_ARGS on the master job to suppress the spurious ERROR line. Validated end-to-end against a kind cluster on a cgroup v2 host: busybox receives exactly 60 of 60 expected requests, master job succeeds, load test reaches finished. Co-Authored-By: OpenCode <noreply@opencode.ai>
klemmster
marked this pull request as ready for review
November 12, 2026 15:57
etiennechabert
approved these changes
Nov 12, 2026
klemmster
deleted the
fix/jmeter-master-jmx-disable-cgroupv2-integration-tests
branch
November 12, 2026 16:45
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Fixes the JMeter integration test verification that fails in CI:
Seen in https://github.com/hellofresh/kangal/actions/runs/43230416554/job/131257234249?pr=409 (and before that on 2026-07-31). The failure is unrelated to the PRs that hit it — it is environmental.
Root cause
Reproduced deterministically against a kind cluster (
kindest/node:v1.21.1, same image as CI) on a cgroup v2 host, which is what GitHub runners now use:The JMeter master image (
hellofresh/kangal-jmeter-master:latest, built 2022, JDK 11 + JMeter 5.5 + log4j2) throws aNullPointerExceptionfromjdk.internal.platform.cgroupv2.CgroupV2Subsystem.getInstancewhen log4j2 re-registers its JMX MBeans. Log4j2 logs it as:The master container's
launcher.shtreats any line containingERRORin the JMeter output as fatal andexit 1— even though this line is harmless.The master job dies ~10s in, while the workers are still ramping (2 workers × 30 threads over 30s ⇒ ~2 req/s ⇒ ~16 requests reach the target before teardown). With
BackoffLimit: 1a second master pod starts, hits the same NPE (andEngine is busy - cannot create JMeter engineon the workers), and fails too.determineLoadTestPhaseFromJobmaps the failed master job to phasefinished, soTestIntegrationJMeterpasses,t.Cleanupdeletes the LoadTest and the workers are torn down mid-ramp.verify_jmeter_integration_testinci/integration-tests.shthen counts only 16 of the 60 expectedresponse:200lines in the busybox logs ⇒ CI fails.This is why the tests were green until early 2026 (cgroup v1 runners) and red afterwards. Captured master pod logs from the repro:
Fix
Set
JVM_ARGS=-Dlog4j2.disable.jmx=trueon the JMeter master job so log4j2 never touches JMX and the spurious ERROR line is never emitted. Verified empirically that the master image produces zeroERRORlines with this flag.Validation
finished.make test-unitpasses; new unit testTestNewJMeterMasterJobEnvVarslocks in the env vars.Follow-ups (out of scope here)
hellofresh/kangal-jmeter:launcher.sh'sgrep "ERROR"kill-switch is far too broad — any benign ERROR-level line can abort a load test. The JDK 11 base image is also the underlying trigger and could do with a rebuild.determineLoadTestPhaseFromJobreports a failed master job asfinished; arguably it should beerrored. That is a behavior change for existing users, so it deserves its own PR/discussion.