[CI] Integration tests failed in nightly CI

[christophfroehlich]

Describe the bug
unsure what could be the root cause

2026-04-24T04:50:28.9859837Z     FAIL: test_arm (robot_launch_test.TestFixture.test_arm)
2026-04-24T04:50:28.9860002Z     ----------------------------------------------------------------------
2026-04-24T04:50:28.9860086Z     Traceback (most recent call last):
2026-04-24T04:50:28.9860627Z       File "/home/runner/work/ros2_control_ci/ros2_control_ci/.work/target_ws/src/ros-controls/mujoco_ros2_control/mujoco_ros2_control_tests/test/robot_launch_test.py", line 239, in test_arm
2026-04-24T04:50:28.9860843Z         self.wait_for_joint_positions({"joint1": 0.5, "joint2": -0.5}, delta=0.05, timeout=15.0)
2026-04-24T04:50:28.9861402Z       File "/home/runner/work/ros2_control_ci/ros2_control_ci/.work/target_ws/src/ros-controls/mujoco_ros2_control/mujoco_ros2_control_tests/test/robot_launch_test.py", line 167, in wait_for_joint_positions
2026-04-24T04:50:28.9861469Z         self.fail(
2026-04-24T04:50:28.9861927Z     AssertionError: joints did not reach the commanded position in 15.0s. expected: {'joint1': 0.5, 'joint2': -0.5}, checked: {'joint1': 1.5732284405525438, 'joint2': -6.3227644161271614}
2026-04-24T04:50:28.9861997Z     
2026-04-24T04:50:28.9862092Z     ======================================================================
2026-04-24T04:50:28.9862246Z     FAIL: test_gripper (robot_launch_test.TestFixture.test_gripper)
2026-04-24T04:50:28.9862380Z     ----------------------------------------------------------------------
2026-04-24T04:50:28.9862464Z     Traceback (most recent call last):
2026-04-24T04:50:28.9862987Z       File "/home/runner/work/ros2_control_ci/ros2_control_ci/.work/target_ws/src/ros-controls/mujoco_ros2_control/mujoco_ros2_control_tests/test/robot_launch_test.py", line 258, in test_gripper
2026-04-24T04:50:28.9863105Z         self.check_controllers_running_with_retry(cnames)
2026-04-24T04:50:28.9863718Z       File "/home/runner/work/ros2_control_ci/ros2_control_ci/.work/target_ws/src/ros-controls/mujoco_ros2_control/mujoco_ros2_control_tests/test/robot_launch_test.py", line 139, in check_controllers_running_with_retry
2026-04-24T04:50:28.9863933Z         self.fail(f"Controllers not running after {timeout}s: {controller_names}")
2026-04-24T04:50:28.9864263Z     AssertionError: Controllers not running after 15.0s: ['gripper_controller', 'position_controller', 'joint_state_broadcaster']
2026-04-24T04:50:28.9864332Z     
2026-04-24T04:50:28.9864430Z     ======================================================================
2026-04-24T04:50:28.9864645Z     FAIL: test_reset_world_service (robot_launch_test.TestFixture.test_reset_world_service)
2026-04-24T04:50:28.9864808Z     Test that the reset_world service resets robot to initial position.
2026-04-24T04:50:28.9864936Z     ----------------------------------------------------------------------
2026-04-24T04:50:28.9865018Z     Traceback (most recent call last):
2026-04-24T04:50:28.9865576Z       File "/home/runner/work/ros2_control_ci/ros2_control_ci/.work/target_ws/src/ros-controls/mujoco_ros2_control/mujoco_ros2_control_tests/test/robot_launch_test.py", line 309, in test_reset_world_service
2026-04-24T04:50:28.9865697Z         self.check_controllers_running_with_retry(cnames)
2026-04-24T04:50:28.9866281Z       File "/home/runner/work/ros2_control_ci/ros2_control_ci/.work/target_ws/src/ros-controls/mujoco_ros2_control/mujoco_ros2_control_tests/test/robot_launch_test.py", line 139, in check_controllers_running_with_retry
2026-04-24T04:50:28.9866459Z         self.fail(f"Controllers not running after {timeout}s: {controller_names}")
2026-04-24T04:50:28.9866793Z     AssertionError: Controllers not running after 15.0s: ['gripper_controller', 'position_controller', 'joint_state_broadcaster']
2026-04-24T04:50:28.9866853Z     
2026-04-24T04:50:28.9866982Z     ----------------------------------------------------------------------
2026-04-24T04:50:28.9867065Z     Ran 12 tests in 65.061s

To Reproduce
ros-controls/ros2_control_ci#757
ros-controls/ros2_control_ci#756

Environment (please complete the following information):

• OS: [e.g. Ubuntu]
• Version Rolling+Kilted

[eholum-nasa]

I'm hoping that reducing scope of the functional tests in #204 will help with this, as I'm thinking that the runners are just overloaded by work and the simulation isn't progressing as it should...

[omnilink-tech]

The numbers in the original paste look inconsistent with the overloaded-runner
explanation, and I think there may be two different failures filed here.

test_arm: the reported joint2 is outside what the simulation can produce

expected: {'joint1': 0.5, 'joint2': -0.5}
checked:  {'joint1': 1.5732284405525438, 'joint2': -6.3227644161271614}

joint2 is declared revolute with lower="-3.14" upper="3.14" in
test_robot.urdf, and test_robot.xml carries range="-3.14 3.14". Loading
that MJCF directly (mujoco 3.11.0) confirms the range is actually enforced --
autolimits defaults on, so limited comes back 1 even though the file never
writes it:

joint1                       limited=1  range=[-3.1400, 3.1400]
joint2                       limited=1  range=[-3.1400, 3.1400]
gripper_left_finger_joint    limited=1  range=[-0.0450, 0.0050]
gripper_right_finger_joint   limited=1  range=[-0.0050, 0.0450]

So -6.3228 is 3.1828 rad outside the range the solver is enforcing. MuJoCo
cannot have put the joint there.

That matters for the diagnosis: if the simulation were merely failing to
progress, the reading would be a legal in-range value close to where the joint
started. An out-of-range value says the number did not come from the simulated
joint angle at all -- it came from somewhere further down the state pipeline.

joint1 = 1.5732 is in range, so only one of the two is impossible.

A candidate, offered as arithmetic rather than a conclusion

test_robot.urdf couples the two joints through a DifferentialTransmission
with reductions 2.0 / 0.5, and the test's own forward map is

actuator1 = joint1*jr1 + joint2*jr2
actuator2 = joint1*jr1 - joint2*jr2

Inverting that, joint2 = (a1 - a2) / (2*jr2) with jr2 = 0.5, so joint2 in
this space spans +/-6.28 when the two actuators sit at opposite ends of their
in-limit travel. The observed -6.3228 sits essentially at that bound, which
is at least suggestive of joint-space and actuator-space values being mixed
somewhere rather than of a physics problem. Treating the reported pair as
actuator-space values and inverting gives a1 = -0.0149, a2 = +6.3078
(2*pi = 6.2832), where the same run should have produced a1 = 0.75,
a2 = 1.25 at the commanded targets.

I have not reproduced your CI, so I would not put weight on that reading -- the
part I would stand behind is the paragraph above it: the value is outside the
enforced range, so it is not a simulated joint angle.

The second failure has a different signature

AssertionError: Controllers not running after 15.0s:
  ['gripper_controller', 'position_controller', 'joint_state_broadcaster']

Controllers never reaching active is consistent with @eholum-nasa's
overloaded-runner hypothesis, and with #204 quarantining the flake. But it is a
startup/timing signature, whereas the test_arm one is a state-value signature.
Worth separating -- otherwise a fix for one will look like it flaked away rather
than fixed anything.

One small change that would settle it from the log alone

wait_for_joint_positions budgets its timeout on time.time()
(robot_launch_test.py:134-135), while the simulation publishes /clock. On a
loaded runner those two diverge, so the failure message cannot currently
distinguish "15 wall-seconds elapsed but the sim advanced 2 seconds" from "the
sim advanced 15 seconds and the joint went somewhere wrong".

Adding two things to that message would make every future instance
self-diagnosing:

1. sim-time elapsed (from /clock, which the fixture already subscribes to at
   :229) alongside the wall-clock timeout, and
2. a flag when a reported position lies outside the joint's declared limit --
   which in this instance would have pointed at the state pipeline immediately
   instead of at the runners.

Happy to send that as a PR if it is useful. It touches only the test helper.

Context: we run MuJoCo under a hardware_interface::SystemInterface on our own
simulator and have been bitten by the wall-clock/sim-clock split in a way that
was invisible from the logs -- a diff_drive_controller whose cmd_vel_timeout
is evaluated on the simulation clock silently expired between messages from a
20 Hz wall-clock publisher, the robot moved 0.0000 m, and every layer reported
success. That is why the clock separation above is the first thing I would
instrument here, even though I do not think it is the cause of the out-of-range
reading.