Silent permanent hang under free-threaded Python 3.14t: cross-lock race in LibevLoop._run_loop reading _live_conns

[mykaul]

Symptom

CI job "test libev (3.14t)" (free-threaded / no-GIL Python 3.14 build) on PR #910 hung for ~6 hours with zero output/traceback, then was cancelled:

• PR: fix: detect ScyllaDB via SUPPORTED protocol extensions, not shard count #910
• Run: https://github.com/scylladb/python-driver/actions/runs/31791129329/job/94738053735

The last thing that ran was tests/integration/standard/test_metadata.py::GroupPerHost::test_group_keys_by_host, which passed. The job log shows that test's PASSED line, then nothing at all until the run was force-cancelled ~6 hours later:

...test_metadata.py::GroupPerHost::test_group_keys_by_host PASSED [ 12%]
##[error]The operation was canceled.

That test's teardown (tearDownClass → drop_keyspace_shutdown_cluster()) calls Cluster.shutdown() right before moving on to test_metrics.py. Cluster.shutdown() joins its executor (executor.shutdown(wait=True)), which blocks forever if a connection's I/O never completes.

This is confirmed unrelated to PR #910's own diff — the same job passed on that branch in an earlier run, and separately passes on master.

Root cause analysis

cassandra/io/libevreactor.py, class LibevLoop:

• self._live_conns is a set that's rebound (never mutated in place) under self._conn_set_lock in connection_created() (line ~161-169) and connection_destroyed() (line ~171-185).
• _run_loop's exit check (line ~99-111) reads self._live_conns under a different lock, self._lock:

def _run_loop(self):
    while True:
        self._loop.start()
        # there are still active watchers, no deadlock
        with self._lock:
            if not self._shutdown and self._live_conns:
                log.debug("Restarting event loop")
                continue
            else:
                # all Connections have been closed, no active watchers
                log.debug("All Connections currently closed, event loop ended")
                self._started = False
                break

LibevLoop is a process-global singleton (_global_loop in the same module), shared by every Cluster/connection created in the process. maybe_start() only spawns a new reactor thread when self._started is False; a freshly created connection's constructor calls connection_created() then maybe_start() exactly once, with no retry.

Sequence that produces a permanent lost wakeup:

1. Reactor thread finishes an iteration of self._loop.start() and is about to evaluate the exit condition.
2. Concurrently, another thread creates a new connection: connection_created() rebinds self._live_conns to a new set containing it (under _conn_set_lock).
3. The reactor thread's read of self._live_conns (under self._lock, not _conn_set_lock) does not observe the new connection yet, decides the loop is idle, sets self._started = False, and the thread exits.
4. The new connection's maybe_start() call (also under self._lock) happens to observe self._started == True from just before the reactor's transition, so it does nothing — assuming an already-running thread will pick up the new connection.
5. Nobody restarts the reactor. The new connection's read/write watchers are never serviced. Anything that blocks on that connection's I/O — including a subsequent Cluster.shutdown() → executor.shutdown(wait=True) — hangs forever with no error and no traceback.

Under the GIL, self._lock's two critical sections (the _started check in maybe_start() and the exit-check+_started=False in _run_loop) are already mutually exclusive, so in practice the specific bad interleaving above is rare — a connection is almost always created "during" an already-decided restart, or a nearby connection's own maybe_start() call ends up "retrying" the wakeup. It's a genuine pre-existing race, not a free-threading-only bug, but the GIL's coarse serialization between Python bytecodes made the bad window rare in practice. On free-threaded (3.14t, no GIL) builds, true parallel execution removes that incidental protection, making the window dramatically easier to hit — consistent with this only having ever been observed on the 3.14t job.

Proposed fix

Read _live_conns under the same lock it's written under, so the exit decision can't race a concurrent connection_created():

--- a/cassandra/io/libevreactor.py
+++ b/cassandra/io/libevreactor.py
@@ -101,7 +101,12 @@ class LibevLoop(object):
-                if not self._shutdown and self._live_conns:
+                # _live_conns is written under _conn_set_lock; read it under
+                # the same lock so this exit decision can't race a concurrent
+                # connection_created() (safe under the GIL, not free-threaded).
+                with self._conn_set_lock:
+                    live_conns = bool(self._live_conns)
+                if not self._shutdown and live_conns:
                     log.debug("Restarting event loop")
                     continue
                 else:

(Line numbers are current as of master at the time of filing; please re-verify before applying, since surrounding code may have shifted.)

Reproduction status

I wrote a standalone stress harness (not committed) that drives the real, unmodified LibevLoop class directly: it installs a fake cassandra.io.libevwrapper module (the C libev extension isn't built in the sandbox this was tested in) providing no-op Loop/Async/Prepare/Timer/IO stand-ins, so LibevLoop.__init__, connection_created, connection_destroyed, maybe_start, and _run_loop all run as real production code against fake (hashable, no-op) connection objects. FakeLoop.start() returns immediately, so the background _run_loop thread spins as fast as possible re-checking its exit condition, maximizing race attempts per second.

Each trial: spin up N worker threads hammering connection_created/maybe_start/connection_destroyed for a short burst, stop them, then create one final, unaccompanied connection (nothing follows it to "retry" a stuck wakeup — mirroring the last connection settling right before a Cluster.shutdown()), call maybe_start(), settle briefly, and check whether that connection is still registered in _live_conns while the reactor thread is not alive / _started is False (a genuine permanent lost wakeup).

• Obtained a free-threaded interpreter via uv python install 3.14t (cpython-3.14.6+freethreaded, confirmed sys._is_gil_enabled() == False).
• Ran the harness as a control under the regular GIL 3.14.6 build (300 trials, 12 worker threads each) and under the free-threaded 3.14t build with PYTHON_GIL=0 (800 trials total across two runs: 300 workers=12, then 500 workers=32, to push contention harder).
• Result: no reproduction in either configuration (0/300 stuck on GIL 3.14.6, 0/800 stuck on free-threaded 3.14t). The harness never observed the target connection left stranded (registered in _live_conns with the reactor thread not alive / _started == False).

I don't want to overstate this: it's a negative result on a synthetic proxy, not a disproof of the race. The real CI hang involved real sockets, real GC pressure, and a much longer-running, more heavily loaded process (a full pytest suite) before the moment it happened — conditions this lightweight harness (fake connections, no real I/O, short bursts) doesn't reproduce. The harness may simply not be hitting the precise instruction-level interleaving needed, even with _loop.start() faked to return instantly to keep the exit-check hot. I'm filing this issue regardless of the reproduction outcome, since the static analysis of the lock mismatch is solid and matches the observed hang's timing/location exactly, per repo owner's request.

Fix status: I drafted the fix above and, since it's a minimal, obviously-safe change (read under the lock the writer already uses), committed it locally on a scratch branch for verification purposes; it has NOT been pushed or opened as a PR pending review here.

Related

#717 — "Segfault in free-threaded Python 3.14t during cluster shutdown (logging race in Cythonized cluster.so)". Same neighborhood (Cluster.shutdown() exposing GIL-dependent assumptions under 3.14t), but a different mechanism: #717 is a crash from formatting a concurrently-GC'd Host object during shutdown logging, whereas this issue is a cross-lock read/write race in the libev reactor's connection bookkeeping causing a silent, permanent hang (no crash, no traceback).