Skip to content

connectd: don't stall peer reads when lightningd attaches no subd - #9370

Open
morehouse wants to merge 1 commit into
ElementsProject:masterfrom
morehouse:handle_channeld_race
Open

connectd: don't stall peer reads when lightningd attaches no subd#9370
morehouse wants to merge 1 commit into
ElementsProject:masterfrom
morehouse:handle_channeld_race

Conversation

@morehouse

@morehouse morehouse commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

When connectd reads a peer message whose channel_id has no attached subd, it creates a subd with conn == NULL and sends connectd_peer_spoke to lightningd, asking it to start up a subdaemon and respond with connectd_peer_connect_subd and the file descriptor that should be assigned to conn. While connectd is waiting for lightningd's response, it stops reading all messages from the peer.

There were six cases in lightningd's handle_peer_spoke where lightningd would fail to respond, so when those occurred connectd would end up waiting indefinitely while never reading or responding to the peer's messages. Essentially the connection would become a "zombie" until either the next gossip flush (if any) or the ping timeout caused the connection to be dropped.

Add a new connectd_peer_no_subd message for lightningd to send connectd when it will not be creating a subd, so connectd can then free the subd structure and continue servicing the connection as usual.

Notably the connectd_peer_no_subd message uses a new spoke_id to uniquely identify the connectd_peer_spoke message that it is responding to, since multiple connectd_peer_spoke messages may be in flight at a time and because the channel_id (which is the more natural choice) can be changed by connectd before lightningd's response is received, thereby causing the wrong subd to be freed.

Addresses all 6 cases from #9369, though there is special handling for case 4, which does not need a response if the peer_connected hook will cause the subdaemon to be created shortly. There may still be some corner case where this can lead to a zombie connection, but I was unable to find a way to trigger it.

A regression test is also added that demonstrates the connection getting zombified after data loss recovery (case 1). This was the best candidate for a test since it was both simple and deterministically repoducible. Case 2 was the one detected by smite, and I verified that smite can no longer reproduce the bug with this PR applied. Any tests for cases 2, 3, or 4 would be nondeterministic since they need to trigger specific race conditions. Cases 5 and 6 are extreme corner cases that can not be easily tested with the existing tooling (they would require FD exhaustion).

Fixes: #9369

@morehouse
morehouse force-pushed the handle_channeld_race branch 2 times, most recently from 26aeffe to 4e35018 Compare August 3, 2026 15:29
@morehouse
morehouse marked this pull request as draft August 3, 2026 15:45
@morehouse

Copy link
Copy Markdown
Contributor Author

Converted to draft, as I'm not sure if this is the right approach yet. I'm still working through some issues with this change.

@morehouse
morehouse force-pushed the handle_channeld_race branch from 4e35018 to 451eaf4 Compare August 3, 2026 19:17
@morehouse
morehouse marked this pull request as ready for review August 3, 2026 22:01
When connectd reads a peer message whose channel_id has no attached
subd, it creates a subd with conn == NULL and sends connectd_peer_spoke
to lightningd, asking it to start up a subdaemon and respond with
connectd_peer_connect_subd and the file descriptor that should be
assigned to conn.  While connectd is waiting for lightningd's response,
it stops reading *all* messages from the peer.

There were six cases in lightningd's handle_peer_spoke where lightningd
would fail to respond, so when those occurred connectd would end up
waiting indefinitely while never reading or responding to the peer's
messages.  Essentially the connection would become a "zombie" until
either the next gossip flush (if any) or the ping timeout caused the
connection to be dropped.

Add a new connectd_peer_no_subd message for lightningd to send connectd
when it will not be creating a subdaemon, so connectd can then remove
the subd and continue servicing the connection as usual.

Notably the connectd_peer_no_subd message uses a new spoke_id to
uniquely identify the connectd_peer_spoke message that it is responding
to, since multiple connectd_peer_spoke messages may be in flight at a
time and because the channel_id (which is the more natural choice) can
be changed by connectd before lightningd's response is received, thereby
causing the wrong subd to be freed.

Fixes: ElementsProject#9369
Changelog-Fixed: connectd: no longer stop servicing a peer connection for ~80s when a channel fails.
@morehouse
morehouse force-pushed the handle_channeld_race branch from 451eaf4 to c6648c3 Compare August 3, 2026 22:03
@morehouse

Copy link
Copy Markdown
Contributor Author

AFAICT all related CI failures have been addressed. test_emergencyrecover had started failing because it hit case 4 of #9369 and we were freeing the subd even though the peer_connected hook was still running and would have created the subdaemon within a few ms. I've added an exception for this case -- if the hook is still running then lightningd returns without sending connectd_peer_no_subd so that the hook can take care of attaching the subdaemon.

This is ready for review now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Zombie connection after lightningd ignores connectd_peer_spoke

1 participant