Fix minion crash with RuntimeError: Event loop is closed on daemon retry - #70180
Open
twangboy wants to merge 3 commits into
Open
Fix minion crash with RuntimeError: Event loop is closed on daemon retry#70180twangboy wants to merge 3 commits into
twangboy wants to merge 3 commits into
Conversation
The minion logged loud "unclosed publish server", "unclosed SyncWrapper", and "unclosed publisher client" WARNING messages after losing its master connection with no working failover. Root cause was two gaps in MinionManager's teardown: - MinionManager.destroy() never closed event_publisher/event; only stop_async() (the SIGTERM path) did. - cli/daemons.py's Minion.start() never called shutdown()/destroy() when _real_start() raised SaltClientError because self.minion.restart was True -- it just returned, leaving event_publisher/event to be reclaimed only by __del__'s GC-time safety net. - salt/minion.py: MinionManager.destroy() now also closes event_publisher and destroys event, mirroring stop_async(). - salt/cli/daemons.py: Minion.start() now destroys the MinionManager on SaltClientError, both before retrying (daemonized multi-master failover) and before falling through to the final break/return. Fixes saltstack#70175
When salt-minion runs in classic daemon mode (-d) and loses its master connection with no working failover, MinionManager.tune_in() closes its io_loop and returns. cli.daemons.Minion.start()'s retry loop called _real_start() -> tune_in() again on the same, now-stale MinionManager, whose io_loop.run_forever() immediately raised RuntimeError: Event loop is closed, crashing the process. - salt/cli/daemons.py: Minion.start() now constructs a fresh MinionManager (mirroring the narrow construction step from prepare(), not prepare() itself, which would re-daemonize/re-fork) before retrying, instead of reusing the stale instance. - salt/minion.py: MinionManager.stop() now guards against a SIGTERM landing in the destroy()-to-reconstruct window and hitting a MinionManager whose loop is already closed; tune_in()'s final close() now checks is_closed() first, matching sibling Minion/ SyndicManager tune_in() methods. - salt/transport/tcp.py, salt/utils/event.py: destroy()'s own cleanup chain (PublishClient.close()/on_recv(), Subscriber.close(), PubServer.close(), RequestClient.close(), SaltEvent.close_pub()/ close_pull()) had several unguarded Task.cancel()/stream.close() calls that raised the same RuntimeError when invoked after their loop was already closed -- guard them the same way the zeromq transport already does, so destroy() can never itself crash. Fixes saltstack#70178
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
When
salt-minionruns in classic daemon mode (-d) and loses its master connection with no working failover,MinionManager.tune_in()closes itsio_loopand returns.cli.daemons.Minion.start()'s retry loop called_real_start()->tune_in()again on the same, now-staleMinionManager, whoseio_loop.run_forever()immediately raisedRuntimeError: Event loop is closed, crashing the process.salt/cli/daemons.py: Minion.start()now constructs a freshMinionManager(mirroring the narrow construction step fromprepare(), notprepare()itself, which would re-daemonize/re-fork) before retrying, instead of reusing the stale instance.salt/minion.py: MinionManager.stop()now guards against aSIGTERMlanding in thedestroy()-to-reconstruct window and hitting aMinionManagerwhose loop is already closed;tune_in()'s finalclose()now checksis_closed()first, matching sibling Minion/SyndicManagertune_in()methods.salt/transport/tcp.py,salt/utils/event.py: destroy()'s own cleanup chain (PublishClient.close()/on_recv(),Subscriber.close(),PubServer.close(),RequestClient.close(),SaltEvent.close_pub()/close_pull()) had several unguardedTask.cancel()/stream.close()calls that raised the sameRuntimeErrorwhen invoked after their loop was already closed -- guard them the same way the zeromq transport already does, sodestroy()can never itself crash.What issues does this PR fix or reference?
Fixes #70178
Merge requirements satisfied?
[NOTICE] Bug fixes or features added to Salt require tests.
Commits signed with GPG?
Yes