Skip to content

[master] Merge forward from 3008.x - #70191

Open
dwoz wants to merge 494 commits into
saltstack:masterfrom
dwoz:merge/3008.x/master-08-28-26
Open

[master] Merge forward from 3008.x#70191
dwoz wants to merge 494 commits into
saltstack:masterfrom
dwoz:merge/3008.x/master-08-28-26

Conversation

@dwoz

@dwoz dwoz commented Aug 30, 2026

Copy link
Copy Markdown
Contributor

Summary

Merge-forward from 3008.x into master. Standard cadence.

  • 493 commits from 3008.x
  • 13 files had merge conflicts, all resolved per-file with cross-branch semantic checks
  • Silent-drop audit ran across all 781 diverging files: zero silent SOURCE-content regressions (contrast to the recent 3007.x→3008.x merge where the x509_v2 cluster silently reverted to DEST content)
  • SRP-managed pins carried forward intact (cryptography 50.0.1, gitpython 3.1.60, msgpack 1.2.1, pyasn1 0.6.4, pyopenssl 26.4.0, requests 2.34.2, attrs 26.1.0, charset-normalizer 3.5.1)

Notable resolutions

  • .github/workflows/templates/layout.yml.jinja — kept master's hash-files.py infra rework (post-3008.x), restored shell: bash, removed dead jinja var
  • Generated workflow YAMLs regenerated from the resolved template via pre-commit
  • .pre-commit-config.yaml — took 3008.x's platform-split lint hook aliases (compile-ci-<plat>-lint-<pyver>-requirements)
  • pkg/macos/install_salt.sh — carried 3008.x's LDFLAGS=-Wl,-undefined,dynamic_lookup fix (relenv 0.22.25 / py3.14 sysconfig workaround) and the PIP_LOG=$(mktemp) error-surfacing block
  • requirements/base.txt — max(master, 3008.x) floors for every SRP-controlled package; dropped master's py3.13 msgpack carve-out (1.2.1 works there)
  • tests/pytests/pkg/integration/test_version.py — kept master's PR test_version: assert current_release() contract, not a hardcoded codename #70168 rewrite (_DARWIN_PKG_SYMLINK_TO_BINKEY lookup)
  • tests/pytests/unit/{cli/test_batch,grains/test_core}.py — union of both sides' independent test additions

Test plan

  • All CI (test:full)
  • Silent-drop audit report at agents/reports/silent_drop_audit_3008_to_master.md (in the branch) — reviewer sanity check

charzl and others added 30 commits August 10, 2026 17:10
The "short" unit covers both inode counts (Minion Inodes) and FD/process
counts (Master & API Resource Usage), which had been collapsed into the
same generic "count" label. Label each by what it actually counts instead.
``PubServer.publish_payload`` awaited each subscriber's write future
sequentially. A single slow subscriber (kernel TCP send buffer full)
made ``await future`` never resolve; every subsequent publish_payload
piled up more coroutines all blocked on the same subscriber, wedging
EventPublisher's io_loop. EP stopped draining
``master_event_pull.ipc``, MWorker's ``fire_event -> stream.write``
blocked in the kernel, SyncWrapper's ``thread.join()`` never
returned, MWorkers deadlocked, MWQ's DEALER ``send()`` blocked,
cascading to minion request timeouts and TCP churn.

Move writes to fire-and-forget: ``asyncio.ensure_future`` per
subscriber, wrapping the write future in
``asyncio.wait_for(..., timeout=publish_drain_timeout)``.  On timeout
or ``StreamClosedError``, remove the subscriber from ``self.clients``
and close its stream via new ``_discard_slow_client`` helper.

New config: ``publish_drain_timeout: 5.0`` on master.

Fixes saltstack#69988
``PubServer.publish_payload`` awaited each subscriber's write future
sequentially (via ``yield future``). A single slow subscriber
(kernel TCP send buffer full) made ``yield future`` never resolve;
every subsequent publish_payload piled up more coroutines all
blocked on the same subscriber, wedging EventPublisher's io_loop.
EP stopped draining ``master_event_pull.ipc``, MWorker's
``fire_event -> stream.write`` blocked in the kernel, SyncWrapper's
``thread.join()`` never returned, MWorkers deadlocked, MWQ's
DEALER ``send()`` blocked, cascading to minion request timeouts
and TCP churn.

Move writes to fire-and-forget via ``io_loop.spawn_callback``. Each
drain coroutine wraps the write future in
``salt.ext.tornado.gen.with_timeout(io_loop.time() + drain_timeout, future)``.
On timeout, ``StreamClosedError``, or any other exception, remove
the subscriber from ``self.clients`` and close its stream via new
``_discard_slow_client`` helper. Complements the existing
concurrent-write commit ``73c6970351b``.

New config: ``publish_drain_timeout: 5.0`` on master.

3006.x-specific counterpart to 3008.x PR saltstack#69995 (which uses
``asyncio.ensure_future`` instead).

Fixes saltstack#69988
The 5s per-subscriber drain timeout was too aggressive for the
existing ``test_issue_36469_tcp`` regression test, which pushes
20x750KB payloads through a Python collector.  On slower CI VMs
the collector took >5s to drain a single write, tripping
``_discard_slow_client`` and disconnecting the subscriber
mid-stream.  The test then hung in ``__exit__`` broadcasting a
stop sentinel to a peerless publisher and pytest-timeout killed
it at 90s.

Bumping the default to 60s preserves the wedge-recovery
guarantee (a truly stuck subscriber is still evicted within a
minute) without falsely killing subscribers that are alive but
slow to drain large payloads.  Verified locally: the test now
completes in ~24s.
Since 3006.26 (b45b721) the loader recorded a failed __virtual__() under
the module's __virtualname__ in missing_modules.  When two files share a
virtualname (e.g. deb_postgres.py and postgres.py both use "postgres")
and the failing one was processed first due to non-deterministic
directory iteration, the virtualname was marked missing and the real
module was skipped on subsequent lookups, breaking postgres_user/state
runs on RHEL/Rocky.

Stop reassigning module_name to virtualname on failure in
_process_virtual, so missing_modules is only keyed by the actual file
basename.  Track per-virtualname failure reasons in a new
missing_virtualnames mapping consulted by missing_fun_string(), so
collision error surfacing (issue saltstack#68625) is preserved without the
poisoning race.

Fixes saltstack#69806
Under sustained load a busy MWorker rebuilds the ``cryptography`` +
libcrypto RSA state on every public-key operation. ``memray`` on a
stressed master showed thousands of ``RSAX931Verifier.__init__``
calls per minute against a matching ``PublicKey.decrypt`` count.
Same pattern on the sign side.

Three layers of caching (mirrors 3008.x PR saltstack#69996):
1. Lazy per-instance ``_verifier`` / ``_signer`` on
   ``PublicKey`` / ``PrivateKey``. ``self.key`` is immutable after
   ``__init__``, so the derived libcrypto bridge can be reused for
   the instance's lifetime.
2. Path-level cache on ``PublicKey.from_file`` keyed on
   ``(path, mtime)``. Key rotation on disk bumps mtime and
   invalidates the cache automatically.
3. Retry-on-verify-fail in ``PublicKey.verify`` / ``.decrypt``.
   Preserves the pre-cache 'always fresh' behavior for edge cases
   where a rotation preserves mtime (``cp -p``, some NFS setups).

Orthogonal to the existing ``_get_key_with_evict`` memoize (which
caches at the private-key file-loading layer).

Fixes saltstack#69989
test_verify_signature was calling verify_signature() with a fake path
`/keydir/keyname.pub` and patching fopen. PublicKey.from_file now takes
os.path.getmtime(path) for the cache key, which raises FileNotFoundError
on a fake path. Stub the mtime lookup and clear the pub-key cache so the
mocked fopen is actually consulted.

test_when_async_req_channel_with_syndic_role_... patched
`salt.crypt.PublicKey` and asserted the class was invoked with the
syndic master pubkey path. verify_signature now calls
`PublicKey.from_file(path)` so the path lands on the from_file
classmethod call rather than on the class itself.
``RequestClient.close()`` was dropping a ``(None, None)`` sentinel and
immediately closing ``self.socket`` + destroying ``self.context``. The
running ``send_recv_task`` coroutine was still on the io_loop with its
locals holding a reference to the socket; the close raced the task's
finally, leaving the underlying socketpair + mailbox FDs unfreed.
Under sustained ``saltutil.refresh_pillar`` / re-auth churn this
leaked ~451 socketpairs (~902 FDs) per minion, tripping the 1024-FD
ulimit throttle at ~924/1024 within minutes.

Port the graceful-drain pattern from ``AsyncReqMessageClient`` (from
twangboy's saltstack#68637 chain: ``6ad90a51d65``, ``aa317c67e51``,
``bcb3778c9c1``, ``b96ddd58ce6``): add ``_send_recv_exit_future``
that ``_send_recv`` sets in a try/finally, then rewrite ``close()`` to
schedule an async ``_drain_and_close`` task that awaits the future
(with 5s timeout) before closing the socket + destroying the context.

Also add ``SyncWrapper.__del__`` that emits ``ResourceWarning`` if the
wrapper was GC'd without an explicit ``close()`` -- mirrors
``SaltEvent.__del__`` (``salt/utils/event.py:278-321``) and surfaces
future missed-close bugs in tests and sentry rather than silently
leaking loops.

Validated in a 10-min stress bench: median FD/minion dropped from
924 (throttle threshold) to 35 (idle baseline); zero minions crossed
500 FDs across 50 stressed minions.

Fixes saltstack#69991
The graceful-drain patch scheduled ``_drain_and_close`` on the io_loop
via ``call_soon_threadsafe`` and returned immediately.  When the caller
was already on the loop thread (async production code, or a sync
fixture-teardown after an async test) and did not yield control to the
loop before it was torn down, the drain task was destroyed while
pending -- socket + context never got closed, and the ``zmq.Context``
finalizer later blocked in ``__del__`` -> ``term()`` under GC.  All 20
functional zeromq CI jobs hung until the 3h workflow timeout after the
first ``test_request_client_send_recv_socket_closed`` finished cleanly
and pytest tried to run the next test (repro: 2h45m gap between last
PASSED and cancellation in Debian 11 job 93340807019, py-spy dump
showed the main thread stuck in ``zmq.sugar.context.term``).

Split the teardown into three branches:

1. Same-thread + loop-running: the shutdown sentinel is already queued;
   fall through to the sync teardown so socket/context close
   deterministically before we return.  ``_send_recv`` picks up the
   sentinel on the next iteration and drops its socket ref, matching
   base-branch behavior that the ``send_recv_socket_closed`` test
   asserts on.
2. Cross-thread + loop-running: schedule the drain and block on a
   ``threading.Event`` (6s cap, matching the 5s drain timeout) so the
   caller doesn't tear down its loop while our task is still pending.
3. Loop not running: sync teardown directly (unchanged fallback).

Restores test_request_client.py to all-passing on the functional
transport suite (16 passed + 1 xfailed, was hanging indefinitely after
the third test).
``OptsDict.__getitem__`` was allocating a fresh ``DictProxy`` or
``ListProxy`` on every read of a mutable value.  Master hot paths
(``opts["file_roots"]``, ``opts["pillar_roots"]``) called this
thousands of times per minute, causing continuous object allocation +
GC churn.

Add a per-instance ``_proxy_cache: dict[str, tuple[Any, int]]`` keyed
on the config key, storing ``(proxy, id(underlying))``. Return the
cached proxy if the underlying object hasn't been replaced; otherwise
rebuild. Invalidate on ``__setitem__`` / ``__delitem__`` (pop key
from cache).

Fixes saltstack#69990
…_file

Under sustained load a busy MWorker rebuilds cryptography + libcrypto RSA
state on every public-key operation. memray on a stressed 3008.x master
showed ~5,000 RSAX931Verifier.__init__ calls per 60 seconds against a
matching PublicKey.decrypt call count. Same pattern on the sign side.

Three layers of caching:

1. Lazy per-instance _verifier / _signer on PublicKey / PrivateKey. self.key
   is immutable after __init__, so the derived libcrypto bridge object can
   be reused for the lifetime of the instance.
2. Path-level cache on PublicKey.from_file keyed on (path, mtime).  A key
   rotation on disk bumps mtime and invalidates the cache automatically.
3. Retry-on-verify-fail in PublicKey.verify / .decrypt.  Preserves the
   pre-cache "always fresh" behavior for edge cases where a rotation
   preserves mtime (cp -p, NFS mtime cache, atomic rename with preserved
   timestamps).  On the first failure the cache entry is evicted and one
   reload-and-retry is attempted.  Genuine bad signatures still return
   False / raise ValueError; the retry costs one extra file read + PEM
   parse per forged attempt.

Fixes saltstack#69940
Sibling entry to the cherry-picked fix; complements the file-layer
mtime-eviction fix (saltstack#69941) already on 3008.x.
test_verify_signature was calling verify_signature() with a fake path
`/keydir/keyname.pub` and patching fopen. PublicKey.from_file now takes
os.path.getmtime(path) for the cache key, which raises FileNotFoundError
on a fake path. Stub the mtime lookup and clear the pub-key cache so the
mocked fopen is actually consulted.

test_when_async_req_channel_with_syndic_role_... patched
`salt.crypt.PublicKey` and asserted the class was invoked with the
syndic master pubkey path. verify_signature now calls
`PublicKey.from_file(path)` so the path lands on the from_file
classmethod call rather than on the class itself.

(cherry picked from commit 61f55fc)
test_verify_retries_after_rotation_without_mtime_bump was calling
PrivateKey.sign() with the default PKCS1v15-SHA1 algorithm, which is
rejected at the salt boundary in FIPS mode.  Parameterize the test on
FIPS_TESTRUN so it uses PKCS1v15-SHA224 under FIPS and still exercises
the same retry-on-verify-fail code path on both toolchains.
An empty 'grains:' config option parses to None instead of a dict,
which crashed the minion during startup with "TypeError: 'NoneType'
object is not iterable" when the loader tried to build the
__grains__ NamespacedDictWrapper.

Default the option to an empty dict in both places that read it:
apply_minion_config, and salt.loader.grains(), which independently
re-reads the raw config file off disk. Log a warning in each case
pointing out that 'grains: {}' should be used instead.

Fixes saltstack#61321
Only warn/default the 'grains' config option when it is explicitly
present and None, not merely absent from opts. The looser
opts.get("grains") is None check could not distinguish "grains: set
to empty" from "grains key never set at all" (e.g. a sparse defaults
dict passed into apply_minion_config without a 'grains' key), causing
a false-positive warning in that case. This matches the equivalent
check already used in salt.loader.grains().

Also move log_file = factory.config["log_file"] in the new
integration test out of the try block, since it doesn't depend on the
test.ping call succeeding and reads more clearly next to the log file
assertions it's used for.
Broaden the empty-grains guards in apply_minion_config and
salt.loader.grains() from an explicit None check to
isinstance(value, dict), so any non-mapping value (empty string,
list, scalar, ...) is defaulted to an empty dict, not just an
explicitly empty 'grains:' key. Drop the runtime warning in favor of
documenting the required shape in conf/minion and
doc/ref/configuration/minion.rst.

Parametrize the existing tests over a range of non-dict grains values
instead of just None, and drop the now-irrelevant log/warning
assertions.
Keep the script available for background PowerShell/cmd/POSIX runs
via a self-cleaning wrapper, then remove it after exit. Refs saltstack#69959 saltstack#50273
Do not exec the real script from the /bin/sh wrapper; exec replaces the
shell and skips the EXIT trap, leaving the tempfile behind. Refs saltstack#69959 saltstack#50273
Use salt.utils.files.fopen in cmdmod bg wrapper unit tests
Tornado's HTTPClient enforces a default max_buffer_size of 100MiB
independently of max_body_size. When a server doesn't send a
Content-Length header (as some winrepo_ng HTTP servers don't), Salt
read the response until the connection closed and silently truncated
downloads over 100MiB instead of raising an error. Pass max_buffer_size
alongside max_body_size so both track http_max_body.

Also harden fileclient.get_url() to compare bytes received against any
advertised Content-Length and raise MinionError on mismatch instead of
caching a partial file, and fix the requests backend to stream via
iter_content() and catch RequestException so connection failures
surface as clean errors instead of unhandled exceptions.

Fixes saltstack#69916
SaltEvent.fire_event() re-raised send failures without resetting
self.pusher/self.cpush, so once an MWorker's IPC pusher stream broke
(e.g. a stale epoll fd after EventPublisher restarts), every
subsequent job return on that worker hit the same exception forever,
silently dropping the return before it reached the job cache and
burning memory/CPU on repeated thread+IOLoop churn. Close the pusher
on failure, mirroring the existing reconnect pattern on the subscribe
side, so the next fire_event() call reconnects instead.
dwoz requested a functional/integration test on PR saltstack#69937 since the
existing unit test only exercised a mocked pusher. Add a test that
spins up a real EventPublisher and a real SaltEvent pusher, fakes a
send() failure at the IPCMessageClient boundary to reproduce the
reported FileNotFoundError deterministically, and asserts the pusher
is dropped and a subsequent fire_event() reconnects and actually
delivers the event to a live listener.
…altstack#65088)

subproxy_post_master_init builds each sub-proxy's opts with a shallow
opts.copy(), so proxyopts["schedule"] and proxyopts["beacons"] were the same
dict objects as the control minion's. The schedule and beacon helpers mutate
those dicts in place -- Schedule.add_job does opts["schedule"].update(...) --
so every sub-proxy's add_job("__proxy_keepalive", ...) overwrote the same key
in the one shared dict. Only the last sub-proxy kept a keepalive job, so only
one of N sub-proxies got a __proxy_keepalive (the reported symptom); per
sub-proxy beacons collided the same way.

Give each sub-proxy its own schedule and beacon storage. Their jobs and beacons
come from their own pillar plus the per-sub-proxy keepalive added below, so they
were never meant to share the control minion's dicts (the sharing was an
accident of the shallow copy).
Move tests/unit/modules/test_rh_ip.py to
tests/pytests/unit/modules/test_rh_ip.py, converting the legacy
unittest.TestCase style (LoaderModuleMockMixin, self.assertX) to modern
pytest (configure_loader_modules fixture, plain asserts, pytest.raises).
Faithful style-only migration: all 26 tests preserved and passing; the
legacy file (which no longer runs under the current pytest/py3.12 test
env) is removed.

This lets the rh_ip test suite run in the maintained pytests tree, ahead
of adding NetworkManager-aware provider selection for saltstack#54791.
network.managed has been broken on RedHat-family systems since EL8. The
rh_ip provider writes /etc/sysconfig/network-scripts/ifcfg-* files and
brings interfaces up with ifup/ifdown from the network-scripts package.
That package is not installed by default on EL8+ and is removed entirely
on EL10, so the state fails with

    Unable to run command '['ifdown', 'eth1']' ... No such file or
    directory: 'ifdown'

and configures nothing. Verified on AlmaLinux 8, 9 and 10.

nm_ip writes NetworkManager keyfiles under
/etc/NetworkManager/system-connections/ and applies them with nmcli, the
supported way to manage networking on modern RedHat systems. It supports
ethernet (static/dhcp/disabled, dual-stack), bond, vlan and bridge, with
bond/bridge members written as their own port keyfiles. A deterministic
per-interface connection uuid keeps build_interface output identical to
the keyfile NetworkManager reads back, so the state stays idempotent.

Provider selection is a single condition both modules test: nmcli
present, /run/NetworkManager exists, and no ifup/ifdown on PATH. nm_ip
claims the ip virtual when that holds; rh_ip defers to it. Hosts that
still have network-scripts installed keep the legacy rh_ip behavior.

Also addresses saltstack#68252 and saltstack#62844, which share this root cause.

Validated end to end on AlmaLinux 8, 9 and 10 VMs: nm_ip is selected,
network.managed brings the interface up, a second apply is a no-op, and
the address persists across reboot.
3006.27 was released 2026-07-01 without this provider, so the next
available 3006.x release is 3006.28.
… Amazon deferral

The direct tests call nm_ip.build_interface at the exact altitude
network.managed uses (name, iface_type, enabled, **kwargs with the
state-injected test flag): test=True must return the rendered keyfile
lines for the diff without writing anything (including bond port
keyfiles), and the same call with test=False must write those exact
lines. The inverse tests guard the refactored rh_ip.__virtual__ against
overcorrection: Amazon Linux 2 with network-scripts must still claim ip
and Amazon Linux 1 must still decline, verified against the base branch
module as well.
Close a set of coverage gaps in the nm_ip (NetworkManager keyfile) ip
provider so it maps more of the network.managed schema and moves closer to
rh_ip parity, and address review feedback on keyfile permissions and code
duplication.

Coverage:

- mtu on bond/bridge/vlan is now emitted via a separate [ethernet]
  (802-3-ethernet) section attached to the connection, matching how NM sets
  MTU on virtual devices. Previously it was accepted and silently dropped
  because the native [bond]/[bridge]/[vlan] settings have no mtu key.
  Ethernet interfaces keep folding mtu into their own [ethernet] section.
- hwaddr pins a connection to a NIC's permanent MAC (802-3-ethernet
  mac-address, or bridge.mac-address for bridges), honouring the auto/none
  sentinels. macaddr sets the in-use MAC (802-3-ethernet cloned-mac-address)
  and is mutually exclusive with hwaddr, matching rh_ip.
- The autoneg/speed/duplex ethtool link parameters map to [ethernet]
  auto-negotiate/speed/duplex (speed and duplex must be set together)
  instead of being rejected. Offload/channel/advertise ethtool knobs, which
  have no keyfile equivalent, are still refused.
- Bond options are passed through to [bond] from the full kernel bonding set
  (ad_select, fail_over_mac, primary_reselect, arp_validate,
  all_slaves_active, min_links, ...) rather than a fixed ten-key allow-list.
  Option names are validated and mode stays required.
- dns-search is written under [ipv6] as well as [ipv4], so search domains
  survive on IPv6-only hosts; a disabled family no longer carries a dead
  dns-search line.
- vlan reorder_hdr/gvrp/loose_binding fold into the [vlan] flags bitmask
  (emitted only when it diverges from NM's default), and wol maps to
  [ethernet] wake-on-lan.

Hardening / cleanup:

- Write keyfiles with salt.utils.files.fpopen(mode=0o600) so the connection
  file is created with 0600 permissions before any content is written,
  instead of chmod'ing an already-populated file.
- Extract the shared NetworkManager provider-selection check into
  salt.utils.network.nm_managed; nm_ip.nm_managed and rh_ip._nm_managed now
  both call it so exactly one provider claims the ip module.

Add direct and inverse unit tests for each gap under
tests/pytests/unit/modules/test_nm_ip.py.
…d-mode error

- _write_keyfile writes to a 0600 mkstemp temp in the keyfile's own directory
  and os.replace()s it onto the target, so NetworkManager (which watches these
  files via inotify) never sees a half-written or briefly world-readable
  keyfile. Replaces the in-place fpopen write; a copy that preserves the
  destination mode (salt.utils.files.copyfile) would let an existing 0644
  keyfile stay world-readable, so it is deliberately not used here.
- _listify also splits on ';', NetworkManager's on-disk array delimiter, so a
  pillar value pre-formatted that way parses into individual entries.
- The missing-bond-mode error names example modes and explains why mode is
  required rather than defaulted (rh_ip requires it too).

Adds direct tests for the atomic 0600 write (including rewriting an existing
0644 keyfile) and for semicolon _listify.
dwoz and others added 11 commits August 27, 2026 11:54
The `SaltVersionsInfo` table in `salt/version.py` had CHLORINE (3007)
marked `released=True` but ARGON (3008) not, even though 3008.0,
3008.1, and 3008.2 have all been tagged and shipped.

`SaltVersionsInfo.current_release()` iterates the version list and
returns the last codename with `released=True`. With ARGON missing
the flag it returns CHLORINE, which then gets used as the baseline
in `__discover_version()`. On the 3008.x branch that produces version
strings like `3007.14+2643.gca4940b8f0` for nightly builds -- packages
end up named `salt-3007.14+N-0.x86_64.rpm` instead of the expected
`salt-3008.2+N-0.x86_64.rpm`.

Regression window on 3008.x: introduced by the 2026-08-25 merge from
3007.x (commit 17c1f44 "Merge remote-tracking branch
'origin/3007.x' into merge/3007.x/3008.x-08-25-26"). Prior nightlies
were correctly labeled -- e.g. build 210 from 2026-08-19 shipped as
`salt-3008.2+210.g9a41326f33`. Master has the same bug and was
similarly affected (though master nightlies have been failing for
other reasons and nobody noticed).

Fix is a one-line flip:

    ARGON = SaltVersion("Argon", info=3008, released=True)

Needs to be backported to 3008.x. Any active supported release
branch whose `current_release()` should return ARGON needs the same.

Discovered while verifying nightly RPM signing on a live 3008.x
build (salt-nightlies run 33103981309): "Prepare Release" step
printed "3007.14+2643.gca4940b8f0" from a 3008.x checkout.
…008.x

[3008.x] version: mark ARGON (3008) as released
The mirror workflow on saltstack/salt-nightlies force-pushes branches
from saltstack/salt daily. Those push events trigger ci.yml here to
re-run the exact same tests that just ran on saltstack/salt before the
mirror — pure duplicate, no additional signal, real runner-minute cost.

Gate the prepare-workflow job with
  if: github.repository != 'saltstack/salt-nightlies'
so CI cascade-skips on the fork (all subsequent jobs `needs:
prepare-workflow`). Set via prepare_workflow_if_check in ci.yml.jinja
so the change survives future template regeneration; the generated
ci.yml is updated in the same commit for immediate effect (would
otherwise wait for the next `tools ci` regen).

Follow-ups if warranted (not in this PR): similar treatment for
scheduled.yml, staging.yml, doc-linkcheck.yml, backport.yml,
dependabot-sync.yml, release.yml.
saltstack#70141 added RPM signing for nightly builds via the reusable
build-packages.yml workflow. DEBs were left unsigned -- build-packages.yml
had sign-rpm-packages / sign-macos-packages / sign-windows-packages
inputs but no sign-deb-packages, and the build-deb-packages job had
no environment: declaration (so it couldn't read secrets), no gnupg
setup, no debsigs invocation. Verified today: run 33107756715 on
saltstack/salt-nightlies produced 16 signed RPMs (RSA/SHA256, Key ID
52f404520060a19b) but all 16 DEBs came out unsigned (no _gpgorigin
in the ar archive).

Closes that gap by mirroring the RPM signing plumbing for DEBs:

tools/pkg/build.py:
  Add `key_id` argument to the `deb` command. When set, run
  `debsigs --sign=origin --default-key $KEY_ID <pkg>.deb` on each
  .deb debuild produced in the parent directory. Same style of
  post-build signing loop the `rpm` command uses with `rpmsign`.

.github/workflows/build-packages.yml:
  - Add `sign-deb-packages` boolean input (default false).
  - Add `environment: ${{ inputs.environment }}` to the
    build-deb-packages job so it can read caller-inherited secrets
    (previously only build-rpm-packages had this).
  - Install debsigs in the deb build container's dependency step.
  - Add a Setup GnuPG step gated on `inputs.sign-deb-packages`,
    identical shape to the RPM job's version: imports SIGNING_GPG_KEY
    with SIGNING_PASSPHRASE, then discovers the imported key's
    fingerprint via `gpg --list-secret-keys` and exports SIGN_KEY_ID
    to $GITHUB_ENV.
  - Thread `--key-id=${env.SIGN_KEY_ID}` through to
    `tools pkg build deb` when signing is enabled.

templates/build-packages.yml.jinja:
  Add `sign-deb-packages: true` for nightly gh_environment, following
  the sign-rpm-packages pattern. Regenerates nightly.yml with the
  flag set on both build-pkgs-onedir and build-pkgs-src callers.

templates/staging.yml.jinja:
  Add matching `sign-deb-packages` input so stable releases can opt
  in explicitly. Default false to preserve current behavior.

Local verification: pre-commit run passes on all modified files
(Generate GitHub Workflow Templates, Lint GitHub Actions Workflows,
isort, black, mypy). End-to-end debsigs signing was independently
tested earlier in a debian:13 container against the same nightly
signing key material -- signature verifies cleanly with
`gpg --verify _gpgorigin <payload>` after extracting the ar archive.

DEB consumer caveat: `apt install` does not check debsigs-style
file signatures by default (it verifies Release-file signatures
from apt repos). Since salt-nightlies live in an Artifactory
generic repo without apt metadata by design, the signed DEBs are
verifiable via `debsigs --verify <file.deb>` for anyone who wants
to before installing. Same posture as the signed nightly RPMs
verified via `rpm --checksig`.

Needs backport to 3008.x (and any other supported release branch
where nightly RPM signing is enabled).
…008.x

[3008.x] nightly: sign DEB packages with debsigs, mirroring RPM signing path
…name

saltstack#70162 marked ARGON released=True on 3008.x so
SaltVersionsInfo.current_release() now returns ARGON. The regression
test test_current_release_matches_maintenance_branch_67061 was
asserting the pre-flip codename by name (CHLORINE), which was
correct for 3007.x but wrong once 3008.x updated the released
flag. Every 3008.x PR started failing here (e.g. saltstack#70157 CI).

Rather than bump the hardcoded codename with every release cycle
(and reintroduce this same failure mode next time POTASSIUM or its
successors flip to released=True), assert the *contract* that
current_release() is documented to satisfy: return the last codename
in SaltVersionsInfo.versions() with released=True.

    released = [v for v in SaltVersionsInfo.versions() if v.released]
    assert released, "SaltVersionsInfo table has no released codenames"
    expected = released[-1]
    current = SaltVersionsInfo.current_release()
    assert current == expected, "..."

Same regression-protection value against the original saltstack#67061 bug
(current_release() walking to the first *un*-released codename would
still fail this assertion), zero per-release-cycle churn. Failure
message names both the expected and actual codenames.

Verified locally: contract-based assertion passes with the current
SaltVersionsInfo table state on this branch (expected=Argon/3008,
current=Argon/3008).
…008.x

[3008.x] ci: skip CI on saltstack/salt-nightlies (duplicate of upstream)
…lease-argon-3008.x

[3008.x] test_version: assert current_release() contract, not a hardcoded codename
Commit f3ffc8f (originally landed on 3007.x) added
`--match v3007.*` to salt.version's git-describe call to keep 3008.x
tags from hijacking version detection on 3007.x when a reverted
mis-merge left 3008.x commits reachable.

That commit got merged forward into 3008.x on 2026-08-25 (via
17c1f44 "Merge remote-tracking branch 'origin/3007.x' into
merge/3007.x/3008.x-08-25-26") without updating the constraint to
this branch's major. Result on 3008.x: git describe skips every
v3008.* tag and reports "v3007.14-N-gSHA". Downstream:

- __discover_version() sees parsed.major=3007 < saltstack_version.major=3008
  and takes the "lift baseline" branch, replacing the parsed base
  with SaltVersionsInfo.current_release() (=ARGON, info=3008 only,
  no minor).
- Minor defaults to 0.
- Every nightly RPM/DEB on 3008.x since the merge has been labeled
  salt-3008.0+N-... instead of salt-3008.2+N-...

Fix is one-line: change v3007.* -> v3008.*. Same intent as the
original 3007.x commit (constrain to the branch's own major),
adapted to 3008.x's calver line. git describe now returns
v3008.2-<ahead>-gSHA, __discover_version() parses it directly
(same major, no lift-baseline needed), and RPM/DEB packages get
the correct salt-3008.2+N-... version prefix.

Verified in the salt-nightlies 3008.x run 33122447777 (post-saltstack#70162
merged): produces salt-3008.0+2651.gfd8241adb4 instead of the
expected salt-3008.2+2651.gfd8241adb4. This one-line fix restores
the correct minor.

Master is unaffected -- it uses `--match v[0-9]*` (no branch-major
constraint).
…tch-3008-on-3008.x

[3008.x] version: constrain git-describe --match to v3008.* on 3008.x
…-08-28-26

# Conflicts:
#	.github/workflows/ci.yml
#	.github/workflows/dependabot-sync.yml
#	.github/workflows/nightly-stress-test.yml
#	.github/workflows/nightly.yml
#	.github/workflows/scheduled.yml
#	.github/workflows/staging.yml
#	.github/workflows/templates/layout.yml.jinja
#	.pre-commit-config.yaml
#	pkg/macos/install_salt.sh
#	requirements/base.txt
#	tests/pytests/pkg/integration/test_version.py
#	tests/pytests/unit/cli/test_batch.py
#	tests/pytests/unit/grains/test_core.py
@dwoz
dwoz requested a review from a team as a code owner August 30, 2026 01:26
@dwoz dwoz added the test:full Run the full test suite label Aug 30, 2026
@dwoz dwoz added this to the Potassium v3009.0 milestone Aug 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

test:full Run the full test suite

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants