Repository navigation
fix: emudhra ca certificate update - #175
nick-hildebrant-etas wants to merge 1 commit into
Conversation
|
@opajonk @lurtz - this is a security fix, is it a chore or a fix? Also, this is quite a jump in uv versions, and I am not so sure if this is a patch level bump (the ca fix should be non-breaking) or if it is a minor due to side-effects related to the big uv jump. I am not so familiar with the version history of uv. |
nick-hildebrant-etas
changed the title
| # Refresh the system trust store before the first TLS download. This must come | ||
| # before load_feature_versions.sh, which fetches yq over HTTPS, so every | ||
| # download in this script validates against a current set of roots. The feature |
There was a problem hiding this comment.
What happens if I run an outdated container image? Should we now delete all container images, which have outdated certificates? This would be in conflict with having long term reproducible builds
There was a problem hiding this comment.
there are definitely conflicts between the ideas of "long term reproducible builds" and "security updates". This isn't resolved here. The assumption I make is security issues have higher priority. If in 10 years you load up a container image which contains virus which infects the build system you delivered with, do you want the same infected build or you want to make a fixed version. There is no right answer here, long term reproducible and security are in conflict. If we prioritize on security, we can be surer we can build the software next week when the exploits start coming...
There was a problem hiding this comment.
I wanted to point out what the consequences are, if we really want to prevent devcontainer images with known security issues.
As far as I can understand your pull request checks that at container build time no revoked CA certificate is used. There is nothing which checks for that in a running container image and I could at the moment download an older container image.
This means we might have to delete old container images each time there is a security update. Or we need a more sophisticated approach, where also older versions might get fixes.
Can we avoid doing such stuff?
There was a problem hiding this comment.
This is a good point, and I don't have an answer. The only solution I know for avoiding the topic is to go full hermetic builds where all deps are packed into the devcontainer and then no CA is required.
The only way to be sure all of our dependencies are really available (even beyond these CA issues, but what if PyPI moves or removes packages) is to copy them locally, but this results in a massive devcontainer which brings everyone to a stop for hours of updates every day... I don't know of a good way. I already tried once setting up a sidecar container for a pypi/cargo repos with our deps pinned, which score devcontainer references everywhere. This moves the churn out of the devcontainer, but updating dependencies for all modules has to be centralized across the whole score project to make that work (maybe the least bad option I know)
Discuss on Monday in the INFRA round?
There was a problem hiding this comment.
We can discuss this on monday. Actually @opajonk and myself would prefer less downloads from the internet. We have always unstable CI at Github, because its internet connection is not stable.
| # System trust store | ||
| # | ||
| # The October 2026 eMudhra incident compromised intermediate CA private keys | ||
| # chaining to "emSign ECC Root CA - G3"; Ubuntu distrusts that root as of | ||
| # ca-certificates 20260816ubuntu0.24.04.1. The install step upgrades the | ||
| # package, but image builds run with `--cache-from`, so a layer assembled | ||
| # before that upgrade could still be reused. Asserting the outcome here means | ||
| # a stale trust store fails CI instead of being published. | ||
| distrusted_root_cn="emSign ECC Root CA - G3" | ||
| distrusted_root_sha256="86:A1:EC:BA:08:9C:4A:8D:3B:BE:27:34:C6:12:BA:34:1D:81:3E:04:3C:F9:E8:A8:62:CD:5C:57:A3:6B:BE:6B" | ||
|
|
||
| # Prove the bundle is readable first; otherwise the absence check below would | ||
| # pass simply because nothing could be parsed out of it. | ||
| check "validate the system CA bundle is present and parsable" bash -c \ | ||
| "[ \"\$(openssl crl2pkcs7 -nocrl -certfile /etc/ssl/certs/ca-certificates.crt | openssl pkcs7 -print_certs -noout | grep -c '^subject=')\" -gt 50 ]" | ||
| check "validate the compromised ${distrusted_root_cn} is not in the system CA bundle" bash -c \ | ||
| "! openssl crl2pkcs7 -nocrl -certfile /etc/ssl/certs/ca-certificates.crt | openssl pkcs7 -print_certs -noout | grep -qF '${distrusted_root_cn}'" | ||
|
|
||
| # Bazel runs its JVM against this keystore (see the bazel feature's | ||
| # /etc/bazel.bazelrc), so it needs the same guarantee as the OpenSSL bundle. | ||
| # Matched on fingerprint because the keytool alias is derived from a filename. | ||
| check "validate the compromised ${distrusted_root_cn} is not in the Java keystore" bash -c \ | ||
| "! keytool -list -keystore /etc/ssl/certs/java/cacerts -storepass changeit | grep -qiF '${distrusted_root_sha256}'" | ||
|
|
There was a problem hiding this comment.
does it make sense to have this test in a separate file?
| apt-get update | ||
|
|
||
| # Refresh the system trust store before the first TLS download. This must come | ||
| # before load_feature_versions.sh, which fetches yq over HTTPS, so every | ||
| # download in this script validates against a current set of roots. The base | ||
| # image carries the CA bundle from its own build date and this feature runs | ||
| # with `upgradePackages: false`, so a distrusted CA would otherwise persist. | ||
| # Deliberately not pinned in versions.yaml: a pinned trust store is a stale | ||
| # trust store, and it has to be current before versions.yaml can even be read. | ||
| apt-get install -y --no-install-recommends ca-certificates | ||
| update-ca-certificates --fresh |
There was a problem hiding this comment.
This is now the second time I see this. Would it not be enough to have it in the Dockerfile?
|
|
||
| # Derive the GitHub repo from the first binary URL: | ||
| # https://github.com/<owner>/<repo>/releases/download/v<ver>/<asset> | ||
| # https://github.com/<owner>/<repo>/releases/download/<tag>/<asset> |
There was a problem hiding this comment.
maybe then @FScholPer does not need to fix this script anymore
a-zw
left a comment
a-zw
left a comment
There was a problem hiding this comment.
Looks fine. Only two comments are suspicious.
| # `0.13.5`. Reuse the prefix the pinned URL already encodes, because | ||
| # Renovate rewrites the version inside that URL but never the prefix. |
There was a problem hiding this comment.
You sure about the Renovate behavior? My LLM believes this comment is wrong and might mislead someone.
| # entries in uv.lock.json separately, so a partial bump would otherwise leave | ||
| # them silently pinned to different releases. |
There was a problem hiding this comment.
You sure about the Renovate behavior? My LLM believes this comment is wrong and might mislead someone.
This pull request focuses on improving the security and reliability of the system trust store handling in devcontainer builds, ensuring that TLS operations always use an up-to-date CA bundle. It also upgrades several Python tools (
uv,uvx, andpre-commit) to their latest versions and makes related changes to lockfiles, documentation, and test coverage. Additionally, the lockfile refresh logic is improved for better compatibility with upstream release tagging conventions.response to: https://access.redhat.com/security/vulnerabilities/RHSB-2026-012
Security and Trust Store Improvements:
.devcontainer/Dockerfile,with-proxy-vars.Dockerfile,bazel-feature/install.sh,s-core-local/install.sh) to prevent usage of stale or distrusted CAs, with explanatory comments clarifying the rationale. [1] [2] [3] [4]test_default.sh) to assert that compromised or distrusted CAs (e.g., "emSign ECC Root CA - G3") are not present in either the OpenSSL or Java trust stores, ensuring CI fails if the trust store is stale.Python Tool Upgrades and Lockfile Consistency:
uvanduvxfrom version 0.10.4 to 0.13.5 and updated their lockfile entries and checks to ensure both always track the same upstream release, preventing accidental version skew. [1] [2]pre-commitfrom version 4.5.1 to 4.7.0 in the tool catalog, lockfile, and corresponding tests. [1] [2] [3]uv.lock.jsonupdates automatically, ensuring bothuvanduvxare kept in sync with upstream security releases.Lockfile Refresh Logic Improvements:
refresh_lockfile_digests.pyto correctly handle upstream release tags that may or may not be prefixed withv(e.g.,v1.27.0vs0.13.5), ensuring URLs and API calls match the actual release format and preventing broken upgrades for tools likeuvandruff. [1] [2]Test and Documentation Updates:
uvanduvxare always pinned to the same release, and that the correct versions are installed and functioning.tools/README.mdto reflect the new versions ofuv,uvx, andpre-commit.