Skip to content

feat: pull the faster of ghcr.io and the NJU mirror - #1141

Open
cocofhu wants to merge 4 commits into
mainfrom
feat/fastest-ghcr-mirror
Open

cocofhu wants to merge 4 commits into
mainfrom
feat/fastest-ghcr-mirror

Conversation

@cocofhu

@cocofhu cocofhu commented Nov 20, 2026

Copy link
Copy Markdown
Collaborator

What changed

./start.sh and ./start.sh pull time ghcr.io against ghcr.nju.edu.cn and pull the faster registry. A faster Nanjing image is tagged back to the name pinned in compose.yaml. Digests that disagree stay on ghcr.io. MIRROR=ghcr.io or MIRROR=<host> still forces that host, and the choice is not written to .env.

Scope

  • Public adapter / deployment
  • Platform core / API / runner
  • Classic
  • Portable
  • Shared behavior contract
  • Installer / CLI
  • Documentation only

Compatibility

Install and ./start.sh pull now contact both registries unless MIRROR is set. A local image with the default missing policy is kept and is not timed again. Explicit MIRROR=<host> behavior is unchanged.

Validation

python3 scripts/test-install.py

55 tests passed, including the new speed-test cases with a fake curl. The Go, public-adapter, Local, and secret-scan commands in the template were not run; this change is the container install script and its docs.

Safety

  • I did not include credentials, internal endpoints, production data, or organization-specific infrastructure.
  • I preserved user files and unrelated changes.
  • I updated generated Classic host bundles when applicable.
  • I documented third-party content and licensing changes when applicable.

Made with Cursor

cocofhu00 and others added 2 commits November 20, 2026 17:06
Time both registries during install and pull so a domestic link does not have to set MIRROR by hand.

Co-authored-by: Cursor <cursoragent@cursor.com>
Keep the speed test in Unreleased and the 0.4.0 notes from main. The installer tests keep both the SQL log and the mirror probe environment.

Co-authored-by: Cursor <cursoragent@cursor.com>
@crispinhu27

Copy link
Copy Markdown
Collaborator

安装器复现截图

对 d70557d 实测 GHCR 发布镜像 0.4.0。同一个 layer 请求返回 307,跟随跳转后收到 262,144 字节的 HTTP 206 响应。

本地 Docker 用两个仓库夹具复验 ./start.sh pull。两边的 manifest 和测速样本都能取得。较快的镜像在实际拉取时缺少配置 blob,命令退出 1。上游夹具的完整镜像由同一个 Docker daemon 拉取成功。

GHCR 的 307 跳转和本地 Docker 拉取对照

@crispinhu27 crispinhu27 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

复审了 d70557d。Linux 容器中的 57 个安装测试通过。真实 GHCR 请求和本地 Docker 拉取复现了测速跳转与自动拉取回退问题,另外核对了默认镜像的摘要验证和 Python 3.6 兼容性。

本地复现截图

Comment thread scripts/ghcr-mirror.sh Outdated
return 'timed out'
return 'failed'

if ghcr.digest and nju.digest and ghcr.digest != nju.digest:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] 自动选镜像前验证官方摘要,并按该摘要拉取

这里仅在两边都返回摘要且不同的时候拒绝镜像。默认安装在拿不到官方 manifest 摘要时仍会自动拉取南京镜像并标成官方镜像,test_nju_wins_when_ghcr_times_out 覆盖的就是这条路径。即使测速时摘要相同,后续按可变 tag 拉取,也无法保证启动的是已验证的镜像。

请在自动模式下用官方 GHCR 或发布记录提供的可信摘要验证镜像,并按该摘要拉取。拿不到可信摘要时,应保留官方路径或让用户显式选择 MIRROR。

Comment thread scripts/ghcr-mirror.sh
f'https://{host}/v2/{repo}/blobs/{layer}',
auth + ['Range: bytes=0-262143'],
)
if blob is None or blob[0] not in (200, 206):

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] 跟随 GHCR 的 layer 重定向再测速

对本 PR 固定的 ghcr.io/tencent/loopforge:0.4.0 实测,layer 的 Range 请求返回 307 和 0 字节,跳转到 CDN 后返回 206 和 262,144 字节。当前 curl 不跟随跳转,此处会把可用的官方源标成失败。只要镜像源直接返回样本,就可能被当作最快源。

请支持受限的 HTTPS 跳转,并对最终响应取样。跨主机跳转应保持认证头的作用域。回归用例需要覆盖 307 → 206。Distribution blob API

Comment thread scripts/ghcr-mirror.sh
fi
fi

local mirror_image="${mirror_host}/${image#ghcr.io/}"

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] 自动选中的镜像拉取失败后回到官方源

自动测速选中镜像后,实际 docker pull 失败会直接 exit 1。测速只验证 manifest 和一个 layer 样本,完整镜像仍可能拉取失败。

在本地 Docker 上,两边 manifest 相同且都能测速,较快的镜像缺少配置 blob。./start.sh pull 退出 1,官方路径未被调用。同一个 Docker daemon 直接拉取上游夹具成功。请在自动选择的镜像拉取失败时回退到现有官方拉取路径。

Comment thread scripts/ghcr-mirror.sh Outdated
for header in headers:
cmd.extend(['-H', header])
cmd.append(url)
proc = subprocess.run(cmd, capture_output=True, text=True)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P3] 兼容安装器允许的 Python 3.6

usable_python 接受 Python 3.6,capture_output 和 text 从 3.7 才支持。Docker Python 3.6.15 下复现了 unexpected keyword argument 'capture_output',异常被两个探测线程吞掉,测速恒回退。可用现有最低版本支持的 stdout、stderr 和 universal_newlines 参数。Python subprocess 文档

@crispinhu27 crispinhu27 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

请补齐自动选源的官方摘要验证,并按已验证的摘要拉取。请同时处理 GHCR layer 重定向、自动选中的镜像拉取失败后的回退,以及 Python 3.6 兼容性。

行内评论和复现记录

Automatic selection now follows layer redirects without forwarding the token, and a failed mirror pull falls back to ghcr.io instead of stopping the install.

Co-authored-by: Cursor <cursoragent@cursor.com>
@cocofhu

cocofhu commented Nov 20, 2026

Copy link
Copy Markdown
Collaborator Author

Addressed the review on scripts/ghcr-mirror.sh:

  • Automatic Nanjing selection now requires a verified ghcr.io digest (sha256 + 64 hex) that the mirror returns unchanged. The pull is ghcr.nju.edu.cn/<repo>@sha256:…, then tagged back to the pinned name. If the official digest cannot be verified, the install stays on the compose pull.
  • Layer Range requests follow up to three absolute HTTPS redirects. Authorization is dropped when the host changes. A 307→206 case is covered.
  • If that automatic pull fails, the script returns to the official pull instead of exiting. An explicit MIRROR= pull failure still exits 1 and does not retag.
  • The probe uses stdout/stderr/universal_newlines so it runs on the installer's Python 3.6.

python3 scripts/test-install.py: 60 tests OK.

@crispinhu27

Copy link
Copy Markdown
Collaborator

macOS Bash 3.2 安装复验

本地 Docker 上复验 c3f98c3。没有缓存镜像时,首次安装在 docker image inspect 后退出 1。自动选择镜像且实际拉取失败时,./start.sh pull 同样提前退出。

对照只在临时夹具中把 command docker 放入子 shell。同一个拉取错误随后进入官方回退分支,Compose 成功拉取本地上游夹具。

macOS Bash 3.2 首次安装和镜像回退复验

@crispinhu27 crispinhu27 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

复审了 c3f98c3。摘要验证与按摘要拉取、GHCR 重定向、Python 3.6 均已验证。60 个 Linux 安装测试与 GitHub Actions 通过。macOS 自带 Bash 3.2 仍会在首次安装和镜像拉取失败时提前退出,请修复下方的阻塞项。

本地 Docker 复现截图

Comment thread scripts/ghcr-mirror.sh
local mirror_image="${mirror_host}/${image#ghcr.io/}"
# A re-run that already has the pinned image does not need a speed test.
# An explicit mirror still reports that it kept the local image.
if [[ "$policy" != always ]] && docker_access image inspect "$image" >/dev/null 2>&1; then

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] 修复 macOS 默认 Bash 3.2 下的首次安装和镜像回退

在 macOS 的 /bin/bash 3.2.57 上,set -e 下 docker_access 内的 command docker 一旦返回非零,外层 shell 会直接退出,即使调用在 if 条件里。自动测速默认开启后,未缓存镜像的用户会走到这里。

用本地 Docker 复验 c3f98c3,无缓存时 scripts/install.sh 在 docker image inspect 后退出 1,尚未开始拉取。./start.sh pull 自动选中一个测速成功的镜像后,实际镜像拉取失败也会直接退出,跳过第 395 行后的回退分支。

在临时夹具中把 docker_access 的 command docker 调用包进子 shell 后,同一个错误进入回退分支,Compose 成功拉取上游夹具。请让调用方能收到 Docker 的失败状态,并用 Bash 3.2 覆盖冷安装和自动回退。

macOS /bin/bash aborts set -e when a command fails inside docker_access, so a cold install and a failed automatic mirror pull never reach the official pull.

Co-authored-by: Cursor <cursoragent@cursor.com>
@cocofhu

cocofhu commented Nov 21, 2026

Copy link
Copy Markdown
Collaborator Author

docker_access now runs Docker in a subshell, so Bash 3.2 returns the real exit status instead of aborting set -e inside the function.

  • A cold install whose pinned image is not cached continues after docker image inspect fails.
  • An automatic mirror docker pull failure still falls back to the official Compose pull. An explicit MIRROR= failure still exits 1.

Covered with Bash 3.2.57: test_bash32_cold_install_continues_when_the_image_is_missing and test_bash32_auto_pull_failure_falls_back. python3 scripts/test-install.py under that Bash: 62 tests OK. On Bash 4+ those two tests run when /bin/bash is 3.2.

@crispinhu27 crispinhu27 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

复审了 627cb8f。在 macOS 自带 Bash 3.2.57 上用本地 Docker 仓库夹具验证,冷安装遇到未缓存镜像时继续进入 Compose 拉取,自动选中的镜像实际拉取失败后回退到 Compose 并成功拉取上游夹具。5 个 Bash 3.2 定向用例、60 个 Linux 安装用例和三条 GitHub Actions 工作流均通过。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants