embed_docsite: replace per-directory GitHub fetch with Trees API + cache

[IZO-Ong]

Short Description

Replaces embed_docsite's per-directory GitHub Contents API walk with a single Trees API call with an on-disk cache keyed by ETag and per-file blob SHA, cutting a full run of 20 GitHub API requests down to 1.

Fixes #182

Implementation Details

github_utils.py used to fetch docs/ and adaptors/ from OpenFn/docs by recursing into every directory via the Contents API (GET /repos/{owner}/{repo}/contents/{path}) with one request per directory, then downloading each file's download_url separately. Against the live repo that is 20 requests (16 directories under docs/, 4 under adaptors/) plus 191 unconditional file downloads, on every run, with no change detection. GitHub's unauthenticated limit of 60/hr therefore allowed three runs an hour, and the fourth failed without naming its cause: the rate-limit branch returned silently with an empty file list, and the caller then crashed on IndexError: list index out of range.

#182 proposed fixing this with git clone --depth 1. This PR takes a different route by using GitHub Trees API (GET /repos/OpenFn/docs/git/trees/main?recursive=1), which returns the entire file tree, path and blob SHA for every file in one request, plus a persistent on-disk cache (services/embed_docsite/docsite_cache/, gitignored) that stores the tree's ETag and each cached file's blob SHA.

• Steady-state cost: a full run (general_docs + adaptor_docs + adaptor_functions) now spends 1 GitHub API request. Measured against the live repo: it comes back 304 when nothing changed, and 0 files are re-downloaded. Only files whose blob SHA actually moved get re-fetched, from raw.githubusercontent.com, which is not rate-limited.
• Refreshing and reading are separate: refresh_cache(docs_types) brings the whole corpus up to date in one conditional Trees request and runs once per run; read_docs(docs_type) then serves each docs type from disk without touching the network. Only the requested docs types are refreshed, so warming just adaptor_functions leaves the markdown corpus alone.
• Degradation: if GitHub is unreachable or rate-limited, the cached copy is served and a warning is logged; only a failure with an empty cache is fatal (503 UPSTREAM_ERROR). A cached file missing from disk degrades to an ApolloError instead of an
  unhandled FileNotFoundError, and reading an unpopulated cache raises rather than reporting the docs as empty.
• Syncs GITHUB_TOKEN (in .env.example) to the docs fetch, raising the limit from 60 to 5000 requests/hour.
• New mode: {"refresh_cache_only": true} warms the cache for docs_to_upload without needing Pinecone/OpenAI keys, for pre-warming in CI or locally.
• Known limitation: get_repo_tree raises if GitHub reports the tree as truncated (~100,000 files from Github's API). Not a concern at OpenFn/docs' current size (702 files).

Tests: unit tests fully mocked (services/embed_docsite/tests/unit/), plus integration tests over the real cached corpus (services/embed_docsite/tests/integration/test_docsite_cache_corpus.py, skipped unless the cache is warmed) which hold the read path to making no network calls at all.

AI Usage

Please disclose whether you've used AI in this work (it's cool, we just want to know!):

• Yes, I have used AI
• No, I have not used AI

You can read more details in our
Responsible AI Policy

[josephjclark]

Hey @IZO-Ong! Before I get too into this, why are you using the Tree API and not git clone?

I also think the theory behind that was to just pull the content we needed and ignore the rest. But presumably we're pulling so much content anyway that it's easier just to download it all?

[IZO-Ong]

I actually was working on this before seeing issue #182, so I was not aware of the git clone method. I was thinking about the git trees approach for a while, and I think the git clone method is probably better given that we can delegate the caching to git itself. I will probably close this PR and set up a new one with git clone, and maybe test the time taken between both of them!