Build the offline Docs source parser and fail-closed parity census

[alexeygrigorev]

Parent epic: #41

Normative authority:

• Spec 03 — source ownership and Docs adapter
• Spec 03 — sync safety
• Spec 02 — exact URL/link/fragment compatibility
• Development process
• Implement the Django compatibility, redirect, link, and SEO parity gate #35 supplies the generic fail-closed compatibility vocabulary; Adopt secure direct-sync content ingestion and management #38 owns later direct sync and Build Django-owned PostgreSQL search and cross-source graph activation #44 owns later unified search.

Product outcome

Make the currently checked Docs projection auditable from one caller-supplied immutable DataTalksClub/docs checkout. This issue delivers only a deterministic, network-free parser, selection ledger, schema validation, and parity report. It must expose present source/projection drift as a typed BLOCKED result; it does not bless, repair, activate, or serve a different projection.

Public repository/edit controls are forbidden by spec 03. The existing source path, checksum, immutable revision, and moving-main edit target may be retained as management-only provenance, but no public DTO, template, page, metadata, link, or browser control may expose them.

Exact baseline to bind

Repository baseline is DataTalksClub/website@face8e4808d65afbf0374d1ced7a88079950d663.

Docs source declared by the checked projection:

• repository: DataTalksClub/docs;
• commit: 3f23e006ffdaa498bbc69697408853b6f5eb37dc;
• tree: 585e440a6c7c32e8a3936f431a8c0a1d8709b1a7;
• complete source tree: 285 blobs, including 119 Markdown files and 54 image files;
• adopted checked selection: 106 pages and 39 image assets;
• excluded Markdown census: 13 paths; unselected image census: 15 paths.

Checked target evidence at face8e4:

• content/docs_projection.json: 381,637 bytes, SHA-256 15d9491fee4683303ce3cc0ba00c702599016149c010923e200a85d37084fc8a;
• deterministic 39-file content/docs_assets/ path/checksum manifest SHA-256: 8cce1dd8b6d0339c42879817cfa9d07f45684c9084ac7445dec3c8f6056f601f;
• projection records: 106 pages, 39 assets, 106 management-only edit_url values;
• _docs/compatibility/generated-path-baseline.jsonl: 174 dtc-docs rows, all preserve: 107 HTML, 63 assets, 2 JSON, and 2 XML.

The older inventory claim that the projection digest is 1abd84ab... is stale. The checked projection also contains at least one proven source contradiction: courses/llm-zoomcamp/project.md retains the pinned raw-source checksum 6a3ae6c9... while its body contains the later AI Shipping Blog destinations absent from the declared source commit. The parser must discover and report the complete mismatch set, not special-case this example.

Moving upstream main is not an input. It is currently f6c8527d349fceec6e8e89cc255c6d484ffbf22f / tree 2e856b9a818e0c3123b9efaf3879c2b999c04899, four commits ahead of the declared pin, and changes four adopted page files plus repository-maintenance files. This issue neither adopts nor copies those bytes.

Scope and owned files

Implement the smallest self-contained offline lane in these file families:

• new content/docs_source_adapter.py: bounded source selection, frontmatter decoding, hierarchy/path derivation, asset reference selection, checksums, and canonical projection records;
• new scripts/build_docs_projection.py: a uv-run CLI that accepts an already verified checkout path plus exact commit/tree arguments and writes only to an explicit output directory under .tmp/, or compares with --check;
• new _docs/compatibility/docs-source-selection.schema.json and docs-source-selection.json: exact 285-blob classification ledger with one disposition and bounded reason for every source blob;
• new _docs/compatibility/docs-projection.schema.json: strict schema for the current version-1 projection object, page records, asset records, and management-only provenance fields;
• new focused fixtures/tests under content/tests/fixtures/docs_source_adapter/, content/tests/test_docs_source_adapter.py, and tests_ci/test_docs_projection_generator.py;
• one Makefile check target and only the minimal CI ownership/change-selection entries needed to run it.

Do not edit content/docs_projection.json, content/docs_assets/**, content/docs_projection.py, content/docs_presentation.py, views, URLs, templates, browser tests, models, migrations, source pins, or compatibility observations in this issue. An unexpected need to touch those files returns the issue to PM.

Parser and selection contract

• The adapter performs no Git, GitHub, HTTP, DNS, database, subprocess, template, or filesystem write operation. It accepts a local checkout plus scalar expected repository/commit/tree identities supplied by the CLI.
• Reject a wrong/moving SHA, wrong tree, dirty or symlinked selection, traversal, duplicate normalized path, unsupported file type, oversized file/tree, YAML alias/tag, malformed or duplicate frontmatter key, non-UTF-8 text, and unknown required field with stable content-free diagnostics.
• Classify all 285 source blobs exactly once. Adopt the exact 106 page sources rooted at index.md, activities/, courses/, and general/; every other Markdown/config/theme/tool/draft file requires an explicit ledger disposition and reason. Do not infer publication merely from .md.
• Preserve exact source bytes/checksums and decoded body. Parse only the bounded Docs frontmatter fields title, description, nav_order, parent, grand_parent, has_children, has_toc, and permalink; reject wrong scalar types and unknown publication-affecting metadata.
• Derive canonical /docs/ pretty paths, explicit permalinks, parent_path, and grand_parent_path deterministically. Missing, ambiguous duplicate-title, contradictory parent/grand-parent, cycle, orphan, duplicate public path, wrong-root permalink, query, fragment, slash, case, Unicode, or percent-escape collision fails the whole result; no title/path guessing or partial projection is allowed.
• Preserve Markdown/Liquid as source data for the existing renderer. Recognize and validate root-relative relative_url links/images without executing Liquid. Select only the exact referenced adopted JPEG/PNG/SVG files, verify content type/size/SHA-256, and reject traversal, remote/protocol-relative media, missing/duplicate asset ownership, unsafe SVG, or unreferenced asset presented as adopted.
• Emit records in deterministic source/navigation order with the exact current projection fields. edit_url is management provenance only and must be derived from the allowlisted repository plus bounded source path; public exposure is outside this adapter and forbidden.
• Canonical JSON and asset output must be byte-identical across two clean invocations from the same checkout. All diagnostics and reports are bounded, path-safe, secret-free, and contain no source bodies.

Parity and compatibility report

The check command compares a generated scratch output with the frozen checked projection/assets and separately accounts for all 174 dtc-docs compatibility rows.

• Every source blob, generated page/asset, checked page/asset, and compatibility row is classified exactly once. Missing, extra, duplicate, changed, or unexplained entries block.
• The report binds website SHA, source SHA/tree, parser/schema/selection-ledger digests, checked and generated projection/asset-manifest digests, and the checked compatibility-manifest digest.
• The present expected verdict is BLOCKED, with stable findings for every source/checksum/body/projection mismatch and every one of the 29 compatibility paths not represented by the 106-page/39-asset projection (one HTML 404, 24 other assets, two JSON, and two XML). No finding is automatically approved as redirect, retirement, or parity.
• The known courses/llm-zoomcamp/project.md declared-source/body contradiction and stale 1abd84ab... audit binding must be detected from general rules. Hard-coding those paths, URLs, or hashes as exceptions fails acceptance.
• PASS is impossible while any selected source byte cannot reproduce the checked record, any public compatibility path lacks a separately approved expectation, or any binding is stale. A blocked report never invokes a write, activation, cutover, or public-reader callback.

Acceptance criteria

• Strict schemas and loaders reject unknown versions/keys, duplicate/missing classifications, bad hashes/counts/pins, malformed paths, and hostile YAML/path/media fixtures with stable bounded codes.
• The selection ledger covers exactly 285 source blobs: 106 adopted pages, 39 adopted images, and 140 explicitly reasoned exclusions; it binds the exact source SHA/tree and regenerates canonically.
• Real-pin generation produces 106 page and 39 asset records twice with byte-identical scratch trees and no network or repository write.
• Navigation/path fixtures cover explicit permalink, nested parent/grand-parent, order, duplicate titles, ambiguity, orphan/cycle, case, Unicode, percent spelling, query/fragment, and trailing slash.
• Content fixtures cover tables, fenced code, callouts, Mermaid source, raw HTML, images, safe relative_url, malformed Liquid, unsafe HTML/URL/media, missing asset, and unsafe SVG without executing source code.
• The full real-pin report deterministically returns BLOCKED, accounts for all 285 source blobs, 145 projection records, and 174 compatibility rows, and reports the complete drift/unimplemented-path census with no auto-approval.
• Focused tests, schema checks, make docs-projection-check, formatting, lint, type checking, migration drift, Django checks, and the graph-selected verification envelope pass. For backend-only/no-render impact, screenshots are not_applicable only when the computed graph proves render.impact=false with no required captures; the graph-selected Playwright smoke tier remains required.
• The frozen checked projection/assets and all public routes/rendering remain byte-for-byte untouched.

Test scenarios

1. Parse a compact real-shaped hierarchy and assert canonical records, checksums, ordering, edit provenance, referenced assets, and exact repeat output.
2. Mutate each pin, selection, frontmatter, hierarchy, path, source-body/checksum, Liquid/media, asset, schema, and count boundary independently; assert one deterministic blocking diagnostic and no partial output.
3. Run the full supplied immutable checkout twice into separate .tmp/ outputs, compare complete trees, then compare to the checked projection/assets and the 174-row compatibility scope.
4. Tamper with generated output, checked projection, checked asset bytes, source ledger, schema, or compatibility manifest after report creation; validation must fail stale/contradictory rather than reuse evidence.

Non-goals

No source-pin advance, moving-branch adoption, upstream repository edit, public content repair, generated-artifact replacement, public edit/search control, template/view/route change, Markdown renderer redesign, database model/migration, direct upsert, webhook/reconciliation, Studio/admin API, search indexing, public authority cutover, redirect/410 approval, provider/production access, deployment, or source checkout committed to this repository. No new Python dependency.

Dependencies and handoff

Direct prerequisite #35 is closed. This offline issue is deliberately independent of open #38, #44, #253, and production/provider state because it activates nothing and accepts a caller-supplied checkout.

A later #41 phase may select a new immutable Docs revision, repair/replace the checked projection, wire direct sync, approve compatibility dispositions, switch public authority, and build unified search only after this report, #38’s source-rollout/public-authority decisions and accepted children, and #44’s applicable interface exist. Evidence from this issue cannot authorize those later actions.

[alexeygrigorev]

Grooming Complete

This is the bounded, engineer-ready first child of #41.

It is intentionally network-free and non-activating: exact source/tree/projection/asset/compatibility counts and digests are pinned, the allowed file families and strict schemas are named, hostile parser/path/hierarchy/media cases are specified, and the current required result is a deterministic BLOCKED parity report—not a rewritten projection.

Public edit/source controls are explicitly forbidden; the existing edit_url values are management-only provenance. The issue is independent of #38/#44/#253 because it performs no sync, cutover, search, or provider action. Engineer may proceed through the normal uncommitted implementation → independent tester → PM lifecycle on a current clean base.

[alexeygrigorev]

Software Engineer Report

Candidate

• Issue: Build the offline Docs source parser and fail-closed parity census #292 only.
• Worktree: /home/alexey/git/dtc-website/.tmp/issue-292-engineer
• Branch: issue-292-docs-source-adapter
• Base and observed HEAD: face8e4808d65afbf0374d1ced7a88079950d663 (origin/main at assignment).
• State: frozen and uncommitted; no commit, push, merge, runtime activation, public edit link, provider access, or production access was performed.
• Canonical candidate-content manifest SHA-256 (sorted sha256sum rows for all 15 changed files): d812a0e973a49d57fd05d53e38b7cb58d15fb7f1d0762a69c88480afcb569fbd.
• Pinned source: DataTalksClub/docs@3f23e006ffdaa498bbc69697408853b6f5eb37dc, tree 585e440a6c7c32e8a3936f431a8c0a1d8709b1a7; the 285-blob source tree was exported locally to .tmp/docs-source-3f23e006 and processed offline.

Behavior implemented

• Added a bounded, read-only docs source adapter and CLI. It validates the exact source repository/commit/tree/blob ledger, rejects missing/extra/dirty/symlink/unsafe or ambiguous input, strictly parses bounded front matter and supported Markdown/Liquid references, validates hierarchy/permalinks/fragments/assets, and emits deterministic projection plus parity evidence only into an explicitly supplied new child of repository-local .tmp/.
• Added versioned JSON schemas and a complete 285-blob selection ledger: 106 adopted pages, 39 adopted assets, and 140 reasoned exclusions.
• Added general fixture and pinned-tree regression tests, including static capability-denial tests. The implementation has no Git, subprocess, database, template, network, or provider capability and no public-write path.
• Added make docs-projection-check, which performs two independent clean builds, requires the groomed baseline census and BLOCKED verdict, and compares the complete output trees byte-for-byte.
• No checked runtime projection, runtime asset, presentation, URL, search, Adopt secure direct-sync content ingestion and management #38 activation, or Build Django-owned PostgreSQL search and cross-source graph activation #44 search file was changed.

Changed files:

• Makefile
• _docs/compatibility/docs-projection.schema.json
• _docs/compatibility/docs-source-selection.json
• _docs/compatibility/docs-source-selection.schema.json
• content/docs_source_adapter.py
• content/tests/fixtures/docs_source_adapter/valid/assets/images/diagram.svg
• content/tests/fixtures/docs_source_adapter/valid/courses/index.md
• content/tests/fixtures/docs_source_adapter/valid/courses/sample/index.md
• content/tests/fixtures/docs_source_adapter/valid/courses/sample/overview.md
• content/tests/fixtures/docs_source_adapter/valid/courses/sample/section.md
• content/tests/fixtures/docs_source_adapter/valid/courses/sample/topic.md
• content/tests/fixtures/docs_source_adapter/valid/index.md
• content/tests/test_docs_source_adapter.py
• scripts/build_docs_projection.py
• tests_ci/test_docs_projection_generator.py

Projection/parity evidence

• Command: make docs-projection-check DOCS_SOURCE_CHECKOUT=.tmp/docs-source-3f23e006 — PASS twice with byte-identical output.
• Outputs: .tmp/docs-projection-check-1655854-{a,b}.
• Report SHA-256 in both outputs: 4f72d8f46aa39281fda47bd4a76b5856edebf9cf14013ae02a4363d2bdb1148e.
• Generated projection SHA-256 in both outputs: a24a03298fe4e13d5aed45f9deb49e52a6fc24813d5430a614e79a1580c28f92.
• Checked projection binding: 15d9491fee4683303ce3cc0ba00c702599016149c010923e200a85d37084fc8a.
• Checked/generated asset-manifest binding: 8cce1dd8b6d0339c42879817cfa9d07f45684c9084ac7445dec3c8f6056f601f.
• Exact census: 285 source blobs; 106 pages + 39 assets = 145 generated/checked records; 174 compatibility rows (107 HTML, 63 asset, 2 JSON, 2 XML); 29 unimplemented paths (1 HTML, 24 assets, 2 JSON, 2 XML).
• Expected verdict: BLOCKED. General comparison discovers one checked page mismatch (courses/llm-zoomcamp/project.md, fields body and body_sha256), the corresponding checked projection byte mismatch, 29 unimplemented compatibility paths, and two stale projection bindings. This is product evidence and was not weakened or converted to success.

Verification plan and dispositions

• Command: make verification-plan VERIFY_BASE_SHA=face8e4808d65afbf0374d1ced7a88079950d663 VERIFY_HEAD_SHA=face8e4808d65afbf0374d1ced7a88079950d663 VERIFY_OUTPUT_DIR=.tmp/verification-292-final VERIFY_CONSUMER=engineer — PASS.
• Plan: .tmp/verification-292-final/verification-plan.json; file SHA-256 13b0a76156758442a650581a4073660aa7d9a82c2d2cc127233c14b14bb271d7; graph SHA-256 05ffeee99244c9becd5e98e40be08188f1c5e2a61204daece984c3beb1d1bce6.
• Risk flags: dependency_toolchain, deployment_runtime, global_fixture, test_infrastructure; full profile, smoke browser profile.
• rerun: selector (always_fresh_control_plane), evidence validation (always_fresh_control_plane), container, Django, Playwright, and quality (fresh_full_regression), content invariants (large_content_invariants_required).
• reused: none.
• skipped: none.
• not_applicable: compatibility (outside_impact_closure); screenshots (render_inputs_unchanged_without_reusable_visual). No screenshots were created.
• Required Playwright is pending_independent_tester and was not run by this engineer under the assigned non-browser-only verification scope. Consequently a complete evidence-directory report was not claimed.

Commands and results

Passing candidate and required non-browser gates:

• uv run pytest -q content/tests/test_docs_source_adapter.py tests_ci/test_docs_projection_generator.py — 45 passed.
• targeted Ruff — pass; targeted mypy over the four new Python/test files — pass.
• make test-ci — 652 passed.
• make lint, make lock-check, make database-portability-check, make migrations-check django-check, make deployment-check, make security-check — pass.
• plan validation — pass.
• make verification-content-invariants ... — pass; .tmp/verification-292-final/evidence/content-invariants.json, SHA-256 ca998e744c056a8c9c23b6ddfb04d4652711d425fa5fa8951397340e638661a1.
• make verification-container ... — pass; .tmp/verification-292-final/evidence/container-check.json, SHA-256 b3954adbc0731524e6d993d4c23d25d208c97a1d256118e79be796b7bb1f8647.
• git diff --check — pass.

Unrelated base blockers observed and left untouched:

• make format-check — fails on six pre-existing files outside this candidate: content/podcast_routes.py, content/public_data.py, content/tests/test_public_projection_builder.py, core/tests/test_interactive_surfaces.py, events/tests/test_prepare_registration_sources.py, and playwright_tests/test_interactive_surfaces.py.
• make typecheck — four pre-existing errors in content/event_speaker_bio_normalization.py at lines 505, 544, 550, and 564. Targeted candidate mypy passes.
• make terminology-check — three pre-existing frozen digest mismatches in content/public_projection/{podcasts,wiki,wiki_search}.json.
• make test-django-full discovered 2,553 tests but aborted on the pre-existing content.tests.test_collection_hub.CollectionHubTemplateTests.test_controls_are_one_labelled_landmark_with_one_current_page stale markup expectation, followed by Django parallel-runner traceback pickling failure. The failing source/template are outside this candidate; the user has confirmed the new headings are correct.

Limitations and handoff

• This issue intentionally establishes an offline parser/selection/parity gate. It does not activate docs runtime delivery or modify frozen artifacts. The deterministic BLOCKED baseline remains until the separately owned runtime/compatibility work closes the reported gaps.
• The candidate is frozen for independent tester review. The tester must recompute the plan from the exact frozen state, address/attribute the base-suite blockers through their owning issues, run the required browser tier, and validate final evidence before PM acceptance.

[alexeygrigorev]

Product Manager Lifecycle Review — HOLD

Verdict: HOLD at the engineering-to-tester boundary. This is not Product Acceptance.

I audited the frozen candidate and the engineer report against _docs/PROCESS.md and the graph-selected plan.

Evidence that is valid

• Worktree/branch remains uncommitted at base and observed HEAD face8e4808d65afbf0374d1ced7a88079950d663, with exactly the reported 15 changed files.
• Recomputing the sorted changed-file checksum rows yields candidate manifest SHA-256 d812a0e973a49d57fd05d53e38b7cb58d15fb7f1d0762a69c88480afcb569fbd, matching the handoff.
• Both offline outputs are byte-identical. Each parity report has SHA-256 4f72d8f46aa39281fda47bd4a76b5856edebf9cf14013ae02a4363d2bdb1148e; each generated projection has SHA-256 a24a03298fe4e13d5aed45f9deb49e52a6fc24813d5430a614e79a1580c28f92.
• The required product result is correctly BLOCKED, not PASS: the report accounts for 285 source blobs, 106 generated pages, 39 generated assets, 145 checked records, and all 174 compatibility rows. Its 33 general-rule findings are one checked page-record mismatch, one checked projection-byte mismatch, 29 unimplemented compatibility paths, and two stale projection bindings. This deterministic product BLOCKED verdict is expected and must not be treated as a lifecycle test failure or weakened.
• Screenshots are correctly not_applicable for this candidate: plan digest 13b0a76156758442a650581a4073660aa7d9a82c2d2cc127233c14b14bb271d7, graph digest 05ffeee99244c9becd5e98e40be08188f1c5e2a61204daece984c3beb1d1bce6, render.impact=false, no render reasons, and no required captures. No unrelated screenshots are requested.

Why the lifecycle cannot advance

The same plan selects a full profile and explicitly requires fresh Django, quality, and Playwright evidence. Those requirements are incomplete or red:

• Playwright is rerun|required with command make test-playwright-smoke; it is still pending_independent_tester.
• Full Django aborted on the inherited collection-hub assertion/parallel-runner traceback.
• Required quality remains red through the reported six-file formatting failure, four event-speaker type errors, and three frozen terminology digest mismatches.
• There is no complete validated evidence report or independent tester-final report.

Inherited failures remain failures under the process; attribution does not waive a required component. The user-confirmed headings Events and Community knowledgebase remain the approved product copy. Any assertion expecting the older headings is stale and must be repaired by its owning issue, but that confirmation does not turn a red suite green.

The engineer report phrase “assigned non-browser-only verification scope” has no issue or process authority. Backend-only/no-render impact waives screenshots, not browser regression. The independent tester owns the required smoke execution, so the engineer need not duplicate it solely for role symmetry; however, Playwright remains mandatory in the final tester envelope and cannot be called outside scope.

I corrected the ambiguous acceptance-criterion sentence accordingly: render.impact=false can make screenshots not_applicable, while graph-selected Playwright smoke remains required.

Required next handoff

1. Keep this candidate frozen; do not commit, merge, push, or accept it.
2. Resolve the separately owned base failures. Because a base/source/graph input change invalidates this plan, reconstruct or rebase the unchanged Build the offline Docs source parser and fail-closed parity census #292 scope onto the resulting current clean base and recompute all candidate/plan/report digests.
3. Engineer reruns and reports all required non-browser components green while preserving the expected deterministic product BLOCKED parity result.
4. A separate tester recomputes the plan from the exact frozen state, runs the required make test-playwright-smoke plus all other required verification, and posts a complete tester-final report. Screenshots remain N/A only if the recomputed plan again proves no render impact.
5. Return to PM only after that tester PASS.