Parse the pinned legacy tools and conferences source corpus

[alexeygrigorev]

Parent: #39

Product outcome

Provide the first independently deliverable part of the remaining legacy-main adapter: a deterministic, network-free, non-rendering parser for the exact two tool and two conference records at the accepted immutable legacy-main pin. It returns typed source records, source-owned asset descriptors, opaque Person keys, and bounded diagnostics. It changes no public or database authority.

Normative authority

• _docs/specs/03-github-content-and-people.md: legacy-main source ownership, the remaining-main-site adapter, opaque Person identity, sync safety, and no executable content;
• _docs/specs/01-platform-architecture.md: GitHub-owned tools/conferences and no repository/network/parse work on public requests;
• _docs/specs/02-url-link-seo-compatibility.md: later preservation of established /tools/<slug>.html and /conferences/<slug>.html canonicals, but no route delivery in this issue;
• _docs/specs/open-decisions.md#1-github-backed-editing-resolved-by-12-and-226: GitHub stays read-only and direct sync later consumes accepted adapters;
• _docs/architecture/app-boundaries.md: source parsing belongs to content_sync and may not move into content models/views;
• _docs/specs/09-migration-rollout-roadmap.md, _docs/specs/10-verification-strategy.md, and _docs/PROCESS.md; and
• the corrected parent boundary in #39.

Immutable source and complete selection

Repository: https://github.com/DataTalksClub/datatalksclub.github.io

Commit: ee43d3fa0929faf691178d79f19528e6f15a83e5

Tree: 1666d11893a9f001e916409f3bac65a9ef0ff072

The adapter accepts only a caller-supplied checkout already proven to have that normalized origin, exact HEAD/tree, and an empty tracked/untracked status. It performs no clone, fetch, HTTP request, DNS lookup, provider call, or moving-branch lookup.

Every _tools/*.md and _conferences/*.md blob at the pin is selected—there is no collection-record exception. The only selected assets are the eleven exact files listed below. Every other repository path is excluded by rule and must neither be walked nor emitted. A missing/extra collection record, selected path of a type/mode other than regular 100644, symlink, submodule, traversal, case/Unicode-normalization collision, or file outside the exact allowlist rejects the entire parse.

Selected record	Git blob	Bytes	SHA-256
_conferences/2021-feb.md	75d7c2f9c0e3acdfd18c0cfe66b41bdcbdd39d97	18,261	df494d6bcfbd02b7b0745e75f149eb6029f6c249b5b6c6df29a349bc4e19b77b
_conferences/2021-summer-marathon.md	3a541ddec2d81d4e1fb8495cbca3c6f6333b03d4	11,089	db239c7ff07b6bebceae144fbf588eb92d53e26a57ff4db457d17f03b564f71d
_tools/modelstore.md	de13d7d514af7e8187d54ec0ce73b5d0b33453ac	256	9a66d4fba23b1616d1126ee1b892e97f25e5124389db6025ae2838dc41d18131
_tools/obsei.md	bcfb109854ea6b92f41a11568eac8b5fbe098e45	246	13fe3b6e0b873add7fb9da83a2e2c0e175be54dfaebe1f5cd2f3255d5a9ed027

Selected asset	Git blob	Bytes	SHA-256
images/other/conference-2021-feb-cover.jpg	538b8f819e0fec88b5b7e8f87cd0201b69324b84	41,644	8354a6e53e06d227f3b1d291f1d2abb96e7112fd2ec42e146cea3d142eb5b333
images/other/conference-2021-summer-marathon-cover.jpg	b245b80a9d31a35df7c68012ee39f1c00b1e24c3	101,807	ad6bfe867d477d3f5415ce69620127060036900eb92cfdc625c1c27ab6f0dd83
images/partners/aicamp.png	921c9be8fb88e440fc81f05c2cd3e9e2269ebadd	34,290	690d35cd88677dd2cbb292ad23dc4bf17929125c565437d52e8632f76177acf4
images/partners/confetti.png	04854a28b8070eaf37854b664212cf634d50c193	10,859	720f4dca0dff44fdb1919f0ffe2ac52b7abf7a3c6e7c0be068a7c84bf6f66635
images/partners/dphi.png	891bd1efe77a5cd0e76fb3dc7413848422fc65c3	9,484	7cbfc95fc1c43f3c2a44dcb730de9dc5c61bcadd73508f8655a9f827e35f8373
images/partners/mlops-community.jpg	1551e92e19ccebf7476fa3ffbca03eb9319c3af8	10,863	32a3fa7eabe88f25bde91382d691b25f500b4befe440b152ce55a5cf9d4fc678
images/partners/odsc.png	c881cdfd70a5c69c5adfb37b9e9ae21ce3ad8005	17,435	e1641a1bbaf28ead0dc1edaa824d02ad6da8aa719b5299cdd19e216cdce8dbe9
images/partners/oreilly.jpg	13e7959e3bc8063c43c545af5817319631ba3e5f	11,815	21841b32991025990810707e5e97b84daf0ad57bcc386c2d5dd0474b9bcd0ee7
images/partners/pydata-london.png	76f636d791516c4f10ee57d824fcfcddd17af882	12,083	ae2fff85ac104b2d4058105bfd93368aee407e2d365d902937b8275eacd0d7f6
images/partners/thenewstack.png	7c85efa0a784ee39cad0a28f99d1772c3d32574c	15,884	b850bc6e59da36eb350fa0a2198de022d402ef720be68537715244a6ad90c478
images/partners/thesequence.png	0378326d4778663076df4991eaf41a7eaf2b097a	18,342	a58cf7c9ff06dfe6b0e7a5d4c4d619441074cb80446ce55418314ec1f7334d85

The 15 sorted manifest lines use exact UTF-8 path<TAB>git_blob_sha<TAB>byte_size<TAB>sha256<LF> bytes. Their SHA-256 is 8a7ebde5ea63116efc4fe3651439ac7dceeb724609e0055332cddf3f22c84b6c; total selected bytes are 314,358. The MLOps.community image is referenced twice but emitted once by source path.

Exact parser and output boundary

Implement a pure content_sync legacy-main adapter/contract. Its input is an exact verified checkout path plus the explicit commit/tree values. Its immutable output contains only:

• corpus provenance: repository, commit, tree, parser/schema version, selected-manifest digest, and counts;
• ordered ToolSourceRecord values: key (filename stem), source_path, source blob/SHA-256, title, name, description, one HTTPS github_url, ordered categories, one HTTPS demo_url, and ordered opaque maintainer_keys from who;
• ordered ConferenceSourceRecord values: key, provenance, title, description, normalized root-relative cover_asset_path, exact source layout="page", ordered tracks, ordered talks, ordered partners, and the checksum-bound legacy-body evidence described below;
• tracks preserve their source order and explicit schedule variant. 2021-feb uses track_date_talk_time (name, naive date lexeme, positive Eventbrite ID normalized to a decimal string, HTTPS YouTube URL, talks). 2021-summer-marathon uses track_window_talk_datetime (name, naive start/end lexemes, positive Eventbrite ID string, talks);
• talks normalize to name, exact source abstract_fragment, opaque speaker_key, optional source speaker_company, and only the fields present for their schedule variant: quoted HH:MM time plus optional HTTPS YouTube for February, or naive date, positive Eventbrite ID string, and optional HTTPS YouTube for summer;
• partners normalize to ordered name, HTTPS link, and root-relative image_asset_path; and
• AssetSourceRecord values sorted by source path with path, blob, byte size, SHA-256, detected MIME family (image/jpeg or image/png), and referencing record/pointer census. Asset bytes are not duplicated into normalized JSON.

YAML timestamps are parsed from their exact scalar lexemes and remain timezone-free strings; this issue does not infer UTC, Europe/Berlin, event state, or an Event row. Entity-looking title text such as – and the apparent source typo in Obsei's GitHub URL are preserved; the parser does not editorially repair or dereference them. Eventbrite IDs are opaque positive decimal strings, not database identifiers.

Stable output ordering is record kind then source path, source order within lists, and asset path byte order. A canonical evidence serialization uses UTF-8 JSON, sorted mapping keys, compact separators, ensure_ascii=false, and one trailing LF. Two parses of independently copied exact checkouts must be byte-identical. No current public-projection digest is an acceptance input and no output is written into content/public_projection.

Exact source census

The accepted result is exactly 2 tools, 2 conferences, 6 tracks, 26 talks, 10 partner entries, 12 asset references resolving to 11 unique assets, and 28 opaque Person references resolving to 27 unique source strings. The only repeated Person key is elenasamuylova, once in each conference. The ordered opaque keys are:

• tools: neallathia, lalitpagaria;
• February talks: philwinder, himanshuupreti, mahmoudaziz, eugeneyan, eliasnema, susanwalsh, elenasamuylova, danbecker, dannyma, admondleekinlim, parulpandey, vinvashishta, larysavisengeriyeva, mikiobraun, sararobinson, elleobrien;
• summer talks: svpino, dalianaliu, andreaskretz, roksolanadiachuk, elenasamuylova, janzawadzki, benwilson, linaweichbrodt, dougturnbull, fabianaclemente.

The adapter validates opaque keys only against the current Person-key grammar ^[A-Za-z0-9_][A-Za-z0-9._()-]{0,199}$ with .. forbidden. It does not read _people, resolve existence, infer by name/company/email/case, create relations, or decide required/optional Person behavior; #40 owns all of that.

Legacy body and fragment quarantine

The tool bodies are empty. Conference frontmatter is semantic source; the bodies are legacy Jekyll presentation programs and are not renderable input. They contain Liquid, inline HTML/style/event handlers, javascript: links, inline JavaScript, and a remote Eventbrite script. This issue neither evaluates, sanitizes, rewrites, copies, nor emits those bodies as HTML.

The sole reviewed exception is checksum-bound opaque evidence for the two exact bodies (bytes after the second delimiter line through EOF):

• 2021-feb: 3,111 bytes, SHA-256 336a008c02a4bdbb13a76ecf11358ef555270ff822ed797944cc3efbffe366c8;
• 2021-summer-marathon: 3,866 bytes, SHA-256 bcdcb8359fe49e66342566fd01a91735709ef876208d1f5c11a8bfc5ed717ac2.

Any body drift or any non-empty tool body fails closed and returns to PM; there is no generic executable-body allowlist. Frontmatter abstract strings may contain only text plus the observed inert fragment tags <ul>, <li>, <br>, and <br/>, with no attributes. Scripts, styles, event attributes, loading elements, javascript:/data/protocol-relative URLs, Liquid/Jinja delimiters, malformed tags, or any other element in a semantic string fail. Fragments remain source strings in this issue and are never marked safe or rendered; later presentation must sanitize/render from normalized data under its own issue.

Fail-closed limits and diagnostics

Read no more than the exact 15 selected files. Limits are 32 KiB per Markdown record, 128 KiB per asset, 512 KiB total, 24 KiB frontmatter, 8 KiB legacy body, YAML depth 16, YAML nodes 2,048, 64 tracks per conference, 64 talks per track, 32 partners, 20 diagnostics, 2,048 characters per URL, 200 per opaque key, 256 per title/name/company/category, 4,096 per description, and 16,384 per abstract. UTF-8 is required. YAML safe parsing forbids aliases, custom tags/object construction, merge keys, duplicate mapping keys, non-string mapping keys, non-finite numbers, unknown fields, and scalar coercion outside the exact schemas. Booleans are never integers.

Structured external URLs are absolute HTTPS without userinfo, control characters, backslashes, protocol-relative forms, fragments where not present in the source contract, or synchronous fetch. Asset references normalize one optional leading slash, must remain under exact images/other/ or images/partners/, and must resolve to the selected regular blob with exact size/digest and JPEG/PNG signature.

A rejected corpus produces no partial records/assets. Diagnostics are deterministic, path/pointer/code sorted, shaped only as {code, source_path, pointer}, bounded to 20 with a fixed overflow marker, and never contain source text, titles, abstracts, raw URLs, exception text, checkout absolute paths, credentials, account/Person data, or production data. Required fixed codes include:

• checkout/corpus: source_origin_mismatch, source_revision_mismatch, source_tree_mismatch, source_checkout_dirty, source_path_missing, source_path_unexpected, source_path_type_invalid, source_path_escape, source_file_size_invalid, source_corpus_size_invalid;
• framing/YAML/schema: source_utf8_invalid, frontmatter_invalid, yaml_alias_forbidden, yaml_duplicate_key, yaml_depth_exceeded, yaml_node_limit_exceeded, schema_field_unknown, schema_required_value_missing, schema_value_type_invalid, schema_limit_exceeded;
• values/references: stable_key_invalid, person_key_invalid, schedule_value_invalid, eventbrite_id_invalid, url_invalid, url_scheme_forbidden, abstract_fragment_unsafe, asset_path_invalid, asset_missing, asset_digest_mismatch, asset_type_invalid;
• quarantine/internal: legacy_body_digest_mismatch and fixed content-free legacy_main_parser_failed for an unexpected internal failure.

KeyboardInterrupt and SystemExit propagate. Parser state is invocation-local; a failure/retry cannot reuse partial output or a cross-run cache.

Acceptance criteria

• The exact origin/commit/tree, four regular record blobs, eleven regular asset blobs, 15-line manifest digest, sizes, SHA-256 values, and exclusion rule above are encoded in a versioned legacy-main contract and verified before parsing.
• One pure network-free adapter emits the exact typed schema and census: 2 tools, 2 conferences, 6 ordered tracks, 26 ordered talks, 10 ordered partners, 12/11 asset references/assets, and 28/27 opaque Person references/keys.
• Both legacy schedule variants normalize without timezone/Event inference; optional summer YouTube remains absent where absent; identifiers remain opaque; source order and exact source text are preserved.
• All eleven assets validate by path, regular-file boundary, byte size, digest, and JPEG/PNG signature. No other legacy-main asset is scanned or emitted.
• Conference bodies are accepted only by the two exact opaque body digests and are absent from renderable output. Tool bodies are empty. Semantic abstract fragments accept only the observed inert tag subset and are never marked safe/rendered.
• Person values are validated and emitted only as the 28 ordered opaque keys. There is zero _people read, Person resolution/relation, account/member lookup, fuzzy inference, or authority side effect.
• Two independent exact-checkout parses produce byte-identical canonical evidence and identical bundle digests. Source/tree/file/body/asset/schema/order drift and every adversarial case fail atomically with the exact bounded diagnostics.
• Tests prove the parser performs zero network/subprocess/provider calls and zero model/query/migration/file-write/runtime side effects; importing the module does not register a route, job, command, signal, source row, or setting.
• A current change-selective CI plan records exact base/head, graph/plan digests, and every disposition. uv-backed format/lint/type checks, focused real-corpus and synthetic security tests, graph-selected Django/Playwright/container gates, independent tester PASS, and PM acceptance precede commit.

Required test envelope

1. Parse two independently copied exact legacy checkouts and compare complete typed values plus canonical bytes/digest; assert every pinned path/blob/size/SHA, all exact counts, both schedule variants, optional fields, ordering, 28 opaque keys, and shared asset deduplication.
2. Mutate each source file, body, asset, source origin/SHA/tree/status, selected path set/type/case, ordering, scalar kind, required/unknown field, date/time/Eventbrite grammar, URL/path, opaque key, and collection/byte limit. Assert whole-corpus rejection, stable diagnostic code/pointer/order, no partial output, and no leaked value/exception/absolute path.
3. Cover duplicate YAML keys, aliases/merge/custom tags, depth/node bombs, invalid UTF-8, symlink/FIFO/directory/submodule/traversal, digest-valid wrong image signatures, URL userinfo/control/backslash/protocol-relative/unsafe schemes, dangerous fragment tags/attributes/Liquid, and diagnostic overflow.
4. Install hard-fail seams around network clients, subprocess, DNS/socket, Django ORM/model writes, filesystem writes, and runtime registration; exact parse must remain green and negative cases must not cross a seam.
5. Prove public URLConf, migrations, models, checked projection, sitemap/search/SEO output, templates/static assets, Studio/admin API/OpenAPI, jobs, and settings are byte-identical to the base.

This source-only issue has no render impact. Screenshots are not_applicable if the independently recomputed change graph confirms the strict boundary above; the applicable Playwright tier is backend-only smoke. Any public artifact/template/route/runtime impact is scope drift and returns to engineering/PM rather than being covered with unrelated screenshots.

Dependencies and delivery order

Satisfied inputs: #12, #34, #35, #37, and #226. Parent: #39.

There is no product dependency on #38, #40, or #253 for this four-record source-only parser. It neither uses the disputed #253 projection nor resolves Person keys. However, engineering starts only from a clean current base with required baseline CI green under _docs/PROCESS.md; a red or dirty shared main is an operational hold, not permission to weaken this contract.

Downstream order:

1. implement/test/accept this issue as the isolated source parser;
2. repair/accept Reconcile non-reproducible public projection after face8e4 #253, then finalize/accept Establish canonical Person short source and relationship resolution #40;
3. separately re-groom Complete the remaining legacy-main content adapter and route parity #39's Person-resolved tool/conference relationship and public render/route/parity slice against the accepted parser and exact Establish canonical Person short source and relationship resolution #40 output;
4. Adopt secure direct-sync content ingestion and management #38 later owns direct-sync registration/execution and any public database-authority cutover with its separate HUMAN authority manifest.

Any source revision, selected corpus, schema, asset, body, or architecture drift returns this issue to PM before engineering continues.

Explicit non-goals

No public tool/conference hub or detail route, alias, canonical, sitemap/search/SEO entry, template, navigation, asset serving, checked projection change, Person lookup/relation, Event/Course import, model, migration, database read/write, source registration, direct-sync orchestration, webhook/reconciliation, Studio/admin API/OpenAPI, job, provider/network call, source/GitHub write, production access, browser UI, content editorial repair, legacy Liquid/JavaScript execution, commit, push, merge, deployment, or #253 hash blessing.

Lifecycle

This body is the PM-groomed product/architecture contract. Engineering may be dispatched only after the orchestrator supplies the clean green current-base gate. The engineer works uncommitted in an isolated worktree, posts the complete verification handoff, and freezes it for a separate tester. Tester PASS and PM acceptance remain mandatory before any focused commit or local merge.

[alexeygrigorev]

PM grooming complete — source-only parser lane

Disposition: engineer-ready product contract, subject to the process-wide clean/green current-base gate. needs grooming has been removed.

Read-only source inspection bound the complete legacy-main corpus at ee43d3fa0929faf691178d79f19528e6f15a83e5 / tree 1666d11893a9f001e916409f3bac65a9ef0ff072: four regular Markdown records, eleven unique regular image assets, 314,358 selected bytes, and selected-manifest SHA-256 8a7ebde5ea63116efc4fe3651439ac7dceeb724609e0055332cddf3f22c84b6c.

The accepted normalized census is exact: 2 tools, 2 conferences, 6 tracks, 26 talks, 10 partner entries, 12 references to 11 assets, and 28 opaque Person references containing 27 unique keys. The two conference bodies are checksum-bound opaque legacy presentation evidence only; their Liquid/HTML/inline JavaScript/Eventbrite code is never emitted, sanitized, evaluated, or rendered. #40 retains Person resolution and #39 retains all route/render/parity work.

The body now specifies exact paths/blobs/sizes/digests, both legacy schedule schema variants, immutable output ordering/serialization, source-text and asset boundaries, strict parser limits, fixed content-free diagnostics, two-run determinism, adversarial/security coverage, and zero route/runtime/model/migration/network/write side effects. Screenshots are not_applicable only under an independently recomputed no-render graph; backend-only smoke remains applicable.

No repository file, checkout, source, generated artifact, model/database, route, test, browser, provider, production state, commit, push, merge, or deployment was changed during grooming.