Sync podcast source pages and expand wiki graph

Author: alexeygrigorev

AGENTS.md

@@ -50,8 +50,10 @@ publish separate `Archive Evidence`, `Guest Descriptions`, `Maintenance Notes`,
 `Episode Evidence`, `Recurring Archive Themes`, `Contents`, or `Search Intent`
 sections in reader-facing pages.
 
-When the source episode is known, link to the specific podcast page:
-`https://datatalks.club/podcast/<source-file-slug>.html`. Use
+When the source episode is known, link wiki and article pages to the local
+podcast page: `{{ '/podcasts/<source-file-slug>/' | relative_url }}`. That
+local page links to the original
+`https://datatalks.club/podcast/<source-file-slug>.html` episode. Use
 `https://datatalks.club/podcast.html` only as a temporary fallback when the
 specific episode slug is unknown.
 
@@ -77,11 +79,17 @@ and replace them with podcast-backed synthesis when the topic becomes important.
 4. Update the target exploration page with synthesized takeaways, not just lists
    of links.
 5. Add cross-links to related wiki pages, articles, people, and podcast summaries.
-6. Add podcast evidence links in the body. Public episode links should point to
-   the specific `https://datatalks.club/podcast/<slug>.html` page when known.
-7. For graph/search changes, run `make graph` and
+6. Add podcast evidence links in the body. Use local
+   `/podcasts/<source-file-slug>/` links when the source episode is known.
+7. For source-derived podcast and people pages, run `make sources`.
+8. For graph/search changes, run `make graph` and
    `python scripts/build_search_index.py`, or simply run `make check`.
 
+For broad podcast-topic work, first extract episodes, people, chapter summaries,
+and topic candidates with `make sources`. Then keep five subagents running on
+non-overlapping episode batches. Subagents should produce grounded topic reports
+with local podcast links and guest links before writing wiki or article pages.
+
 ## Insight Hub Target
 
 Insight pages should follow the shape requested in DataTalksClub issue #111:
@@ -147,9 +155,12 @@ transcripts.
 The graph UI lives at `graph.md` and `assets/graph.js`. It should remain static:
 use `graph/graph.json`, avoid external runtime dependencies, and keep node links
 stable through graph URL hashes. `graph/graph.json` is generated from collection
-frontmatter and internal links by `scripts/build_graph.py`. Episode nodes should
-link to local podcast summaries when available and otherwise to
-`https://datatalks.club/podcast.html`.
+frontmatter and internal links by `scripts/build_graph.py`; do not maintain it
+as separate editorial content. Run `make sources` to sync source-derived
+podcast and people pages before graph generation. The source documents are
+`_wiki/`, `_articles/`, `_people/`, and `_podcast_summaries/`. Episode nodes
+should link to local podcast pages, which then link to the original
+DataTalks.Club podcast pages.
 
 Run `python scripts/check_links.py` after a static build to validate generated
 internal links. GitHub Pages runs the same checker with the deployed base path.

CONTENT_GUIDE.md

@@ -20,8 +20,10 @@ Do not publish maintenance notes, agent instructions, evidence appendixes, guest
 description appendixes, or SEO scaffolding such as `Search Intent` sections.
 Use inline references in the relevant section, like Wikipedia citations.
 
-When the source episode is known, link to the specific podcast page:
-`https://datatalks.club/podcast/<source-file-slug>.html`. Use
+When the source episode is known, wiki and article pages should link to the
+local podcast page: `{{ '/podcasts/<source-file-slug>/' | relative_url }}`. The
+local podcast page links to the original
+`https://datatalks.club/podcast/<source-file-slug>.html` episode. Use
 `https://datatalks.club/podcast.html` only when the specific episode slug is not
 known yet.
 
@@ -75,6 +77,7 @@ knowledge pages built from the podcast archive.
 Podcast summaries live in `_podcast_summaries/`. They are not replacement podcast
 pages and must not copy full transcripts. Their job is to help future agents decide
 whether to open the original source episode in `../datatalksclub.github.io`.
+They are also the internal citation targets for wiki and article pages.
 
 Required structure:
 
@@ -84,6 +87,29 @@ Required structure:
 - useful-for / probably-skip-if guidance for agents
 - source pointers back to `https://datatalks.club/podcast.html` and source files
 
+Generate or refresh these local podcast pages with
+`python scripts/sync_podcast_pages.py`. The source of truth is still
+`../datatalksclub.github.io/_podcast`.
+
+Before topic writeups, run `make sources`. This extracts podcast episodes,
+people, chapter summaries, and topic candidates into Markdown pages and
+`artifacts/podcast/source-index.json`. Agents should use that index and the
+local `/podcasts/<slug>/` pages as their first pass before opening a full
+episode transcript.
+
+For large topic discovery work, keep five subagents running in parallel. Split
+the source episodes into non-overlapping batches. Each subagent should return
+candidate topics, local podcast links, guest links, and chapter evidence before
+any wiki or article page is written.
+
+## Graph Source
+
+The graph is derived from Markdown documents, not maintained as editorial
+content. Wiki pages, articles, people pages, and local podcast pages are the
+source. `graph/graph.json` is generated by `python scripts/build_graph.py` and
+should not be hand-edited. Run `make sources` before `make graph` when podcast
+or people source files changed.
+
 ## People Pages
 
 People pages live in `_people/`. They collect guest/contributor information from

CONTENT_TODO.md

@@ -36,15 +36,24 @@ These notes capture the current cleanup direction and should not be lost.
   collecting evidence in a separate appendix.
 - Make every page link-heavy: visible links to specific podcast interviews,
   related wiki pages, relevant articles, and useful local podcast summaries.
-- Use specific podcast interview URLs, not only
-  `https://datatalks.club/podcast.html`, whenever the source episode slug is
-  known.
+- Use local podcast links such as
+  `{{ '/podcasts/<source-file-slug>/' | relative_url }}` whenever the source
+  episode slug is known. The local podcast page links to the original
+  DataTalks.Club episode.
 - Make related links visually obvious in CSS. Related pages should look like
   links, not muted tags.
 - Add a graph-driven "See also" or related graph nodes section to wiki and
   article pages, especially stub pages such as `_wiki/tools.md`.
-- Change `/podcasts/` into a list view closer to the original DataTalks.Club
-  podcast structure, not a generic card/grid page.
+- Keep `/podcasts/` as a list view closer to the original DataTalks.Club
+  podcast structure. The pages in `_podcast_summaries/` are the internal
+  citation targets for wiki/articles.
+- Treat Markdown pages as the source for the graph. Do not maintain
+  `graph/graph.json` separately; regenerate it from the collections.
+- Keep `make sources` as the first step for broad podcast work. It syncs local
+  podcast pages, people pages, chapter summaries, and the source index used by
+  subagents.
+- For broad topic discovery, keep five subagents running on non-overlapping
+  podcast batches and collect grounded topic reports before writing pages.
 - Keep MLOps and DataOps as separate concept pages. Use `_wiki/mlops.md` for
   model lifecycle operations, `_wiki/dataops.md` for data delivery operations,
   and `_articles/mlops-vs-dataops.md` for the comparison.

Makefile

@@ -1,9 +1,14 @@
-.PHONY: help graph index lambda-package build serve links clean check
+.PHONY: help sources graph index lambda-package build serve links clean check
 
 help:
 	@grep -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | awk 'BEGIN {FS = ":.*?## "}; {printf "  %-12s %s\n", $$1, $$2}'
 
-graph: ## Build the static graph data used by the site
+sources: ## Sync source-derived podcast and people pages for graph/search
+	python scripts/sync_podcast_pages.py
+	python scripts/sync_people_pages.py
+	python scripts/extract_podcast_sources.py
+
+graph: sources ## Build the static graph data used by the site
 	python scripts/build_graph.py
 
 index: graph ## Build the zerosearch artifact used by Lambda