DataOps

DataOps is the combined DataTalks.Club operations portal.

Version 1 focuses on operations docs and tasks:

• process docs, SOPs, templates, references, playbooks, prompts, and search
• task workflows, bundles, recurring work, required links, and execution state
• AWS Lambda deployment with GitHub Actions OIDC

The deployed V1 app is the DataTalks.Club operations portal for ops.dtcdev.click, served from the DataTalksClub/dataops repository and the dataops-v1 stack. The single backend lives under backend/ and serves the frontend, docs, search, and work APIs from one TypeScript Lambda, and the podcast assistant lives under assistants/podcast/ as a DataOps assistant workflow.

Layout

• content/ — operational documentation (SOPs, templates, references, playbooks, prompts) and its image assets.
• backend/ — single TypeScript backend: docs content API, search, work execution, portal/auth, and assistant-job orchestration.
• assistants/podcast/ — DataOps podcast workflow assistant module, process docs, guest-intake template, knowledge-base builder, and tests.
• docs/ — repo-meta docs (this README, STRUCTURE.md, sop-format*.md, archived materials).
• _docs/ — DataOps merge plan, development process, and planning notes.
• frontend/ — static vanilla-JS app and its Dockerfile.
• infra/ — SAM/CloudFormation templates, OIDC deploy roles, runtime secrets.
• tools/content_tools/ — pure-Python content validation CLIs (docs link validation, knowledge-repo validation, process-quality reports).
• scripts/ — repo tooling (planning-docs validator).

Architecture

DataOps is a portal served by a single TypeScript backend Lambda. GitHub is the source of truth for all markdown content; execution state lives in DynamoDB.

Component	Path	Runtime	Responsibility
Frontend shell	frontend/	Static vanilla JS	SOP block editor, search UI, filters, publish dialog. No build step; served as static assets.
Backend	backend/	TypeScript / Node (Lambda)	Single backend: serves the frontend, docs content API, zerosearch-node search, work execution, and GitHub-backed content editing from one Function URL.
Podcast assistant	assistants/podcast/	Python module	Guest-intake, knowledge-base builder, and draft generation for the podcast operations workflow.

flowchart TB
  Browser[Browser]

  subgraph Portal
    Frontend[Static frontend shell]
    Backend[Backend Lambda - TypeScript]
  end

  Podcast[Podcast assistant - Python module]

  GitHub[(GitHub repo: content/ markdown)]
  Dynamo[(DynamoDB: execution state)]
  Secrets[(AWS Secrets Manager)]

  Browser -->|Basic auth| Backend
  Backend --> Frontend
  Backend -->|docs API + zerosearch-node| Index[/Lambda /tmp cache + index/]
  Backend -->|GitHub Contents API| GitHub
  Backend --> Dynamo
  Podcast -->|drafts + knowledge base| GitHub
  Portal -->|runtime secrets| Secrets

Loading

How the pieces fit:

• Content is GitHub-backed. Edits made in the UI are committed straight to GitHub by the backend through the Contents API; the Lambda keeps a /tmp cache and builds a zerosearch-node index from it. No SQLite, no EFS.
• One backend serves everything. The frontend, docs API, search, and work APIs run in a single TypeScript Lambda with a single Function URL. The old three-process split (frontend proxy + Python docs Lambda + work engine) is gone. Local dev is make dev (one origin on port 3000).
• The podcast assistant runs as a Python module, not a web service — it drives the podcast intake/drafting workflow and writes into the knowledge base.
• Infra is SAM/CloudFormation (templates under infra/), deployed by GitHub Actions through an AWS OIDC role; runtime secrets live in AWS Secrets Manager rather than GitHub Actions secrets.

See _docs/TARGET_ARCHITECTURE.md for the target layout and design rationale.

Planning

• Portal Analysis
• Shared Project Plan
• Merge Plan
• Development Process

Running locally

Local dev is a single backend that serves the frontend + docs + work APIs from one origin (port 3000):

make setup
make dev

Development commands

make help lists the current setup, dev, test, SOP lint, SAM validation/build, and CI-parity targets. These targets are thin wrappers around the package-local commands documented below so failures stay visible and package commands remain useful for troubleshooting.

Common verification targets:

make seed-backend
make test-backend
make typecheck-backend
make build-backend
make test-backend-e2e
make test-assistant
make sam-validate
make sam-build
make ci

Run make sop-lint FILES="content/path/to/sop.md" for marked SOP files. make sam-validate is local template validation only: it uses empty AWS config and credentials files under .tmp/aws-empty/, disables EC2 metadata lookup, and does not require live AWS credentials or run sam deploy.

Sponsor CRM’s four indexes are added one at a time by a separate protected, manual migration workflow. The normal application workflow never performs that migration: immediately after OIDC credentials and before the deployment build, a read-only guard requires the exact retained table identity, CloudFormation ownership, and all four canonical indexes to be active with no backfill. If they are incomplete, deployment stops and directs the operator to dispatch the protected migration first.

Node workspace

DataOps uses npm workspaces from the repository root. The single workspace is backend/.

Install Node dependencies from the repo root:

npm ci

Common backend lifecycle commands are available from the root:

npm run dev:backend
npm run test:backend
npm run test:e2e:backend
npm run typecheck:backend
npm run build:backend
npm run seed:backend
npm run export:templates:backend
npm run validate:export:backend -- <export-dir>
npm run dry-run:import:backend -- <export-dir>
npm run clean:backend

package-lock.json at the repo root is the npm lockfile for the workspace. The backend Lambda packaging, CI cache, and Docker Lambda image all use that root lockfile; there is no nested backend/package-lock.json.

What the editor does

• Opens every SOP in a block view — Section / Group / Step / Free-form / Screenshot / TODO. Click any text to edit inline; Cmd/Ctrl+Enter to commit.
• Add, delete, drag-reorder steps; cross-group moves; convert flat ↔ grouped procedures; renumber automatically. Roundtrips through the SOP linter.
• Image upload via file picker, drag-and-drop onto a step, or clipboard paste.
• Frontmatter editor: doc_type, summary, tags, systems (chips).
• Pending-changes panel aggregates every local draft; Save all from the sidebar, then review lint and commit from the publish dialog.
• Search (server-side), tag/system/domain/type filters, quick-nav palette (Cmd/Ctrl+P), sidebar tree filter, "Recently edited" + "Pinned" sections.
• Diff view between draft and saved version. Lint dashboard for the whole corpus in the publish dialog. Loom + YouTube + Vimeo embeds. Lightbox for screenshots.
• Dark mode, resizable sidebar, mobile layout.

Keyboard shortcuts

• / focuses sidebar search.
• Cmd/Ctrl + K focuses search from anywhere.
• Cmd/Ctrl + P opens quick navigation.
• Cmd/Ctrl + S saves the current doc.
• Cmd/Ctrl + Shift + S saves all drafts.
• Cmd/Ctrl + Enter commits an inline edit.
• Esc cancels inline edits and closes modals.
• ? shows keyboard shortcut help.

SOP format

Every SOP is structured-markdown with HTML-comment markers (<!-- sop-section-start: ... --> etc.) — invisible on GitHub but machine-readable. See docs/sop-format.md for the strict spec and docs/sop-format-design.md for the design log.

The Node CLI (backend/scripts/sop.ts) provides parse and lint subcommands; the content-validation Python tools live in tools/content_tools/.