Skip to content

readme.md

Latest commit

 

History

History
25 lines (16 loc) · 1.6 KB

readme.md

File metadata and controls

25 lines (16 loc) · 1.6 KB
Raw

Parsing (ETL) for Unstructured and Semi-Structured Sources

This directory contains parsers that convert intermediate source formats into LlamaIndex Document objects with consistent metadata.

High-level pipeline:

  1. Parse source artifacts into an intermediate representation (for PDFs, typically LlamaParse JSON).
  2. Chunk and normalize content into Document objects (with provenance metadata like source, page_number, bbox).
  3. Optionally emit specialized Document types (for example type="table" for CSV table chunks) that downstream KG extractors can consume.

For the knowledge-graph extraction step, see packages/src/ingestion/kg_extractors/README.md.

Unstructured PDF parsing/extraction (high level)

  • pdf_parser.py: generic LlamaParse JSON loader that emits Document chunks with basic provenance metadata.
  • form1040_parser.py: IRS Form 1040 instructions parser that creates section-level documents and emits type="table" docs when a table is encountered.
  • title26_parser.py: Title 26 parser that extracts legal hierarchy markers (section, subsection, paragraph, etc.) and writes hierarchy metadata used downstream by Title26HierarchyExtractor.

Title 26 and Form1040 parsing is documented in-depth separately to avoid duplication:

Structured parsing

  • tax_history_csv_parser.py: parses tax_data.csv into row-level narrative documents and aggregated summary documents using strict Pydantic models (see packages/src/ingestion/models/csv_data_model.py).