open-energy-rag

Demo

OpenEnergy-compressed.mp4

RAG assistant for Brazilian electric sector regulations (PRODIST/ANEEL, ONS, MME). Every answer cites document, section and item. 100% public dataset.

Status: Phase 2 closed. 12/12 PDFs ingested (1065 chunks), hybrid dense + sparse retrieval (RRF) enabled, benchmark published at evals/results/latest.md. Full roadmap in docs/ROADMAP.md.

Why it exists

1. Bridges two lines of work: IFSC academic research on multi-agent systems for the electric sector and production-grade hybrid RAG.
2. Underexplored domain: almost no public repository does RAG on Brazilian sector regulation.
3. Citation-first: no citation, no answer.

Benchmark

25 curated questions, ground truth versioned in evals/questions.yaml. Both modes share the same Qdrant collection (dense vector e5-large + sparse vector BM25 in named vectors); only the search strategy changes.

Metric	Dense-only	Hybrid (RRF)	Δ
recall@1	0.720	0.760	+0.040
recall@3	0.800	0.840	+0.040
recall@5	0.840	0.880	+0.040
recall@10	0.920	1.000	+0.080
MRR	0.782	0.819	+0.037

Hybrid beats dense on every metric. Recall@10 hits 100% (the gold chunk appears in the top-10 for every question in the eval set). Full per-query report at evals/results/latest.md. Reproduce with cd backend && uv run python -m src.eval.runner.

Stack

Layer	Choice
Backend	FastAPI + async
Vector DB	Qdrant (named vectors: dense + sparse, server-side RRF)
Dense embeddings	intfloat/multilingual-e5-large via fastembed (query: / passage: prefixes applied)
Sparse embeddings	Qdrant/bm25 via fastembed (BM25, language-agnostic)
PDF extraction	docling
LLM	Groq (Llama 3.3 70B) by default. OpenRouter, Gemini, OpenAI and Ollama configurable via .env
Observability	Langfuse (optional)
Eval	in-house harness: recall@k, MRR, metadata-based matching

Deliberate decision: no LangChain. Done well, RAG is a small amount of boilerplate; over-abstracting gets in the way of understanding what retrieval is actually doing.

How to run

Prerequisites: Docker, Python 3.12, uv.

1. Qdrant

docker compose up -d qdrant

Available at http://localhost:6333.

2. Backend

cd backend
uv sync
cp .env.example .env
# fill in GROQ_API_KEY (or another provider) in .env

uv run python run.py

API at http://localhost:8000. Docs at /docs.

3. Ingest the PDFs

PRODIST and ONS PDFs live under data/raw/ (gitignored). To repopulate the collection:

cd backend
uv run python -m src.ingestion.cli ../data/raw/*.pdf

Ingestion converts each PDF via docling (caching the markdown in data/processed/), chunks it preserving section/item, embeds dense + sparse and upserts into Qdrant.

4. Ask

curl -s -X POST http://localhost:8000/ask \
  -H "Content-Type: application/json" \
  -d '{"query":"Como classificar a tensão de atendimento?","top_k":4}' | jq

The response includes answer (generated by the LLM) and sources (chunks used, with metadata).

5. Endpoints

Method	Endpoint	Notes
GET	/health	liveness, returns active provider + model
POST	/search	top-k chunks; accepts "mode": "hybrid" (default) or "dense"
POST	/ask	retrieval + generation with citations; same mode

6. Reproduce the benchmark

cd backend
uv run python -m src.eval.runner
# report at evals/results/<timestamp>.md and evals/results/latest.md

7. Frontend (optional)

A Next.js 15 single-page UI lives under frontend/. Citation-first chat, dark/light aware, calls the backend through a server-side proxy so the API URL stays out of the browser bundle.

cd frontend
pnpm install            # or npm install
cp .env.example .env.local   # BACKEND_URL=http://localhost:8000
pnpm dev                # http://localhost:3000

The backend already serves CORS for http://localhost:3000 by default (configurable via CORS_ALLOWED_ORIGINS).

Layout

open-energy-rag/
├── backend/
│   ├── src/
│   │   ├── api/          FastAPI: /search, /ask, /health
│   │   ├── config/       pydantic-settings
│   │   ├── ingestion/    docling → structural chunker → embed → upsert
│   │   ├── retrieval/    dense + sparse embeddings, qdrant hybrid + RRF
│   │   ├── llm/          provider abstraction (Groq / OpenRouter / Gemini / OpenAI / Ollama)
│   │   └── eval/         metrics (recall@k, MRR) + benchmark runner
│   ├── pyproject.toml
│   ├── run.py
│   └── .env.example
├── frontend/
│   ├── src/
│   │   ├── app/          Next.js App Router (page + /api/ask proxy)
│   │   ├── components/   Header, QuestionInput, AnswerCard, SourceCard, ...
│   │   └── lib/          types and fetch wrapper
│   ├── package.json
│   └── .env.example
├── evals/
│   ├── questions.yaml    curated eval set (25 queries)
│   └── results/          generated benchmark reports
├── data/raw/             [gitignored] raw PDFs
├── data/processed/       [gitignored] markdowns produced by docling
├── docs/                 ROADMAP
└── docker-compose.yml    Qdrant

Known limitations

• Tables inside PDFs are extracted but the structural chunker does not preserve them as blocks: their content is folded into the surrounding text.
• The chunker only records the first numbered item of each chunk in metadata.item. Adjacent glossary items end up grouped in the same chunk; the eval set's ground truth handles this by using items: [] when it happens.
• No reranker in the MVP. The roadmap includes testing bge-reranker-v2-m3 over the hybrid top-20.
• The LLM only answers in PT-BR and strictly follows citation-first: with no information in the context, it returns "Não encontrei isso nos documentos indexados.".

License

MIT (to be confirmed before public release).