Invoice RAG Agent
IN PROGRESSDescription
Built a local-first invoice extraction and RAG Q&A agent that turns PDF invoices into Pydantic-validated structured data and answers cross-invoice questions with source citations, keeping all financial data on-device. A deterministic LangGraph pipeline (extract → validate → retry → review) constrains a small local LLM: arithmetic and schema failures trigger an automatic re-prompt, and results that still can't be verified hard-stop into a human-review queue instead of being trusted silently. A scored eval harness measures Straight-Through Processing rate and per-field accuracy against explicit correctness bars (60–80% STP, ≥ 95% on money fields) over a synthetic multi-currency invoice set, and validated invoices are indexed into Chroma for filtered, cited retrieval.
Highlights
- —Local-first invoice extraction + RAG Q&A agent: PDF invoices become Pydantic-validated structured data, and cross-invoice questions are answered with source citations — all financial data stays on-device.
- —Deterministic LangGraph pipeline (extract → validate → retry → review) constraining a small local LLM, auto re-prompting on arithmetic and schema failures.
- —Unverifiable results hard-stop into a human-review queue rather than being silently trusted — the agent never emits a number it couldn't validate.
- —Scored eval harness measuring Straight-Through Processing rate and per-field accuracy against defined correctness bars (60–80% STP, ≥ 95% on money fields) over a synthetic multi-currency invoice set.
- —Validated invoices indexed into Chroma for filtered, cited RAG Q&A across the whole invoice set.
Architecture
PDF invoice → LangGraph extract node (small local LLM via LM Studio) → Pydantic schema + arithmetic validation → bounded retry loop that re-prompts on failure → human-review queue for anything still unverifiable. Validated invoices are embedded and indexed into Chroma, where filtered retrieval backs a cited RAG Q&A layer. A scored eval harness gates the pipeline on Straight-Through Processing rate and per-field accuracy (60–80% STP, ≥ 95% money fields) against a synthetic multi-currency invoice set.