POC exploration
Frameworks Are Not the Architecture: Testing LangGraph Against a Production RAG Baseline
5 runnable phases · production hybrid RAG is the control
I built this lab because job descriptions kept asking for LangChain and LangGraph. The awkward part was that I had already shipped the underlying ideas: query rewrite, a bounded agent loop, deterministic profile fan-out, groundedness checks, and evals. I knew the work. I did not have the framework names attached to it.
So I rebuilt the system in Python, one phase at a time. The easy version would compare every new technique with dense cosine retrieval and declare a win. That baseline would be convenient and false. The live portfolio had already moved to Qdrant hybrid retrieval, Titan dense vectors plus BM25 sparse vectors fused with reciprocal-rank fusion. If corrective RAG or GraphRAG could not beat that where it mattered, the extra graph was just extra machinery.
Hybrid became the control. Nova Pro stayed as the generator, which meant a score could move only when retrieval, routing, or orchestration changed. The lab starts with plain RAG, expresses the same flow as a LangGraph state graph, adds corrective retrieval, then builds a small entity graph and an adaptive router. The last phase adds reranking and a scorecard against production. Each phase has runnable code, notes, and an artifact that shows the active line, input, output, full state, and state change.
Cost had to stay visible. Grading a handful of retrieved documents can multiply model calls before the user sees one answer. Nova Lite handles classification work while Nova Pro writes the response. Embeddings and reranking run locally by default, and NetworkX is enough for a 16-file corpus. Starting with a hosted graph database would have taught me more about operating a database than about whether graph retrieval helped.
LangGraph gave familiar ideas stable names and a useful execution model. It did not replace architecture judgment. CRAG still has to earn its grading cost. GraphRAG still has to improve the queries that need relationships. The framework can show the path through a system; it cannot decide whether that path deserves to exist.
The decision
Use the live portfolio's Qdrant hybrid retrieval as the control, hold generation constant, and attribute score changes to retrieval or orchestration rather than a stronger model.
Alternatives considered
- ·Compare against dense cosine retrieval. Easy to beat, but irrelevant because that stack is no longer the production baseline.
- ·Upgrade the generator while adding the graph. That would improve answers while making it impossible to say which change helped.
- ·Start with a hosted graph database. Too much operational weight for a 16-file learning corpus.
Tech stack
The challenge
Learn LangChain, LangGraph, CRAG, and GraphRAG without creating a toy benchmark that makes every new technique look better than it is.
Architecture approach
- ·Built five runnable phases: vanilla RAG, LangGraph, corrective RAG, GraphRAG, then reranking and evaluation.
- ·Kept Nova Pro as the generator while cheaper models handled grading, routing, and extraction.
- ·Used local embeddings and an in-memory graph by default so learning runs did not quietly become an infrastructure bill.
- ·Traced the phases in Langfuse and created interactive artifacts that expose code, state, and state changes step by step.
Results
- A runnable lab with the same corpus shape and questions as the production system
- A fair comparison where hybrid retrieval is the control, not the strawman
- Reproducible phases that separate framework vocabulary from architecture judgment
What I'd do differently at production scale
- ·Do not promote CRAG or GraphRAG because the diagram looks more advanced. Promote it only when the eval moves on the question class it is meant to fix.
- ·Track the cost of grading per document; corrective loops can multiply calls faster than the final answer suggests.
- ·Move beyond NetworkX only when corpus size, update frequency, or multi-hop latency earns a persistent graph store.