Two Failures From My Own RAG Demo

This case study comes from the live RAG demo on arupdas.com, not slideware, a system visitors can actually use. Upload a Markdown file, ask questions, get cited answers from Qdrant hybrid retrieval plus Bedrock generation. I am putting two of its failures here on purpose, because debugging my own demo in public is more honest than a clean architecture diagram.

The first was a retrieval-quality miss. During testing, a user uploaded a short story, confirmed broad questions worked, then asked about "Narrow Gate." The answer was right there in the file, Thomas Reeve's boat, but the system said it had no context. The upload was indexed and the vector store had the chunk; the old dense-only semantic path had simply ranked portfolio chunks higher, because "Narrow Gate" out of context looks nothing like a boat in embedding space. That is not a deletion bug or a UI bug, it is a lesson. Semantic search is excellent for meaning and paraphrase and weaker for rare proper nouns, codes, and entity names that appear once inside a long chunk. I shipped pragmatic mitigations first, reserved slots for upload chunks, a corpus-busy lock during indexing and deletion, session-scoped history aligned with the UI, and then shipped the production fix: Qdrant hybrid search, Titan dense vectors plus BM25 sparse vectors fused at query time.

The second failure is more recent and I took three wrong turns through it. The goal was to let the demo answer questions about me, my USP, my strengths, why someone should hire me, not just questions about individual projects. My first instinct was to write one confident "about me" document and index it. It worked on the surface and was bad retrieval: that summary was such a strong match for any query about me that it dominated, and every answer cited a single self-congratulatory page and zero real projects. For an audience evaluating a retrieval engineer, watching the system pull one bio doc is the opposite of convincing.

So I made the bio a thin map that points to the real projects and told the agent to gather evidence across several and cite each, textbook model-driven multi-hop. It worked beautifully for "why hire me, give the top reasons" and was unreliable everywhere else: asked for my "USP," the same system retrieved the map, decided it could not find a document literally about "USP," and gave up. The model's willingness to decompose a question depended on its phrasing, which is not something you ship to strangers. Along the way I also found that a long conversation could condition the agent to skip retrieval entirely and answer from memory, and that a sloppy regex silently failed to match a plural, both caught only because the trace and evals made the behavior visible.

The fix was to stop trusting the model to multi-hop and make the system do it: the bio became a router that holds no conclusions of its own, profile questions deterministically fan out across my projects and synthesize a cited answer, and a guardrail forces a retrieval if the model ever tries to answer without one. It is the opposite trade-off from the research-grade approaches, less elegant, more reliable, and profile questions now cost a few extra retrievals, which the cost-and-latency readout shows rather than hides. The lesson that ties both failures together: a document or a model that takes a shortcut will quietly collapse your answers, and the only reason I caught either was that I had built the system to show its own work.