Production Target

Both backends also keep a server-side Ask transcript today, viewable while a session is live and after it ends: EKS reuses the per-run store it already had, and the Bedrock console gets its own durable chat store in this app. In Chat mode, that durable transcript IS the model's conversational context now (Ops' Chat / Single question toggle makes this real and switchable per question) - the RAG API Lambda's own short-term session memory is skipped entirely rather than combined with it, which is what keeps a resent history from being billed twice. Single question mode reads and writes neither store, for a genuinely memoryless turn.

A retention tension worth naming: chat turns are kept for a year, while uploaded documents expire after 24 hours (ADR-005's "don't leave data lying around" reasoning does not automatically extend to a year of visitor questions). See ADR-015.

The live quota panel is deployed and verified: the browser subscribes directly to AWS IoT Core over MQTT/WSS for the remaining-questions count, so no Vercel function is held open for the life of a tab. Proven by subscribing as a browser would, a retained message arrived immediately on connect showing 15 remaining, and a push arrived showing 14 while a query ran. One AWS-specific detail was worth the trouble it caused: IoT Core rejects a SigV4 signature computed over the session token, so the token is appended after signing. Until that was found the URL looked correct in every respect and the socket simply closed. See ADR-016.

Sets from fields, stories from retrieval

A visitor asked this demo "what are the different projects Arup has contributed in?" and it refused, over a corpus that is almost entirely his projects. Three causes stacked behind that one refusal, and each was hiding the next. The corpus survey (ADR-017) exists to answer exactly that shape of question by enumerating every document rather than searching, and it never fired: the detector matched a noun list that stopped at "skills", so "projects" fell through to ordinary top-k. Asked with the word "skills" instead, the same system answered correctly. One absent noun separated a full corpus survey from a refusal.

With that fixed the survey fired, and exposed the deeper problem. It handed the model all fourteen document summaries as prose and let it work out what each one was. The answer listed "AI Certifications and Formal Training" and "AI Architecture Patterns" as projects, and omitted three production wins. Nothing in the payload distinguished work delivered from material about the practice, so the model guessed, and guessed badly.

Typed fields were the obvious repair: docType, domains and techniques derived at ingestion and written into the vector payload as indexed, filterable keys. It was also not enough, twice over, and both discoveries were worth more than the repair.

The corpus was the wrong size. /work rendered 22 projects at the time (the list has since grown). The corpus had documents for five. The demo was answering from roughly a quarter of the real work, and no retrieval strategy fixes an index that does not contain the answer. It had been written by hand alongside content the site already held as typed data, and the two drifted: the corpus technology list for the migration omitted Rails, PostgreSQL and Redis that /work had all along. So the corpus is now derived from /work rather than maintained beside it: the site publishes its projects as JSON, the seed reads it, and projects with a hand-written narrative keep it while the rest get a generated spine. Fourteen documents became 27. Presence on /work is also what decides whether something is a project, which retired a rule that had been guessing from filename prefixes.

Naming a type is not enforcing it. With every document typed and in the prompt, the model still answered with reference material listed as projects. A field the model is asked to respect is a suggestion. So membership stopped being its job: a question asking which projects, technologies or domains is now answered by computing the set from the indexed fields in code, and the model only describes what it is handed. Sets come from fields, stories come from retrieval. The trace says enumerate rather than survey when that happens, because a field read is not a search and should not be dressed as one.

Then the guardrails ate the answer. With all 24 computed and handed over, the reply still named four or five. The prompt was not the problem. A grounding guard exists to catch an answer produced without retrieving, and an enumerate calls no tools, so it fired: it discarded the answer and rebuilt one from the top-scoring citations. Enumerated members carry a score of zero on purpose, because they were read from a field and never ranked, so they sorted last and were cut every time. A complete list of 24 was being thrown away and regenerated from six retrieved chunks. The coverage check that appended the missing nineteen was not a safety net doing its job; it was patching damage inflicted three steps earlier.

The grounding self-check made the same mistake one step later, flagging every enumerated answer for "case studies not mentioned in the provided context" (the projects it had just asked for). Each member is stored carrying the whole list as its text, and the checker built its context one entry per citation, truncated at 500 characters, so it saw the same list two dozen times with every copy cut off around the fourth project. Its verdict was correct about its own input and wrong about the world. Both guards were applying retrieval logic to an enumeration; both now know the difference between a field read and a search, and the answer went from four or five to 21 of 24 named directly.

One honesty guard fell out of it. Technology answers keep "used in these projects" separate from "certified in", because SageMaker and Comprehend appear only in the certifications document. Merging them would have the system claim Arup used tools the corpus only supports him being trained in.

Derived, never authored is the thread running through all of it. The tempting fix at every stage was to write the answer down: an index document listing the projects. This repo disproves it against itself. corpus/README.mdstill says "4 production wins" when there are six; it drifted the moment documents were added. A client is worse again, because you cannot hand them a heading convention and ask them to restructure documents they already have. Derivation puts the burden on the pipeline, where it can be validated. Seeding reports documents that yield no metadata rather than indexing them silently, because a document that vanishes from breadth answers is invisible in a success count.

Held to the usual bar: 39 of 44 eval cases passed at the time, average top score 0.5077, unchanged across the re-index and improved slightly by citing documents with their real titles instead of their filenames; the title feeds the BM25 sparse vector, so that was measured rather than assumed. The suite has since grown to 60 cases with 57 passing at 0.5476, and the failures are kept red on purpose: they measure plain similarity search, which cannot enumerate a corpus, which is the entire reason the enumerate route exists. What this does not claim: some runs still saw a few projects missed and appended by the coverage check. That is a model declining to finish a list it was handed, which is a different and smaller problem than the pipeline destroying its own correct work, and closing it means either splitting the answer across calls or accepting a more mechanical format, both of which cost more than three appended lines do. The corpus is also small enough to read in one pass; at a few thousand documents it would not be, and hierarchical summarisation is the next step rather than a bigger prompt.