Decisions

Architecture decisions

The reasoning behind the live Portfolio Ops system, recorded as decisions with their alternatives and their costs. You can check each one against the running system.

Every record names the alternatives I rejected and the costs I took on. Where a decision is visible in the live system, there is a link to go check it.

ADR-001Shipped

Pinecone serverless for vector search, not pgvector or OpenSearch

Context

The corpus is small: about ten documents and a few dozen chunks, plus short-lived visitor uploads. One person maintains it, and the demo has to stay cheap while idle. The retrieval layer should not turn into an operations job.

Decision

Pinecone serverless, with 1024-dim Titan v2 embeddings and cosine similarity. searchSimilar() scopes uploads to the asking session via a sessionId metadata filter, so a visitor's uploaded document is reachable to them, in its own Q&A box and as a reference for the probe, and never to anyone else.

Alternatives considered

  • pgvector on RDS: another stateful service to run and scale, which is overkill at this volume.
  • OpenSearch or self-hosted vectors: heavier to operate and slower to stand up, with cost even at rest.
  • An in-memory index in the Lambda: nothing persists across cold starts, and uploads have no shared state.

Consequences

  • Almost no cost while idle, and no index to operate. That suits a solo-run demo.
  • Embeddings run in ap-south-2 while the index lives in us-east-1, so each search pays a cross-region hop. That latency shows up in Inspect.
  • If the corpus ever grew large, the right move is to colocate the regions and add sparse-dense vectors. Written down, not pre-built.
Superseded by ADR-011: retrieval now runs on Qdrant hybrid
ADR-002Shipped · evolving

A relevance floor over dense retrieval; the hybrid it called for shipped as ADR-011

Context

Dense cosine handles semantic questions well; real matches in the golden suite score between 0.33 and 0.79. But a short, keyword-heavy query exposed the classic dense-only gap. When nothing relevant is indexed, the closest documents still come back around 0.10, which is noise, and the model, handed that thin context, can write a confident denial. On a demo whose whole pitch is transparent retrieval, that is the worst thing that can happen.

Decision

Keep dense-only as the base and add a calibrated relevance floor. When the top score drops below about 0.20, the system says "no strong match in the index" instead of letting the model speak from noise. Hybrid retrieval (BM25 or sparse fused with dense) was written down as the next step rather than pre-built, so the failure would earn it instead of a roadmap guess. It did: hybrid shipped in ADR-011. The floor is still in force on top of it, because a hybrid match can be weak too.

Alternatives considered

  • Ship hybrid right away: more infrastructure and a reindex, so it was deferred until the floor proved insufficient.
  • Do nothing, and leave the model free to deny things from noise. Rejected, because it breaks the honesty invariant.

Consequences

  • Off-corpus questions now get an honest empty-context answer, with the top score shown in Inspect.
  • Keyword recall for rare tokens stayed limited while this was dense-only. That gap was written down rather than hidden, and ADR-011 closed it with sparse + dense fusion.
  • The floor is a single configurable number, calibrated against the eval suite so it never trips a real match.
Top-score + below-floor signal in /ops Inspect
ADR-003Shipped

Amazon Nova Pro + Titan Embed v2 on Bedrock

Context

The system needs embeddings, generation, and tool-use on AWS-native infrastructure, with one IAM and billing surface and a predictable cost.

Decision

Titan Embed Text v2 (1024-dim) for embeddings and Nova Pro for generation, both on Bedrock. Generation goes through the Converse API, which gives native tool-use. That is what made the agentic loop possible without bolting on a separate framework.

Alternatives considered

  • External OpenAI or Anthropic APIs: cross-cloud egress, plus a second billing and key-management surface.
  • Self-hosted open models: inference operations and GPU cost a demo does not justify.

Consequences

  • One cloud, one IAM model, one bill, so least-privilege per Lambda stays simple.
  • Converse tool-use unlocked the agent directly (see ADR-004).
  • Model choice is now mostly about cost and latency rather than capability, and the readout in Inspect makes that explicit.
Token usage + timings in /ops Inspect
ADR-004Shipped

A bounded agentic loop with a self-check, not an open-ended ReAct agent

Context

The hero claim is agentic systems, so the demo should run a real agent. But the most common production failure for agents is the runaway loop: unbounded tool calls that burn latency and money. Multi-agent sprawl would add risk here without adding signal.

Decision

One read-only retrieve tool, a hard cap of three tool iterations, and an LLM-as-judge self-check that confirms the answer is grounded in retrieved context before it is finalized. Two guardrails reinforce the bound without loosening it: if the model ever tries to answer without retrieving, the harness forces a single grounding retrieval; and questions about Arup himself draw evidence from a separate, deterministic fan-out (ADR-008) instead of relying on the loop to multi-hop. Below-floor or ungrounded answers are flagged honestly, never invented. Every step is emitted as a visible trace.

Alternatives considered

  • An unbounded ReAct loop: open-ended cost and latency, the exact failure this guards against.
  • Multi-agent orchestration: more moving parts, no better answers at this scale.
  • No self-check: faster, but it lets ungrounded claims through.
  • Trusting the model to multi-hop on its own: unreliable per-phrasing on this model, which is why broad questions use the fan-out instead.

Consequences

  • A predictable cost and latency ceiling per query, narrow questions usually take one tool call.
  • Restraint instead of theatrics. The hard cap is the safety story, and it is what the agent-eval asserts on.
  • A genuinely hard question can stop a step early. That is an accepted trade for bounded, inspectable behavior.
  • The grounding guardrail makes a skipped retrieval impossible, so a stale conversation can't produce a false 'not in the corpus' for indexed content.
Live trace + Agent Evals (16/16) in /ops
ADR-005Shipped

Session-scoped, TTL-expiring, quota-bounded visitor uploads

Context

Visitors can upload a document and query it live. Anything public that touches storage and a model has to be safe by default. It cannot leak across visitors, run up unbounded cost, or leave data lying around.

Decision

Uploads are scoped to a session and expire on a 24-hour TTL (an S3 lifecycle rule plus a vector-store cleanup path, now Qdrant). They sit behind per-day global upload and query quotas, with a corpus-busy lock during index mutations. The retrieve tool is read-only, so an instruction injected into an uploaded file has no action it can take.

Alternatives considered

  • Persistent uploads: storage, cost, and privacy obligations a demo should not carry.
  • No quotas: an open door to abuse and runaway spend.
  • A shared, unscoped index: cross-visitor data leakage.

Consequences

  • Safe to leave open to the public. The blast radius of a malicious upload is one short-lived session.
  • Ephemerality becomes a feature, since the teardown is part of the demo.
  • By design, it is not a durable multi-user document store.
Upload → query → delete in /ops Upload
ADR-006Shipped

The corpus panel is generated from the live index, credibility is the product

Context

The homepage panel that shows what's indexed started as a hardcoded list, and it drifted. It advertised documents that were never indexed while hiding the ones that were. A visitor could ask about an advertised document and watch the model correctly say it isn't there, a contradiction sitting two inches from the claim.

Decision

Generate the panel from a GET /corpus endpoint that lists what is actually in the vector store (a Qdrant scroll over the corpus/ path prefix), with a real-corpus static fallback. The panel cannot advertise a document that isn't indexed.

Alternatives considered

  • A hardcoded or hand-curated list, which is the thing that drifted in the first place.

Consequences

  • The panel can never show a document that isn't there, and new documents appear once they are indexed.
  • One cheap, cached API call on load.
  • This is the honesty invariant in practice: the whole demo's value is that nothing on screen is faked.
The corpus panel on the homepage
ADR-007Shipped

Defense in depth against prompt injection (OWASP LLM01)

Context

The agent reads visitor-uploaded documents and its own corpus, then feeds that text to a model. Prompt injection is OWASP's top LLM risk: a document or a question that says "ignore your instructions and do X." Neither RAG nor fine-tuning removes it, so it needs layered defense rather than a single filter.

Decision

Four layers. The retrieve tool is read-only, so there is no privileged action to hijack. The system prompt treats the question and all retrieved text as data, never as instructions. An injection-pattern guard surfaces attempts in the trace so they are visible rather than silent. The grounding self-check and the relevance floor keep the answer tied to real sources, and input is stripped of control tokens before it reaches the model.

Alternatives considered

  • A single regex filter as the defense: brittle, and it gives false confidence. Detection here only adds visibility on top of the structural layers.
  • Trust the model to behave: injection is a known, repeatable failure, and hope is not a control.

Consequences

  • An injected instruction has no action it can take and does not change the agent's behavior; the attempt shows up in the trace as a guard step.
  • Pattern detection will miss novel phrasings, which is exactly why it is the visible layer and not the load-bearing one.
  • With session isolation and TTL (ADR-005), the blast radius of a malicious upload stays inside one short-lived session.
Type an injection into /ops Ask and watch the guard step
ADR-008Shipped

Deterministic fan-out for questions about Arup, not model-driven multi-hop

Context

Questions about Arup himself, USP, strengths, why-hire, setbacks, should be answered from the breadth of his real projects, not a single bio document. The first attempt told the agent to multi-hop across projects on its own. It worked for some phrasings and gave up on others (asked for the 'USP', it retrieved the map and decided the corpus had nothing). Model-driven decomposition was too phrasing-fragile on Nova Pro to ship to strangers.

Decision

A profile question triggers a deterministic fan-out: the harness issues a fixed set of focused queries across Arup's distinct projects, gathers the evidence, and the model synthesises a cited answer from it. The about-arup document is a routing map that holds no conclusions of its own, so it can't shortcut the answer. The agentic loop stays bounded at three (ADR-004); the fan-out is a separate, fixed pre-retrieval step, not extra loop iterations.

Alternatives considered

  • Model-driven multi-hop (PRISM-style): more elegant and general, but unreliable per-phrasing on this model.
  • A single 'about me' summary doc: dominates retrieval and collapses every answer to one self-referential source.
  • Multi-agent planning (MA-RAG-style): more machinery than a single, known question-class needs here.

Consequences

  • Profile questions reliably cite several real projects instead of one bio page, at the cost of a few extra retrievals, shown honestly in the cost-and-latency readout.
  • A deliberate trade of agentic elegance for reliability. The upgrade path is to hand decomposition back to the model once a stronger tool-use model makes it dependable, keeping the fan-out as a fallback.
  • Narrow project questions are unaffected, they still do a single retrieve.
Ask 'Why hire Arup?' in /ops and watch the fan-out
ADR-009Shipped

Layered spend caps against denial-of-wallet, not a single limit

Context

The query path is a public, unauthenticated Lambda Function URL behind CloudFront, and an agentic answer calls Bedrock several times. A Function URL has no built-in throttling, so a flood could run up an invocation, DynamoDB, and model bill before any single guard noticed. The per-call token caps and the global daily quota already hard-capped the model bill, but a flood still paid for Lambda and DynamoDB work, and one visitor could drain the whole day's quota by accident.

Decision

Cap spend in independent layers, each cheap and each failing safe. At the edge, a WAF rate rule drops a single IP above 100 requests per five minutes before it costs an invocation. In the Lambda, a per-IP daily sub-quota of 8 sits under the global quota of 20; the real client IP is read from X-Forwarded-For since traffic is behind CloudFront. That per-IP cap is a FAIRNESS mechanism, not a security one, and is worth stating plainly because it is easy to mistake for a wallet guard: anyone willing to rotate IPs walks straight through it, and it deliberately fails open on a DynamoDB hiccup. What it actually buys is that one ordinary visitor cannot accidentally consume the whole day and leave the demo dead for everyone else. The thing that genuinely bounds spend is the atomic global#<date> counter, backed by the $5/day budget. At the function tier, the account's Lambda concurrency ceiling is already low enough to bound how wide a flood can fan out, so no slice was carved out for the query function: reserving one would only starve the sibling functions out of the same small pool, and provisioned concurrency was never on the table since it bills around the clock. An account-wide $5/day cost budget is the backstop that catches whatever the other layers miss.

Alternatives considered

  • One global rate limit and nothing else: still safe for the wallet, since the global counter is the real bound, but one visitor could exhaust the shared day and a flood would still pay for invocations the quota never stops. The per-IP cap is kept because it costs nothing, not because it closes an attack.
  • Provisioned concurrency: bills around the clock for warm capacity the demo does not need, which is the opposite of the goal here.
  • API keys or auth on the endpoint: friction for a public portfolio whose whole point is that a stranger can try it without signing up.

Consequences

  • A flood is dropped at the edge, slowed per IP, held under the account's concurrency ceiling, and bounded again in total dollars. Only the global counter and the budget are load-bearing against a determined attacker; the other layers raise the effort. The edge, quota, and budget layers each deploy and revert on their own.
  • The WAF rule and the budget add a few dollars a month, the only recurring cost the hardening introduces.
  • The per-IP quota fails open on a DynamoDB hiccup, leaving the global quota and the WAF rule as the hard backstops, so a storage blip never blocks a real visitor.
Edge rate-limit and caps shown on /architecture
ADR-010Shipped

Deterministic concept routing for abstract project questions

Context

The live agentic loop works well when a visitor asks in the same language the corpus uses: project names, outcomes, and implementation details. It gets shakier when the question comes in as abstract engineering language and the corpus stores the evidence in concrete project terms instead. A question like "Is there an example of race condition handling, and what is its relation to idempotency?" is answerable from the portfolio. The problem is that the first retrieval path kept circling the abstract words and never landed on the project language underneath them: double-bookings, retries, double-clicks, Redis locks, replay, dedup. The issue was not missing evidence. The issue was that the system had no reliable bridge into it.

Decision

Add a deterministic concept-routing layer ahead of the bounded agentic loop for a small class of abstract project questions. The router classifies concept-style queries such as example-of, relation-between, where-did-you-use, and difference-between. For those classes, it expands canonical concepts into corpus-native terms and known project candidates before retrieval. The LLM still reads the evidence, explains it, and writes the final answer. What changes is that it no longer has to guess its way from abstract phrasing to the right project evidence every time. A companion guard blocks answers that cite only routing documents like about-arup without citing any real project document.

Alternatives considered

  • Trust the model to reformulate the query on its own. Nice when it works, too brittle for a public demo.
  • Add glossary-style explainer documents for every concept. That might help one phrasing, but it pulls the portfolio away from project evidence and toward textbook filler.
  • Rely on hybrid retrieval alone. That should help lexical recall, but it will not fully solve relation-style questions because they still need a bridge into the right projects.
  • Make everything deterministic, including answer composition. More rigid, and it gives up the part the model is actually useful for.

Consequences

  • Abstract engineering questions became less wording-sensitive, while narrow project questions keep the simpler path.
  • The architecture becomes less purely agent-driven at retrieval time, but more dependable and easier to inspect.
  • A small maintained concept map becomes part of the harness. That is extra structure to own, but it is cheaper than letting repeated public misses pile up.
  • The split becomes cleaner: deterministic logic owns routing, candidate narrowing, and evidence gating; the model owns synthesis and explanation.
Ask concept-style questions in /ops and inspect the route and evidence mix
ADR-011Shipped

Migrate from Pinecone dense-only retrieval to Qdrant hybrid retrieval

Context

The original Pinecone decision was right for a tiny, mostly semantic corpus and almost-zero idle cost. The later retrieval failures were not operational failures, they were retrieval-quality failures: rare proper nouns, explicit failure-language sections, and exact engineering terms were all the kinds of queries dense-only retrieval predictably drops. Once those misses were visible in the live evals, the system had earned a real hybrid retrieval layer rather than more app-side patches.

Decision

Keep the public API contract stable and swap the retrieval store under it. The production path now uses Qdrant Cloud with one hybrid collection: Titan v2 dense vectors, BM25 sparse vectors, reciprocal-rank fusion at query time, the same session-scoped payload filtering for uploads, and the same calibrated confidence floor before the model answers. The migration is documented as an evolution after ADR-001 and ADR-002, not a rewrite of them.

Alternatives considered

  • Stay on Pinecone and keep patching dense-only misses with routing and upload-slot heuristics. That helps specific cases, but not the base lexical recall problem.
  • Run dense and sparse search in separate systems and fuse in the application layer. More moving parts, with no upside at this scale.
  • Add a reranker before fixing first-stage recall. Better ordering would not solve the chunks that never got retrieved.

Consequences

  • Entity names, exact terms, and failure-language queries are now much more likely to surface the right chunks on the first retrieval pass.
  • The frontend and deployed API shape stay stable, so the migration does not force a public UI rewrite.
  • The historical Pinecone and dense-only ADRs remain true as records of the earlier system; ADR-011 explains why the later system changed.
See the hybrid retrieval path in /architecture and the live retrieval diagnostics in /ops
ADR-012Shipped

Treat Qdrant hybrid retrieval as the current and future architecture baseline

Context

Once the migration was complete, the architecture story had two truths that needed to coexist cleanly. Historically, Pinecone was the right first retrieval store for a tiny dense-only corpus with almost no idle cost. Operationally, that system no longer exists in production. The live stack, the inspect tooling, the eval harness, and the public API now run on Qdrant hybrid retrieval. Leaving current or destination architecture views anchored on Pinecone would blur the line between historical record and present system.

Decision

Keep the old Pinecone ADRs unchanged as history, and append a new baseline ADR that says the live and forward-looking architecture is Qdrant-first. Current-state and eventual architecture references should describe one hybrid Qdrant collection, Titan dense vectors, sparse lexical retrieval, reciprocal-rank fusion, and the existing session-scoped filtering and confidence floor. Pinecone remains part of the story only where the goal is to explain the earlier system and why it changed.

Alternatives considered

  • Rewrite the original Pinecone ADRs so every record reads as if Qdrant had always been the choice. Rejected because it erases the real evolution of the system.
  • Leave current and target architecture copy split between Pinecone-era and Qdrant-era descriptions. Rejected because it confuses readers about what is actually live.
  • Maintain separate 'historical' and 'current' architecture pages for the same stack. Rejected as too much ceremony for a small public system.

Consequences

  • The architecture narrative stays historically honest without making the live system look ambiguous.
  • Future diagrams, inspect copy, and production architecture notes can all assume Qdrant hybrid retrieval as the baseline retrieval layer.
  • Pinecone remains visible where it should: as the earlier decision that was later superseded, not as the current production dependency.
The live and target retrieval paths shown in /architecture and the production diagnostics in /ops
ADR-013Shipped

Disable the About Arup retrieval shortcut while keeping the profile fan-out

Context

Hybrid retrieval fixed the lexical-recall problem, but it did not make the old biography shortcut healthy. The about-arup document was still a tempting answer-collapse path: even when the system had enough project evidence, one broad self-summary could reappear in traces and citations and make profile answers look sourced from a bio rather than from the work itself. That is the wrong shape for a public architecture whose credibility depends on project-grounded evidence.

Decision

Keep the deterministic profile fan-out, but disable corpus/about-arup.md from active retrieval, live corpus listings, and future seeding. The file stays in the repository for now as historical material and for controlled testing, but the production retrieval path should synthesise broad profile answers directly from project documents and case studies. Hybrid Qdrant retrieval remains the first-stage recall layer; the fan-out remains the control-plane step for broad profile questions.

Alternatives considered

  • Leave the thin About Arup map indexed and trust guardrails to keep it from dominating. Rejected because the shortcut still leaks into citations and weakens the evidence story.
  • Remove the deterministic profile fan-out now that hybrid retrieval is live. Rejected because hybrid improves recall, not cross-project aggregation discipline.
  • Delete the file immediately from the repository and rewrite the earlier ADR trail around it. Rejected because the file is still useful as historical evidence while the new path is being tested.

Consequences

  • Profile answers now have to stand on project evidence, not a single self-summary document.
  • The public corpus panel stays honest: disabled documents are no longer advertised as live retrieval sources.
  • Testing can continue with the file preserved on disk, while production behavior reflects the new architecture baseline.
Ask broad profile questions in /ops and inspect the citations and fan-out behavior
ADR-014Shipped · evolving

Split ephemeral EKS into keep vs destroy stacks

Context

An EKS LangGraph lab only proves something useful if it looks production-shaped: real cluster, Helm, a public load balancer, traces, teardown. Running that stack all month would burn money on the control plane, nodes, load balancer, and public IPv4 even while nobody is watching. The design problem is not 'can we run EKS'; it is 'can we run EKS for thirty minutes and trust that expensive leftovers cannot linger.'

Decision

Split resources by billing risk. Foundation (keep): ECR images, S3 artifacts and Terraform remote state, IAM/OIDC trust, optional minimal VPC shell without NAT, reused Qdrant Cloud collection, LangSmith/Langfuse project, and the control-plane run store plus EventBridge reconciler schedule. Ephemeral (destroy every run): EKS cluster and node group, the Service load balancer (a Classic ELB in practice), public IPv4 allocations, Helm agent releases, demo log groups with short retention, and any demo-specific DNS. Never create EKS without a provider-side one-time teardown schedule. Never report Destroyed until a tag-based cleanup audit confirms the expensive set is gone.

Alternatives considered

  • Keep a long-lived EKS cluster and only redeploy the app. Rejected: control-plane and node hours dominate cost and teach the wrong ops lesson.
  • Destroy everything including ECR, IAM, and Terraform state after each demo. Rejected: recreating trust and registry setup every run slows demos without reducing meaningful spend.
  • Rely only on an in-app timer to tear down. Rejected: if the portfolio app is down, billing continues; a provider-side schedule and reconciler are required.
  • Put NAT gateways and interface VPC endpoints in the foundation stack 'for convenience.' Rejected: they have hourly charges and defeat the near-zero idle goal.

Consequences

  • Idle cost stays near zero: between demos there is no EKS, load balancer, or public IPv4 to bill.
  • Demo setup stays fast because images, state, and IAM do not churn.
  • The Ops and Architecture UIs can tell an honest story: Idle = keep stack only; Running = keep + destroy stack; Destroyed and verified = scrubber passed.
  • Cleanup failures surface as Cleanup needs attention instead of a false Destroyed status.
Toggle Idle vs Running on /architecture · Ephemeral EKS, and walk the lifecycle on /ops
ADR-015Shipped

Chat mode: one durable transcript, never combined with Bedrock's own memory

Context

Bedrock was already a chat backend before this shipped: session.ts kept a rolling 20-message window (10 turns) and query-rewrite.ts already rewrote conversational follow-ups against it, on every query, with no UI ever saying so. Adding an explicit chat mode on top risked the worst possible failure: if the site ALSO sent a durable transcript from its own store while the backend kept reading its own session history, every prior turn would land in the model's payload twice and inputTokens would roughly double for the same conversation, a correct-looking readout of a silently doubled bill.

Decision

One source of truth for conversation context, never two. query.ts accepts an optional `transcript`; when present it REPLACES the session.ts read and write entirely rather than adding to it (verified by direct measurement against live Bedrock: doubling the same history roughly doubled inputTokens, confirming the mechanism the guard prevents). A `single` mode sends `noHistory: true` instead, reading no history and writing none, so switching back to `chat` resumes exactly where it left off. Chat turns persist for 1 year in `portfolio-ops-chat-<stage>` (one item per turn, not one fat item per chat, to avoid the DynamoDB 400 KB ceiling and the read-modify-write race a single fat item would reintroduce). The system prompt's TEXT is never rendered anywhere, even in the 'View original' viewer, only its role, line count, and an approximate token size, because agent.ts actively refuses to reveal that prompt and printing it in Inspect would make that refusal theatre.

Alternatives considered

  • Send the transcript AND let query.ts keep reading its own session history. Rejected: this is the doubling trap above, and it would have shipped invisibly, since Inspect would faithfully report the inflated (wrong) token counts as if they were correct.
  • Drop session.ts entirely once the durable chat store exists. Rejected: the site's other, non-Ops callers of /query (e.g. the homepage widget) don't send a transcript and still benefit from short-lived server-side memory; removing it would silently regress their behavior.
  • Cap the per-chat quota above the model's 10-turn context window. Rejected: the 5-turn cap was chosen specifically to stay under that window, so the visible transcript never shows more turns than the model actually still has in context.

Consequences

  • A visible mode toggle in /ops is the honesty fix and the demo at once: the same follow-up scores worse in `single` mode (no history reaches the model) and better in `chat` mode (rewritten against the prior turn), run by the visitor rather than asserted by copy.
  • Chat turns are kept for a year while uploaded documents expire in 24 hours (ADR-005), a real and deliberate difference in privacy posture on the same site that is now surfaced in Inspect rather than left implicit.
  • A chat idle for an hour is CLOSED, not resumable (added 2026-08-08): the server refuses to replay or append to it and the console rotates to a fresh chat, so a returning visitor is never greeted by a stale transcript. Closing is distinct from retention. The records still live out their year server-side, but they drop out of view: Ask history and Inspect follow the current chat, so closed means gone from the console, by design. Enforced server-side in the store (the client mirror only shapes what an open tab sends next), the same trust boundary as the 5-turn cap.
  • EKS's `AskRequest.session_id` (declared and silently discarded) is replaced by an equivalent `transcript` field, capped server-side; a client-supplied cap is never trusted.
Toggle Chat vs Single question in /ops Ask, then compare Inspect's per-turn token growth against the same follow-up in each mode
ADR-016Shipped

Push live quota over AWS IoT Core, not a held Vercel connection

Context

The quota panel showed a stale count until the visitor's next question. An earlier draft assumed serverless has no push at all, which is wrong - SSE genuinely is push, the server holds the connection and writes when it chooses. The real problem was narrower: on Vercel, the SSE handler and the query handler that decrements the counter are separate invocations with no shared memory, so a held SSE stream would have no channel to learn the counter changed without some other broker in between anyway.

Decision

Let the browser subscribe directly to AWS IoT Core over MQTT/WebSocket instead of holding any Vercel connection open. A site route mints a short-lived SigV4-presigned wss:// URL (reusing the existing AWS_ROLE_ARN Vercel-OIDC role, so no Cognito identity pool is needed); the query path publishes the new remaining count onto one topic with the MQTT retain flag after each decrement, so a newly-connecting tab gets the current value immediately instead of a blank gap. The IAM policy behind that presigned URL is deliberately subscribe-only on exactly that one topic - no publish, no wildcard - since this is a public, unauthenticated browser connection and, per AWS's own authorization model for SigV4/WebSocket clients, that policy governs every MQTT frame the connection sends for its whole lifetime, not just the initial handshake.

Alternatives considered

  • Keep polling GET /quota on an interval: simple, but every visitor's tab pays a request for a value that changes at most ~20 times a day.
  • SNS, EventBridge, or SQS fan-out to a Vercel webhook: none of these can push into a browser tab that's already holding a connection open - they deliver to endpoints (Lambda, SQS, HTTPS), which is a fresh invocation per message, not a channel into an existing session.
  • A Cognito identity pool for browser credentials: unnecessary machinery here, since the site already holds a Vercel-OIDC role capable of being scoped down to mint the presigned URL directly.

Consequences

  • No Vercel function is held open for the life of a viewer's tab, and IoT Core's billing (connection-minutes) is the only new cost - closing the connection on visibilitychange keeps a backgrounded tab from paying for nothing.
  • The security posture rests entirely on that one IAM policy staying subscribe-only, single-topic - the comment in portfolio-rag-infra's quota-broadcast-stack.ts says so explicitly, so a future edit can't widen it by accident.
  • Only the query path publishes. readQueryQuota(), which serves the REST fallback the browser reads on mount, must never publish, because it reads the counter without decrementing it and would push a value that had not changed.
  • The endpoint is discovered rather than configured, which reverses the original handoff's reasoning. That doc argued a hand-set env var was cheaper than an iot:DescribeEndpoint permission and a round trip. The round trip is cached per warm instance, and the hand-set value is exactly what left this feature dark after every other piece was provisioned, so the cheap option was the one that failed. iot:DescribeEndpoint supports no resource-level permissions and so needs a wildcard resource; it lives in a SEPARATE policy for that reason, keeping the browser-facing one provably wildcard-free.
  • All three pieces are deployed and the path is observed end to end, not inferred: subscribing exactly as a browser does, a retained message arrived immediately on connect showing 15 remaining, and a live push arrived showing 14 while a query ran. One AWS-specific detail cost the most and is worth recording: @smithy/signature-v4 puts X-Amz-Security-Token inside the canonical query string whenever credentials carry one, which Vercel OIDC always does, and IoT Core rejects a signature computed over the token. Every connection closed with no connect event and no error frame while the URL looked correct in every respect. The token is now appended after signing. Nothing short of deploying and actually subscribing would have found it.
Open /ops and watch the remaining-questions count drop the moment a query runs, pushed over MQTT rather than polled
ADR-017Shipped

Breadth questions are enumerated, not searched

Context

A visitor asked what technologies Arup had worked with and got three AI items, then asked for anything apart from AI and got the AWS ML stack, TypeScript and Zod, every one of them AI. Two distinct failures sat underneath that. The exclusion was one, fixed separately by lifting the negation out of the embedding and into a must_not filter, since negation is not representable in embedding space. The other is that "what technologies has he worked with" is a global question, the class chunk-retrieval RAG structurally cannot answer: retrieval returns short contiguous passages, and no single passage in this corpus frames the whole of it. The information was there, webhooks in six documents, Redis in three, dual-write and feature flags in two each, but it lived inside project narratives that do not resemble the abstract word "technologies".

Decision

Two steps, and the first alone was not enough, which is worth recording rather than smoothing over. First, add one '## At a glance' section per corpus document, stating what it is, its domain, and the technologies it names, in the vocabulary breadth questions use. The existing header chunker turns each into exactly one chunk, so that was fourteen extra chunks and no retrieval code. It moved the hard case from 0.123 to 0.173 against a floor of 0.2, and still refused. Second, and this is what actually closed it: breadth questions ENUMERATE that layer instead of searching it. Every document summary is fetched by filter, unranked, and placed in the prompt. Retrieval was simply the wrong verb for a question with no single passage to match, and no amount of better matching was going to fix that.

Alternatives considered

  • GraphRAG: entity extraction, community detection, and map-reduce over community summaries. It answers global questions by ENUMERATING every community rather than searching, which is a stronger guarantee than a summary layer gives. It is also machinery whose entire purpose is to avoid reading a corpus too large to enumerate, and this corpus is about 10,600 tokens, roughly three percent of the model's context window. Paying that complexity for a constraint that does not bind would be architecture theatre.
  • RAPTOR proper: recursive clustering and summarisation into a tree. Same objection at this scale, and clusters spanning fourteen documents would mostly reproduce the documents.
  • Stopping at the summary layer and leaving the hard case refusing. Defensible for a while, since an honest refusal beats a confident wrong answer, but it left a question the corpus could clearly answer going unanswered. Superseded by the enumeration above, which shipped with the labelling requirement met rather than waived: the survey is its own step kind and the strategy label renames itself, because a fan-out over every document is not retrieval and this site's claim is that visitors are watching real retrieval happen.
  • Lower the relevance floor when a filter has already constrained the candidate set. Rejected as a hidden threshold. Lowering a floor to get an answer out is how confident wrong answers are manufactured, which is the exact failure this system exists to prevent.

Consequences

  • Breadth questions now have something shaped like an answer to retrieve, and the fix is legible: it is corpus content, reviewable in a diff, with no new retrieval path to reason about.
  • The reported question now answers correctly. Asked what technologies Arup has worked with apart from AI, the system surveys eight documents (six excluded on request) and returns PostgreSQL, Django, React, feature flags, dual-write, Lambda, Iridium, offline-first sync and face verification, with no AI document cited. The arc took four steps and each was necessary: exclusion filter, query reformulation, summary layer, enumeration. None alone was sufficient, which is why none of them was a reason to lower the floor.
  • The relevance floor was never lowered. It is skipped only when a survey ran, because a search-derived similarity says nothing about an enumeration that read every document, and the GROUNDING self-check is untouched: this relaxes the relevance guard, never the honesty one. Whether 0.2 is the right number for ordinary retrieval is still open, and still the separate decision it always was, now without a failing case pushing on it.
  • Summaries are claims, so they carry an invention risk the prose does not. The no-invention check is what makes the layer safe to extend; it caught two false positives during authoring.
  • Enumeration is visible, not substituted. The survey emits its own step kind, the retrieval diagnostics rename the strategy when one runs, and survey citations carry a score of zero because nothing was ranked, since a plausible-looking similarity would be a fabricated number in the Inspect panel. The exclusion filter binds inside the enumeration too, so reading everything never became a way around a filter the visitor asked for.
  • Qdrant refuses outright to filter on an unindexed payload field, so the keyword index on 'kind' is not an optimisation, it is what makes the survey work at all. Found before deploying only because a filtered scroll errored while an unfiltered one did not.
Ask what technologies Arup has worked with in /ops, then ask for anything apart from AI, and watch the trace show a survey rather than a retrieve
ADR-018Shipped

A filtered follow-up over an enumerated set is validated, not trusted

Context

A visitor asked what are different projects Arup were a part of and got the correct answer: all 24 projects, enumerated from the typed docType field per ADR-017. The follow-up, which one of these involve generative AI, came back with AI Architecture Patterns I Use in Production - docType reference, one of the three documents that is explicitly not a project, and never a member of the 24 the visitor had just been shown. classifySetQuestion, the narrow gate that routes a question to the enumerate path, returns null on that phrasing by design: it only fires on an explicit enumeration cue or the canonical whole-corpus phrasing, precisely so it does not misfire on a question that is actually scoped to one project. A filtered follow-up has neither signal, so it fell through to ordinary similarity search, where reference documents are fair game again. Turn one was deterministic about membership. Turn two was back to whatever embedded closest.

Decision

Widening classifySetQuestion's noun list to catch which of these was the obvious fix and the wrong one: that gate sits under isBreadthQuestion specifically so a project-scoped question like which technology did the anti-overbooking project use for locking, which isBreadthQuestion already matches on its own noun list, does not get the enumerate path's there are exactly N, list all N instruction. Widening it to catch a filter would have reintroduced that false global-count injection through a side door. So the fix is a second, separate gate, isFilteredSetFollowUp, that only recognises the referent shape itself: a filter-question cue, which, what, any, how many, do, does, is, are, within a short window of a demonstrative, these, those, them. On its own that gate proves nothing; it only matters combined with re-running the ORIGINAL classifier against the PRIOR turn's question, confirming turn one really was a set answer before turn two is treated as a filter over it. Once both hold, the set is recomputed fresh from the same typed fields ADR-017 reads, never trusted from conversation history, and handed to the model with a machine-checkable tag on every member, a project's path or a technology's normalised key. The model selects a subset against the visitor's semantic criterion, which is real judgement and the part it is good at. Then every line of its answer is checked against the tag set, and any line naming something untagged, ai-patterns among them, is dropped before the visitor ever sees it. The model chooses. It does not get to invent.

Alternatives considered

  • Widen classifySetQuestion's noun list to also match which of these. Rejected: that gate is deliberately narrow so a project-scoped question does not get a false there-are-N-list-all-N instruction, and a filter needs different handling entirely, not a wider trigger for the same one.
  • Trust the model to keep membership straight from the 24 titles already sitting in its own conversation history. Rejected on the evidence: turn one's answer had all 24 titles in it, in plain prose, and the model still reached past them for a document that was never in the list. Prose in a history window is not a membership check.
  • Persist the computed set somewhere so turn two can reuse it instead of recomputing. Rejected: listProjects and its technology/domain equivalents are a filtered Qdrant scroll, cheap enough that persistence would only be a second copy of the set to keep from drifting out of sync with the first, for a cost saving that does not exist.

Consequences

  • The same recompute-tag-validate mechanism covers all three set kinds, projects, technologies, and domains, not only the one the bug report happened to use - the machinery is identical across all three, so there was no reason to special-case projects.
  • A genuine none of them qualify answer is preserved rather than misread as a hallucination and stripped: the validator checks for that sentinel before it starts policing tags, so an honest empty result still reads as one.
  • The agent trace gained its own step kind, filter, distinct from enumerate, so the trace never implies a retrieve() ran or that the model's own sense of membership was trusted unchecked - what got dropped, and why, is a visible guard step, not a silent edit.
  • The gate only looks one turn back, matching the reported shape exactly. A visitor filtering a set established two turns earlier still falls through to ordinary retrieval today, the same fail-open behaviour an unrecognised phrasing has always had here.
Ask what are the different projects Arup was part of, then which one of these involve generative AI, in /ops and watch the filter and guard trace steps
ADR-019Shipped

Every answer takes a second model pass before a visitor sees it

Context

Raw Nova Pro output reads like a model: em-dash chains, double spaces, the cadence of a system prompt being obeyed. For a portfolio whose product is credibility, an answer that reads machine-written undercuts the content before anyone evaluates it. The first fix was a text normaliser, and it caused the worse bug: its punctuation rules matched any whitespace, so it flattened markdown - a newline before a list bullet is whitespace too - and every multi-paragraph answer collapsed into one run-on block. The same normaliser had by then been copied into the website and the EKS agent, so the flattening shipped three times.

Decision

A dedicated humanizer pass: after the answer model finishes, a second Bedrock call rewrites the draft for tone and rhythm, on both the single-shot and agentic paths, and its tokens are folded into the reported usage rather than hidden - the Inspect panel's answer phase is the pass's cost, visible. The draft that streamed internally is discarded and the humanized text is what the visitor's stream actually carries. The punctuation normaliser was rewritten to match horizontal whitespace only, so newlines are structurally untouchable, and that exact rule now lives, deliberately identical, in all three repos that render or emit answer text.

Alternatives considered

  • Fold style instructions into the answer prompt and keep one model call. Rejected: the answer prompt's one job is grounding in retrieved evidence; every style constraint added to it competes with that job on the same token budget.
  • Clean up client-side in the website. Rejected: three consumers render this text (site, Ops panel, EKS run history), and client-side rules drift; the API is the single owner of the visible answer.
  • Skip the pass on the cheap single-shot path. Rejected: the two modes would read like two different authors, and the difference would look like a bug.

Consequences

  • Every answer costs two generation calls minimum. That is real money and it is reported, not buried - usage.byPhase.answer includes the pass.
  • The Bedrock trace does not yet emit a step for this pass, so Inspect shows its tokens but not its existence; the EKS port emits an explicit humanizer step. That asymmetry is a known gap, stated here rather than papered over.
  • The horizontal-whitespace-only rule is duplicated in three repos by design and must change in lockstep; each copy carries a comment naming the other two.
Ask anything in /ops and open Inspect: the answer phase carries the second pass's tokens
ADR-020Shipped

An exclusion becomes a retrieval filter; negation never reaches the embedding

Context

A visitor asked what Arup has built apart from the AI work, and got AI projects back. Embedding similarity cannot represent negation: 'apart from AI' embeds close to 'AI', so the vector search returned more of exactly what was excluded, and the model then argued with its own evidence. The failure is structural, not a prompt problem - as long as the excluded topic's words appear in the query text, the excluded documents rank first.

Decision

Exclusions are handled where they can actually be enforced: retrieval. A small detector matches only unambiguous exclusion phrasing - apart from, aside from, other than, besides, except, anything but, non-X - and resolves the excluded phrase against a hand-maintained topic-to-paths list. A match becomes a Qdrant must_not on document paths, merged into the same must_not that carries the corpus policy so neither can override the other, and it rides every retrieval in the turn including the fan-out and the survey. The question is then re-embedded as its positive remainder - what is being asked for, with the exclusion clause stripped - falling back to the prior turn's question when too little survives. 'Not' and 'without' deliberately never fire: a false exclusion silently hides documents, which is a worse failure than a redundant answer.

Alternatives considered

  • Prompt the model not to mention the excluded topic. Rejected: retrieval had already spent the top-K budget on excluded documents, so the model would be summarising evidence it is told to ignore.
  • An LLM classifier for exclusion intent. Rejected: a wrong guess is invisible - nothing in the answer says a filter was applied that shouldn't have been - so the trigger must be auditable and conservative.
  • Embedding the negated query and hoping the model sorts it out. Rejected on the evidence: that was the bug.

Consequences

  • The topic-to-paths lists are hand-maintained and drift when the corpus changes; removing documents once broke the AI topic list, and the resolver now guards against topics whose members are gone.
  • Unknown topics no-op - fail open to normal retrieval - so the worst case of a missed exclusion is the old behaviour, never a hidden document.
  • The trace stays honest: when the embedded text differs from the visitor's words, the retrieve step reports what was actually searched.
Ask 'besides the AI work, what has Arup built?' in /ops and watch the guard step and searched query
ADR-021Shipped

The corpus is a build artifact of the site, and re-seeding reconciles deletions

Context

/work rendered 22 projects at the time (23 now); the hand-written corpus had documents for five. The demo could not name most of the work it existed to prove, and the two copies also disagreed on vocabulary. Separately, seeding only upserted: a shortened document left its stale tail chunks serving, and a disabled document kept answering from vectors that no source file justified - that is how a removed biography went on being quoted after its removal.

Decision

The corpus became a derived artifact. Seeding fetches the site's own content.json - the same typed objects /work renders - and generates a project spine document per entry, plus typed metadata (docType, domains, techniques) written into the vector payload as indexed, filterable keys. Narrative case-study files stay hand-written; prose is never generated over prose. And every seed run reconciles: chunks whose source no longer exists are pruned by the exact point ids a scroll returned, scoped to the portfolio source so visitor uploads are untouched, and the seed log names what it removed.

Alternatives considered

  • Keep dual authorship and add a review checklist. Rejected: the checklist was already implicitly in place and had already failed - 17 of the 22 projects had no corpus document.
  • Generate the narrative documents too. Rejected: the writing is the evidence; flattening it into templated prose destroys what retrieval is meant to surface.
  • A one-off backfill script. Rejected: drift is a rate, not an event; it returns the day after the backfill.

Consequences

  • Deploy order matters and is documented: the site's content route ships before the API, because a seed run that cannot reach content.json degrades to files-only and the prune pass would then delete every generated document. CONTENT_SOURCE_URL set empty is the deliberate kill switch for that coupling.
  • Hand-maintained counts stopped being authoritative anywhere - the corpus README's own stale '4 production wins' is preserved as the cautionary example.
  • A standing script enforces the summary invariant - a document summary may compress its body, never add to it - with word-boundary matching so 'SSE' cannot pass on 'assessed'.
The /ops corpus panel lists the generated site/* documents next to the hand-written case studies
ADR-022Shipped

Self-description prose is banned from the evidence corpus; surviving facts become typed data

Context

ADR-013 disabled the About-Arup shortcut. This is the general policy it turned out to imply. Three more self-description documents - AI patterns, RAG stack, certifications - were measured crowding out the projects they summarised: a question about how Arup builds production RAG cited the two summary documents at 46% and 44% and nothing else, with all three real RAG case studies lost. Worse, one document's 'What I'd Change at Scale' section was cited as current practice, so the agent claimed a semantic cache and a reranker that do not exist - and the grounding check passed, because the words genuinely were in the corpus. Conditional prose in an evidence corpus becomes a false capability claim.

Decision

All three documents were removed from the index, enforced at three independent layers: skipped at seed time, filtered at query time, and dropped at citation read time, so no single layer's failure re-admits them. The facts that died with them were republished as typed data, not prose: a generated certifications document with one section per issuer, so a truncated chunk cannot silently drop a credential, and a one-chunk identity document that says who Arup is and deliberately refuses to say how good he is. The eval suite was corrected in the same change - two cases had been asserting the removed biography's presence, which means the suite demanded a policy violation; assertions moved to the layer that owns the behaviour.

Alternatives considered

  • Tune ranking to bury the summary documents. Rejected: a general document beating specific ones on a general query is embeddings working correctly; the bug was the document's presence, so ranking would have been a bandage.
  • Rewrite the documents to be purely factual. Rejected: the facts worth keeping are typed facts - certifications, identity - and typed site content is their home; the rest was self-assessment, which the corpus policy exists to exclude.

Consequences

  • An accepted, stated cost: certifications were unanswerable for a window until the typed content shipped, pinned by eval cases that flipped to positive assertions when it landed.
  • General how-does-he-build-it answers now stand on project case studies - with the residual weakness named openly: the learning-lab writeups still outrank the production case studies on those queries.
  • The policy is mechanical: DISABLED_CORPUS_PATHS is one list, enforced three places, and the eval suite owns a must-not-include assertion so a regression is red, not anecdotal.
Ask 'what certifications does Arup hold?' in /ops - the answer cites the typed site document
ADR-023Shipped

Retrieval-shaped guards stand down when the answer is an enumeration

Context

ADR-017 made breadth questions enumerate from typed fields, and the guards built for retrieval answers then destroyed the result, three separate ways. The forced-grounding guard saw zero tool calls, discarded the complete 24-member list, and rebuilt an answer from six retrieved chunks. The self-check judged the list against 500-character truncations of itself and flagged honest members as ungrounded. And the default 800-token budget cut the list short in a way indistinguishable from the model stopping early. Each guard was correct for the path it was built for and actively destructive on this one.

Decision

Enumerated and filtered turns are exempted from exactly the guards whose assumptions they break, and no others. The forced-grounding guard skips them - set members carry score 0 by design, so any re-ranked rebuild loses them structurally, not accidentally. The relevance floor skips them for the same reason. The self-check still runs, but its context is overridden to the whole member list once, verbatim and untruncated, instead of two dozen clipped copies. Set answers get a 2000-token budget. And because the model can still omit, a deterministic coverage check - no second model call - scores each member against the answer's best line at a measured 0.7 threshold (present members scored at or above 0.818, absent at or below 0.4) and appends what is missing as a visible guard step.

Alternatives considered

  • Raise top-K until the re-sorted rebuild keeps all members. Rejected: score-0 members lose to any ranked chunk at every K; the failure is structural.
  • Skip the self-check entirely for sets. Rejected: a set answer can still hallucinate attributes about real members; the fix was giving the judge the right context, not removing the judge.
  • Prompt harder for completeness instead of checking. Rejected: the coverage check exists because 'list all 24' still produced four or five - instructions are a request, the append is a guarantee.

Consequences

  • The exemptions are load-bearing and travel together: the EKS LangGraph port carries all of them explicitly, because porting the guards without the exemptions reintroduces every one of these bugs.
  • A coverage append is visible in the trace with the count of missing members, so a partial model answer is a recorded event, not a silent patch.
  • Non-enumerated turns keep every guard at full strength; the exemption is keyed on the deterministic classifier, never on model output.
Ask for all projects in /ops, then check the enumerate step and any coverage guard step
ADR-024Shipped

Setback questions get their own fan-out; strengths and failures are different retrievals

Context

The profile fan-out (ADR-008) gathered evidence with strength-shaped queries (differentiators, seniority signals, results). Asked about weaknesses, the system answered that the corpus documents no setbacks. It does: the case studies carry explicit what-went-wrong sections, written precisely so that question has an honest answer. The strength queries never surfaced them, because results-and-intro chunks outrank failure retrospectives on every strength-shaped embedding, and the denial then read as evasion, the one impression a failure question must not produce.

Decision

Failure questions route to a second, separate query set whose text names each project together with its own what-went-wrong language, exploiting the header-prefixed chunking to land on the retrospective sections directly. A generic 'tell me about Arup' runs both sets so the evidence is never one-sided. In the same change the identity query was pinned top-1 to the typed identity document (left open, it dragged in the certifications page, which reads as evidence of quality rather than identity), and the self-check was made path-aware so a fan-out answer citing a lower-ranked source is not falsely flagged.

Alternatives considered

  • One larger blended query set. Rejected: ranked fusion buries failure chunks under confident results prose (the exact mechanism that caused the denial).
  • Prompt the model to acknowledge imperfection. Rejected: it cannot cite what retrieval never fetched; an uncited admission is theatre.
  • Author a dedicated weaknesses document. Rejected: it would be self-description prose, which ADR-022 bans; the retrospectives inside real case studies are the honest source.

Consequences

  • Setback answers now cite real retrospectives, and the eval suite pins the absence of the old denial as a standing assertion.
  • Two hand-curated query lists must grow as the corpus grows; a new project with a retrospective needs its failure query added.
  • The fan-out counts as one tool call regardless of its internal query count, so the iteration cap and cost reporting stay comparable across question types.
Ask 'what went wrong in Arup's projects?' in /ops and check the citations point at retrospectives
ADR-025Shipped

The EKS agent pod gets its own minimal identity, not the provisioner's credentials

Context

The ephemeral EKS agent originally read AWS credentials from a Kubernetes Secret, and the credentials were the provisioner's own session: eks:*, ec2:*, iam:CreateRole, iam:PassRole, cloudformation:*, and ssm:GetParameter on *. That role sat inside a pod behind an internet-facing load balancer whose ask endpoint has no authentication. Anyone reaching the pod could have re-provisioned infrastructure with it. The same credentials also expired after an hour inside a demo window that can run ninety minutes, so the design was both over-privileged and self-breaking.

Decision

The pod assumes its own per-run role via EKS Pod Identity, scoped to exactly the Bedrock invoke actions on the three model ARNs the agent uses: no wildcard on foundation models, no other service. The Pod Identity agent add-on is ordered after the node group so the association exists before the workload schedules, and the trust policy grants sts:TagSession alongside sts:AssumeRole, because Pod Identity fails silently without both. AWS keys were removed from the chart, the provision script, and the image-roll workflow. The roll workflow would otherwise have quietly re-injected the old secret on every upgrade. Because /health and /ready pass with no AWS credential at all, the provision pipeline ends with a paid /v1/ask smoke test: the only signal that proves the pod can actually reach Bedrock.

Alternatives considered

  • IRSA. Rejected for this shape: it needs an OIDC provider registered per cluster, which is per-run ceremony for an ephemeral cluster; Pod Identity associates a role in one API call after the add-on is up.
  • Keep the Secret but scope the role down. Rejected: still a static credential inside the cluster, still expiring mid-run, still one exec away from exfiltration.

Consequences

  • The resume path that skips Terraform reuses a cluster without re-applying the association. The paid smoke test is what catches that, and it is why the smoke test is not optional.
  • Readiness and liveness stay credential-free by design, so a broken identity surfaces as a failed ask, not a crash-looping pod.
  • The pod can reach Bedrock and nothing else. Any future checkpointer or store for the agent needs its own explicit IAM statement, which is a feature, not a limitation.
See the workload-identity row on /production and the Pod Identity note on the EKS map
ADR-026Shipped

Teardown is verified, never assumed, and a rehearsal must report nothing

Context

Three separate incidents taught the same lesson. The hard-expiry kill switch called delete_cluster while node groups still existed; AWS refused with ResourceInUseException; a broad except swallowed it and returned ok=False, which EventBridge treats as success, so no retry ever fired and billed clusters ran on indefinitely while the UI showed green. The destroy workflow's dry-run rehearsal reported its teardown steps as done, falsifying the public run history for runs where nothing was destroyed. And nothing anywhere checked that destroy achieved anything: exit status is not evidence.

Decision

Every teardown layer now proves its claim. The reconciler follows an explicit contract (return only for terminal outcomes, raise for anything retryable, so the scheduler's retry policy actually engages), waits for node groups via ListNodegroups before touching the cluster, and carries an age-based backstop that destroys any demo-tagged cluster older than three hours using EKS's own creation timestamp. Destroy defaults dry_run=false while provision defaults dry_run=true, deliberately inverted: forgetting a flag on provision costs nothing, forgetting it on destroy leaves a live cluster billing behind a green run. A rehearsal emits a loud warning and reports nothing to the control plane. And the workflow ends by asserting the cluster is gone (describe-cluster must fail), plus a tag-based sweep of clusters, load balancers, addresses, and volumes that ends in a cleanup-needs-attention state rather than a polite log line when leftovers persist.

Alternatives considered

  • Retry the delete harder. Rejected: retrying a call that cannot succeed while node groups exist just extends the billing window; the fix was sequencing, not persistence.
  • Trust terraform destroy's exit code. Rejected: Terraform does not own the Kubernetes-created load balancers, so its success is structurally incomplete here.

Consequences

  • Five teardown layers exist (site TTL, destroy script, workflow assertion, per-run scheduler, recurring reconciler), each added after a real leftover, and the map draws all five.
  • The run history is honest by construction: a run's Destroyed and verified state means the scrubber found nothing, not that a script exited zero.
  • One specified safeguard is still unbuilt and named openly: resources that match the demo prefix but lost their tags are skipped, not quarantined for human review.
Open the teardown row on /production and the safety lane on the EKS map
ADR-027Shipped

Two backends, one evidence index

Context

The EKS LangGraph agent began life on a toy Chroma index seeded with sample text. Two backends answering from different corpora would make every cross-backend comparison meaningless. The point of running the same bounded loop twice is that differences show the runtime, not the data. But ingest, seeding, and pruning all live in the Bedrock repo's pipeline, and giving a second writer access to the same collection invites deletion races with the prune pass.

Decision

The EKS agent reads the production Qdrant collection and writes nothing. Its upload endpoint stays a stub by design; ingest remains the Bedrock pipeline's job alone. The corpus policy (the disabled self-description documents of ADR-022) is re-enforced locally in the EKS agent's own retrieval filter, because a reader cannot rely on the writer's prune timing to keep excluded vectors gone. The hybrid parameters travel with the collection: same fusion constant, same relevance floor, same sparse-coverage boost, so a score means the same thing on either backend.

Alternatives considered

  • A replicated collection per backend. Rejected: a second copy to drift, and reseed ordering across two pipelines is exactly the class of consistency bug the derived-corpus decision (ADR-021) exists to kill.
  • Grant the EKS path write access for uploads. Rejected: two writers and one prune pass is a deletion race; the demo's value is the runtime comparison, not a second upload path.

Consequences

  • A reseed by the Bedrock pipeline changes EKS answers mid-run. Accepted for a demo whose runs last thirty minutes.
  • Collection name and retrieval constants are cross-repo contracts that must move in lockstep, and each side's config names the other.
  • The EKS surface inherits corpus fixes for free: the same day the evidence corpus improved, both backends improved.
Ask both backends the same question in /ops and compare citations: same documents, same scores
ADR-028Shipped · evolving

When production drifts ahead of the repo, the repo moves to production; then the leak is closed

Context

Four separate times, what was running in production was not what the infrastructure repo would deploy. The Qdrant migration ran live while main still provisioned Pinecone. The IoT endpoint-lookup policy that the live quota badge depends on existed only as an uncommitted working-tree edit. A clean deploy from any other machine would have removed it, and the badge would have silently fallen back to polling. The CloudFront upload and delete routes vanished because empty config hostnames make conditional constructs silently absent, which presented to browsers as a CORS error. And the WAF rate rule, flipped to BLOCK by hand after its observe window, reverted to COUNT on every CI deploy because the flag lived only in the manual command.

Decision

Each rescue follows the same two-step pattern, now named. First, a commit records deployed reality exactly as it runs: zero behaviour change, honest about what could not be verified (where IAM denied reading the deployed policy, the commit says the contents were not diffed). Reverting production to match the repo is never the move: production was the version that worked, and reverting reintroduces the outage that forced the hotfix. Second, the mechanism that allowed the drift is closed in the same arc: cross-region hostnames became concrete config strings because CloudFormation exports cannot cross regions; the seed-verify step asserts the application-level result instead of the transport result; and the wafBlock decision moved into the CI command where the deploy actually happens.

Alternatives considered

  • Treat each incident as a one-off mistake. Rejected: four instances is a failure class, and a failure class gets a pattern, not an apology.
  • Lock deploys to CI only so hand-applied changes are impossible. Rejected for a one-person system: the hand-applied hotfix is sometimes the responsible move; the discipline is committing it afterwards, not forbidding it.

Consequences

  • Conditional-construct-from-empty-config is now a named hazard in the infra repo: a missing value silently removes a route instead of failing the build, so route configs are asserted, not assumed.
  • The WAF regression fix rides in the infra repo and this record is marked evolving until it merges; until then, every main push reverts the prod rule to observe-only, which this ADR states rather than hides.
  • As-deployed commits make git history honest about when decisions actually took effect, at the cost of commits whose diff is not the interesting part; the message is.
The denial-of-wallet guardrails on the cloud diagram reflect what CI actually deploys
ADR-029Shipped

The human in the loop is the visitor, and the graph genuinely pauses for them

Context

The EKS agent had six independent 'I am not confident' signals (the relevance floor, the grounding judge, the filter validator, the coverage check, the transcript-numbers guard, injection detection), and every one of them decided silently: discard, rewrite, refuse, append. Two of those silent decisions deserved a human. An unscoped projects enumeration ('what all projects has he done?') fails open and guesses which reading was meant. And at the iteration cap with thin evidence, the loop force-answers from whatever it holds, when the honest question is whether to spend more. The classic answer is an admin approval queue, and this system already has one for provisioning, which is exactly why it was wrong here: an email a sleeping admin reads tomorrow cannot answer a question a visitor asked ten seconds ago.

Decision

Two checkpoints, both facing the person asking, both real LangGraph interrupts against a checkpointer, never a UI-side confirm. Clarify fires in the seed node before any model call, only when a projects enumeration has no topic in its wording: the graph parks, the Ask tab renders the readings (all N, or one of the top typed domains), and the graph resumes down the chosen branch. The fail-open guess became a question. Extend fires at most once, at the iteration cap, only when the model was still asking for tools and the best evidence score is under a threshold: the visitor sees the turns used, the top score, and the priced cost of more, and either authorizes a bounded extension or takes the honest partial answer. The cap never moves without a human saying so. Both are opt-in per request (smoke tests and evals never pause), and both decisions land in the trace as a visible human step. The checkpointer is in-memory, deliberately: one replica, thirty-minute runs, and a pod restart forgetting a paused ask degrades to 'please ask again' rather than justifying a durable store the pod's Bedrock-only role (ADR-025) would have to widen for.

Alternatives considered

  • Admin-approval of held answers via the existing signed-email machinery. Rejected: the reviewer latency is unbounded and the visitor is right there; an approval the human cannot see arrive is theatre.
  • A UI-side confirm without pausing the graph. Rejected: nothing is actually parked, so the demo would claim an interrupt the runtime never performed (the exact class of false trace claim the Ops console exists to avoid).
  • Interrupt before every retrieval, the textbook example. Rejected: retrieval here is read-only and costs milliseconds; pausing for it is ceremony, and ceremony teaches visitors to click through checkpoints.
  • A durable checkpointer so pauses survive restarts. Rejected for this shape: it needs a new IAM statement on a deliberately Bedrock-only pod role and a store that outlives a cluster whose whole point is to burn.

Consequences

  • The agent went beyond its Bedrock sibling for the first time: when this shipped, Bedrock's loop could not pause mid-graph, and the comparison was a real architectural difference rather than a port. ADR-030 has since closed that gap with a checkpoint-and-resume mechanism, so the difference is now the plumbing, not the capability (see ADR-REVISIT).
  • An unanswered checkpoint holds until the run's own TTL tears the cluster down. The demo's existing cost ceiling is the timeout, so no new one was invented.
  • The extension is the one place a visitor can raise the spend of a single ask, and it is bounded (two turns), priced in the payload, and offered once. The denial-of-wallet story survives contact with HITL.
  • Non-interactive callers keep the exact old behaviour, which is what lets the paid provision smoke test stay a one-shot curl.
Start the EKS demo, ask 'what all projects has he done?', and answer the clarify gate the trace then shows
ADR-030Shipped · evolving

The Bedrock loop pauses too: checkpoint to DynamoDB, resume in a fresh invocation

Context

ADR-029 put two visitor-in-the-loop checkpoints on the EKS agent and named the gap it left: the Bedrock backend, the always-on production track, could not pause mid-graph, and the comparison between the backends was a real architectural difference rather than a port. That gap was real but not structural. What a Lambda genuinely cannot do is park an in-memory graph and wait, because the invocation that holds the loop ends when the stream ends. What it can do is write the loop's state down and stop.

Decision

The same two checkpoints as EKS land on the Bedrock backend with a different mechanism: clarify fires before any model call when a projects enumeration has no topic in its wording (list all of them, or focus on one area), and extend the loop fires at the three-turn cap with weak evidence, offering up to two more priced turns or an honest answer from what exists. A pause checkpoints the loop state to the existing DynamoDB session table (30 minute TTL) and ends the stream; the visitor's decision resumes it in a fresh invocation from that snapshot. The wire semantics match EKS exactly: a paused status, the options, a priced extend, and the resume decision recorded in the trace as a visible human step. Quota is charged once per ask, not once per invocation, and the TTL means an abandoned decision card self-cleans instead of accumulating. Opt-in per request (interactive: true), so every non-interactive caller keeps the old behaviour, and nothing runs and nothing is spent while the card is on screen.

Alternatives considered

  • Hold the Lambda open on the SSE stream, waiting for the visitor to decide. Rejected: it burns paid duration against the 90 second timeout and couples a human decision to a network connection staying alive.
  • A separate /resume CloudFront path with its own Lambda. Rejected: new edge behaviour plus the hostname-injection dance for another route, and a forgotten behaviour 404s with no CORS, the silent-route failure class ADR-028 already catalogued.
  • A durable LangGraph-style checkpointer library. Rejected: the loop is a few hundred lines of TypeScript; a serialized snapshot in the existing session table is the whole job, and a framework around it would be more code than the feature.

Consequences

  • The paid production proof is pending: this lands via a site PR and a rag-api PR, and the backend only deploys through the infra repo workflow, so at authoring time the checkpoints are merged but not yet live in production. This record says so instead of rounding up.
  • The two backends now share the interrupt contract but not the mechanism: EKS parks an in-memory graph behind one replica, Bedrock serializes to DynamoDB and resumes in a fresh invocation. The comparison is honest again, just narrower.
  • ADR-029's consequence that Bedrock's loop cannot pause mid-graph is superseded; that line was updated in place with the supersession noted, the original wording is preserved verbatim in ADR-REVISIT, and the counter-evidence trail lives there too.
  • A pause that outlives its 30 minute TTL is gone by design; resuming after that means asking again, which keeps abandoned cards from becoming standing state.
In the /ops Ask tab with the Bedrock backend, ask an unscoped projects question and answer the decision card the trace then shows
AboutEMpathWritingProductionConnect