Decisions

Architecture decisions

The reasoning behind the live Portfolio Ops system, recorded as decisions with their alternatives and their costs. You can check each one against the running system.

Every record names the alternatives I rejected and the costs I took on. Where a decision is visible in the live system, there is a link to go check it.

ADR-001Shipped

Pinecone serverless for vector search, not pgvector or OpenSearch

Context

The corpus is small: about ten documents and a few dozen chunks, plus short-lived visitor uploads. One person maintains it, and the demo has to stay cheap while idle. The retrieval layer should not turn into an operations job.

Decision

Pinecone serverless, with 1024-dim Titan v2 embeddings and cosine similarity. searchSimilar() scopes uploads to the asking session via a sessionId metadata filter, so a visitor's uploaded document is reachable to them, in its own Q&A box and as a reference for the probe, and never to anyone else.

Alternatives considered

  • pgvector on RDS: another stateful service to run and scale, which is overkill at this volume.
  • OpenSearch or self-hosted vectors: heavier to operate and slower to stand up, with cost even at rest.
  • An in-memory index in the Lambda: nothing persists across cold starts, and uploads have no shared state.

Consequences

  • Almost no cost while idle, and no index to operate. That suits a solo-run demo.
  • Embeddings run in ap-south-2 while the index lives in us-east-1, so each search pays a cross-region hop. That latency shows up in Inspect.
  • If the corpus ever grew large, the right move is to colocate the regions and add sparse-dense vectors. Written down, not pre-built.
Superseded by ADR-011 — retrieval now runs on Qdrant hybrid
ADR-002Shipped · evolving

A relevance floor over dense retrieval; the hybrid it called for shipped as ADR-011

Context

Dense cosine handles semantic questions well; real matches in the golden suite score between 0.33 and 0.79. But a short, keyword-heavy query exposed the classic dense-only gap. When nothing relevant is indexed, the closest documents still come back around 0.10, which is noise, and the model, handed that thin context, can write a confident denial. On a demo whose whole pitch is transparent retrieval, that is the worst thing that can happen.

Decision

Keep dense-only as the base and add a calibrated relevance floor. When the top score drops below about 0.20, the system says "no strong match in the index" instead of letting the model speak from noise. Hybrid retrieval (BM25 or sparse fused with dense) was written down as the next step rather than pre-built, so the failure would earn it instead of a roadmap guess. It did: hybrid shipped in ADR-011. The floor is still in force on top of it, because a hybrid match can be weak too.

Alternatives considered

  • Ship hybrid right away: more infrastructure and a reindex, so it was deferred until the floor proved insufficient.
  • Do nothing, and leave the model free to deny things from noise. Rejected, because it breaks the honesty invariant.

Consequences

  • Off-corpus questions now get an honest empty-context answer, with the top score shown in Inspect.
  • Keyword recall for rare tokens stayed limited while this was dense-only. That gap was written down rather than hidden, and ADR-011 closed it with sparse + dense fusion.
  • The floor is a single configurable number, calibrated against the eval suite so it never trips a real match.
Top-score + below-floor signal in /ops Inspect
ADR-003Shipped

Amazon Nova Pro + Titan Embed v2 on Bedrock

Context

The system needs embeddings, generation, and tool-use on AWS-native infrastructure, with one IAM and billing surface and a predictable cost.

Decision

Titan Embed Text v2 (1024-dim) for embeddings and Nova Pro for generation, both on Bedrock. Generation goes through the Converse API, which gives native tool-use. That is what made the agentic loop possible without bolting on a separate framework.

Alternatives considered

  • External OpenAI or Anthropic APIs: cross-cloud egress, plus a second billing and key-management surface.
  • Self-hosted open models: inference operations and GPU cost a demo does not justify.

Consequences

  • One cloud, one IAM model, one bill, so least-privilege per Lambda stays simple.
  • Converse tool-use unlocked the agent directly (see ADR-004).
  • Model choice is now mostly about cost and latency rather than capability, and the readout in Inspect makes that explicit.
Token usage + timings in /ops Inspect
ADR-004Shipped

A bounded agentic loop with a self-check, not an open-ended ReAct agent

Context

The hero claim is agentic systems, so the demo should run a real agent. But the most common production failure for agents is the runaway loop: unbounded tool calls that burn latency and money. Multi-agent sprawl would add risk here without adding signal.

Decision

One read-only retrieve tool, a hard cap of three tool iterations, and an LLM-as-judge self-check that confirms the answer is grounded in retrieved context before it is finalized. Two guardrails reinforce the bound without loosening it: if the model ever tries to answer without retrieving, the harness forces a single grounding retrieval; and questions about Arup himself draw evidence from a separate, deterministic fan-out (ADR-008) instead of relying on the loop to multi-hop. Below-floor or ungrounded answers are flagged honestly, never invented. Every step is emitted as a visible trace.

Alternatives considered

  • An unbounded ReAct loop: open-ended cost and latency, the exact failure this guards against.
  • Multi-agent orchestration: more moving parts, no better answers at this scale.
  • No self-check: faster, but it lets ungrounded claims through.
  • Trusting the model to multi-hop on its own: unreliable per-phrasing on this model, which is why broad questions use the fan-out instead.

Consequences

  • A predictable cost and latency ceiling per query, narrow questions usually take one tool call.
  • Restraint instead of theatrics. The hard cap is the safety story, and it is what the agent-eval asserts on.
  • A genuinely hard question can stop a step early. That is an accepted trade for bounded, inspectable behavior.
  • The grounding guardrail makes a skipped retrieval impossible, so a stale conversation can't produce a false 'not in the corpus' for indexed content.
Live trace + Agent Evals (7/7) in /ops
ADR-005Shipped

Session-scoped, TTL-expiring, quota-bounded visitor uploads

Context

Visitors can upload a document and query it live. Anything public that touches storage and a model has to be safe by default. It cannot leak across visitors, run up unbounded cost, or leave data lying around.

Decision

Uploads are scoped to a session and expire on a 24-hour TTL (an S3 lifecycle rule plus a vector-store cleanup path, now Qdrant). They sit behind per-day global upload and query quotas, with a corpus-busy lock during index mutations. The retrieve tool is read-only, so an instruction injected into an uploaded file has no action it can take.

Alternatives considered

  • Persistent uploads: storage, cost, and privacy obligations a demo should not carry.
  • No quotas: an open door to abuse and runaway spend.
  • A shared, unscoped index: cross-visitor data leakage.

Consequences

  • Safe to leave open to the public. The blast radius of a malicious upload is one short-lived session.
  • Ephemerality becomes a feature, since the teardown is part of the demo.
  • By design, it is not a durable multi-user document store.
Upload → query → delete in /ops Upload
ADR-006Shipped

The corpus panel is generated from the live index, credibility is the product

Context

The homepage panel that shows what's indexed started as a hardcoded list, and it drifted. It advertised documents that were never indexed while hiding the ones that were. A visitor could ask about an advertised document and watch the model correctly say it isn't there, a contradiction sitting two inches from the claim.

Decision

Generate the panel from a GET /corpus endpoint that lists what is actually in the vector store (a Qdrant scroll over the corpus/ path prefix), with a real-corpus static fallback. The panel cannot advertise a document that isn't indexed.

Alternatives considered

  • A hardcoded or hand-curated list, which is the thing that drifted in the first place.

Consequences

  • The panel can never show a document that isn't there, and new documents appear once they are indexed.
  • One cheap, cached API call on load.
  • This is the honesty invariant in practice: the whole demo's value is that nothing on screen is faked.
The corpus panel on the homepage
ADR-007Shipped

Defense in depth against prompt injection (OWASP LLM01)

Context

The agent reads visitor-uploaded documents and its own corpus, then feeds that text to a model. Prompt injection is OWASP's top LLM risk: a document or a question that says "ignore your instructions and do X." Neither RAG nor fine-tuning removes it, so it needs layered defense rather than a single filter.

Decision

Four layers. The retrieve tool is read-only, so there is no privileged action to hijack. The system prompt treats the question and all retrieved text as data, never as instructions. An injection-pattern guard surfaces attempts in the trace so they are visible rather than silent. The grounding self-check and the relevance floor keep the answer tied to real sources, and input is stripped of control tokens before it reaches the model.

Alternatives considered

  • A single regex filter as the defense: brittle, and it gives false confidence. Detection here only adds visibility on top of the structural layers.
  • Trust the model to behave: injection is a known, repeatable failure, and hope is not a control.

Consequences

  • An injected instruction has no action it can take and does not change the agent's behavior; the attempt shows up in the trace as a guard step.
  • Pattern detection will miss novel phrasings, which is exactly why it is the visible layer and not the load-bearing one.
  • With session isolation and TTL (ADR-005), the blast radius of a malicious upload stays inside one short-lived session.
Type an injection into /ops Ask and watch the guard step
ADR-008Shipped

Deterministic fan-out for questions about Arup, not model-driven multi-hop

Context

Questions about Arup himself, USP, strengths, why-hire, setbacks, should be answered from the breadth of his real projects, not a single bio document. The first attempt told the agent to multi-hop across projects on its own. It worked for some phrasings and gave up on others (asked for the 'USP', it retrieved the map and decided the corpus had nothing). Model-driven decomposition was too phrasing-fragile on Nova Pro to ship to strangers.

Decision

A profile question triggers a deterministic fan-out: the harness issues a fixed set of focused queries across Arup's distinct projects, gathers the evidence, and the model synthesises a cited answer from it. The about-arup document is a routing map that holds no conclusions of its own, so it can't shortcut the answer. The agentic loop stays bounded at three (ADR-004); the fan-out is a separate, fixed pre-retrieval step, not extra loop iterations.

Alternatives considered

  • Model-driven multi-hop (PRISM-style): more elegant and general, but unreliable per-phrasing on this model.
  • A single 'about me' summary doc: dominates retrieval and collapses every answer to one self-referential source.
  • Multi-agent planning (MA-RAG-style): more machinery than a single, known question-class needs here.

Consequences

  • Profile questions reliably cite several real projects instead of one bio page, at the cost of a few extra retrievals, shown honestly in the cost-and-latency readout.
  • A deliberate trade of agentic elegance for reliability. The upgrade path is to hand decomposition back to the model once a stronger tool-use model makes it dependable, keeping the fan-out as a fallback.
  • Narrow project questions are unaffected, they still do a single retrieve.
Ask 'Why hire Arup?' in /ops and watch the fan-out
ADR-009Shipped

Layered spend caps against denial-of-wallet, not a single limit

Context

The query path is a public, unauthenticated Lambda Function URL behind CloudFront, and an agentic answer calls Bedrock several times. A Function URL has no built-in throttling, so a flood could run up an invocation, DynamoDB, and model bill before any single guard noticed. The per-call token caps and the global daily quota already hard-capped the model bill, but a flood still paid for Lambda and DynamoDB work, and one visitor could drain the whole day's quota by accident.

Decision

Cap spend in independent layers, each cheap and each failing safe. At the edge, a WAF rate rule drops a single IP above 100 requests per five minutes before it costs an invocation. In the Lambda, a per-IP daily sub-quota of 8 sits under the global quota of 20; the real client IP is read from X-Forwarded-For since traffic is behind CloudFront. That per-IP cap is a FAIRNESS mechanism, not a security one, and is worth stating plainly because it is easy to mistake for a wallet guard: anyone willing to rotate IPs walks straight through it, and it deliberately fails open on a DynamoDB hiccup. What it actually buys is that one ordinary visitor cannot accidentally consume the whole day and leave the demo dead for everyone else. The thing that genuinely bounds spend is the atomic global#<date> counter, backed by the $5/day budget. At the function tier, the account's Lambda concurrency ceiling is already low enough to bound how wide a flood can fan out, so no slice was carved out for the query function: reserving one would only starve the sibling functions out of the same small pool, and provisioned concurrency was never on the table since it bills around the clock. An account-wide $5/day cost budget is the backstop that catches whatever the other layers miss.

Alternatives considered

  • One global rate limit and nothing else: still safe for the wallet, since the global counter is the real bound, but one visitor could exhaust the shared day and a flood would still pay for invocations the quota never stops. The per-IP cap is kept because it costs nothing, not because it closes an attack.
  • Provisioned concurrency: bills around the clock for warm capacity the demo does not need, which is the opposite of the goal here.
  • API keys or auth on the endpoint: friction for a public portfolio whose whole point is that a stranger can try it without signing up.

Consequences

  • A flood is dropped at the edge, slowed per IP, held under the account's concurrency ceiling, and bounded again in total dollars. Only the global counter and the budget are load-bearing against a determined attacker; the other layers raise the effort. The edge, quota, and budget layers each deploy and revert on their own.
  • The WAF rule and the budget add a few dollars a month, the only recurring cost the hardening introduces.
  • The per-IP quota fails open on a DynamoDB hiccup, leaving the global quota and the WAF rule as the hard backstops, so a storage blip never blocks a real visitor.
Edge rate-limit and caps shown on /architecture
ADR-010Shipped

Deterministic concept routing for abstract project questions

Context

The live agentic loop works well when a visitor asks in the same language the corpus uses: project names, outcomes, and implementation details. It gets shakier when the question comes in as abstract engineering language and the corpus stores the evidence in concrete project terms instead. A question like "Is there an example of race condition handling, and what is its relation to idempotency?" is answerable from the portfolio. The problem is that the first retrieval path kept circling the abstract words and never landed on the project language underneath them: double-bookings, retries, double-clicks, Redis locks, replay, dedup. The issue was not missing evidence. The issue was that the system had no reliable bridge into it.

Decision

Add a deterministic concept-routing layer ahead of the bounded agentic loop for a small class of abstract project questions. The router classifies concept-style queries such as example-of, relation-between, where-did-you-use, and difference-between. For those classes, it expands canonical concepts into corpus-native terms and known project candidates before retrieval. The LLM still reads the evidence, explains it, and writes the final answer. What changes is that it no longer has to guess its way from abstract phrasing to the right project evidence every time. A companion guard blocks answers that cite only routing documents like about-arup without citing any real project document.

Alternatives considered

  • Trust the model to reformulate the query on its own. Nice when it works, too brittle for a public demo.
  • Add glossary-style explainer documents for every concept. That might help one phrasing, but it pulls the portfolio away from project evidence and toward textbook filler.
  • Rely on hybrid retrieval alone. That should help lexical recall, but it will not fully solve relation-style questions because they still need a bridge into the right projects.
  • Make everything deterministic, including answer composition. More rigid, and it gives up the part the model is actually useful for.

Consequences

  • Abstract engineering questions became less wording-sensitive, while narrow project questions keep the simpler path.
  • The architecture becomes less purely agent-driven at retrieval time, but more dependable and easier to inspect.
  • A small maintained concept map becomes part of the harness. That is extra structure to own, but it is cheaper than letting repeated public misses pile up.
  • The split becomes cleaner: deterministic logic owns routing, candidate narrowing, and evidence gating; the model owns synthesis and explanation.
Ask concept-style questions in /ops and inspect the route and evidence mix
ADR-011Shipped

Migrate from Pinecone dense-only retrieval to Qdrant hybrid retrieval

Context

The original Pinecone decision was right for a tiny, mostly semantic corpus and almost-zero idle cost. The later retrieval failures were not operational failures, they were retrieval-quality failures: rare proper nouns, explicit failure-language sections, and exact engineering terms were all the kinds of queries dense-only retrieval predictably drops. Once those misses were visible in the live evals, the system had earned a real hybrid retrieval layer rather than more app-side patches.

Decision

Keep the public API contract stable and swap the retrieval store under it. The production path now uses Qdrant Cloud with one hybrid collection: Titan v2 dense vectors, BM25 sparse vectors, reciprocal-rank fusion at query time, the same session-scoped payload filtering for uploads, and the same calibrated confidence floor before the model answers. The migration is documented as an evolution after ADR-001 and ADR-002, not a rewrite of them.

Alternatives considered

  • Stay on Pinecone and keep patching dense-only misses with routing and upload-slot heuristics. That helps specific cases, but not the base lexical recall problem.
  • Run dense and sparse search in separate systems and fuse in the application layer. More moving parts, with no upside at this scale.
  • Add a reranker before fixing first-stage recall. Better ordering would not solve the chunks that never got retrieved.

Consequences

  • Entity names, exact terms, and failure-language queries are now much more likely to surface the right chunks on the first retrieval pass.
  • The frontend and deployed API shape stay stable, so the migration does not force a public UI rewrite.
  • The historical Pinecone and dense-only ADRs remain true as records of the earlier system; ADR-011 explains why the later system changed.
See the hybrid retrieval path in /architecture and the live retrieval diagnostics in /ops
ADR-012Shipped

Treat Qdrant hybrid retrieval as the current and future architecture baseline

Context

Once the migration was complete, the architecture story had two truths that needed to coexist cleanly. Historically, Pinecone was the right first retrieval store for a tiny dense-only corpus with almost no idle cost. Operationally, that system no longer exists in production. The live stack, the inspect tooling, the eval harness, and the public API now run on Qdrant hybrid retrieval. Leaving current or destination architecture views anchored on Pinecone would blur the line between historical record and present system.

Decision

Keep the old Pinecone ADRs unchanged as history, and append a new baseline ADR that says the live and forward-looking architecture is Qdrant-first. Current-state and eventual architecture references should describe one hybrid Qdrant collection, Titan dense vectors, sparse lexical retrieval, reciprocal-rank fusion, and the existing session-scoped filtering and confidence floor. Pinecone remains part of the story only where the goal is to explain the earlier system and why it changed.

Alternatives considered

  • Rewrite the original Pinecone ADRs so every record reads as if Qdrant had always been the choice. Rejected because it erases the real evolution of the system.
  • Leave current and target architecture copy split between Pinecone-era and Qdrant-era descriptions. Rejected because it confuses readers about what is actually live.
  • Maintain separate 'historical' and 'current' architecture pages for the same stack. Rejected as too much ceremony for a small public system.

Consequences

  • The architecture narrative stays historically honest without making the live system look ambiguous.
  • Future diagrams, inspect copy, and production architecture notes can all assume Qdrant hybrid retrieval as the baseline retrieval layer.
  • Pinecone remains visible where it should: as the earlier decision that was later superseded, not as the current production dependency.
The live and target retrieval paths shown in /architecture and the production diagnostics in /ops
ADR-013Shipped

Disable the About Arup retrieval shortcut while keeping the profile fan-out

Context

Hybrid retrieval fixed the lexical-recall problem, but it did not make the old biography shortcut healthy. The about-arup document was still a tempting answer-collapse path: even when the system had enough project evidence, one broad self-summary could reappear in traces and citations and make profile answers look sourced from a bio rather than from the work itself. That is the wrong shape for a public architecture whose credibility depends on project-grounded evidence.

Decision

Keep the deterministic profile fan-out, but disable corpus/about-arup.md from active retrieval, live corpus listings, and future seeding. The file stays in the repository for now as historical material and for controlled testing, but the production retrieval path should synthesise broad profile answers directly from project documents and case studies. Hybrid Qdrant retrieval remains the first-stage recall layer; the fan-out remains the control-plane step for broad profile questions.

Alternatives considered

  • Leave the thin About Arup map indexed and trust guardrails to keep it from dominating. Rejected because the shortcut still leaks into citations and weakens the evidence story.
  • Remove the deterministic profile fan-out now that hybrid retrieval is live. Rejected because hybrid improves recall, not cross-project aggregation discipline.
  • Delete the file immediately from the repository and rewrite the earlier ADR trail around it. Rejected because the file is still useful as historical evidence while the new path is being tested.

Consequences

  • Profile answers now have to stand on project evidence, not a single self-summary document.
  • The public corpus panel stays honest: disabled documents are no longer advertised as live retrieval sources.
  • Testing can continue with the file preserved on disk, while production behavior reflects the new architecture baseline.
Ask broad profile questions in /ops and inspect the citations and fan-out behavior
ADR-014Shipped · evolving

Split ephemeral EKS into keep vs destroy stacks

Context

An EKS LangGraph lab only proves something useful if it looks production-shaped: real cluster, Helm, a public load balancer, traces, teardown. Running that stack all month would burn money on the control plane, nodes, load balancer, and public IPv4 even while nobody is watching. The design problem is not 'can we run EKS' — it is 'can we run EKS for thirty minutes and trust that expensive leftovers cannot linger.'

Decision

Split resources by billing risk. Foundation (keep): ECR images, S3 artifacts and Terraform remote state, IAM/OIDC trust, optional minimal VPC shell without NAT, reused Qdrant Cloud collection, LangSmith/Langfuse project, and the control-plane run store plus EventBridge reconciler schedule. Ephemeral (destroy every run): EKS cluster and node group, the Service load balancer (a Classic ELB in practice), public IPv4 allocations, Helm agent releases, demo log groups with short retention, and any demo-specific DNS. Never create EKS without a provider-side one-time teardown schedule. Never report Destroyed until a tag-based cleanup audit confirms the expensive set is gone.

Alternatives considered

  • Keep a long-lived EKS cluster and only redeploy the app. Rejected: control-plane and node hours dominate cost and teach the wrong ops lesson.
  • Destroy everything including ECR, IAM, and Terraform state after each demo. Rejected: recreating trust and registry setup every run slows demos without reducing meaningful spend.
  • Rely only on an in-app timer to tear down. Rejected: if the portfolio app is down, billing continues; a provider-side schedule and reconciler are required.
  • Put NAT gateways and interface VPC endpoints in the foundation stack 'for convenience.' Rejected: they have hourly charges and defeat the near-zero idle goal.

Consequences

  • Idle cost stays near zero: between demos there is no EKS, load balancer, or public IPv4 to bill.
  • Demo setup stays fast because images, state, and IAM do not churn.
  • The Ops and Architecture UIs can tell an honest story: Idle = keep stack only; Running = keep + destroy stack; Destroyed and verified = scrubber passed.
  • Cleanup failures surface as Cleanup needs attention instead of a false Destroyed status.
Toggle Idle vs Running on /architecture · Ephemeral EKS, and walk the lifecycle on /ops
ADR-015Shipped

Chat mode: one durable transcript, never combined with Bedrock's own memory

Context

Bedrock was already a chat backend before this shipped: session.ts kept a rolling 20-message window (10 turns) and query-rewrite.ts already rewrote conversational follow-ups against it, on every query, with no UI ever saying so. Adding an explicit chat mode on top risked the worst possible failure: if the site ALSO sent a durable transcript from its own store while the backend kept reading its own session history, every prior turn would land in the model's payload twice and inputTokens would roughly double for the same conversation, a correct-looking readout of a silently doubled bill.

Decision

One source of truth for conversation context, never two. query.ts accepts an optional `transcript`; when present it REPLACES the session.ts read and write entirely rather than adding to it (verified by direct measurement against live Bedrock: doubling the same history roughly doubled inputTokens, confirming the mechanism the guard prevents). A `single` mode sends `noHistory: true` instead, reading no history and writing none, so switching back to `chat` resumes exactly where it left off. Chat turns persist for 1 year in `portfolio-ops-chat-<stage>` (one item per turn, not one fat item per chat, to avoid the DynamoDB 400 KB ceiling and the read-modify-write race a single fat item would reintroduce). The system prompt's TEXT is never rendered anywhere, even in the 'View original' viewer, only its role, line count, and an approximate token size, because agent.ts actively refuses to reveal that prompt and printing it in Inspect would make that refusal theatre.

Alternatives considered

  • Send the transcript AND let query.ts keep reading its own session history. Rejected: this is the doubling trap above, and it would have shipped invisibly, since Inspect would faithfully report the inflated (wrong) token counts as if they were correct.
  • Drop session.ts entirely once the durable chat store exists. Rejected: the site's other, non-Ops callers of /query (e.g. the homepage widget) don't send a transcript and still benefit from short-lived server-side memory; removing it would silently regress their behavior.
  • Cap the per-chat quota above the model's 10-turn context window. Rejected: the 5-turn cap was chosen specifically to stay under that window, so the visible transcript never shows more turns than the model actually still has in context.

Consequences

  • A visible mode toggle in /ops is the honesty fix and the demo at once: the same follow-up scores worse in `single` mode (no history reaches the model) and better in `chat` mode (rewritten against the prior turn), run by the visitor rather than asserted by copy.
  • Chat turns are kept for a year while uploaded documents expire in 24 hours (ADR-005), a real and deliberate difference in privacy posture on the same site that is now surfaced in Inspect rather than left implicit.
  • EKS's `AskRequest.session_id` — declared and silently discarded — is replaced by an equivalent `transcript` field, capped server-side; a client-supplied cap is never trusted.
Toggle Chat vs Single question in /ops Ask, then compare Inspect's per-turn token growth against the same follow-up in each mode
ADR-016Shipped

Push live quota over AWS IoT Core, not a held Vercel connection

Context

The quota panel showed a stale count until the visitor's next question. An earlier draft assumed serverless has no push at all, which is wrong - SSE genuinely is push, the server holds the connection and writes when it chooses. The real problem was narrower: on Vercel, the SSE handler and the query handler that decrements the counter are separate invocations with no shared memory, so a held SSE stream would have no channel to learn the counter changed without some other broker in between anyway.

Decision

Let the browser subscribe directly to AWS IoT Core over MQTT/WebSocket instead of holding any Vercel connection open. A site route mints a short-lived SigV4-presigned wss:// URL (reusing the existing AWS_ROLE_ARN Vercel-OIDC role, so no Cognito identity pool is needed); the query path publishes the new remaining count onto one topic with the MQTT retain flag after each decrement, so a newly-connecting tab gets the current value immediately instead of a blank gap. The IAM policy behind that presigned URL is deliberately subscribe-only on exactly that one topic - no publish, no wildcard - since this is a public, unauthenticated browser connection and, per AWS's own authorization model for SigV4/WebSocket clients, that policy governs every MQTT frame the connection sends for its whole lifetime, not just the initial handshake.

Alternatives considered

  • Keep polling GET /quota on an interval: simple, but every visitor's tab pays a request for a value that changes at most ~20 times a day.
  • SNS, EventBridge, or SQS fan-out to a Vercel webhook: none of these can push into a browser tab that's already holding a connection open - they deliver to endpoints (Lambda, SQS, HTTPS), which is a fresh invocation per message, not a channel into an existing session.
  • A Cognito identity pool for browser credentials: unnecessary machinery here, since the site already holds a Vercel-OIDC role capable of being scoped down to mint the presigned URL directly.

Consequences

  • No Vercel function is held open for the life of a viewer's tab, and IoT Core's billing (connection-minutes) is the only new cost - closing the connection on visibilitychange keeps a backgrounded tab from paying for nothing.
  • The security posture rests entirely on that one IAM policy staying subscribe-only, single-topic - the comment in portfolio-rag-infra's quota-broadcast-stack.ts says so explicitly, so a future edit can't widen it by accident.
  • Only the query path publishes. readQueryQuota(), which serves the REST fallback the browser reads on mount, must never publish, because it reads the counter without decrementing it and would push a value that had not changed.
  • The endpoint is discovered rather than configured, which reverses the original handoff's reasoning. That doc argued a hand-set env var was cheaper than an iot:DescribeEndpoint permission and a round trip. The round trip is cached per warm instance, and the hand-set value is exactly what left this feature dark after every other piece was provisioned, so the cheap option was the one that failed. iot:DescribeEndpoint supports no resource-level permissions and so needs a wildcard resource; it lives in a SEPARATE policy for that reason, keeping the browser-facing one provably wildcard-free.
  • All three pieces are deployed and the path is observed end to end, not inferred: subscribing exactly as a browser does, a retained message arrived immediately on connect showing 15 remaining, and a live push arrived showing 14 while a query ran. One AWS-specific detail cost the most and is worth recording: @smithy/signature-v4 puts X-Amz-Security-Token inside the canonical query string whenever credentials carry one, which Vercel OIDC always does, and IoT Core rejects a signature computed over the token. Every connection closed with no connect event and no error frame while the URL looked correct in every respect. The token is now appended after signing. Nothing short of deploying and actually subscribing would have found it.
Open /ops and watch the remaining-questions count drop the moment a query runs, pushed over MQTT rather than polled
ADR-017Shipped

Breadth questions are enumerated, not searched

Context

A visitor asked what technologies Arup had worked with and got three AI items, then asked for anything apart from AI and got the AWS ML stack, TypeScript and Zod, every one of them AI. Two distinct failures sat underneath that. The exclusion was one, fixed separately by lifting the negation out of the embedding and into a must_not filter, since negation is not representable in embedding space. The other is that "what technologies has he worked with" is a global question, the class chunk-retrieval RAG structurally cannot answer: retrieval returns short contiguous passages, and no single passage in this corpus frames the whole of it. The information was there, webhooks in six documents, Redis in three, dual-write and feature flags in two each, but it lived inside project narratives that do not resemble the abstract word "technologies".

Decision

Two steps, and the first alone was not enough, which is worth recording rather than smoothing over. First, add one '## At a glance' section per corpus document, stating what it is, its domain, and the technologies it names, in the vocabulary breadth questions use. The existing header chunker turns each into exactly one chunk, so that was fourteen extra chunks and no retrieval code. It moved the hard case from 0.123 to 0.173 against a floor of 0.2, and still refused. Second, and this is what actually closed it: breadth questions ENUMERATE that layer instead of searching it. Every document summary is fetched by filter, unranked, and placed in the prompt. Retrieval was simply the wrong verb for a question with no single passage to match, and no amount of better matching was going to fix that.

Alternatives considered

  • GraphRAG: entity extraction, community detection, and map-reduce over community summaries. It answers global questions by ENUMERATING every community rather than searching, which is a stronger guarantee than a summary layer gives. It is also machinery whose entire purpose is to avoid reading a corpus too large to enumerate, and this corpus is about 10,600 tokens, roughly three percent of the model's context window. Paying that complexity for a constraint that does not bind would be architecture theatre.
  • RAPTOR proper: recursive clustering and summarisation into a tree. Same objection at this scale, and clusters spanning fourteen documents would mostly reproduce the documents.
  • Stopping at the summary layer and leaving the hard case refusing. Defensible for a while, since an honest refusal beats a confident wrong answer, but it left a question the corpus could clearly answer going unanswered. Superseded by the enumeration above, which shipped with the labelling requirement met rather than waived: the survey is its own step kind and the strategy label renames itself, because a fan-out over every document is not retrieval and this site's claim is that visitors are watching real retrieval happen.
  • Lower the relevance floor when a filter has already constrained the candidate set. Rejected as a hidden threshold. Lowering a floor to get an answer out is how confident wrong answers are manufactured, which is the exact failure this system exists to prevent.

Consequences

  • Breadth questions now have something shaped like an answer to retrieve, and the fix is legible: it is corpus content, reviewable in a diff, with no new retrieval path to reason about.
  • The reported question now answers correctly. Asked what technologies Arup has worked with apart from AI, the system surveys eight documents (six excluded on request) and returns PostgreSQL, Django, React, feature flags, dual-write, Lambda, Iridium, offline-first sync and face verification, with no AI document cited. The arc took four steps and each was necessary: exclusion filter, query reformulation, summary layer, enumeration. None alone was sufficient, which is why none of them was a reason to lower the floor.
  • The relevance floor was never lowered. It is skipped only when a survey ran, because a search-derived similarity says nothing about an enumeration that read every document, and the GROUNDING self-check is untouched: this relaxes the relevance guard, never the honesty one. Whether 0.2 is the right number for ordinary retrieval is still open, and still the separate decision it always was, now without a failing case pushing on it.
  • Summaries are claims, so they carry an invention risk the prose does not. The no-invention check is what makes the layer safe to extend; it caught two false positives during authoring.
  • Enumeration is visible, not substituted. The survey emits its own step kind, the retrieval diagnostics rename the strategy when one runs, and survey citations carry a score of zero because nothing was ranked, since a plausible-looking similarity would be a fabricated number in the Inspect panel. The exclusion filter binds inside the enumeration too, so reading everything never became a way around a filter the visitor asked for.
  • Qdrant refuses outright to filter on an unindexed payload field, so the keyword index on 'kind' is not an optimisation, it is what makes the survey work at all. Found before deploying only because a filtered scroll errored while an unfiltered one did not.
Ask what technologies Arup has worked with in /ops, then ask for anything apart from AI, and watch the trace show a survey rather than a retrieve
ADR-018Shipped

A filtered follow-up over an enumerated set is validated, not trusted

Context

A visitor asked what are different projects Arup were a part of and got the correct answer: all 24 projects, enumerated from the typed docType field per ADR-017. The follow-up, which one of these involve generative AI, came back with AI Architecture Patterns I Use in Production - docType reference, one of the three documents that is explicitly not a project, and never a member of the 24 the visitor had just been shown. classifySetQuestion, the narrow gate that routes a question to the enumerate path, returns null on that phrasing by design: it only fires on an explicit enumeration cue or the canonical whole-corpus phrasing, precisely so it does not misfire on a question that is actually scoped to one project. A filtered follow-up has neither signal, so it fell through to ordinary similarity search, where reference documents are fair game again. Turn one was deterministic about membership. Turn two was back to whatever embedded closest.

Decision

Widening classifySetQuestion's noun list to catch which of these was the obvious fix and the wrong one: that gate sits under isBreadthQuestion specifically so a project-scoped question like which technology did the anti-overbooking project use for locking, which isBreadthQuestion already matches on its own noun list, does not get the enumerate path's there are exactly N, list all N instruction. Widening it to catch a filter would have reintroduced that false global-count injection through a side door. So the fix is a second, separate gate, isFilteredSetFollowUp, that only recognises the referent shape itself: a filter-question cue, which, what, any, how many, do, does, is, are, within a short window of a demonstrative, these, those, them. On its own that gate proves nothing; it only matters combined with re-running the ORIGINAL classifier against the PRIOR turn's question, confirming turn one really was a set answer before turn two is treated as a filter over it. Once both hold, the set is recomputed fresh from the same typed fields ADR-017 reads, never trusted from conversation history, and handed to the model with a machine-checkable tag on every member, a project's path or a technology's normalised key. The model selects a subset against the visitor's semantic criterion, which is real judgement and the part it is good at. Then every line of its answer is checked against the tag set, and any line naming something untagged, ai-patterns among them, is dropped before the visitor ever sees it. The model chooses. It does not get to invent.

Alternatives considered

  • Widen classifySetQuestion's noun list to also match which of these. Rejected: that gate is deliberately narrow so a project-scoped question does not get a false there-are-N-list-all-N instruction, and a filter needs different handling entirely, not a wider trigger for the same one.
  • Trust the model to keep membership straight from the 24 titles already sitting in its own conversation history. Rejected on the evidence: turn one's answer had all 24 titles in it, in plain prose, and the model still reached past them for a document that was never in the list. Prose in a history window is not a membership check.
  • Persist the computed set somewhere so turn two can reuse it instead of recomputing. Rejected: listProjects and its technology/domain equivalents are a filtered Qdrant scroll, cheap enough that persistence would only be a second copy of the set to keep from drifting out of sync with the first, for a cost saving that does not exist.

Consequences

  • The same recompute-tag-validate mechanism covers all three set kinds, projects, technologies, and domains, not only the one the bug report happened to use - the machinery is identical across all three, so there was no reason to special-case projects.
  • A genuine none of them qualify answer is preserved rather than misread as a hallucination and stripped: the validator checks for that sentinel before it starts policing tags, so an honest empty result still reads as one.
  • The agent trace gained its own step kind, filter, distinct from enumerate, so the trace never implies a retrieve() ran or that the model's own sense of membership was trusted unchecked - what got dropped, and why, is a visible guard step, not a silent edit.
  • The gate only looks one turn back, matching the reported shape exactly. A visitor filtering a set established two turns earlier still falls through to ordinary retrieval today, the same fail-open behaviour an unrecognised phrasing has always had here.
Ask what are the different projects Arup was part of, then which one of these involve generative AI, in /ops and watch the filter and guard trace steps
AboutEMpathWritingProductionConnect