Change Impact Analyst — what's in the tree

The plain-English pitch, no code. Why "what breaks if I change this" is a graph question and not a search question — a subway map, not a librarian — and why git history can grade the system for free. Read this one first.

The technical front door: quickstart commands, the stage roadmap (1–3a done, 3b next: blend vector search into the baseline), the stack table (Neo4j AuraDB, Voyage voyage-code-3, LangGraph, Claude), and the current httpx numbers for both the call-graph baseline and the embedding index.

Append-only decision log, D001–D015. The most load-bearing file in the repo for understanding why the code looks the way it does — e.g. why builtin calls are excluded from the resolution-rate denominator (D005), why 45% of the eval set turned out to be unanswerable before it was fixed (D008), why the combined baseline number leaks against its own eval set (D012), why Neo4j moved from local Docker to AuraDB Free (D013), why the vector index embeds source code instead of the originally-planned docstrings (D014), and why the first Stage 3b vector-recall measurement (75.5%) was actually an unbounded-prediction bug, not a result (D015).

Package metadata and dependency groups. Core install is light (typer, rich, neo4j); everything else is gated behind optional extras so Stage 1/2 stay fast to set up: vectors (voyageai — alone drags in langchain-core, tokenizers, huggingface-hub, numpy, and pillow transitively, D014, far more than "one embeddings call" implies), agents (langgraph / anthropic, Stage 4), mcp (Stage 5), observability (opentelemetry-sdk, Stage 6 — genuinely optional, D017: agent.py runs the same without it, just untraced). Also pins [tool.uv.workspace] exclude so a cloned analysis corpus never gets mistaken for a workspace member (D007). Ruff runs in strict mode; the ignore list is annotated per rule rather than blanket-disabled — e.g. FBT001/FBT003 are excused because --open/--no-open boolean flags are the Typer API, and PLC0415 is excused only in cli.py, only for the commands that lazily import an optional extra so every other command still runs without it.

NEO4J_URI (AuraDB Free's neo4j+s:// connection string, D013 — a commented-out local-Docker fallback is included too), NEO4J_PASSWORD, VOYAGE_API_KEY, and ANTHROPIC_API_KEY (Stage 4+ — the LangGraph agent and, via --api-key passthrough, nothing else needs it). .env itself is gitignored; cli.py loads it via python-dotenv on startup.

Stage 6: ruff check + pytest on every push/PR to main, all extras installed so lint/tests cover every stage, not just Stage 1/2. Deliberately does not run labgo baseline or agent-eval against live services — confirmed first that all 46 tests need zero live credentials (grepped the suite for driver/client construction before wiring this up), so a free, fast lint+test gate is the honest scope for CI, not a simulated full eval run that would need secrets and, for the agent, real per-token cost on every push (D017).

Neo4j Community edition, single container. No longer the primary target — Stage 2 points at AuraDB Free by default (D013) — but kept as an offline/ scratch fallback: swap three lines in .env and labgo load --clear works identically against it.

Resolved dependency lockfile, and the usual ignore rules (.venv, __pycache__, .pytest_cache, .env, data/). Nothing project-specific here.

The entry point. Twelve Typer commands. Stage 1, no database or LLM: ingest (repo → code graph), history (repo → raw co-change + eval set), benchmark (repo → pinned, filtered exam), verify (does a repo still match a pinned benchmark?). Stage 2, Neo4j: load (graph.json → Neo4j, --clear to wipe first) and baseline (score an impact prediction against a pinned benchmark — --method calls, the default, is the unchanged Stage 2 path; vectors/hybrid route to hybrid.py, Stage 3b, D015). Stage 3a, +Voyage: embed and search — both import labgo.embed lazily inside the command body (PLC0415 ignored, deliberately) so the other commands keep working without the vectors extra installed; labgo.hybrid has no such dependency, so it's imported normally at the top. Stage 4, +Anthropic/LangGraph: agent (ask one impact question, interactive) and agent-eval (score the agent against a sample of a pinned benchmark — the full 236 cases through a multi-turn LLM loop is real money and time, so the sample size is a disclosed, deterministic choice, not a shortcut hidden from the number). Stage 5, +mcp: mcp serves the same five tools over the Model Context Protocol (stdio) instead of driving them itself — opposite direction of control from agent. All three lazily import their extras (labgo.agent, labgo.mcp_server), same reasoning as embed/search. view is the odd one out — no repo argument, only --data: a ThreadingHTTPServer serves the prebuilt viewer/dist/ bundle, with _view_handler() intercepting /graph.json and /cochange.json to route them to whatever's on disk right now, so the committed bundle never serves a graph baked in at someone else's build time.

Python's own ast module walks every file and builds a call/import graph in two passes: collect every definition, then resolve every call site against it. Each CALLS edge is tagged with a confidence tier (exact / self / local / heuristic) rather than pretending Python's dynamic dispatch is fully resolvable — 27.2% of in-scope calls resolve on httpx, and the module is explicit about why the rest can't be, short of running the program.

The graph schema shared by everything upstream: Node / Edge dataclasses, the NodeKind / EdgeKind / Confidence enums, and ExtractionStats — which deliberately separates "external" calls (builtins/stdlib/third-party) from the resolution-rate denominator, since counting them as unresolved understated the real number by 7 points (D005). Plain dataclasses on purpose, so the extractor needs no database to run or test.

Shells out to git log --no-merges --name-only and turns the output into two things: co-change pairs (files that changed together ≥2 times) and an EvalCase per commit — one seed file, the rest as the "expected" answer. Excludes merges and oversized commits, because pairs grow as N² and five commits out of 620 otherwise supply 30% of all coupling evidence (D010). The parser also survived two subprocess/encoding bugs recorded in D006 — one of which silently returned zero commits. leave_one_out_neighbors() (D012) subtracts one commit's own contribution from the raw pairs before applying min_count — the fix for scoring a case against evidence that includes the very commit being predicted.

Makes a score mean something over time. A benchmark = the corpus pinned at an exact commit SHA + the filtered eval cases + the extraction parameters, written together to benchmarks/<name>/. filter_answerable() drops any case referencing a file no longer in the tree — on httpx that was 58% of raw cases (D008) — and verify_corpus() refuses to score a benchmark against a repo that has since moved past that commit. write_benchmark() also pins the raw, unfiltered co-change pairs to evidence.json (D012) — load_evidence() reads them back so baseline.py can subtract a case's own commit at scoring time, which needs every count, not just the ones that already cleared min_count.

Stage 2's persistence layer. connect() reads NEO4J_* from explicit args or env, and fails loudly — a RuntimeError, not a silent connection to nothing — if no password is set. load_graph() MERGEs nodes and edges into Neo4j in batches, idempotent so a re-ingest can always be re-loaded; every node keeps a generic :Node label alongside its specific kind so one uniqueness constraint (and later, one vector index) covers all three kinds at once. CO_CHANGED is loaded directed but meant to be queried undirected — git history carries no inherent direction between two files that changed together.

Stage 2's actual deliverable: a number, measured before any LLM touches the problem. predict_impact_calls() is pure Cypher — functions the seed file contains, callers reachable within hops traversals of CALLS — structural, so it carries no leakage risk. Measured on httpx: 22.4% recall, 28.8% hit rate, the honest floor. predict_impact() adds Neo4j's global CO_CHANGED edges on top — fine for a live, one-off query, but scoring a benchmark against it leaks: every eval case's own commit is inside its own co-change evidence window (D012). predict_impact_loo() is the fix used for scoring — same call-graph query, but co-change comes from gitlog.leave_one_out_neighbors() with the scored case's own commit subtracted first. Leak-free result: 92.1% recall, 5.9% precision — barely below the leaky 94.9%, because checking predicted-set size found the real problem: it predicts 40.5 of ~60 files per case, the same "returns most of the corpus" failure D015 caught for an early vector measurement. score_baseline() now requires pairs= (from benchmark.load_evidence()) whenever use_cochange=True, and raises rather than silently scoring leaky. D010's follow-up sweep found the actual fix: at --min-count 25 (a matched ~5-file budget — the curve plateaus there, since httpx has no pairs left above count ~24), the combined signal reaches 26.9% recall, 15.7% precision — beats the floor on both axes, and evidence_max_files turned out to barely matter once min_count is tuned properly.

Stage 3a. Embeds each Function/Class node's source code — not the originally-planned docstrings, which only 19.8% of httpx functions have (D014) — read straight off the corpus via the line range the AST extractor already recorded, so nothing needs duplicating into the graph itself. Batches through voyage-code-3 (100 texts/call, comfortably under Voyage's 1,000-text/ 120K-token caps), writes vectors onto Neo4j's generic :Node label, and creates a native vector index (node_embedding, cosine similarity) that simply excludes any node without an embedding rather than erroring. Measured on httpx: 1,229 candidates, 1,229 embedded, 162,884 tokens billed. semantic_search() is the librarian half of the WHY.md distinction, on its own — no graph traversal.

Stage 3b: checks D001's claim with a number instead of an assertion. predict_impact_vector() ranks candidate files by their single best-matching node and caps the result at k files — the first version instead unioned every seed function's own top-k neighbors unfiltered, and scored a shiny 75.5% recall by sweeping in ~30% of httpx's 60-file corpus per case, the vector equivalent of D012's leakage bug (D015). At a budget matched to the call-graph baseline (~4 files/case), vectors alone score 19.3% — call-graph's 22.4% holds up. But predict_impact_hybrid(), a plain set union of both signals, reaches 40.1% recall / 14.8% precision — better than either alone, meaning the two signals miss different things. Imports predict_impact_calls / predict_impact_loo and CaseScore from baseline.py (D012: use_cochange=True here needs pairs= too, same leave-one-out requirement as baseline.score_baseline) and nothing from embed.py, so it has no voyageai dependency: scoring reuses vectors labgo embed already wrote, no live Voyage call needed to rerun it. D015 follow-up: turning cochange back on with D010's matched min_count=25 lifts the hybrid to 43.3% recall / 15.4% precision — real lift, not the leakage D012 caught, though it shrinks as k grows since a wide vector net starts catching the same hits on its own. Ranking the union stays unbuilt on purpose — no measurement yet shows the plain union is the bottleneck.

Stage 4: the first stage where an LLM decides anything — a LangGraph tool-calling loop answering "if I change X, what breaks, which tests must run, and who should review it?" (README's opening question, in full, for the first time). Five tools: call_graph_traverse and co_change_neighbors wrap what baseline.py/graph.py already measure, semantic_search wraps embed.py, and two are new — test_coverage (walks TESTS edges) and likely_reviewer (most frequent historical committer, plain git log, no graph). State carries Anthropic's own message/tool-use JSON unchanged — no langchain_core message type, no create_react_agent: the graph is agent ⇄ tools with a conditional edge, so the routing decision that is this stage's actual point (D001) stays visible instead of hidden inside a prebuilt loop. A finalize node (tools disabled) is the escape hatch at max_turns, guaranteeing termination in a fixed number of turns rather than hoping the model stops calling tools on its own. The system prompt lists every file id in the corpus so predictions are scored on reasoning, not on inventing correctly-spelled paths; parse_impacted_files() drops anything unrecognized rather than guessing, and keeps the drop visible on AgentResult.unrecognized_mentions instead of silently discarding it. Measured on httpx (n=40/236 sample, Haiku 4.5, D016): 30.3% recall / 5.8% precision — does not beat the 43.3% / 15.4% deterministic hybrid+cochange floor. Diagnosed, not just reported: mean predicted size is 13.05 files/case, 57% larger than the deterministic baseline's own matched budget — every deterministic method has an explicit size cap (hops/k/min_count), the agent has none, so it lists everything it gathered rather than committing to a budget. Same category of mistake D012 and D015 each paid for once: a plausible number produced by predicting too much.

Stage 5: the same five tools, opposite direction of control. Where agent.py puts an LLM inside this project deciding which tool to call, mcp_server.py puts this project's tools inside any MCP client's own model — Claude Code, Claude Desktop, anything that speaks the protocol. Owns no logic of its own: build_server() wires MCPServer.tool() decorators around agent.py's tool_call_graph / tool_co_change / tool_semantic_search / tool_test_coverage / tool_likely_reviewer — promoted off their old _tool_* private names specifically for this reuse, two callers sharing one implementation rather than two copies drifting apart. A labgo://files resource (not a tool — static context a client reads once, not an action to invoke repeatedly) exposes every file id, the same grounding agent.py's system prompt gives the LangGraph loop. Connects to Neo4j once at process startup, not per-call like every other CLI command — an MCP server is a long-lived stdio process, not a one-shot invocation. Verified against a real mcp.ClientSession over actual stdio JSON-RPC (handshake, tool listing, tool calls, resource read) — not just calling the server object's methods directly.

Stage 6: real OpenTelemetry spans, console-exported — SimpleSpanProcessor(ConsoleSpanExporter()), not Batch: labgo's commands are short CLI processes, and a batched processor can still be holding spans when one exits. Chose this over Langfuse specifically to avoid one more account/API key that can be unavailable overnight (D017) — swapping in an OTLP exporter later is one line, not a rewrite. span() is a no-op context manager when opentelemetry-sdk (the observability extra) isn't installed — same optional-dependency shape as voyageai (D014) and mcp (Stage 5); every caller runs identically either way, just untraced. agent.py's run_agent() wraps the whole invocation in one root agent.run span so every child agent.llm_call / agent.tool_call span nests under it and shares a trace_id — the first version didn't do this, and checking directly showed every span getting its own trace, a pile of spans rather than a trace of one run. LLM spans carry input_tokens / output_tokens / latency_ms; mcp_server.py's tool spans carry latency_ms only — that model call is the client's, not one this project pays for.

Package markers for labgo/ and labgo/ingest/. No content.

The whole viewer, one component. Two modes over the same force-directed graph (react-force-graph-2d): Explore shows the raw graph with CALLS edges visible by default (CONTAINS is 1,200+ edges and would drown out everything else). Impact is click-driven, not hover-driven — pick a node and a BFS walks the reverse CALLS adjacency (callee → caller, the "what breaks if I change this" direction) out to an adjustable hop depth, colored by hop distance, plus separately-highlighted co-change neighbors from cochange.json — also threshold-adjustable now (a "Co-change ≥" slider next to Hops, min pinned at 2 since that's cochange.json's own baked-in floor, max computed from whatever count actually appears in the loaded data). A hint nudges toward ≥25 below that, citing D010's matched-budget finding directly in the UI — low thresholds visibly pull in most of the corpus, the same thing that made the default unfit for scoring. Camera auto-fits to whatever's currently in frame via zoomToFit, clamped to a min/max zoom so a single isolated node doesn't fill the screen.

TypeScript mirror of the JSON schema ingest/models.py writes (RawGraph / RawGraphNode / RawGraphEdge) plus RawCochangeEdge from gitlog.py's output, and the GraphNode/GraphLink shapes react-force-graph-2d wants once its simulation mutates edges to point at live node objects.

Bootstrap (mounts <App /> in strict mode) and styling. No project-specific logic.

Dev-only: copies data/graph.json and data/cochange.json into public/ so npm run dev's Vite server has something to fetch. Never runs as part of npm run build — the production bundle must not bake in whichever corpus the maintainer last happened to ingest, which is exactly what cli.py's _view_handler exists to avoid at the other end.

The prebuilt bundle labgo view actually serves — checked into the repo so end users need no Node at runtime. Regenerate with cd viewer && npm install && npm run build; only necessary when src/ changes.

Standard Vite + React 19 + TypeScript scaffolding. One runtime dep beyond React: react-force-graph-2d for the canvas. Linted with oxlint rather than ESLint.

Exercises pyast.py and gitlog.py: import/self call resolution, the builtin-exclusion fix from D005, syntax errors counted rather than fatal, and — pointedly — that the git-log parser never silently returns zero commits (the exact failure mode from D006).

Exercises benchmark.py: cases with a dead seed or a dead expected file get dropped, the drop-percentage arithmetic is correct including the 0/0 case, a corpus at the wrong commit raises CorpusMismatch with an actionable fix command rather than silently scoring the wrong world, and (D012) write_benchmark()/load_evidence() round-trip raw co-change pairs unchanged — including counts below min_count, which leave-one-out needs and the old, already-filtered cochange.json couldn't provide.

D012's actual fix, isolated: leave_one_out_neighbors() keeps a neighbor whose count survives other commits' support, drops one whose only support was the excluded commit (the bug this exists to catch), leaves untouched pairs alone, and never subtracts more than the one commit's single contribution even when min_count is raised past it. Pure Python, no live Neo4j.

The pure-Python slice of graph.py: graph.json round-trips through read_graph_json() unchanged, cochange.json is picked up when present, and — pointedly — connect() raises RuntimeError rather than silently connecting to nothing when NEO4J_PASSWORD is unset. No live database for any of these.

The scoring math in isolation, no Neo4j: recall is share-of-expected covered, precision is share-of-prediction-correct, and precision is None (not 0) when nothing was predicted at all — averaging that as zero would punish an empty prediction twice, since it already tanks recall.

The pure-Python slice of embed.py: read_source() extracts the right line range, and returns None rather than raising when line numbers are missing or the file's gone — no live Voyage or Neo4j needed.

The pure-Python slice of hybrid.py: aggregate_scores() averages recall/precision correctly, hit rate counts cases with any recall, empty predictions don't drag the precision mean to zero (same guarantee test_baseline.py checks for CaseScore itself), and zero cases aggregates to zeros rather than a division-by-zero crash. No live Neo4j — the two Cypher-querying functions this module adds (predict_impact_vector, predict_impact_hybrid) aren't unit tested here, matching how baseline.py's own predict_impact isn't either; both were instead checked directly against Aura while diagnosing D015's inflated-prediction bug.

The pure-Python slice of agent.py, no live Anthropic or Neo4j: parse_impacted_files() keeps recognized file ids and drops unrecognized ones without crashing (including a model that ignores the format entirely), and _route_after_agent() — the loop's actual routing logic — continues to tools under the turn cap, forces finalize at it, and ends once the model stops calling tools on its own. A hallucinated tool name dispatches to an error payload, not an exception. The tool-calling loop itself was exercised directly against Aura, same testing philosophy as baseline.predict_impact / hybrid.predict_impact_vector.

Tests tracing.span()'s contract without special-casing whether opentelemetry-sdk is installed — that would defeat the point of it being optional (D017): usable as a context manager either way, yields something attribute-settable or None, accepts no attributes, and — the one that would actually matter in production — never swallows an exception raised inside the traced block.

Output of labgo ingest on httpx: 1,301 nodes, 2,100 edges.

Raw output of labgo history: 1,745 co-change edges and 610 unfiltered eval cases. Explicitly labelled "not yet usable for scoring" until run through benchmark.

The committed, pinned exam: corpus SHA b5addb64, 236 answerable cases surviving from 565 raw (D008), plus the extraction parameters and filter rule that produced them — everything needed to reproduce or invalidate the score later.

Added for D012: the raw, unfiltered co-change pairs (3,348 of them, every count from 1 to 39) pinned alongside the same exam. Deliberately not filtered to min_count — leave-one-out has to subtract a commit's contribution before the threshold is applied, or a pair that only clears it because of the excluded commit would look like it never existed.

Everything above still regenerates from the corpus; this is the one piece of project state that doesn't live in the repo. labgo load writes graph.json's nodes/edges plus CO_CHANGED into a single free Aura instance (D013, switched from local Docker the same day); labgo embed adds an embedding vector property + native vector index on top of the same nodes. One instance, no local/prod split — --clear is destructive against the only copy, though recoverable in one command since data/graph.json stays the source of truth. Free tier auto-pauses after 72h idle and is deleted 30 days after that; expected and accepted for a project touched in bursts.