diff --git a/CHANGELOG.md b/CHANGELOG.md index 8d8ab6f79..552dc2c76 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,47 +4,159 @@ All notable changes to CodeGraph are documented here. Each entry also ships as a [GitHub Release](https://github.com/colbymchenry/codegraph/releases) tagged `vX.Y.Z`, which is where most people will look. +Each release opens with a short **Highlights** list — the handful of things most +users will notice — followed by the full notes. + This project follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ## [Unreleased] + +## [1.6.0] - 2026-08-26 + +### Highlights + +- **GitHub Copilot is now supported** — `codegraph install` sets it up in VS Code, the Copilot CLI, and JetBrains IDEs, next to the agents it already knew. +- **Set up in one command** — `codegraph install --yes --init` wires up your agents and indexes your project with no prompts, ideal for a fresh container or CI. +- **Better answers for your agent** — `codegraph_explore` no longer repeats code it already showed you, always brings back the files and symbols you asked for by name, and spends its space on the code that actually answers the question rather than look-alikes, generated files, and type shims. +- **Your graph stays right as you keep coding** — a long-running index no longer drifts from a fresh one, and edits to `codegraph.json` (such as `exclude`) apply immediately without a restart. +- **No more silent crashes or hangs** — deeply nested C/C++ files, Swift Vapor projects, and large sync batches that used to kill or stall indexing now finish cleanly. +- **A disk-space leak is fixed** — force-killed sessions could leave the database log growing to tens of gigabytes; it's now capped and cleaned up automatically. +- **Works from a workspace or monorepo root** — the MCP server finds your indexed project when launched from a folder above it, and says so clearly when it can't find one. +- **More accurate code graphs** for TypeScript, Rust, Erlang, C/C++, and Python projects — see the full list below. +- **Also new**: per-project Codex setup (`codegraph install --location=local`), a `deprioritize` setting to keep helper folders from outranking your real code, the `codegraph context` command, and usage stats that now stay entirely on CodeGraph's own servers. + +After upgrading, run `codegraph index` once in each project so your existing graph picks up these fixes — `codegraph status` reminds you when it's needed. + ### New Features -- Anonymous usage telemetry is now stored entirely on CodeGraph's own first-party infrastructure — no third-party analytics vendor receives any of it, and the endpoint that receives it makes no outbound requests at all. Individual events are deleted after 90 days, leaving only anonymous daily totals. Nothing about what is collected changed, your IP address is still never read or stored, and every off-switch works exactly as before (`codegraph telemetry off`, `CODEGRAPH_TELEMETRY=0`, `DO_NOT_TRACK=1`). `TELEMETRY.md` remains the complete field-by-field list. +- GitHub Copilot is now a supported agent: `codegraph install` can configure Copilot Chat in VS Code (`copilot-vscode`), the GitHub Copilot CLI (`copilot-cli`), and the Copilot plugin in JetBrains IDEs (`copilot-jetbrains`). Installed Copilot surfaces are auto-detected like every other agent, existing MCP server entries in their config files are preserved, and `codegraph uninstall` reverses the setup cleanly. Restart VS Code or your JetBrains IDE after installing so Copilot picks up the server. + +- `codegraph install --init` wires up your agents and builds the current project's index in one command, and `codegraph init --yes` runs without any prompts — so a fresh container or CI job can bootstrap CodeGraph with a single non-interactive line (`codegraph install --yes --init`). The installer still never indexes anything unless you ask for it with the flag, and the usual safety refusal for a home directory or filesystem root applies. (#1578) + +- Codex CLI can now be set up per project instead of only user-wide: `codegraph install --location=local` writes `./.codex/config.toml` and the CodeGraph block in your project's `AGENTS.md`, so CodeGraph is wired into that repo only rather than every Codex session on the machine. `codegraph uninstall --location=local` reverses it, and the global install is untouched either way. Codex only applies a project's config once you've marked the project trusted, so the installer says so after a local install. Thanks @maxmilian. (#1531) + +- A new `deprioritize` setting in `codegraph.json` keeps the paths you name from outranking your product code in search and `codegraph_explore` answers, without removing anything from the index. It takes gitignore-style patterns just like `exclude`, but is ranking-only: helper-script trees, generated output, or optional add-on directories whose generic symbol names (`usage`, `run`, `status`) would otherwise crowd out the code that actually answers a query stay fully indexed and findable — and a query that genuinely targets such a tree still returns it. Thanks @maxmilian. (#982) - `codegraph_explore` no longer re-sends source it already returned earlier in the same conversation. A file it has already shown you comes back as a short pointer — the path, the symbols and the exact line range, with confirmation that the file hasn't changed since — and the space that frees is spent on code you haven't seen yet, so a follow-up call covers new ground instead of repeating the last one. If a file was edited in between, its source is always shown again in full. Set `CODEGRAPH_EXPLORE_DEDUP=0` to turn this off. - When an agent connects over MCP, CodeGraph now states up front that it indexes 30+ languages — TypeScript/JavaScript, Python, Go, Rust, Java, C#, C/C++, PHP, Ruby, Swift, Kotlin, and more — so agents no longer assume a language isn't supported and skip the graph. (#671) -- GitHub Copilot is now a supported agent: `codegraph install` can configure Copilot Chat in VS Code (`copilot-vscode`), the GitHub Copilot CLI (`copilot-cli`), and the Copilot plugin in JetBrains IDEs (`copilot-jetbrains`). Installed Copilot surfaces are auto-detected like every other agent, existing MCP server entries in their config files are preserved, and `codegraph uninstall` reverses the setup cleanly. Restart VS Code or your JetBrains IDE after installing so Copilot picks up the server. +- Anonymous usage telemetry is now stored entirely on CodeGraph's own first-party infrastructure — no third-party analytics vendor receives any of it, and the endpoint that receives it makes no outbound requests at all. Individual events are deleted after 90 days, leaving only anonymous daily totals. Nothing about what is collected changed, your IP address is still never read or stored, and every off-switch works exactly as before (`codegraph telemetry off`, `CODEGRAPH_TELEMETRY=0`, `DO_NOT_TRACK=1`). `TELEMETRY.md` remains the complete field-by-field list. ### Fixes -- C, C++, Objective-C and Rust unions are now indexed as first-class `union` nodes. A `union` declaration previously produced no symbol at all, so it never appeared in search or `codegraph_explore`, and anything attached to it disappeared with it — in Rust, every `impl SomeTrait for MyUnion` lost its edge, the methods from that impl were left pointing at a type the graph did not contain, and asking which types implement a trait quietly skipped the union ones. A union-shaped dispatch table in C now resolves its function pointers like a struct-shaped one. A `typedef union { … } Name;` in C keeps the typedef's name and is no longer mistaken for a plain type alias. Thanks @ctype-lab. Re-index after upgrading to pick up unions in existing projects. (#1515) +#### Better answers from `codegraph_explore` + +- Naming a file by its path in a `codegraph_explore` query now works reliably: the path is resolved against the index and that file is guaranteed a place at the top of the answer. Previously the path was broken into fragments — bracketed route segments like SvelteKit's `[id]` made this worst — and pieces like `page` or `runs` matched every sibling file, so the file you actually named could be crowded out of the answer entirely. A path that doesn't match any indexed file is now called out instead of silently ignored. + +- Naming a kebab-case file **without its extension** in a `codegraph_explore` query — `background-image-table` rather than `background-image-table.tsx`, the way import paths and prose spell it — now returns that exact file too. Previously the name was split at the hyphens, and in a kebab-cased frontend those pieces (`background`, `image`, `table`) are among the most common words in the codebase, so look-alike sibling files filled the answer while the named file never appeared. Hyphenated words that don't name an indexed file, like "cross-call" or "non-blocking", are left alone. + +- Plainly-worded `codegraph_explore` questions now find camelCase code: a query like "auto-scroll to bottom" can reach a function named `scrollFeedToBottom`, because query words are matched against the words inside identifiers, not just whole names. + +- Variables and constants now count when `codegraph_explore` picks its starting symbols, so state held in plain variables — `$state`-style variables in Svelte, for example — no longer gets overlooked. -- A long-lived index no longer drifts away from what a fresh `codegraph index` would produce. When a file gained or lost a symbol, references to that name in files the sync never touched kept pointing at the definition that was correct before the change, and — because nothing distinguished two same-named definitions — the winner could come down to the order files happened to be written, which differs between a full index and a sync. On this project's own repository, replaying 80 commits through `sync` left 5.7% of connections wrong; it is now 1.3%, and the wrong-answers-still-being-asserted half drops by 99.7%. Since call edges are what flow questions follow and what `codegraph_explore` ranks files by, this quietly degraded answers as an index aged, with nothing to indicate it. Syncing is unchanged in speed, and an edit that only changes a function's body does no extra work at all. Set `CODEGRAPH_NO_REBIND=1` to opt out. - `codegraph_explore` now concentrates its answer on the code that actually answers your question instead of spreading it across files that merely share a word with it, so more of the answer arrives in a single call. Thanks @LeDuyViet for the detailed measurements and reproduction. (#1500) + - Files only weakly related to your question now come back as a name, symbol and line number instead of spending the answer on their source — name one of them in a follow-up `codegraph_explore` to get it back in full. (#1500) + - A generated CRUD or protobuf layer no longer crowds out the hand-written code sitting beside it: generated files are now recognized by the `// Code generated by … DO NOT EDIT.` style banner written at the top of the file, not just by a filename that looks generated. Re-index after upgrading to pick up the new detection. (#1500) + - Test and spec files in a repository's top-level `test/` or `spec/` directory are now recognized as such, so they no longer take room from the code you asked about. (#1500) + - Generated type-declaration files that announce themselves with a "Generated by … by running …" banner — Cloudflare Wrangler's `worker-configuration.d.ts` is the common one — are now recognized as generated. Previously a file like that could take most of a `codegraph_explore` answer on nothing more than a few common words, pushing the hand-written code you asked about out of the response entirely. Re-index after upgrading to pick up the new detection. + - A hand-written type-declaration file — an ambient `.d.ts` of global shims, vendored typings, module augmentation — no longer takes over a `codegraph_explore` answer about how something works. Files like these declare common names (`Body`, `Message`, `ImageMetadata`) and nothing else, so a plainly-worded question could match one strongly enough that it ranked first and crowded the actual handler out of the answer. They are now ranked lower for questions about behaviour, and are still listed by name so one follow-up call fetches them. Asking about a type by name still returns its declaration first, and a shared types module the rest of your code imports is unaffected. -- A CodeGraph process that gets force-killed — by the stuck-process watchdog, a crash, or the OS — no longer leaves the database's write-ahead log behind to grow without bound. Previously each killed session stacked more data onto the same log file and nothing ever shrank it, which on machines where sessions were killed regularly could quietly eat tens of gigabytes of disk. The log is now capped, and any oversized leftover is reclaimed automatically the next time the project is opened. Thanks @tiendungdev for the exceptional Windows report that pinned this down. (#1431) -- The background server's watchdog no longer kills a healthy server that is just waiting on a slow disk: like indexing already does, it now checks whether the database files are still making progress before concluding the process is stuck. Fewer spurious kills also means fewer leftover write-ahead logs. (#1431) -- `codegraph status` now shows the write-ahead log's size next to the database size and warns when killed sessions have left it oversized, and every line in the background server's log now carries a timestamp so kills and restarts can be placed in time. (#1431) -- On Windows, the Claude Code prompt hook written by `codegraph install` failed with "command not found" when hooks run through Git Bash, which needs the `.cmd` extension to find the launcher. The installer now writes the platform-correct command, and re-running `codegraph install` (or `codegraph upgrade`) repairs an existing install in place. (#1466) -- Python classes used as values — `return SomeSerializer` from a factory method, `handler = SomeClass` aliases, registry dicts and lists, and classes passed as arguments — now produce reference edges in the graph. Previously these idioms were invisible, so on Django and Django REST Framework projects, asking for a serializer's callers or the impact of editing it missed the views that actually use it. Re-index after upgrading to pick up the new edges. (#1478) -- When a file changed on disk after its last index sync, `codegraph_node` and `codegraph_explore` could return a different symbol's code under the requested name — current file bytes cut at outdated line positions — while presenting it as verbatim, trustworthy source. This hit hardest on projects queried through `projectPath` (for example, sub-projects of a monorepo), which have no live file watcher to flag pending edits. Both tools now verify each file against the index before showing sliced code: an out-of-date file is either shown whole with its full current source, or its code is withheld with a clear "changed on disk" notice — never served as a wrong slice. A fresh re-index restores normal output automatically. Thanks @inth3shadows for the thorough report and verification passes. (#1474) + +- When you name a symbol in a `codegraph_explore` query, its definition now actually comes back. Two cases previously lost it. If the symbols you named don't call one another — sibling functions inside the same factory or module are the everyday example — CodeGraph stopped treating them as symbols you had asked for, and answered with whatever sat at the top of their file instead; on one 1,400-line file that meant a same-stem `QueuedMessage` interface on line 70 came back while the `queueMessage` function on line 1087 did not. And when an answer had to be trimmed to fit, it was trimmed from the bottom of the file down, so a symbol near the end of a long file was always the first thing cut. Trimming now protects the definitions you named wherever they sit in the file. + - A file built around one very long function no longer takes the whole `codegraph_explore` answer for itself — or disappears from it. Previously such a file was shown in full however big it was, which used up the room every file after it needed, and when the function was larger than the entire response the file was dropped without a word. These files now come back as a bounded window on whole lines — the signature and the top of the body, plus the call site when the call path runs through it — with the rest one follow-up `codegraph_explore` away. + - `codegraph_explore` no longer lets the first file in an answer spend the room set aside for the files below it, so the rest of the answer still arrives. Previously a large file near the top could quietly use up everything left, and the files ranked under it — each already judged relevant enough to include — were dropped with no source at all; on one question only one of six made it into the answer. Every file now keeps what it was given, and a question that really is about one file still concentrates on that file. + - When a `codegraph_explore` answer runs right up against its size limit, it now drops the trailing notes rather than a whole file's source. Previously the last file was cut even though trimming the notes alone would have fit, so a file that had already been read, ranked and rendered was thrown away at the last moment. Across a range of real projects this returns one more file and up to 20% more source per call. + - Every file `codegraph_explore` decides to include now actually arrives. A file shown in full could still spend room set aside for files below it — the fix above covered files shown as excerpts but not files shown whole — and the answer's own size bookkeeping under-counted each file's heading, so the answer ran past its limit and a fully prepared file was discarded at the end. A file that no longer fits whole is now shown as excerpts instead of vanishing, and one that overshoots by a little is trimmed to fit rather than dropped. + - The list of files an answer could not cover — the "explore these names for their source" pointers — is no longer thrown away when the answer is full. It is now budgeted for and trimmed to fit, so a full answer still tells you what it left out and which names to ask for next, instead of ending with no pointers at all. + - When a `codegraph_explore` answer shows a file as excerpts, a large excerpt that no longer fit was dropped entirely instead of being shortened. If the file's first excerpt happened to be a trivial one — an import block, a one-line helper next to the code you asked about — the excerpt carrying the actual answer was the one thrown away, and the file came back with a quarter of the room it had been given. On real projects that meant the top-ranked file delivered a fraction of its share while a far less relevant file took the rest. Excerpts are now shortened to fit, whole method by whole method, and only dropped when what is left is too small to hold anything readable. -- When you name a symbol in a `codegraph_explore` query, its definition now actually comes back. Two cases previously lost it. If the symbols you named don't call one another — sibling functions inside the same factory or module are the everyday example — CodeGraph stopped treating them as symbols you had asked for, and answered with whatever sat at the top of their file instead; on one 1,400-line file that meant a same-stem `QueuedMessage` interface on line 70 came back while the `queueMessage` function on line 1087 did not. And when an answer had to be trimmed to fit, it was trimmed from the bottom of the file down, so a symbol near the end of a long file was always the first thing cut. Trimming now protects the definitions you named wherever they sit in the file. + - The blast-radius section of `codegraph_explore` flagged "no covering tests found" whenever no test called a symbol directly — falsely branding helpers that tests exercise through their callers as untested (about 40% of flagged symbols in a measured sample). The check now follows caller chains up to 3 hops and reports indirect coverage as "tested via callers"; when nothing is found it states exactly what was checked instead of an unconditional warning. Thanks @inth3shadows for measuring the false-positive rate. (#1475) +- When a file changed on disk after its last index sync, `codegraph_node` and `codegraph_explore` could return a different symbol's code under the requested name — current file bytes cut at outdated line positions — while presenting it as verbatim, trustworthy source. This hit hardest on projects queried through `projectPath` (for example, sub-projects of a monorepo), which have no live file watcher to flag pending edits. Both tools now verify each file against the index before showing sliced code: an out-of-date file is either shown whole with its full current source, or its code is withheld with a clear "changed on disk" notice — never served as a wrong slice. A fresh re-index restores normal output automatically. Thanks @inth3shadows for the thorough report and verification passes. (#1474) + +#### Finding your project, live updates, and the CLI + +- The MCP server now finds your project when it's launched from a workspace folder above it: if the launch directory has no index of its own but exactly one indexed project sits below it (a repo container, an agent workspace, a monorepo root), that project becomes the session's default — live file watching and the shared daemon included — instead of every tool call failing until a `projectPath` or `--path` is supplied. Thanks @nakisen. (#1606) + +- When no project can be resolved at all, the MCP server now says so instead of starting silently: a startup log line names the directory it searched, and tool calls list the indexed sub-projects it can see nearby so you can pass one as `projectPath`. Previously the server looked healthy from the outside while every tool quietly had no project to answer from. Thanks @nakisen. (#1607) + +- Editing `codegraph.json`'s `exclude` or `include` (or a `.gitignore`) while the MCP server is running now takes effect immediately. Previously the running file watcher kept the scope it had when it started, so a newly excluded file was removed by `codegraph sync` and then quietly re-added by the watcher seconds later — which looked like `exclude` not working at all — until the server was restarted. A scope change now refreshes the watcher and triggers a full reconcile, and a changed file the watcher hands to sync is re-checked against the current scope first, so the CLI and the live server can no longer disagree about what belongs in the index. Thanks @K1nG11. (#1590) + +- `codegraph status` now sees new files inside brand-new directories. Git reports an entirely-untracked directory as a single collapsed entry, so source files created there — a freshly scaffolded `frontend/`, for example — were missing from the pending-changes report, which could claim everything was up to date while those files had not yet been indexed. Thanks @maxmilian. (#1213) + +- The `codegraph context ` command documented in the CLI help now actually exists — it builds a ready-to-inject context pack for a task (relevant symbols, their relationships, and code) in markdown or JSON, restoring the contract external integrations like Memorix rely on (`--path`, `--format json`, `--max-nodes`, `--no-code`). (#1611) + +- On Windows, the Claude Code prompt hook written by `codegraph install` failed with "command not found" when hooks run through Git Bash, which needs the `.cmd` extension to find the launcher. The installer now writes the platform-correct command, and re-running `codegraph install` (or `codegraph upgrade`) repairs an existing install in place. (#1466) + +- Looking a symbol up by name no longer reads the whole graph. Every search made one full pass over all indexed symbols for each word you typed, and a question that named several symbols made two more passes per name — including for a word that matches nothing, which is the common case. The cost therefore grew with the size of the project, and it was paid again on every message when the prompt hook is enabled. These lookups now go through the name index instead. Results are identical; only the time to get them changes, and it no longer grows with the project. Thanks @maxmilian. + +#### Indexing reliability and disk usage + +- Indexing no longer crashes the whole process — a segmentation fault with no message and no partial index — on a C/C++ (or any other) file with extremely deep nesting, such as the parser stress-test fixtures in the clang and gcc test suites or a fuzzer corpus. Such a file is now handed to the fallback parser and recorded with a parse warning while the rest of the repository indexes normally. Thanks @apollo600 for the exact diagnosis. (#1581) + +- Indexing no longer hangs on a Swift Vapor project containing a call with a long argument list. A single `.get(...)`-style call with many labeled arguments and no `use:` handler — the shape generated request builders produce — could stall `codegraph index`, `codegraph sync`, and the MCP server indefinitely. Route detection now handles such files in milliseconds, and every previously-recognized route shape still parses exactly as before. Thanks @maxmilian. (#1544) (Swift) + +- Syncing a large batch of changed files no longer crashes with "Maximum call stack size exceeded" partway through. The crash aborted reference resolution after the files' symbols were already stored, leaving the graph with far fewer connections than a fresh index would have — and it hit exactly the scenarios that re-parse many files at once, including the automatic repair above. Thanks @netbrah for pinpointing the failure. (#1558) + +- A long-lived index no longer drifts away from what a fresh `codegraph index` would produce. When a file gained or lost a symbol, references to that name in files the sync never touched kept pointing at the definition that was correct before the change, and — because nothing distinguished two same-named definitions — the winner could come down to the order files happened to be written, which differs between a full index and a sync. On this project's own repository, replaying 80 commits through `sync` left 5.7% of connections wrong; it is now 1.3%, and the wrong-answers-still-being-asserted half drops by 99.7%. Since call edges are what flow questions follow and what `codegraph_explore` ranks files by, this quietly degraded answers as an index aged, with nothing to indicate it. Syncing is unchanged in speed, and an edit that only changes a function's body does no extra work at all. Set `CODEGRAPH_NO_REBIND=1` to opt out. + +- Fixed a v1.5.0 regression where a perfectly valid file could be permanently recorded as having 0 symbols, with no error reported. When a file's first parse attempt was interrupted — a parsing worker crash or timeout, most likely on slow or heavily loaded machines — the automatic retry stored an empty result for any language on the native extraction path, so the file's functions and classes silently vanished from search, callers, and impact until the file was next edited. Retries now store the file's real symbols, and a file already recorded as symbol-free is detected and repaired automatically by the next sync or re-index after upgrading. Thanks @Baiae for the report. (#1541) + +- A CodeGraph process that gets force-killed — by the stuck-process watchdog, a crash, or the OS — no longer leaves the database's write-ahead log behind to grow without bound. Previously each killed session stacked more data onto the same log file and nothing ever shrank it, which on machines where sessions were killed regularly could quietly eat tens of gigabytes of disk. The log is now capped, and any oversized leftover is reclaimed automatically the next time the project is opened. Thanks @tiendungdev for the exceptional Windows report that pinned this down. (#1431) + +- The background server's watchdog no longer kills a healthy server that is just waiting on a slow disk: like indexing already does, it now checks whether the database files are still making progress before concluding the process is stuck. Fewer spurious kills also means fewer leftover write-ahead logs. (#1431) + +- `codegraph status` now shows the write-ahead log's size next to the database size and warns when killed sessions have left it oversized, and every line in the background server's log now carries a timestamp so kills and restarts can be placed in time. (#1431) + +- A background daemon left behind by an out-of-memory kill or force-kill can no longer block every future session when the operating system reuses its process ID: daemon management now verifies a recorded process is really a CodeGraph daemon before trusting or signaling it, and `codegraph unlock` clears stale daemon artifacts as well as the indexing lock. Thanks @hcg1023 for the report and @danusha2345 for the fix. (#1553) + +- When indexing has to fall back to parsing a file with its comment lines stripped — a last-resort recovery after repeated parser crashes — the file is now flagged with a visible warning instead of being reported as cleanly indexed. The recovered result can be incomplete, and reporting success made a fresh index quietly disagree with a later re-parse of the same unchanged file. Thanks @jeremypetz for the precise init-versus-sync symbol accounting that exposed this. (#1565) + +- Data-only C/C++ headers near the file-size limit no longer hold a parser worker for several minutes before timing out; the default large-file parse budget is now bounded, while an explicitly configured larger timeout is still honored. (#1555) + +- Reopening an index after a crash during bulk loading now restores every dropped database index, and a successful recovery sync marks the index complete instead of leaving it permanently flagged as interrupted. (#1556) + +- Files skipped because they are too large or repeatedly fail to parse are now recorded with the reason, so unchanged rejected files are no longer rediscovered and retried on every status check and sync — and a later successful parse of such a file replaces the record with its real symbols. Thanks @netbrah for the exceptional failure analysis behind this batch, and @danusha2345 for the fixes. (#1557) + +- C/C++ function-pointer analysis now bounds its compiled-pattern caches, so very large repositories can no longer exhaust the JavaScript engine's regular-expression code space during indexing. (#1559) + +- JSX rendering analysis now runs only on JavaScript-family files, so JSX-looking strings in C/C++ (or any other language) no longer create impossible call edges — in pure-C projects and in mixed-language monorepos alike. (#1560) + +#### Language and framework accuracy + +- Calls to the methods of an exported object-literal constant — `export const api = { call() { … } }` used as a module's namespace, a common way to organize a TypeScript API surface — now resolve to the method, both in the defining file and through imports. Previously such a call linked to nothing (or to the constant itself), so `codegraph callers` and impact analysis reported zero callers for methods that are called from everywhere. Re-index after upgrading to pick up the edges. Thanks @IAliceBobI for the precise report and root-cause. (#1573) + +- Import aliases defined in a shared TypeScript config are now picked up. Nx-style monorepos keep every `@scope/...` alias in a `tsconfig.base.json` that the root `tsconfig.json` only inherits through `extends`, so CodeGraph found no aliases at all and every cross-package import fell back to matching on name alone — which quietly attaches results to unrelated symbols that happen to share a name, exactly where a monorepo needs `codegraph_impact` and `codegraph_callers` to be right. Chains several configs deep, a config inherited from a package in `node_modules`, and a `baseUrl` declared in an inherited config are all followed now, and a `tsconfig.base.json` is read directly when the root `tsconfig.json` is only a project-references shell or isn't there at all. Re-index after upgrading. Thanks @maxmilian. (#1534) + +- Methods implemented in a generic or lifetime-parameterized `impl` block (`impl Source for BufSource`, `impl<'a> Iterator for Parents<'a>`) are now recorded under the implementing type instead of the trait. Previously such a method could not be found by its type — "who calls `BufSource::read`" had no answer — and it collided with the trait's own declaration, which could even invent a call-graph edge out of an impl body that contains no call at all. Impls on a reference (`impl Trait for &Foo`) and on a module-qualified type (`impl Trait for m::Foo`) are attributed to their type too. Re-index after upgrading. Thanks @Dshuishui. (#1588) (Rust) + +- A method call on a struct field — `self.inner.run()` with `inner: Inner` — now resolves to the method on the field's declared type. Previously the call was reduced to the bare method name and matched whichever same-named method was nearest, which was often the calling method itself, recording recursion that isn't in the source (a few hundred such self-edges in ripgrep alone), or a method of an unrelated type. References and `Box`/`Rc`/`Arc` fields are looked through, as Rust's own method calls are; a field whose type is external (a std or third-party type), a generic parameter, or a container like `Option`/`Vec` is left unresolved rather than guessed. Re-index after upgrading. Thanks @Dshuishui. (#1585) (Rust) + +- Erlang functions that share a name but differ in arity are now separate symbols with the language's own `module:fun/arity` identity, so the everyday `f/1` delegating to `f/2` shows as a real call edge instead of a self-loop, each arity keeps its own `-spec` and source span, `-export([f/1])` marks exactly that arity as public, and asking `codegraph_explore` for a symbol the way Erlang spells it — `cowboy_req:header/3` — returns that definition. Re-index Erlang projects after upgrading. Thanks @Dshuishui. (#1610) (Erlang) + +- Erlang behaviour dispatch no longer miscounts a call site's arity when an argument is a binary literal like `<<1,2,3>>` — the commas inside were counted as argument separators, which silently dropped (or could mislink) the dispatch edge to the behaviour callback. (#1358) (Erlang) + +- A C++ `.h` header whose only C++ construct is a plain derived type — `struct Derived : Base` with no export macro, `class` keyword, or access section — is now recognized as C++ (previously only the export-macro form was). Such a header was read as C, so the derived struct vanished from the index and a phantom function named after the base type appeared in its place. The check now also covers the whole file rather than its first few kilobytes, so a long C-compatible preamble no longer hides the signal. Re-index after upgrading to pick up affected headers. Thanks @Jaysenpeng. (#1592) + +- C, C++, Objective-C and Rust unions are now indexed as first-class `union` nodes. A `union` declaration previously produced no symbol at all, so it never appeared in search or `codegraph_explore`, and anything attached to it disappeared with it — in Rust, every `impl SomeTrait for MyUnion` lost its edge, the methods from that impl were left pointing at a type the graph did not contain, and asking which types implement a trait quietly skipped the union ones. A union-shaped dispatch table in C now resolves its function pointers like a struct-shaped one. A `typedef union { … } Name;` in C keeps the typedef's name and is no longer mistaken for a plain type alias. Thanks @ctype-lab. Re-index after upgrading to pick up unions in existing projects. (#1515) + +- Python classes used as values — `return SomeSerializer` from a factory method, `handler = SomeClass` aliases, registry dicts and lists, and classes passed as arguments — now produce reference edges in the graph. Previously these idioms were invisible, so on Django and Django REST Framework projects, asking for a serializer's callers or the impact of editing it missed the views that actually use it. Re-index after upgrading to pick up the new edges. (#1478) + +- SAP HANA `.xsjs` / `.xsjslib` imports now resolve across files: an extensionless `import { x } from './helpers'` in a `.xsjs` file finds `helpers.xsjslib`, so the cross-file call edge is created and `codegraph_callers` / `codegraph_impact` see it. Previously the import path resolved to nothing and the call fell back to same-name matching, which could bind the edge to an unrelated file that happened to export the same symbol. Complements the `.xsjs` / `.xsjslib` extraction support. Thanks @maxmilian. (#556) + ## [1.5.0] - 2026-07-21 # ⚡ The Rust engine release — with near-instant sync @@ -747,3 +859,4 @@ Thanks @andreinknv for the substantive draft this release was based on. [1.4.0]: https://github.com/colbymchenry/codegraph/releases/tag/v1.4.0 [1.4.1]: https://github.com/colbymchenry/codegraph/releases/tag/v1.4.1 [1.5.0]: https://github.com/colbymchenry/codegraph/releases/tag/v1.5.0 +[1.6.0]: https://github.com/colbymchenry/codegraph/releases/tag/v1.6.0 diff --git a/CLAUDE.md b/CLAUDE.md index 4f24f1c0d..0b2434919 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -217,6 +217,7 @@ Formatting rules for any entry (anywhere — `[Unreleased]` or otherwise): 3. **Strip the internals.** No internal file paths (`src/...`), no internal symbol / function / class names, no benchmark numbers / percentages / node-or-edge counts. **Keep:** language & framework names (Go, Spring, NestJS, …), things a user types or sets (`codegraph install`, `codegraph_explore`, the `CODEGRAPH_*` env vars), agent / IDE names (Claude Code, Cursor, opencode, Kiro, …), and a brief `Thanks @user` when a contributor is credited. 4. Issue / PR references in entries are by number (`(#403)` etc.); the GitHub renderer auto-links them in the published release notes. 5. **Don't add a `[X.Y.Z]: https://...` link reference yourself** — `prepare-release.mjs` appends it automatically when it promotes the version (idempotent: a re-run is a no-op if it already exists). +6. **Every release opens with a `### Highlights` block — the only part most people read.** At most ~8 one-line bullets, in plain language for someone who doesn't read code, ordered by what a typical user notices first (new agent/IDE support and setup changes, then answer quality, then reliability), plus a one-sentence upgrade note when a re-index is needed. Write or refresh it in `[Unreleased]` when a release is being prepared — not per PR — and keep the detailed `### New Features` / `### Fixes` entries below it. When `### Fixes` grows past ~15 entries, group them under `####` sub-headings (`Better answers from codegraph_explore`, `Finding your project, live updates, and the CLI`, `Indexing reliability and disk usage`, `Language and framework accuracy`) so a skimmer can find their area. Multi-word headings like `### New Features` are safe on the normal release path: `prepare-release.mjs` **Case A** moves the whole `[Unreleased]` body verbatim into `[X.Y.Z]`. (Only its rarely-used **Case B** *merge* splits sub-sections with a single-word `^### (\w+)$` regex that wouldn't match them — and Case B fires only if a `[X.Y.Z]` block was pre-created, which rule above already forbids.) diff --git a/FORK.md b/FORK.md index 6d9fee95c..0128e9dc6 100644 --- a/FORK.md +++ b/FORK.md @@ -3,13 +3,15 @@ FM-Agent's maintenance fork of [colbymchenry/codegraph](https://github.com/colbymchenry/codegraph). -**Pinned base:** upstream `main` at `c6aaa20358cd6adcd04b87bdef8e5803ad146f3a` -(2026-08-07). That is *past* `v1.5.0` and before whatever upstream tags next: -`v1.5.0` was 104 commits behind, and the incremental-sync convergence work, the -first-class `union` nodes and the WAL growth fix in between all matter to how -FM-Agent reads the graph. The version marker keeps saying `1.5.0-fmagent.N` -because it is the last upstream *release* this descends from — the exact base is -this commit. +**Pinned base:** upstream `v1.6.0` (2026-08-26) — a tagged release, as the policy +below prefers. It carries fixes for the three issues reported upstream from this +project since the previous base: Rust field-receiver resolution +([#1585](https://github.com/colbymchenry/codegraph/issues/1585)), generic `impl` +ownership ([#1588](https://github.com/colbymchenry/codegraph/issues/1588)) and +Erlang per-arity identity +([#1610](https://github.com/colbymchenry/codegraph/issues/1610)). The previous +base was `c6aaa20` (upstream `main`, 2026-08-07), 27 commits behind this tag; see +issue #10 for the full rationale of this sync. Upstream shipped the C macro-attribute extraction fix (issue #1211, PR #1311) in v1.5.0, so the base carries it natively; the fork no longer needs its own patch @@ -31,9 +33,9 @@ tree or changing how the runner discovers it: our file would stay where it is, q stop being collected, and nothing would fail. Check the suite's file count after a sync, not just that it is green. -**Version marker:** `codegraph --version` → `1.5.0-fmagent.N` identifies a build -from this fork. Note this is a SemVer pre-release of `1.5.0`, so it sorts *below* -plain `1.5.0`; the updater must therefore point at this fork (see below), never +**Version marker:** `codegraph --version` → `1.6.0-fmagent.N` identifies a build +from this fork. Note this is a SemVer pre-release of `1.6.0`, so it sorts *below* +plain `1.6.0`; the updater must therefore point at this fork (see below), never upstream, or it would advertise a "downgrade to upstream" as an upgrade. **All install/upgrade entry points point at this fork,** so a fork install never diff --git a/README.md b/README.md index f8bb60bbc..7f1ed7316 100644 --- a/README.md +++ b/README.md @@ -388,6 +388,7 @@ The installer **wires up your agents only — it does not index your code.** Aft ```bash codegraph install --yes # auto-detect agents, install global +codegraph install --yes --init # same, then build the current project's index (one-shot bootstrap) codegraph install --target=cursor,claude --yes # explicit target list codegraph install --target=auto --location=local # detected agents, project-local codegraph install --target=copilot-vscode,copilot-cli,copilot-jetbrains --yes # GitHub Copilot everywhere @@ -400,6 +401,7 @@ codegraph install --print-config copilot-vscode # same, for Copilot in VS C | `--target` | `auto`, `all`, `none`, or csv (`claude,cursor,...`) | prompt | | `--location` | `global`, `local` | prompt | | `--yes` | (boolean) | prompt every step | +| `--init` | (boolean) run `codegraph init` in the current directory after wiring agents | — | | `--no-permissions` | (boolean) skip Claude auto-allow list | permissions on | | `--print-config ` | dump snippet for one agent and exit | — | @@ -414,7 +416,7 @@ cd your-project codegraph init ``` -Builds the per-project knowledge graph index, which then auto-syncs on every file change. A single global `codegraph install` works in every project you open — no need to re-run the installer per project. +Builds the per-project knowledge graph index, which then auto-syncs on every file change. A single global `codegraph install` works in every project you open — no need to re-run the installer per project. Add `--yes` to skip every prompt (scripts / CI / container bootstraps). That's it — your agent will use CodeGraph tools automatically when a `.codegraph/` directory exists. @@ -669,6 +671,26 @@ CodeGraph discovers those files off disk, overriding `.gitignore`, on index, sync, and watch. An explicit `exclude` still wins, and built-in skips (`node_modules`, `dist`, `.git`) are never re-included. +Sometimes a directory shouldn't leave the index — you still want to find things +in it — it just shouldn't *outrank* your real code. A `scripts/` or +`optional-skills/` tree whose helpers use generic names (`usage`, `status`, +`run`) can win on an exact name match and crowd out the product code that +actually answers the query. Name those trees under `deprioritize`: + +```json +{ + "deprioritize": ["optional-skills/", "scripts/"] +} +``` + +This is the ranking counterpart to `exclude`: those paths stay indexed and +findable — searching for them directly still works — they just stop winning +against first-party code. It applies to `query` / `search` and to `explore`'s +ranking. It is *not* a filter: unlike the built-in `example/`, `sample/`, +`fixture/`, `benchmark/` and `demo/` handling — which also drops those files +from some result sets outright — `deprioritize` only ever changes rank. Reach +for `exclude` when you want something gone. + ### Custom file extensions If your project uses a non-standard extension for a [supported diff --git a/__tests__/cli-context-command.test.ts b/__tests__/cli-context-command.test.ts new file mode 100644 index 000000000..fddc56cf0 --- /dev/null +++ b/__tests__/cli-context-command.test.ts @@ -0,0 +1,114 @@ +/** + * `codegraph context` CLI command (#1611). + * + * The usage header has advertised `codegraph context Build context for + * a task` since the first release, and the ContextBuilder behind the public + * `buildContext` API has always shipped in the package — but the command was + * never registered with commander, so external integrations built against the + * documented contract (`codegraph context --path --format json + * --max-nodes 8 --no-code `, e.g. Memorix) got `unknown command + * 'context'` and fell back to their own heuristics. + * + * Exercised end-to-end against the built binary, mirroring + * cli-query-command.test.ts. + */ + +import { describe, it, expect, beforeEach, afterEach } from 'vitest'; +import { execFileSync } from 'child_process'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; + +const BIN = path.resolve(__dirname, '../dist/bin/codegraph.js'); + +const ENV = { ...process.env, CODEGRAPH_NO_DAEMON: '1', CODEGRAPH_WASM_RELAUNCHED: '1' }; + +function runContext(cwd: string, extraArgs: string[], taskParts: string[] = ['parseToken', 'expiry', 'handling']): string { + return execFileSync(process.execPath, [BIN, 'context', ...extraArgs, '-p', cwd, ...taskParts], { + encoding: 'utf-8', + env: ENV, + stdio: ['ignore', 'pipe', 'ignore'], // drop stderr (SQLite experimental warning) + }); +} + +describe('codegraph context — registered CLI command (#1611)', () => { + let tempDir: string; + + beforeEach(async () => { + tempDir = fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-context-cmd-')); + fs.mkdirSync(path.join(tempDir, 'src')); + fs.writeFileSync( + path.join(tempDir, 'src/auth.ts'), + 'export function parseToken(t: string){ return parseTokenExpiry(t) + t.trim().length; }\n' + + 'export function parseTokenExpiry(t: string){ return Date.parse(t); }\n', + ); + const cg = CodeGraph.initSync(tempDir); + await cg.indexAll(); + cg.close(); + }); + + afterEach(() => { + fs.rmSync(tempDir, { recursive: true, force: true }); + }); + + it('--format json emits clean machine-parseable JSON on stdout', () => { + const parsed = JSON.parse(runContext(tempDir, ['--format', 'json'])); + expect(parsed.query).toBe('parseToken expiry handling'); + expect(Array.isArray(parsed.nodes)).toBe(true); + expect(parsed.nodes.length).toBeGreaterThan(0); + expect(Array.isArray(parsed.codeBlocks)).toBe(true); + expect(parsed.codeBlocks.length).toBeGreaterThan(0); + }); + + it('--max-nodes bounds the returned symbol set', () => { + const parsed = JSON.parse(runContext(tempDir, ['--format', 'json', '--max-nodes', '1'])); + expect(parsed.nodes.length).toBeLessThanOrEqual(1); + }); + + it('--no-code omits code blocks (the Memorix contract shape)', () => { + // The exact documented invocation: --format json --max-nodes 8 --no-code + const parsed = JSON.parse( + runContext(tempDir, ['--format', 'json', '--max-nodes', '8', '--no-code']), + ); + expect(parsed.codeBlocks).toEqual([]); + expect(parsed.nodes.length).toBeGreaterThan(0); + }); + + it('defaults to markdown output', () => { + const out = runContext(tempDir, []); + expect(out).toContain('## Code Context'); + expect(out).toContain('**Query:** parseToken expiry handling'); + }); + + it('fails cleanly on an uninitialized project', () => { + const empty = fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-context-empty-')); + try { + execFileSync(process.execPath, [BIN, 'context', '-p', empty, 'some', 'task'], { + encoding: 'utf-8', + env: ENV, + stdio: ['ignore', 'pipe', 'pipe'], + }); + throw new Error('expected non-zero exit'); + } catch (err: any) { + expect(err.status).toBe(1); + expect(String(err.stderr)).toContain('not initialized'); + } finally { + fs.rmSync(empty, { recursive: true, force: true }); + } + }); + + it('rejects an unknown --format value', () => { + try { + execFileSync(process.execPath, [BIN, 'context', '--format', 'yaml', '-p', tempDir, 'task'], { + encoding: 'utf-8', + env: ENV, + stdio: ['ignore', 'pipe', 'pipe'], + }); + throw new Error('expected non-zero exit'); + } catch (err: any) { + expect(err.status).toBe(1); + expect(String(err.stderr)).toContain('Unknown format'); + } + }); +}); diff --git a/__tests__/cli-install-init.test.ts b/__tests__/cli-install-init.test.ts new file mode 100644 index 000000000..f647e884b --- /dev/null +++ b/__tests__/cli-install-init.test.ts @@ -0,0 +1,108 @@ +/** + * `codegraph install --init` and `codegraph init --yes` (#1578): the one-shot, + * non-interactive "wire agents + build this project's index" bootstrap a fresh + * container / CI job needs. + * + * Exercised end-to-end against the built binary so the CLI wiring (the shared + * `runInit` flow, the flag plumbing, exit codes) is what's covered. Every run + * uses `--target none`, so the installer touches no agent config on the + * machine running the suite; the only side effect is the temp project's + * `.codegraph/`. + */ + +import { describe, it, expect, beforeEach, afterEach } from 'vitest'; +import { execFileSync } from 'child_process'; +import * as fs from 'fs'; +import * as path from 'path'; +import * as os from 'os'; + +const BIN = path.resolve(__dirname, '../dist/bin/codegraph.js'); + +interface RunResult { + status: number; + stdout: string; + stderr: string; +} + +/** Run the CLI with stdin closed — a prompt that blocks would hang / fail here. */ +function runCodegraph(args: string[], cwd: string): RunResult { + try { + const stdout = execFileSync(process.execPath, [BIN, ...args], { + cwd, + encoding: 'utf-8', + env: { + ...process.env, + CODEGRAPH_NO_DAEMON: '1', + CODEGRAPH_TELEMETRY: '0', + DO_NOT_TRACK: '1', + NO_COLOR: '1', + }, + stdio: ['ignore', 'pipe', 'pipe'], + timeout: 120_000, + }); + return { status: 0, stdout, stderr: '' }; + } catch (err) { + const e = err as { status?: number | null; stdout?: string | Buffer; stderr?: string | Buffer }; + return { + status: e.status ?? -1, + stdout: String(e.stdout ?? ''), + stderr: String(e.stderr ?? ''), + }; + } +} + +describe('codegraph install --init / init --yes (#1578)', () => { + let tempDir: string; + + beforeEach(() => { + tempDir = fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-install-init-')); + fs.writeFileSync( + path.join(tempDir, 'a.ts'), + `export function greet(name: string) { return hello(name); }\n` + + `export function hello(n: string) { return 'hi ' + n; }\n`, + ); + }); + + afterEach(() => { + fs.rmSync(tempDir, { recursive: true, force: true }); + }); + + it('install --yes --target none --init builds the current project\'s index in one command', () => { + const r = runCodegraph(['install', '--yes', '--target', 'none', '--init'], tempDir); + expect(r.status, r.stdout + r.stderr).toBe(0); + // The installer ran (and had nothing to wire) … + expect(r.stdout).toContain('No agent targets selected'); + // … and the init ran afterwards, in cwd. + expect(r.stdout).toContain(`Initialized in ${fs.realpathSync(tempDir)}`); + expect(fs.existsSync(path.join(tempDir, '.codegraph', 'codegraph.db'))).toBe(true); + }); + + it('install --init on an already-initialized project reports that and still exits 0', () => { + expect(runCodegraph(['init', '--yes'], tempDir).status).toBe(0); + const r = runCodegraph(['install', '--yes', '--target', 'none', '--init'], tempDir); + expect(r.status, r.stdout + r.stderr).toBe(0); + expect(r.stdout).toContain('Already initialized'); + }); + + it('install --init refuses an unsafe root (filesystem root) with exit code 1, like init does', () => { + // `/` (or the drive root on Windows) is the canonical unsafe root: the + // refusal fires before anything is created, so nothing is written there. + const root = path.parse(process.cwd()).root; + const r = runCodegraph(['install', '--yes', '--target', 'none', '--init'], root); + expect(r.status).toBe(1); + expect(r.stdout).toContain('Refusing to initialize'); + expect(fs.existsSync(path.join(root, '.codegraph'))).toBe(false); + }); + + it('init --yes runs non-interactively with stdin closed and builds the index', () => { + const r = runCodegraph(['init', '--yes'], tempDir); + expect(r.status, r.stdout + r.stderr).toBe(0); + expect(r.stdout).toContain('Initialized in'); + expect(fs.existsSync(path.join(tempDir, '.codegraph', 'codegraph.db'))).toBe(true); + }); + + it('documents the new flags in --help', () => { + expect(runCodegraph(['init', '--help'], tempDir).stdout).toMatch(/-y, --yes\b/); + expect(runCodegraph(['install', '--help'], tempDir).stdout).toMatch(/-i, --init\b/); + }); +}); diff --git a/__tests__/cli-unlock.test.ts b/__tests__/cli-unlock.test.ts new file mode 100644 index 000000000..9db3b7a02 --- /dev/null +++ b/__tests__/cli-unlock.test.ts @@ -0,0 +1,102 @@ +import { afterEach, beforeEach, describe, expect, it } from 'vitest'; +import { execFile, execFileSync } from 'child_process'; +import * as fs from 'fs'; +import * as net from 'net'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; +import { getDaemonPidPath, getDaemonSocketPath } from '../src/mcp/daemon-paths'; +import { CodeGraphPackageVersion } from '../src/mcp/version'; + +const BIN = path.resolve(__dirname, '../dist/bin/codegraph.js'); + +function runCodegraph(args: string[], cwd: string): string { + return execFileSync(process.execPath, [BIN, ...args], { + cwd, + encoding: 'utf8', + env: { ...process.env, CODEGRAPH_NO_DAEMON: '1' }, + stdio: ['ignore', 'pipe', 'pipe'], + }); +} + +function runCodegraphAsync(args: string[], cwd: string): Promise { + return new Promise((resolve, reject) => { + execFile( + process.execPath, + [BIN, ...args], + { cwd, encoding: 'utf8', env: { ...process.env, CODEGRAPH_NO_DAEMON: '1' } }, + (error, stdout, stderr) => { + if (error) reject(new Error(`${error.message}\n${stderr}`)); + else resolve(stdout); + }, + ); + }); +} + +describe('codegraph unlock — daemon artifact recovery (#1553)', () => { + let tempDir: string; + + beforeEach(() => { + tempDir = fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-unlock-')); + const cg = CodeGraph.initSync(tempDir); + cg.close(); + }); + + afterEach(() => { + fs.rmSync(tempDir, { recursive: true, force: true }); + }); + + it('removes indexing and phantom-daemon artifacts, then permits indexing', () => { + const graphDir = path.join(tempDir, '.codegraph'); + const pidPath = getDaemonPidPath(tempDir); + const socketPath = getDaemonSocketPath(tempDir); + fs.writeFileSync(path.join(graphDir, 'codegraph.lock'), 'stale\n'); + fs.writeFileSync(pidPath, JSON.stringify({ + pid: process.pid, + version: CodeGraphPackageVersion, + socketPath, + startedAt: Date.now() - 60_000, + })); + if (process.platform !== 'win32') fs.writeFileSync(socketPath, 'stale\n'); + + const output = runCodegraph(['unlock', tempDir], tempDir); + + expect(output).toContain('Removed stale lock artifacts'); + expect(fs.existsSync(path.join(graphDir, 'codegraph.lock'))).toBe(false); + expect(fs.existsSync(pidPath)).toBe(false); + if (process.platform !== 'win32') expect(fs.existsSync(socketPath)).toBe(false); + expect(() => process.kill(process.pid, 0)).not.toThrow(); + expect(() => runCodegraph(['index', '--quiet', tempDir], tempDir)).not.toThrow(); + }); + + it('preserves artifacts when the recorded live daemon answers the socket hello', async () => { + const pidPath = getDaemonPidPath(tempDir); + const socketPath = getDaemonSocketPath(tempDir); + const server = net.createServer((socket) => { + socket.end(JSON.stringify({ + codegraph: CodeGraphPackageVersion, + pid: process.pid, + socketPath, + protocol: 1, + }) + '\n'); + }); + await new Promise((resolve, reject) => { + server.once('error', reject); + server.listen(socketPath, resolve); + }); + fs.writeFileSync(pidPath, JSON.stringify({ + pid: process.pid, + version: CodeGraphPackageVersion, + socketPath, + startedAt: Date.now(), + })); + + try { + const output = await runCodegraphAsync(['unlock', tempDir], tempDir); + expect(output).toContain('No stale lock files found'); + expect(fs.existsSync(pidPath)).toBe(true); + } finally { + await new Promise((resolve) => server.close(() => resolve())); + } + }); +}); diff --git a/__tests__/daemon-registry.test.ts b/__tests__/daemon-registry.test.ts index 55bafc45a..aa13ec100 100644 --- a/__tests__/daemon-registry.test.ts +++ b/__tests__/daemon-registry.test.ts @@ -1,6 +1,7 @@ import { describe, it, expect, beforeEach, afterEach } from 'vitest'; import { spawn } from 'child_process'; import * as fs from 'fs'; +import * as net from 'net'; import * as os from 'os'; import * as path from 'path'; import { @@ -9,8 +10,11 @@ import { registerDaemon, deregisterDaemon, listDaemons, + listVerifiedDaemons, + stopDaemonAt, type DaemonRecord, } from '../src/mcp/daemon-registry'; +import { encodeLockInfo, getDaemonPidPath } from '../src/mcp/daemon-paths'; /** A pid that's guaranteed dead: spawn a trivial process, let it exit, reap it. */ async function deadPid(): Promise { @@ -100,4 +104,55 @@ describe('daemon-registry', () => { const live = listDaemons(); expect(live.map((d) => d.root)).toEqual(['/proj/new', '/proj/old']); }); + + it('keeps a registry entry whose socket hello matches its PID and version', async () => { + const root = fs.mkdtempSync(path.join(tmpHome, 'verified-')); + const socketPath = process.platform === 'win32' + ? `\\\\.\\pipe\\cg-reg-${process.pid}-${Date.now()}` + : path.join(tmpHome, 'verified.sock'); + const server = net.createServer((socket) => { + socket.end(JSON.stringify({ + protocol: 1, + pid: process.pid, + codegraph: '1.5.0', + socketPath, + }) + '\n'); + }); + await new Promise((resolve, reject) => { + server.once('error', reject); + server.listen(socketPath, resolve); + }); + try { + registerDaemon({ root, pid: process.pid, version: '1.5.0', socketPath, startedAt: 1 }); + expect((await listVerifiedDaemons()).map((d) => d.root)).toEqual([root]); + } finally { + await new Promise((resolve) => server.close(() => resolve())); + } + }); + + it('never signals a reused live PID when no matching daemon answers (#1553)', async () => { + const root = fs.mkdtempSync(path.join(tmpHome, 'project-')); + const pidPath = getDaemonPidPath(root); + fs.mkdirSync(path.dirname(pidPath), { recursive: true }); + fs.writeFileSync(pidPath, encodeLockInfo({ + pid: process.pid, + version: '1.5.0', + socketPath: path.join(root, '.codegraph', 'missing.sock'), + startedAt: Date.now() - 60_000, + })); + + registerDaemon({ + root, + pid: process.pid, + version: '1.5.0', + socketPath: path.join(root, '.codegraph', 'missing.sock'), + startedAt: Date.now() - 60_000, + }); + + expect(await listVerifiedDaemons()).toEqual([]); + const result = await stopDaemonAt(root); + expect(result).toMatchObject({ pid: process.pid, outcome: 'not-running' }); + expect(isProcessAlive(process.pid)).toBe(true); + expect(fs.existsSync(pidPath)).toBe(false); + }); }); diff --git a/__tests__/deprioritize-config.test.ts b/__tests__/deprioritize-config.test.ts new file mode 100644 index 000000000..b1cac1d08 --- /dev/null +++ b/__tests__/deprioritize-config.test.ts @@ -0,0 +1,280 @@ +/** + * `codegraph.json` → `deprioritize` — user-extensible ranking de-prioritization (#982). + * + * `matchesNonProductionDir` hardcodes example/sample/fixture/benchmark/demo, so a + * peripheral tree only the project knows about — `optional-skills/`, `scripts/` — + * gets no de-prioritization. When helpers in such a tree carry generic symbol + * names, an exact name match hands them a large bonus and they crowd out the + * product code that actually answers the query. + * + * This is the *ranking* half of #982, deliberately distinct from the corpus- + * frequency discount: that one keys on a name being COMMON, and is near-inert on + * #982's own 8-file repro where only two symbols are named `usage`. The fixture + * here IS that repro, which is the point — the two levers cover different shapes. + * + * It is also distinct from `exclude`, which is a recall lever. De-prioritized + * paths stay indexed and findable; they just stop winning. Locked below. + */ + +import { describe, it, expect, beforeAll, afterAll } from 'vitest'; +import * as fs from 'fs'; +import * as path from 'path'; +import * as os from 'os'; +import { CodeGraph } from '../src'; +import { initGrammars, loadAllGrammars } from '../src/extraction/grammars'; +import { loadDeprioritizePatterns } from '../src/project-config'; +import { nameMatchBonus, scorePathRelevance } from '../src/search/query-utils'; +import { DEPRIORITIZED_NAME_BONUS_SCALE } from '../src/db/queries'; + +const QUERY = 'desktop status bar context window usage'; + +/** #982's minimal reproduction layout. */ +function writeRepro(root: string): void { + const mk = (rel: string, content: string) => { + const p = path.join(root, rel); + fs.mkdirSync(path.dirname(p), { recursive: true }); + fs.writeFileSync(p, content); + }; + + // Product code. No symbol here is literally named `usage`. + mk( + 'apps/desktop/statusbar/StatusBar.ts', + [ + 'export class DesktopStatusBar {', + ' render(): string { return this.refresh(); }', + ' refresh(): string { return "status bar"; }', + ' mount(): void {}', + '}', + ].join('\n') + ); + mk( + 'apps/desktop/statusbar/StatusBarController.ts', + [ + "import { DesktopStatusBar } from './StatusBar';", + 'export class StatusBarController {', + ' constructor(private readonly bar: DesktopStatusBar) {}', + ' show(): string { return this.bar.render(); }', + '}', + ].join('\n') + ); + mk( + 'apps/desktop/context/ContextWindowMeter.ts', + [ + 'export class ContextWindowMeter {', + ' read(): number { return this.recompute(); }', + ' recompute(): number { return estimateTokens("context window"); }', + '}', + 'export function estimateTokens(text: string): number { return text.length; }', + ].join('\n') + ); + mk( + 'apps/desktop/context/format.ts', + 'export function formatTokens(n: number): string { return `${n} tokens`; }\n' + ); + mk('gateway/server/server.ts', 'export function startServer(): void {}\n'); + mk('packages/core/util/strings.ts', 'export function slugify(s: string): string { return s; }\n'); + + // The peripheral tree: two standalone helpers, each with a module-level `usage`. + for (const skill of ['bodyfat', 'nutrition']) { + mk( + `optional-skills/${skill}/scripts/${skill}_calc.ts`, + ['export function usage(): void {', ` console.log("usage: ${skill}_calc [options]");`, '}'].join('\n') + ); + } +} + +const isHelper = (r: { node: { name: string; filePath: string } }): boolean => + r.node.name.toLowerCase() === 'usage' && r.node.filePath.includes('optional-skills'); + +describe('codegraph.json deprioritize — parsing', () => { + let dir: string; + + beforeAll(() => { + dir = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-deprio-cfg-')); + }); + + afterAll(() => { + fs.rmSync(dir, { recursive: true, force: true }); + }); + + const write = (config: unknown): string => { + const sub = fs.mkdtempSync(path.join(dir, 'p-')); + fs.writeFileSync(path.join(sub, 'codegraph.json'), JSON.stringify(config)); + return sub; + }; + + it('defaults to empty with no config file', () => { + const sub = fs.mkdtempSync(path.join(dir, 'none-')); + expect(loadDeprioritizePatterns(sub)).toEqual([]); + }); + + it('keeps gitignore-style patterns verbatim, trimmed', () => { + const sub = write({ deprioritize: ['optional-skills/', ' tools/gen ', 'vendor/**'] }); + expect(loadDeprioritizePatterns(sub)).toEqual(['optional-skills/', 'tools/gen', 'vendor/**']); + }); + + it('warns-and-skips a non-array value instead of throwing', () => { + const sub = write({ deprioritize: 'optional-skills/' }); + expect(loadDeprioritizePatterns(sub)).toEqual([]); + }); + + it('drops blank and non-string entries, keeping the rest', () => { + const sub = write({ deprioritize: ['optional-skills/', '', 42, ' ', 'scripts/'] }); + expect(loadDeprioritizePatterns(sub)).toEqual(['optional-skills/', 'scripts/']); + }); + + it('does not disturb the other config keys', () => { + const sub = write({ deprioritize: ['optional-skills/'], exclude: ['static/'] }); + expect(loadDeprioritizePatterns(sub)).toEqual(['optional-skills/']); + }); +}); + +describe('#982 minimal repro — ranking with and without deprioritize', () => { + let baseDir: string; + let cfgDir: string; + let baseCg: CodeGraph; + let cfgCg: CodeGraph; + + beforeAll(async () => { + await initGrammars(); + await loadAllGrammars(); + + baseDir = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-deprio-base-')); + writeRepro(baseDir); + baseCg = CodeGraph.initSync(baseDir); + await baseCg.indexAll(); + + cfgDir = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-deprio-on-')); + writeRepro(cfgDir); + fs.writeFileSync( + path.join(cfgDir, 'codegraph.json'), + JSON.stringify({ deprioritize: ['optional-skills/'] }, null, 2) + ); + cfgCg = CodeGraph.initSync(cfgDir); + await cfgCg.indexAll(); + }, 180_000); + + afterAll(() => { + baseCg?.destroy(); + cfgCg?.destroy(); + for (const d of [baseDir, cfgDir]) if (d) fs.rmSync(d, { recursive: true, force: true }); + }); + + it('control: without the config the usage() helpers still take the top ranks', () => { + // This is the status quo the issue reports, and the shape the corpus-frequency + // discount cannot fix (only two symbols are named `usage` here, so it is rare). + const results = baseCg.searchNodes(QUERY, { limit: 20 }); + expect(results.length).toBeGreaterThanOrEqual(2); + expect(results.slice(0, 2).every(isHelper)).toBe(true); + }); + + it('with deprioritize, product code outranks the peripheral helpers', () => { + const results = cfgCg.searchNodes(QUERY, { limit: 20 }); + const firstHelper = results.findIndex(isHelper); + const firstProduct = results.findIndex((r) => r.node.filePath.includes('apps/desktop')); + expect(firstProduct).toBeGreaterThanOrEqual(0); + expect(firstHelper === -1 || firstProduct < firstHelper).toBe(true); + }); + + it('is a ranking lever, not exclude: the helpers stay indexed and findable', () => { + // The whole point of keeping this distinct from `exclude` — recall is intact. + expect(cfgCg.getNodesByName('usage').length).toBe(2); + const direct = cfgCg.searchNodes('usage', { limit: 20 }); + expect(direct.some(isHelper)).toBe(true); + }); + + it('leaves paths outside the patterns alone', () => { + // gateway/ and packages/ are not named, so their scores must not move. + const score = (cg: CodeGraph, file: string): number | undefined => + cg.searchNodes('slugify', { limit: 20 }).find((r) => r.node.filePath.includes(file))?.score; + const baseline = score(baseCg, 'packages/core/util/strings.ts'); + expect(baseline).toBeDefined(); + expect(score(cfgCg, 'packages/core/util/strings.ts')).toBe(baseline); + }); + + it('a query that genuinely targets the tree still ranks it, competitor present', () => { + // The "discount, don't erase" edge case #982 calls out. `bodyfat_calc` lives + // only in the de-prioritized tree; a query naming it must still find it + // first, even with product code competing for the same terms. + const results = cfgCg.searchNodes('bodyfat calc usage', { limit: 20 }); + expect(results.length).toBeGreaterThan(0); + expect(results[0].node.filePath).toContain('optional-skills/bodyfat'); + }); + + it('explore ranking honours the setting, not just search', () => { + // #982's reproduction rows B/C/D are all `codegraph explore`. Explore ranks + // through its own path scorer as well as through searchNodes, so a + // search-only fix would leave the reported surface unchanged. + const matcher = (cfgCg as unknown as { queries: { getDeprioritizedPathMatcher(): ((p: string) => boolean) | undefined } }) + .queries.getDeprioritizedPathMatcher(); + expect(matcher).toBeDefined(); + expect(matcher!('optional-skills/bodyfat/scripts/bodyfat_calc.ts')).toBe(true); + expect(matcher!('apps/desktop/statusbar/StatusBar.ts')).toBe(false); + }); + + it('picks up a config written after the project was opened', async () => { + // wireLayers runs once per open, so a matcher captured there would freeze + // at open time — and the MCP server keeps one CodeGraph per root alive for + // its whole lifetime, which would make an edited config look like a no-op. + const late = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-deprio-late-')); + writeRepro(late); + const cg = CodeGraph.initSync(late); + try { + await cg.indexAll(); + const before = cg.searchNodes(QUERY, { limit: 20 }); + expect(before.length).toBeGreaterThanOrEqual(2); + expect(before.slice(0, 2).every(isHelper)).toBe(true); + + fs.writeFileSync( + path.join(late, 'codegraph.json'), + JSON.stringify({ deprioritize: ['optional-skills/'] }) + ); + const after = cg.searchNodes(QUERY, { limit: 20 }); + const firstHelper = after.findIndex(isHelper); + const firstProduct = after.findIndex((r) => r.node.filePath.includes('apps/desktop')); + expect(firstProduct).toBeGreaterThanOrEqual(0); + expect(firstHelper === -1 || firstProduct < firstHelper).toBe(true); + } finally { + cg.destroy(); + fs.rmSync(late, { recursive: true, force: true }); + } + }, 180_000); +}); + +describe('scorePathRelevance — the two deliberate asymmetries (#982)', () => { + it('docks a path that is both test-like and de-prioritized only once', () => { + const both = 'example/a/foo.ts'; + const asTestOnly = scorePathRelevance(both, 'foo'); + const asBoth = scorePathRelevance(both, 'foo', undefined, true); + expect(asBoth).toBe(asTestOnly); + }); + + it('does not waive the user penalty for a test-y query, unlike the built-ins', () => { + // The built-in classification is inferred, so a test-y query waives it. A + // `deprioritize` pattern is a standing statement by the project, so it + // stands. Asserted so the difference is a decision, not an accident. + const builtIn = scorePathRelevance('example/a/foo.ts', 'foo test'); + const userDeclared = scorePathRelevance('optional-skills/a/foo.ts', 'foo test', undefined, true); + expect(userDeclared).toBe(builtIn - 15); + }); +}); + +describe('the name-bonus damping constant is derived, not picked (#982)', () => { + it('keeps a damped exact match above the prefix arm, so it cannot lose to one', () => { + // A de-prioritized node keeps `80 * SCALE` of the whole-query exact bonus + // and also takes the -15 path penalty. The prefix arm tops out below 40, so + // `80 * SCALE - 15 > 40` is what guarantees the exact match still wins — + // "discount, don't erase" stated as arithmetic instead of taste. + expect(nameMatchBonus('child', 'child')).toBe(80); + expect(nameMatchBonus('children', 'child')).toBeLessThan(40); + expect(80 * DEPRIORITIZED_NAME_BONUS_SCALE - 15).toBeGreaterThan(40); + }); + + it('would fail at the originally proposed 0.25, which is why it moved', () => { + // Measured on a 62k-node django index with `deprioritize: ["tests/"]`: at + // 0.25 the exact-name queries `child`, `parent` and `method` lost rank 1 to + // the prefix matches `children`, `all_parents` and `method_decorator`. + // Asserted so nobody lowers the constant back without meeting the bound. + expect(80 * 0.25 - 15).toBeLessThan(40); + }); +}); diff --git a/__tests__/erlang-arity-resolution.test.ts b/__tests__/erlang-arity-resolution.test.ts new file mode 100644 index 000000000..9351f81b3 --- /dev/null +++ b/__tests__/erlang-arity-resolution.test.ts @@ -0,0 +1,147 @@ +/** + * Erlang arity-aware resolution (#1610). + * + * Arity is part of a function's identity: `f/1` and `f/2` are unrelated + * definitions. Extraction gives each arity its own node (`mod::f/1`) and + * stamps refs with the call-site arity; resolution must land each ref on the + * def of exactly that arity — the everyday `header/2 -> header/3` delegation + * must be a real edge, never a self-loop — and refuse to guess a sibling + * arity when the named one doesn't exist. + */ +import { describe, it, expect, beforeEach, afterEach } from 'vitest'; +import * as fs from 'node:fs'; +import * as path from 'node:path'; +import * as os from 'node:os'; +import { CodeGraph } from '../src'; + +describe('erlang arity-aware resolution', () => { + let dir: string; + beforeEach(() => { dir = fs.mkdtempSync(path.join(os.tmpdir(), 'erlang-arity-')); }); + afterEach(() => { fs.rmSync(dir, { recursive: true, force: true }); }); + + async function callEdges(d: string): Promise> { + const cg = await CodeGraph.init(d, { silent: true }); + await cg.indexAll(); + const db = (cg as any).db.db; + const rows = db + .prepare( + `SELECT s.qualified_name sq, t.qualified_name tq + FROM edges e JOIN nodes s ON s.id = e.source JOIN nodes t ON t.id = e.target + WHERE e.kind IN ('calls','references') AND s.kind = 'function'` + ) + .all(); + cg.destroy(); + return rows; + } + + it('resolves the f/N -> f/N+1 delegation to a real edge, not a self-loop', async () => { + fs.mkdirSync(path.join(dir, 'src'), { recursive: true }); + fs.writeFileSync( + path.join(dir, 'src', 'deleg.erl'), + `-module(deleg). +-export([header/2]). + +header(Name, Req) -> + header(Name, Req, undefined). + +-spec header(binary(), map(), any()) -> any(). +header(Name, Headers, Default) -> + maps:get(Name, Headers, Default). +` + ); + const edges = await callEdges(dir); + expect(edges).toContainEqual({ sq: 'deleg::header/2', tq: 'deleg::header/3' }); + // No self-loop in either direction. + expect(edges.some((e) => e.sq === e.tq && e.sq.startsWith('deleg::header'))).toBe(false); + }); + + it('resolves remote calls to the called arity and refuses a sibling arity', async () => { + fs.mkdirSync(path.join(dir, 'src'), { recursive: true }); + fs.writeFileSync( + path.join(dir, 'src', 'store.erl'), + `-module(store). +-export([get/1, get/2]). + +get(K) -> get(K, undefined). +get(K, Default) -> {K, Default}. +` + ); + fs.writeFileSync( + path.join(dir, 'src', 'client.erl'), + `-module(client). +-export([fetch/1, broken/1]). + +fetch(K) -> + store:get(K, nil). + +broken(K) -> + store:get(K, nil, extra). +` + ); + const edges = await callEdges(dir); + expect(edges).toContainEqual({ sq: 'client::fetch/1', tq: 'store::get/2' }); + // store:get/3 doesn't exist — the ref must resolve to NOTHING, not /1 or /2. + expect(edges.some((e) => e.sq === 'client::broken/1' && e.tq.startsWith('store::get'))).toBe(false); + }); + + it('resolves an arity-less dynamic MFA ref only when exactly one arity exists', async () => { + fs.mkdirSync(path.join(dir, 'src'), { recursive: true }); + fs.writeFileSync( + path.join(dir, 'src', 'single.erl'), + `-module(single). +-export([work/1]). + +work(X) -> X. +` + ); + fs.writeFileSync( + path.join(dir, 'src', 'multi.erl'), + `-module(multi). +-export([job/1, job/2]). + +job(X) -> X. +job(X, Y) -> {X, Y}. +` + ); + fs.writeFileSync( + path.join(dir, 'src', 'spawner.erl'), + `-module(spawner). +-export([go/1]). + +go(Args) -> + erlang:spawn(single, work, Args), + erlang:spawn(multi, job, Args). +` + ); + const edges = await callEdges(dir); + // `Args` is dynamic, so both refs are arity-less. single:work has exactly + // one arity — it resolves; multi:job has two — silent beats wrong. + expect(edges).toContainEqual({ sq: 'spawner::go/1', tq: 'single::work/1' }); + expect(edges.some((e) => e.sq === 'spawner::go/1' && e.tq.startsWith('multi::job'))).toBe(false); + }); + + it('lands `fun mod:f/1` references on the written arity', async () => { + fs.mkdirSync(path.join(dir, 'src'), { recursive: true }); + fs.writeFileSync( + path.join(dir, 'src', 'lib_m.erl'), + `-module(lib_m). +-export([bump/1, bump/2]). + +bump(X) -> X + 1. +bump(X, N) -> X + N. +` + ); + fs.writeFileSync( + path.join(dir, 'src', 'user_m.erl'), + `-module(user_m). +-export([run/1]). + +run(L) -> + lists:map(fun lib_m:bump/1, L). +` + ); + const edges = await callEdges(dir); + expect(edges).toContainEqual({ sq: 'user_m::run/1', tq: 'lib_m::bump/1' }); + expect(edges.some((e) => e.sq === 'user_m::run/1' && e.tq === 'lib_m::bump/2')).toBe(false); + }); +}); diff --git a/__tests__/erlang-behaviour-synthesizer.test.ts b/__tests__/erlang-behaviour-synthesizer.test.ts index d5d33f4ef..f3e11fc47 100644 --- a/__tests__/erlang-behaviour-synthesizer.test.ts +++ b/__tests__/erlang-behaviour-synthesizer.test.ts @@ -187,4 +187,45 @@ on_event(Ev) -> {seen, Ev}. const rows = await synthEdges(dir); expect(rows.map((r) => path.basename(r.tf))).toEqual(['public_impl.erl']); }); + + it('counts dispatch-site arity across <> literals (#1358)', async () => { + fs.mkdirSync(path.join(dir, 'src'), { recursive: true }); + fs.writeFileSync( + path.join(dir, 'src', 'codec_behaviour.erl'), + `-module(codec_behaviour). + +-callback decode(binary(), list()) -> term(). + +-export([run/3]). + +run(Mod, Bin, Opts) -> + Mod:decode(Bin, Opts). +` + ); + fs.writeFileSync( + path.join(dir, 'src', 'json_codec.erl'), + `-module(json_codec). +-behaviour(codec_behaviour). +-export([decode/2]). + +decode(Bin, _Opts) -> Bin. +` + ); + // The dispatch site passes a binary literal whose commas previously + // inflated the computed arity (4 instead of 2), so the edge was dropped. + fs.writeFileSync( + path.join(dir, 'src', 'probe.erl'), + `-module(probe). +-export([go/1]). + +go(Mod) -> + Mod:decode(<<1,2,3>>, []). +` + ); + + const rows = await synthEdges(dir); + const fromProbe = rows.filter((r) => r.source === 'go').map((r) => `${path.basename(r.tf)}:${r.target}`); + expect(fromProbe).toEqual(['json_codec.erl:decode']); + expect(rows.every((r) => r.via === 'codec_behaviour:decode/2' || r.source !== 'go')).toBe(true); + }); }); diff --git a/__tests__/explore-path-pinning.test.ts b/__tests__/explore-path-pinning.test.ts new file mode 100644 index 000000000..b79ba7d12 --- /dev/null +++ b/__tests__/explore-path-pinning.test.ts @@ -0,0 +1,122 @@ +/** + * End-to-end gate for query-path pinning + the segment-vocab supplement + + * variable seeding, on the bug that motivated all three: an agent named a + * SvelteKit route file by exact path plus behavior words ("scrollToBottom, + * onscroll, atBottom tracking") and got back neither the file's scroll code + * nor the file itself at full weight — the bracketed path was tokenizer + * shrapnel (`runId` seeded as a named symbol, every sibling `+page` admitted) + * and the camelCase scroll symbols were FTS-opaque. + * + * The fixture mirrors that shape in plain TS (bracket/paren directories are + * the crux, not the language): a target file under + * `src/routes/m/projects/[id]/runs/[runId]/` holding `feedAtBottom` / + * `handleFeedScroll` / `pinFeedIfNearBottom`, a decoy chat-window page under + * a `(protected)` route group, and a runs-store decoy defining `runId` and + * `Scope` — the two symbols that headlined the original junk blast radius. + */ +import { describe, it, expect, beforeAll, afterAll } from 'vitest'; +import * as fs from 'fs'; +import * as path from 'path'; +import * as os from 'os'; +import CodeGraph from '../src/index'; +import { ToolHandler } from '../src/mcp/tools'; + +const FIXTURE = 'explore-path-pinning'; +const TARGET = 'src/routes/m/projects/[id]/runs/[runId]/+page.ts'; +const DECOY_CHAT = 'src/routes/(protected)/chat-window/+page.ts'; + +let dir: string; +let cg: CodeGraph; + +async function explore(query: string): Promise { + const res = await new ToolHandler(cg).execute('codegraph_explore', { query }); + return res.content?.[0]?.text ?? ''; +} + +/** The response renders a source section for `file`. */ +const hasSection = (response: string, file: string): boolean => + response.includes('**`' + file + '`'); + +beforeAll(async () => { + dir = fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-path-pin-')); + fs.cpSync(path.join(__dirname, 'fixtures', FIXTURE), dir, { recursive: true }); + fs.rmSync(path.join(dir, '.codegraph'), { recursive: true, force: true }); + cg = CodeGraph.initSync(dir); + await cg.indexAll(); +}, 180_000); + +afterAll(() => { + cg?.destroy(); + if (dir && fs.existsSync(dir)) fs.rmSync(dir, { recursive: true, force: true }); +}); + +describe('fixture shape — if this rots, the gates below mean nothing', () => { + it('indexes the bracketed-path target with its scroll symbols', () => { + const names = cg.getNodesInFile(TARGET).map((n) => n.name); + expect(names).toContain('feedAtBottom'); + expect(names).toContain('handleFeedScroll'); + expect(names).toContain('pinFeedIfNearBottom'); + }); +}); + +describe('path pinning (fix 1)', () => { + it('a pure-path query renders the named file and says it was pinned', async () => { + const out = await explore(TARGET); + expect(hasSection(out, TARGET)).toBe(true); + expect(out).toContain('pinned from the query'); + }); + + it('the original bug-shaped query renders the pinned file, not path shrapnel', async () => { + const out = await explore( + `run page auto-scroll to bottom logic in ${TARGET} — scrollToBottom, onscroll, atBottom tracking`, + ); + expect(hasSection(out, TARGET)).toBe(true); + // The path fragments must not seed: `runId` (runs-store decoy) and the + // bracketed segment's namesakes headlined the original junk blast radius. + const blast = out.split('**Relationships**')[0]!; + expect(blast).not.toMatch(/`runId` \(src\/lib\/runs-store\.ts/); + // The chat decoy MAY render — it genuinely holds scroll-pinning code the + // segment supplement now finds — but the pinned file must rank first. + // (Pre-fix, `+page`/`runs` shrapnel admitted the siblings ABOVE the named + // file and the envelope truncated it.) + const decoyAt = out.indexOf('**`' + DECOY_CHAT + '`'); + const targetAt = out.indexOf('**`' + TARGET + '`'); + expect(targetAt).toBeGreaterThan(-1); + if (decoyAt !== -1) expect(targetAt).toBeLessThan(decoyAt); + }); + + it('an unresolvable path is reported, not silently dropped', async () => { + const out = await explore('crash in src/routes/gone/missing-page.ts on load'); + expect(out).toContain('No indexed file uniquely matches'); + expect(out).toContain('src/routes/gone/missing-page.ts'); + }); +}); + +describe('extension-less kebab basenames (the amnisphere gap)', () => { + const KEBAB_TARGET = 'src/lib/background-image-table.ts'; + + it('a bare kebab basename — no slash, no extension — pins and renders its file', async () => { + // Pre-fix this query never opened the path gate; FTS shredded the token + // into `background`/`image`/`table` and served the fragment decoy instead. + const out = await explore('background-image-table Source column'); + expect(hasSection(out, KEBAB_TARGET)).toBe(true); + expect(out).toContain('pinned from the query'); + }); + + it('kebab prose that names no file is not reported as an unresolved path', async () => { + const out = await explore('how does cross-call dedup interact with feed scroll pinning'); + expect(out).not.toContain('No indexed file uniquely matches'); + }); +}); + +describe('segment supplement + variable seeding (fixes 2–3)', () => { + it('word-level scroll terms reach the camelCase scroll code without a path', async () => { + const out = await explore('feed auto-scroll to bottom pinning behavior'); + expect(hasSection(out, TARGET)).toBe(true); + }); + + it('a camel infix naming only $state-style variables still finds their file', async () => { + const out = await explore('where does the atBottom flag get reset'); + expect(hasSection(out, TARGET)).toBe(true); + }); +}); diff --git a/__tests__/explore-pinned-allocation.test.ts b/__tests__/explore-pinned-allocation.test.ts new file mode 100644 index 000000000..e093c8ad2 --- /dev/null +++ b/__tests__/explore-pinned-allocation.test.ts @@ -0,0 +1,83 @@ +/** + * Pinned files in `allocateExploreBudget` (see query-paths.ts): a file the + * query named by PATH must survive every allocation guard. Its score is + * whatever the path-stripped query happened to match — for a pure-path query, + * nearly nothing — so without the pinned floor the proportional split would + * fund the one file the agent explicitly asked for worst of all, and the + * cliff would zero it outright. + */ +import { describe, it, expect } from 'vitest'; +import { allocateExploreBudget, getExploreOutputBudget, EXPLORE_ALLOCATION } from '../src/mcp/tools'; +import type { ExploreAllocationCandidate } from '../src/mcp/tools'; + +const cand = ( + path: string, + score: number, + extra: Partial = {}, +): ExploreAllocationCandidate => ({ path, score, worth: 1, spine: false, ...extra }); + +const budget = getExploreOutputBudget(1000); + +describe('allocateExploreBudget — pinned files', () => { + it('never cliffs a pinned file, however low it scores', () => { + const { allowances, cliffed } = allocateExploreBudget( + [ + cand('pinned.svelte', 0.1, { pinned: true }), + cand('hub.ts', 200), + cand('noise.ts', 0.1), + ], + budget, + 8, + ); + expect(cliffed).toContain('noise.ts'); + expect(cliffed).not.toContain('pinned.svelte'); + expect(allowances.has('pinned.svelte')).toBe(true); + }); + + it('funds a pinned file at least as well as the strongest candidate', () => { + const { allowances } = allocateExploreBudget( + [ + cand('pinned.svelte', 0.5, { pinned: true }), + cand('hub.ts', 300), + cand('helper.ts', 40), + ], + budget, + 8, + ); + expect(allowances.get('pinned.svelte')!).toBeGreaterThanOrEqual(allowances.get('hub.ts')!); + expect(allowances.get('pinned.svelte')!).toBeGreaterThan(allowances.get('helper.ts')!); + }); + + it('keeps pinned files through the affordability trim', () => { + // Smallest tier: affordable = floor(13000 / (MIN_CHARS + FILE_OVERHEAD)) = 14 + // slots. 18 equal-weight candidates admitted → the trim must cut 4. The + // pinned file sits last with a TIED weight (the pinned floor lifts it to + // the top weight), so the stable by-weight sort would slice it off — only + // the explicit spine/pinned keep saves it. + const tiny = getExploreOutputBudget(10); + const fleet = Array.from({ length: 17 }, (_, i) => cand(`f${i}.ts`, 100)); + fleet.push(cand('pinned.svelte', 0.1, { pinned: true })); + const { allowances, cliffed } = allocateExploreBudget(fleet, tiny, 18); + expect(allowances.has('pinned.svelte')).toBe(true); + expect(cliffed).not.toContain('pinned.svelte'); + expect(allowances.size).toBeLessThan(18); + }); + + it('an all-pinned zero-score call still allocates (pure-path query)', () => { + const { allowances, pool } = allocateExploreBudget( + [cand('a.svelte', 0, { pinned: true }), cand('b.svelte', 0, { pinned: true })], + budget, + 8, + ); + expect(pool).toBeGreaterThan(0); + expect(allowances.get('a.svelte')!).toBeGreaterThanOrEqual(EXPLORE_ALLOCATION.MIN_CHARS); + expect(allowances.get('b.svelte')!).toBeGreaterThanOrEqual(EXPLORE_ALLOCATION.MIN_CHARS); + }); + + it('unpinned behavior is unchanged when no candidate is pinned', () => { + const before = allocateExploreBudget( + [cand('a.ts', 40), cand('b.ts', 10)], budget, 8, + ); + expect(before.allowances.get('a.ts')!).toBeGreaterThan(before.allowances.get('b.ts')!); + }); +}); diff --git a/__tests__/extraction.test.ts b/__tests__/extraction.test.ts index 292658822..ad0ba2374 100644 --- a/__tests__/extraction.test.ts +++ b/__tests__/extraction.test.ts @@ -175,6 +175,53 @@ class ENGINE_API UNetConnectionRepControl : public UObject expect(detectLanguage('cfoo.h', '#ifndef CFOO_H\nstruct Point { int x; int y; };\nvoid f(struct Point p);\n#endif\n')).toBe('c'); }); + it('should detect a .h whose only C++ signal is a plain base clause as cpp (#1592)', () => { + // No export macro, no `class` keyword, no access section, no `virtual`: + // the derived struct's base clause is the only C++ construct, and the + // #1159 branch only knows the macro-annotated form. Misdetected as C, the + // C extractor drops `Derived` and mints a phantom `function Base`. + expect(detectLanguage('min.h', 'struct Base {};\nstruct Derived : Base {};\n')).toBe('cpp'); + expect(detectLanguage('pub.h', 'struct Derived : public Base {};\n')).toBe('cpp'); + expect(detectLanguage('scoped.h', 'struct Derived : ns::Base {};\n')).toBe('cpp'); + expect(detectLanguage('tmpl.h', 'struct Derived : Base> {};\n')).toBe('cpp'); + expect(detectLanguage('final.h', 'struct Derived final : Base {};\n')).toBe('cpp'); + expect(detectLanguage('multi.h', 'class Derived : public A, private B\n{\n};\n')).toBe('cpp'); + expect(detectLanguage('virt.h', 'struct Derived : virtual Base {};\n')).toBe('cpp'); + + // The base clause sits PAST the 8 KB sample, behind a long C-compatible + // preamble (guards, defines, plain typedefs) — the second pass must scan + // the whole file, not just the sample. + const preamble = '#ifndef BIG_H\n#define BIG_H\n' + '#define VALUE_0 0\n'.repeat(700); + expect(preamble.length).toBeGreaterThan(8192); + expect(detectLanguage('big.h', `${preamble}struct Base {};\nstruct Derived : Base {};\n#endif\n`)).toBe('cpp'); + + // Controls — all genuine C, none may flip to C++: + // a bit-field (`:` after a member name inside the body), + expect(detectLanguage('bits.h', 'struct S { unsigned int a : 3; unsigned int b : 5; };\n')).toBe('c'); + // a ternary whose `:` follows a `sizeof(struct …)` / cast, + expect(detectLanguage('tern.h', 'static inline int sz(int x) { return x ? sizeof(struct foo) : 0; }\n#define P(a,b) ((a) ? (struct foo *)(a) : (b))\n')).toBe('c'); + // a label / identifier that merely starts with `struct`, + expect(detectLanguage('label.h', 'static void g(void) {\nstruct_end:\n return;\n}\nint struct_a, struct_b;\n')).toBe('c'); + // a doc comment whose prose reads like a base clause, + expect(detectLanguage('doc.h', '/* struct timeval: seconds, microseconds */\nstruct timeval { long tv_sec; long tv_usec; };\n// struct foo: x, y\n')).toBe('c'); + // and the two existing controls. + expect(detectLanguage('cfoo.h', '#ifndef CFOO_H\nstruct Point { int x; int y; };\nvoid f(struct Point p);\n#endif\n')).toBe('c'); + expect(detectLanguage('stdio.h', '#ifndef STDIO_H\nvoid printf();\n#endif\n')).toBe('c'); + }); + + it('should extract a derived struct from a plain base-clause .h, with no phantom function (#1592)', () => { + const result = extractFromSource('src/min.h', 'struct Base {};\nstruct Derived : Base {};\n'); + const derived = result.nodes.find((n) => n.name === 'Derived'); + expect(derived).toBeDefined(); + expect(derived?.kind).toBe('struct'); + expect(derived?.language).toBe('cpp'); + // The C mis-route read `Derived : Base {}` as a K&R-ish function `Base` + // returning `Derived` — that phantom must be gone. + expect(result.nodes.some((n) => n.name === 'Base' && n.kind === 'function')).toBe(false); + expect(result.nodes.filter((n) => n.name === 'Base')).toHaveLength(1); + expect(result.nodes.find((n) => n.name === 'Base')?.kind).toBe('struct'); + }); + it('should return unknown for unsupported extensions', () => { expect(detectLanguage('styles.css')).toBe('unknown'); expect(detectLanguage('data.json')).toBe('unknown'); @@ -1131,6 +1178,146 @@ impl Cache for MyCache { expect(implRef?.fromNodeId).toBe(myCacheNode?.id); }); + it('qualifies methods of a generic or lifetime impl by the implementing type, not the trait (#1588)', () => { + const code = ` +pub trait Source { + fn read(&mut self) -> usize; +} + +pub struct FileSource { pub n: usize } +impl Source for FileSource { + fn read(&mut self) -> usize { self.n } +} + +pub struct BufSource { pub inner: T } +impl Source for BufSource { + fn read(&mut self) -> usize { 0 } +} + +pub struct Parents<'a> { cur: &'a u32 } +impl<'a> Iterator for Parents<'a> { + type Item = u32; + fn next(&mut self) -> Option { None } +} + +pub struct Wrapper { pub n: usize } +impl Source for &Wrapper { + fn read(&mut self) -> usize { 1 } +} + +pub mod m { pub struct Scoped { pub n: usize } } +impl Source for m::Scoped { + fn read(&mut self) -> usize { 2 } +} + +pub struct Own { pub n: usize } +impl From for Own { + fn from(n: u32) -> Self { Own { n: n as usize } } +} +`; + const result = extractFromSource('src.rs', code); + + // Every impl method is qualified by the IMPLEMENTING type. Before, a + // parameterized implementing type (`BufSource`, `Parents<'a>`, `&Wrapper`) + // left the trait's identifier as the only bare type_identifier child of the + // impl, so those methods were recorded as `Source::read` / `Iterator::next`. + const methodQns = result.nodes + .filter((n) => n.kind === 'method') + .map((n) => n.qualifiedName) + .sort(); + expect(methodQns).toEqual([ + 'BufSource::read', + 'FileSource::read', + 'Own::from', + 'Parents::next', + 'Scoped::read', + 'Source::read', + 'Wrapper::read', + ]); + // The trait's qualified name now names exactly one node: its declaration. + const traitRead = result.nodes.filter((n) => n.qualifiedName === 'Source::read'); + expect(traitRead).toHaveLength(1); + expect(traitRead[0]!.startLine).toBe(3); + + // The implements back-reference comes FROM the implementing type's node + // for every impl shape, named by the trait's full text. + const implementsFrom = (typeName: string): string[] => { + const typeNode = result.nodes.find((n) => n.name === typeName && n.kind === 'struct'); + expect(typeNode, typeName).toBeDefined(); + return result.unresolvedReferences + .filter((r) => r.referenceKind === 'implements' && r.fromNodeId === typeNode!.id) + .map((r) => r.referenceName); + }; + expect(implementsFrom('FileSource')).toEqual(['Source']); + expect(implementsFrom('BufSource')).toEqual(['Source']); + expect(implementsFrom('Parents')).toEqual(['Iterator']); + expect(implementsFrom('Wrapper')).toEqual(['Source']); + expect(implementsFrom('Scoped')).toEqual(['Source']); + expect(implementsFrom('Own')).toEqual(['From']); + + // …and the owner `contains` edge lands on the implementing type too. + const buf = result.nodes.find((n) => n.name === 'BufSource' && n.kind === 'struct')!; + const bufRead = result.nodes.find((n) => n.qualifiedName === 'BufSource::read')!; + expect( + result.edges.some((e) => e.kind === 'contains' && e.source === buf.id && e.target === bufRead.id) + ).toBe(true); + }); + + it('keeps the owner-field shape for `self..()` and collapses every other receiver (#1585)', () => { + const code = ` +pub struct Outer { pub inner: Inner, pub deep: Deep } +impl Outer { + pub fn run(&mut self) { + self.inner.run(); + self.deep.inner.run(); + self.make().run(); + (self.inner).run(); + self.run(); + let local = Inner { n: 0 }; + local.run(); + } +} +`; + const result = extractFromSource('outer.rs', code); + const calls = result.unresolvedReferences + .filter((r) => r.referenceKind === 'calls') + .map((r) => r.referenceName); + // Exactly one call keeps the `self.` prefix — the single-hop field + // receiver whose type the resolver can read off the owner struct. + expect(calls.filter((c) => c.startsWith('self.'))).toEqual(['self.inner.run']); + // A local receiver keeps its name as before… + expect(calls).toContain('local.run'); + // …and the deeper chain, the call receiver, the parenthesized receiver and + // the bare `self` receiver all still collapse to the method name. + expect(calls.filter((c) => c === 'run')).toHaveLength(4); + expect(calls).toContain('make'); + const outerRun = result.nodes.find((n) => n.qualifiedName === 'Outer::run'); + expect(outerRun).toBeDefined(); + const fieldRef = result.unresolvedReferences.find((r) => r.referenceName === 'self.inner.run'); + expect(fieldRef?.fromNodeId).toBe(outerRun!.id); + expect(fieldRef?.line).toBe(5); + }); + + it('gives no receiver to an impl whose target names no single type', () => { + // A tuple / `dyn Trait` / primitive implementing type has no struct to + // hang the methods off, so they are extracted as plain functions — the + // pre-#1588 behavior for these shapes, minus the trait mis-qualification. + const code = ` +pub trait Base { fn id(&self) -> u32; } +impl Base for (u32, u32) { + fn id(&self) -> u32 { 0 } +} +impl Base for dyn Base { + fn id(&self) -> u32 { 1 } +} +`; + const result = extractFromSource('src.rs', code); + const ids = result.nodes.filter((n) => n.name === 'id'); + expect(ids.map((n) => n.qualifiedName).sort()).toEqual(['Base::id', 'id', 'id']); + expect(ids.filter((n) => n.kind === 'function')).toHaveLength(2); + expect(result.unresolvedReferences.filter((r) => r.referenceKind === 'implements')).toHaveLength(0); + }); + it('should extract trait supertraits as extends references', () => { const code = ` pub trait Display {} @@ -6936,6 +7123,54 @@ export function multiply(a: number, b: number): number { cg.close(); }); + it('should resolve an ES import from a .xsjs file to a .xsjslib file (#556)', async () => { + // Exercises the JS import-path resolution list: `./helpers` must resolve to + // `helpers.xsjslib`. `decoy.js` exports the same symbol name and is never + // imported — without .xsjs/.xsjslib in the list the import resolves to + // nothing and the call falls back to same-name matching, which binds the + // edge to the decoy. The decoy is what makes this test fail on a regression: + // with a lone helpers.xsjslib the fallback happens to pick the right file. + fs.writeFileSync( + path.join(tempDir, 'helpers.xsjslib'), + 'export function buildQuery(table) {\n return "SELECT * FROM " + table;\n}\n' + ); + fs.writeFileSync( + path.join(tempDir, 'decoy.js'), + 'export function buildQuery(table) {\n return "DECOY " + table;\n}\n' + ); + fs.writeFileSync( + path.join(tempDir, 'service.xsjs'), + 'import { buildQuery } from "./helpers";\n\nfunction run() {\n return buildQuery("users");\n}\n' + ); + + const cg = CodeGraph.initSync(tempDir); + await cg.indexAll(); + + const run = cg.getNodesInFile('service.xsjs').find((n) => n.name === 'run'); + const buildQuery = cg.getNodesInFile('helpers.xsjslib').find((n) => n.name === 'buildQuery'); + const decoy = cg.getNodesInFile('decoy.js').find((n) => n.name === 'buildQuery'); + expect(run).toBeDefined(); + expect(buildQuery).toBeDefined(); + expect(decoy).toBeDefined(); + + expect( + cg.getFileDependencies('service.xsjs'), + "'./helpers' should resolve to helpers.xsjslib, not the same-named decoy" + ).toEqual(['helpers.xsjslib']); + + const outgoing = cg.getOutgoingEdges(run!.id); + expect( + outgoing.find((e) => e.target === buildQuery!.id), + 'run() should resolve buildQuery across the .xsjs -> .xsjslib import' + ).toBeDefined(); + expect( + outgoing.find((e) => e.target === decoy!.id), + 'run() must not bind to the unrelated same-named export in decoy.js' + ).toBeUndefined(); + + cg.close(); + }); + it('should count the full file-level tracked class (yaml/twig/properties) in indexFiles()', async () => { fs.writeFileSync(path.join(tempDir, 'app.yaml'), 'name: test\n'); fs.writeFileSync(path.join(tempDir, 'view.twig'), '{{ title }}\n'); @@ -10131,7 +10366,81 @@ helper() -> ok. const ns = result.nodes.find((n) => n.kind === 'namespace'); expect(ns?.name).toBe('my_server'); const start = result.nodes.find((n) => n.kind === 'function' && n.name === 'start'); - expect(start?.qualifiedName).toBe('my_server::start'); + // Arity is part of an Erlang function's identity — qualifiedName carries it (#1610). + expect(start?.qualifiedName).toBe('my_server::start/0'); + }); + + it('should give same-name different-arity functions separate arity-qualified nodes (#1610)', () => { + const code = `-module(gap). +-export([f/1, f/2]). + +f(X) -> X + 1. +f(X, Y) -> X + Y. +`; + const result = extractFromSource('src/gap.erl', code); + const fns = result.nodes.filter((n) => n.kind === 'function' && n.name === 'f'); + expect(fns).toHaveLength(2); + expect(fns.map((n) => n.qualifiedName).sort()).toEqual(['gap::f/1', 'gap::f/2']); + const f1 = fns.find((n) => n.qualifiedName === 'gap::f/1')!; + const f2 = fns.find((n) => n.qualifiedName === 'gap::f/2')!; + expect([f1.startLine, f1.endLine]).toEqual([4, 4]); + expect([f2.startLine, f2.endLine]).toEqual([5, 5]); + expect(f1.signature).toBe('f(X)'); + expect(f2.signature).toBe('f(X, Y)'); + }); + + it('should split interleaved same-name defs by arity with distinct qualified names', () => { + const code = `-module(inter). + +f(X) -> X + 1; +f(Y) -> Y. +g() -> ok. +f(X, Y) -> X + Y. +`; + const result = extractFromSource('src/inter.erl', code); + const fs = result.nodes.filter((n) => n.kind === 'function' && n.name === 'f'); + expect(fs).toHaveLength(2); + expect(fs.map((n) => n.qualifiedName).sort()).toEqual(['inter::f/1', 'inter::f/2']); + // Clauses of the same arity still merge into one span. + const f1 = fs.find((n) => n.qualifiedName === 'inter::f/1')!; + expect([f1.startLine, f1.endLine]).toEqual([3, 4]); + }); + + it('should flag exported per arity (#1610)', () => { + const code = `-module(m). +-export([f/1]). + +f(X) -> X. +f(X, Y) -> {X, Y}. +`; + const result = extractFromSource('src/m.erl', code); + expect(result.nodes.find((n) => n.qualifiedName === 'm::f/1')?.isExported).toBe(true); + expect(result.nodes.find((n) => n.qualifiedName === 'm::f/2')?.isExported).toBe(false); + }); + + it('should attach a -spec sitting between two arities to the arity it names (#1610)', () => { + const code = `-module(deleg). +-export([header/2, header/3]). + +header(Name, Req) -> + header(Name, Req, undefined). + +-spec header(binary(), map(), any()) -> any(). +header(Name, Headers, Default) -> + maps:get(Name, Headers, Default). +`; + const result = extractFromSource('src/deleg.erl', code); + const h2 = result.nodes.find((n) => n.qualifiedName === 'deleg::header/2')!; + const h3 = result.nodes.find((n) => n.qualifiedName === 'deleg::header/3')!; + expect(h2.signature).toBe('header(Name, Req)'); + expect(h3.signature).toBe('-spec header(binary(), map(), any()) -> any().'); + expect([h2.startLine, h2.endLine]).toEqual([4, 5]); + expect(h3.startLine).toBe(8); + // The delegation call carries the callee's arity — no more self-loop. + const calls = result.unresolvedReferences + .filter((r) => r.referenceKind === 'calls') + .map((r) => r.referenceName); + expect(calls).toContain('header/3'); }); it('should flag exported functions and honor -compile(export_all)', () => { @@ -10266,11 +10575,31 @@ prepare(X) -> X. `; const result = extractFromSource('src/m.erl', code); const calls = result.unresolvedReferences.filter((r) => r.referenceKind === 'calls').map((r) => r.referenceName); - expect(calls).toContain('prepare'); - // `mod:fn(...)` is emitted as `mod::fn` — the same shape the module - // namespace gives every function's qualifiedName, so it resolves via - // the qualified-name matcher. - expect(calls).toContain('other_mod::process'); + expect(calls).toContain('prepare/1'); + // `mod:fn(...)` is emitted as `mod::fn/arity` — the same shape the + // module namespace + arity suffix gives every function's qualifiedName, + // so it resolves via the qualified-name matcher (#1610). + expect(calls).toContain('other_mod::process/1'); + }); + + it('should carry written arity on fun references and static MFA lists (#1610)', () => { + const code = `-module(m). +-export([go/0]). + +go() -> + lists:map(fun bump/1, [1]), + Prod = fun other_mod:produce/2, + proc_lib:spawn_link(?MODULE, work, [a, b]), + Prod. + +bump(X) -> X + 1. +work(_A, _B) -> ok. +`; + const result = extractFromSource('src/m.erl', code); + const refs = result.unresolvedReferences; + expect(refs.some((r) => r.referenceKind === 'references' && r.referenceName === 'bump/1')).toBe(true); + expect(refs.some((r) => r.referenceKind === 'references' && r.referenceName === 'other_mod::produce/2')).toBe(true); + expect(refs.some((r) => r.referenceKind === 'calls' && r.referenceName === 'work/2')).toBe(true); }); it('should not emit calls for dynamic dispatch (var module / var fun)', () => { @@ -10283,9 +10612,9 @@ run(Mod, F) -> `; const result = extractFromSource('src/m.erl', code); const calls = result.unresolvedReferences.filter((r) => r.referenceKind === 'calls').map((r) => r.referenceName); - expect(calls).not.toContain('handle'); - expect(calls).not.toContain('Mod::handle'); - expect(calls).not.toContain('F'); + expect(calls.some((c) => c.startsWith('handle'))).toBe(false); + expect(calls.some((c) => c.startsWith('Mod::'))).toBe(false); + expect(calls.some((c) => c === 'F' || c.startsWith('F/'))).toBe(false); }); it('should connect gen_server self-calls to the module handlers', () => { @@ -10313,9 +10642,10 @@ handle_cast({put, K, V}, S) -> {noreply, maps:put(K, V, S)}. const result = extractFromSource('src/kv_store.erl', code); const calls = result.unresolvedReferences.filter((r) => r.referenceKind === 'calls').map((r) => r.referenceName); // ?SERVER (defined as ?MODULE), ?MODULE, and the module's own atom all - // count as self — public API wrappers connect to their handlers. - expect(calls.filter((c) => c === 'kv_store::handle_call')).toHaveLength(2); - expect(calls).toContain('kv_store::handle_cast'); + // count as self — public API wrappers connect to their handlers, at + // OTP's fixed handler arities (#1610). + expect(calls.filter((c) => c === 'kv_store::handle_call/3')).toHaveLength(2); + expect(calls).toContain('kv_store::handle_cast/2'); }); it('should connect gen_server calls to a registered-name module, directly or via an atom macro', () => { @@ -10335,8 +10665,8 @@ evict(Key) -> // OTP's {local, ?MODULE} convention names a server after its module — // a cross-module registered name targets that module's handlers. A name // matching no module simply never resolves downstream. - expect(calls).toContain('kv_store::handle_call'); - expect(calls).toContain('kv_store::handle_cast'); + expect(calls).toContain('kv_store::handle_call/3'); + expect(calls).toContain('kv_store::handle_cast/2'); }); it('should not connect gen_server calls with dynamic targets', () => { @@ -10369,10 +10699,10 @@ monitor_loop(_P) -> ok. `; const result = extractFromSource('src/m.erl', code); const calls = result.unresolvedReferences.filter((r) => r.referenceKind === 'calls').map((r) => r.referenceName); - expect(calls).toContain('request_process'); // ?MODULE → bare, same-file resolution - expect(calls).toContain('monitor_loop'); - expect(calls).toContain('other_mod::handle'); - expect(calls).toContain('other_mod::tick'); + expect(calls).toContain('request_process/2'); // ?MODULE → bare-with-arity, same-file resolution + expect(calls).toContain('monitor_loop/1'); + expect(calls).toContain('other_mod::handle/1'); + expect(calls).toContain('other_mod::tick/0'); }); it('should stay silent on dynamic spawn/apply (var module, fun value, or plain fun)', () => { @@ -10389,8 +10719,8 @@ helper() -> ok. const result = extractFromSource('src/m.erl', code); const calls = result.unresolvedReferences.filter((r) => r.referenceKind === 'calls').map((r) => r.referenceName); // The fun body's call is still walked; no phantom MFA targets appear. - expect(calls).toContain('helper'); - expect(calls.filter((c) => c !== 'spawn' && c !== 'apply' && c !== 'helper')).toHaveLength(0); + expect(calls).toContain('helper/0'); + expect(calls.filter((c) => !['spawn/3', 'spawn/1', 'apply/3', 'helper/0'].includes(c))).toHaveLength(0); }); it('should treat ?MODULE:fn calls as local calls', () => { @@ -10404,7 +10734,7 @@ work() -> ok. `; const result = extractFromSource('src/m.erl', code); const calls = result.unresolvedReferences.filter((r) => r.referenceKind === 'calls').map((r) => r.referenceName); - expect(calls).toContain('work'); + expect(calls).toContain('work/0'); }); it('should capture fun name/arity values as function references', () => { @@ -10419,8 +10749,8 @@ notify(_P) -> ok. `; const result = extractFromSource('src/m.erl', code); const refs = result.unresolvedReferences.filter((r) => r.referenceKind === 'references').map((r) => r.referenceName); - expect(refs).toContain('notify'); - expect(refs).toContain('m::notify'); + expect(refs).toContain('notify/1'); + expect(refs).toContain('m::notify/1'); }); it('should reference records used in bodies and argument patterns', () => { @@ -10456,8 +10786,8 @@ second(X) -> X. const calls = result.unresolvedReferences.filter( (r) => r.referenceKind === 'calls' && r.fromNodeId === handle?.id ).map((r) => r.referenceName); - expect(calls).toContain('first'); - expect(calls).toContain('second'); + expect(calls).toContain('first/1'); + expect(calls).toContain('second/1'); }); }); @@ -10479,8 +10809,8 @@ analyze(Path) -> expect(fns).toContain('main'); expect(fns).toContain('analyze'); const calls = result.unresolvedReferences.filter((r) => r.referenceKind === 'calls').map((r) => r.referenceName); - expect(calls).toContain('analyze'); - expect(calls).toContain('io::format'); + expect(calls).toContain('analyze/1'); + expect(calls).toContain('io::format/2'); }); it('should link an app resource file to its callback module and dependency apps', () => { @@ -10526,7 +10856,7 @@ do_thing(X) -> const refsFrom = (id?: string) => result.unresolvedReferences.filter((r) => r.fromNodeId === id).map((r) => `${r.referenceKind}:${r.referenceName}`); // The body's remote call belongs to the macro node — true exactly once. - expect(refsFrom(macro?.id)).toContain('calls:audit_logger::log'); + expect(refsFrom(macro?.id)).toContain('calls:audit_logger::log/2'); // The use site joins the call chain: do_thing -calls→ LOG_AUDIT. expect(refsFrom(doThing?.id)).toContain('calls:LOG_AUDIT'); }); @@ -10559,7 +10889,7 @@ prepare() -> ok. const result = extractFromSource('src/m.erl', code); const refs = result.unresolvedReferences.map((r) => r.referenceName); // The nested call inside the macro's arguments still attributes to check/0. - expect(refs).toContain('prepare'); + expect(refs).toContain('prepare/0'); // ?assertEqual (an OTP header macro) is emitted and simply never resolves… expect(refs).toContain('assertEqual'); // …but predefined macros have no definition to link. @@ -10579,7 +10909,7 @@ prepare() -> ok. const alias = result.nodes.find((n) => n.kind === 'constant' && n.name === 'ALIAS'); const refsFrom = (id?: string) => result.unresolvedReferences.filter((r) => r.fromNodeId === id).map((r) => `${r.referenceKind}:${r.referenceName}`); - expect(refsFrom(target?.id)).toContain('calls:target_fn'); + expect(refsFrom(target?.id)).toContain('calls:target_fn/0'); expect(refsFrom(alias?.id)).toContain('references:TARGET'); }); }); diff --git a/__tests__/fixtures/explore-path-pinning/package.json b/__tests__/fixtures/explore-path-pinning/package.json new file mode 100644 index 000000000..c17caa943 --- /dev/null +++ b/__tests__/fixtures/explore-path-pinning/package.json @@ -0,0 +1,5 @@ +{ + "name": "explore-path-pinning-fixture", + "version": "1.0.0", + "private": true +} diff --git a/__tests__/fixtures/explore-path-pinning/src/lib/background-image-table.ts b/__tests__/fixtures/explore-path-pinning/src/lib/background-image-table.ts new file mode 100644 index 000000000..e77d83334 --- /dev/null +++ b/__tests__/fixtures/explore-path-pinning/src/lib/background-image-table.ts @@ -0,0 +1,21 @@ +/** Table of background images for a training set — source-column rendering. */ + +export interface BackgroundImageRow { + id: string; + sourceUrl: string; + label: string; +} + +let tableRows: BackgroundImageRow[] = []; + +export function loadTableRows(rows: BackgroundImageRow[]): void { + tableRows = rows; +} + +export function renderSourceColumn(row: BackgroundImageRow): string { + return `${row.label}: ${row.sourceUrl}`; +} + +export function sortRowsBySource(): BackgroundImageRow[] { + return [...tableRows].sort((a, b) => a.sourceUrl.localeCompare(b.sourceUrl)); +} diff --git a/__tests__/fixtures/explore-path-pinning/src/lib/background-store.ts b/__tests__/fixtures/explore-path-pinning/src/lib/background-store.ts new file mode 100644 index 000000000..541986e26 --- /dev/null +++ b/__tests__/fixtures/explore-path-pinning/src/lib/background-store.ts @@ -0,0 +1,11 @@ +/** Uploaded-background registry — shares the `background` fragment with the table file. */ + +let backgrounds: string[] = []; + +export function addBackground(url: string): void { + backgrounds.push(url); +} + +export function listBackgrounds(): string[] { + return [...backgrounds]; +} diff --git a/__tests__/fixtures/explore-path-pinning/src/lib/runs-store.ts b/__tests__/fixtures/explore-path-pinning/src/lib/runs-store.ts new file mode 100644 index 000000000..b3a39d889 --- /dev/null +++ b/__tests__/fixtures/explore-path-pinning/src/lib/runs-store.ts @@ -0,0 +1,31 @@ +/** In-memory registry of task runs, keyed by run id. */ + +export interface Scope { + projectId: string; + label: string; +} + +export const runId = 'run-000'; + +const runs = new Map(); + +export function registerRun(id: string, scope: Scope): void { + runs.set(id, { id, scope, status: 'queued' }); +} + +export function getRun(id: string): { id: string; scope: Scope; status: string } | null { + return runs.get(id) ?? null; +} + +export function listRuns(scope: Scope): string[] { + return [...runs.values()] + .filter((r) => r.scope.projectId === scope.projectId) + .map((r) => r.id); +} + +export function stopRun(id: string): boolean { + const run = runs.get(id); + if (!run) return false; + run.status = 'cancelled'; + return true; +} diff --git a/__tests__/fixtures/explore-path-pinning/src/routes/(protected)/chat-window/+page.ts b/__tests__/fixtures/explore-path-pinning/src/routes/(protected)/chat-window/+page.ts new file mode 100644 index 000000000..3023ba459 --- /dev/null +++ b/__tests__/fixtures/explore-path-pinning/src/routes/(protected)/chat-window/+page.ts @@ -0,0 +1,28 @@ +/** Detached chat window page — session presence + streaming state. */ + +let chatAtBottom = true; +let isStreaming = false; +let messages: string[] = []; + +export function handleMessagesScroll(distance: number): void { + chatAtBottom = distance < 50; +} + +export function sendMessage(text: string): void { + messages = [...messages, text]; + isStreaming = true; +} + +export function stopResponse(): void { + isStreaming = false; +} + +export function redock(): void { + messages = []; + isStreaming = false; + chatAtBottom = true; +} + +export function chatSnapshot(): { messages: string[]; streaming: boolean } { + return { messages: [...messages], streaming: isStreaming }; +} diff --git a/__tests__/fixtures/explore-path-pinning/src/routes/m/projects/[id]/runs/[runId]/+page.ts b/__tests__/fixtures/explore-path-pinning/src/routes/m/projects/[id]/runs/[runId]/+page.ts new file mode 100644 index 000000000..c5fe2b77c --- /dev/null +++ b/__tests__/fixtures/explore-path-pinning/src/routes/m/projects/[id]/runs/[runId]/+page.ts @@ -0,0 +1,67 @@ +/** Mobile run feed — event stream + scroll pinning for the run page. */ + +export interface FeedEvent { + id: string; + kind: 'output' | 'tool' | 'error'; + content: string; +} + +const EVENT_CAP = 300; + +let events: FeedEvent[] = []; +let workingLine: string | null = null; + +/** Whether the reader is at the tail of the feed (within 50px). */ +let feedAtBottom = true; + +interface FeedElement { + scrollTop: number; + scrollHeight: number; + clientHeight: number; +} + +let feedEl: FeedElement | null = null; + +export function bindFeedElement(el: FeedElement | null): void { + feedEl = el; +} + +/** Track the reader's position; called from the feed's scroll listener. */ +export function handleFeedScroll(): void { + const el = feedEl; + if (!el) return; + feedAtBottom = el.scrollHeight - el.scrollTop - el.clientHeight < 50; +} + +/** Re-pin after async content growth (image loads), only when at the tail. */ +export function pinFeedIfNearBottom(): void { + const el = feedEl; + if (!el) return; + if (feedAtBottom) { + el.scrollTop = el.scrollHeight; + } +} + +export function appendEvent(event: FeedEvent): void { + events = events.length >= EVENT_CAP + ? [...events.slice(-(EVENT_CAP - 1)), event] + : [...events, event]; + if (feedAtBottom) { + pinFeedIfNearBottom(); + } +} + +export function setWorkingLine(line: string | null): void { + workingLine = line; + pinFeedIfNearBottom(); +} + +export function resetFeed(): void { + events = []; + workingLine = null; + feedAtBottom = true; +} + +export function feedSnapshot(): { events: FeedEvent[]; workingLine: string | null } { + return { events: [...events], workingLine }; +} diff --git a/__tests__/fixtures/kernel-parity/torture.rs b/__tests__/fixtures/kernel-parity/torture.rs index 1e14b7b7a..8fb8382be 100644 --- a/__tests__/fixtures/kernel-parity/torture.rs +++ b/__tests__/fixtures/kernel-parity/torture.rs @@ -72,6 +72,16 @@ impl Widget { self.n * mul() } + /// Receiver shapes (#1585): only `self..()` keeps the + /// owner-field prefix; deeper / parenthesized / call / bare-self collapse. + fn via_field(&self) -> u32 { + self.field.deep_call(); + self.field.z.clone(); + self.method_a().chain_b(); + (self.field).deep_call(); + self.area() + } + fn clone_self(&self) -> Self { Self::assoc(); Widget { @@ -107,6 +117,71 @@ impl Render for Container { fn render(&self) {} } +/// Receiver = the impl_item's `type` field (#1588): generic, lifetime, +/// reference, scoped, and generic-trait impls all qualify by the TYPE. +pub trait Source { + fn read(&mut self) -> usize; +} + +pub struct FileSource { + pub n: usize, +} + +impl Source for FileSource { + fn read(&mut self) -> usize { + self.n + } +} + +pub struct BufSource { + pub inner: T, +} + +impl Source for BufSource { + fn read(&mut self) -> usize { + 0 + } +} + +pub struct Parents<'a> { + cur: &'a u32, +} + +impl<'a> Iterator for Parents<'a> { + type Item = u32; + fn next(&mut self) -> Option { + None + } +} + +impl Container { + fn dup(&self) -> T { + self.item.clone() + } +} + +impl Base for &Widget {} + +impl Render for &mut BufSource { + fn render(&self) {} +} + +impl Base for self::Deep {} + +impl From for FileSource { + fn from(n: u32) -> Self { + FileSource { n: n as usize } + } +} + +impl Base for (u32, u32) {} + +impl Render for dyn Base { + fn render(&self) {} +} + +impl Base for u32 {} + impl Later { fn touch(&self) {} } diff --git a/__tests__/foundation.test.ts b/__tests__/foundation.test.ts index 12c136445..b7616272a 100644 --- a/__tests__/foundation.test.ts +++ b/__tests__/foundation.test.ts @@ -120,6 +120,41 @@ describe('CodeGraph Foundation', () => { cg.close(); }); + it('restores every secondary index after a crash inside bulk parse load (#1556)', () => { + const dbPath = getDatabasePath(tempDir); + const first = DatabaseConnection.initialize(dbPath); + const before = (first.getDb() + .prepare("SELECT name FROM sqlite_master WHERE type = 'index' ORDER BY name") + .all() as Array<{ name: string }>).map((r) => r.name); + first.beginBulkParseLoad(); + first.close(); + + const reopened = DatabaseConnection.open(dbPath); + const after = (reopened.getDb() + .prepare("SELECT name FROM sqlite_master WHERE type = 'index' ORDER BY name") + .all() as Array<{ name: string }>).map((r) => r.name); + reopened.close(); + + expect(after).toEqual(before); + }); + + it('skips secondary-index DDL when the schema is already healthy', () => { + const dbPath = getDatabasePath(tempDir); + const connection = DatabaseConnection.initialize(dbPath); + const db = connection.getDb(); + const originalExec = db.exec.bind(db); + let execCalls = 0; + db.exec = (sql: string) => { + execCalls++; + originalExec(sql); + }; + + (connection as any).healBulkSecondaryIndexes(); + connection.close(); + + expect(execCalls).toBe(0); + }); + it('should return correct database size', () => { const cg = CodeGraph.initSync(tempDir); const stats = cg.getStats(); diff --git a/__tests__/frameworks.test.ts b/__tests__/frameworks.test.ts index cc7e3555f..6064706d7 100644 --- a/__tests__/frameworks.test.ts +++ b/__tests__/frameworks.test.ts @@ -1472,6 +1472,47 @@ func boot(routes: RoutesBuilder) throws { const { nodes } = vaporResolver.extract!('configure.swift', src); expect(nodes).toHaveLength(0); }); + + // A `.METHOD(...)` call with many comma-separated args and no `use:` used to + // make the route regex backtrack exponentially (60 args hung for minutes). + it('does not backtrack exponentially on a long arg list without use:', () => { + const args = Array.from({ length: 60 }, (_, i) => `arg${i}: value${i}`).join(', '); + const src = `app.get(${args})\n`; + const start = performance.now(); + const { nodes } = vaporResolver.extract!('routes.swift', src); + const elapsed = performance.now() - start; + expect(nodes).toHaveLength(0); + expect(elapsed).toBeLessThan(250); + }); + + it('still parses every Vapor route shape after the arg-list rewrite', () => { + const src = ` +admin.get(use: self.list) +app.get("users", use: listUsers) +router.post("users", User.parameter, "edit", use: UserController.edit) +app.patch(":id" , "meta" , use: update) +app.get( + "multi", + "line", + use: multiLine +) +`; + const { nodes, references } = vaporResolver.extract!('routes.swift', src); + expect(nodes.map((n) => n.name)).toEqual([ + 'GET /', + 'GET /users', + 'POST /users/edit', + 'PATCH /:id/meta', + 'GET /multi/line', + ]); + expect(references.map((r) => r.referenceName)).toEqual([ + 'list', + 'listUsers', + 'edit', + 'update', + 'multiLine', + ]); + }); }); import { reactResolver } from '../src/resolution/frameworks/react'; diff --git a/__tests__/git-changed-untracked-dir.test.ts b/__tests__/git-changed-untracked-dir.test.ts new file mode 100644 index 000000000..699c01ec2 --- /dev/null +++ b/__tests__/git-changed-untracked-dir.test.ts @@ -0,0 +1,68 @@ +/** + * Regression test for #1213: `codegraph sync` silently skips untracked files + * that live inside an untracked directory. + * + * `git status --porcelain` collapses an entirely-untracked directory into a + * single `?? frontend/` entry. getGitChangedFiles must still surface the source + * files inside it (via `-uall`) rather than dropping the whole directory. + */ + +import { describe, it, expect, afterEach } from 'vitest'; +import { execFileSync } from 'child_process'; +import * as fs from 'fs'; +import * as path from 'path'; +import * as os from 'os'; +import { getGitChangedFiles } from '../src/extraction/index'; + +function git(cwd: string, args: string[]): void { + execFileSync('git', args, { cwd, stdio: 'pipe' }); +} + +describe('getGitChangedFiles — untracked directories (#1213)', () => { + const dirs: string[] = []; + + function makeRepo(): string { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-1213-')); + dirs.push(dir); + git(dir, ['init']); + git(dir, ['config', 'user.email', 'test@example.com']); + git(dir, ['config', 'user.name', 'test']); + fs.writeFileSync(path.join(dir, 'root.js'), 'function foo() {}\n'); + git(dir, ['add', 'root.js']); + git(dir, ['commit', '-m', 'init']); + return dir; + } + + afterEach(() => { + while (dirs.length) { + fs.rmSync(dirs.pop()!, { recursive: true, force: true }); + } + }); + + it('detects source files inside a fully-untracked directory', () => { + const dir = makeRepo(); + fs.mkdirSync(path.join(dir, 'frontend')); + fs.writeFileSync(path.join(dir, 'frontend', 'app.js'), 'function bar() {}\n'); + + const changes = getGitChangedFiles(dir); + + expect(changes).not.toBeNull(); + expect(changes!.added).toContain('frontend/app.js'); + }); + + it('still recurses into an untracked embedded git repo (no -uall regression)', () => { + // `-uall` must not break the embedded-repo path: git collapses a nested + // repo to `?? embedded/` regardless of `-uall`, so its files are only + // reachable through collectGitStatus's recursion. + const dir = makeRepo(); + const embedded = path.join(dir, 'embedded'); + fs.mkdirSync(embedded); + git(embedded, ['init']); + fs.writeFileSync(path.join(embedded, 'inner.js'), 'function baz() {}\n'); + + const changes = getGitChangedFiles(dir); + + expect(changes).not.toBeNull(); + expect(changes!.added).toContain('embedded/inner.js'); + }); +}); diff --git a/__tests__/identifier-segments.test.ts b/__tests__/identifier-segments.test.ts index 11884bf0a..477c702a1 100644 --- a/__tests__/identifier-segments.test.ts +++ b/__tests__/identifier-segments.test.ts @@ -2,6 +2,7 @@ import { describe, it, expect } from 'vitest'; import { splitIdentifierSegments, extractProseCandidates, + extractSegmentSearchWords, normalizeProseWord, segmentLookupVariants, } from '../src/search/identifier-segments'; @@ -101,3 +102,32 @@ describe('segmentLookupVariants — light plural folding', () => { expect(segmentLookupVariants('boxes')).toEqual(['boxes']); // -es strip would go sub-minimum }); }); + +describe('extractSegmentSearchWords — query words for the search-side vocab supplement', () => { + it('keeps prose words and adds camel-token segments', () => { + const words = extractSegmentSearchWords('auto-scroll to bottom — atBottom tracking'); + // Prose candidates survive as before… + expect(words).toContain('scroll'); + expect(words).toContain('bottom'); + expect(words).toContain('tracking'); + // …and the camel token contributed its ≥4-char segments ("at" is under + // the prose minimum; "bottom" arrives from the split even when the prose + // pass missed it). + expect(extractSegmentSearchWords('where is atBottom set')).toContain('bottom'); + }); + + it('splits multi-hump tokens into every usable segment', () => { + const words = extractSegmentSearchWords('trace pinFeedIfNearBottom please'); + expect(words).toEqual(expect.arrayContaining(['feed', 'near', 'bottom'])); + }); + + it('does not invent segments for plain prose', () => { + const words = extractSegmentSearchWords('how does checkout work'); + expect(words).toContain('checkout'); + expect(words).not.toContain('check'); + }); + + it('returns nothing for an empty query', () => { + expect(extractSegmentSearchWords('')).toEqual([]); + }); +}); diff --git a/__tests__/installer-targets.test.ts b/__tests__/installer-targets.test.ts index 0d185a9d8..4ec3e5903 100644 --- a/__tests__/installer-targets.test.ts +++ b/__tests__/installer-targets.test.ts @@ -240,6 +240,42 @@ describe('Installer targets — partial-state idempotency', () => { expect(mdEntry?.action).toBe('updated'); }); + it('codex: local install writes ./.codex/config.toml and the project-root ./AGENTS.md block (#1531)', () => { + const codex = getTarget('codex')!; + const result = codex.install('local', { autoAllow: false }); + const paths = result.files.map((f) => f.path.replace(/\\/g, '/')); + // macOS realpath shenanigans (/var vs /private/var) — suffix match. + expect(paths.some((p) => p.endsWith('/.codex/config.toml'))).toBe(true); + // AGENTS.md sits at the project root, NOT under .codex/ — that's the + // file Codex reads for repo instructions. + expect(paths.some((p) => p.endsWith('/AGENTS.md') && !p.includes('/.codex/'))).toBe(true); + + const toml = fs.readFileSync(path.join(process.cwd(), '.codex', 'config.toml'), 'utf-8'); + expect(toml).toContain('[mcp_servers.codegraph]'); + expect(fs.readFileSync(path.join(process.cwd(), 'AGENTS.md'), 'utf-8')).toContain('codegraph explore'); + + // The project layer is only applied in a trusted project, so say so + // instead of reporting a silent success. + expect(result.notes?.join(' ')).toMatch(/trusted/); + + // Global config is untouched by a local install. + expect(fs.existsSync(path.join(tmpHome, '.codex', 'config.toml'))).toBe(false); + }); + + it('codex: local uninstall reverses the local install and leaves the global entry alone (#1531)', () => { + const codex = getTarget('codex')!; + codex.install('global', { autoAllow: false }); + codex.install('local', { autoAllow: false }); + expect(codex.detect('local').alreadyConfigured).toBe(true); + + codex.uninstall('local'); + + expect(codex.detect('local').alreadyConfigured).toBe(false); + expect(codex.detect('global').alreadyConfigured).toBe(true); + expect(fs.readFileSync(path.join(tmpHome, '.codex', 'config.toml'), 'utf-8')) + .toContain('[mcp_servers.codegraph]'); + }); + it('opencode: prefers .jsonc when both .json and .jsonc exist', () => { const opencode = getTarget('opencode')!; const dir = path.join(tmpHome, '.config', 'opencode'); diff --git a/__tests__/kernel-deep-nesting.test.ts b/__tests__/kernel-deep-nesting.test.ts new file mode 100644 index 000000000..7529ff3b3 --- /dev/null +++ b/__tests__/kernel-deep-nesting.test.ts @@ -0,0 +1,281 @@ +/** + * Deep-nesting safety for the native kernel (#1581). + * + * The kernel's per-language walkers recurse once per AST level. tree-sitter's + * parser is iterative, so a pathologically nested file — clang's + * `clang/test/Parser/parser_overflow.c` nests 16,384 `{`; fuzzer corpora go + * deeper — parses fine and then overflowed the WALKER's native stack. A native + * overflow is uncatchable: the parse worker is a thread of the `codegraph` + * process, so the SIGSEGV killed the whole indexer with no message, no partial + * index, no per-file fallback. Worker threads get Node's 4 MiB default stack; + * the 8 MiB main thread only moved the cliff (100k levels still died). + * + * The kernel now guards its recursion against the calling thread's real stack + * bounds (codegraph-kernel/src/stack.rs) and turns an imminent overflow into + * its `defer:` routing signal, so the file takes the wasm path — whose walker + * catches its own JS `RangeError` per file and stores a partial result with a + * `parse_error`. These tests pin that contract on every default-routed + * language, on the main thread AND inside a default-sized worker, and + * end-to-end through the built CLI. + * + * Like the other kernel suites: skipped without a staged .node; CI that + * builds the kernel sets CODEGRAPH_KERNEL_EXPECT=1 so a missing binary FAILS. + */ + +import { describe, it, expect, beforeAll, beforeEach, afterEach } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { execFileSync } from 'child_process'; +import { Worker } from 'worker_threads'; +import { extractFromSource } from '../src/extraction'; +import { initGrammars, loadGrammarsForLanguages } from '../src/extraction/grammars'; +import { kernelRoutes, resetKernelForTests } from '../src/extraction/kernel'; +import type { Language } from '../src/types'; + +const REPO = path.resolve(__dirname, '..'); +const KERNEL_PATH = path.join( + REPO, + 'codegraph-kernel', + 'prebuilds', + `${process.platform}-${process.arch}`, + 'codegraph-kernel.node' +); +const kernelBuilt = fs.existsSync(KERNEL_PATH); +const expectKernel = process.env.CODEGRAPH_KERNEL_EXPECT === '1'; +const BIN = path.join(REPO, 'dist', 'bin', 'codegraph.js'); +const DIST_KERNEL = path.join(REPO, 'dist', 'extraction', 'kernel'); +const distBuilt = fs.existsSync(BIN) && fs.existsSync(path.join(DIST_KERNEL, 'index.js')); + +/** Deep enough to overflow an 8 MiB main-thread stack on every walker. */ +const PARENS_DEPTH = 60_000; +/** The reporter's exact shape: clang's parser_overflow.c nests 16,384 `{`. */ +const BRACES_DEPTH = 16_384; + +const CANDIDATES: Language[] = [ + 'typescript', 'tsx', 'javascript', 'jsx', 'java', 'python', 'go', 'c', 'cpp', + 'rust', 'csharp', 'ruby', 'php', 'swift', 'kotlin', 'r', 'lua', 'luau', 'scala', 'dart', +]; + +const EXT: Record = { + typescript: 'ts', tsx: 'tsx', javascript: 'js', jsx: 'jsx', java: 'java', python: 'py', + go: 'go', c: 'c', cpp: 'cpp', rust: 'rs', csharp: 'cs', ruby: 'rb', php: 'php', + swift: 'swift', kotlin: 'kt', r: 'R', lua: 'lua', luau: 'luau', scala: 'scala', dart: 'dart', +}; + +/** A function `f` whose body is a `depth`-deep parenthesized expression. */ +function deepParens(language: Language, depth: number): string { + const open = '('.repeat(depth); + const close = ')'.repeat(depth); + switch (language) { + case 'typescript': case 'tsx': case 'javascript': case 'jsx': + return `function f() { return ${open}1${close}; }\n`; + case 'java': + return `class A {\n int f() { return ${open}1${close}; }\n}\n`; + case 'python': + return `def f():\n return ${open}1${close}\n`; + case 'go': + return `package p\n\nfunc f() int { return ${open}1${close} }\n`; + case 'c': + return `int f(void) { return ${open}1${close}; }\n`; + case 'cpp': + return `int f() { return ${open}1${close}; }\n`; + case 'rust': + return `fn f() -> i32 { ${open}1${close} }\n`; + case 'csharp': + return `class A {\n int f() { return ${open}1${close}; }\n}\n`; + case 'ruby': + return `def f\n ${open}1${close}\nend\n`; + case 'php': + return ` Int { return ${open}1${close} }\n`; + case 'kotlin': + return `fun f(): Int { return ${open}1${close} }\n`; + case 'r': + return `f <- function() {\n ${open}1${close}\n}\n`; + case 'lua': case 'luau': + return `local function f()\n return ${open}1${close}\nend\n`; + case 'scala': + return `object A {\n def f(): Int = ${open}1${close}\n}\n`; + case 'dart': + return `int f() { return ${open}1${close}; }\n`; + default: + throw new Error(`no deep fixture for ${language}`); + } +} + +/** The reporter's repro: a C function body of `depth` nested blocks. */ +function deepBraces(depth: number): string { + return `void foo(void) {\n${'{'.repeat(depth)}${'}'.repeat(depth)}\n}\n`; +} + +const ENV_KEYS = ['CODEGRAPH_KERNEL', 'CODEGRAPH_KERNEL_LANGS', 'CODEGRAPH_KERNEL_PATH'] as const; +let savedEnv: Record; + +describe.skipIf(!kernelBuilt)('kernel deep-nesting guard (#1581)', () => { + let routed: Language[] = []; + + beforeAll(async () => { + resetKernelForTests(); + routed = CANDIDATES.filter((l) => kernelRoutes(l)); + expect(routed.length).toBeGreaterThan(0); + await initGrammars(); + await loadGrammarsForLanguages(routed); + }); + + beforeEach(() => { + savedEnv = Object.fromEntries(ENV_KEYS.map((k) => [k, process.env[k]])); + resetKernelForTests(); + }); + + afterEach(() => { + for (const k of ENV_KEYS) { + if (savedEnv[k] === undefined) delete process.env[k]; + else process.env[k] = savedEnv[k]; + } + resetKernelForTests(); + }); + + it('every default-routed language survives a 60k-deep expression on the main thread', () => { + const failures: string[] = []; + for (const language of routed) { + const file = `deep.${EXT[language]}`; + const source = deepParens(language, PARENS_DEPTH); + // The ONLY acceptable outcomes: a clean result (the thread's stack was + // big enough for the walk), or the wasm fallback's partial result with + // its parse_error. A native overflow would have killed this process. + const result = extractFromSource(file, source, language); + const fn = result.nodes.find((n) => n.name === 'f' && (n.kind === 'function' || n.kind === 'method')); + // R's wasm walker mints `f <- function()` only after walking the + // assignment's value, so its partial result for a file this deep holds + // just the file node — the same shape main's wasm-only path produces + // (verified with CODEGRAPH_KERNEL=0). Pre-existing and out of scope + // here; what this test pins for R is that the process survives. + if (!fn && language !== 'r') failures.push(`${language}: no function node 'f' (nodes=${result.nodes.map((n) => `${n.kind}:${n.name}`).join(',')})`); + for (const e of result.errors) { + if (!/Maximum call stack|parse_error|Parse error/.test(`${e.code} ${e.message}`)) { + failures.push(`${language}: unexpected error ${e.message}`); + } + } + } + expect(failures).toEqual([]); + }, 120_000); + + it("the reporter's 16,384-brace C file is indexed (partial) instead of killing the process", () => { + const result = extractFromSource('deep.c', deepBraces(BRACES_DEPTH), 'c'); + expect(result.nodes.some((n) => n.kind === 'function' && n.name === 'foo')).toBe(true); + }, 60_000); + + it('shallow files still take the kernel path (the guard never trips on normal code)', () => { + // Sanity for the perf-neutral claim: a 200-deep expression is far inside + // any thread's stack, so it must come back clean with no parse_error. + for (const language of routed) { + const result = extractFromSource(`ok.${EXT[language]}`, deepParens(language, 200), language); + expect(result.errors, language).toEqual([]); + expect(result.nodes.some((n) => n.name === 'f'), language).toBe(true); + } + }, 60_000); + + describe.skipIf(!distBuilt)('inside a default-sized (4 MiB) parse worker, through dist/', () => { + let tmp: string; + beforeEach(() => { + tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-deep-')); + }); + afterEach(() => { + fs.rmSync(tmp, { recursive: true, force: true }); + }); + + /** + * Run the kernel's raw extraction for `file` inside a Worker with Node's + * DEFAULT resourceLimits — exactly how ParseWorkerPool runs it. Resolves + * with the worker's exit code and what it reported; a native overflow + * would SIGSEGV/SIGILL this whole vitest process instead. + */ + function runInWorker(file: string, source: string, language: Language): Promise<{ exitCode: number; outcome: string }> { + const script = path.join(tmp, 'worker.cjs'); + fs.writeFileSync( + script, + [ + `const { parentPort, workerData } = require('worker_threads');`, + `const { tryKernelExtractRaw } = require(${JSON.stringify(DIST_KERNEL)});`, + `const raw = tryKernelExtractRaw(workerData.file, workerData.source, workerData.language);`, + `parentPort.postMessage(raw ? 'kernel:' + raw.counts.nodes : 'deferred');`, + ].join('\n') + ); + return new Promise((resolve, reject) => { + let outcome = 'no message'; + const w = new Worker(script, { workerData: { file, source, language } }); + w.on('message', (m: string) => { outcome = m; }); + w.on('error', reject); + w.on('exit', (exitCode) => resolve({ exitCode, outcome })); + }); + } + + it("defers the reporter's deep.c instead of crashing the worker", async () => { + const r = await runInWorker('deep.c', deepBraces(BRACES_DEPTH), 'c'); + expect(r.exitCode).toBe(0); + expect(r.outcome).toBe('deferred'); + }, 60_000); + + it('defers a 60k-deep expression in every default-routed language', async () => { + for (const language of routed) { + const r = await runInWorker(`deep.${EXT[language]}`, deepParens(language, PARENS_DEPTH), language); + expect(r.exitCode, language).toBe(0); + // Either the guard tripped (deferred) or the walk fit — never a crash. + expect(['deferred', 'kernel'].some((p) => r.outcome.startsWith(p)), `${language}: ${r.outcome}`).toBe(true); + } + }, 180_000); + + it('still extracts a normal file natively in the worker', async () => { + const r = await runInWorker('ok.c', 'int add(int a, int b) { return a + b; }\n', 'c'); + expect(r.exitCode).toBe(0); + expect(r.outcome).toMatch(/^kernel:/); + }, 30_000); + }); + + describe.skipIf(!distBuilt)('end-to-end: codegraph init on a repo holding the deep file', () => { + let tmp: string; + beforeEach(() => { + tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-deep-cli-')); + }); + afterEach(() => { + fs.rmSync(tmp, { recursive: true, force: true }); + }); + + it('exits 0 and records deep.c alongside the normal files', () => { + fs.writeFileSync(path.join(tmp, 'deep.c'), deepBraces(BRACES_DEPTH)); + fs.writeFileSync(path.join(tmp, 'ok.c'), 'int add(int a, int b) { return a + b; }\n'); + execFileSync(process.execPath, [BIN, 'init', '.'], { + cwd: tmp, + encoding: 'utf-8', + stdio: ['ignore', 'pipe', 'pipe'], + timeout: 120_000, + env: { + ...process.env, + CODEGRAPH_NO_DAEMON: '1', + CODEGRAPH_WASM_RELAUNCHED: '1', + CODEGRAPH_TELEMETRY: '0', + DO_NOT_TRACK: '1', + CODEGRAPH_NO_PROMPT_HOOK: '1', + }, + }); + const { DatabaseSync } = require('node:sqlite') as typeof import('node:sqlite'); + const db = new DatabaseSync(path.join(tmp, '.codegraph', 'codegraph.db'), { readOnly: true }); + try { + const files = (db.prepare('SELECT path FROM files ORDER BY path').all() as Array<{ path: string }>).map((r) => r.path); + expect(files).toEqual(['deep.c', 'ok.c']); + const fns = (db.prepare("SELECT name FROM nodes WHERE kind = 'function' ORDER BY name").all() as Array<{ name: string }>).map((r) => r.name); + expect(fns).toEqual(['add', 'foo']); + } finally { + db.close(); + } + }, 180_000); + }); +}); + +describe.skipIf(!expectKernel)('kernel presence (CODEGRAPH_KERNEL_EXPECT=1)', () => { + it('the staged .node exists so the deep-nesting suite actually ran', () => { + expect(kernelBuilt).toBe(true); + }); +}); diff --git a/__tests__/kernel-retry-materialize.test.ts b/__tests__/kernel-retry-materialize.test.ts new file mode 100644 index 000000000..23965bd2d --- /dev/null +++ b/__tests__/kernel-retry-materialize.test.ts @@ -0,0 +1,150 @@ +/** + * Kernel results must be DECODED before they are persisted (#1541). + * + * The bulk-index parse workers return kernel extractions as an undecoded + * buffer transport: `nodes`/`edges`/`unresolvedReferences` are EMPTY and the + * real tables ride in `kernelBuffers`. The main loop decodes (or hands the + * buffers to the store worker), but indexAll's retry passes used to store the + * transport as-is — the storage gate passed via `errors.length === 0`, zero + * nodes were inserted, and the file was permanently recorded as + * "(0 symbols)" with the retry counted as a success. Any worker + * crash/timeout whose in-flight file was a kernel-routed language silently + * wiped that file's symbols (issue #1541: v1.5.0 indexes a valid Python file + * as 0 symbols; v1.4.1, pre-kernel, indexed it correctly). + * + * This pins the store boundary: storeExtractionResult must materialize a + * buffer-transport result before persisting, so every caller — including the + * retry passes — stores the real nodes. + * + * Skips when no kernel binary is staged (same gating as the parity suites). + */ +import { describe, it, expect, beforeEach, afterEach } from 'vitest'; +import * as fs from 'node:fs'; +import * as path from 'node:path'; +import * as os from 'node:os'; +import { CodeGraph } from '../src'; +import { initGrammars, loadGrammarsForLanguages } from '../src/extraction/grammars'; +import { tryKernelExtractRaw } from '../src/extraction/kernel'; +import type { ExtractionResult } from '../src/types'; + +const KERNEL_PATH = path.join( + __dirname, + '..', + 'codegraph-kernel', + 'prebuilds', + `${process.platform}-${process.arch}`, + 'codegraph-kernel.node' +); +const kernelBuilt = fs.existsSync(KERNEL_PATH); + +describe.skipIf(!kernelBuilt)('kernel buffer-transport storage (#1541)', () => { + let dir: string; + let cg: CodeGraph; + + beforeEach(async () => { + dir = fs.mkdtempSync(path.join(os.tmpdir(), 'kernel-retry-mat-')); + cg = await CodeGraph.init(dir); + await initGrammars(); + await loadGrammarsForLanguages(['python']); + }); + + afterEach(() => { + cg.destroy(); + fs.rmSync(dir, { recursive: true, force: true }); + }); + + it('storeExtractionResult persists the decoded nodes of a raw kernel result', async () => { + const source = + 'def target_fn(root, mission_path):\n' + + ' return (root, mission_path)\n' + + '\n' + + 'class Adapter:\n' + + ' def adapt(self):\n' + + ' return target_fn(1, 2)\n'; + const filePath = 'adapter.py'; + fs.writeFileSync(path.join(dir, filePath), source); + + // A genuine undecoded transport, exactly as parse-worker builds it. + const raw = tryKernelExtractRaw(filePath, source, 'python'); + expect(raw).not.toBeNull(); + expect(raw!.counts.nodes).toBeGreaterThan(0); + const transport: ExtractionResult = { + nodes: [], + edges: [], + unresolvedReferences: [], + errors: raw!.errors, + durationMs: 0, + kernelBuffers: raw!.buffers, + kernelCounts: raw!.counts, + }; + + const stats = fs.statSync(path.join(dir, filePath)); + const orchestrator = (cg as unknown as { orchestrator: { storeExtractionResult(f: string, c: string, l: string, s: fs.Stats, r: ExtractionResult): Promise } }).orchestrator; + await orchestrator.storeExtractionResult(filePath, source, 'python', stats, transport); + + // The files row must carry the real symbol count, not the transport's + // empty array — a 0 here is the #1541 "(python, 0 symbols)" wipe. + const file = cg.getFile(filePath); + expect(file).not.toBeNull(); + expect(file!.nodeCount).toBe(raw!.counts.nodes); + + // And the nodes themselves must be queryable. + const nodes = cg.getNodesInFile(filePath); + expect(nodes.length).toBe(raw!.counts.nodes); + expect(nodes.map((n) => n.name)).toContain('target_fn'); + expect(nodes.map((n) => n.name)).toContain('Adapter'); + }); +}); + +/** + * Self-heal for rows the released bug already wiped: a files row recorded + * with zero nodes on a symbol-bearing language can only be a #1541 casualty + * (every real extraction stores at least the file node), and its content + * hash matches the on-disk bytes, so hash-based reconciles skip it forever. + * The full-reconcile sync and indexAll now drop such rows so the file + * re-indexes. Kernel-independent — the wipe is simulated at the DB. + */ +describe('zero-node row self-heal (#1541)', () => { + let dir: string; + let cg: CodeGraph; + + beforeEach(async () => { + dir = fs.mkdtempSync(path.join(os.tmpdir(), 'zero-node-heal-')); + fs.writeFileSync( + path.join(dir, 'adapter.py'), + 'def target_fn(root, mission_path):\n' + + ' return (root, mission_path)\n' + + '\n' + + 'class Adapter:\n' + + ' def adapt(self):\n' + + ' return target_fn(1, 2)\n' + ); + cg = await CodeGraph.init(dir); + await cg.indexAll(); + }); + + afterEach(() => { + cg.destroy(); + fs.rmSync(dir, { recursive: true, force: true }); + }); + + it('sync repairs a wiped row even though the content hash is unchanged', async () => { + const before = cg.getFile('adapter.py'); + expect(before).not.toBeNull(); + expect(before!.nodeCount).toBeGreaterThan(0); + + // Simulate the released-v1.5.0 wipe: nodes gone, row says 0 symbols, + // content hash still matching the file on disk. + const db = (cg as unknown as { db: { getDb(): { prepare(sql: string): { run(...args: unknown[]): unknown } } } }).db.getDb(); + db.prepare('DELETE FROM nodes WHERE file_path = ?').run('adapter.py'); + db.prepare('UPDATE files SET node_count = 0 WHERE path = ?').run('adapter.py'); + expect(cg.getFile('adapter.py')!.nodeCount).toBe(0); + + await cg.sync(); + + const after = cg.getFile('adapter.py'); + expect(after).not.toBeNull(); + expect(after!.nodeCount).toBe(before!.nodeCount); + expect(cg.getNodesInFile('adapter.py').map((n) => n.name)).toContain('target_fn'); + }); +}); diff --git a/__tests__/kernel-rustlang-parity.test.ts b/__tests__/kernel-rustlang-parity.test.ts index b6d792e67..07c897b5e 100644 --- a/__tests__/kernel-rustlang-parity.test.ts +++ b/__tests__/kernel-rustlang-parity.test.ts @@ -4,8 +4,8 @@ * Asserts the native walker (codegraph-kernel/src/rustlang.rs) produces the * SAME ExtractionResult as the wasm TreeSitterExtractor — nodes, edges, and * unresolved refs compared as canonicalized multisets — over the checked-in - * torture fixture (torture.rs: impl/trait quirks incl. the - * `impl Trait for Generic` trait-receiver bug, unit-struct skip, phantom + * torture fixture (torture.rs: impl/trait quirks incl. generic / lifetime / + * reference / scoped / generic-trait impl receivers (#1588), unit-struct skip, phantom * const identifiers, use-binding refs incl. nested groups + wildcard-emits- * nothing, chained-call re-encode, turbofish, Rocket route macros body-only, * fn-ref shapes, value-ref shadowing, attribute-broken docstrings, dead-code diff --git a/__tests__/large-corpus-regressions.test.ts b/__tests__/large-corpus-regressions.test.ts new file mode 100644 index 000000000..109032cb8 --- /dev/null +++ b/__tests__/large-corpus-regressions.test.ts @@ -0,0 +1,143 @@ +import { describe, it, expect, beforeEach, afterEach } from 'vitest'; +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; +import CodeGraph from '../src/index'; +import { QueryBuilder } from '../src/db/queries'; + +describe('large-corpus regression fixes', () => { + it('collects a dense unresolved-reference chunk without spreading it onto the V8 stack (#1558)', () => { + const row = { + id: 1, + from_node_id: 'source', + reference_name: 'target', + reference_kind: 'calls', + line: 1, + col: 1, + candidates: null, + file_path: 'dense.c', + language: 'c', + status: 'pending', + name_tail: 'target', + }; + const denseRows = new Array(200_000).fill(row); + const db = { prepare: () => ({ all: () => denseRows }) }; + const queries = new QueryBuilder(db as any); + expect(queries.getUnresolvedReferencesByFiles(['dense.c'])).toHaveLength(200_000); + }); + + it('records an oversized file during a fresh index so sync does not retry it (#1557)', async () => { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-skipped-file-')); + try { + fs.writeFileSync(path.join(dir, 'oversized.py'), 'value = 1\n'.repeat(120_000)); + const cg = await CodeGraph.init(dir, { silent: true }); + const indexed = await cg.indexAll(); + expect(indexed.filesSkipped).toBe(1); + expect(cg.getFiles().find((f) => f.path === 'oversized.py')?.errors?.[0]?.code).toBe('size_exceeded'); + const synced = await cg.sync(); + expect(synced.filesAdded).toBe(0); + cg.close(); + } finally { + fs.rmSync(dir, { recursive: true, force: true }); + } + }); + + it('records an oversized file through the single-file indexing path (#1557)', async () => { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-single-skipped-file-')); + try { + fs.writeFileSync(path.join(dir, 'oversized.py'), 'value = 1\n'.repeat(120_000)); + const cg = await CodeGraph.init(dir, { silent: true }); + const indexed = await cg.indexFiles(['oversized.py']); + expect(indexed.filesSkipped).toBe(1); + expect(cg.getFiles().find((f) => f.path === 'oversized.py')?.errors?.[0]?.code).toBe('size_exceeded'); + const synced = await cg.sync(); + expect(synced.filesAdded).toBe(0); + expect(synced.filesModified).toBe(0); + cg.close(); + } finally { + fs.rmSync(dir, { recursive: true, force: true }); + } + }); +}); + +describe('JSX synthesis language boundary (#1560)', () => { + let dir: string; + beforeEach(() => { dir = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-jsx-gate-')); }); + afterEach(() => { fs.rmSync(dir, { recursive: true, force: true }); }); + + it('does not create jsx-render edges from JSX-looking text in a C-only project', async () => { + fs.writeFileSync( + path.join(dir, 'only.c'), + 'void Foo(void) {}\nvoid parent(void) { const char *s = ""; }\n' + ); + const cg = await CodeGraph.init(dir, { silent: true }); + await cg.indexAll(); + const rows = (cg as any).db.db.prepare( + "SELECT count(*) AS c FROM edges WHERE json_extract(metadata, '$.synthesizedBy') = 'jsx-render'" + ).get() as { c: number }; + cg.close(); + expect(rows.c).toBe(0); + }); + + it('runs for JavaScript while excluding C parents in the same project', async () => { + fs.writeFileSync( + path.join(dir, 'native.c'), + 'void Widget(void) {}\nvoid native_parent(void) { const char *s = ""; }\n' + ); + fs.writeFileSync( + path.join(dir, 'ui.jsx'), + 'export function Widget() { return ; }\nexport function App() { return ; }\n' + ); + const cg = await CodeGraph.init(dir, { silent: true }); + await cg.indexAll(); + const rows = (cg as any).db.db.prepare(` + SELECT source.file_path AS source_file, target.name AS target_name + FROM edges e + JOIN nodes source ON source.id = e.source + JOIN nodes target ON target.id = e.target + WHERE json_extract(e.metadata, '$.synthesizedBy') = 'jsx-render' + `).all() as Array<{ source_file: string; target_name: string }>; + cg.close(); + expect(rows).toContainEqual({ source_file: 'ui.jsx', target_name: 'Widget' }); + expect(rows.some((row) => row.source_file === 'native.c')).toBe(false); + }); +}); + +describe('failure markers vs later real results (#1557 × #1541)', () => { + it('a failure marker never blocks storing a later successful parse of the same bytes', async () => { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-marker-override-')); + try { + const rel = 'flaky.py'; + const content = 'def real_fn():\n return 1\n\nclass RealClass:\n def m(self):\n return 2\n'; + fs.writeFileSync(path.join(dir, rel), content); + const cg = await CodeGraph.init(dir, { silent: true }); + const { initGrammars, loadGrammarsForLanguages } = await import('../src/extraction/grammars'); + await initGrammars(); + await loadGrammarsForLanguages(['python']); + const orch = (cg as any).orchestrator; + const stats = fs.statSync(path.join(dir, rel)); + + // What recordParseFailure persists when a parse worker dies: a marker + // row under the SAME content hash the retry will store with. + await orch.storeExtractionResult(rel, content, 'python', stats, { + nodes: [], edges: [], unresolvedReferences: [], + errors: [{ message: 'Worker exited with code 1', filePath: rel, severity: 'error', code: 'parse_error' }], + durationMs: 0, + }); + expect(cg.getFile(rel)?.nodeCount).toBe(0); + + // The retry pass succeeds with identical bytes — the marker must be + // replaced, not treated as "no changes". + const { extractFromSource } = await import('../src/extraction/tree-sitter'); + const real = extractFromSource(rel, content, 'python'); + expect(real.nodes.length).toBeGreaterThan(0); + await orch.storeExtractionResult(rel, content, 'python', stats, real); + + expect(cg.getFile(rel)?.nodeCount).toBe(real.nodes.length); + expect(cg.getNodesInFile(rel).map((n: { name: string }) => n.name)).toContain('real_fn'); + cg.close(); + } finally { + fs.rmSync(dir, { recursive: true, force: true }); + } + }); +}); diff --git a/__tests__/mcp-daemon.test.ts b/__tests__/mcp-daemon.test.ts index ab7613664..c73ac564c 100644 --- a/__tests__/mcp-daemon.test.ts +++ b/__tests__/mcp-daemon.test.ts @@ -39,6 +39,7 @@ import * as os from 'os'; import * as path from 'path'; import { CodeGraph } from '../src'; import { getDaemonSocketPath } from '../src/mcp/daemon-paths'; +import { CodeGraphPackageVersion } from '../src/mcp/version'; const BIN = path.resolve(__dirname, '../dist/bin/codegraph.js'); @@ -336,6 +337,44 @@ describe('Shared MCP daemon (issue #411)', () => { expect(isAlive(livePid!)).toBe(true); }, 40000); + it('takes over after SIGKILL even when the stale PID has been reused (#1553)', async () => { + const env = { CODEGRAPH_DAEMON_IDLE_TIMEOUT_MS: '30000' }; + const first = spawnServer(tempDir, env); + servers.push(first); + sendInitialize(first.child, `file://${tempDir}`, 1); + await waitFor(() => findResponse(first.stdout, 1), 10000); + await waitFor(() => countListeningLines(realRoot) >= 1, 10000); + const killedPid = readLockPid(realRoot)!; + + process.kill(killedPid, 'SIGKILL'); + expect(await waitProcessExit(killedPid, 8000)).toBe(true); + + // Model OS PID reuse without risking another process: the stale lock now + // names this live vitest worker, but no daemon answers the leftover socket. + fs.writeFileSync( + path.join(realRoot, '.codegraph', 'daemon.pid'), + JSON.stringify({ + pid: process.pid, + version: CodeGraphPackageVersion, + socketPath: getDaemonSocketPath(realRoot), + startedAt: Date.now() - 60_000, + }), + ); + + const second = spawnServer(tempDir, env); + servers.push(second); + sendInitialize(second.child, `file://${tempDir}`, 2); + const response = await waitFor(() => findResponse(second.stdout, 2), 12000); + expect(response.result.serverInfo.name).toBe('codegraph'); + await waitFor(() => countListeningLines(realRoot) >= 2, 10000); + + const replacementPid = readLockPid(realRoot)!; + expect(replacementPid).not.toBe(killedPid); + expect(replacementPid).not.toBe(process.pid); + expect(isAlive(replacementPid)).toBe(true); + expect(isAlive(process.pid)).toBe(true); + }, 50000); + it('proxy falls back to direct mode on a daemon version mismatch', async () => { const net = await import('net'); const sockPath = getDaemonSocketPath(realRoot); diff --git a/__tests__/mcp-subproject-adoption.test.ts b/__tests__/mcp-subproject-adoption.test.ts new file mode 100644 index 000000000..39abac038 --- /dev/null +++ b/__tests__/mcp-subproject-adoption.test.ts @@ -0,0 +1,188 @@ +/** + * MCP workspace sub-project adoption + no-default diagnostics (#1606, #1607). + * + * When an MCP host launches the server from a workspace root whose indexed + * projects live in CHILD directories (a repo container, a monorepo root), the + * upward walk finds nothing. The server now runs the same bounded down-scan + * the front-load hook uses: + * - exactly ONE indexed sub-project → adopted as the session's default; + * - zero or several → no default, but the state is SAID: + * stderr names what was searched/found, and tool calls list the indexed + * sub-projects so the agent can pass one as `projectPath`; + * - non-workspace base (no manifest, no .git) → no scan at all. + * + * Same real-subprocess harness as mcp-roots.test.ts — no mocking. + */ +import { describe, it, expect, beforeEach, afterEach } from 'vitest'; +import { spawn, ChildProcessWithoutNullStreams } from 'child_process'; +import * as fs from 'fs'; +import * as path from 'path'; +import * as os from 'os'; +import { CodeGraph } from '../src'; + +const BIN = path.resolve(__dirname, '../dist/bin/codegraph.js'); + +function spawnServer(cwd: string): ChildProcessWithoutNullStreams { + // --no-watch keeps the test deterministic; CODEGRAPH_NO_DAEMON keeps the + // session in direct mode so no detached daemon outlives the test. + return spawn(process.execPath, [BIN, 'serve', '--mcp', '--no-watch'], { + cwd, + stdio: ['pipe', 'pipe', 'pipe'], + env: { ...process.env, CODEGRAPH_NO_DAEMON: '1', CODEGRAPH_WASM_RELAUNCHED: '1' }, + }) as ChildProcessWithoutNullStreams; +} + +function collectMessages(child: ChildProcessWithoutNullStreams): Array> { + const messages: Array> = []; + let buf = ''; + child.stdout.on('data', (chunk) => { + buf += chunk.toString('utf8'); + let idx; + while ((idx = buf.indexOf('\n')) !== -1) { + const line = buf.slice(0, idx).trim(); + buf = buf.slice(idx + 1); + if (!line) continue; + try { messages.push(JSON.parse(line)); } catch { /* ignore non-JSON */ } + } + }); + return messages; +} + +function collectStderr(child: ChildProcessWithoutNullStreams): { text: () => string } { + let buf = ''; + child.stderr.on('data', (chunk) => { buf += chunk.toString('utf8'); }); + return { text: () => buf }; +} + +function waitForMessage( + messages: ReadonlyArray>, + predicate: (m: Record) => boolean, + timeoutMs: number, +): Promise> { + return new Promise((resolve, reject) => { + const started = Date.now(); + const tick = () => { + const hit = messages.find(predicate); + if (hit) return resolve(hit); + if (Date.now() - started > timeoutMs) { + return reject(new Error(`Timed out. Messages so far: ${JSON.stringify(messages)}`)); + } + setTimeout(tick, 20); + }; + tick(); + }); +} + +function send(child: ChildProcessWithoutNullStreams, msg: object): void { + child.stdin.write(JSON.stringify(msg) + '\n'); +} + +const CLIENT_INFO = { name: 'test', version: '0.0.0' }; + +/** Create ws/ with one source file and an initialized .codegraph/. */ +async function makeIndexedChild(ws: string, name: string): Promise { + const dir = path.join(ws, name); + fs.mkdirSync(dir, { recursive: true }); + fs.writeFileSync(path.join(dir, 'a.ts'), `export function hello_${name}() { return 1; }\n`); + const cg = await CodeGraph.init(dir); + cg.close(); + return dir; +} + +/** initialize (no rootUri, no roots capability) → initialized → codegraph_status. */ +async function driveStatusCall( + child: ChildProcessWithoutNullStreams, + messages: Array>, +): Promise<{ initResult: Record; statusText: string }> { + send(child, { + jsonrpc: '2.0', id: 0, method: 'initialize', + params: { protocolVersion: '2025-11-25', capabilities: {}, clientInfo: CLIENT_INFO }, + }); + const initResult = await waitForMessage(messages, (m) => m.id === 0 && !!m.result, 5000); + send(child, { jsonrpc: '2.0', method: 'notifications/initialized' }); + send(child, { jsonrpc: '2.0', id: 1, method: 'tools/call', params: { name: 'codegraph_status', arguments: {} } }); + const resp = await waitForMessage(messages, (m) => m.id === 1, 10000); + return { initResult, statusText: resp.result.content[0].text as string }; +} + +describe('MCP workspace sub-project adoption (#1606) + no-default diagnostics (#1607)', () => { + let ws: string; + let child: ChildProcessWithoutNullStreams | null = null; + + beforeEach(() => { + ws = fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-mcp-ws-')); + }); + + afterEach(() => { + if (child && !child.killed) { + child.kill('SIGKILL'); + child = null; + } + fs.rmSync(ws, { recursive: true, force: true }); + }); + + it('adopts the single indexed sub-project below a workspace root as the default project', async () => { + fs.mkdirSync(path.join(ws, '.git')); // workspace marker — no manifest needed + await makeIndexedChild(ws, 'service-a'); + + child = spawnServer(ws); + const messages = collectMessages(child); + const stderr = collectStderr(child); + + const { initResult, statusText } = await driveStatusCall(child, messages); + + // The default project works without any projectPath. + expect(statusText).toContain('CodeGraph Status'); + expect(statusText).not.toContain('No CodeGraph project is loaded'); + // The adoption is announced on stderr (#1607 discoverability). + expect(stderr.text()).toContain('adopted the single indexed sub-project'); + expect(stderr.text()).toContain('service-a'); + // Instructions match what the engine adopted: the FULL single-project + // playbook, not the per-project variant. + const instructions = initResult.result.instructions as string; + expect(instructions).not.toContain('per-project; pass projectPath'); + }, 20000); + + it('lists several indexed sub-projects instead of adopting one, in stderr and in tool responses', async () => { + fs.mkdirSync(path.join(ws, '.git')); + await makeIndexedChild(ws, 'service-a'); + await makeIndexedChild(ws, 'service-b'); + + child = spawnServer(ws); + const messages = collectMessages(child); + const stderr = collectStderr(child); + + const { initResult, statusText } = await driveStatusCall(child, messages); + + // No default was adopted — ambiguous — but the state is said, not silent. + expect(statusText).toContain('No CodeGraph project is loaded'); + // Protocol-reachable listing (#1607): the tool response names what IS there. + expect(statusText).toContain('Indexed sub-projects were found below it'); + expect(statusText).toContain('service-a'); + expect(statusText).toContain('service-b'); + expect(statusText).toContain('projectPath'); + // stderr carries the same facts for the host's log. + expect(stderr.text()).toContain('no default project, live sync disabled'); + expect(stderr.text()).toContain('Indexed sub-projects found:'); + // Ambiguous root → per-project instructions variant. + const instructions = initResult.result.instructions as string; + expect(instructions).toContain('per-project; pass projectPath'); + }, 20000); + + it('does not scan below a base that is not a workspace (no manifest, no .git)', async () => { + // NO .git and no manifest at ws — the gate must keep the scan off even + // though an indexed child exists. + await makeIndexedChild(ws, 'service-a'); + + child = spawnServer(ws); + const messages = collectMessages(child); + const stderr = collectStderr(child); + + const { statusText } = await driveStatusCall(child, messages); + + expect(statusText).toContain('No CodeGraph project is loaded'); + expect(statusText).not.toContain('Indexed sub-projects were found below it'); + expect(stderr.text()).toContain('no default project, live sync disabled'); + expect(stderr.text()).not.toContain('Indexed sub-projects found:'); + }, 20000); +}); diff --git a/__tests__/name-lookup-index.test.ts b/__tests__/name-lookup-index.test.ts new file mode 100644 index 000000000..dce2f994c --- /dev/null +++ b/__tests__/name-lookup-index.test.ts @@ -0,0 +1,206 @@ +/** + * Exact-name lookups must seek `idx_nodes_lower_name` + * + * `nodes` carries two name indexes and neither one can serve + * `WHERE name = ? COLLATE NOCASE`: + * + * - `idx_nodes_name` is BINARY-collated, so NOCASE equality can't use it; + * - `idx_nodes_lower_name` is an expression index on `lower(name)`, and the + * planner only matches it against the same expression. + * + * So every exact-name lookup written that way degrades to a full table scan. + * The `LIMIT`s on those queries do not save them: SQLite can only stop early + * once it has produced `LIMIT` rows, and the common cases — a query term that + * is not a symbol at all, or a name with only a handful of definitions — never + * reach it and scan the whole table. + * + * These tests read the planner's own verdict rather than a wall-clock number, + * so they are deterministic and fail loudly if a lookup regresses to a scan. + * `lower(name) = lower(?)` (not a JS-side `.toLowerCase()`) is the required + * form — see the folding-parity test at the bottom for why. + */ + +import { describe, it, expect, beforeAll, afterAll } from 'vitest'; +import * as fs from 'fs'; +import * as path from 'path'; +import * as os from 'os'; +import { DatabaseConnection } from '../src/db'; +import { QueryBuilder } from '../src/db/queries'; +import { SqliteDatabase } from '../src/db/sqlite-adapter'; +import { Node } from '../src/types'; + +function makeNode(id: string, name: string, filePath = 'src/a.ts'): Node { + return { + id, + kind: 'function', + name, + qualifiedName: name, + filePath, + language: 'typescript', + startLine: 1, + endLine: 2, + startColumn: 0, + endColumn: 0, + updatedAt: Date.now(), + }; +} + +/** Wraps a db so every `prepare()` is recorded, then delegates unchanged. */ +function recordingDb(raw: SqliteDatabase): { db: SqliteDatabase; sqls: string[] } { + const sqls: string[] = []; + const db: SqliteDatabase = { + prepare(sql: string) { + sqls.push(sql); + return raw.prepare(sql); + }, + exec: (sql: string) => raw.exec(sql), + pragma: (str: string, options?: { simple?: boolean }) => raw.pragma(str, options), + transaction: (fn: (...args: any[]) => T) => raw.transaction(fn), + close: () => raw.close(), + get open() { + return raw.open; + }, + }; + return { db, sqls }; +} + +/** SQL that filters `nodes` on whole-name equality, in either spelling. */ +function exactNameLookups(sqls: string[]): string[] { + return sqls.filter( + (s) => + /\bFROM\s+nodes\b/i.test(s) && + (/\bname\s*(COLLATE\s+NOCASE\s*)?=\s*\?(\s*COLLATE\s+NOCASE)?/i.test(s) || + /\blower\(name\)\s*=/i.test(s)) + ); +} + +/** The planner's access path for the `nodes` table in a statement. */ +function nodesAccessPath(raw: SqliteDatabase, sql: string): string { + const args = new Array((sql.match(/\?/g) ?? []).length).fill('x'); + const rows = raw.prepare(`EXPLAIN QUERY PLAN ${sql}`).all(...args) as { detail: string }[]; + const detail = rows.map((r) => r.detail).find((d) => /\bnodes\b/.test(d)); + return detail ?? rows.map((r) => r.detail).join(' | '); +} + +describe('exact-name lookups seek idx_nodes_lower_name', () => { + let dir: string; + let conn: DatabaseConnection; + let raw: SqliteDatabase; + + beforeAll(() => { + dir = fs.mkdtempSync(path.join(os.tmpdir(), 'name-lookup-index-')); + conn = DatabaseConnection.initialize(path.join(dir, 'test.db')); + raw = conn.getDb(); + const seed = new QueryBuilder(raw); + + // A corpus wide enough that a scan and a seek can't accidentally agree on + // ordering, with `handleRequest` deliberately rare (2 nodes) — the shape + // the LIMITs never short-circuit on. + const nodes: Node[] = []; + for (let i = 0; i < 300; i++) { + nodes.push(makeNode(`filler-${i}`, `filler${i}Symbol`, `src/pkg${i % 7}/f${i}.ts`)); + } + nodes.push(makeNode('hr-1', 'handleRequest', 'src/server/router.ts')); + nodes.push(makeNode('hr-2', 'HandleRequest', 'src/server/legacy.ts')); + for (const n of nodes) seed.insertNode(n); + }); + + afterAll(() => { + conn.close(); + fs.rmSync(dir, { recursive: true, force: true }); + }); + + it('searchNodes issues its exact-name supplement as an index seek', () => { + const { db, sqls } = recordingDb(raw); + const q = new QueryBuilder(db); + + const results = q.searchNodes('handleRequest'); + expect(results.length).toBeGreaterThan(0); + + const lookups = exactNameLookups(sqls); + // Guard against a vacuous pass: the supplement must actually have run. + expect(lookups.length).toBeGreaterThan(0); + + for (const sql of lookups) { + expect(nodesAccessPath(raw, sql)).toMatch(/SEARCH nodes USING .*idx_nodes_lower_name/); + } + }); + + it('findNodesByExactName issues both of its passes as index seeks', () => { + const { db, sqls } = recordingDb(raw); + const q = new QueryBuilder(db); + + const results = q.findNodesByExactName(['handleRequest']); + expect(results.length).toBeGreaterThan(0); + + const lookups = exactNameLookups(sqls); + // Two passes: the file_path probe and the row fetch. + expect(lookups.length).toBeGreaterThanOrEqual(2); + + for (const sql of lookups) { + expect(nodesAccessPath(raw, sql)).toMatch(/SEARCH nodes USING .*idx_nodes_lower_name/); + } + }); + + it('getNodesByLowerName seeks the index and does not depend on the caller lowering', () => { + const { db, sqls } = recordingDb(raw); + const q = new QueryBuilder(db); + + // Previously this took an already-lowered string on trust: anything with an + // uppercase letter in it silently returned nothing. + expect(q.getNodesByLowerName('handlerequest').map((n) => n.id).sort()).toEqual([ + 'hr-1', + 'hr-2', + ]); + expect(q.getNodesByLowerName('HandleRequest').map((n) => n.id).sort()).toEqual([ + 'hr-1', + 'hr-2', + ]); + expect(q.getNodesByLowerName('HANDLEREQUEST').map((n) => n.id).sort()).toEqual([ + 'hr-1', + 'hr-2', + ]); + + const lookups = exactNameLookups(sqls); + expect(lookups.length).toBeGreaterThan(0); + for (const sql of lookups) { + expect(nodesAccessPath(raw, sql)).toMatch(/SEARCH nodes USING .*idx_nodes_lower_name/); + } + }); + + it('still matches case-insensitively across both call sites', () => { + const q = new QueryBuilder(raw); + + const exact = q.findNodesByExactName(['HANDLEREQUEST']); + expect(exact.map((r) => r.node.id).sort()).toEqual(['hr-1', 'hr-2']); + + const searched = q.searchNodes('HandleRequest'); + const ids = new Set(searched.map((r) => r.node.id)); + expect(ids.has('hr-1')).toBe(true); + expect(ids.has('hr-2')).toBe(true); + }); + + it('folds exactly what COLLATE NOCASE folded — ASCII only', () => { + // SQLite's NOCASE and its `lower()` are both ASCII-only. JavaScript's + // `.toLowerCase()` is not, so lowering the parameter in JS and comparing + // against `lower(name)` would silently stop matching non-ASCII names that + // NOCASE used to match. `lower(?)` keeps both sides on SQLite's rules. + const probe = new QueryBuilder(raw); + probe.insertNode(makeNode('uni-1', 'Ünïcode', 'src/i18n/a.ts')); + + const found = probe.findNodesByExactName(['Ünïcode']); + expect(found.map((r) => r.node.id)).toContain('uni-1'); + + // The mixed-ASCII half still folds, as NOCASE did. + probe.insertNode(makeNode('uni-2', 'Ünïcodeloader', 'src/i18n/b.ts')); + const folded = probe.findNodesByExactName(['ÜnïcodeLOADER']); + expect(folded.map((r) => r.node.id)).toContain('uni-2'); + + // Same rule for the fuzzy-match lookup. Note what this does NOT claim: a + // caller that lowers in JavaScript first still hands over `ünïcode`, which + // is not what SQLite's `lower()` makes of `Ünïcode`, so the gap stays open + // on that side. + expect(probe.getNodesByLowerName('Ünïcode').map((n) => n.id)).toContain('uni-1'); + expect(probe.getNodesByLowerName('ÜnïcodeLOADER').map((n) => n.id)).toContain('uni-2'); + }); +}); diff --git a/__tests__/parse-pool.test.ts b/__tests__/parse-pool.test.ts index 641d24d12..6211481a4 100644 --- a/__tests__/parse-pool.test.ts +++ b/__tests__/parse-pool.test.ts @@ -11,7 +11,7 @@ * parallelism safe. */ import { describe, it, expect } from 'vitest'; -import { ParseWorkerPool, resolveParsePoolSize, resolveParseTimeoutMs, type ParsePoolWorker, type ParseTask } from '../src/extraction/parse-pool'; +import { ParseWorkerPool, resolveParseBudgetMs, resolveParsePoolSize, resolveParseTimeoutMs, type ParsePoolWorker, type ParseTask } from '../src/extraction/parse-pool'; import type { Language, ExtractionResult } from '../src/types'; const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms)); @@ -94,6 +94,17 @@ describe('resolveParseTimeoutMs', () => { }); }); +describe('resolveParseBudgetMs', () => { + it('caps near-limit blob headers at a 20s soft / 60s hard window (#1555)', () => { + expect(resolveParseBudgetMs(10_000, 940_800)).toBe(20_000); + expect(resolveParseBudgetMs(10_000, 857_376)).toBe(20_000); + }); + + it('does not clamp an explicit larger base timeout', () => { + expect(resolveParseBudgetMs(45_000, 940_800)).toBe(45_000); + }); +}); + describe('resolveParsePoolSize', () => { it('treats explicit 0 and 1 as a single worker (the rollback path)', () => { expect(resolveParsePoolSize('0', 8)).toBe(1); diff --git a/__tests__/query-paths.test.ts b/__tests__/query-paths.test.ts new file mode 100644 index 000000000..f1e3c0e2e --- /dev/null +++ b/__tests__/query-paths.test.ts @@ -0,0 +1,238 @@ +/** + * File-path recognition in explore queries (src/search/query-paths.ts). + * + * The originating bug: an agent named two SvelteKit route files by exact path + * (`src/routes/m/projects/[id]/runs/[runId]/+page.svelte`) and the explore + * pipeline shredded them — the seeding tokenizer splits on brackets, so the + * fragments `runId`/`scope` seeded as "named symbols" and headlined the blast + * radius, while FTS admitted every sibling `+page.svelte` off the `page`/`runs` + * fragments. These tests pin the module that stops that: path spans resolve + * against the indexed file list, matching files pin, and the spans leave the + * query. Resolution IS the detector — slash-bearing non-paths stay untouched. + */ +import { describe, it, expect } from 'vitest'; +import { extractQueryPaths, queryMightContainPaths } from '../src/search/query-paths'; + +const INDEX = [ + 'src/routes/m/projects/[id]/runs/[runId]/+page.svelte', + 'src/routes/m/projects/[id]/chat/[scope]/+page.svelte', + 'src/routes/m/projects/[id]/+page.svelte', + 'src/routes/(protected)/chat-window/+page.svelte', + 'src/lib/chat-manager.ts', + 'src/lib/task-runner-manager.ts', + 'src/lib/stores/sqlite-store.ts', + 'src/lib/stores/postgresql-store.ts', + // Kebab-case frontend shapes (the amnisphere extension-less-basename bug): + 'src/components/training-set-page/training-set-page.tsx', + 'src/components/training-set-page/training-set-page-background-images.tsx', + 'src/components/training-set-page/training-set-page.module.scss', + 'src/components/training-set-page/background-image-table.tsx', + 'src/components/modal/add-to-training-set/add-to-training-set.tsx', + 'src/pages/library-page-layout.tsx', + 'src/api/job-manager/backgrounds.ts', + 'src/x/generic-modal.tsx', + 'src/y/generic-modal.tsx', + 'scripts/pre-commit', + 'src/a/user-profile.tsx', + 'src/b/user-profile.tsx', + 'src/c/user-profile.tsx', + 'src/d/user-profile.tsx', +]; + +describe('queryMightContainPaths — the cheap pre-gate', () => { + it('fires on slashes and dotted basenames', () => { + expect(queryMightContainPaths('look at src/lib/chat-manager.ts')).toBe(true); + expect(queryMightContainPaths('look at chat-manager.ts please')).toBe(true); + }); + + it('stays quiet on plain prose and Class.method spans', () => { + expect(queryMightContainPaths('how does the scroll pinning work')).toBe(false); + // `.isPackaged` is 10 chars — past the 8-char extension cap. + expect(queryMightContainPaths('what reads app.isPackaged here')).toBe(false); + }); + + it('fires on extension-less kebab basenames — with or without wrapping', () => { + expect(queryMightContainPaths('background-image-table Source column')).toBe(true); + expect(queryMightContainPaths('the `library-page-layout` wrapper')).toBe(true); + expect(queryMightContainPaths('usage, add-to-training-set.')).toBe(true); + }); + + it('stays quiet on flags, snake_case, and snake-with-a-dash hybrids', () => { + expect(queryMightContainPaths('run it with --no-cache maybe')).toBe(false); + expect(queryMightContainPaths('where is background_image_table used')).toBe(false); + expect(queryMightContainPaths('the foo_bar-baz helper')).toBe(false); + }); +}); + +describe('extractQueryPaths — resolution and stripping', () => { + it('resolves a bracketed SvelteKit path and strips it from the query', () => { + const q = 'auto-scroll logic in src/routes/m/projects/[id]/runs/[runId]/+page.svelte — atBottom tracking'; + const out = extractQueryPaths(q, INDEX); + expect(out.pinnedFiles).toEqual(['src/routes/m/projects/[id]/runs/[runId]/+page.svelte']); + expect(out.strippedQuery).not.toContain('+page.svelte'); + expect(out.strippedQuery).not.toContain('runId'); + expect(out.strippedQuery).toContain('atBottom tracking'); + expect(out.unresolvedPathSpans).toEqual([]); + }); + + it('pins multiple named files in appearance order', () => { + const q = 'compare src/routes/m/projects/[id]/chat/[scope]/+page.svelte and src/routes/m/projects/[id]/runs/[runId]/+page.svelte'; + const out = extractQueryPaths(q, INDEX); + expect(out.pinnedFiles).toEqual([ + 'src/routes/m/projects/[id]/chat/[scope]/+page.svelte', + 'src/routes/m/projects/[id]/runs/[runId]/+page.svelte', + ]); + }); + + it('resolves a (protected) route-group path — parens are path characters', () => { + const out = extractQueryPaths('read src/routes/(protected)/chat-window/+page.svelte', INDEX); + expect(out.pinnedFiles).toEqual(['src/routes/(protected)/chat-window/+page.svelte']); + }); + + it('resolves an absolute path by walking suffixes to the indexed relative path', () => { + const q = 'fix /Users/colby/dev/beads-live-dashboard/src/lib/chat-manager.ts'; + const out = extractQueryPaths(q, INDEX); + expect(out.pinnedFiles).toEqual(['src/lib/chat-manager.ts']); + }); + + it('resolves a unique basename and a partial path', () => { + expect(extractQueryPaths('see chat-manager.ts', INDEX).pinnedFiles) + .toEqual(['src/lib/chat-manager.ts']); + expect(extractQueryPaths('see stores/sqlite-store.ts', INDEX).pinnedFiles) + .toEqual(['src/lib/stores/sqlite-store.ts']); + }); + + it('strips wrapping punctuation and line references', () => { + const out = extractQueryPaths('the bug (see `src/lib/chat-manager.ts:243`).', INDEX); + expect(out.pinnedFiles).toEqual(['src/lib/chat-manager.ts']); + const hash = extractQueryPaths('regression at src/lib/task-runner-manager.ts#L88-L120', INDEX); + expect(hash.pinnedFiles).toEqual(['src/lib/task-runner-manager.ts']); + }); + + it('treats an over-ambiguous basename as unresolved — stripped and reported', () => { + const out = extractQueryPaths('why do all +page.svelte files flash', INDEX); + expect(out.pinnedFiles).toEqual([]); + expect(out.unresolvedPathSpans).toEqual(['+page.svelte']); + expect(out.strippedQuery).toBe('why do all files flash'); + }); + + it('strips and reports a clearly-path-shaped span that matches nothing', () => { + const out = extractQueryPaths('crash in src/routes/gone/missing-page.svelte on load', INDEX); + expect(out.pinnedFiles).toEqual([]); + expect(out.unresolvedPathSpans).toEqual(['src/routes/gone/missing-page.svelte']); + expect(out.strippedQuery).toBe('crash in on load'); + }); + + it('leaves slash-bearing non-paths alone', () => { + const q = 'does gen_server:call/2 block and/or timeout'; + const out = extractQueryPaths(q, INDEX); + expect(out.pinnedFiles).toEqual([]); + expect(out.unresolvedPathSpans).toEqual([]); + expect(out.strippedQuery).toBe(q); + }); + + it('dedupes a path named twice and honors maxPins', () => { + const twice = extractQueryPaths( + 'src/lib/chat-manager.ts wraps src/lib/chat-manager.ts', INDEX, + ); + expect(twice.pinnedFiles).toEqual(['src/lib/chat-manager.ts']); + + const capped = extractQueryPaths( + 'src/lib/chat-manager.ts src/lib/task-runner-manager.ts', INDEX, { maxPins: 1 }, + ); + expect(capped.pinnedFiles).toEqual(['src/lib/chat-manager.ts']); + }); + + it('matches case-insensitively but returns the indexed spelling', () => { + const out = extractQueryPaths('SRC/LIB/CHAT-MANAGER.TS', INDEX); + expect(out.pinnedFiles).toEqual(['src/lib/chat-manager.ts']); + }); + + it('passes through untouched when nothing resolves', () => { + const q = 'plain prose question about scrolling'; + const out = extractQueryPaths(q, INDEX); + expect(out).toEqual({ strippedQuery: q, pinnedFiles: [], unresolvedPathSpans: [] }); + }); +}); + +describe('extractQueryPaths — extension-less kebab basenames', () => { + it('pins the file a bare kebab basename names and consumes the token', () => { + const out = extractQueryPaths('background-image-table Source column', INDEX); + expect(out.pinnedFiles) + .toEqual(['src/components/training-set-page/background-image-table.tsx']); + expect(out.strippedQuery).toBe('Source column'); + expect(out.unresolvedPathSpans).toEqual([]); + }); + + it('resolves with no slash or extension anywhere in the query (session-4 shape)', () => { + const out = extractQueryPaths( + 'TrainingSetPage train modal library-page-layout AddToTrainingSetModal usage', INDEX, + ); + expect(out.pinnedFiles).toEqual(['src/pages/library-page-layout.tsx']); + // Identifier-shaped tokens stay for the named-symbol seeder. + expect(out.strippedQuery).toBe('TrainingSetPage train modal AddToTrainingSetModal usage'); + }); + + it('pins every named file in a mixed dotted + kebab query (session-1 shape)', () => { + const out = extractQueryPaths( + 'add-to-training-set training-set-page-background-images backgrounds.ts background-image-table Source column', + INDEX, + ); + expect(out.pinnedFiles).toEqual([ + // The dotted pass runs first, so the explicit basename pins ahead of the kebabs. + 'src/api/job-manager/backgrounds.ts', + 'src/components/modal/add-to-training-set/add-to-training-set.tsx', + 'src/components/training-set-page/training-set-page-background-images.tsx', + 'src/components/training-set-page/background-image-table.tsx', + ]); + expect(out.strippedQuery).toBe('Source column'); + }); + + it('leaves kebab prose that names no indexed file untouched — and unreported', () => { + const q = 'how does cross-call dedup make explore non-blocking'; + const out = extractQueryPaths(q, INDEX); + expect(out).toEqual({ strippedQuery: q, pinnedFiles: [], unresolvedPathSpans: [] }); + }); + + it('leaves a stem shared by too many files alone — one hot name must not pin half the repo', () => { + const q = 'refactor the user-profile rendering'; + const out = extractQueryPaths(q, INDEX); + expect(out).toEqual({ strippedQuery: q, pinnedFiles: [], unresolvedPathSpans: [] }); + }); + + it('pins all files sharing a stem when within the ambiguity budget', () => { + const out = extractQueryPaths('generic-modal close behavior', INDEX); + expect(out.pinnedFiles).toEqual(['src/x/generic-modal.tsx', 'src/y/generic-modal.tsx']); + }); + + it('matches case-insensitively and through wrapping punctuation', () => { + expect(extractQueryPaths('see `Background-Image-Table`.', INDEX).pinnedFiles) + .toEqual(['src/components/training-set-page/background-image-table.tsx']); + }); + + it('stems drop only the last extension — a kebab token cannot pin a .module.scss sibling', () => { + const out = extractQueryPaths('training-set-page props flow', INDEX); + expect(out.pinnedFiles) + .toEqual(['src/components/training-set-page/training-set-page.tsx']); + }); + + it('pins an extension-less indexed file by its exact name', () => { + expect(extractQueryPaths('what does the pre-commit hook run', INDEX).pinnedFiles) + .toEqual(['scripts/pre-commit']); + }); + + it('skips tokens the dotted pass consumed and dedupes a file named both ways', () => { + const out = extractQueryPaths('src/lib/chat-manager.ts vs chat-manager internals', INDEX); + expect(out.pinnedFiles).toEqual(['src/lib/chat-manager.ts']); + expect(out.strippedQuery).toBe('vs internals'); + }); + + it('explicit paths win the shared maxPins budget over kebab tokens', () => { + const out = extractQueryPaths( + 'background-image-table then src/lib/chat-manager.ts', INDEX, { maxPins: 1 }, + ); + expect(out.pinnedFiles).toEqual(['src/lib/chat-manager.ts']); + // The kebab token was not consumed once the budget was spent — it stays for FTS. + expect(out.strippedQuery).toBe('background-image-table then'); + }); +}); diff --git a/__tests__/refs-by-files-spread.test.ts b/__tests__/refs-by-files-spread.test.ts new file mode 100644 index 000000000..7d93429fb --- /dev/null +++ b/__tests__/refs-by-files-spread.test.ts @@ -0,0 +1,74 @@ +/** + * getUnresolvedReferencesByFiles must survive dense result sets (#1558). + * + * The input file-path list is chunked under SQLite's parameter limit, but the + * ROWS a chunk returns are unbounded — and appending them with + * `rows.push(...chunkRows)` passes every row as a call argument, so a dense + * chunk (a recovery sync re-indexing many files at once, e.g. the #1541 + * self-heal) exceeded V8's argument limit and killed the whole sync with + * "Maximum call stack size exceeded" after the store phase, leaving every + * re-indexed file's references unresolved. Reproduced for real on a + * cpython-stdlib-sized heal (919 files, 234k refs). The append is now a loop; + * this pins it with a result set well past V8's argument ceiling (~124k). + */ +import { describe, it, expect, beforeEach, afterEach } from 'vitest'; +import * as fs from 'node:fs'; +import * as path from 'node:path'; +import * as os from 'node:os'; +import { CodeGraph } from '../src'; +import type { UnresolvedReference } from '../src/types'; + +describe('unresolved-ref loads with dense result sets (#1558)', () => { + let dir: string; + let cg: CodeGraph; + + beforeEach(async () => { + dir = fs.mkdtempSync(path.join(os.tmpdir(), 'refs-spread-')); + fs.writeFileSync(path.join(dir, 'anchor.py'), 'def anchor():\n return 1\n'); + cg = await CodeGraph.init(dir); + await cg.indexAll(); + }); + + afterEach(() => { + cg.destroy(); + fs.rmSync(dir, { recursive: true, force: true }); + }); + + it('returns 200k pending refs from few files without exhausting the call stack', () => { + const queries = (cg as unknown as { + queries: { + insertUnresolvedRefsBatch(refs: UnresolvedReference[]): void; + getUnresolvedReferencesByFiles(paths: string[]): UnresolvedReference[]; + }; + }).queries; + + const FILES = 200; + const TOTAL = 200_000; + const paths: string[] = Array.from({ length: FILES }, (_, i) => `src/f${i}.py`); + // unresolved_refs.from_node_id is FK-constrained — anchor on a real node. + const anchorId = cg.getNodesInFile('anchor.py')[0]!.id; + + const batch: UnresolvedReference[] = []; + for (let i = 0; i < TOTAL; i++) { + batch.push({ + fromNodeId: anchorId, + referenceName: `ref_${i}`, + referenceKind: 'call', + line: (i % 1000) + 1, + column: 0, + filePath: paths[i % FILES]!, + language: 'python', + }); + if (batch.length === 20_000) { + queries.insertUnresolvedRefsBatch(batch); + batch.length = 0; + } + } + if (batch.length > 0) queries.insertUnresolvedRefsBatch(batch); + + // All 200 paths fit in ONE SQLite parameter chunk, so a single query + // returns all 200k rows — the exact shape that blew the argument limit. + const rows = queries.getUnresolvedReferencesByFiles(paths); + expect(rows.length).toBe(TOTAL); + }); +}); diff --git a/__tests__/resolution.test.ts b/__tests__/resolution.test.ts index 637b4a9d0..decaadee5 100644 --- a/__tests__/resolution.test.ts +++ b/__tests__/resolution.test.ts @@ -1119,6 +1119,168 @@ impl Describe for Ctl { fn describe(&self) -> String { "ctl".into() } } ).toBe('interface-impl'); }); + it('qualifies a generic impl by its type, so trait dispatch reaches it and no edge is invented from its body (#1588)', async () => { + // `impl Source for BufSource`: the implementing type parses as a + // generic_type, so the old positional receiver scan picked the TRAIT. + // The impl's `read` was recorded as `Source::read` — unaddressable as + // `BufSource::read` — and, carrying the trait's name, the interface-impl + // synthesizer treated its body (`{ 0 }`, no call at all) as a second + // declaration and gave it a dispatch edge to FileSource's implementation. + fs.writeFileSync( + path.join(tempDir, 'lib.rs'), + `pub trait Source { + fn read(&mut self) -> usize; +} + +pub struct FileSource { pub n: usize } +impl Source for FileSource { + fn read(&mut self) -> usize { self.n } +} + +pub struct BufSource { pub inner: T } +impl Source for BufSource { + fn read(&mut self) -> usize { 0 } +} +` + ); + + cg = await CodeGraph.init(tempDir, { index: true }); + + const methods = cg.getNodesByKind('method'); + const traitDecls = methods.filter((n) => n.qualifiedName === 'Source::read'); + expect(traitDecls, 'only the declaration carries the trait-qualified name').toHaveLength(1); + const traitMethod = traitDecls[0]!; + expect(traitMethod.startLine).toBe(2); + const fileImpl = methods.find((n) => n.qualifiedName === 'FileSource::read'); + const bufImpl = methods.find((n) => n.qualifiedName === 'BufSource::read'); + expect(fileImpl).toBeDefined(); + expect(bufImpl, 'the generic impl is addressable by its type').toBeDefined(); + + const synth = (id: string) => + cg.getOutgoingEdges(id).filter((e) => e.kind === 'calls' && e.provenance === 'heuristic'); + // Dispatch fans out from the declaration to BOTH implementations… + const fromTrait = synth(traitMethod.id); + expect(new Set(fromTrait.map((e) => e.target))).toEqual(new Set([fileImpl!.id, bufImpl!.id])); + for (const e of fromTrait) { + expect( + (e.metadata as { synthesizedBy?: string } | undefined)?.synthesizedBy + ).toBe('interface-impl'); + expect(e.line, 'registered at the declaration, never at an impl body').toBe(2); + } + // …and neither implementation body sprouts a synthesized call of its own. + expect(synth(fileImpl!.id)).toHaveLength(0); + expect(synth(bufImpl!.id)).toHaveLength(0); + }); + + // ── Rust `self..()` receivers (#1585) ─────────────────── + // A Cargo layout (Cargo.toml + src/) so `use crate::…` paths resolve. + function writeRustCrate(root: string, files: Record): void { + fs.writeFileSync( + path.join(root, 'Cargo.toml'), + '[package]\nname = "repro"\nversion = "0.1.0"\nedition = "2021"\n' + ); + fs.mkdirSync(path.join(root, 'src'), { recursive: true }); + for (const [rel, content] of Object.entries(files)) { + fs.writeFileSync(path.join(root, 'src', rel), content); + } + } + const callsFrom = (qualifiedName: string) => { + const from = cg.getNodesByKind('method').find((n) => n.qualifiedName === qualifiedName); + expect(from, qualifiedName).toBeDefined(); + return cg + .getOutgoingEdges(from!.id) + .filter((e) => e.kind === 'calls') + .map((e) => ({ + target: cg.getNode(e.target)?.qualifiedName, + resolvedBy: (e.metadata as { resolvedBy?: string } | undefined)?.resolvedBy, + provenance: e.provenance ?? undefined, // a resolved (non-synthesized) edge stores NULL + })); + }; + + it("resolves `self.field.method()` to the method on the field's declared type, never to the caller itself (#1585)", async () => { + // The issue's repro: `Outer::run` forwards to `Inner::run` through the + // typed field `inner`. The call used to collapse to the bare name `run` + // and exact-match the nearest same-named method — the calling method — + // recording recursion the source does not contain. + writeRustCrate(tempDir, { + 'lib.rs': 'pub mod inner;\npub mod outer;\n', + 'inner.rs': 'pub struct Inner {\n pub n: usize,\n}\n\nimpl Inner {\n pub fn run(&mut self) {\n self.n += 1;\n }\n}\n', + 'outer.rs': 'use crate::inner::Inner;\n\npub struct Outer {\n pub inner: Inner,\n}\n\nimpl Outer {\n pub fn run(&mut self) {\n self.inner.run();\n }\n}\n', + }); + cg = await CodeGraph.init(tempDir, { index: true }); + expect(callsFrom('Outer::run')).toEqual([ + { target: 'Inner::run', resolvedBy: 'instance-method', provenance: undefined }, + ]); + }); + + it('leaves a `self.field.method()` call unresolved when the field type is external, instead of guessing a same-named local method', async () => { + // `its` is a std type with no project node. Before, `self.its.next()` + // became the bare `next`, which exact-matched a local `next` — the + // calling method (self-edge) or the unrelated `Other::next` decoy. + writeRustCrate(tempDir, { + 'lib.rs': + 'pub struct Scanner {\n its: std::vec::IntoIter,\n}\n\nimpl Scanner {\n pub fn next(&mut self) -> Option {\n self.its.next()\n }\n}\n\n' + + 'pub struct Other { pub n: u8 }\nimpl Other {\n pub fn next(&mut self) -> Option {\n None\n }\n}\n', + }); + cg = await CodeGraph.init(tempDir, { index: true }); + expect(callsFrom('Scanner::next')).toEqual([]); + }); + + it('looks through references and owning smart pointers, but not through containers (#1585)', async () => { + // Method-call auto-deref reaches the pointee of `Box`/`&mut`, so those + // fields resolve to `Inner::run`. `Option` does not auto-deref — + // `self.inner.take()` is Option's method, so it must NOT become + // `Inner::take` even though Inner declares a `take` too. + writeRustCrate(tempDir, { + 'lib.rs': + 'pub struct Inner { pub n: usize }\nimpl Inner {\n pub fn run(&mut self) { self.n += 1; }\n pub fn take(&mut self) {}\n}\n\n' + + 'pub struct Boxed { inner: Box }\nimpl Boxed {\n pub fn go(&mut self) { self.inner.run(); }\n}\n\n' + + "pub struct Borrowed<'a> { inner: &'a mut Inner }\nimpl<'a> Borrowed<'a> {\n pub fn go(&mut self) { self.inner.run(); }\n}\n\n" + + 'pub struct Optional { inner: Option }\nimpl Optional {\n pub fn go(&mut self) { self.inner.take(); }\n}\n', + }); + cg = await CodeGraph.init(tempDir, { index: true }); + expect(callsFrom('Boxed::go').map((c) => c.target)).toEqual(['Inner::run']); + expect(callsFrom('Borrowed::go').map((c) => c.target)).toEqual(['Inner::run']); + expect(callsFrom('Optional::go')).toEqual([]); + }); + + it('leaves a call through a generic-typed field unresolved, and keeps genuine `self.method()` recursion (#1585)', async () => { + writeRustCrate(tempDir, { + 'lib.rs': + 'pub struct Inner { pub n: usize }\nimpl Inner {\n pub fn run(&mut self) {}\n}\n\n' + + 'pub struct Holder { item: T }\nimpl Holder {\n pub fn go(&mut self) { self.item.run(); }\n}\n\n' + + 'pub struct Countdown { pub n: usize }\nimpl Countdown {\n pub fn run(&mut self) {\n if self.n > 0 {\n self.n -= 1;\n self.run();\n }\n }\n}\n', + }); + cg = await CodeGraph.init(tempDir, { index: true }); + // `T` names no project type: no edge, and in particular not `Inner::run`. + expect(callsFrom('Holder::go')).toEqual([]); + // A bare `self` receiver is untouched — real recursion stays a self-edge. + expect(callsFrom('Countdown::run').map((c) => c.target)).toEqual(['Countdown::run']); + }); + + it('resolves a trait-object field to the trait method and typed fields to the right implementation (#1585, #1588)', async () => { + // The #1588 repro's second half: `UsesFile::go` / `UsesBuf::go` each + // forward through a typed field, and a `Box` field lands on + // the trait's declaration — from which the interface-impl synthesizer + // fans out to every implementation. + writeRustCrate(tempDir, { + 'lib.rs': + 'pub trait Source {\n fn read(&mut self) -> usize;\n}\n\n' + + 'pub struct FileSource { pub n: usize }\nimpl Source for FileSource {\n fn read(&mut self) -> usize { self.n }\n}\n\n' + + 'pub struct BufSource { pub inner: T }\nimpl Source for BufSource {\n fn read(&mut self) -> usize { 0 }\n}\n\n' + + 'pub struct UsesFile { pub src: FileSource }\nimpl UsesFile {\n pub fn go(&mut self) -> usize { self.src.read() }\n}\n\n' + + 'pub struct UsesBuf { pub src: BufSource }\nimpl UsesBuf {\n pub fn go(&mut self) -> usize { self.src.read() }\n}\n\n' + + 'pub struct UsesDyn { pub src: Box }\nimpl UsesDyn {\n pub fn go(&mut self) -> usize { self.src.read() }\n}\n', + }); + cg = await CodeGraph.init(tempDir, { index: true }); + expect(callsFrom('UsesFile::go').map((c) => c.target)).toEqual(['FileSource::read']); + expect(callsFrom('UsesBuf::go').map((c) => c.target)).toEqual(['BufSource::read']); + expect(callsFrom('UsesDyn::go').map((c) => c.target)).toEqual(['Source::read']); + // …and dispatch continues from the trait declaration to both impls. + const fanOut = callsFrom('Source::read').filter((c) => c.provenance === 'heuristic').map((c) => c.target).sort(); + expect(fanOut).toEqual(['BufSource::read', 'FileSource::read']); + }); + it('records instantiates for C++ stack/brace construction, targeting the class (#1035)', async () => { // `Calculator calc(0)` (direct-init) and `Widget w{1, 2}` (brace-init) // carry the constructor args directly on the declarator — there's no @@ -2913,6 +3075,131 @@ export function callFromImportedFile(): void { }, 30000); }); + describe('Object-literal namespace members (#1573)', () => { + // `export const api = { call() {…}, get: () => {…} }` used as the module's + // API surface: the members are plain functions with bare names inside the + // constant's extent, so `api.call()` resolved to nothing in the defining + // file and to the CONSTANT through an import — zero callers everywhere. + const setup = (files: Record) => { + const tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-1573-')); + for (const [name, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(tmpDir, name)), { recursive: true }); + fs.writeFileSync(path.join(tmpDir, name), content); + } + return tmpDir; + }; + const callersOf = async (cg: CodeGraph, name: string, kind: string, filePath?: string) => { + const target = (await cg.searchNodes(name, { limit: 20 })).find( + (r) => r.node.kind === kind && r.node.name === name && (!filePath || r.node.filePath === filePath) + ); + expect(target).toBeDefined(); + return (await cg.getCallers(target!.node.id)).map((c) => c.node.name).sort(); + }; + + it('resolves same-file and imported calls to the literal member, never to the constant (#1573)', async () => { + const tmpDir = setup({ + 'a.ts': `export const obj = { m() { return 1; } }; +export class C { static s() { return 2; } } +export function sameFileCallers() { return obj.m() + C.s(); } +`, + 'b.ts': `import { obj, C } from "./a"; +export function crossFileCaller() { return obj.m() + C.s(); } +`, + // A same-named top-level function elsewhere must never be chosen. + 'decoy.ts': `export function m() { return 'decoy'; } +`, + }); + try { + const cg = CodeGraph.initSync(tmpDir); + await cg.indexAll(); + + expect(await callersOf(cg, 'm', 'function', 'a.ts')).toEqual(['crossFileCaller', 'sameFileCallers']); + expect(await callersOf(cg, 'm', 'function', 'decoy.ts')).toEqual([]); + // The class static next to it resolves exactly as before (#825). + expect(await callersOf(cg, 's', 'method')).toEqual(['crossFileCaller', 'sameFileCallers']); + + // The import edge no longer lands on the constant itself. + const obj = (await cg.searchNodes('obj', { limit: 5 })).find((r) => r.node.kind === 'constant'); + expect(obj).toBeDefined(); + const caller = (await cg.searchNodes('crossFileCaller', { limit: 5 })).find((r) => r.node.kind === 'function'); + const toConstant = cg + .getOutgoingEdges(caller!.node.id) + .filter((e) => e.kind === 'calls' && e.target === obj!.node.id); + expect(toConstant).toHaveLength(0); + cg.close(); + } finally { + fs.rmSync(tmpDir, { recursive: true, force: true }); + } + }, 30000); + + it('covers method and arrow-property members, and skips a declaration nested in a member body', async () => { + const tmpDir = setup({ + 'src/api.ts': `export const api = { + call: () => { return 1; }, + get() { + function call() { return 'nested in get, not a member'; } + return call(); + }, +}; +`, + 'src/use.ts': `import { api } from './api'; +export function useCall() { return api.call(); } +export function useGet() { return api.get(); } +`, + }); + try { + const cg = CodeGraph.initSync(tmpDir); + await cg.indexAll(); + + const calls = (await cg.searchNodes('call', { limit: 20 })) + .map((r) => r.node) + .filter((n) => n.name === 'call' && n.filePath === 'src/api.ts' && (n.kind === 'function' || n.kind === 'method')); + // The member is the arrow on line 2; the nested declaration sits + // inside `get`'s body on line 4 and must never be taken for it. + const member = calls.find((n) => n.startLine === 2); + const nested = calls.find((n) => n.startLine === 4); + expect(member).toBeDefined(); + expect(nested).toBeDefined(); + expect((await cg.getCallers(member!.id)).map((c) => c.node.name)).toContain('useCall'); + expect((await cg.getCallers(nested!.id)).map((c) => c.node.name)).not.toContain('useCall'); + expect(await callersOf(cg, 'get', 'function')).toEqual(['useGet']); + cg.close(); + } finally { + fs.rmSync(tmpDir, { recursive: true, force: true }); + } + }, 30000); + + it('leaves a non-literal value receiver on its existing path', async () => { + const tmpDir = setup({ + 'src/mk.ts': `export function m() { return 'top-level, unrelated to obj'; } +export const obj = makeObj(); +export function makeObj(): { m(): number } { return { m: () => 1 } as { m(): number }; } +export function localUse() { return obj.m(); } +`, + 'src/use.ts': `import { obj } from './mk'; +export function remoteUse() { return obj.m(); } +`, + }); + try { + const cg = CodeGraph.initSync(tmpDir); + await cg.indexAll(); + // `obj` holds a call result, not a literal: the same-named top-level + // `m` lies outside its declaration, so containment finds nothing and + // both calls keep today's behavior (unresolved in the defining file; + // the constant edge through the import) rather than guessing. + expect(await callersOf(cg, 'm', 'function')).toEqual([]); + const obj = (await cg.searchNodes('obj', { limit: 5 })).find((r) => r.node.kind === 'constant'); + const remote = (await cg.searchNodes('remoteUse', { limit: 5 })).find((r) => r.node.kind === 'function'); + expect( + cg.getOutgoingEdges(remote!.node.id).some((e) => e.kind === 'calls' && e.target === obj!.node.id) + ).toBe(true); + cg.close(); + } finally { + fs.rmSync(tmpDir, { recursive: true, force: true }); + } + }, 30000); + }); + describe('C++ namespace-qualified static method calls to out-of-line definitions (#1291)', () => { // The issue's exact shape: nested types + out-of-line static method // definition inside `namespace simulator { }` in the .cpp, called via the diff --git a/__tests__/sync.test.ts b/__tests__/sync.test.ts index f26c05e1f..3ce6a0a86 100644 --- a/__tests__/sync.test.ts +++ b/__tests__/sync.test.ts @@ -149,6 +149,28 @@ describe('Sync Module', () => { expect(result.filesRemoved).toBe(0); expect(result.filesChecked).toBeGreaterThan(0); }); + + it('persists an oversized skipped file so later syncs do not retry it (#1557)', async () => { + const filePath = path.join(testDir, 'src', 'oversized.ts'); + fs.writeFileSync(filePath, 'const value = 1;\n'.repeat(70_000)); + + const first = await cg.sync(); + expect(first.filesAdded).toBe(1); + expect(cg.getFiles().find((f) => f.path === 'src/oversized.ts')?.errors?.[0]?.code).toBe('size_exceeded'); + + const second = await cg.sync(); + expect(second.filesAdded).toBe(0); + expect(second.filesModified).toBe(0); + }); + + it('marks a successfully recovered indexing state complete (#1556)', async () => { + (cg as any).queries.setMetadata('index_state', 'indexing'); + await cg.sync({ paths: ['src/index.ts'] }); + expect(cg.getIndexState()).toBe('indexing'); + + await cg.sync(); + expect(cg.getIndexState()).toBe('complete'); + }); }); }); @@ -829,4 +851,33 @@ describe('Scoped sync parity (#watcher-scoped)', () => { // b.ts untouched and still present expect(cg.searchNodes('beta').length).toBeGreaterThan(0); }); + + it('a scoped path that codegraph.json now excludes is removed, never re-parsed (#1590)', async () => { + // The daemon's watcher hands sync the exact edited path. If the project's + // scope changed underneath it, that path must be treated the way the full + // scan treats it — out of scope, hence gone — never parsed on trust. + const cfg = path.join(testDir, 'codegraph.json'); + fs.writeFileSync(cfg, JSON.stringify({ exclude: ['src/b.ts'] })); + fs.writeFileSync(path.join(testDir, 'src', 'b.ts'), `export function beta() { return 2; }\nexport function gamma() { return 3; }`); + const scoped = await cg.sync({ paths: ['src/b.ts'] }); + expect(scoped.filesRemoved).toBe(1); + expect(scoped.filesModified).toBe(0); + expect(scoped.filesAdded).toBe(0); + expect(cg.searchNodes('gamma').length).toBe(0); + expect(cg.searchNodes('beta').filter((r) => r.node.filePath === 'src/b.ts').length).toBe(0); + // Idempotent: the file stays out on a repeat scoped sync. + const again = await cg.sync({ paths: ['src/b.ts'] }); + expect(again.filesRemoved).toBe(0); + expect(again.filesAdded).toBe(0); + + // Dropping the exclude readmits it through the same scoped path. The + // scope matcher is mtime-keyed, so give the rewrite a distinct mtime even + // on a coarse-timestamp filesystem. + fs.writeFileSync(cfg, JSON.stringify({})); + const later = new Date(Date.now() + 5000); + fs.utimesSync(cfg, later, later); + const readmitted = await cg.sync({ paths: ['src/b.ts'] }); + expect(readmitted.filesAdded).toBe(1); + expect(cg.searchNodes('gamma').length).toBe(1); + }); }); diff --git a/__tests__/tsconfig-extends-aliases.test.ts b/__tests__/tsconfig-extends-aliases.test.ts new file mode 100644 index 000000000..8010c0811 --- /dev/null +++ b/__tests__/tsconfig-extends-aliases.test.ts @@ -0,0 +1,182 @@ +/** + * `compilerOptions.paths` behind an `extends` chain (#1534). + * + * Nx-style TypeScript monorepos keep every alias in `tsconfig.base.json` and + * let the root `tsconfig.json` inherit it with a bare `"extends"`. The v1 + * loader read only the root file's own `compilerOptions`, so those repos got + * `null` back — every cross-package import fell through to name-based + * matching, silently, with no unresolved-import warning. + * + * What is locked in here: + * - `paths` is picked up through one and through several `extends` hops + * - `extends` targets resolve as relative paths (with or without `.json`), + * and as `node_modules` package specifiers + * - inherited `paths` resolve against the config that DECLARED them (a base + * config one directory down must not have its targets read as root-relative) + * - an explicit `baseUrl` still wins, and is itself relative to the file that + * declared it + * - the nearest config wins: a child's own `paths` replaces (not merges with) + * the parent's, which is what `tsc` does + * - a cyclic `extends` terminates instead of blowing the stack + */ +import { describe, it, expect, beforeEach, afterEach } from 'vitest'; +import * as fs from 'fs'; +import * as path from 'path'; +import * as os from 'os'; +import { loadProjectAliases, applyAliases } from '../src/resolution/path-aliases'; + +function write(file: string, content: unknown): void { + fs.mkdirSync(path.dirname(file), { recursive: true }); + fs.writeFileSync(file, typeof content === 'string' ? content : JSON.stringify(content, null, 2)); +} + +describe('tsconfig `extends` chains (#1534)', () => { + let root: string; + + beforeEach(() => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-tsextends-')); + }); + + afterEach(() => { + fs.rmSync(root, { recursive: true, force: true }); + }); + + it('picks up paths from an extended tsconfig.base.json (Nx layout)', () => { + write(path.join(root, 'tsconfig.base.json'), { + compilerOptions: { + baseUrl: '.', + paths: { '@scope/lib-name': ['libs/lib-name/src/index.ts'], '@scope/*': ['libs/*/src/index.ts'] }, + }, + }); + write(path.join(root, 'tsconfig.json'), { extends: './tsconfig.base.json', compilerOptions: {} }); + + const aliases = loadProjectAliases(root); + expect(aliases).not.toBeNull(); + expect(applyAliases('@scope/lib-name', aliases!, root)).toEqual(['libs/lib-name/src/index.ts']); + expect(applyAliases('@scope/other', aliases!, root)).toEqual(['libs/other/src/index.ts']); + }); + + it('follows a multi-hop chain and accepts an extensionless relative target', () => { + write(path.join(root, 'tsconfig.root.json'), { + compilerOptions: { baseUrl: '.', paths: { '@app/*': ['packages/*/src'] } }, + }); + write(path.join(root, 'tsconfig.mid.json'), { extends: './tsconfig.root' }); + write(path.join(root, 'tsconfig.json'), { extends: './tsconfig.mid.json' }); + + const aliases = loadProjectAliases(root); + expect(applyAliases('@app/ui', aliases!, root)).toEqual(['packages/ui/src']); + }); + + it('resolves an `extends` package specifier through node_modules', () => { + write(path.join(root, 'node_modules/@acme/tsconfig/tsconfig.json'), { + // Anchored at the package's own directory: node_modules/@acme/tsconfig + compilerOptions: { paths: { '@acme/*': ['../../../src/*'] } }, + }); + write(path.join(root, 'tsconfig.json'), { extends: '@acme/tsconfig' }); + + const aliases = loadProjectAliases(root); + expect(applyAliases('@acme/thing', aliases!, root)).toEqual(['src/thing']); + }); + + it('resolves inherited paths against the config that declared them, not the root', () => { + // No baseUrl anywhere: tsc anchors `paths` at the declaring config's own + // directory. Reading `src/*` as root-relative would silently point every + // alias at the wrong tree. + write(path.join(root, 'config/tsconfig.base.json'), { + compilerOptions: { paths: { '~/*': ['src/*'] } }, + }); + write(path.join(root, 'tsconfig.json'), { extends: './config/tsconfig.base.json' }); + + const aliases = loadProjectAliases(root); + expect(applyAliases('~/foo', aliases!, root)).toEqual(['config/src/foo']); + }); + + it('honours an inherited baseUrl relative to the file that declared it', () => { + write(path.join(root, 'config/tsconfig.base.json'), { + compilerOptions: { baseUrl: '..', paths: { '~/*': ['src/*'] } }, + }); + write(path.join(root, 'tsconfig.json'), { extends: './config/tsconfig.base.json' }); + + const aliases = loadProjectAliases(root); + expect(applyAliases('~/foo', aliases!, root)).toEqual(['src/foo']); + }); + + it('lets the nearest config override inherited paths and baseUrl', () => { + write(path.join(root, 'tsconfig.base.json'), { + compilerOptions: { baseUrl: 'base-dir', paths: { '@x/*': ['from-base/*'] } }, + }); + write(path.join(root, 'tsconfig.json'), { + extends: './tsconfig.base.json', + compilerOptions: { baseUrl: 'own-dir', paths: { '@x/*': ['from-child/*'] } }, + }); + + const aliases = loadProjectAliases(root); + expect(applyAliases('@x/y', aliases!, root)).toEqual(['own-dir/from-child/y']); + }); + + it('terminates on a cyclic extends chain and still uses what it reached', () => { + // The paths live INSIDE the cycle, so this only passes if the chain is + // actually walked — and only returns at all if the cycle is cut. + write(path.join(root, 'tsconfig.json'), { extends: './a.json' }); + write(path.join(root, 'a.json'), { + extends: './b.json', + compilerOptions: { paths: { '@cycle/*': ['from-a/*'] } }, + }); + write(path.join(root, 'b.json'), { extends: './a.json' }); + + const aliases = loadProjectAliases(root); + expect(applyAliases('@cycle/x', aliases!, root)).toEqual(['from-a/x']); + }); + + it('falls back to tsconfig.base.json behind a solution-style root config', () => { + // What `nrwl/nx` itself ships: the root tsconfig.json is a project- + // references shell with no `extends` and no `paths`, so following the + // chain from it reaches nothing. The aliases are all in the base. + write(path.join(root, 'tsconfig.base.json'), { + compilerOptions: { baseUrl: '.', paths: { '@scope/*': ['libs/*/src/index.ts'] } }, + }); + write(path.join(root, 'tsconfig.json'), { + compileOnSave: false, + files: [], + include: [], + references: [{ path: './libs/lib-name' }], + }); + + const aliases = loadProjectAliases(root); + expect(aliases).not.toBeNull(); + expect(applyAliases('@scope/lib-name', aliases!, root)).toEqual(['libs/lib-name/src/index.ts']); + }); + + it('falls back to tsconfig.base.json when no root tsconfig.json exists', () => { + // The classic Nx integrated layout: only per-project tsconfigs and a + // base at the root. Nothing to follow an `extends` chain from. + write(path.join(root, 'tsconfig.base.json'), { + compilerOptions: { baseUrl: '.', paths: { '@scope/*': ['libs/*/src/index.ts'] } }, + }); + + const aliases = loadProjectAliases(root); + expect(aliases).not.toBeNull(); + expect(applyAliases('@scope/lib-name', aliases!, root)).toEqual(['libs/lib-name/src/index.ts']); + }); + + it('still prefers the root tsconfig.json when both files carry paths', () => { + // Precedence guard for the fallback: base is consulted only when the + // root config yields nothing. + write(path.join(root, 'tsconfig.base.json'), { + compilerOptions: { baseUrl: '.', paths: { '@x/*': ['from-base/*'] } }, + }); + write(path.join(root, 'tsconfig.json'), { + compilerOptions: { baseUrl: '.', paths: { '@x/*': ['from-root/*'] } }, + }); + + const aliases = loadProjectAliases(root); + expect(applyAliases('@x/y', aliases!, root)).toEqual(['from-root/y']); + }); + + it('still returns null when nothing in the chain declares paths', () => { + write(path.join(root, 'tsconfig.base.json'), { compilerOptions: { strict: true } }); + write(path.join(root, 'tsconfig.json'), { extends: './tsconfig.base.json' }); + + expect(loadProjectAliases(root)).toBeNull(); + }); +}); diff --git a/__tests__/watcher.test.ts b/__tests__/watcher.test.ts index f493a96cc..942fd5bcd 100644 --- a/__tests__/watcher.test.ts +++ b/__tests__/watcher.test.ts @@ -545,6 +545,134 @@ describe('FileWatcher', () => { }); }); + describe('scope config refresh (#1590)', () => { + // The matcher used to be built once in start() and kept for the watcher's + // lifetime, so a `codegraph.json` written AFTER the daemon started was + // invisible to the live watcher while `codegraph sync` honoured it: the + // CLI removed a newly excluded file and the watcher re-added it. + it('a codegraph.json edit rebuilds the matcher and forces a full sync', async () => { + const syncFn = vi.fn().mockResolvedValue({ filesChanged: 0, durationMs: 0 }); + const watcher = newWatcher(syncFn, { debounceMs: 100 }); + watcher.start(); + await watcher.waitUntilReady(); + + // Scope the project after the watcher is already running. + fs.mkdirSync(path.join(testDir, 'skipme')); + fs.writeFileSync(path.join(testDir, 'skipme', 'b.ts'), 'export const b = 1;\n'); + fs.writeFileSync(path.join(testDir, 'codegraph.json'), JSON.stringify({ exclude: ['skipme/'] })); + __emitWatchEventForTests(testDir, 'codegraph.json'); + + // The config edit schedules a FULL sync (no scoped path list): only the + // scan-diff can find the files the new scope drops or admits. + await waitFor(() => syncFn.mock.calls.length > 0); + expect(syncFn.mock.calls.length).toBe(1); + expect(syncFn.mock.calls[0]![0]).toBeUndefined(); + expect(watcher.getPendingFiles()).toEqual([]); + await new Promise((r) => setTimeout(r, 50)); // let runSync settle + + // An edit inside the newly excluded tree is dropped by the LIVE matcher: + // not pending, and no sync scheduled for it. + __emitWatchEventForTests(testDir, 'skipme/b.ts'); + expect(watcher.getPendingFiles().map((p) => p.path)).not.toContain('skipme/b.ts'); + await new Promise((r) => setTimeout(r, 300)); // > debounce + expect(syncFn.mock.calls.length).toBe(1); + + // In-scope edits still sync, scoped to the edited path as before. + __emitWatchEventForTests(testDir, 'src/index.ts'); + await waitFor(() => syncFn.mock.calls.length > 1); + expect(syncFn.mock.calls[1]![0]).toEqual(['src/index.ts']); + + watcher.stop(); + }); + + it('a root .gitignore edit is a scope change too', async () => { + const syncFn = vi.fn().mockResolvedValue({ filesChanged: 0, durationMs: 0 }); + const watcher = newWatcher(syncFn, { debounceMs: 100 }); + watcher.start(); + await watcher.waitUntilReady(); + + fs.mkdirSync(path.join(testDir, 'gen')); + fs.writeFileSync(path.join(testDir, 'gen', 'out.ts'), 'export const g = 1;\n'); + fs.writeFileSync(path.join(testDir, '.gitignore'), 'gen/\n'); + __emitWatchEventForTests(testDir, '.gitignore'); + + await waitFor(() => syncFn.mock.calls.length > 0); + expect(syncFn.mock.calls[0]![0]).toBeUndefined(); + await new Promise((r) => setTimeout(r, 50)); + + __emitWatchEventForTests(testDir, 'gen/out.ts'); + expect(watcher.getPendingFiles().map((p) => p.path)).not.toContain('gen/out.ts'); + await new Promise((r) => setTimeout(r, 300)); + expect(syncFn.mock.calls.length).toBe(1); + + watcher.stop(); + }); + + it('a nested .gitignore inside the scope forces a full sync', async () => { + const syncFn = vi.fn().mockResolvedValue({ filesChanged: 0, durationMs: 0 }); + const watcher = newWatcher(syncFn, { debounceMs: 100 }); + watcher.start(); + await watcher.waitUntilReady(); + + fs.mkdirSync(path.join(testDir, 'sub')); + fs.writeFileSync(path.join(testDir, 'sub', '.gitignore'), 'build/\n'); + __emitWatchEventForTests(testDir, 'sub/.gitignore'); + + await waitFor(() => syncFn.mock.calls.length > 0); + expect(syncFn.mock.calls[0]![0]).toBeUndefined(); + + watcher.stop(); + }); + + it('a .gitignore under an ignored tree (npm install churn) schedules nothing', async () => { + const syncFn = vi.fn().mockResolvedValue({ filesChanged: 0, durationMs: 0 }); + const watcher = newWatcher(syncFn, { debounceMs: 100 }); + watcher.start(); + await watcher.waitUntilReady(); + + fs.mkdirSync(path.join(testDir, 'node_modules', 'pkg'), { recursive: true }); + fs.writeFileSync(path.join(testDir, 'node_modules', 'pkg', '.gitignore'), 'lib/\n'); + __emitWatchEventForTests(testDir, 'node_modules/pkg/.gitignore'); + + await new Promise((r) => setTimeout(r, 300)); + expect(syncFn).not.toHaveBeenCalled(); + + watcher.stop(); + }); + + it('removing the exclude again readmits the tree', async () => { + const syncFn = vi.fn().mockResolvedValue({ filesChanged: 0, durationMs: 0 }); + const watcher = newWatcher(syncFn, { debounceMs: 100 }); + watcher.start(); + await watcher.waitUntilReady(); + + fs.mkdirSync(path.join(testDir, 'skipme')); + fs.writeFileSync(path.join(testDir, 'skipme', 'b.ts'), 'export const b = 1;\n'); + const cfg = path.join(testDir, 'codegraph.json'); + fs.writeFileSync(cfg, JSON.stringify({ exclude: ['skipme/'] })); + __emitWatchEventForTests(testDir, 'codegraph.json'); + await waitFor(() => syncFn.mock.calls.length > 0); + await new Promise((r) => setTimeout(r, 50)); + __emitWatchEventForTests(testDir, 'skipme/b.ts'); + expect(watcher.getPendingFiles().map((p) => p.path)).not.toContain('skipme/b.ts'); + + // Drop the exclude. The loader is mtime-keyed, so make sure the second + // write carries a distinct mtime even on a coarse-timestamp filesystem. + fs.writeFileSync(cfg, JSON.stringify({})); + const later = new Date(Date.now() + 5000); + fs.utimesSync(cfg, later, later); + __emitWatchEventForTests(testDir, 'codegraph.json'); + await waitFor(() => syncFn.mock.calls.length > 1); + expect(syncFn.mock.calls[1]![0]).toBeUndefined(); + await new Promise((r) => setTimeout(r, 50)); + + __emitWatchEventForTests(testDir, 'skipme/b.ts'); + expect(watcher.getPendingFiles().map((p) => p.path)).toContain('skipme/b.ts'); + + watcher.stop(); + }); + }); + describe('pending file tracking (#403)', () => { it('should expose edited paths via getPendingFiles before sync fires', async () => { // Slow debounce — pending entries are visible until the debounce fires. diff --git a/codegraph-kernel/Cargo.lock b/codegraph-kernel/Cargo.lock index 65a70e115..a53782029 100644 --- a/codegraph-kernel/Cargo.lock +++ b/codegraph-kernel/Cargo.lock @@ -47,6 +47,7 @@ name = "codegraph-kernel" version = "0.1.0" dependencies = [ "cc", + "libc", "napi", "napi-build", "napi-derive", diff --git a/codegraph-kernel/Cargo.toml b/codegraph-kernel/Cargo.toml index 3cb15d7a4..f40626ae1 100644 --- a/codegraph-kernel/Cargo.toml +++ b/codegraph-kernel/Cargo.toml @@ -62,6 +62,12 @@ tree-sitter-luau = "=1.2.0" # kotlin grammar C (see build.rs — no kotlin crate dep is possible). tree-sitter-language = "0.1" +# Stack-bounds queries for the walker stack guard (src/stack.rs, #1581): +# pthread_getattr_np / pthread_get_stackaddr_np. Already in the lock file +# transitively; Windows uses a hand-declared kernel32 extern instead. +[target.'cfg(unix)'.dependencies] +libc = "0.2" + [build-dependencies] napi-build = "2" cc = "1" diff --git a/codegraph-kernel/src/ccpp/mod.rs b/codegraph-kernel/src/ccpp/mod.rs index 9f058f495..78877d7da 100644 --- a/codegraph-kernel/src/ccpp/mod.rs +++ b/codegraph-kernel/src/ccpp/mod.rs @@ -776,6 +776,7 @@ impl<'t> Walker<'t> { // --- visitNode ----------------------------------------------------------- fn visit_node(&mut self, node: Node<'t>) { + stack_guard!(); let kind = node.kind(); let mut skip_children = false; @@ -852,6 +853,7 @@ impl<'t> Walker<'t> { // --- extractors ---------------------------------------------------------- fn extract_function(&mut self, node: Node<'t>) { + stack_guard!(); // Receiver present (out-of-line `Cls::method` def) → method instead. if self.variant == Variant::Cpp && self.receiver_type_of(node).is_some() { self.extract_method(node); @@ -892,6 +894,7 @@ impl<'t> Walker<'t> { } fn extract_method(&mut self, node: Node<'t>) { + stack_guard!(); let receiver_type = if self.variant == Variant::Cpp { self.receiver_type_of(node) } else { None }; if !self.inside_class_like() && receiver_type.is_none() { @@ -956,6 +959,7 @@ impl<'t> Walker<'t> { /// extractClass for cpp class_specifier (skipBodilessClass, #1093). fn extract_class(&mut self, node: Node<'t>) { + stack_guard!(); let Some(body) = node.child_by_field_name("body") else { return }; let name = self.extract_name(node); let extra = Extra { @@ -976,6 +980,7 @@ impl<'t> Walker<'t> { /// Extract a struct-like declaration while preserving its semantic kind. fn extract_aggregate(&mut self, node: Node<'t>, kind: &'static str) { + stack_guard!(); let Some(body) = node.child_by_field_name("body") else { return }; let name = self.extract_name(node); let extra = Extra { @@ -995,6 +1000,7 @@ impl<'t> Walker<'t> { } fn extract_enum(&mut self, node: Node<'t>) { + stack_guard!(); let Some(body) = node.child_by_field_name("body") else { return }; let name = self.extract_name(node); let extra = Extra { @@ -1028,6 +1034,7 @@ impl<'t> Walker<'t> { /// extractTypeAlias for type_definition / alias_declaration. Returns true /// when children were consumed (typedef struct/enum bodies). fn extract_type_alias(&mut self, node: Node<'t>) -> bool { + stack_guard!(); let name = self.extract_name(node); if name == "" { return false; @@ -1502,10 +1509,12 @@ impl<'t> Walker<'t> { // --- function bodies ----------------------------------------------------- fn visit_function_body(&mut self, body: Node<'t>) { + stack_guard!(); self.visit_for_calls_and_structure(body); } fn visit_for_calls_and_structure(&mut self, node: Node<'t>) { + stack_guard!(); let kind = node.kind(); self.maybe_capture_fn_refs(node); @@ -1591,6 +1600,7 @@ impl<'t> Walker<'t> { /// grammars: base_class_clause (#1043), the field_declaration Go-embedding /// shape, and the field_declaration_list recursion that reaches it. fn extract_inheritance(&mut self, node: Node<'t>, class_row: u32) { + stack_guard!(); let extends_kind = edge_kind_index("extends").unwrap(); for i in 0..node.named_child_count() { let Some(child) = node.named_child(i) else { continue }; @@ -1703,6 +1713,7 @@ impl<'t> Walker<'t> { /// normalizeValue for cFamilySpec: bare identifiers, and the /// pointer_expression unwrap (`&fn`; `&Cls::m` keeps the qualified name). fn normalize_fn_ref_value(&mut self, v: Node<'t>, from: u32, mode: Mode, explicit_ref: bool, depth: u32) { + stack_guard!(); if depth > 4 { return; } @@ -1749,6 +1760,7 @@ impl<'t> Walker<'t> { /// scanFnRefSubtree: capture-only walk of subtrees the main walkers skip /// (variable-declaration initializers). Halts at nested functions/lambdas. fn scan_fn_ref_subtree(&mut self, node: Node<'t>, depth: u32) { + stack_guard!(); if depth > 12 { return; } diff --git a/codegraph-kernel/src/csharp.rs b/codegraph-kernel/src/csharp.rs index 4bed3a996..f370a0e54 100644 --- a/codegraph-kernel/src/csharp.rs +++ b/codegraph-kernel/src/csharp.rs @@ -500,6 +500,7 @@ impl<'t> Walker<'t> { // --- the dispatcher (visitNode, C#-relevant branches) ----------------------- fn visit_node(&mut self, node: Node<'t>) { + stack_guard!(); let kind = node.kind(); let mut skip_children = false; @@ -571,10 +572,12 @@ impl<'t> Walker<'t> { // --- visitFunctionBody ------------------------------------------------------ fn visit_function_body(&mut self, body: Node<'t>) { + stack_guard!(); self.visit_for_calls_and_structure(body); } fn visit_for_calls_and_structure(&mut self, node: Node<'t>) { + stack_guard!(); let kind = node.kind(); self.maybe_capture_fn_refs(node); @@ -626,6 +629,7 @@ impl<'t> Walker<'t> { // --- extractors -------------------------------------------------------------- fn extract_class(&mut self, node: Node<'t>) { + stack_guard!(); // skipBodilessClass unset: a bodiless `record Empty;` still mints a node. let name = self.extract_name(node); let extra = Extra { @@ -656,6 +660,7 @@ impl<'t> Walker<'t> { } fn extract_struct(&mut self, node: Node<'t>) { + stack_guard!(); // Body gate — EXCEPT C# positional records (`record struct M(…);`, // node type record_declaration), complete definitions with no body. // A bodiless `struct Fwd;` mints NO node. (#831) @@ -685,6 +690,7 @@ impl<'t> Walker<'t> { } fn extract_interface(&mut self, node: Node<'t>) { + stack_guard!(); let name = self.extract_name(node); let extra = Extra { docstring: preceding_docstring(node, self.src), @@ -703,6 +709,7 @@ impl<'t> Walker<'t> { } fn extract_enum(&mut self, node: Node<'t>) { + stack_guard!(); let Some(body) = node.child_by_field_name("body") else { return }; let name = self.extract_name(node); let extra = Extra { @@ -890,6 +897,7 @@ impl<'t> Walker<'t> { /// extractMethod (1737) — method_declaration + constructor_declaration. /// Signature is ALWAYS undefined (no getSignature hook); isAsync is real. fn extract_method(&mut self, node: Node<'t>) { + stack_guard!(); if !self.inside_class_like() { // Unreachable on non-erroring C# (top-level `void M(){}` parses as // local_function_statement; erroring files defer) — mirror the TS @@ -924,6 +932,7 @@ impl<'t> Walker<'t> { /// extractFunction — only reachable for a method outside any class /// (unreachable on non-erroring C#; kept faithful to the generic tail). fn extract_function(&mut self, node: Node<'t>) { + stack_guard!(); let name = self.extract_name(node); if name == "" { if let Some(body) = node.child_by_field_name("body") { @@ -1087,6 +1096,7 @@ impl<'t> Walker<'t> { /// (object initializers are initializer_expression), so this is /// unreachable — mirrored from the shared TS path like java.rs. fn extract_anonymous_class(&mut self, node: Node<'t>, body: Node<'t>) { + stack_guard!(); let type_node = node .child_by_field_name("constructor") .or_else(|| node.child_by_field_name("type")) @@ -1243,6 +1253,7 @@ impl<'t> Walker<'t> { /// walkCsharpTypePosition (5955). fn walk_type_position(&mut self, node: Node<'t>, from_row: u32) { + stack_guard!(); match node.kind() { "predefined_type" => {} "identifier" => { @@ -1362,6 +1373,7 @@ impl<'t> Walker<'t> { /// normalizeValue (function-ref.ts:525) for CSHARP_SPEC: bare identifiers, /// the transparent `argument` layer, and the `this.Member` special. fn normalize_fn_ref_value(&mut self, v: Node<'t>, from: u32, depth: u32) { + stack_guard!(); if depth > 4 { return; } @@ -1412,6 +1424,7 @@ impl<'t> Walker<'t> { } fn scan_fn_ref_subtree(&mut self, node: Node<'t>, depth: u32) { + stack_guard!(); if depth > 12 { return; } diff --git a/codegraph-kernel/src/dart.rs b/codegraph-kernel/src/dart.rs index a0df27ac0..748410b10 100644 --- a/codegraph-kernel/src/dart.rs +++ b/codegraph-kernel/src/dart.rs @@ -596,6 +596,7 @@ impl<'t> Walker<'t> { // --- the main walk (visitNode, tree-sitter.ts:936-1303) --------------- fn visit(&mut self, node: Node<'t>) { + stack_guard!(); // The visitNode hook (dart.ts:144-157) — the constants branch. if node.kind() == "static_final_declaration" { let mut cursor = node.walk(); @@ -669,6 +670,7 @@ impl<'t> Walker<'t> { // --- extractFunction / extractMethod (:1517 / :1737) ------------------ fn extract_function(&mut self, node: Node<'t>) { + stack_guard!(); // No receiver hook. Name first (resolveName inside extract_name). let name = self.extract_name(node); if name == "" { @@ -770,6 +772,7 @@ impl<'t> Walker<'t> { // --- extractClass (:1679) — classes, mixins, extensions --------------- fn extract_class(&mut self, node: Node<'t>) { + stack_guard!(); let resolved_body = self.resolve_body(node); // No skipBodilessClass. Anonymous `extension on String` → the name // fallback finds the ON type's type_identifier — a class named after @@ -800,6 +803,7 @@ impl<'t> Walker<'t> { // --- extractEnum (:1914) ---------------------------------------------- fn extract_enum(&mut self, node: Node<'t>) { + stack_guard!(); let body = match self.resolve_body(node) { Some(b) => b, None => return, @@ -1221,6 +1225,7 @@ impl<'t> Walker<'t> { } fn type_refs_from_subtree(&mut self, node: Node<'t>, from_row: u32) { + stack_guard!(); if node.kind() == "type_identifier" { let name = self.text(node); if !name.is_empty() && !is_builtin_type(name) { @@ -1239,6 +1244,7 @@ impl<'t> Walker<'t> { // --- visitFunctionBody (:5129-5286) — dart rows ----------------------- fn visit_body(&mut self, node: Node<'t>) { + stack_guard!(); self.maybe_capture_fn_refs(node); let kind = node.kind(); @@ -1348,6 +1354,7 @@ impl<'t> Walker<'t> { /// normalizeValue with DART_SPEC's one layer (`argument` → fan out). /// Named arguments are NOT captured (named_argument is not a layer). fn normalize_fn_ref_value(&mut self, v: Node<'t>, from: u32, depth: u32) { + stack_guard!(); if depth > 4 { return; } @@ -1378,6 +1385,7 @@ impl<'t> Walker<'t> { } fn scan_fn_ref_subtree(&mut self, node: Node<'t>, depth: u32) { + stack_guard!(); if depth > 12 { return; } diff --git a/codegraph-kernel/src/go.rs b/codegraph-kernel/src/go.rs index 71081cb6c..66bb352f1 100644 --- a/codegraph-kernel/src/go.rs +++ b/codegraph-kernel/src/go.rs @@ -380,6 +380,7 @@ impl<'t> Walker<'t> { // --- visitNode ------------------------------------------------------------ fn visit_node(&mut self, node: Node<'t>) { + stack_guard!(); let kind = node.kind(); let mut skip_children = false; @@ -417,10 +418,12 @@ impl<'t> Walker<'t> { } fn visit_function_body(&mut self, body: Node<'t>) { + stack_guard!(); self.visit_for_calls_and_structure(body); } fn visit_for_calls_and_structure(&mut self, node: Node<'t>) { + stack_guard!(); let kind = node.kind(); self.maybe_capture_fn_refs(node); @@ -448,6 +451,7 @@ impl<'t> Walker<'t> { // --- extractors -------------------------------------------------------------- fn extract_function(&mut self, node: Node<'t>) { + stack_guard!(); // (getReceiverType only matches method_declaration's receiver field — // function_declaration has none, so no reroute happens here) let name = self.extract_name(node); @@ -524,6 +528,7 @@ impl<'t> Walker<'t> { /// extractTypeAlias for Go: type_spec → struct / interface / plain alias. fn extract_type_alias(&mut self, node: Node<'t>) -> bool { + stack_guard!(); let name = self.extract_name(node); if name == "" { return false; @@ -842,6 +847,7 @@ impl<'t> Walker<'t> { /// (constraint_elem) and struct embedding (field_declaration without a /// field_identifier), plus the field_declaration_list recursion. fn extract_inheritance(&mut self, node: Node<'t>, class_row: u32) { + stack_guard!(); let extends_kind = edge_kind_index("extends").unwrap(); for i in 0..node.named_child_count() { let Some(child) = node.named_child(i) else { continue }; @@ -894,6 +900,7 @@ impl<'t> Walker<'t> { } fn extract_type_refs_from_subtree(&mut self, node: Node<'t>, from_row: u32) { + stack_guard!(); if node.kind() == "type_identifier" { let type_name = self.text(node).to_string(); if !type_name.is_empty() && !is_builtin_type(&type_name) { @@ -980,6 +987,7 @@ impl<'t> Walker<'t> { /// normalizeValue with GO_SPEC's transparent layers (literal_element, /// expression_list — both fan out to named children). fn normalize_fn_ref_value(&mut self, v: Node<'t>, from: u32, depth: u32) { + stack_guard!(); if depth > 4 { return; } @@ -1010,6 +1018,7 @@ impl<'t> Walker<'t> { } fn scan_fn_ref_subtree(&mut self, node: Node<'t>, depth: u32) { + stack_guard!(); if depth > 12 { return; } diff --git a/codegraph-kernel/src/java.rs b/codegraph-kernel/src/java.rs index d2017a093..7065ab565 100644 --- a/codegraph-kernel/src/java.rs +++ b/codegraph-kernel/src/java.rs @@ -485,6 +485,7 @@ impl<'t> Walker<'t> { // --- the dispatcher (visitNode, Java-relevant branches) ----------------------- fn visit_node(&mut self, node: Node<'t>) { + stack_guard!(); let kind = node.kind(); let mut skip_children = false; @@ -534,10 +535,12 @@ impl<'t> Walker<'t> { // --- visitFunctionBody ---------------------------------------------------------- fn visit_function_body(&mut self, body: Node<'t>) { + stack_guard!(); self.visit_for_calls_and_structure(body); } fn visit_for_calls_and_structure(&mut self, node: Node<'t>) { + stack_guard!(); let kind = node.kind(); self.maybe_capture_fn_refs(node); @@ -577,6 +580,7 @@ impl<'t> Walker<'t> { // --- extractors -------------------------------------------------------------- fn extract_class(&mut self, node: Node<'t>) { + stack_guard!(); let name = self.extract_name(node); let extra = Extra { docstring: preceding_docstring(node, self.src), @@ -600,6 +604,7 @@ impl<'t> Walker<'t> { } fn extract_method(&mut self, node: Node<'t>) { + stack_guard!(); if !self.inside_class_like() { // (object-literal parents don't exist in Java; a stray top-level // method extracts as a function, mirroring extractMethod's tail) @@ -627,6 +632,7 @@ impl<'t> Walker<'t> { /// extractFunction — only reachable for a method outside any class. fn extract_function(&mut self, node: Node<'t>) { + stack_guard!(); let name = self.extract_name(node); if name == "" { if let Some(body) = node.child_by_field_name("body") { @@ -653,6 +659,7 @@ impl<'t> Walker<'t> { } fn extract_interface(&mut self, node: Node<'t>) { + stack_guard!(); let name = self.extract_name(node); let extra = Extra { docstring: preceding_docstring(node, self.src), @@ -671,6 +678,7 @@ impl<'t> Walker<'t> { } fn extract_enum(&mut self, node: Node<'t>) { + stack_guard!(); let Some(body) = node.child_by_field_name("body") else { return }; let name = self.extract_name(node); let extra = Extra { @@ -902,6 +910,7 @@ impl<'t> Walker<'t> { /// extractAnonymousClass — `new T() { ... }`. fn extract_anonymous_class(&mut self, node: Node<'t>, body: Node<'t>) { + stack_guard!(); let type_node = node .child_by_field_name("constructor") .or_else(|| node.child_by_field_name("type")) @@ -1101,6 +1110,7 @@ impl<'t> Walker<'t> { } fn extract_type_refs_from_subtree(&mut self, node: Node<'t>, from_row: u32) { + stack_guard!(); if node.kind() == "type_identifier" { let type_name = self.text(node).to_string(); if !type_name.is_empty() && !is_builtin_type(&type_name) { @@ -1193,6 +1203,7 @@ impl<'t> Walker<'t> { } fn scan_fn_ref_subtree(&mut self, node: Node<'t>, depth: u32) { + stack_guard!(); if depth > 12 { return; } diff --git a/codegraph-kernel/src/kotlin.rs b/codegraph-kernel/src/kotlin.rs index ee47d8758..58fde4285 100644 --- a/codegraph-kernel/src/kotlin.rs +++ b/codegraph-kernel/src/kotlin.rs @@ -648,6 +648,7 @@ impl<'t> Walker<'t> { // --- the dispatcher (visitNode, Kotlin-relevant branches) ----------------------- fn visit_node(&mut self, node: Node<'t>) { + stack_guard!(); if self.try_visit_hook(node) { self.scan_fn_ref_subtree(node, 0); return; @@ -719,10 +720,12 @@ impl<'t> Walker<'t> { // --- visitFunctionBody ---------------------------------------------------------- fn visit_function_body(&mut self, body: Node<'t>) { + stack_guard!(); self.visit_for_calls_and_structure(body); } fn visit_for_calls_and_structure(&mut self, node: Node<'t>) { + stack_guard!(); let kind = node.kind(); self.maybe_capture_fn_refs(node); @@ -777,6 +780,7 @@ impl<'t> Walker<'t> { // --- extractors ------------------------------------------------------------------ fn extract_function(&mut self, node: Node<'t>) { + stack_guard!(); // getReceiverType short-circuit (1522) — extension fns at any scope. if self.receiver_type_of(node).is_some() { self.extract_method(node); @@ -810,6 +814,7 @@ impl<'t> Walker<'t> { } fn extract_method(&mut self, node: Node<'t>) { + stack_guard!(); let receiver = self.receiver_type_of(node); let name = self.extract_name(node); let qualified_override = receiver.as_ref().map(|r| format!("{r}::{name}")); @@ -861,6 +866,7 @@ impl<'t> Walker<'t> { } fn extract_class(&mut self, node: Node<'t>) { + stack_guard!(); let resolved_body = self.resolve_body(node); let name = self.extract_name(node); let extra = Extra { @@ -887,6 +893,7 @@ impl<'t> Walker<'t> { } fn extract_interface(&mut self, node: Node<'t>) { + stack_guard!(); let name = self.extract_name(node); let extra = Extra { docstring: preceding_docstring(node, self.src), @@ -905,6 +912,7 @@ impl<'t> Walker<'t> { } fn extract_enum(&mut self, node: Node<'t>) { + stack_guard!(); let Some(body) = self.resolve_body(node) else { return }; let name = self.extract_name(node); let extra = Extra { @@ -1283,6 +1291,7 @@ impl<'t> Walker<'t> { } fn normalize_fn_ref_value(&mut self, v: Node<'t>, from: u32, depth: u32) { + stack_guard!(); if depth > 4 { return; } @@ -1362,6 +1371,7 @@ impl<'t> Walker<'t> { } fn scan_fn_ref_subtree(&mut self, node: Node<'t>, depth: u32) { + stack_guard!(); if depth > 12 { return; } diff --git a/codegraph-kernel/src/lib.rs b/codegraph-kernel/src/lib.rs index d600b0cda..7411fce0f 100644 --- a/codegraph-kernel/src/lib.rs +++ b/codegraph-kernel/src/lib.rs @@ -16,6 +16,20 @@ #![deny(clippy::all)] +/// First statement of every recursive walker function (see stack.rs, #1581): +/// once the stack pointer is inside the red zone, stop descending — the +/// latched flag makes `stack::run_guarded` discard the walk and defer the +/// file to wasm. `Default::default()` covers every walker return type in use +/// (`()`, `bool`, `Option<_>`, `String`); a hook returning `false` just sends +/// its caller down the generic child walk, whose own guard returns at once. +macro_rules! stack_guard { + () => { + if $crate::stack::exhausted() { + return ::core::default::Default::default(); + } + }; +} + mod buffers; mod ccpp; mod cfnptr; @@ -33,6 +47,7 @@ mod rlang; mod ruby; mod rustlang; mod scala; +mod stack; mod swift; mod textutil; mod python; @@ -216,23 +231,28 @@ pub fn cfnptr_strip_c(text: String) -> String { #[napi] pub fn extract_file(file_path: String, content: String, language: String) -> Result { - let out = match language.as_str() { - "java" => java::extract(&file_path, &content).map_err(Error::from_reason)?, - "python" => python::extract(&file_path, &content).map_err(Error::from_reason)?, - "go" => go::extract(&file_path, &content).map_err(Error::from_reason)?, - "c" | "cpp" => ccpp::extract(&file_path, &content, &language).map_err(Error::from_reason)?, - "rust" => rustlang::extract(&file_path, &content).map_err(Error::from_reason)?, - "csharp" => csharp::extract(&file_path, &content).map_err(Error::from_reason)?, - "ruby" => ruby::extract(&file_path, &content).map_err(Error::from_reason)?, - "php" => php::extract(&file_path, &content).map_err(Error::from_reason)?, - "swift" => swift::extract(&file_path, &content).map_err(Error::from_reason)?, - "kotlin" => kotlin::extract(&file_path, &content).map_err(Error::from_reason)?, - "r" => rlang::extract(&file_path, &content).map_err(Error::from_reason)?, - "lua" | "luau" => lua::extract(&file_path, &content, &language).map_err(Error::from_reason)?, - "scala" => scala::extract(&file_path, &content).map_err(Error::from_reason)?, - "dart" => dart::extract(&file_path, &content).map_err(Error::from_reason)?, - _ => tsjs::extract(&file_path, &content, &language).map_err(Error::from_reason)?, - }; + // The whole walk runs under the stack guard (stack.rs, #1581): a file + // nested deeply enough to overflow this thread's stack comes back as a + // `defer:` error — the TS side's routine "take the wasm path" signal — + // instead of a SIGSEGV that kills the entire indexer process. + let out = stack::run_guarded(|| match language.as_str() { + "java" => java::extract(&file_path, &content), + "python" => python::extract(&file_path, &content), + "go" => go::extract(&file_path, &content), + "c" | "cpp" => ccpp::extract(&file_path, &content, &language), + "rust" => rustlang::extract(&file_path, &content), + "csharp" => csharp::extract(&file_path, &content), + "ruby" => ruby::extract(&file_path, &content), + "php" => php::extract(&file_path, &content), + "swift" => swift::extract(&file_path, &content), + "kotlin" => kotlin::extract(&file_path, &content), + "r" => rlang::extract(&file_path, &content), + "lua" | "luau" => lua::extract(&file_path, &content, &language), + "scala" => scala::extract(&file_path, &content), + "dart" => dart::extract(&file_path, &content), + _ => tsjs::extract(&file_path, &content, &language), + }) + .map_err(Error::from_reason)?; Ok(ExtractBuffers { meta: out.meta.into(), nodes: out.nodes.into(), diff --git a/codegraph-kernel/src/lua.rs b/codegraph-kernel/src/lua.rs index f9ab3ab2b..b3cd32a76 100644 --- a/codegraph-kernel/src/lua.rs +++ b/codegraph-kernel/src/lua.rs @@ -418,6 +418,7 @@ impl<'t> Walker<'t> { // --- the main walk (visitNode, tree-sitter.ts:936-1303) --------------- fn visit(&mut self, node: Node<'t>) { + stack_guard!(); let kind = node.kind(); // The visitNode hook (lua.ts:105-151) runs FIRST. @@ -487,6 +488,7 @@ impl<'t> Walker<'t> { // --- extractFunction / extractMethod (1517 / 1737) -------------------- fn extract_function(&mut self, node: Node<'t>) { + stack_guard!(); // :1522 receiver short-circuit IS the method routing. if let Some(receiver) = self.receiver_type(node) { let receiver = receiver.to_string(); @@ -522,6 +524,7 @@ impl<'t> Walker<'t> { } fn extract_method(&mut self, node: Node<'t>, receiver: String) { + stack_guard!(); let name = self.extract_name(node); let docstring = preceding_docstring(node, self.src); let signature = self.signature_of(node); @@ -654,6 +657,7 @@ impl<'t> Walker<'t> { // --- visitFunctionBody (5129-5286) — the hook-free body walk ---------- fn visit_body(&mut self, node: Node<'t>) { + stack_guard!(); // maybeCaptureFnRefs (5137) fires in the body walker too. self.maybe_capture_fn_refs(node); @@ -750,6 +754,7 @@ impl<'t> Walker<'t> { /// normalizeValue with LUA_SPEC's one transparent layer (expression_list /// fans out to named children). fn normalize_fn_ref_value(&mut self, v: Node<'t>, from: u32, depth: u32) { + stack_guard!(); if depth > 4 { return; } @@ -780,6 +785,7 @@ impl<'t> Walker<'t> { } fn scan_fn_ref_subtree(&mut self, node: Node<'t>, depth: u32) { + stack_guard!(); if depth > 12 { return; } diff --git a/codegraph-kernel/src/php.rs b/codegraph-kernel/src/php.rs index 3369e311b..13a8626b6 100644 --- a/codegraph-kernel/src/php.rs +++ b/codegraph-kernel/src/php.rs @@ -530,6 +530,7 @@ impl<'t> Walker<'t> { // --- the dispatcher (visitNode, PHP-relevant branches) ------------------------ fn visit_node(&mut self, node: Node<'t>) { + stack_guard!(); if self.try_visit_hook(node) { self.scan_fn_ref_subtree(node, 0); return; @@ -609,10 +610,12 @@ impl<'t> Walker<'t> { // --- visitFunctionBody -------------------------------------------------------- fn visit_function_body(&mut self, body: Node<'t>) { + stack_guard!(); self.visit_for_calls_and_structure(body); } fn visit_for_calls_and_structure(&mut self, node: Node<'t>) { + stack_guard!(); let kind = node.kind(); self.maybe_capture_fn_refs(node); @@ -671,6 +674,7 @@ impl<'t> Walker<'t> { // --- extractors ---------------------------------------------------------------- fn extract_function(&mut self, node: Node<'t>) { + stack_guard!(); let name = self.extract_name(node); if name == "" { if let Some(body) = node.child_by_field_name("body") { @@ -696,6 +700,7 @@ impl<'t> Walker<'t> { } fn extract_method(&mut self, node: Node<'t>) { + stack_guard!(); let name = self.extract_name(node); let extra = Extra { docstring: preceding_docstring(node, self.src), @@ -715,6 +720,7 @@ impl<'t> Walker<'t> { } fn extract_class(&mut self, node: Node<'t>, kind: &'static str) { + stack_guard!(); let name = self.extract_name(node); let extra = Extra { docstring: preceding_docstring(node, self.src), @@ -736,6 +742,7 @@ impl<'t> Walker<'t> { } fn extract_interface(&mut self, node: Node<'t>) { + stack_guard!(); let name = self.extract_name(node); let extra = Extra { docstring: preceding_docstring(node, self.src), @@ -754,6 +761,7 @@ impl<'t> Walker<'t> { } fn extract_enum(&mut self, node: Node<'t>) { + stack_guard!(); let Some(body) = node.child_by_field_name("body") else { return }; let name = self.extract_name(node); let extra = Extra { @@ -1080,6 +1088,7 @@ impl<'t> Walker<'t> { /// nests inside `anonymous_class`, so findAnonymousClassBody finds no /// DIRECT child) — mirrored from the shared TS path for shape. fn extract_anonymous_class(&mut self, node: Node<'t>, body: Node<'t>) { + stack_guard!(); let type_node = node .child_by_field_name("constructor") .or_else(|| node.child_by_field_name("type")) @@ -1206,6 +1215,7 @@ impl<'t> Walker<'t> { } fn walk_php_type_position(&mut self, node: Node<'t>, from_row: u32) { + stack_guard!(); match node.kind() { "primitive_type" => {} "name" => { @@ -1249,6 +1259,7 @@ impl<'t> Walker<'t> { } fn normalize_fn_ref_value(&mut self, v: Node<'t>, from: u32, depth: u32) { + stack_guard!(); if depth > 4 { return; } @@ -1337,6 +1348,7 @@ impl<'t> Walker<'t> { } fn scan_fn_ref_subtree(&mut self, node: Node<'t>, depth: u32) { + stack_guard!(); if depth > 12 { return; } diff --git a/codegraph-kernel/src/python.rs b/codegraph-kernel/src/python.rs index 93cb10a9d..b2397facd 100644 --- a/codegraph-kernel/src/python.rs +++ b/codegraph-kernel/src/python.rs @@ -320,6 +320,7 @@ impl<'t> Walker<'t> { // --- visitNode ------------------------------------------------------------ fn visit_node(&mut self, node: Node<'t>) { + stack_guard!(); let kind = node.kind(); let mut skip_children = false; @@ -356,10 +357,12 @@ impl<'t> Walker<'t> { } fn visit_function_body(&mut self, body: Node<'t>) { + stack_guard!(); self.visit_for_calls_and_structure(body); } fn visit_for_calls_and_structure(&mut self, node: Node<'t>) { + stack_guard!(); let kind = node.kind(); self.maybe_capture_fn_refs(node); @@ -390,6 +393,7 @@ impl<'t> Walker<'t> { // --- extractors -------------------------------------------------------------- fn extract_function(&mut self, node: Node<'t>) { + stack_guard!(); let name = self.extract_name(node); if name == "" { if let Some(body) = node.child_by_field_name("body") { @@ -414,6 +418,7 @@ impl<'t> Walker<'t> { } fn extract_method(&mut self, node: Node<'t>) { + stack_guard!(); let name = self.extract_name(node); let extra = Extra { docstring: preceding_docstring(node, self.src), @@ -431,6 +436,7 @@ impl<'t> Walker<'t> { } fn extract_class(&mut self, node: Node<'t>) { + stack_guard!(); let name = self.extract_name(node); let extra = Extra { docstring: preceding_docstring(node, self.src), @@ -789,6 +795,7 @@ impl<'t> Walker<'t> { } fn scan_fn_ref_subtree(&mut self, node: Node<'t>, depth: u32) { + stack_guard!(); if depth > 12 { return; } diff --git a/codegraph-kernel/src/rlang.rs b/codegraph-kernel/src/rlang.rs index 04be9dccb..9c85f0cf2 100644 --- a/codegraph-kernel/src/rlang.rs +++ b/codegraph-kernel/src/rlang.rs @@ -298,6 +298,7 @@ impl<'t> Walker<'t> { // else plain recursion over namedChildren in order. fn visit(&mut self, node: Node<'t>) { + stack_guard!(); if self.hook(node) { return; } @@ -313,6 +314,7 @@ impl<'t> Walker<'t> { /// The visitNode hook (r.ts:180-309). Returns true when consumed. fn hook(&mut self, node: Node<'t>) -> bool { + stack_guard!(); match node.kind() { "call" => self.hook_call(node), "binary_operator" => self.hook_binary_operator(node), @@ -321,6 +323,7 @@ impl<'t> Walker<'t> { } fn hook_call(&mut self, node: Node<'t>) -> bool { + stack_guard!(); let fname = match self.callee_name(node) { Some(f) => f, None => return false, @@ -405,6 +408,7 @@ impl<'t> Walker<'t> { } fn hook_binary_operator(&mut self, node: Node<'t>) -> bool { + stack_guard!(); let op = match node.child_by_field_name("operator") { Some(o) => self.text(o), None => return false, @@ -477,6 +481,7 @@ impl<'t> Walker<'t> { /// `list(…)` entries become methods. Non-method argument subtrees are /// NEVER visited (`representation(…)`, `signature(…)` invisible). fn extract_class_members(&mut self, class_call: Node<'t>, class_row: u32) { + stack_guard!(); let args = match class_call.child_by_field_name("arguments") { Some(a) => a, None => return, @@ -543,6 +548,7 @@ impl<'t> Walker<'t> { /// `method` node positioned at the ARGUMENT node, signature from the raw /// parameters text, body walked hook-aware inside the method scope. fn emit_method_arg(&mut self, entry: Node<'t>) { + stack_guard!(); let entry_name = match entry.child_by_field_name("name") { Some(n) => n, None => return, diff --git a/codegraph-kernel/src/ruby.rs b/codegraph-kernel/src/ruby.rs index 18d8407b1..35cd2a713 100644 --- a/codegraph-kernel/src/ruby.rs +++ b/codegraph-kernel/src/ruby.rs @@ -383,6 +383,7 @@ impl<'t> Walker<'t> { /// the source of the module multiply-capture quirk (scan runs with the /// module already POPPED, so candidates re-attribute to the outer scope). fn try_visit_hook(&mut self, node: Node<'t>) -> bool { + stack_guard!(); let kind = node.kind(); if kind == "call" && node.child_by_field_name("receiver").is_none() { if let Some(method) = node.child_by_field_name("method") { @@ -451,6 +452,7 @@ impl<'t> Walker<'t> { // --- the dispatcher (visitNode, Ruby-relevant branches) ---------------------- fn visit_node(&mut self, node: Node<'t>) { + stack_guard!(); // Language hook FIRST (tree-sitter.ts:943) — a handled subtree is // scanned for fn-ref candidates and never reaches the ladder (or the // maybeCaptureFnRefs call below). @@ -522,10 +524,12 @@ impl<'t> Walker<'t> { // --- visitFunctionBody ------------------------------------------------------ fn visit_function_body(&mut self, body: Node<'t>) { + stack_guard!(); self.visit_for_calls_and_structure(body); } fn visit_for_calls_and_structure(&mut self, node: Node<'t>) { + stack_guard!(); let kind = node.kind(); self.maybe_capture_fn_refs(node); @@ -596,6 +600,7 @@ impl<'t> Walker<'t> { // --- extractors -------------------------------------------------------------- fn extract_function(&mut self, node: Node<'t>) { + stack_guard!(); let name = self.extract_name(node); if name == "" { if let Some(body) = node.child_by_field_name("body") { @@ -619,6 +624,7 @@ impl<'t> Walker<'t> { } fn extract_method(&mut self, node: Node<'t>) { + stack_guard!(); let name = self.extract_name(node); let extra = Extra { docstring: preceding_docstring(node, self.src), @@ -634,6 +640,7 @@ impl<'t> Walker<'t> { } fn extract_class(&mut self, node: Node<'t>) { + stack_guard!(); let name = self.extract_name(node); let extra = Extra { docstring: preceding_docstring(node, self.src), @@ -872,6 +879,7 @@ impl<'t> Walker<'t> { /// qualify); `block_argument` is a transparent layer; specials are the /// `method(:sym)` call form and hook-DSL `simple_symbol`s. fn normalize_fn_ref_value(&mut self, v: Node<'t>, from: u32, depth: u32) { + stack_guard!(); if depth > 4 { return; } @@ -938,6 +946,7 @@ impl<'t> Walker<'t> { } fn scan_fn_ref_subtree(&mut self, node: Node<'t>, depth: u32) { + stack_guard!(); if depth > 12 { return; } diff --git a/codegraph-kernel/src/rustlang.rs b/codegraph-kernel/src/rustlang.rs index 16447c243..3e880fa11 100644 --- a/codegraph-kernel/src/rustlang.rs +++ b/codegraph-kernel/src/rustlang.rs @@ -11,10 +11,11 @@ //! - impl blocks push NO scope: members re-dispatch at file scope, so an impl //! associated `const` becomes a FILE-level `variable`, and the method↔owner //! `contains` edge is a source-order name scan (an impl ABOVE its struct -//! gets no edge). `impl Trait for Generic`'s receiver resolves to the -//! TRAIT (the only direct type_identifier), and methods get QN -//! `Trait::method` — preserve, never "fix" via the grammar's trait:/type: -//! fields. +//! gets no edge). The receiver (method QN prefix, `contains` owner, +//! `implements` source) is the impl_item's `type` field via +//! impl_type_name — both sides moved to the grammar's trait:/type: fields +//! together in #1588 (the earlier positional scan qualified every +//! parameterized impl's methods by the TRAIT). //! - `const_item`/`static_item` ride the generic extractVariable fallback: //! kind is always `variable`, no signature, and EVERY direct `identifier` //! child mints a node (`const MAX: u32 = OTHER;` → two nodes, `MAX` + the @@ -22,10 +23,12 @@ //! - Unit structs (`struct Unit;`, no body field) mint NO node; `mod_item` //! mints no module node and adds no QN prefix. //! - Chained-call re-encode is scoped_identifier-gated (`Foo::new().bar()` → -//! `Foo::new().bar`); instance chains, parens, `.await`, 2-hop fields, and -//! `self` receivers all collapse to the bare method name (`self` is node -//! kind `self`, not `identifier`, so it dodges SKIP_RECEIVERS by falling -//! through). Turbofish callees keep the raw `helper::` text. +//! `Foo::new().bar`); a call through a field of the enclosing type keeps +//! the owner-field shape (`self.inner.run()` → `self.inner.run`, #1585); +//! instance chains, parens, `.await`, deeper/non-self field chains, and +//! bare `self` receivers all collapse to the bare method name (`self` is +//! node kind `self`, not `identifier`, so it dodges SKIP_RECEIVERS by +//! falling through). Turbofish callees keep the raw `helper::` text. //! - `use` emits an import node named by the ROOT module (`crate`/`self`/…), //! one root `imports` ref, then one FULL-path `imports` ref per binding; //! `use x::*` (use_wildcard) emits nothing at all. @@ -399,33 +402,35 @@ impl<'t> Walker<'t> { Some(if last == "Self" { "self".to_string() } else { last.to_string() }) } - /// rustExtractor.getReceiverType: parent-walk to the nearest impl_item; - /// LAST direct type_identifier child wins (for `impl Trait for Generic` - /// that's the TRAIT — bug preserved); else the first generic_type's inner - /// type_identifier. + /// rustImplTypeName (languages/rust.ts) — the implementing type's simple + /// name for an impl block, from the grammar's `type` field (#1588): + /// `impl Tr for G` / `impl<'a> Iterator for Parents<'a>` / + /// `impl Tr for &Foo` / `impl Tr for m::Foo` → `G` / `Parents` / `Foo` / + /// `Foo`. Shapes naming no single type (tuple, `dyn Tr`, pointer, + /// primitive, fn type…) → None. Mirrored byte-for-byte — change both. + fn impl_type_name(&self, ty: Option) -> Option { + let ty = ty?; + match ty.kind() { + "type_identifier" | "identifier" => Some(self.text(ty).to_string()), + "generic_type" => self.impl_type_name(ty.child_by_field_name("type")), + "scoped_type_identifier" | "scoped_identifier" => { + self.impl_type_name(ty.child_by_field_name("name")) + } + "reference_type" => self.impl_type_name(ty.child_by_field_name("type")), + _ => None, + } + } + + /// rustExtractor.getReceiverType: parent-walk to the nearest impl_item and + /// read its `type` field (impl_type_name). The pre-#1588 rule took the + /// LAST direct type_identifier child, which for `impl Trait for Generic` + /// was the TRAIT — so every parameterized impl's methods were qualified by + /// the trait. fn receiver_type_of(&self, node: Node) -> Option { let mut parent = node.parent(); while let Some(p) = parent { if p.kind() == "impl_item" { - let type_idents: Vec = (0..p.named_child_count()) - .filter_map(|i| p.named_child(i)) - .filter(|c| c.kind() == "type_identifier") - .collect(); - if let Some(last) = type_idents.last() { - return Some(self.text(*last).to_string()); - } - let generic = (0..p.named_child_count()) - .filter_map(|i| p.named_child(i)) - .find(|c| c.kind() == "generic_type"); - if let Some(g) = generic { - let inner = (0..g.named_child_count()) - .filter_map(|i| g.named_child(i)) - .find(|c| c.kind() == "type_identifier"); - if let Some(inner) = inner { - return Some(self.text(inner).to_string()); - } - } - return None; + return self.impl_type_name(p.child_by_field_name("type")); } parent = p.parent(); } @@ -435,6 +440,7 @@ impl<'t> Walker<'t> { // --- visitNode ------------------------------------------------------------ fn visit_node(&mut self, node: Node<'t>) { + stack_guard!(); let kind = node.kind(); let mut skip_children = false; @@ -498,6 +504,7 @@ impl<'t> Walker<'t> { /// impl method's body, whose parent walk passes through the outer fn) or /// the stack top is class-like (trait members). fn extract_fn_or_method(&mut self, node: Node<'t>) { + stack_guard!(); let receiver = self.receiver_type_of(node); let as_method = receiver.is_some() || self.inside_class_like(); @@ -564,6 +571,7 @@ impl<'t> Walker<'t> { /// extractInterface — kind `trait` (interfaceKind), inheritance from /// trait_bounds, body children visited with the trait pushed. fn extract_interface(&mut self, node: Node<'t>) { + stack_guard!(); let name = self.extract_name(node); let extra = Extra { docstring: preceding_docstring(node, self.src), @@ -584,6 +592,7 @@ impl<'t> Walker<'t> { /// Extract a Rust struct or union with a body; unit structs remain skipped. fn extract_aggregate(&mut self, node: Node<'t>, kind: &'static str) { + stack_guard!(); let Some(body) = node.child_by_field_name("body") else { return }; let name = self.extract_name(node); let extra = Extra { @@ -606,6 +615,7 @@ impl<'t> Walker<'t> { /// extractEnum — body required; enum_variant children → enum_member nodes /// (name field only, payloads never walked); other children re-dispatched. fn extract_enum(&mut self, node: Node<'t>) { + stack_guard!(); let Some(body) = node.child_by_field_name("body") else { return }; let name = self.extract_name(node); let extra = Extra { @@ -700,6 +710,7 @@ impl<'t> Walker<'t> { /// getRootModule (languages/rust.ts:124). fn root_module(&self, n: Node) -> String { + stack_guard!(); let Some(first) = n.named_child(0) else { return self.text(n).to_string(); }; @@ -719,6 +730,7 @@ impl<'t> Walker<'t> { if prefix.is_empty() { seg.to_string() } else { format!("{prefix}::{seg}") } } fn collect<'t>(w: &Walker<'t>, n: Node<'t>, prefix: &str, paths: &mut Vec<(String, Node<'t>)>) { + stack_guard!(); match n.kind() { "identifier" => paths.push((join(prefix, w.text(n)), n)), "scoped_identifier" => { @@ -835,9 +847,29 @@ impl<'t> Walker<'t> { callee_name = method_name.to_string(); } } + "field_expression" => { + // `self..()` — a call through a + // field of the enclosing type (#1585): keep the + // `self.` prefix so the resolver can type the + // field from the owner struct's declaration + // (or leave it unresolved). Any other + // field_expression receiver — a deeper chain, + // a non-self base — keeps the bare name. + let base = r.child_by_field_name("value"); + let field = r.child_by_field_name("field"); + match (base, field) { + (Some(b), Some(f)) + if b.kind() == "self" && f.kind() == "field_identifier" => + { + let field_name = self.text(f); + callee_name = format!("self.{field_name}.{method_name}"); + } + _ => callee_name = method_name.to_string(), + } + } _ => { - // field_expression 2-hop, parenthesized, - // await_expression, `self` — bare method name. + // parenthesized, await_expression, `self` — + // bare method name. callee_name = method_name.to_string(); } } @@ -966,6 +998,7 @@ impl<'t> Walker<'t> { /// every field has a field_identifier), and the field_declaration_list /// recursion that reaches it. fn extract_inheritance(&mut self, node: Node<'t>, class_row: u32) { + stack_guard!(); let extends_kind = edge_kind_index("extends").unwrap(); for i in 0..node.named_child_count() { let Some(child) = node.named_child(i) else { continue }; @@ -1024,36 +1057,19 @@ impl<'t> Walker<'t> { } } - /// extractRustImplItem — `impl Trait for Type` back-reference: positional - /// type-node filter (NEVER the grammar's trait:/type: fields), ≥2 needed, - /// target found by FIRST earlier node of kind struct/enum/class (never - /// trait); ref FROM the type's node, named by the trait's full text. + /// extractRustImplItem — `impl Trait for Type` back-reference from the + /// grammar's `trait` / `type` fields (#1588; an inherent impl has no + /// `trait` field and emits nothing). Target = FIRST earlier node of kind + /// struct/union/enum/class (never trait) named by impl_type_name; ref FROM + /// the type's node, named by the trait's full text (scoped path / generic + /// args kept), at the trait node's position. fn extract_rust_impl_item(&mut self, node: Node<'t>) { - let has_for = (0..node.child_count()) - .filter_map(|i| node.child(i)) - .any(|c| c.kind() == "for" && !c.is_named()); - if !has_for { + let Some(trait_node) = node.child_by_field_name("trait") else { return; - } - let type_idents: Vec = (0..node.named_child_count()) - .filter_map(|i| node.named_child(i)) - .filter(|c| matches!(c.kind(), "type_identifier" | "generic_type" | "scoped_type_identifier")) - .collect(); - if type_idents.len() < 2 { - return; - } - let trait_node = type_idents[0]; - let type_node = type_idents[type_idents.len() - 1]; - + }; let trait_name = self.text(trait_node).to_string(); - let type_name = if type_node.kind() == "generic_type" { - (0..type_node.named_child_count()) - .filter_map(|i| type_node.named_child(i)) - .find(|c| c.kind() == "type_identifier") - .map(|c| self.text(c).to_string()) - .unwrap_or_else(|| self.text(type_node).to_string()) - } else { - self.text(type_node).to_string() + let Some(type_name) = self.impl_type_name(node.child_by_field_name("type")) else { + return; }; let target_row = self @@ -1086,6 +1102,7 @@ impl<'t> Walker<'t> { } fn extract_type_refs_from_subtree(&mut self, node: Node<'t>, from_row: u32) { + stack_guard!(); if node.kind() == "type_identifier" { let type_name = self.text(node).to_string(); if !type_name.is_empty() && !is_builtin_type(&type_name) { @@ -1103,10 +1120,12 @@ impl<'t> Walker<'t> { // --- visitFunctionBody ----------------------------------------------------- fn visit_function_body(&mut self, body: Node<'t>) { + stack_guard!(); self.visit_for_calls_and_structure(body); } fn visit_for_calls_and_structure(&mut self, node: Node<'t>) { + stack_guard!(); let kind = node.kind(); self.maybe_capture_fn_refs(node); @@ -1262,6 +1281,7 @@ impl<'t> Walker<'t> { } fn scan_fn_ref_subtree(&mut self, node: Node<'t>, depth: u32) { + stack_guard!(); if depth > 12 { return; } diff --git a/codegraph-kernel/src/scala.rs b/codegraph-kernel/src/scala.rs index eb7a0bd3d..fc80feae7 100644 --- a/codegraph-kernel/src/scala.rs +++ b/codegraph-kernel/src/scala.rs @@ -471,6 +471,7 @@ impl<'t> Walker<'t> { /// scalaBaseTypeName (tree-sitter.ts:201-224). fn scala_base_type_name(&self, node: Option>) -> Option { + stack_guard!(); let node = node?; match node.kind() { "type_identifier" | "identifier" => Some(self.text(node).to_string()), @@ -495,6 +496,7 @@ impl<'t> Walker<'t> { /// emitScalaTypeRefs (scala.ts:27-45) — the hook's own builtin set. fn emit_scala_type_refs(&mut self, type_node: Node<'t>, from_row: u32) { + stack_guard!(); if type_node.kind() == "type_identifier" { let name = self.text(type_node); if !name.is_empty() && !is_scala_builtin(name) { @@ -529,6 +531,7 @@ impl<'t> Walker<'t> { // --- the main walk (visitNode, tree-sitter.ts:936-1303) --------------- fn visit(&mut self, node: Node<'t>) { + stack_guard!(); // The visitNode hook (scala.ts:131-198) runs FIRST. if self.hook(node) { self.scan_fn_ref_subtree(node, 0); @@ -591,6 +594,7 @@ impl<'t> Walker<'t> { /// The visitNode hook (scala.ts:131-198). Returns true when consumed. fn hook(&mut self, node: Node<'t>) -> bool { + stack_guard!(); match node.kind() { "val_definition" | "var_definition" => { let is_val = node.kind() == "val_definition"; @@ -691,6 +695,7 @@ impl<'t> Walker<'t> { // --- extractMethod → extractFunction routing (:1737 / :1517) ---------- fn extract_method_or_function(&mut self, node: Node<'t>) { + stack_guard!(); // No receiver hook, no methodsAreTopLevel: inside class-like → method, // else → function (the object/object_expression parent check never // matches scala node kinds). @@ -735,6 +740,7 @@ impl<'t> Walker<'t> { // --- extractClass (:1679) — classes, objects, traits ------------------ fn extract_class(&mut self, node: Node<'t>, kind: &'static str) { + stack_guard!(); let resolved_body = node.child_by_field_name("body"); // template_body // No skipBodilessClass — bodiless mints (scala-complete). let name = self.extract_name(node); @@ -765,6 +771,7 @@ impl<'t> Walker<'t> { // --- extractEnum (:1914) ---------------------------------------------- fn extract_enum(&mut self, node: Node<'t>) { + stack_guard!(); let body = match node.child_by_field_name("body") { Some(b) => b, None => return, // bodiless enum mints nothing @@ -812,6 +819,7 @@ impl<'t> Walker<'t> { // --- extractImport (:3170-3236) --------------------------------------- fn extract_import(&mut self, node: Node<'t>) { + stack_guard!(); let import_text = self.text(node).trim(); // extractImport hook (scala.ts:200-211): `path` field is FIRST-MATCH- // WINS → the FIRST dotted segment names the import. @@ -1133,6 +1141,7 @@ impl<'t> Walker<'t> { } fn type_refs_from_subtree(&mut self, node: Node<'t>, from_row: u32) { + stack_guard!(); if node.kind() == "type_identifier" { let name = self.text(node); if !name.is_empty() && !is_builtin_type(name) { @@ -1151,6 +1160,7 @@ impl<'t> Walker<'t> { // --- visitFunctionBody (:5129-5286) — scala rows ---------------------- fn visit_body(&mut self, node: Node<'t>) { + stack_guard!(); self.maybe_capture_fn_refs(node); let kind = node.kind(); @@ -1266,6 +1276,7 @@ impl<'t> Walker<'t> { /// normalizeValue with SCALA_SPEC's unwrap (postfix_expression → first /// named child — eta-expansion `handler _`). No layers. fn normalize_fn_ref_value(&mut self, v: Node<'t>, from: u32, depth: u32) { + stack_guard!(); if depth > 4 { return; } @@ -1294,6 +1305,7 @@ impl<'t> Walker<'t> { } fn scan_fn_ref_subtree(&mut self, node: Node<'t>, depth: u32) { + stack_guard!(); if depth > 12 { return; } diff --git a/codegraph-kernel/src/stack.rs b/codegraph-kernel/src/stack.rs new file mode 100644 index 000000000..e5b044b23 --- /dev/null +++ b/codegraph-kernel/src/stack.rs @@ -0,0 +1,273 @@ +//! Stack-budget guard for the recursive walkers (#1581). +//! +//! Every language walker recurses per AST level (`visit_node` → +//! `visit_for_calls_and_structure` → …). tree-sitter's own parser is +//! iterative, so a pathologically nested file — clang's +//! `parser_overflow.c` nests 16,384 `{`, fuzzer corpora go deeper — parses +//! fine and then overflows the WALKER's native stack. A native overflow is +//! uncatchable: the parse worker is a thread of the `codegraph` process, so +//! the SIGSEGV takes the whole indexer down with no message, no partial +//! index and no per-file fallback. Worker threads get Node's 4 MiB default +//! stack; the main thread's 8 MiB only moves the cliff (100k levels still +//! kill it). +//! +//! The guard turns "about to overflow" into the kernel's existing `defer:` +//! routing signal: `exhausted()` is checked at the top of every recursive +//! walker function (the `stack_guard!` macro in lib.rs), returns `true` once +//! the stack pointer is within `RED_ZONE` of the thread's stack limit, and +//! latches a per-thread flag. `run_guarded` wraps a whole extraction: when +//! the flag is set afterwards the result is discarded and replaced by a +//! `defer:` error, which the TS side (`src/extraction/kernel/index.ts`) +//! already treats as "this file takes the wasm path" — and the wasm walker +//! catches its own JS `RangeError` per file, so the file lands as a partial +//! result with a recorded parse error instead of a dead process. +//! +//! The per-thread stack bounds come from the OS (glibc/musl +//! `pthread_getattr_np`, macOS `pthread_get_stackaddr_np`, Win32 +//! `GetCurrentThreadStackLimits`), computed once per thread and cached, so +//! the guard is exact on the 4 MiB worker, the 8 MiB main thread and any +//! `resourceLimits.stackSizeMb` alike. Where the bounds are unavailable the +//! guard falls back to a fixed descent budget measured from the entry stack +//! pointer. Hot path: one thread-local load and one compare. + +use std::cell::Cell; + +/// Headroom kept free below the deepest walker frame: the napi return path, +/// tree-sitter's node accessors and the error formatting all still need to +/// run after the guard trips, and the frames BETWEEN two guard checks (an +/// `extract_class` between two `visit_node`s) are never more than a few KiB. +const RED_ZONE: usize = 256 * 1024; + +/// Descent budget when the OS can't report the thread's stack bounds — safe +/// on anything from Node's 4 MiB worker default upwards. +const FALLBACK_BUDGET: usize = 1024 * 1024; + +thread_local! { + /// Lowest stack-pointer value the walker may reach before the guard + /// trips. `0` = not computed yet for this thread. + static THRESHOLD: Cell = const { Cell::new(0) }; + /// `true` when the thread's threshold came from real OS bounds (fixed + /// for the thread's lifetime) rather than the per-call fallback budget. + static THRESHOLD_IS_OS: Cell = const { Cell::new(false) }; + /// Latched by `exhausted()`; read by `run_guarded` after the walk. + static OVERFLOWED: Cell = const { Cell::new(false) }; +} + +/// Approximate current stack pointer: the address of a local. Stacks grow +/// downward on every target the kernel ships for (x86_64 / aarch64). +#[inline(always)] +fn current_sp() -> usize { + let marker = 0u8; + std::hint::black_box(&marker) as *const u8 as usize +} + +/// Low (deepest) address of the calling thread's stack, from the OS. +#[cfg(target_os = "linux")] +fn os_stack_low() -> Option { + // SAFETY: plain pthread queries on the calling thread; `attr` is + // initialised by pthread_getattr_np and destroyed before returning. + unsafe { + let mut attr: libc::pthread_attr_t = std::mem::zeroed(); + if libc::pthread_getattr_np(libc::pthread_self(), &mut attr) != 0 { + return None; + } + let mut addr: *mut libc::c_void = std::ptr::null_mut(); + let mut size: libc::size_t = 0; + let rc = libc::pthread_attr_getstack(&attr, &mut addr, &mut size); + libc::pthread_attr_destroy(&mut attr); + if rc != 0 || addr.is_null() || size == 0 { + return None; + } + Some(addr as usize) + } +} + +#[cfg(target_os = "macos")] +fn os_stack_low() -> Option { + // SAFETY: plain pthread queries on the calling thread. + unsafe { + let me = libc::pthread_self(); + // pthread_get_stackaddr_np returns the HIGH end (the stack base). + let high = libc::pthread_get_stackaddr_np(me) as usize; + let size = libc::pthread_get_stacksize_np(me); + if high == 0 || size == 0 || size > high { + return None; + } + Some(high - size) + } +} + +#[cfg(windows)] +fn os_stack_low() -> Option { + #[link(name = "kernel32")] + extern "system" { + // Win8+ (the bundled Node runtime needs Win10 anyway). Reports the + // full RESERVED range; Windows commits pages on demand down to it. + fn GetCurrentThreadStackLimits(low_limit: *mut usize, high_limit: *mut usize); + } + let mut low: usize = 0; + let mut high: usize = 0; + // SAFETY: both out-pointers are valid for the duration of the call. + unsafe { GetCurrentThreadStackLimits(&mut low, &mut high) }; + if low == 0 || high <= low { + return None; + } + Some(low) +} + +#[cfg(not(any(target_os = "linux", target_os = "macos", windows)))] +fn os_stack_low() -> Option { + None +} + +/// Arm the guard for one extraction on the calling thread: clear the latch +/// and (re)compute the threshold. OS bounds are computed once per thread; +/// the fallback budget is re-anchored at every call's entry stack pointer. +pub fn begin() { + OVERFLOWED.with(|o| o.set(false)); + let cached = THRESHOLD.with(|t| t.get()); + if cached != 0 && THRESHOLD_IS_OS.with(|f| f.get()) { + return; + } + match os_stack_low() { + Some(low) => { + THRESHOLD.with(|t| t.set(low.saturating_add(RED_ZONE))); + THRESHOLD_IS_OS.with(|f| f.set(true)); + } + None => { + THRESHOLD.with(|t| t.set(current_sp().saturating_sub(FALLBACK_BUDGET).max(1))); + THRESHOLD_IS_OS.with(|f| f.set(false)); + } + } +} + +/// `true` once the walker has descended to within `RED_ZONE` of the stack +/// limit. Latches `OVERFLOWED` so `run_guarded` can discard the result. A +/// thread that never called `begin()` (a direct unit-test call) arms itself +/// lazily from the current position. +#[inline(always)] +pub fn exhausted() -> bool { + let threshold = THRESHOLD.with(|t| t.get()); + if threshold == 0 { + begin(); + return exhausted(); + } + if current_sp() < threshold { + OVERFLOWED.with(|o| o.set(true)); + true + } else { + false + } +} + +/// Whether the guard tripped since the last `begin()`. +pub fn overflowed() -> bool { + OVERFLOWED.with(|o| o.get()) +} + +/// Run one extraction under the guard. A walk that tripped the guard returns +/// a `defer:` error — the TS side's routine "take the wasm path" signal — +/// regardless of what the truncated walk produced. +pub fn run_guarded(f: impl FnOnce() -> Result) -> Result { + begin(); + let out = f(); + if overflowed() { + return Err( + "defer: nesting too deep for the native walker — wasm recovery handles it".to_string(), + ); + } + out +} + +#[cfg(test)] +mod tests { + use super::*; + + /// 1 MiB is a quarter of Node's worker default; a guard that holds here + /// holds on every real thread. Without the guard these walks SIGSEGV the + /// test process instead of failing an assertion. + const SMALL_STACK: usize = 1 << 20; + const DEPTH: usize = 30_000; + + fn on_small_stack(f: impl FnOnce() -> T + Send + 'static) -> T { + std::thread::Builder::new() + .stack_size(SMALL_STACK) + .spawn(f) + .expect("spawn") + .join() + .expect("walker thread panicked") + } + + fn nested_parens(prefix: &str, suffix: &str) -> String { + format!("{prefix}{}1{}{suffix}", "(".repeat(DEPTH), ")".repeat(DEPTH)) + } + + #[test] + fn os_bounds_are_sane_on_this_platform() { + // Every shipped target has an OS implementation; the fallback budget + // is only for platforms the kernel is not built for. + let low = os_stack_low().expect("OS stack bounds available"); + let sp = current_sp(); + assert!(low < sp, "stack low {low:#x} must be below the current sp {sp:#x}"); + assert!(sp - low < 1 << 31, "implausible stack size {}", sp - low); + } + + #[test] + fn small_stack_reports_its_own_bounds() { + on_small_stack(|| { + let low = os_stack_low().expect("OS stack bounds available"); + let used = current_sp() - low; + // std/the OS round the requested size up a little (macOS reports + // 1,060,864 for a 1 MiB request); the point is that the bounds + // describe THIS thread's small stack, not the main thread's. + assert!( + used <= SMALL_STACK + 128 * 1024, + "used {used} is not within the {SMALL_STACK}-byte stack" + ); + }); + } + + #[test] + fn deep_braces_c_defer_instead_of_crashing() { + let src = format!("void foo(void) {{\n{}{}\n}}\n", "{".repeat(DEPTH), "}".repeat(DEPTH)); + let r = on_small_stack(move || run_guarded(|| crate::ccpp::extract("deep.c", &src, "c"))); + let err = r.err().expect("deep nesting must defer"); + assert!(err.starts_with("defer:"), "unexpected error: {err}"); + } + + type Extract = fn(&str) -> Result; + + #[test] + fn deep_parens_cpp_rust_ts_python_defer_instead_of_crashing() { + let cases: [(&str, Extract, String); 4] = [ + ("deep.cpp", |s| crate::ccpp::extract("deep.cpp", s, "cpp"), nested_parens("int f() { return ", "; }\n")), + ("deep.rs", |s| crate::rustlang::extract("deep.rs", s), nested_parens("fn f() -> i32 { ", " }\n")), + ("deep.ts", |s| crate::tsjs::extract("deep.ts", s, "typescript"), nested_parens("function f() { return ", "; }\n")), + ("deep.py", |s| crate::python::extract("deep.py", s), nested_parens("def f():\n return ", "\n")), + ]; + for (name, extract, src) in cases { + let r = on_small_stack(move || run_guarded(|| extract(&src))); + let err = r.err().unwrap_or_else(|| panic!("{name}: deep nesting must defer")); + assert!(err.starts_with("defer:"), "{name}: unexpected error: {err}"); + } + } + + #[test] + fn normal_files_are_untouched_by_the_guard() { + let src = "int add(int a, int b) { return a + b; }\nint main(void) { return add(1, 2); }\n"; + let r = on_small_stack(move || run_guarded(|| crate::ccpp::extract("ok.c", src, "c"))); + assert!(r.is_ok(), "a shallow file must not defer: {:?}", r.err()); + assert!(!overflowed()); + } + + #[test] + fn latch_resets_between_runs() { + let deep = format!("void foo(void) {{\n{}{}\n}}\n", "{".repeat(DEPTH), "}".repeat(DEPTH)); + on_small_stack(move || { + assert!(run_guarded(|| crate::ccpp::extract("deep.c", &deep, "c")).is_err()); + // The latch from the deep file must not poison the next, shallow one. + let ok = run_guarded(|| crate::ccpp::extract("ok.c", "int x;\n", "c")); + assert!(ok.is_ok(), "latch leaked into the next run: {:?}", ok.err()); + }); + } +} diff --git a/codegraph-kernel/src/swift.rs b/codegraph-kernel/src/swift.rs index e365b7197..7d46c4922 100644 --- a/codegraph-kernel/src/swift.rs +++ b/codegraph-kernel/src/swift.rs @@ -264,6 +264,7 @@ fn first_simple_identifier<'t>(node: Option>) -> Option> { /// lastNamedOfType (function-ref.ts:600): rightmost matching DESCENDANT in /// document order (deeper matches override). fn last_simple_identifier<'t>(node: Node<'t>) -> Option> { + stack_guard!(); let mut found: Option> = None; for i in 0..node.named_child_count() { let Some(child) = node.named_child(i) else { continue }; @@ -560,6 +561,7 @@ impl<'t> Walker<'t> { // --- the dispatcher (visitNode, Swift-relevant branches) ----------------------- fn visit_node(&mut self, node: Node<'t>) { + stack_guard!(); let kind = node.kind(); let mut skip_children = false; @@ -632,6 +634,7 @@ impl<'t> Walker<'t> { /// THE DEDICATED PROPERTY BRANCH (tree-sitter.ts:1113-1193, #1020). /// Returns skipChildren. fn dedicated_property_branch(&mut self, node: Node<'t>) -> bool { + stack_guard!(); let owner_row = self.top_row(); let info = self.swift_property_info(node); let mut computed_prop: Option<(u32, String)> = None; @@ -707,6 +710,7 @@ impl<'t> Walker<'t> { } fn walk_attr_args(&mut self, n: Node<'t>) { + stack_guard!(); self.extract_static_member_ref(n); for i in 0..n.named_child_count() { if let Some(c) = n.named_child(i) { @@ -718,10 +722,12 @@ impl<'t> Walker<'t> { // --- visitFunctionBody --------------------------------------------------------- fn visit_function_body(&mut self, body: Node<'t>) { + stack_guard!(); self.visit_for_calls_and_structure(body); } fn visit_for_calls_and_structure(&mut self, node: Node<'t>) { + stack_guard!(); let kind = node.kind(); self.maybe_capture_fn_refs(node); @@ -775,6 +781,7 @@ impl<'t> Walker<'t> { // --- extractors ----------------------------------------------------------------- fn extract_function(&mut self, node: Node<'t>) { + stack_guard!(); let name = self.extract_name(node); if name == "" { if let Some(body) = node.child_by_field_name("body") { @@ -802,6 +809,7 @@ impl<'t> Walker<'t> { } fn extract_method(&mut self, node: Node<'t>) { + stack_guard!(); let name = self.extract_name(node); let extra = Extra { docstring: preceding_docstring(node, self.src), @@ -823,6 +831,7 @@ impl<'t> Walker<'t> { } fn extract_class(&mut self, node: Node<'t>) { + stack_guard!(); let name = self.extract_name(node); let extra = Extra { docstring: preceding_docstring(node, self.src), @@ -845,6 +854,7 @@ impl<'t> Walker<'t> { } fn extract_struct(&mut self, node: Node<'t>) { + stack_guard!(); // Body gate (:1876) — bodiless mints nothing (record exemption is C#). let Some(body) = node.child_by_field_name("body") else { return }; let name = self.extract_name(node); @@ -866,6 +876,7 @@ impl<'t> Walker<'t> { } fn extract_enum(&mut self, node: Node<'t>) { + stack_guard!(); let Some(body) = node.child_by_field_name("body") else { return }; let name = self.extract_name(node); let extra = Extra { @@ -900,6 +911,7 @@ impl<'t> Walker<'t> { } fn extract_interface(&mut self, node: Node<'t>) { + stack_guard!(); let name = self.extract_name(node); let extra = Extra { docstring: preceding_docstring(node, self.src), @@ -1151,6 +1163,7 @@ impl<'t> Walker<'t> { } fn extract_type_refs_from_subtree(&mut self, node: Node<'t>, from_row: u32) { + stack_guard!(); if node.kind() == "type_identifier" { let type_name = self.text(node).to_string(); if !type_name.is_empty() && !is_builtin_type(&type_name) { @@ -1317,6 +1330,7 @@ impl<'t> Walker<'t> { } fn normalize_fn_ref_value(&mut self, v: Node<'t>, from: u32, depth: u32) { + stack_guard!(); if depth > 4 { return; } @@ -1381,6 +1395,7 @@ impl<'t> Walker<'t> { } fn scan_fn_ref_subtree(&mut self, node: Node<'t>, depth: u32) { + stack_guard!(); if depth > 12 { return; } diff --git a/codegraph-kernel/src/tsjs/extractors.rs b/codegraph-kernel/src/tsjs/extractors.rs index ad0440d6d..3e4814736 100644 --- a/codegraph-kernel/src/tsjs/extractors.rs +++ b/codegraph-kernel/src/tsjs/extractors.rs @@ -478,6 +478,7 @@ impl<'t> Walker<'t> { } fn find_initializer_returned_object(&self, call: Node<'t>, depth: u32) -> Option> { + stack_guard!(); if depth > 4 { return None; } @@ -499,6 +500,7 @@ impl<'t> Walker<'t> { fn function_returned_object(&self, fn_node: Node<'t>) -> Option> { fn as_object<'t>(n: Node<'t>) -> Option> { + stack_guard!(); match n.kind() { "object" | "object_expression" => Some(n), "parenthesized_expression" => { @@ -873,6 +875,7 @@ impl<'t> Walker<'t> { fn extract_ts_tuple_contract_names(&mut self, value: Node<'t>, alias_row: u32, alias_name: &str) { let mut tuples: Vec = Vec::new(); fn collect<'t>(n: Node<'t>, depth: u32, out: &mut Vec>) { + stack_guard!(); if depth > 6 { return; } @@ -1230,6 +1233,7 @@ impl<'t> Walker<'t> { // --- extractInheritance (TS/JS clauses) --------------------------------------------------- pub(super) fn extract_inheritance(&mut self, node: Node<'t>, class_row: u32) { + stack_guard!(); let extends_kind = edge_kind_index("extends").unwrap(); let implements_kind = edge_kind_index("implements").unwrap(); for i in 0..node.named_child_count() { @@ -1298,6 +1302,7 @@ impl<'t> Walker<'t> { } fn extract_type_refs_from_subtree(&mut self, node: Node<'t>, from_row: u32) { + stack_guard!(); if node.kind() == "type_identifier" { let type_name = self.text(node).to_string(); if !type_name.is_empty() && !is_builtin_type(&type_name) { diff --git a/codegraph-kernel/src/tsjs/mod.rs b/codegraph-kernel/src/tsjs/mod.rs index f61cf93e2..7559ae3f8 100644 --- a/codegraph-kernel/src/tsjs/mod.rs +++ b/codegraph-kernel/src/tsjs/mod.rs @@ -550,6 +550,7 @@ impl<'t> Walker<'t> { /// scanFnRefSubtree: capture-only walk of subtrees the main walkers skip. fn scan_fn_ref_subtree(&mut self, node: Node<'t>, depth: u32) { + stack_guard!(); if depth > 12 { return; } @@ -607,6 +608,7 @@ impl<'t> Walker<'t> { // --- the dispatcher (visitNode) -------------------------------------------- fn visit_node(&mut self, node: Node<'t>) { + stack_guard!(); let kind = node.kind(); let mut skip_children = false; @@ -686,6 +688,7 @@ impl<'t> Walker<'t> { } fn visit_for_calls_and_structure(&mut self, node: Node<'t>) { + stack_guard!(); let kind = node.kind(); self.maybe_capture_fn_refs(node); diff --git a/docs/design/rust-kernel-migration-plan.md b/docs/design/rust-kernel-migration-plan.md index 2f58bc7c6..14ed2673e 100644 --- a/docs/design/rust-kernel-migration-plan.md +++ b/docs/design/rust-kernel-migration-plan.md @@ -148,9 +148,11 @@ them are the ORIGINAL plan and carry expectations that measurement later correct tokio node sections IDENTICAL, small precision-positive edge churn only, full suite green), walker `codegraph-kernel/src/rustlang.rs` (survey artifact: rust-lang-kernel-port-checklist.md — isAsync dead-code, - impl-pushes-no-scope, trait-receiver bug on `impl Trait for Generic`, - phantom const identifiers, use-binding triple emission, all preserved - bug-for-bug). Gates: parity sweeps **0 diffs** on ripgrep (101/101, + impl-pushes-no-scope, trait-receiver bug on `impl Trait for Generic` + (fixed on both sides together in #1588 — receiver now comes from the + impl_item's `type` field), phantom const identifiers, use-binding + triple emission, all preserved bug-for-bug). Gates: parity sweeps + **0 diffs** on ripgrep (101/101, 0 deferred) / tokio (790/790, 0 deferred) / rust-analyzer (1217/1488, 0 diffs; 271 deferrals are token-macro-table sources — `T![~]`, `[$]` — that error on BOTH arms, grammar-inherent like fmt's C++ 42%); full-init @@ -410,6 +412,14 @@ in a different emission order would shift rowids and change resolution — and file whose tree `has_error()` to the wasm extractor** (`defer:` signal, silent, per-file) — parity by construction on erroring files, 99.6%+ keep the fast path, and the harness fails if deferrals exceed 10% (a broken kernel can't hide). + The same signal carries a second, rarer case (#1581): the walkers recurse per + AST level, and a pathologically nested file (clang's 16,384-brace + `parser_overflow.c`, fuzzer corpora) overflowed the native stack — a SIGSEGV + that killed the whole indexer, uncatchable from JS. `src/stack.rs` now + checks the thread's real stack bounds at every recursive entry + (`stack_guard!`) and `run_guarded` turns a tripped walk into `defer:`, so + the file lands on the wasm path (which catches its own `RangeError` per + file) while the process lives. Pinned by `__tests__/kernel-deep-nesting.test.ts`. 3. **Retrieval invariants:** kernel-indexed excalidraw — `mutateElement → renderStaticScene` connects end-to-end via explore (callback + react-render + jsx hops shown); synthesized-edge families present (408 jsx-render / 46 diff --git a/docs/design/rust-lang-kernel-port-checklist.md b/docs/design/rust-lang-kernel-port-checklist.md index 8727d8598..831e3ca18 100644 --- a/docs/design/rust-lang-kernel-port-checklist.md +++ b/docs/design/rust-lang-kernel-port-checklist.md @@ -89,18 +89,21 @@ Hooks PRESENT (port each exactly): - **getVisibility (rust.ts:74)** — direct child of type `visibility_modifier`: text `.includes('pub')` → `'public'` else `'private'`; no modifier → `'private'` (so `pub(crate)`/`pub(super)` are all `'public'`). -- **getReceiverType (rust.ts:83)** — walk PARENT chain to the nearest - `impl_item`; there: filter DIRECT namedChildren of type `type_identifier`; - if ≥1, return the LAST one's source text (`source.substring(startIndex, - endIndex)` — UTF-16 units). If none, find the first `generic_type` child and - return its inner `type_identifier` text; else undefined. Never an impl parent - → undefined. QUIRK/BUG, PRESERVE: for `impl Trait for Generic` the only - direct type_identifier is the TRAIT (probe: `impl Render for Container` → - typeIdents=[`Render`] → receiver = **`Render`**, the trait name — methods get - qualifiedName `Render::render` and a contains edge from the trait node if one - exists in-file). `impl fmt::Display for Fields` is fine - (scoped_type_identifier isn't type_identifier → [Fields]). `impl - Container` → no direct type_identifiers → generic branch → `Container`. +- **getReceiverType (rust.ts)** — walk PARENT chain to the nearest + `impl_item`; there, read the grammar's `type` field through + `rustImplTypeName` (kernel: `impl_type_name`): `type_identifier`/`identifier` + → text; `generic_type` → its `type` field (bare name, never the args); + `scoped_type_identifier`/`scoped_identifier` → its `name` field (last + segment); `reference_type` → its `type` field; anything else (tuple, `dyn`, + pointer, primitive, fn type) → undefined. Never an impl parent → undefined. + **Changed in #1588 on both sides together**: the original rule took the LAST + direct `type_identifier` child, so for `impl Trait for Generic` / + `Parents<'a>` / `&Foo` the only bare identifier was the TRAIT's (probe: + `impl Render for Container` → receiver **`Render`** → methods + `Render::render`, colliding with the trait declaration and feeding the + interface-impl synthesizer a phantom declaration). Now `Container`. + `impl fmt::Display for Fields` → `Fields`; `impl Container` → + `Container`; `impl Tr for m::Foo` → `Foo` (was: no receiver). Note `` type_parameters is its own child, its inner T is NOT a direct impl child. - **extractImport (rust.ts:120)** — signature = trimmed full `use …;` text. @@ -187,8 +190,9 @@ undefined; **no isConst means `const_item`/`static_item` extract as kind present AND not class-like — finds the FIRST node in `this.nodes` with `name === receiverType && filePath === this.filePath && kind ∈ {struct,class,enum,trait}`. Source-order dependent: an impl ABOVE its struct - gets no contains edge. `impl Trait for Generic` (receiver=trait bug) links - to the TRAIT node if it's in-file.** Then type annotations, decorators + gets no contains edge. Since #1588 `impl Trait for Generic` links to the + implementing TYPE's node (it used to link to the TRAIT node, the receiver + bug).** Then type annotations, decorators (no-op), body walk with the method pushed. - **Nested `fn` inside an impl-method's body**: visitFunctionBody:5245 → named → extractFunction → getReceiverType walks parents THROUGH the outer fn @@ -222,9 +226,13 @@ Generic else-branch (4312+), `func = childForFieldName('function') ?? namedChild (4455) → `Foo::new().bar()` → ref `Foo::new().bar`; an instance chain `x.foo().bar()` (innerFn field_expression) → bare `bar`. When not re-encoding, calleeName = bare methodName. - - receiver anything else (`field_expression` 2-hop `v.field.method()`, - `parenthesized_expression`, `await_expression`, `self`) → bare - methodName (probed all four). + - receiver `field_expression` whose `value` is `self` and whose `field` is + a `field_identifier` (`self.inner.run()`) → `self.inner.run` — the + owner-field shape the resolver types from the struct declaration + (#1585, both sides together). + - receiver anything else (`field_expression` with a non-self base + `v.field.method()` / deeper `self.a.b.m()`, `parenthesized_expression`, + `await_expression`, `self`) → bare methodName (probed all four). 2. `func.type === 'scoped_identifier'` (4499) → calleeName = FULL text (`Foo::new`, `m::helper2`, `std::mem::swap` — whatever the source spells, whitespace included). diff --git a/package-lock.json b/package-lock.json index b10985a80..c40a4033f 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,12 +1,12 @@ { "name": "@colbymchenry/codegraph", - "version": "1.5.0-fmagent.2", + "version": "1.6.0-fmagent.1", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "@colbymchenry/codegraph", - "version": "1.5.0-fmagent.2", + "version": "1.6.0-fmagent.1", "license": "MIT", "dependencies": { "@clack/prompts": "^1.3.0", diff --git a/package.json b/package.json index 5cc4ede48..5b593b3d0 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "@colbymchenry/codegraph", - "version": "1.5.0-fmagent.2", + "version": "1.6.0-fmagent.1", "description": "Supercharge AI coding agents with semantic code intelligence — surgical context, fewer tool calls, faster answers. 100% local.", "repository": { "type": "git", diff --git a/repro-1373/native.c b/repro-1373/native.c new file mode 100644 index 000000000..56d41a602 --- /dev/null +++ b/repro-1373/native.c @@ -0,0 +1,7 @@ +#define NATIVE_FN(name) int name(void) + +NATIVE_FN(get_version) { return 1; } + +int use_it(void) { return get_version(); } + +int plain_func(void) { return 42; } diff --git a/repro-1373/native.cpp b/repro-1373/native.cpp new file mode 100644 index 000000000..8b2e2adfe --- /dev/null +++ b/repro-1373/native.cpp @@ -0,0 +1,7 @@ +#define NATIVE_FN(name) int name(void) + +NATIVE_FN(get_version_cpp) { return 1; } + +int use_it_cpp(void) { return get_version_cpp(); } + +int plain_func_cpp(void) { return 42; } diff --git a/repro-1373/native.m b/repro-1373/native.m new file mode 100644 index 000000000..c5d1448d8 --- /dev/null +++ b/repro-1373/native.m @@ -0,0 +1,5 @@ +#define NATIVE_FN(name) int name(void) + +NATIVE_FN(get_version_objc) { return 1; } + +int use_it_objc(void) { return get_version_objc(); } diff --git a/src/bin/codegraph.ts b/src/bin/codegraph.ts index 067cdd3e0..19038df1b 100644 --- a/src/bin/codegraph.ts +++ b/src/bin/codegraph.ts @@ -405,6 +405,16 @@ function printIndexResult(clack: typeof import('@clack/prompts'), result: IndexR for (const w of result.errors.filter((e) => e.code === 'index_partial')) { clack.log.warn(w.message); } + // Files salvaged from comment-stripped source after repeated parser + // failures are indexed but possibly incomplete — say so here, or the run + // reads as fully clean and the index quietly disagrees with a later + // re-parse of the same bytes (#1565). + const salvaged = result.errors.filter((e) => e.code === 'salvaged_stripped'); + if (salvaged.length > 0) { + const sample = salvaged.slice(0, 3).map((e) => e.filePath).filter(Boolean).join(', '); + const more = salvaged.length > 3 ? ', ...' : ''; + clack.log.warn(`${formatNumber(salvaged.length)} file(s) indexed from comment-stripped source after repeated parse failures ${getGlyphs().dash} symbols may be incomplete (${sample}${more})`); + } } else if (hasErrors) { clack.log.error(`Indexing failed ${getGlyphs().dash} all ${formatNumber(result.filesErrored)} files had errors`); } else { @@ -443,9 +453,16 @@ function printIndexResult(clack: typeof import('@clack/prompts'), result: IndexR clack.log.info(`The index is fully usable ${getGlyphs().dash} only the failed files are missing.`); } } else if (projectPath) { - const logPath = path.join(getCodeGraphDir(projectPath), 'errors.log'); - if (fs.existsSync(logPath)) { - fs.unlinkSync(logPath); + // No hard errors. Salvaged-file warnings still belong in the log — it + // carries the per-file detail behind the one-line summary above. + if (result.errors.some((e) => e.code === 'salvaged_stripped')) { + writeErrorLog(projectPath, result.errors); + clack.log.info('See .codegraph/errors.log for details'); + } else { + const logPath = path.join(getCodeGraphDir(projectPath), 'errors.log'); + if (fs.existsSync(logPath)) { + fs.unlinkSync(logPath); + } } } } @@ -590,94 +607,111 @@ async function recordIndexTelemetry( // ============================================================================= /** - * codegraph init [path] + * The `init` flow — shared by `codegraph init` and `codegraph install --init` + * (#1578): refuse an unsafe root, create `.codegraph/`, build the initial + * index under supervision, then the post-index offers. `yes` makes every + * offer non-interactive (defaults only), so a container / CI bootstrap never + * blocks on a prompt. An unsafe root sets `process.exitCode = 1` and returns + * (no `--force` is implied by any caller); an index failure exits 1. */ -program - .command('init [path]') - .description('Initialize CodeGraph in a project directory and build the initial index') - .option('-i, --index', 'Deprecated: indexing now runs by default; flag accepted for backward compatibility') - .option('-f, --force', 'Initialize even if the path looks like your home directory or a filesystem root') - .option('-v, --verbose', 'Show detailed worker lifecycle and memory info') - .action(async (pathArg: string | undefined, options: { index?: boolean; force?: boolean; verbose?: boolean }) => { - const projectPath = path.resolve(pathArg || process.cwd()); - const clack = await importESM('@clack/prompts'); - - clack.intro('Initializing CodeGraph'); - - try { - // Refuse to index your home directory / a filesystem root — it pulls in - // caches, other projects, and your whole tree (a multi-GB index + watcher - // churn, and on pre-1.0 macOS a machine-crashing fd blowup, #845). - const unsafe = unsafeIndexRootReason(projectPath); - if (unsafe && !options.force) { - clack.log.error(`Refusing to initialize in ${projectPath} — it looks like ${unsafe}.`); - clack.log.info('Run this inside a specific project directory, or pass --force if you really mean to index everything under it.'); - clack.outro(''); - process.exitCode = 1; - return; - } - - if (isInitialized(projectPath)) { - clack.log.warn(`Already initialized in ${projectPath}`); - clack.log.info('Use "codegraph index" to re-index or "codegraph sync" to update'); - try { - const { offerWatchFallback } = await import('../installer'); - await offerWatchFallback(clack, projectPath); - } catch { /* non-fatal */ } - clack.outro(''); - return; - } - - const { default: CodeGraph, getDatabasePath } = await loadCodeGraph(); - const cg = await CodeGraph.init(projectPath, { index: false }); - clack.log.success(`Initialized in ${projectPath}`); +async function runInit( + projectPath: string, + options: { index?: boolean; force?: boolean; verbose?: boolean; yes?: boolean }, +): Promise { + const clack = await importESM('@clack/prompts'); - // Indexing runs by default now. The legacy -i/--index flag is still - // accepted (so existing muscle memory and scripts don't break) but is a - // no-op — initializing always builds the initial index. - // Supervise the index: self-terminate if orphaned or wedged (#999). - // The DB + WAL paths let the liveness watchdog tell a slow store on - // degraded storage from a true wedge (#1231). - // A closure so we can re-run the exact same supervised, progress-rendered - // index if the user opts gitignored child repos in below (#1156). - const dbPath = getDatabasePath(projectPath); - const runIndex = async (): Promise => { - const supervision = installCommandSupervision('init', { progressPaths: [dbPath, `${dbPath}-wal`] }); - try { - if (options.verbose) { - return await cg.indexAll({ onProgress: createVerboseProgress(), verbose: true }); - } - process.stdout.write(`${colors.dim}${getGlyphs().rail}${colors.reset}\n`); - const progress = createShimmerProgress(); - const r = await cg.indexAll({ onProgress: progress.onProgress }); - await progress.stop(); - return r; - } finally { - supervision.stop(); - } - }; - const result = await runIndex(); - printIndexResult(clack, result, projectPath); - await recordIndexTelemetry(cg, result); + clack.intro('Initializing CodeGraph'); - // An empty graph at a git super-repo usually means `.gitignore` excludes - // the child repos that hold the code — surface them and offer to opt in - // rather than leaving the user with a silent 0-node "Done". (#1156) - if (result.nodesCreated === 0) { - await offerIndexIgnoredRepos(clack, projectPath, runIndex, { interactive: true }); - } + try { + // Refuse to index your home directory / a filesystem root — it pulls in + // caches, other projects, and your whole tree (a multi-GB index + watcher + // churn, and on pre-1.0 macOS a machine-crashing fd blowup, #845). + const unsafe = unsafeIndexRootReason(projectPath); + if (unsafe && !options.force) { + clack.log.error(`Refusing to initialize in ${projectPath} — it looks like ${unsafe}.`); + clack.log.info('Run this inside a specific project directory, or pass --force if you really mean to index everything under it.'); + clack.outro(''); + process.exitCode = 1; + return; + } + if (isInitialized(projectPath)) { + clack.log.warn(`Already initialized in ${projectPath}`); + clack.log.info('Use "codegraph index" to re-index or "codegraph sync" to update'); try { const { offerWatchFallback } = await import('../installer'); - await offerWatchFallback(clack, projectPath); + await offerWatchFallback(clack, projectPath, { yes: options.yes }); } catch { /* non-fatal */ } + clack.outro(''); + return; + } - clack.outro('Done'); - cg.destroy(); - } catch (err) { - clack.log.error(`Failed: ${err instanceof Error ? err.message : String(err)}`); - process.exit(1); + const { default: CodeGraph, getDatabasePath } = await loadCodeGraph(); + const cg = await CodeGraph.init(projectPath, { index: false }); + clack.log.success(`Initialized in ${projectPath}`); + + // Indexing runs by default now. The legacy -i/--index flag is still + // accepted (so existing muscle memory and scripts don't break) but is a + // no-op — initializing always builds the initial index. + // Supervise the index: self-terminate if orphaned or wedged (#999). + // The DB + WAL paths let the liveness watchdog tell a slow store on + // degraded storage from a true wedge (#1231). + // A closure so we can re-run the exact same supervised, progress-rendered + // index if the user opts gitignored child repos in below (#1156). + const dbPath = getDatabasePath(projectPath); + const runIndex = async (): Promise => { + const supervision = installCommandSupervision('init', { progressPaths: [dbPath, `${dbPath}-wal`] }); + try { + if (options.verbose) { + return await cg.indexAll({ onProgress: createVerboseProgress(), verbose: true }); + } + process.stdout.write(`${colors.dim}${getGlyphs().rail}${colors.reset}\n`); + const progress = createShimmerProgress(); + const r = await cg.indexAll({ onProgress: progress.onProgress }); + await progress.stop(); + return r; + } finally { + supervision.stop(); + } + }; + const result = await runIndex(); + printIndexResult(clack, result, projectPath); + await recordIndexTelemetry(cg, result); + + // An empty graph at a git super-repo usually means `.gitignore` excludes + // the child repos that hold the code — surface them and offer to opt in + // rather than leaving the user with a silent 0-node "Done". (#1156) + // Under --yes the offer prints its one-line opt-in snippet instead of + // prompting (same as a non-TTY run). + if (result.nodesCreated === 0) { + await offerIndexIgnoredRepos(clack, projectPath, runIndex, { interactive: !options.yes }); } + + try { + const { offerWatchFallback } = await import('../installer'); + await offerWatchFallback(clack, projectPath, { yes: options.yes }); + } catch { /* non-fatal */ } + + clack.outro('Done'); + cg.destroy(); + } catch (err) { + clack.log.error(`Failed: ${err instanceof Error ? err.message : String(err)}`); + process.exit(1); + } +} + +/** + * codegraph init [path] + */ +program + .command('init [path]') + .description('Initialize CodeGraph in a project directory and build the initial index') + .option('-i, --index', 'Deprecated: indexing now runs by default; flag accepted for backward compatibility') + .option('-f, --force', 'Initialize even if the path looks like your home directory or a filesystem root') + .option('-v, --verbose', 'Show detailed worker lifecycle and memory info') + .option('-y, --yes', 'Non-interactive: skip every prompt and take the defaults (for scripts / CI / container bootstraps)') + .action(async (pathArg: string | undefined, options: { index?: boolean; force?: boolean; verbose?: boolean; yes?: boolean }) => { + await runInit(path.resolve(pathArg || process.cwd()), options); }); /** @@ -1216,6 +1250,65 @@ program } }); +/** + * codegraph context + * + * The CLI face of the public `buildContext` API (ContextBuilder): FTS entry + * points + graph expansion + code blocks, formatted as markdown or JSON. + * Advertised in the usage header since the first release but never actually + * registered (#1611); external integrations (e.g. Memorix) invoke it as + * `codegraph context --path --format json --max-nodes 8 --no-code `. + */ +program + .command('context ') + .description('Build context for a task: relevant symbols, relationships, and code blocks') + .option('-p, --path ', 'Project path') + .option('-f, --format ', 'Output format: markdown or json', 'markdown') + .option('-n, --max-nodes ', 'Maximum number of symbols to include') + .option('--no-code', 'Omit code blocks (structure only)') + .action(async (taskParts: string[], options: { path?: string; format?: string; maxNodes?: string; code?: boolean }) => { + const projectPath = resolveProjectPath(options.path); + + const format = options.format ?? 'markdown'; + if (format !== 'markdown' && format !== 'json') { + error(`Unknown format "${options.format}" — use "markdown" or "json".`); + process.exit(1); + } + let maxNodes: number | undefined; + if (options.maxNodes !== undefined) { + maxNodes = parseInt(options.maxNodes, 10); + if (Number.isNaN(maxNodes) || maxNodes < 1) { + error(`--max-nodes expects a positive integer, got "${options.maxNodes}".`); + process.exit(1); + } + } + + try { + if (!isInitialized(projectPath)) { + error(`CodeGraph not initialized in ${projectPath}`); + process.exit(1); + } + + const { default: CodeGraph } = await loadCodeGraph(); + const cg = await CodeGraph.open(projectPath); + + const result = await cg.buildContext(taskParts.join(' '), { + format, + includeCode: options.code !== false, + ...(maxNodes !== undefined ? { maxNodes } : {}), + }); + + // Both supported formats return a formatted string; print it verbatim so + // `--format json` stays machine-parseable on stdout (error()/warnings go + // to stderr only). + console.log(typeof result === 'string' ? result : JSON.stringify(result, null, 2)); + cg.destroy(); + } catch (err) { + error(`Context build failed: ${err instanceof Error ? err.message : String(err)}`); + process.exit(1); + } + }); + /** * codegraph prompt-hook (hidden) * @@ -1691,10 +1784,10 @@ program .aliases(['daemons']) .description('Manage running CodeGraph background daemons — pick one and press enter to stop it') .action(async () => { - const { listDaemons, stopDaemonAt, stopAllDaemons } = await import('../mcp/daemon-registry'); + const { listVerifiedDaemons, stopDaemonAt, stopAllDaemons } = await import('../mcp/daemon-registry'); const { runDaemonPicker } = await import('../mcp/daemon-manager'); - const daemons = listDaemons(); + const daemons = await listVerifiedDaemons(); if (daemons.length === 0) { info('No CodeGraph daemons running.'); return; @@ -1717,7 +1810,7 @@ program const clack = await importESM('@clack/prompts'); clack.intro('CodeGraph daemons'); await runDaemonPicker({ - list: listDaemons, + list: listVerifiedDaemons, stop: stopDaemonAt, stopAll: stopAllDaemons, cwdRoot, @@ -1823,14 +1916,15 @@ program } const lockPath = path.join(getCodeGraphDir(projectPath), 'codegraph.lock'); - - if (!fs.existsSync(lockPath)) { - info(`No lock file found ${getGlyphs().dash} nothing to do`); - return; - } - - fs.unlinkSync(lockPath); - success('Removed lock file. You can now run indexing again.'); + let removed = false; + if (fs.existsSync(lockPath)) { + fs.unlinkSync(lockPath); + removed = true; + } + const { clearStaleDaemonArtifacts } = await import('../mcp/daemon-registry'); + removed = await clearStaleDaemonArtifacts(projectPath) || removed; + if (removed) success('Removed stale lock artifacts. You can now run indexing again.'); + else info(`No stale lock files found ${getGlyphs().dash} nothing to do`); } catch (err) { error(`Failed to remove lock: ${err instanceof Error ? err.message : String(err)}`); process.exit(1); @@ -2250,6 +2344,7 @@ program .option('-t, --target ', 'Target agent(s): comma-separated ids, or "auto"|"all"|"none". Default: prompt') .option('-l, --location ', 'Install location: "global" or "local". Default: prompt') .option('-y, --yes', 'Non-interactive: defaults to --location=global --target=auto, auto-allow on') + .option('-i, --init', 'After wiring agents, also run `codegraph init` in the current directory — builds this project’s index, so install + index is one command (combine with --yes for an unattended bootstrap)') .option('--no-permissions', 'Skip writing the auto-allow permissions list (Claude Code only)') .option('--print-config ', 'Print MCP config snippet for the named agent and exit (no file writes)') .option('--refresh', 'Rewrite what previous installs configured, for already-configured agents only (never adds new ones). Run automatically by `codegraph upgrade`') @@ -2257,6 +2352,7 @@ program target?: string; location?: string; yes?: boolean; + init?: boolean; permissions?: boolean; printConfig?: string; refresh?: boolean; @@ -2334,6 +2430,18 @@ program error(err instanceof Error ? err.message : String(err)); process.exit(1); } + + // --init: the one-shot "wire agents AND build this project's index" + // bootstrap (#1578). The installer itself never indexes implicitly (a + // surprise index of $HOME is the thing we refuse) — an explicit flag is + // the user choosing. Runs after a successful install, including the + // `--target none` / nothing-detected case (the installer returns normally + // there), and shares every guard with `codegraph init`: an unsafe root + // is refused (exit 1, no implied --force), an already-initialized + // project just says so. `--yes` flows through so no offer prompts. + if (opts.init) { + await runInit(process.cwd(), { yes: opts.yes }); + } }); /** diff --git a/src/context/index.ts b/src/context/index.ts index ad4d63bc0..b3c5a5726 100644 --- a/src/context/index.ts +++ b/src/context/index.ts @@ -171,6 +171,7 @@ const DEFAULT_FIND_OPTIONS: Required = { minScore: 0.3, edgeKinds: [], nodeKinds: HIGH_VALUE_NODE_KINDS, // Filter out imports/exports by default + seedNames: [], // Segment-vocab supplement — filled by the facade }; // Re-export the low-confidence sentinel (defined in a dependency-free leaf so @@ -187,6 +188,7 @@ export { LOW_CONFIDENCE_MARKER } from './markers'; export class ContextBuilder { private projectRoot: string; private queries: QueryBuilder; + private traverser: GraphTraverser; constructor( @@ -199,6 +201,21 @@ export class ContextBuilder { this.traverser = traverser; } + /** + * Whether the project's `codegraph.json` `deprioritize` patterns cover this + * path (#982). Explore ranks through its own path scorer as well as through + * `searchNodes`, so the lever has to be applied here too or the setting would + * only half-work — and explore is the surface #982 actually reports on. + * + * Only the -15 relevance penalty is shared. Explore's hard `continue` filters + * and its non-production budget cap are deliberately NOT joined: those REMOVE + * content, and `deprioritize` is a ranking lever by definition — `exclude` is + * the lever for taking things out of reach. + */ + private isDeprioritized(filePath: string): boolean { + return this.queries.getDeprioritizedPathMatcher()?.(filePath) ?? false; + } + /** * Build context for a task * @@ -460,13 +477,37 @@ export class ContextBuilder { // Step 2: Look up exact matches for extracted symbols let exactMatches: SearchResult[] = []; - if (symbolsFromQuery.length > 0) { + if (symbolsFromQuery.length > 0 || opts.seedNames.length > 0) { try { - // Get more results so we can apply co-location boosting before trimming - exactMatches = this.queries.findNodesByExactName(symbolsFromQuery, { - limit: Math.ceil(opts.searchLimit * 5), - kinds: opts.nodeKinds && opts.nodeKinds.length > 0 ? opts.nodeKinds : undefined, - }); + if (symbolsFromQuery.length > 0) { + // Get more results so we can apply co-location boosting before trimming + exactMatches = this.queries.findNodesByExactName(symbolsFromQuery, { + limit: Math.ceil(opts.searchLimit * 5), + kinds: opts.nodeKinds && opts.nodeKinds.length > 0 ? opts.nodeKinds : undefined, + }); + } + + // Step 2a: segment-vocabulary seeds. Word-level query terms cannot + // reach camelCase names through FTS (one token per name), so the + // caller resolves query words → names via the segment vocab and hands + // them in as seedNames. Merged at a dampened score — a symbol the + // query names outright must outrank a segment-derived one — but + // BEFORE the co-location boost below, because several seeds landing + // in one file (pinFeedIfNearBottom + feedAtBottom + handleFeedScroll) + // is exactly the evidence that file is the answer. + if (opts.seedNames.length > 0) { + const seedResults = this.queries.findNodesByExactName(opts.seedNames, { + limit: Math.ceil(opts.searchLimit * 3), + kinds: opts.nodeKinds && opts.nodeKinds.length > 0 ? opts.nodeKinds : undefined, + }); + const known = new Set(exactMatches.map((r) => r.node.id)); + for (const r of seedResults) { + if (known.has(r.node.id)) continue; + known.add(r.node.id); + exactMatches.push({ ...r, score: r.score * 0.6 }); + } + logDebug('Segment seed matches', { seedNames: opts.seedNames, added: known.size }); + } // Co-location boost: when multiple extracted symbols appear in the same file, // those results are much more likely to be what the user is looking for. @@ -812,7 +853,12 @@ export class ContextBuilder { if (searchIdSet.has(r.node.id)) continue; if (isTestFile(r.node.filePath) && !isTestQuery) continue; - const pathScore = scorePathRelevance(r.node.filePath, query); + const pathScore = scorePathRelevance( + r.node.filePath, + query, + undefined, + this.isDeprioritized(r.node.filePath), + ); const brevityBonus = Math.max(0, 6 - (name.length - titleCased.length) / 4); termCandidates.push({ node: r.node, score: 8 + brevityBonus + pathScore }); } @@ -899,7 +945,12 @@ export class ContextBuilder { const compoundResults: SearchResult[] = []; for (const [, entry] of compoundTermMap) { if (entry.terms.size >= 2) { - const pathScore = scorePathRelevance(entry.node.filePath, query); + const pathScore = scorePathRelevance( + entry.node.filePath, + query, + undefined, + this.isDeprioritized(entry.node.filePath), + ); const brevityBonus = Math.max(0, 6 - entry.node.name.length / 8); compoundResults.push({ node: entry.node, diff --git a/src/db/index.ts b/src/db/index.ts index 4d52b0c6c..f01d195d1 100644 --- a/src/db/index.ts +++ b/src/db/index.ts @@ -145,6 +145,7 @@ export class DatabaseConnection { // beginBulkNodeLoad and endBulkNodeLoad): the FTS triggers are missing and // nodes_fts is stale. Rebuild + recreate so search stays in sync. conn.healBulkNodeLoad(); + conn.healBulkSecondaryIndexes(); // Self-heal a killed session's leftover oversized WAL (#1431) — one // statSync when healthy, off-thread checkpoint+truncate when not. @@ -363,6 +364,28 @@ export class DatabaseConnection { this.endBulkNodeLoad(); } + /** Recreate every secondary index a killed bulk parse/ref/edge window may leave dropped. */ + private healBulkSecondaryIndexes(): void { + const names = [...new Set([ + ...DatabaseConnection.BULK_PARSE_INDEX_NAMES, + ...DatabaseConnection.BULK_REF_INDEX_NAMES, + ...DatabaseConnection.BULK_EDGE_INDEX_NAMES, + ])]; + const placeholders = names.map(() => '?').join(','); + const row = this.db + .prepare(`SELECT count(*) AS c FROM sqlite_master WHERE type = 'index' AND name IN (${placeholders})`) + .get(...names) as { c: number } | undefined; + if ((row?.c ?? 0) >= names.length) return; + + const schemaPath = path.join(__dirname, 'schema.sql'); + const schema = fs.readFileSync(schemaPath, 'utf-8'); + for (const idx of names) { + const m = schema.match(new RegExp(`CREATE INDEX IF NOT EXISTS ${idx}\\b[^;]*;`)); + if (!m) throw new Error(`schema.sql: index ${idx} not found for crash recovery`); + this.db.exec(m[0]); + } + } + /** * Recreate the FTS sync triggers from schema.sql — extracted from the file * rather than duplicated here so the DDL cannot drift from the schema. diff --git a/src/db/queries.ts b/src/db/queries.ts index 451c6a90f..af19b14cf 100644 --- a/src/db/queries.ts +++ b/src/db/queries.ts @@ -55,6 +55,23 @@ function isLowValueFile(filePath: string, generated?: ReadonlySet): bool const SQLITE_PARAM_CHUNK_SIZE = 500; +/** + * How much of the exact-name bonus a `deprioritize`d path keeps (#982). Damped + * rather than zeroed: a query that genuinely targets that tree must still rank + * it, the same "discount, don't erase" rule the path penalty follows. + * + * Derived rather than picked. `nameMatchBonus`'s prefix arm tops out below + * `10 + 30 = 40`, and a de-prioritized node also takes the -15 path penalty, so + * `80 * SCALE - 15 > 40` is what stops a damped WHOLE-QUERY exact match from + * losing to a mere prefix match. 0.75 clears it (45). Measured on a 62k-node + * django index: at 0.25 that invariant breaks in practice — `child`, `parent` + * and `method` lose rank 1 to `children`, `all_parents` and `method_decorator` + * — while crowd-out removal is almost flat between 0.75 and 0.5 (39 vs 40 of 88 + * peripheral top-10 slots cleared), so a deeper discount buys little and costs + * the invariant. Pinned by a test. + */ +export const DEPRIORITIZED_NAME_BONUS_SCALE = 0.75; + /** * Database row types (snake_case from SQLite) */ @@ -128,8 +145,11 @@ interface UnresolvedRefRow { * refs against newly-added node names. */ function referenceNameTail(referenceName: string): string { - const idx = Math.max(referenceName.lastIndexOf('.'), referenceName.lastIndexOf(':')); - return idx >= 0 ? referenceName.slice(idx + 1) : referenceName; + // Erlang refs carry a written arity (`f/1`, `mod::fn/2` — #1610); the tail a + // new symbol's plain name could match is the arity-less function name. + const base = referenceName.replace(/\/\d{1,3}$/, '') || referenceName; + const idx = Math.max(base.lastIndexOf('.'), base.lastIndexOf(':')); + return idx >= 0 ? base.slice(idx + 1) : base; } /** @@ -204,6 +224,7 @@ export class QueryBuilder { // whole project, not a symbol, so it carries no discriminative signal (#720). // Set once by the CodeGraph instance; empty by default (no down-weighting). private projectNameTokens: Set = new Set(); + private isDeprioritizedPath: ((filePath: string) => boolean) | undefined; // Node cache for frequently accessed nodes (LRU-style, max 1000 entries) private nodeCache: Map = new Map(); @@ -328,6 +349,21 @@ export class QueryBuilder { return this.projectNameTokens; } + /** + * Set the predicate that marks a path as de-prioritized by the project's + * `codegraph.json` `deprioritize` patterns (#982). Ranking-only: those paths + * stay indexed and findable, they just stop outranking first-party code. + * Called once when the project opens; undefined disables the lever. + */ + setDeprioritizedPathMatcher(matcher: ((filePath: string) => boolean) | undefined): void { + this.isDeprioritizedPath = matcher; + } + + /** The `deprioritize` predicate (#982), so other rankers apply the same lever. */ + getDeprioritizedPathMatcher(): ((filePath: string) => boolean) | undefined { + return this.isDeprioritizedPath; + } + // =========================================================================== // Node Operations // =========================================================================== @@ -1168,15 +1204,26 @@ export class QueryBuilder { } /** - * Get nodes by lowercase name match (uses idx_nodes_lower_name expression index) + * Get nodes by name, case-insensitively (seeks the idx_nodes_lower_name + * expression index). + * + * The parameter is lowered in SQL rather than trusted to arrive lowered, so + * the lookup means the same thing whatever casing a caller hands it. Written + * as a bare `lower(name) = ?` it silently returned nothing for any input + * carrying an uppercase letter, and — because SQLite's `lower()` folds ASCII + * only while JavaScript's `.toLowerCase()` folds Unicode — a caller that + * pre-lowered in JavaScript could not match a non-ASCII name at all. + * + * Note this hardens the query, not its one caller: `matchFuzzy` still lowers + * in JavaScript before calling, so the non-ASCII gap remains open there. */ - getNodesByLowerName(lowerName: string): Node[] { + getNodesByLowerName(name: string): Node[] { if (!this.stmts.getNodesByLowerName) { this.stmts.getNodesByLowerName = this.db.prepare( - 'SELECT * FROM nodes WHERE lower(name) = ?' + 'SELECT * FROM nodes WHERE lower(name) = lower(?)' ); } - const rows = this.stmts.getNodesByLowerName.all(lowerName) as NodeRow[]; + const rows = this.stmts.getNodesByLowerName.all(name) as NodeRow[]; return rows.map(rowToNode); } @@ -1242,12 +1289,25 @@ export class QueryBuilder { // pushing them past the FTS fetch limit before post-hoc scoring can help. // Use the max BM25 score as the base so the nameMatchBonus (exact=30 vs // prefix=20) actually differentiates them after rescoring. + // + // Whole-name equality MUST be written as `lower(name) = lower(?)` so it + // seeks `idx_nodes_lower_name`. The equivalent `name = ? COLLATE NOCASE` + // matches no index — `idx_nodes_name` is BINARY-collated and the expression + // index only matches the same expression — and degrades to a full table + // scan. The `LIMIT 20` does not rescue it: SQLite can only stop early once + // it has produced 20 rows, and this runs once per query term, most of which + // name nothing in the corpus. Measured per term on an unmatched term: + // 0.08ms on gin (2.5k nodes), 0.39ms on excalidraw (11k), 2.4ms on django + // (62k) — and growing with the corpus, where the seek is flat at ~0.002ms. + // Lowering the parameter in SQL rather than in JS is deliberate: SQLite's + // `lower()` and NOCASE both fold ASCII only, while JS `.toLowerCase()` + // folds Unicode, which would silently stop matching non-ASCII names. if (results.length > 0 && query) { const existingIds = new Set(results.map(r => r.node.id)); const maxFtsScore = Math.max(...results.map(r => r.score)); const terms = query.split(/\s+/).filter(t => t.length >= 2); for (const term of terms) { - let sql = 'SELECT * FROM nodes WHERE name = ? COLLATE NOCASE'; + let sql = 'SELECT * FROM nodes WHERE lower(name) = lower(?)'; const params: (string | number)[] = [term]; if (kinds && kinds.length > 0) { sql += ` AND kind IN (${kinds.map(() => '?').join(',')})`; @@ -1271,13 +1331,24 @@ export class QueryBuilder { // Apply multi-signal scoring if (results.length > 0 && (text || query)) { const scoringQuery = text || query; - results = results.map(r => ({ - ...r, - score: r.score - + kindBonus(r.node.kind) - + scorePathRelevance(r.node.filePath, scoringQuery, this.projectNameTokens) - + nameMatchBonus(r.node.name, scoringQuery), - })); + results = results.map(r => { + // A path the project de-prioritized is saying its symbol NAMES are not + // the answer, so the exact-name bonus has to be damped too. The -15 path + // penalty alone cannot do it: the bonus is additive and larger (measured + // on #982's repro, a `usage()` helper sat at 74.8 vs 51.2 for the top + // product symbol — -15 lands at 59.8, still ahead). Damped, not zeroed, + // so the tree stays findable when it genuinely is what you asked for. + // Evaluated once and reused: the predicate stats the config file. + const deprioritized = this.isDeprioritizedPath?.(r.node.filePath) ?? false; + const nameBonus = nameMatchBonus(r.node.name, scoringQuery); + return { + ...r, + score: r.score + + kindBonus(r.node.kind) + + scorePathRelevance(r.node.filePath, scoringQuery, this.projectNameTokens, deprioritized) + + (deprioritized ? Math.round(nameBonus * DEPRIORITIZED_NAME_BONUS_SCALE) : nameBonus), + }; + }); results.sort((a, b) => b.score - a.score); // Trim to requested limit after rescoring if (results.length > limit) { @@ -1547,9 +1618,16 @@ export class QueryBuilder { // Pass 2: Query each name, boosting results that co-locate with distinctive symbols. // Pass 1: Find files containing each queried name, identify distinctive names + // + // Both passes spell whole-name equality as `lower(name) = lower(?)` so they + // seek `idx_nodes_lower_name` — see the note in `searchNodes` for why the + // `name = ? COLLATE NOCASE` form full-scans instead. This path is the one + // that hurts most: it runs both passes for every symbol extracted from the + // query, and extraction is generous, so most of those names are absent from + // the corpus and never reach either LIMIT. const nameToFiles = new Map>(); for (const name of names) { - let sql = 'SELECT DISTINCT file_path FROM nodes WHERE name COLLATE NOCASE = ?'; + let sql = 'SELECT DISTINCT file_path FROM nodes WHERE lower(name) = lower(?)'; const params: (string | number)[] = [name]; if (kinds && kinds.length > 0) { sql += ` AND kind IN (${kinds.map(() => '?').join(',')})`; @@ -1577,7 +1655,7 @@ export class QueryBuilder { let sql = ` SELECT nodes.*, 1.0 as score FROM nodes - WHERE name COLLATE NOCASE = ? + WHERE lower(name) = lower(?) `; const params: (string | number)[] = [name]; @@ -2359,7 +2437,12 @@ export class QueryBuilder { const chunkRows = this.db .prepare(`SELECT * FROM unresolved_refs WHERE status = 'pending' AND file_path IN (${placeholders})`) .all(...chunk) as UnresolvedRefRow[]; - rows.push(...chunkRows); + // Append with a loop, never a spread: the INPUT chunk is bounded, but + // the RESULT rows per chunk are not — a dense recovery sync (e.g. the + // #1541 self-heal re-indexing hundreds of files) returns more rows than + // V8 allows as arguments, and `push(...chunkRows)` dies with "Maximum + // call stack size exceeded", aborting resolution mid-sync (#1558). + for (const row of chunkRows) rows.push(row); } return rows.map((row) => ({ @@ -2541,7 +2624,10 @@ export class QueryBuilder { const chunkRows = this.db .prepare(`SELECT * FROM unresolved_refs WHERE status = 'failed' AND name_tail IN (${placeholders})`) .all(...chunk) as UnresolvedRefRow[]; - rows.push(...chunkRows); + // Loop, not spread — same V8 argument-limit hazard as + // getUnresolvedReferencesByFiles (#1558): a large definition delta can + // select an unbounded number of failed rows per chunk. + for (const row of chunkRows) rows.push(row); } return rows.map((row) => ({ diff --git a/src/directory.ts b/src/directory.ts index fd4a1aa0b..ff5c3b842 100644 --- a/src/directory.ts +++ b/src/directory.ts @@ -233,6 +233,65 @@ export function findIndexedSubprojectRoots( return out; } +/** Result of {@link resolveServerRoot}. */ +export interface ServerRootResolution { + /** The project root to serve as the default, or null when none resolved. */ + root: string | null; + /** True when `root` was adopted from the down-scan rather than the up-walk. */ + viaSubScan: boolean; + /** + * Indexed sub-projects the down-scan saw when it ran but could NOT adopt + * (zero or several candidates). Empty when the up-walk resolved or the scan + * was skipped. Callers surface these so "no default project" errors can say + * what IS reachable (#1607). + */ + candidates: string[]; +} + +/** + * Whether `base` is a plausible workspace root for the sub-project down-scan. + * Mirrors `planFrontload`'s manifest gate, widened to accept a bare `.git` + * entry — the #1606 shape is a workspace container holding only agent config + * and a `.git`, with every build manifest living in the indexed children. The + * user's home directory and the filesystem root are never eligible: a stray + * manifest there must not turn server startup into a scan that could adopt an + * unrelated project (#1454 documents that failure mode for the prompt-hook). + */ +function eligibleForSubprojectScan(base: string): boolean { + if (base === path.parse(base).root) return false; + let home: string | null = null; + try { home = os.homedir(); } catch { home = null; } + if (home && (base === home || base === path.resolve(home))) return false; + if (looksLikeProjectRoot(base)) return true; + return fs.existsSync(path.join(base, '.git')); +} + +/** + * Resolve the project root an MCP server should serve as its DEFAULT project + * (#1606). Up-walk first (`findNearestCodeGraphRoot` — the common case, and + * cheap). When nothing is indexed at or above `searchFrom`, run the bounded + * sub-project down-scan `planFrontload` already uses, behind the workspace + * gate above: EXACTLY ONE indexed sub-project is unambiguous and is adopted + * as the root; zero or several yield no root, with the candidates carried so + * the caller can name them instead of failing silently (#1607). + * + * `opts.subprojectScan: false` skips the down-scan entirely (the per-tool-call + * retry path throttles it; the up-walk always runs). + */ +export function resolveServerRoot( + searchFrom: string, + opts: { subprojectScan?: boolean } = {}, +): ServerRootResolution { + const up = findNearestCodeGraphRoot(searchFrom); + if (up) return { root: up, viaSubScan: false, candidates: [] }; + if (opts.subprojectScan === false) return { root: null, viaSubScan: false, candidates: [] }; + const base = path.resolve(searchFrom); + if (!eligibleForSubprojectScan(base)) return { root: null, viaSubScan: false, candidates: [] }; + const subs = findIndexedSubprojectRoots(base); + if (subs.length === 1) return { root: subs[0]!, viaSubScan: true, candidates: subs }; + return { root: null, viaSubScan: false, candidates: subs }; +} + /** * Unicode-aware word-boundary emulation for the keyword lists below. JS's `\b` * is ASCII-only — it fires only at `[A-Za-z0-9_]` edges — so it can never bound diff --git a/src/extraction/grammars.ts b/src/extraction/grammars.ts index d4127631d..84647c3e4 100644 --- a/src/extraction/grammars.ts +++ b/src/extraction/grammars.ts @@ -496,9 +496,40 @@ export function detectLanguage(filePath: string, source?: string, overrides?: Re return lang; } +/** + * A class/struct BASE CLAUSE — `struct Derived : Base {`, `class Foo final : + * public Bar, private Baz {`, `struct D : ns::B {` — which is never valid + * C. In C the only thing that can follow `struct ` is `{`, `;`, `*`, an + * identifier (declarator), or a closing `)`: a bit-field's `:` sits after a + * member NAME inside the body (`unsigned a : 3;`), a ternary's `:` is + * separated from the tag by `)` / `*` / a declarator (`sizeof(struct foo) : + * 0`), and a label such as `struct_end:` has no whitespace after `struct`. An + * optional access specifier / `virtual` after the colon and an optional + * `final` before it cover the spelled-out forms; the base may be scoped + * (`ns::Base`) and carry template arguments, and must be followed by the + * body's `{` or a `,` introducing the next base — prose like + * `struct timeval: seconds and microseconds` inside a string never has that + * terminator. Comments are stripped before the scan (see `looksLikeCpp`). + */ +const CPP_BASE_CLAUSE_RE = + /\b(?:class|struct)\s+\w+\s*(?:final\s*)?:\s*(?:(?:public|protected|private|virtual)\s+)*[A-Za-z_][\w:]*(?:\s*<[^{};]*>)?\s*[{,]/; + +/** Block and line comments, for a code-only scan. Lazy block match → linear. */ +const C_COMMENT_RE = /\/\*[\s\S]*?\*\/|\/\/[^\n]*/g; + /** * Heuristic: does a .h file contain C++ constructs? - * Checks the first ~8KB for patterns that are unique to C++ and never valid C. + * + * Two passes. The first checks the first ~8KB for patterns that are unique to + * C++ and never valid C. The second scans the FULL source for a class/struct + * base clause (`CPP_BASE_CLAUSE_RE`): a large header with a long C-compatible + * preamble — include guards, `#define`s, plain C typedefs — can put its only + * C++ signal past the sample, and the cost of that miss is the C extractor + * (classTypes: []) dropping the derived type entirely and minting a phantom + * `function Base` from the base clause instead (#1592). The base-clause regex + * is anchored on a `struct`/`class` keyword followed by a tag and a colon, a + * shape with no C reading, so widening it to the whole file cannot drag a C + * header over to C++. */ function looksLikeCpp(source: string): boolean { const sample = source.substring(0, 8192); @@ -511,7 +542,15 @@ function looksLikeCpp(source: string): boolean { // routed through the C extractor (which extracts no classes), and its class // definition silently vanishes. The two-token shape (` ` // before a `[:{]`) never occurs in valid C, so this can't misclassify C headers. - return /\bnamespace\b|\bclass\s+\w+\s*[:{]|\b(?:class|struct)\s+[A-Z][A-Z0-9_]+\s+\w+\s*(?:final\s*)?[:{]|\btemplate\s*<|\b(?:public|private|protected)\s*:|\bvirtual\b|\busing\s+(?:namespace\b|\w+\s*=)/.test(sample); + if (/\bnamespace\b|\bclass\s+\w+\s*[:{]|\b(?:class|struct)\s+[A-Z][A-Z0-9_]+\s+\w+\s*(?:final\s*)?[:{]|\btemplate\s*<|\b(?:public|private|protected)\s*:|\bvirtual\b|\busing\s+(?:namespace\b|\w+\s*=)/.test(sample)) { + return true; + } + // Plain `struct Derived : Base` (no export macro, no `class` keyword, no + // explicit access section) — the #1159 branch above only recognizes the + // macro-annotated form. Scanned over the whole file, not the sample, with + // comments removed so a doc comment's prose (`struct foo: x, y`) can't + // flip a C header. + return CPP_BASE_CLAUSE_RE.test(source.replace(C_COMMENT_RE, ' ')); } /** diff --git a/src/extraction/index.ts b/src/extraction/index.ts index 1d63cfce1..67339db40 100644 --- a/src/extraction/index.ts +++ b/src/extraction/index.ts @@ -27,7 +27,7 @@ import { StoreWriter, StoreBundle, finalizeStoreBundle } from './store-writer'; import { materializeKernelResult } from './kernel'; import { detectGeneratedFile } from './generated-detection'; import { detectLanguage, isSourceFile, isLanguageSupported, isFileLevelOnlyLanguage, initGrammars, loadGrammarsForLanguages, readGrammarWasmBytes } from './grammars'; -import { loadExtensionOverrides, loadIncludeIgnoredPatterns, loadExcludePatterns, loadIncludePatterns } from '../project-config'; +import { loadExtensionOverrides, loadIncludeIgnoredPatterns, loadExcludePatterns, loadIncludePatterns, PROJECT_CONFIG_FILENAME } from '../project-config'; import { isCodeGraphDataDir } from '../directory'; import { logDebug, logWarn } from '../errors'; import { validatePathWithinRoot, normalizePath } from '../utils'; @@ -1112,7 +1112,7 @@ interface GitChanges { * case this cannot see (the child status that would report the deletions is gone * with it); a full `codegraph index` reconciles that. */ -function getGitChangedFiles(rootDir: string): GitChanges | null { +export function getGitChangedFiles(rootDir: string): GitChanges | null { try { const changes: GitChanges = { modified: [], added: [], deleted: [] }; // Custom extension → language overrides from the project's codegraph.json, @@ -1128,7 +1128,13 @@ function getGitChangedFiles(rootDir: string): GitChanges | null { function collectGitStatus(repoDir: string, prefix: string, out: GitChanges, overrides?: Record, includeIgnored: Ignore | null = null, exclude: Ignore | null = null): void { const output = execFileSync( 'git', - ['status', '--porcelain', '--no-renames'], + // `-uall` lists individual untracked files instead of collapsing an + // entirely-untracked directory into one `?? dir/` entry, which would + // otherwise be dropped here (only embedded git repos are recursed into + // below). Nested untracked git repos still collapse to `?? repo/` even + // with `-uall` — git never crosses a repo boundary — so the recursion + // still handles them. (#1213) + ['status', '--porcelain', '--no-renames', '-uall'], { cwd: repoDir, encoding: 'utf-8', timeout: 10000, maxBuffer: 50 * 1024 * 1024, stdio: ['pipe', 'pipe', 'pipe'], windowsHide: true } ); @@ -1450,12 +1456,48 @@ export class ExtractionOrchestrator { * hasn't run yet so single-file re-index paths can detect on the spot. */ private detectedFrameworkNames: string[] | null = null; + /** + * Scope matcher for SCOPED syncs, memoized on the mtimes of the two root + * files it is derived from (`codegraph.json`, `.gitignore`). See + * {@link scopedSyncMatcher}. + */ + private scopedMatcher: { key: string; matcher: ScopeIgnore } | null = null; constructor(rootDir: string, queries: QueryBuilder) { this.rootDir = rootDir; this.queries = queries; } + /** + * The scope matcher a scoped sync applies to the paths it was handed — the + * same `buildScopeIgnore` the full scan uses, so an explicitly-passed path + * that is OUT of scope (a user `exclude` in `codegraph.json`, a `.gitignore` + * rule, a built-in default) is treated exactly as the full walk would treat + * it: absent, hence removed if tracked, never parsed (#1590). + * + * Memoized on the root config + root `.gitignore` mtimes: building the + * matcher runs embedded-repo discovery (`git ls-files`), which would defeat + * the scoped path's whole point (skipping O(repo) work) if paid per sync. + * Two `stat`s per sync while nothing changed. An embedded repo created + * between config edits joins the scoped matcher on the next full sync, the + * same lifecycle the watcher's own matcher already has. + */ + private scopedSyncMatcher(): ScopeIgnore { + const key = [PROJECT_CONFIG_FILENAME, '.gitignore'] + .map((name) => { + try { + return String(fs.statSync(path.join(this.rootDir, name)).mtimeMs); + } catch { + return '-'; + } + }) + .join('|'); + if (this.scopedMatcher && this.scopedMatcher.key === key) return this.scopedMatcher.matcher; + const matcher = buildScopeIgnore(this.rootDir); + this.scopedMatcher = { key, matcher }; + return matcher; + } + /** * Build a filesystem-backed ResolutionContext sufficient for framework * detection. Graph-query methods (getNodesByName etc.) return empty because @@ -1585,6 +1627,11 @@ export class ExtractionOrchestrator { }); if (process.env.CODEGRAPH_SYNTH_TIMINGS) console.error(`[phase-timing] scan: ${Date.now() - tScan}ms (${files.length} files)`); + // A re-index over an existing DB skips unchanged-hash files at the store, + // which would preserve wiped zero-node rows (#1541) — drop them first so + // this run stores their files fresh. No-op on a fresh DB. + this.healZeroNodeRows(); + // Detect frameworks once per indexAll run using the scanned file list. // Names are passed to each parse call so framework-specific extractors // (route nodes, middleware, etc.) run after the tree-sitter pass. @@ -1717,7 +1764,7 @@ export class ExtractionOrchestrator { const inFlight = new Set>(); const completed = new Map(); + | { ok: false; filePath: string; content: string; stats: fs.Stats; err: unknown }>(); let nextSeq = 0; // file-order sequence assigned at dispatch let nextToStore = 0; // cursor: next sequence to commit let aborted = false; @@ -1745,27 +1792,25 @@ export class ExtractionOrchestrator { // Store: on the writer thread when active (fresh DB — bundles applied // in the same file order this chain dispatches them), else on the main // thread (SQLite connections are per-thread). - if (nodeCount > 0 || result.errors.length === 0) { - const language = detectLanguage(filePath, content, overrides); - if (storeWriter) { - if (result.kernelBuffers) { - // Buffers go to the writer as-is; the worker decodes + finalizes. - // The main thread's only per-file work stays O(1) + the content hash. - storeWriter.send({ - kernel: true, - filePath, - language, - buffers: result.kernelBuffers, - file: this.buildFileRecord(filePath, content, language, stats, nodeCount, result.errors), - }); - } else { - storeWriter.send(this.buildFreshStoreBundle(filePath, content, language, stats, result)); - } - await storeWriter.waitBelow(STORE_WRITER_WINDOW); + const language = detectLanguage(filePath, content, overrides); + if (storeWriter) { + if (result.kernelBuffers) { + // Buffers go to the writer as-is; the worker decodes + finalizes. + // The main thread's only per-file work stays O(1) + the content hash. + storeWriter.send({ + kernel: true, + filePath, + language, + buffers: result.kernelBuffers, + file: this.buildFileRecord(filePath, content, language, stats, nodeCount, result.errors), + }); } else { - const materialized = materializeKernelResult(result, filePath, language); - await this.storeExtractionResult(filePath, content, language, stats, materialized, commitYield); + storeWriter.send(this.buildFreshStoreBundle(filePath, content, language, stats, result)); } + await storeWriter.waitBelow(STORE_WRITER_WINDOW); + } else { + const materialized = materializeKernelResult(result, filePath, language); + await this.storeExtractionResult(filePath, content, language, stats, materialized, commitYield); } if (result.errors.length > 0) { @@ -1796,16 +1841,19 @@ export class ExtractionOrchestrator { onProgress?.({ phase: 'parsing', current: processed, total, currentFile: filePath }); }; - const recordParseFailure = (filePath: string, err: unknown): void => { - processed++; - filesErrored++; - errors.push({ - message: err instanceof Error ? err.message : String(err), - filePath, - severity: 'error', - code: 'parse_error', + const recordParseFailure = async (filePath: string, content: string, stats: fs.Stats, err: unknown): Promise => { + await storeResult(filePath, content, stats, { + nodes: [], + edges: [], + unresolvedReferences: [], + errors: [{ + message: err instanceof Error ? err.message : String(err), + filePath, + severity: 'error', + code: 'parse_error', + }], + durationMs: 0, }); - onProgress?.({ phase: 'parsing', current: processed, total }); }; // Commit buffered parses to the DB in file order, advancing the cursor over @@ -1828,7 +1876,7 @@ export class ExtractionOrchestrator { completed.delete(nextToStore); nextToStore++; if (item.ok) await storeResult(item.filePath, item.content, item.stats, item.result); - else recordParseFailure(item.filePath, item.err); + else await recordParseFailure(item.filePath, item.content, item.stats, item.err); } } catch (err) { flushError = err; @@ -1847,7 +1895,7 @@ export class ExtractionOrchestrator { const result = await parseFile(filePath, content); completed.set(seq, { ok: true, filePath, content, stats, result }); } catch (parseErr) { - completed.set(seq, { ok: false, filePath, err: parseErr }); + completed.set(seq, { ok: false, filePath, content, stats, err: parseErr }); } flushOrdered(); })(); @@ -1918,15 +1966,18 @@ export class ExtractionOrchestrator { // useful symbols. The single-file extractFile path already enforces // this; the bulk path used to silently skip the check. if (stats.size > MAX_FILE_SIZE) { - processed++; - filesSkipped++; - errors.push({ - message: `File exceeds max size (${stats.size} > ${MAX_FILE_SIZE})`, - filePath, - severity: 'warning', - code: 'size_exceeded', + await storeResult(filePath, content, stats, { + nodes: [], + edges: [], + unresolvedReferences: [], + errors: [{ + message: `File exceeds max size (${stats.size} > ${MAX_FILE_SIZE})`, + filePath, + severity: 'warning', + code: 'size_exceeded', + }], + durationMs: 0, }); - onProgress?.({ phase: 'parsing', current: processed, total }); continue; } @@ -2033,8 +2084,16 @@ export class ExtractionOrchestrator { continue; } + // The pool hands kernel results back as an undecoded buffer transport + // (`nodes`/`edges` EMPTY, tables in kernelBuffers). The main loop + // decodes or forwards to the store worker; this path stores directly, + // so decode here — otherwise a kernel-language retry passes the gate + // below via `errors.length === 0`, stores nothing, and the file is + // permanently recorded as "(0 symbols)" with the error erased (#1541). + const language = detectLanguage(filePath, content, overrides); + result = materializeKernelResult(result, filePath, language); + if (result.nodes.length > 0 || result.errors.length === 0) { - const language = detectLanguage(filePath, content, overrides); const stats = await fsp.stat(path.join(this.rootDir, filePath)); await this.storeExtractionResult(filePath, content, language, stats, result, commitYield); @@ -2083,13 +2142,22 @@ export class ExtractionOrchestrator { continue; } + // Same undecoded-transport hazard as the first retry pass (#1541). + const language = detectLanguage(filePath, fullContent, overrides); + result = materializeKernelResult(result, filePath, language); + if (result.nodes.length > 0 || result.errors.length === 0) { - const language = detectLanguage(filePath, fullContent, overrides); const stats = await fsp.stat(path.join(this.rootDir, filePath)); await this.storeExtractionResult(filePath, fullContent, language, stats, result, commitYield); - const idx = errors.indexOf(errEntry); - if (idx >= 0) errors.splice(idx, 1); + // Salvaged from comment-stripped source: keep a visible trace in + // the summary instead of erasing the failure outright — the + // stored result may be missing whatever the failing parse choked + // on, and a silently "clean" file here is how an index quietly + // disagrees with a later per-file sync of the same bytes (#1565). + errEntry.severity = 'warning'; + errEntry.code = 'salvaged_stripped'; + errEntry.message = `Indexed from comment-stripped source after repeated parse failures (symbols may be incomplete until the file is re-indexed): ${errEntry.message}`; filesErrored--; filesIndexed++; totalNodes += result.nodes.length; @@ -2228,9 +2296,11 @@ export class ExtractionOrchestrator { }; } + const language = detectLanguage(relativePath, content, loadExtensionOverrides(this.rootDir)); + // Check file size if (stats.size > MAX_FILE_SIZE) { - return { + const result: ExtractionResult = { nodes: [], edges: [], unresolvedReferences: [], @@ -2244,10 +2314,11 @@ export class ExtractionOrchestrator { ], durationMs: 0, }; + await this.storeExtractionResult(relativePath, content, language, stats, result, createYielder()); + return result; } // Detect language (honoring the project's codegraph.json extension overrides) - const language = detectLanguage(relativePath, content, loadExtensionOverrides(this.rootDir)); if (!isLanguageSupported(language)) { return { nodes: [], @@ -2265,9 +2336,7 @@ export class ExtractionOrchestrator { const result = extractFromSource(relativePath, content, language, frameworkNames); // Store in database - if (result.nodes.length > 0 || result.errors.length === 0) { - await this.storeExtractionResult(relativePath, content, language, stats, result, createYielder()); - } + await this.storeExtractionResult(relativePath, content, language, stats, result, createYielder()); return result; } @@ -2275,6 +2344,34 @@ export class ExtractionOrchestrator { /** * Store extraction result in database */ + /** + * Delete file rows recorded with ZERO nodes so their files re-index. + * + * No extraction path stores an empty, error-free result for a + * symbol-bearing language — even an empty file keeps its file node — so a + * zero-node row is a wiped one (#1541: an interrupted parse's retry stored + * an undecoded kernel transport). The wiped row's content hash matches the + * on-disk bytes, so every hash-based reconcile skips the file forever; + * deleting the row lets the normal add path repair it. File-level-only + * languages (yaml, twig, properties) are left alone. Deleting a zero-node + * row cascades nothing: it has no nodes, so no edges or refs either. + */ + private healZeroNodeRows(): void { + for (const f of this.queries.getAllFiles()) { + // A zero-node row WITH recorded errors is a deliberate skip marker + // (#1557: oversized / repeatedly-unparseable files are persisted with + // their reason so syncs stop retrying them) — leave those alone. The + // #1541 wipe rows are the error-FREE zero-node rows. + if ( + f.nodeCount === 0 && + !isFileLevelOnlyLanguage(f.language) && + (f.errors === undefined || f.errors.length === 0) + ) { + this.queries.deleteFile(f.path); + } + } + } + private async storeExtractionResult( filePath: string, content: string, @@ -2283,6 +2380,12 @@ export class ExtractionOrchestrator { result: ExtractionResult, onYield?: MaybeYield ): Promise { + // A kernel result can arrive as an undecoded buffer transport (empty + // node/edge arrays, tables riding in kernelBuffers). Decode it before + // storing — persisting the transport as-is records the file as having no + // symbols at all (#1541). No-op for already-decoded results. + result = materializeKernelResult(result, filePath, language); + // Bulk inserts run in bounded sub-transactions with a yield between, so a // giant generated file (tens of thousands of symbols) can't block the // event loop — and the #850 watchdog heartbeat — for the whole store. @@ -2292,10 +2395,20 @@ export class ExtractionOrchestrator { const STORE_CHUNK = 2000; const contentHash = hashContent(content); - // Check if file already exists and hasn't changed + // Check if file already exists and hasn't changed. A skip/failure MARKER + // row (zero nodes + recorded errors, #1557) never blocks a store carrying + // real content: markers are written BEFORE the retry pass under the same + // content hash, so treating them as "no changes" would silently discard a + // successful retry's symbols — a permanent empty file presented as + // recovered (the #1541 wipe, reintroduced through the marker path). const existingFile = this.queries.getFileByPath(filePath); if (existingFile && existingFile.contentHash === contentHash) { - return; // No changes + const existingIsMarker = + existingFile.nodeCount === 0 && (existingFile.errors?.length ?? 0) > 0; + const incomingHasContent = result.nodes.length > 0; + if (!existingIsMarker || !incomingHasContent) { + return; // No changes + } } // Re-decided on every re-index of a changed file, so a banner added (or @@ -2631,7 +2744,23 @@ export class ExtractionOrchestrator { // reads `filesChecked === 0 && durationMs === 0` as the // lock-unavailable signature (#449). const unique = [...new Set(scopedPaths)]; - currentFiles = unique.filter((p) => fs.existsSync(path.join(this.rootDir, p))); + // A scoped path is "present" only if it exists AND is in scope — the + // same two gates the full walk applies (source extension, scope + // matcher). Without the scope gate a caller's stale view of scope + // leaked straight into the index: the watcher re-parsed a file the + // user had just excluded in `codegraph.json` while `codegraph sync` + // removed it (#1590). Out-of-scope paths fall out of `currentFiles`, + // so a tracked one takes the removal branch below, exactly as a full + // sync would treat it. (`include`-forced paths pass: ScopeIgnore + // applies the include precedence itself.) + const scope = this.scopedSyncMatcher(); + const overrides = loadExtensionOverrides(this.rootDir); + currentFiles = unique.filter( + (p) => + isSourceFile(p, overrides) && + !scope.ignores(p) && + fs.existsSync(path.join(this.rootDir, p)) + ); trackedFiles = []; for (const p of unique) { const rec = this.queries.getFileByPath(p); @@ -2644,6 +2773,10 @@ export class ExtractionOrchestrator { if (process.env.CODEGRAPH_SYNTH_TIMINGS) console.error(`[phase-timing] sync-scan: ${Date.now() - tSyncScan}ms (${currentFiles.length} files)`); filesChecked = currentFiles.length; + // Full reconcile only (scoped syncs must not touch rows outside their + // scope): drop zero-node rows so the wiped files re-index as adds below. + this.healZeroNodeRows(); + const tTracked = Date.now(); trackedFiles = this.queries.getAllFiles(); if (process.env.CODEGRAPH_SYNTH_TIMINGS) console.error(`[phase-timing] sync-tracked-load: ${Date.now() - tTracked}ms (${trackedFiles.length} tracked)`); diff --git a/src/extraction/languages/erlang.ts b/src/extraction/languages/erlang.ts index 57e6d2573..2f9f1b4bb 100644 --- a/src/extraction/languages/erlang.ts +++ b/src/extraction/languages/erlang.ts @@ -9,8 +9,11 @@ import type { LanguageExtractor, ExtractorContext } from '../tree-sitter-types'; // extractor, so every symbol-bearing top-level form is dispatched through the // visitNode hook below instead: // - a function's name lives on its CLAUSE, not the fun_decl, and the grammar -// emits one fun_decl PER CLAUSE — consecutive same-name fun_decl forms are -// merged into a single function node here; +// emits one fun_decl PER CLAUSE — consecutive same-name same-ARITY +// fun_decl forms (clauses of one function) are merged into a single +// function node here. Arity is part of an Erlang function's identity +// (`f/1` and `f/2` are unrelated definitions — #1610), so each arity gets +// its own node, qualified `mod::f/1` / `mod::f/2`; // - type-position expressions (-spec/-type/-callback bodies, record field // types) parse as `call` nodes, so descending into them would mint bogus // call refs to type names (`pid()`, `term()`); the hook consumes those @@ -19,9 +22,10 @@ import type { LanguageExtractor, ExtractorContext } from '../tree-sitter-types'; // the generic extractStruct would skip as a forward declaration. // Calls (local `f(X)`, remote `mod:f(X)`, `fun f/1` references, and record // usages) are handled by the erlang branch in extractCall — remote calls are -// emitted as `mod::f`, which matches the qualifiedName the module namespace -// produces (see packageTypes below), so cross-module resolution rides the -// standard qualified-name matcher. +// emitted as `mod::f/2` (arity counted at the call site), byte-identical to +// the qualifiedName above, so cross-module resolution rides the standard +// qualified-name matcher; local calls are emitted `f/2` and resolved by the +// erlang arity step in matchReference. /** Text of an atom with quoted-atom quotes stripped (`'EXIT'` → `EXIT`). */ function atomText(node: SyntaxNode, source: string): string { @@ -35,19 +39,27 @@ function collapseWs(text: string): string { // --- Per-file memos. Extraction is file-sequential within a worker, so a // single-entry memo keyed by filePath is safe (and resets naturally). --- -/** Exported function names for the current file ('all' for -compile(export_all)). */ +/** + * Exported `name/arity` keys for the current file ('all' for + * -compile(export_all)). Keyed by arity because `-export([f/1])` exports + * exactly f/1 — f/2 in the same module stays private (#1610). A malformed + * `fa` with no arity node falls back to the bare name key. + */ let exportsFile = ''; let exportsMemo: Set | 'all' = new Set(); /** - * Clause-merge state: the previous fun_decl's name and node id. A fun_decl - * whose clause repeats that name is a continuation clause (or a same-name - * different-arity definition — deliberately grouped under one node, the way - * overloads are elsewhere) and attaches to the existing node instead of - * creating a duplicate. + * Clause-merge state: the previous fun_decl's name, arity, and node id. A + * fun_decl whose clause repeats that (name, arity) is a continuation clause of + * the SAME function and attaches to the existing node instead of creating a + * duplicate. A same-name DIFFERENT-arity fun_decl is an unrelated function + * (Erlang identity is `name/arity`) and gets its own node (#1610). Keying on + * adjacency stays safe: clauses of one function must be adjacent in Erlang — + * a non-adjacent redefinition of the same name/arity is a compile error. */ let lastFnFile = ''; let lastFnName = ''; +let lastFnArity = -1; let lastFnId = ''; function moduleExports(node: SyntaxNode, source: string, filePath: string): Set | 'all' { @@ -69,7 +81,12 @@ function moduleExports(node: SyntaxNode, source: string, filePath: string): Set< for (const fa of form.namedChildren) { if (fa.type !== 'fa') continue; const fun = getChildByField(fa, 'fun'); - if (fun) result.add(atomText(fun, source)); + if (!fun) continue; + const name = atomText(fun, source); + const arityNode = getChildByField(fa, 'arity'); + const arityValue = arityNode ? getChildByField(arityNode, 'value') : null; + const arity = arityValue ? getNodeText(arityValue, source) : null; + result.add(arity !== null ? `${name}/${arity}` : name); } } } @@ -78,13 +95,27 @@ function moduleExports(node: SyntaxNode, source: string, filePath: string): Set< return result; } -/** The -spec directly above a function (comments may sit between), if it names it. */ -function precedingSpec(node: SyntaxNode, name: string, source: string): SyntaxNode | null { +/** Argument count of a clause/sig: the `args` (expr_args) field's named-child count. */ +function nodeArity(withArgs: SyntaxNode): number { + const args = getChildByField(withArgs, 'args'); + return args ? args.namedChildCount : 0; +} + +/** + * The -spec directly above a function (comments may sit between), if it names + * it AND matches its arity — the spec for `header/3` sitting between the + * `header/2` and `header/3` definitions must attach to /3 only (#1610). A + * spec whose sigs can't be read (defensive) is accepted on the name alone. + */ +function precedingSpec(node: SyntaxNode, name: string, arity: number, source: string): SyntaxNode | null { let prev = node.previousNamedSibling; while (prev && prev.type === 'comment') prev = prev.previousNamedSibling; if (prev?.type === 'spec') { const specFun = getChildByField(prev, 'fun'); - if (specFun && atomText(specFun, source) === name) return prev; + if (specFun && atomText(specFun, source) === name) { + const sigs = prev.namedChildren.filter((c) => c.type === 'type_sig'); + if (sigs.length === 0 || sigs.some((sig) => nodeArity(sig) === arity)) return prev; + } } return null; } @@ -104,10 +135,11 @@ function handleFunDecl(node: SyntaxNode, ctx: ExtractorContext): boolean { if (!nameNode) return true; const name = atomText(nameNode, ctx.source); if (!name) return true; + const arity = nodeArity(first); - // Continuation clause: extend the existing node's span and attribute this - // clause's calls to it. - if (ctx.filePath === lastFnFile && name === lastFnName && lastFnId) { + // Continuation clause of the SAME function (same name AND arity): extend the + // existing node's span and attribute this clause's calls to it. + if (ctx.filePath === lastFnFile && name === lastFnName && arity === lastFnArity && lastFnId) { for (let i = ctx.nodes.length - 1; i >= 0; i--) { const n = ctx.nodes[i]; if (n && n.id === lastFnId) { @@ -121,16 +153,20 @@ function handleFunDecl(node: SyntaxNode, ctx: ExtractorContext): boolean { return true; } - const spec = precedingSpec(node, name, ctx.source); + const spec = precedingSpec(node, name, arity, ctx.source); const exports = moduleExports(node, ctx.source, ctx.filePath); const fn = ctx.createNode('function', name, node, { docstring: getPrecedingDocstring(spec ?? node, ctx.source), signature: spec ? collapseWs(getNodeText(spec, ctx.source)).slice(0, 300) : clauseHeader(first, ctx.source), - isExported: exports === 'all' || exports.has(name), + isExported: exports === 'all' || exports.has(`${name}/${arity}`) || exports.has(name), }); if (!fn) return true; + // Arity is part of the function's identity — carry it on the qualified name + // (`mod::f/2`), the canonical Erlang spelling and the only persisted slot. + // The node NAME stays bare so name search and bare-name matching still work. + fn.qualifiedName = `${fn.qualifiedName}/${arity}`; ctx.pushScope(fn.id); // The whole clause is walked (not just the body) so record patterns in the // arguments and guard calls contribute references too. @@ -138,6 +174,7 @@ function handleFunDecl(node: SyntaxNode, ctx: ExtractorContext): boolean { ctx.popScope(); lastFnFile = ctx.filePath; lastFnName = name; + lastFnArity = arity; lastFnId = fn.id; return true; } diff --git a/src/extraction/languages/rust.ts b/src/extraction/languages/rust.ts index 6d91bf6b8..cef4c208e 100644 --- a/src/extraction/languages/rust.ts +++ b/src/extraction/languages/rust.ts @@ -32,6 +32,45 @@ function extractRustReturnType(node: SyntaxNode, source: string): string | undef return last === 'Self' ? 'self' : last; } +/** + * The implementing type's simple name for an `impl` block, read from the + * grammar's `type` field (#1588). Mirrored byte-for-byte by the native + * kernel's `impl_type_name` (codegraph-kernel/src/rustlang.rs) — change both. + * + * `impl Source for BufSource`, `impl<'a> Iterator for Parents<'a>`, + * `impl Trait for &Foo`, `impl Trait for m::Foo` all yield the implementing + * TYPE (`BufSource`, `Parents`, `Foo`, `Foo`). The previous rule took the last + * bare `type_identifier` child of the `impl_item`; once the implementing type + * carries parameters it parses as a `generic_type`, so the only bare + * identifier left was the TRAIT's — every parameterized impl's methods were + * qualified by the trait (`Source::read`), unaddressable by their type and + * colliding with the trait's own declaration. + * + * Shapes that name no single type (tuples, `dyn Trait`, pointers, primitives, + * function types…) yield undefined: no receiver, and the fn is extracted + * exactly as before. + */ +export function rustImplTypeName(typeNode: SyntaxNode | null, source: string): string | undefined { + if (!typeNode) return undefined; + switch (typeNode.type) { + case 'type_identifier': + case 'identifier': + return getNodeText(typeNode, source); + // `Foo` — the `type` field is the bare (or scoped) name, never the args. + case 'generic_type': + return rustImplTypeName(getChildByField(typeNode, 'type'), source); + // `m::Foo` — the last segment is the type's name. + case 'scoped_type_identifier': + case 'scoped_identifier': + return rustImplTypeName(getChildByField(typeNode, 'name'), source); + // `&Foo` / `&'a mut Foo` — the referenced type. + case 'reference_type': + return rustImplTypeName(getChildByField(typeNode, 'type'), source); + default: + return undefined; + } +} + export const rustExtractor: LanguageExtractor = { // `function_signature_item` is a trait method DECLARATION (`fn render(&self);`, // no body). Extracting it makes a trait's method set first-class, which @@ -88,32 +127,10 @@ export const rustExtractor: LanguageExtractor = { let parent = node.parent; while (parent) { if (parent.type === 'impl_item') { - // For `impl Type { ... }` — the type is a direct type_identifier child - // For `impl Trait for Type { ... }` — the type is the LAST type_identifier - // (the first is part of the trait path) - const children = parent.namedChildren; - // Find all direct type_identifier children (not nested in scoped paths) - const typeIdents = children.filter( - (c: SyntaxNode) => c.type === 'type_identifier' - ); - if (typeIdents.length > 0) { - // Last type_identifier is always the implementing type - const typeNode = typeIdents[typeIdents.length - 1]!; - return source.substring(typeNode.startIndex, typeNode.endIndex); - } - // Handle generic types: impl MyStruct { ... } - const genericType = children.find( - (c: SyntaxNode) => c.type === 'generic_type' - ); - if (genericType) { - const innerType = genericType.namedChildren.find( - (c: SyntaxNode) => c.type === 'type_identifier' - ); - if (innerType) { - return source.substring(innerType.startIndex, innerType.endIndex); - } - } - return undefined; + // The grammar names the implementing type directly (the `type` field) + // for both `impl Type { … }` and `impl Trait for Type { … }` — see + // rustImplTypeName for why the old positional scan was wrong (#1588). + return rustImplTypeName(getChildByField(parent, 'type'), source); } parent = parent.parent; } diff --git a/src/extraction/parse-pool.ts b/src/extraction/parse-pool.ts index 26f8ca055..9b2b100e4 100644 --- a/src/extraction/parse-pool.ts +++ b/src/extraction/parse-pool.ts @@ -61,6 +61,8 @@ const MAX_PARSE_POOL_SIZE = 16; const DEFAULT_RECYCLE_INTERVAL = 250; /** Base per-parse timeout; scaled up for large files by the caller's formula. */ const DEFAULT_PARSE_TIMEOUT_MS = 10_000; +/** Keep the default large-file budget bounded; the hard-kill window is 3× this. */ +const MAX_SCALED_PARSE_TIMEOUT_MS = 20_000; /** * A worker is only killed once a parse has gone this many × its budget with no * result. The base timer firing is NOT proof the parse is still running: after @@ -109,6 +111,17 @@ export function resolveParseTimeoutMs(envVal: string | undefined): number { return DEFAULT_PARSE_TIMEOUT_MS; } +/** + * Per-file soft timeout. Size scaling helps legitimate large sources, but an + * uncapped linear budget gave data-only headers near the 1 MiB file limit a + * 4.5–5 minute hard-kill window (#1555). Explicit larger base overrides remain + * respected for slow storage. + */ +export function resolveParseBudgetMs(baseMs: number, contentLength: number): number { + const scaled = baseMs + Math.floor(contentLength / 100_000) * 10_000; + return Math.min(scaled, Math.max(baseMs, MAX_SCALED_PARSE_TIMEOUT_MS)); +} + export function resolveParsePoolSize(envVal: string | undefined, cpuCount: number): number { if (envVal !== undefined && envVal !== '') { const n = Number(envVal); @@ -201,6 +214,12 @@ export class ParseWorkerPool { this.createWorker = opts.createWorker; } else if (opts.workerScriptPath) { const scriptPath = opts.workerScriptPath; + // Deliberately no `resourceLimits.stackSizeMb`: a bigger worker stack + // only moves the cliff a deeply nested file falls off (#1581 — the + // 8 MiB main thread still dies at 100k levels). The native kernel + // guards its own recursion against THIS thread's real stack bounds + // (codegraph-kernel/src/stack.rs) and defers such a file to the wasm + // path, which catches its JS RangeError per file. this.createWorker = () => new Worker(scriptPath); } else { throw new Error('ParseWorkerPool requires workerScriptPath or createWorker'); @@ -344,7 +363,7 @@ export class ParseWorkerPool { this.parseCounts.set(w, (this.parseCounts.get(w) ?? 0) + 1); // Scale the timeout for large files: base + 10s per 100KB (matches the // original single-worker formula so pathological-file behaviour is unchanged). - const timeoutMs = this.parseTimeoutMs + Math.floor(job.task.content.length / 100_000) * 10_000; + const timeoutMs = resolveParseBudgetMs(this.parseTimeoutMs, job.task.content.length); job.budgetMs = timeoutMs; job.timer = setTimeout(() => this.onTimeout(w, job, timeoutMs), timeoutMs); job.timer.unref?.(); diff --git a/src/extraction/tree-sitter.ts b/src/extraction/tree-sitter.ts index 8d71d7f18..c34dc4716 100644 --- a/src/extraction/tree-sitter.ts +++ b/src/extraction/tree-sitter.ts @@ -22,6 +22,7 @@ import { isGeneratedFile } from './generated-detection'; import type { LanguageExtractor, ExtractorContext } from './tree-sitter-types'; import { EXTRACTORS } from './languages'; import { stripCppTemplateArgs } from './languages/c-cpp'; +import { rustImplTypeName } from './languages/rust'; import { LiquidExtractor } from './liquid-extractor'; import { RazorExtractor } from './razor-extractor'; import { SvelteExtractor } from './svelte-extractor'; @@ -3759,15 +3760,18 @@ export class TreeSitterExtractor { // Erlang: a local call is `call(expr: atom, args)`; a remote call nests it // under `remote(module: remote_module, fun: call)` — the module qualifier - // lives on the PARENT. Remote calls are emitted as `mod::fn`, which is - // byte-identical to the qualifiedName the module namespace gives every - // function (see packageTypes in languages/erlang.ts), so they resolve via - // matchByQualifiedName. A var/macro callee or module (`F(X)`, `?M(X)`, - // `Mod:handle(X)`) has no static target — except `?MODULE:fn(X)`, which the - // bare name + same-file preference resolves correctly. `fun name/1` / - // `fun mod:name/1` values are function REFERENCES (callback registration), - // and record construction/update/index/field-access are `references` to the - // record's struct node. + // lives on the PARENT. Arity is part of a function's identity (#1610), so + // refs carry the call-site arity: remote calls are emitted as `mod::fn/2`, + // byte-identical to the qualifiedName the module namespace + arity suffix + // gives every function (see languages/erlang.ts), so they resolve via + // matchByQualifiedName; local calls are emitted `fn/2` and resolved by the + // erlang arity step in matchReference (same-file first). A var/macro callee + // or module (`F(X)`, `?M(X)`, `Mod:handle(X)`) has no static target — + // except `?MODULE:fn(X)`, which the bare-name-with-arity + same-file + // preference resolves correctly. `fun name/1` / `fun mod:name/1` values + // are function REFERENCES (callback registration) carrying their own + // written arity, and record construction/update/index/field-access are + // `references` to the record's struct node. if (this.language === 'erlang') { const line = node.startPosition.row + 1; const column = node.startPosition.column; @@ -3799,9 +3803,13 @@ export class TreeSitterExtractor { moduleExpr.type === 'macro_call_expr' ? getChildByField(moduleExpr, 'name') : null; if (!macroName || getNodeText(macroName, this.source) !== 'MODULE') return; } + // Arity from the call site's own argument list — part of the callee's + // identity, and what disambiguates `f/1` from `f/2` (#1610). + const callArgsNode = getChildByField(node, 'args'); + const callArity = callArgsNode ? callArgsNode.namedChildCount : 0; this.unresolvedReferences.push({ fromNodeId: callerId, - referenceName: calleeName, + referenceName: `${calleeName}/${callArity}`, referenceKind: 'calls', line, column, @@ -3824,9 +3832,10 @@ export class TreeSitterExtractor { const target = argsNode?.namedChild(0) ?? null; const targetModule = target ? this.resolveErlangGenServerTarget(target) : null; if (targetModule) { + // OTP fixes the handler arities: handle_call/3, handle_cast/2. this.unresolvedReferences.push({ fromNodeId: callerId, - referenceName: `${targetModule}::${fnBare === 'cast' ? 'handle_cast' : 'handle_call'}`, + referenceName: `${targetModule}::${fnBare === 'cast' ? 'handle_cast/2' : 'handle_call/3'}`, referenceKind: 'calls', line, column, @@ -3857,9 +3866,17 @@ export class TreeSitterExtractor { getChildByField(m, 'name') !== null && getNodeText(getChildByField(m, 'name')!, this.source) === 'MODULE'; if (m.type !== 'atom' && !isLocalModule) continue; + // Arity of the spawned/applied function = the length of the + // static args-list literal directly after the (M, F) pair, when + // present (`spawn_link(?MODULE, request_process, [Req, Env])` → + // /2). A var/absent list leaves the ref arity-less; the + // qualified matcher then resolves it only when the module + // defines exactly one arity of that name. + const mfaList = argExprs[i + 2]; + const arityTail = mfaList?.type === 'list' ? `/${mfaList.namedChildCount}` : ''; this.unresolvedReferences.push({ fromNodeId: callerId, - referenceName: isLocalModule ? erlAtom(f) : `${erlAtom(m)}::${erlAtom(f)}`, + referenceName: (isLocalModule ? erlAtom(f) : `${erlAtom(m)}::${erlAtom(f)}`) + arityTail, referenceKind: 'calls', line: f.startPosition.row + 1, column: f.startPosition.column, @@ -3880,6 +3897,11 @@ export class TreeSitterExtractor { if (moduleAtom?.type !== 'atom') return; refName = `${erlAtom(moduleAtom)}::${refName}`; } + // `fun f/1` writes its arity — carry it so the ref lands on the + // matching arity's node (#1610). + const funArityNode = getChildByField(node, 'arity'); + const funArityValue = funArityNode ? getChildByField(funArityNode, 'value') : null; + if (funArityValue) refName = `${refName}/${getNodeText(funArityValue, this.source)}`; this.unresolvedReferences.push({ fromNodeId: callerId, referenceName: refName, @@ -4430,6 +4452,26 @@ export class TreeSitterExtractor { } else { calleeName = methodName; } + } else if ( + this.language === 'rust' && + receiver && + receiver.type === 'field_expression' && + getChildByField(receiver, 'value')?.type === 'self' && + getChildByField(receiver, 'field')?.type === 'field_identifier' + ) { + // Rust `self..()` — a call through a field of the + // enclosing type (#1585). Keep the `self.` prefix: the resolver + // recognizes the shape, reads the field's declared type off the + // owner struct's declaration, and resolves the method on THAT + // type — or leaves the ref unresolved when the type is external + // or unknown. Previously this collapsed to the bare method name, + // which exact-matched whichever same-named method was nearest — + // often the calling method itself, a self-edge not in the source. + // Deeper chains (`self.a.b.m()`), `self.f().m()` and parenthesized + // receivers keep the bare name. Mirrored in the kernel's + // extract_call (rustlang.rs). + const fieldName = getNodeText(getChildByField(receiver, 'field')!, this.source); + calleeName = `self.${fieldName}.${methodName}`; } else if ( (this.language === 'cpp' || this.language === 'c' || @@ -5717,38 +5759,20 @@ export class TreeSitterExtractor { * For plain `impl Type { ... }` (no trait), no inheritance edge is needed. */ private extractRustImplItem(node: SyntaxNode): void { - // Check if this is `impl Trait for Type` by looking for a `for` keyword - const hasFor = node.children.some( - (c: SyntaxNode) => c.type === 'for' && !c.isNamed - ); - if (!hasFor) return; - - // In `impl Trait for Type`, the type_identifiers are: - // first = Trait name, last = implementing Type name - // Also handle generic types like `impl Trait for MyStruct` - const typeIdents = node.namedChildren.filter( - (c: SyntaxNode) => c.type === 'type_identifier' || c.type === 'generic_type' || c.type === 'scoped_type_identifier' - ); - if (typeIdents.length < 2) return; - - const traitNode = typeIdents[0]!; - const typeNode = typeIdents[typeIdents.length - 1]!; - - // Get the trait name (handle scoped paths like std::fmt::Display) - const traitName = traitNode.type === 'scoped_type_identifier' - ? this.source.substring(traitNode.startIndex, traitNode.endIndex) - : getNodeText(traitNode, this.source); - - // Get the implementing type name (extract inner type_identifier for generics) - let typeName: string; - if (typeNode.type === 'generic_type') { - const inner = typeNode.namedChildren.find( - (c: SyntaxNode) => c.type === 'type_identifier' - ); - typeName = inner ? getNodeText(inner, this.source) : getNodeText(typeNode, this.source); - } else { - typeName = getNodeText(typeNode, this.source); - } + // `impl Trait for Type` carries the trait in the grammar's `trait` field; + // an inherent `impl Type { … }` has none and needs no inheritance edge. + const traitNode = getChildByField(node, 'trait'); + if (!traitNode) return; + + // Full text, so a scoped path (`std::fmt::Display`) and a generic trait + // (`From`) keep their spelling. + const traitName = getNodeText(traitNode, this.source); + + // The implementing type from the `type` field (#1588). The old positional + // scan took the LAST type-shaped child, which for a parameterized + // implementing type (`BufSource`, `Parents<'a>`, `&Foo`) was the trait. + const typeName = rustImplTypeName(getChildByField(node, 'type'), this.source); + if (!typeName) return; // Find the struct/type node for the implementing type const typeNodeId = this.findNodeByName(typeName); diff --git a/src/index.ts b/src/index.ts index a05661c82..90397c55c 100644 --- a/src/index.ts +++ b/src/index.ts @@ -53,8 +53,10 @@ import { FileWatcher, WatchOptions, PendingFile, LockUnavailableError } from './ import { EXTRACTION_VERSION } from './extraction/extraction-version'; import { getCodeGraphDir } from './directory'; import { deriveProjectNameTokens } from './search/query-utils'; +import ignore from 'ignore'; +import { loadDeprioritizePatterns } from './project-config'; import { CodeGraphPackageVersion } from './mcp/version'; -import { segmentLookupVariants, splitIdentifierSegments } from './search/identifier-segments'; +import { extractSegmentSearchWords, segmentLookupVariants, splitIdentifierSegments } from './search/identifier-segments'; import { createYielder } from './resolution/cooperative-yield'; import { minRefsForPool } from './resolution/resolver-pool'; @@ -186,6 +188,39 @@ export class CodeGraph { } catch { // Best-effort: ranking still works without it. } + // Down-weight the peripheral trees the project named in `codegraph.json` + // `deprioritize` — indexed and findable, but never outranking real code + // (#982). Ranking-only, so a bad pattern costs relevance, never recall. + // + // Read LAZILY, not once here: `wireLayers` runs from the constructor and + // from `reopenIfReplaced`, so a matcher built here would freeze at whatever + // the config said when the project opened. The MCP server caches one + // CodeGraph per root for its whole lifetime, so editing `codegraph.json` + // would appear to do nothing until the process restarted — `exclude` and + // `include` do not behave that way. `loadDeprioritizePatterns` is + // mtime-cached, so this costs one `stat`; the compiled matcher is memoized + // on the pattern array's identity, which the cache keeps stable. + let cachedPatterns: string[] | undefined; + let cachedMatcher: ReturnType | undefined; + this.queries.setDeprioritizedPathMatcher((filePath: string): boolean => { + try { + const patterns = loadDeprioritizePatterns(this.projectRoot); + if (patterns.length === 0) return false; + if (patterns !== cachedPatterns) { + cachedPatterns = patterns; + cachedMatcher = ignore().add(patterns); + } + const rel = path.isAbsolute(filePath) + ? path.relative(this.projectRoot, filePath) + : filePath; + if (!rel || rel.startsWith('..')) return false; + return cachedMatcher!.ignores(rel.split(path.sep).join('/')); + } catch { + // Ranking must never take the search down with it. + return false; + } + }); + this.orchestrator = new ExtractionOrchestrator(this.projectRoot, this.queries); this.resolver = createResolver(this.projectRoot, this.queries); this.graphManager = new GraphQueryManager(this.queries); @@ -976,6 +1011,14 @@ export class CodeGraph { } } catch { /* vocab is advisory — never fail a sync over it */ } + // A killed full index leaves this marker at `indexing`. Sync repairs + // missing files, pending refs, and (on open) dropped indexes, so a + // successful recovery must also close the metadata state (#1556). + const fullReconcile = !options.paths || options.paths.length === 0; + if (fullReconcile && this.getIndexState() === 'indexing') { + try { this.queries.setMetadata('index_state', 'complete'); } catch { /* advisory */ } + } + return result; } finally { // Mirror indexAll's teardown: stop the valve, then restore the @@ -1823,7 +1866,23 @@ export class CodeGraph { query: string, options?: FindRelevantContextOptions ): Promise { - return this.contextBuilder.findRelevantContext(query, options); + // Segment-vocab supplement: FTS keeps camelCase names as single tokens, + // so a word-level query ("auto-scroll to bottom") can never reach + // `pinFeedIfNearBottom` through search alone. Resolve the query's words + // against name_segment_vocab (same precision rules as the prompt hook: + // co-occurrence, else rare singles, verified against live nodes) and hand + // the names down as dampened exact-name seeds. Callers that pass their + // own seedNames keep them; failures degrade to no supplement. + let seedNames = options?.seedNames; + if (seedNames === undefined) { + try { + seedNames = this.getSegmentMatches(extractSegmentSearchWords(query), 8) + .map((m) => m.name); + } catch { + seedNames = []; + } + } + return this.contextBuilder.findRelevantContext(query, { ...options, seedNames }); } /** diff --git a/src/installer/index.ts b/src/installer/index.ts index edeb4ac94..199a6de75 100644 --- a/src/installer/index.ts +++ b/src/installer/index.ts @@ -137,7 +137,7 @@ export async function runInstallerWithOptions(opts: RunInstallerOptions): Promis } else if (useDefaults) { location = 'global'; } else { - // If every selected target is global-only (e.g. Codex), skip the + // If every selected target is global-only (e.g. the Copilot CLI), skip the // prompt and force user-wide — project-local would just produce // skip warnings. const allGlobalOnly = targets.every((t) => !t.supportsLocation('local')); @@ -285,9 +285,10 @@ export async function runInstallerWithOptions(opts: RunInstallerOptions): Promis // index a surprise directory (e.g. a shell sitting in $HOME). Same next step // regardless of global/local scope. clack.note( - location === 'local' + (location === 'local' ? 'codegraph init # build this project’s graph (one time; auto-syncs after)' - : 'cd \ncodegraph init # build a project’s graph (one time; auto-syncs after)', + : 'cd \ncodegraph init # build a project’s graph (one time; auto-syncs after)') + + '\n# (codegraph install --init does both steps in one command)', 'Next: index a project', ); @@ -328,8 +329,8 @@ export type UninstallStatus = 'removed' | 'not-configured' | 'unsupported'; * Per-target outcome of an uninstall sweep. `removed` means we deleted * at least one thing; `not-configured` means the agent had no codegraph * config at this location (nothing to do); `unsupported` means the - * agent has no config concept for this location (e.g. Codex is - * global-only, so a `local` uninstall skips it). + * agent has no config concept for this location (e.g. the Copilot CLI + * is global-only, so a `local` uninstall skips it). */ export interface UninstallReport { id: TargetId; diff --git a/src/installer/targets/codex.ts b/src/installer/targets/codex.ts index 0a2b8c9c8..d5e361ce3 100644 --- a/src/installer/targets/codex.ts +++ b/src/installer/targets/codex.ts @@ -1,15 +1,29 @@ /** * OpenAI Codex CLI target. * - * - MCP server entry to `~/.codex/config.toml` as the dotted-key - * table `[mcp_servers.codegraph]`. TOML — not JSON — handled by - * the narrow serializer in `./toml.ts`. - * - Instructions to `~/.codex/AGENTS.md`. + * - MCP server entry to `config.toml` as the dotted-key table + * `[mcp_servers.codegraph]`. TOML — not JSON — handled by the + * narrow serializer in `./toml.ts`. + * - Instructions to `AGENTS.md`. * - * Codex CLI as of 2026-05 has no project-local config concept — - * everything lives under `~/.codex/`. `supportsLocation('local')` - * returns false; the orchestrator skips Codex when the user picks - * the local install location. + * Both locations are supported (#1531): + * - global: `~/.codex/config.toml` + `~/.codex/AGENTS.md` + * - local: `/.codex/config.toml` + `/AGENTS.md` + * + * Codex has a first-class project config layer: `.codex/config.toml` + * is layer 4 of the loader's stack, above the user config (layer 6), + * merged recursively top-over-bottom + * (`codex-rs/config/src/loader/README.md` in openai/codex). It landed + * in openai/codex#8354 (2025-12-22), so the "Codex has no + * project-local config" note this file used to carry was never + * accurate. The project layer strips a denylist of settings that + * repo contents shouldn't get to choose (base URLs, model providers, + * `notify`, profiles, otel — `loader/mod.rs`), and `mcp_servers` is + * NOT on it, so a project-scoped `[mcp_servers.codegraph]` is honored. + * + * Caveat surfaced as an install note: project layers are "loaded but + * disabled when untrusted," so a local install only takes effect in a + * project the user has marked trusted. * * No permissions concept. */ @@ -38,14 +52,31 @@ import { buildTomlTable, removeTomlTable, upsertTomlTable } from './toml'; const TOML_HEADER = 'mcp_servers.codegraph'; -function configDir(): string { - return path.join(os.homedir(), '.codex'); +function configDir(loc: Location): string { + return loc === 'global' + ? path.join(os.homedir(), '.codex') + : path.join(process.cwd(), '.codex'); +} +function tomlConfigPath(loc: Location): string { + return path.join(configDir(loc), 'config.toml'); } -function tomlConfigPath(): string { - return path.join(configDir(), 'config.toml'); +function instructionsPath(loc: Location): string { + // Global AGENTS.md lives under ~/.codex/; project-local AGENTS.md + // lives at the project root (NOT under .codex/) — that's the file + // Codex reads for repo instructions, and it matches the local + // layout the opencode and gemini targets already use. + return loc === 'global' + ? path.join(configDir('global'), 'AGENTS.md') + : path.join(process.cwd(), 'AGENTS.md'); } -function instructionsPath(): string { - return path.join(configDir(), 'AGENTS.md'); + +/** + * Project layers are "loaded but disabled when untrusted" (openai/codex + * `loader/mod.rs`), so a local install can be written correctly and + * still do nothing. Say so rather than reporting silent success. + */ +function trustNote(): string { + return `Codex applies ${tomlConfigPath('local')} only in a project marked trusted — otherwise the layer is loaded but disabled. Trust this project in Codex to activate it.`; } class CodexTarget implements AgentTarget { @@ -53,15 +84,12 @@ class CodexTarget implements AgentTarget { readonly displayName = 'Codex CLI'; readonly docsUrl = 'https://github.com/openai/codex'; - supportsLocation(loc: Location): boolean { - return loc === 'global'; + supportsLocation(_loc: Location): boolean { + return true; } detect(loc: Location): DetectionResult { - if (loc !== 'global') { - return { installed: false, alreadyConfigured: false }; - } - const tomlPath = tomlConfigPath(); + const tomlPath = tomlConfigPath(loc); let alreadyConfigured = false; if (fs.existsSync(tomlPath)) { try { @@ -69,34 +97,30 @@ class CodexTarget implements AgentTarget { alreadyConfigured = content.includes(`[${TOML_HEADER}]`); } catch { /* ignore */ } } - const installed = fs.existsSync(configDir()); + // Global: ~/.codex/ existing means Codex has run here. Local: the + // project only counts as "Codex-enabled" once it actually has a + // .codex/ dir or config file of its own. + const installed = fs.existsSync(configDir(loc)) || fs.existsSync(tomlPath); return { installed, alreadyConfigured, configPath: tomlPath }; } install(loc: Location, _opts: InstallOptions): WriteResult { - if (loc !== 'global') { - return { - files: [], - notes: ['Codex CLI has no project-local config — re-run with --location=global to install.'], - }; - } const files: WriteResult['files'] = []; - files.push(writeMcpEntry()); + files.push(writeMcpEntry(loc)); // AGENTS.md gets the short marker-fenced CodeGraph block (#704): // subagents and non-MCP harnesses read AGENTS.md but never the MCP // initialize instructions. Upsert self-heals a stale pre-#529 block. - files.push(upsertInstructionsEntry(instructionsPath())); + files.push(upsertInstructionsEntry(instructionsPath(loc))); - return { files }; + return loc === 'local' ? { files, notes: [trustNote()] } : { files }; } uninstall(loc: Location): WriteResult { - if (loc !== 'global') return { files: [] }; const files: WriteResult['files'] = []; - const tomlPath = tomlConfigPath(); + const tomlPath = tomlConfigPath(loc); if (fs.existsSync(tomlPath)) { const content = fs.readFileSync(tomlPath, 'utf-8'); const { content: nextContent, action } = removeTomlTable(content, TOML_HEADER); @@ -114,22 +138,18 @@ class CodexTarget implements AgentTarget { files.push({ path: tomlPath, action: 'not-found' }); } - files.push(removeInstructionsEntry()); + files.push(removeInstructionsEntry(loc)); return { files }; } printConfig(loc: Location): string { - if (loc !== 'global') { - return '# Codex CLI has no project-local config — use --location=global.\n'; - } const block = buildCodegraphBlock(); - return `# Add to ${tomlConfigPath()}\n\n${block}\n`; + return `# Add to ${tomlConfigPath(loc)}\n\n${block}\n`; } describePaths(loc: Location): string[] { - if (loc !== 'global') return []; - return [tomlConfigPath(), instructionsPath()]; + return [tomlConfigPath(loc), instructionsPath(loc)]; } } @@ -141,8 +161,8 @@ function buildCodegraphBlock(): string { }); } -function writeMcpEntry(): WriteResult['files'][number] { - const file = tomlConfigPath(); +function writeMcpEntry(loc: Location): WriteResult['files'][number] { + const file = tomlConfigPath(loc); const dir = path.dirname(file); if (!fs.existsSync(dir)) fs.mkdirSync(dir, { recursive: true }); @@ -162,12 +182,12 @@ function writeMcpEntry(): WriteResult['files'][number] { } /** - * Strip the marker-delimited CodeGraph block from `~/.codex/AGENTS.md` - * if a prior install wrote one. Used by both install (self-heal on - * upgrade) and uninstall — see issue #529. + * Strip the marker-delimited CodeGraph block from this location's + * AGENTS.md if a prior install wrote one. Used by both install + * (self-heal on upgrade) and uninstall — see issue #529. */ -function removeInstructionsEntry(): WriteResult['files'][number] { - const file = instructionsPath(); +function removeInstructionsEntry(loc: Location): WriteResult['files'][number] { + const file = instructionsPath(loc); const action = removeMarkedSection(file, CODEGRAPH_SECTION_START, CODEGRAPH_SECTION_END); return { path: file, action }; } diff --git a/src/installer/targets/types.ts b/src/installer/targets/types.ts index 022ab28e8..d93680573 100644 --- a/src/installer/targets/types.ts +++ b/src/installer/targets/types.ts @@ -87,10 +87,10 @@ export interface AgentTarget { /** * Whether this target supports the given install location. * - * Some agents (Codex CLI as of 2026-05) have no project-local - * config concept — only a single `~/.codex/` dir. Returning false - * for an unsupported (target, location) pair lets the orchestrator - * skip cleanly with a clear message. + * Some agents (GitHub Copilot CLI, the Copilot JetBrains plugin) + * have no project-local config concept — only a single per-user + * config dir. Returning false for an unsupported (target, location) + * pair lets the orchestrator skip cleanly with a clear message. */ supportsLocation(loc: Location): boolean; detect(loc: Location): DetectionResult; diff --git a/src/mcp/daemon-manager.ts b/src/mcp/daemon-manager.ts index 47a61e077..0c1a991a1 100644 --- a/src/mcp/daemon-manager.ts +++ b/src/mcp/daemon-manager.ts @@ -61,7 +61,7 @@ export function buildPickItems(daemons: DaemonRecord[], cwdRoot: string | null, } export interface PickerDeps { - list: () => DaemonRecord[]; + list: () => DaemonRecord[] | Promise; stop: (root: string) => Promise; stopAll: () => Promise; /** Realpath'd root of the current project's daemon, or null. */ @@ -82,7 +82,7 @@ export interface PickerDeps { */ export async function runDaemonPicker(deps: PickerDeps): Promise { for (;;) { - const daemons = deps.list(); + const daemons = await deps.list(); if (daemons.length === 0) { deps.done('All daemons stopped.'); return; diff --git a/src/mcp/daemon-paths.ts b/src/mcp/daemon-paths.ts index 13f19045f..c860ee76c 100644 --- a/src/mcp/daemon-paths.ts +++ b/src/mcp/daemon-paths.ts @@ -29,6 +29,7 @@ */ import * as crypto from 'crypto'; +import * as net from 'net'; import * as os from 'os'; import * as path from 'path'; import { getCodeGraphDir } from '../directory'; @@ -101,6 +102,55 @@ export interface DaemonLockInfo { startedAt: number; } +/** + * Verify that the process named by a lockfile is the CodeGraph daemon serving + * its socket. A bare PID liveness probe is insufficient because OSes reuse PIDs + * after an OOM/SIGKILL (#1553). + */ +export function probeDaemonIdentity(info: DaemonLockInfo, timeoutMs = 1_000): Promise { + if (!Number.isInteger(info.pid) || info.pid <= 0 || !info.socketPath) return Promise.resolve(false); + return new Promise((resolve) => { + let socket: net.Socket; + let buffer = ''; + let done = false; + const finish = (ok: boolean) => { + if (done) return; + done = true; + clearTimeout(timer); + socket.destroy(); + resolve(ok); + }; + const timer = setTimeout(() => finish(false), timeoutMs); + timer.unref?.(); + try { + socket = net.createConnection(info.socketPath); + } catch { + clearTimeout(timer); + resolve(false); + return; + } + socket.setEncoding('utf8'); + socket.on('data', (chunk) => { + buffer += String(chunk); + if (buffer.length > 4096) return finish(false); + const newline = buffer.indexOf('\n'); + if (newline < 0) return; + try { + const hello = JSON.parse(buffer.slice(0, newline)) as Record; + finish( + hello.protocol === 1 && + hello.pid === info.pid && + (info.version === 'unknown' || hello.codegraph === info.version) + ); + } catch { + finish(false); + } + }); + socket.on('error', () => finish(false)); + socket.on('close', () => finish(false)); + }); +} + /** * Serialize a {@link DaemonLockInfo} for writing to the pidfile. JSON for * human readability — operators occasionally `cat` this when debugging. diff --git a/src/mcp/daemon-registry.ts b/src/mcp/daemon-registry.ts index e1885361c..f731563cf 100644 --- a/src/mcp/daemon-registry.ts +++ b/src/mcp/daemon-registry.ts @@ -22,7 +22,13 @@ import * as fs from 'fs'; import * as os from 'os'; import * as path from 'path'; import * as crypto from 'crypto'; -import { getDaemonPidPath, getDaemonSocketCandidates, decodeLockInfo } from './daemon-paths'; +import { + getDaemonPidPath, + getDaemonSocketCandidates, + decodeLockInfo, + probeDaemonIdentity, + type DaemonLockInfo, +} from './daemon-paths'; export interface DaemonRecord { /** Realpath'd project root the daemon serves. */ @@ -114,6 +120,26 @@ export function listDaemons(opts: { prune?: boolean } = {}): DaemonRecord[] { return live.sort((a, b) => b.startedAt - a.startedAt); } +/** + * Registry entries whose socket hello proves the recorded process is the + * daemon. Used by every user-facing list/stop-all path so a reused PID cannot + * appear as a phantom running daemon (#1553). + */ +export async function listVerifiedDaemons(opts: { prune?: boolean } = {}): Promise { + const prune = opts.prune ?? true; + const candidates = listDaemons({ prune }); + const checks = await Promise.all(candidates.map(async (rec) => ({ + rec, + verified: await probeDaemonIdentity(rec), + }))); + const verified: DaemonRecord[] = []; + for (const check of checks) { + if (check.verified) verified.push(check.rec); + else if (prune) deregisterDaemon(check.rec.root); + } + return verified; +} + /** Remove a stopped daemon's leftover lockfile + socket + registry record. */ function cleanupDaemonArtifacts(root: string): void { try { fs.unlinkSync(getDaemonPidPath(root)); } catch { /* gone */ } @@ -128,6 +154,20 @@ function cleanupDaemonArtifacts(root: string): void { deregisterDaemon(root); } +/** Remove daemon artifacts only when no matching daemon answers the socket hello. */ +export async function clearStaleDaemonArtifacts(root: string): Promise { + const pidPath = getDaemonPidPath(root); + const hadArtifacts = fs.existsSync(pidPath) || ( + process.platform !== 'win32' && getDaemonSocketCandidates(root).some((p) => fs.existsSync(p)) + ); + if (!hadArtifacts) return false; + let info: DaemonLockInfo | null = null; + try { info = decodeLockInfo(fs.readFileSync(pidPath, 'utf8')); } catch { /* missing/corrupt */ } + if (info && isProcessAlive(info.pid) && await probeDaemonIdentity(info)) return false; + cleanupDaemonArtifacts(root); + return true; +} + const sleep = (ms: number): Promise => new Promise((r) => setTimeout(r, ms)); async function waitForDeath(pid: number, timeoutMs: number): Promise { @@ -154,9 +194,10 @@ export interface StopResult { */ export async function stopDaemonAt(root: string): Promise { let pid: number | null = null; + let identity: DaemonLockInfo | null = null; try { - const info = decodeLockInfo(fs.readFileSync(getDaemonPidPath(root), 'utf8')); - pid = info?.pid ?? null; + identity = decodeLockInfo(fs.readFileSync(getDaemonPidPath(root), 'utf8')); + pid = identity?.pid ?? null; } catch { /* no lockfile */ } @@ -165,6 +206,7 @@ export async function stopDaemonAt(root: string): Promise { (r) => path.resolve(r.root) === path.resolve(root) ); pid = rec?.pid ?? null; + if (rec) identity = rec; } if (pid == null) { @@ -175,6 +217,12 @@ export async function stopDaemonAt(root: string): Promise { cleanupDaemonArtifacts(root); return { root, pid, outcome: 'not-running' }; } + // Never signal a process merely because it reused a stale daemon PID. The + // daemon's immediate hello is the process-identity proof (#1553). + if (!identity || !await probeDaemonIdentity(identity)) { + cleanupDaemonArtifacts(root); + return { root, pid, outcome: 'not-running' }; + } // POSIX: SIGTERM runs the daemon's graceful shutdown. Windows: TerminateProcess // (no graceful path), so we always sweep artifacts ourselves below. @@ -192,7 +240,7 @@ export async function stopDaemonAt(root: string): Promise { /** Stop every registered, live daemon. */ export async function stopAllDaemons(): Promise { const results: StopResult[] = []; - for (const rec of listDaemons()) { + for (const rec of await listVerifiedDaemons()) { results.push(await stopDaemonAt(rec.root)); } return results; diff --git a/src/mcp/daemon.ts b/src/mcp/daemon.ts index b1d45328b..500c48a8c 100644 --- a/src/mcp/daemon.ts +++ b/src/mcp/daemon.ts @@ -629,25 +629,31 @@ export function acquireLockViaExclusiveOpen(pidPath: string, info: DaemonLockInf } /** - * Remove a stale pidfile, but only if it still names a dead process. Re-reads - * the file immediately before unlinking so we never delete a lock that a live - * daemon (re)acquired in the meantime. + * Remove a stale pidfile. Re-reads the file immediately before unlinking so a + * different daemon that acquired the lock in the meantime is never disturbed. * * must-fix 1 (issue #411 review): the original unconditionally `unlink`'d, * which let a racing candidate delete a healthy daemon's lock. Passing * `expectedDeadPid` (the pid the caller believed was dead) makes the clear a - * compare-and-delete: bail if the file now holds a different pid, or any live - * pid. Returns true when the stale lock is gone (or was already gone). + * compare-and-delete: bail if the file now holds a different pid. By default a + * live pid is also preserved; `allowLivePid` is reserved for callers that have + * already disproved daemon identity with the socket hello (#1553). Returns true + * when the stale lock is gone (or was already gone). */ -export function clearStaleDaemonLock(pidPath: string, expectedDeadPid?: number): boolean { +export function clearStaleDaemonLock( + pidPath: string, + expectedDeadPid?: number, + opts: { allowLivePid?: boolean } = {} +): boolean { try { const raw = fs.readFileSync(pidPath, 'utf8'); const info = decodeLockInfo(raw); if (info) { // A different pid took over since we read it — not ours to clear. if (expectedDeadPid !== undefined && info.pid !== expectedDeadPid) return false; - // Holder is actually alive — never clear a live daemon's lock. - if (info.pid > 0 && isProcessAlive(info.pid)) return false; + // PID liveness is normally sufficient. The takeover caller may override + // it only after a failed identity handshake proves PID reuse. + if (!opts.allowLivePid && info.pid > 0 && isProcessAlive(info.pid)) return false; } fs.unlinkSync(pidPath); return true; diff --git a/src/mcp/engine.ts b/src/mcp/engine.ts index 9ee132d64..8f2e5b6ef 100644 --- a/src/mcp/engine.ts +++ b/src/mcp/engine.ts @@ -11,8 +11,9 @@ */ import * as os from 'os'; +import * as path from 'path'; import type CodeGraph from '../index'; -import { findNearestCodeGraphRoot } from '../directory'; +import { resolveServerRoot } from '../directory'; import { watchDisabledReason } from '../sync'; import { ToolHandler } from './tools'; import { QueryPool, resolvePoolSize } from './query-pool'; @@ -26,6 +27,9 @@ import { QueryPool, resolvePoolSize } from './query-pool'; const loadCodeGraph = (): typeof import('../index').default => (require('../index') as typeof import('../index')).default; +/** How often the per-tool-call retry may re-run the sub-project down-scan. */ +const RETRY_SUBSCAN_TTL_MS = 5_000; + export interface MCPEngineOptions { /** * Whether to start the file watcher when initializing. Daemon and direct @@ -59,6 +63,9 @@ export class MCPEngine { private projectPath: string | null = null; // Set on first `ensureInitialized` so subsequent sessions don't redo work. private initPromise: Promise | null = null; + // Throttle for the retry path's sub-project down-scan (#1606) — the scan is + // bounded but shouldn't run on every tool call in the no-default state. + private lastRetrySubScanAt = 0; private watcherStarted = false; private opts: Required; private closed = false; @@ -158,8 +165,20 @@ export class MCPEngine { if (this.closed) return; if (this.toolHandler.hasDefaultCodeGraph()) return; this.toolHandler.setDefaultProjectHint(searchFrom); - const resolvedRoot = findNearestCodeGraphRoot(searchFrom); + // Same resolution `doInitialize` used: up-walk, then the bounded workspace + // down-scan (#1606) — this retry is exactly the path that picks up a + // project (root or child) `codegraph init`'d after the server started. The + // down-scan is throttled so the persistent no-default state doesn't pay a + // directory walk on every tool call; the up-walk always runs. + const scanDue = Date.now() - this.lastRetrySubScanAt >= RETRY_SUBSCAN_TTL_MS; + const res = resolveServerRoot(searchFrom, { subprojectScan: scanDue }); + if (scanDue) { + this.lastRetrySubScanAt = Date.now(); + if (!res.root) this.toolHandler.setKnownSubprojects(res.candidates, searchFrom); + } + const resolvedRoot = res.root; if (!resolvedRoot) return; + if (res.viaSubScan) this.logSubprojectAdoption(searchFrom, resolvedRoot); try { // Close any previously failed instance to avoid leaking resources. if (this.cg) { @@ -201,12 +220,32 @@ export class MCPEngine { private async doInitialize(searchFrom: string): Promise { this.toolHandler.setDefaultProjectHint(searchFrom); - const resolvedRoot = findNearestCodeGraphRoot(searchFrom); + // Up-walk first; when nothing is indexed at or above searchFrom, a bounded + // down-scan may adopt a SINGLE indexed sub-project as the default (#1606 — + // the workspace-container shape where only children are indexed). Zero or + // several candidates → no default project, but SAY so (#1607): the silent + // variant of this state read as "CodeGraph is broken" and was diagnosable + // only by knowing to look for a missing ~/.codegraph/daemons/ entry. + const res = resolveServerRoot(searchFrom); + const resolvedRoot = res.root; if (!resolvedRoot) { - // No .codegraph/ above searchFrom. Sessions may still discover one later via roots/list + // Sessions may still discover a project later via roots/list, and the + // per-call retry re-resolves — this state is recoverable, hence stderr + // (not a failure) + candidates surfaced through the tool-call error. this.projectPath = searchFrom; + this.toolHandler.setKnownSubprojects(res.candidates, searchFrom); + process.stderr.write( + `[CodeGraph MCP] No .codegraph/ at or above ${searchFrom}: no default project, live sync disabled.\n` + ); + if (res.candidates.length > 0) { + const rels = res.candidates.map((c) => path.relative(searchFrom, c) || '.'); + process.stderr.write( + `[CodeGraph MCP] Indexed sub-projects found: ${rels.join(', ')}. Pass \`projectPath\` per call, or launch with --path.\n` + ); + } return; } + if (res.viaSubScan) this.logSubprojectAdoption(searchFrom, resolvedRoot); this.projectPath = resolvedRoot; try { @@ -221,6 +260,14 @@ export class MCPEngine { } } + /** One stderr line when the default project came from the down-scan (#1606). */ + private logSubprojectAdoption(searchFrom: string, root: string): void { + const rel = path.relative(searchFrom, root) || root; + process.stderr.write( + `[CodeGraph MCP] No .codegraph/ at ${searchFrom}; adopted the single indexed sub-project ${rel} as the default project.\n` + ); + } + /** * Start file watching on the active CodeGraph instance. Idempotent — the * watcher is per-engine, not per-session, which is why the daemon path diff --git a/src/mcp/explore-diagnostics.ts b/src/mcp/explore-diagnostics.ts index 4de9c605c..0c654e025 100644 --- a/src/mcp/explore-diagnostics.ts +++ b/src/mcp/explore-diagnostics.ts @@ -60,6 +60,8 @@ export interface ExploreCandidateMeta { graphScore: number; termHits: number; nodes: number; + /** The query named this file by PATH — pinned rank/allocation treatment. */ + pinned?: boolean; named: boolean; central: boolean; entry: boolean; @@ -579,6 +581,7 @@ export class ExploreDiagnostics { graphScore: round6(r.graphScore), termHits: r.termHits, nodes: r.nodes, + pinned: r.pinned ?? false, named: r.named, central: r.central, entry: r.entry, @@ -797,6 +800,7 @@ export function renderTable(report: ExploreDiagnosticReport): string { function flagString(f: ExploreDiagnosticFile): string { const flags: string[] = []; + if (f.pinned) flags.push('pinned'); if (f.named) flags.push('named'); if (f.entry) flags.push('entry'); if (f.central) flags.push('central'); diff --git a/src/mcp/index.ts b/src/mcp/index.ts index c7c59f622..3f57024d2 100644 --- a/src/mcp/index.ts +++ b/src/mcp/index.ts @@ -37,7 +37,7 @@ import * as fs from 'fs'; import * as path from 'path'; import { spawn, StdioOptions } from 'child_process'; -import { findNearestCodeGraphRoot, getCodeGraphDir } from '../directory'; +import { resolveServerRoot, getCodeGraphDir } from '../directory'; import { StdioTransport } from './transport'; import { MCPEngine } from './engine'; import { MCPSession } from './session'; @@ -48,7 +48,7 @@ import { tryAcquireDaemonLock, } from './daemon'; import { connectWithHello, runLocalHandshakeProxy } from './proxy'; -import { getDaemonSocketCandidates } from './daemon-paths'; +import { getDaemonSocketCandidates, probeDaemonIdentity } from './daemon-paths'; import { getTelemetry } from '../telemetry'; import { checkForUpdateInBackground } from '../upgrade/update-check'; import { EARLY_PPID } from './early-ppid'; @@ -150,6 +150,12 @@ export function watchdogProgressPaths(root: string | null): { progressPaths?: st * that case the caller must run in direct mode, since the daemon lockfile * and socket both live under `.codegraph/`. * + * Uses the same resolution as the engine (#1606): up-walk first, then the + * bounded workspace down-scan that adopts a SINGLE indexed sub-project. A + * workspace root above one indexed child therefore gets the shared daemon + * (one watcher, one writer, keyed on the child) instead of a direct-mode + * server per host. + * * The result is canonicalized with `realpathSync` so every client converges on * the same socket/lock path regardless of how it expressed the path: a client * launched with cwd under a symlink (e.g. macOS `/var` → `/private/var`, where @@ -159,7 +165,7 @@ export function watchdogProgressPaths(root: string | null): { progressPaths?: st */ function resolveDaemonRoot(explicitPath: string | null): string | null { const candidate = explicitPath ?? process.cwd(); - const root = findNearestCodeGraphRoot(candidate); + const root = resolveServerRoot(candidate).root; if (!root) return null; try { return fs.realpathSync(root); } catch { return root; } } @@ -423,15 +429,22 @@ export class MCPServer { // binding) — we're redundant; exit cleanly so the launcher proxies to it. const existing = lock.existing; if (existing && existing.pid > 0 && isProcessAlive(existing.pid)) { - process.stderr.write( - `[CodeGraph daemon] Another daemon (pid ${existing.pid}) already holds the lock; exiting.\n` - ); - process.exit(0); + // Give a newly-elected daemon time to bind, then require its socket hello + // to match the lock PID/version. PID existence alone accepts an unrelated + // process after OS PID reuse and permanently wedges startup (#1553). + const age = Date.now() - existing.startedAt; + const stillStarting = existing.startedAt > 0 && age >= 0 && age < 10_000; + if (stillStarting || await probeDaemonIdentity(existing)) { + process.stderr.write( + `[CodeGraph daemon] Another daemon (pid ${existing.pid}) already holds the lock; exiting.\n` + ); + process.exit(0); + } } // Holder is dead (or the record is unreadable) — clear it (pid-verified, // so we never delete a live daemon's lock) and retry the acquire. - clearStaleDaemonLock(lock.pidPath, existing?.pid); + clearStaleDaemonLock(lock.pidPath, existing?.pid, { allowLivePid: true }); await sleep(TAKEOVER_RETRY_DELAY_MS); } diff --git a/src/mcp/session.ts b/src/mcp/session.ts index 866e0014a..1d5bd79c3 100644 --- a/src/mcp/session.ts +++ b/src/mcp/session.ts @@ -18,7 +18,7 @@ import { MCPEngine } from './engine'; import { tools } from './tools'; import { SERVER_INSTRUCTIONS, SERVER_INSTRUCTIONS_NO_ROOT_INDEX } from './server-instructions'; import { CodeGraphPackageVersion } from './version'; -import { findNearestCodeGraphRoot } from '../directory'; +import { resolveServerRoot } from '../directory'; import { getTelemetry, ClientInfo } from '../telemetry'; import { getUpdateNotice } from '../upgrade/update-check'; import { ExploreSessionState } from './explore-session-state'; @@ -230,18 +230,23 @@ export class MCPSession { explicitPath = this.explicitProjectPath; } - // Pick the instructions variant by the root's index state — a cheap - // synchronous walk-up (existsSync loop only, no DB open, so the #172 - // respond-fast contract holds). When the root IS indexed, send the full - // single-project playbook. When it ISN'T, send the per-project variant - // (tools are still exposed — see handleToolsList): it tells the agent there - // is no default project and to pass `projectPath` to any project that has a - // `.codegraph/`. Gating tool AVAILABILITY on whether `./` is indexed was the - // #964 bug — it broke monorepos (only sub-projects indexed) and never - // surfaced the tools after a mid-session `codegraph init`. When no explicit - // path is known yet (roots/list dance pending), cwd is the best predictor of - // where the default project will resolve. - const indexed = findNearestCodeGraphRoot(explicitPath ?? process.cwd()) !== null; + // Pick the instructions variant by the root's index state — synchronous + // and bounded (an existsSync walk-up plus, when that misses, the depth- and + // count-bounded workspace down-scan; no DB open, so the #172 respond-fast + // contract holds). This is the SAME resolution the engine's doInitialize + // runs (#1606), so the variant matches what the engine will actually adopt + // — a workspace whose single indexed sub-project becomes the default gets + // the full single-project playbook, race-free by construction (both sides + // compute it independently; no ordering between handshake and engine init + // is assumed). When the root ISN'T indexed (and nothing was adopted), send + // the per-project variant (tools are still exposed — see handleToolsList): + // it tells the agent there is no default project and to pass `projectPath` + // to any project that has a `.codegraph/`. Gating tool AVAILABILITY on + // whether `./` is indexed was the #964 bug — it broke monorepos (only + // sub-projects indexed) and never surfaced the tools after a mid-session + // `codegraph init`. When no explicit path is known yet (roots/list dance + // pending), cwd is the best predictor of where the default will resolve. + const indexed = resolveServerRoot(explicitPath ?? process.cwd()).root !== null; // Respond to the handshake BEFORE doing any heavy init — see issue #172. this.transport.sendResult(request.id, { diff --git a/src/mcp/tools.ts b/src/mcp/tools.ts index d1d013514..5c23f675d 100644 --- a/src/mcp/tools.ts +++ b/src/mcp/tools.ts @@ -32,6 +32,7 @@ import { import type { PendingFile } from '../sync'; import type { Node, Edge, SearchResult, Subgraph, NodeKind } from '../types'; import { isTestFile, normalizeNameToken } from '../search/query-utils'; +import { extractQueryPaths, queryMightContainPaths } from '../search/query-paths'; import { existsSync, readFileSync, @@ -81,7 +82,7 @@ export class NotIndexedError extends Error {} * retry guidance — abandoning this path is the desired agent reaction. */ export class PathRefusalError extends Error {} -import { resolve as resolvePath } from 'path'; +import { resolve as resolvePath, relative as relativePath } from 'path'; /** Maximum output length to prevent context bloat (characters) */ const MAX_OUTPUT_LENGTH = 15000; @@ -121,9 +122,14 @@ const CONTAINER_NODE_KINDS = new Set([ 'class', 'struct', 'union', 'interface', 'trait', 'protocol', 'enum', 'namespace', 'module', ]); -/** Last `::` / `.` / `/`-separated segment of a qualified symbol. */ +/** + * Last `::` / `.` / `/`-separated segment of a qualified symbol. An Erlang + * arity tail (`mod::fn/3`, `fn/3`) is stripped first — the useful last segment + * is the function name, never the digits (#1610). + */ function lastQualifierPart(symbol: string): string { - const parts = symbol.split(/::|[./]/).filter((p) => p.length > 0); + const noArity = symbol.replace(/\/\d{1,3}$/, '') || symbol; + const parts = noArity.split(/::|[./]/).filter((p) => p.length > 0); return parts[parts.length - 1] ?? symbol; } @@ -631,6 +637,13 @@ export interface ExploreAllocationCandidate { worth: number; /** Carries a symbol on the rendered flow spine. */ spine: boolean; + /** + * The query named this file by PATH (see query-paths.ts). Pinned files are + * never cliffed or trimmed, and weigh at least as much as the strongest + * candidate — the agent asked for the file itself, so starving it on text/ + * graph scores (which a pure-path query doesn't produce) defeats the ask. + */ + pinned?: boolean; } export interface ExploreAllocation { @@ -676,7 +689,16 @@ export function allocateExploreBudget( return Number.isFinite(w) ? w : 0; }; - const weights = new Map(candidates.map((c) => [c.path, weightOf(c)])); + // Pinned files weigh at least as much as the strongest raw candidate: their + // score is whatever the stripped query happened to match (for a pure-path + // query, nearly nothing), and a proportional split on that would fund the + // named file worst of all. Floor of 1 covers the all-pinned/zero-score case. + const rawWeights = new Map(candidates.map((c) => [c.path, weightOf(c)])); + const topRaw = Math.max(...rawWeights.values()); + const weights = new Map(candidates.map((c) => [ + c.path, + c.pinned ? Math.max(rawWeights.get(c.path) ?? 0, topRaw, 1) : (rawWeights.get(c.path) ?? 0), + ])); const topWeight = Math.max(...weights.values()); if (!(topWeight > 0)) return empty; @@ -686,7 +708,7 @@ export function allocateExploreBudget( const cliffed: string[] = []; let admitted: ExploreAllocationCandidate[] = []; for (const c of candidates) { - if (!c.spine && (weights.get(c.path) ?? 0) < cliffAt) cliffed.push(c.path); + if (!c.spine && !c.pinned && (weights.get(c.path) ?? 0) < cliffAt) cliffed.push(c.path); else admitted.push(c); } // Never cliff every candidate: an empty response costs a whole round-trip. @@ -705,7 +727,7 @@ export function allocateExploreBudget( if (admitted.length > affordable) { const byWeight = [...admitted].sort((a, b) => (weights.get(b.path) ?? 0) - (weights.get(a.path) ?? 0)); const keep = new Set(byWeight.slice(0, affordable).map((c) => c.path)); - for (const c of admitted) if (c.spine) keep.add(c.path); + for (const c of admitted) if (c.spine || c.pinned) keep.add(c.path); for (const c of admitted) if (!keep.has(c.path)) cliffed.push(c.path); admitted = admitted.filter((c) => keep.has(c.path)); } @@ -1297,6 +1319,13 @@ export class ToolHandler { // The directory the server last searched for a default project. Surfaced in // the "not initialized" error so users can see why detection missed. private defaultProjectHint: string | null = null; + // Indexed sub-projects the engine's bounded down-scan saw below the search + // base when no default project resolved (#1607). Listed in the "not + // initialized" error so the fact is reachable through the protocol, not just + // the host's stderr capture. Engine-maintained (initial resolve + throttled + // retry) — tool calls themselves never scan. + private knownSubprojects: string[] = []; + private knownSubprojectsBase: string | null = null; // Per-start-path cache of the git worktree/index mismatch (issue #155). The // mismatch is a fixed property of (where the request came from → which // .codegraph/ it resolves to), so the up-to-two `git rev-parse` spawns run @@ -1394,6 +1423,27 @@ export class ToolHandler { this.defaultProjectHint = searchedPath; } + /** + * Engine-only: record the indexed sub-projects the workspace down-scan saw + * when it could not adopt a default project (#1606/#1607). An empty list + * clears any previous note. + */ + setKnownSubprojects(roots: string[], base: string): void { + this.knownSubprojects = roots; + this.knownSubprojectsBase = base; + } + + /** One message line naming the indexed sub-projects, or '' when none known. */ + private formatKnownSubprojects(): string { + if (this.knownSubprojects.length === 0) return ''; + const base = this.knownSubprojectsBase; + const rels = this.knownSubprojects.map((r) => (base ? relativePath(base, r) || '.' : r)); + return ( + `Indexed sub-projects were found below it: ${rels.join(', ')} — ` + + 'pass one of them (absolute, or resolved against that directory) as projectPath.\n' + ); + } + /** * Whether a default CodeGraph instance is available */ @@ -1516,6 +1566,7 @@ export class ToolHandler { throw new NotIndexedError( 'No CodeGraph project is loaded for this session.\n' + `Searched for a .codegraph/ directory starting from: ${searched}\n` + + this.formatKnownSubprojects() + 'Either the server root has no index of its own (e.g. a monorepo where only ' + "sub-projects are indexed), or the MCP client launched the server outside your " + 'project without reporting the workspace root. Either way, target the project ' + @@ -3223,6 +3274,34 @@ export class ToolHandler { } const maxFiles = clamp((args.maxFiles as number) || budget.defaultMaxFiles, 1, 20); + // File paths named in the query become PINNED files: guaranteed admission, + // top of the rank order, funded first — and their span is REMOVED from the + // matching query. Runs on the RAW query (normalizeQuerySpelling strips + // `/digits` tails, which would mangle numeric path segments). Without this, + // a SvelteKit path like `runs/[runId]/+page.svelte` was shredded by the + // seeding tokenizer (splits on brackets → `runId` seeded as a "named + // symbol") and by FTS (`page`/`runs` fragments admitted every sibling + // `+page.svelte`), starving the very files the agent asked for. + let pinnedFiles: string[] = []; + let unresolvedPathSpans: string[] = []; + let matchQuery = query; + if (queryMightContainPaths(rawQuery)) { + try { + const extraction = extractQueryPaths( + rawQuery, + cg.getFiles().map((f) => f.path), + { maxPins: maxFiles }, + ); + if (extraction.pinnedFiles.length > 0 || extraction.unresolvedPathSpans.length > 0) { + pinnedFiles = extraction.pinnedFiles; + unresolvedPathSpans = extraction.unresolvedPathSpans; + matchQuery = normalizeQuerySpelling(extraction.strippedQuery); + } + } catch { /* path pinning must never fail an explore call */ } + } + const pinnedSet = new Set(pinnedFiles); + const pinnedOrder = new Map(pinnedFiles.map((p, i) => [p, i])); + // Per-file allocation diagnostic (CG-4). `null` unless CODEGRAPH_EXPLORE_DEBUG // is set — every `diag?.` below is then a no-op and the response is // byte-identical. It only OBSERVES: it must never feed back into rendering. @@ -3279,16 +3358,34 @@ export class ToolHandler { // Use a large maxNodes budget — explore has its own 35k char output limit // that prevents context bloat, so more nodes just means better coverage // across entry points (especially for large files like Svelte components). - const subgraph = await cg.findRelevantContext(query, { + // Matching runs on the path-stripped query; `query` stays for display. + const subgraph = await cg.findRelevantContext(matchQuery, { searchLimit: 8, traversalDepth: 3, maxNodes: 200, minScore: 0.2, }); + // Pinned files' symbols enter the gather unconditionally — the agent named + // the file itself, so its contents ARE the answer regardless of what the + // stripped query text matched (which, for a pure-path query, is nothing). + const PINNED_FILE_NODE_CAP = 300; + for (const fp of pinnedFiles) { + let fileNodes: Node[] = []; + try { fileNodes = cg.getNodesInFile(fp); } catch { continue; } + fileNodes + .filter((n) => n.kind !== 'file' && n.kind !== 'import' && n.kind !== 'export') + .sort((a, b) => a.startLine - b.startLine) + .slice(0, PINNED_FILE_NODE_CAP) + .forEach((n) => { if (!subgraph.nodes.has(n.id)) subgraph.nodes.set(n.id, n); }); + } + if (subgraph.nodes.size === 0) { diag?.finishEmpty('no relevant code found — empty subgraph'); - const empty = `No relevant code found for "${query}"`; + const missNote = unresolvedPathSpans.length > 0 + ? ` (no indexed file uniquely matches ${unresolvedPathSpans.map((s) => `\`${s}\``).join(', ')})` + : ''; + const empty = `No relevant code found for "${query}"${missNote}`; // Still an explore call, so it is still recorded: an empty answer spends a // call against the tier budget even though it emits no source. return this.exploreResult(empty, { @@ -3351,11 +3448,18 @@ export class ToolHandler { { const FILE_EXT = /\.(?:java|kt|kts|ts|tsx|js|jsx|mjs|cjs|cs|py|go|rb|php|swift|rs|cpp|cc|cxx|c|h|hpp|scala|lua|dart|vue|svelte|astro|erl|hrl)$/i; const CALLABLE = new Set(['method', 'function', 'component', 'constructor']); + // Variables/constants seed too: in Svelte/React a `$state` variable + // (`chatAtBottom`, `feedAtBottom`) is exactly the kind of symbol an agent + // names in a query, and the exact-name search channel already returns + // them — only this seeding tier was callable-only. The NL-stopword guard + // below applies unchanged, so bare English words still can't seed a + // same-named local. Callables keep priority via the body-size sort. + const SEEDABLE = new Set([...CALLABLE, 'variable', 'constant']); const isTestPath = (p: string) => /(^|\/)(tests?|specs?|__tests__|testdata|mocks?|fixtures?)\//i.test(p) || /\.(test|spec)\.[a-z]+$/i.test(p); const bodyLines = (n: Node) => Math.max(0, (n.endLine ?? n.startLine) - n.startLine); const callerCount = (n: Node) => { try { return cg.getCallers(n.id).length; } catch { return 0; } }; const tokens = [...new Set( - query.split(/[\s,()[\]]+/) + matchQuery.split(/[\s,()[\]]+/) .map((t) => t.replace(FILE_EXT, '').trim()) .filter((t) => t.length >= 3 && /^[A-Za-z_$][\w$]*(?:(?:::|\.)[\w$]+)*$/.test(t)) )].slice(0, 16); @@ -3430,24 +3534,26 @@ export class ToolHandler { } } let cands = raw - .filter((n) => CALLABLE.has(n.kind) && !isTestPath(n.filePath)) + .filter((n) => SEEDABLE.has(n.kind) && !isTestPath(n.filePath)) .sort((a, b) => (bodyLines(b) > 1 ? 1 : 0) - (bodyLines(a) > 1 ? 1 : 0) || bodyLines(b) - bodyLines(a)); // Field-name seeding fallback (#1196): a camelCase token that names NO // definition of its own is usually an object-literal key / API field // (`profileInfo`) — no node exists, so it contributed zero seeds and // the files that DEFINE it (`getProfileInfoV2` in profileController) - // never surfaced. Seed its camel-infix definers instead: callables - // whose name contains the token at a hump boundary or as a prefix. + // never surfaced. Seed its camel-infix definers instead: seedable + // symbols (callables + variables — `atBottom` must reach the `$state` + // variables `feedAtBottom`/`chatAtBottom`) whose name contains the + // token at a hump boundary or as a prefix. // Exact-empty + camel-shaped only (bare words keep the NL-stopword // guard below), shortest-first, capped so a hot infix can't flood. if (cands.length === 0 && !isQual && /[a-z][A-Z]/.test(t)) { const lcToken = t.toLowerCase(); cands = cg .getNodesByNameSubstring(t, { - kinds: ['function', 'method', 'component'], + kinds: ['function', 'method', 'component', 'variable', 'constant'], limit: 60, }) - .filter((n) => CALLABLE.has(n.kind) && !isTestPath(n.filePath)) + .filter((n) => SEEDABLE.has(n.kind) && !isTestPath(n.filePath)) .filter((n) => { const idx = n.name.toLowerCase().indexOf(lcToken); if (idx < 0) return false; @@ -3624,8 +3730,9 @@ export class ToolHandler { fileGroups.set(node.filePath, group); } - // Extract query terms for relevance checking - const queryTerms = query.toLowerCase().split(/\s+/).filter(t => t.length >= 3); + // Extract query terms for relevance checking (path-stripped: a pinned + // file's own path fragments must not count as "term hits" everywhere) + const queryTerms = matchQuery.toLowerCase().split(/\s+/).filter(t => t.length >= 3); // Test/spec/icon/i18n file detector — used by the pre-floor hard filter, the // rank penalty, and the comparator deprioritization. @@ -3715,9 +3822,10 @@ export class ToolHandler { // keep-minimum then pulled two test files back in as the "spread". let candidateFiles = [...fileGroups.entries()]; { - const queryMentionsTests = /\b(test|tests|testing|spec|verify|verifies)\b/i.test(query); + const queryMentionsTests = /\b(test|tests|testing|spec|verify|verifies)\b/i.test(matchQuery); if (!queryMentionsTests) { - const nonLow = candidateFiles.filter(([p]) => !isLowValue(p)); + // A pinned file is exempt: naming a test file by path IS asking for it. + const nonLow = candidateFiles.filter(([p]) => !isLowValue(p) || pinnedSet.has(p)); if (nonLow.length >= 2) { candidateFiles = nonLow; } @@ -3732,7 +3840,9 @@ export class ToolHandler { SCORE_FLOOR_ABSOLUTE, Math.min(SCORE_FLOOR_MAX, topScore * SCORE_FLOOR_FRACTION_OF_TOP), ); - let relevantFiles = candidateFiles.filter(([, group]) => group.score >= scoreFloor); + let relevantFiles = candidateFiles.filter( + ([fp, group]) => group.score >= scoreFloor || pinnedSet.has(fp), + ); if (relevantFiles.length < SCORE_FLOOR_KEEP_MIN) { // Backfill from what the RELATIVE floor cut, best first, at two strengths: // @@ -3747,8 +3857,11 @@ export class ToolHandler { // worst outcome on the board — the agent falls straight back to grep. const minEvidence = relevantFiles.length === 0 ? Number.EPSILON : SCORE_FLOOR_ABSOLUTE; relevantFiles = candidateFiles - .filter(([, group]) => group.score >= minEvidence) - .sort((a, b) => b[1].score - a[1].score || b[1].nodes.length - a[1].nodes.length) + .filter(([fp, group]) => group.score >= minEvidence || pinnedSet.has(fp)) + .sort((a, b) => + (pinnedSet.has(b[0]) ? 1 : 0) - (pinnedSet.has(a[0]) ? 1 : 0) + || b[1].score - a[1].score + || b[1].nodes.length - a[1].nodes.length) .slice(0, Math.max(SCORE_FLOOR_KEEP_MIN, relevantFiles.length)); } diag?.setScoreFloor(scoreFloor, relevantFiles.length); @@ -3852,7 +3965,8 @@ export class ToolHandler { // never prunes below 2. if (maxGraph > 0) { const gated = relevantFiles.filter(([fp]) => - (fileGraphScore.get(fp) ?? 0) >= maxGraph * 0.06 + pinnedSet.has(fp) + || (fileGraphScore.get(fp) ?? 0) >= maxGraph * 0.06 || centralFiles.has(fp) || entryFiles.has(fp) || changeSurfaceFiles.has(fp) @@ -3908,7 +4022,15 @@ export class ToolHandler { const aPath = a[0].toLowerCase(); const bPath = b[0].toLowerCase(); - // Agent-named files first (it asked for a symbol defined here by name). + // Pinned files first of all — the agent named the FILE by path, which is + // even more explicit than naming a symbol in it. Among pins, keep the + // order they appeared in the query. + const aPin = pinnedSet.has(a[0]) ? 1 : 0; + const bPin = pinnedSet.has(b[0]) ? 1 : 0; + if (aPin !== bPin) return bPin - aPin; + if (aPin && bPin) return (pinnedOrder.get(a[0]) ?? 0) - (pinnedOrder.get(b[0]) ?? 0); + + // Agent-named files next (it asked for a symbol defined here by name). const aNamed = namedSeedFiles.has(a[0]) ? 1 : 0; const bNamed = namedSeedFiles.has(b[0]) ? 1 : 0; if (aNamed !== bNamed) return bNamed - aNamed; @@ -4010,7 +4132,7 @@ export class ToolHandler { // Compute the flow spine once — used both to prepend the Flow section (below) // and to gate adaptive source sizing: files on the spine get full source, // off-spine peers skeletonize. - const flow = this.buildFlowFromNamedSymbols(cg, query); + const flow = this.buildFlowFromNamedSymbols(cg, matchQuery); // Snapshot every ranked candidate's scoring inputs, in final sort order, so // the diagnostic can show what each file's share of the envelope was BOUGHT @@ -4031,6 +4153,7 @@ export class ToolHandler { graphScore: fileGraphScore.get(fp) ?? 0, termHits: fileTermHits.get(fp) ?? 0, nodes: group.nodes.length, + pinned: pinnedSet.has(fp), named: namedSeedFiles.has(fp), central: centralFiles.has(fp), entry: entryFiles.has(fp), @@ -4051,8 +4174,11 @@ export class ToolHandler { sortedFiles.map(([fp, group]) => ({ path: fp, score: group.score, - worth: rankPenalty(fp), + // A pinned file's bytes are worth full price by definition — the agent + // asked for the file itself, generated/test or not. + worth: pinnedSet.has(fp) ? 1 : rankPenalty(fp), spine: group.nodes.some((n) => flow.pathNodeIds.has(n.id)), + pinned: pinnedSet.has(fp), })), budget, maxFiles, @@ -5804,9 +5930,19 @@ export class ToolHandler { g.nodes.filter((n) => n.kind !== 'import' && n.kind !== 'export').map((n) => n.id), ).size; }, 0); - const summaryLine = survivors.length > 0 + let summaryLine = survivors.length > 0 ? `Found ${shownSymbols} symbol${shownSymbols === 1 ? '' : 's'} across ${survivors.length} file${survivors.length === 1 ? '' : 's'}.` : `Found ${subgraph.nodes.size} symbol${subgraph.nodes.size === 1 ? '' : 's'} across ${fileGroups.size} file${fileGroups.size === 1 ? '' : 's'}.`; + // Path pinning is visible, not silent: say which query-named files were + // honored, and which path spans matched nothing so the agent can correct + // them instead of trusting a response that quietly ignored the path. + const pinnedShown = pinnedFiles.filter((fp) => survivors.includes(fp)).length; + if (pinnedShown > 0) { + summaryLine += ` ${pinnedShown} file${pinnedShown === 1 ? '' : 's'} pinned from the query.`; + } + if (unresolvedPathSpans.length > 0) { + summaryLine += ` No indexed file uniquely matches ${unresolvedPathSpans.map((s) => `\`${s}\``).join(', ')}.`; + } finalText = finalText.replace(SUMMARY_SENTINEL, summaryLine); // Emit the allocation diagnostic from the FINAL text, so per-file bytes and @@ -6592,6 +6728,19 @@ export class ToolHandler { * Python — `stage_apply::run` matches a `run` in `stage_apply.rs`) */ private matchesSymbol(node: Node, symbol: string): boolean { + // Erlang arity spelling (`fn/3`, `mod:fn/3` → normalized `mod.fn/3`): when + // the node's qualifiedName carries an arity (`mod::fn/3`, #1610), the + // written arity must match it exactly; the remaining comparison then runs + // on the arity-less spelling. A node with no arity in its qualifiedName + // keeps the original symbol (a `/` there means a path-ish name instead). + const aritySpelling = /^(.+)\/(\d{1,3})$/.exec(symbol); + if (aritySpelling) { + const nodeArity = /\/(\d{1,3})$/.exec(node.qualifiedName ?? '')?.[1]; + if (nodeArity !== undefined) { + if (nodeArity !== aritySpelling[2]) return false; + symbol = aritySpelling[1]!; + } + } // Simple name match if (node.name === symbol) return true; // File basename match (e.g., "product-card" matches "product-card.liquid") diff --git a/src/project-config.ts b/src/project-config.ts index ec388daf7..56c5debe1 100644 --- a/src/project-config.ts +++ b/src/project-config.ts @@ -67,6 +67,21 @@ export interface ProjectConfig { * wins. Absent/empty (the default) forces nothing in. */ include?: string[]; + /** + * Gitignore-style patterns for paths that should still be INDEXED and + * findable, but must not outrank first-party code in search ranking (#982). + * + * The ranking counterpart to `exclude`: `exclude` is a recall lever (the + * content leaves the index entirely), this is a relevance lever (the content + * stays, it just stops winning). It generalizes the built-in + * example/sample/fixture/benchmark de-prioritization to trees only the + * project knows about — an `optional-skills/` or `scripts/` directory whose + * helpers share generic symbol names with real code. Matched against + * project-root-relative paths, so `"optional-skills/"`, a recursive glob, or + * `"tools/gen"` all work. Absent/empty (the default) de-prioritizes nothing + * beyond the built-ins. + */ + deprioritize?: string[]; } /** Parsed, validated view of a project's `codegraph.json`. */ @@ -74,6 +89,7 @@ interface ParsedConfig { extensions: Record; includeIgnored: string[]; exclude: string[]; + deprioritize: string[]; include: string[]; } @@ -97,6 +113,7 @@ const EMPTY_CONFIG: ParsedConfig = Object.freeze({ includeIgnored: Object.freeze([]) as unknown as string[], exclude: Object.freeze([]) as unknown as string[], include: Object.freeze([]) as unknown as string[], + deprioritize: Object.freeze([]) as unknown as string[], }); /** @@ -149,15 +166,17 @@ function parseConfig(file: string): ParsedConfig { const includeIgnored = extractIncludeIgnored(parsed, file); const exclude = extractExclude(parsed, file); const include = extractInclude(parsed, file); + const deprioritize = extractPatternList(parsed, file, 'deprioritize'); if ( extensions === EMPTY_EXTENSIONS && includeIgnored.length === 0 && exclude.length === 0 && - include.length === 0 + include.length === 0 && + deprioritize.length === 0 ) { return EMPTY_CONFIG; } - return { extensions, includeIgnored, exclude, include }; + return { extensions, includeIgnored, exclude, include, deprioritize }; } /** @@ -210,6 +229,29 @@ function extractIncludeIgnored(parsed: object, file: string): string[] { return out; } +/** + * Validate a gitignore-style pattern list under `key`. A non-array value or a + * non-string/blank entry warns-and-skips; never throws. Patterns are kept + * verbatim (trimmed) so they match exactly as a `.gitignore` line would. + */ +function extractPatternList(parsed: object, file: string, key: 'deprioritize'): string[] { + const raw = (parsed as ProjectConfig)[key]; + if (raw === undefined) return []; + if (!Array.isArray(raw)) { + logWarn(`Ignoring "${key}" in ${PROJECT_CONFIG_FILENAME}: must be an array of gitignore-style patterns`, { file }); + return []; + } + const out: string[] = []; + for (const entry of raw) { + if (typeof entry !== 'string' || !entry.trim()) { + logWarn(`Ignoring a "${key}" entry in ${PROJECT_CONFIG_FILENAME}: every pattern must be a non-empty string`, { file }); + continue; + } + out.push(entry.trim()); + } + return out; +} + /** * Validate the `exclude` patterns: an array of non-empty gitignore-style * strings naming paths to keep out of the index even when git-tracked (#999). A @@ -325,6 +367,17 @@ export function loadExcludePatterns(rootDir: string): string[] { return loadParsedConfig(rootDir).exclude; } +/** + * Load the validated `deprioritize` patterns for a project, mtime-cached. + * + * These name indexed paths that must not outrank first-party code (#982) — the + * ranking counterpart to `exclude`. An empty result — the zero-config default — + * de-prioritizes nothing beyond the built-in example/fixture/benchmark dirs. + */ +export function loadDeprioritizePatterns(rootDir: string): string[] { + return loadParsedConfig(rootDir).deprioritize; +} + /** * Load the validated `include` patterns for a project, mtime-cached. * diff --git a/src/resolution/c-fnptr-synthesizer.ts b/src/resolution/c-fnptr-synthesizer.ts index 1ab809918..568b782c9 100644 --- a/src/resolution/c-fnptr-synthesizer.ts +++ b/src/resolution/c-fnptr-synthesizer.ts @@ -1209,7 +1209,7 @@ export async function cFnPointerDispatchEdges( // ---- receiver-type resolution within a function's source ---- // `(?:struct )?TYPE [*]recv` declared in the params or body → TYPE (if a known // fn-pointer-bearing struct). - const recvReCache = new Map(); + const recvReCache = new LRUCache(4096); const recvTypeIn = (fnSrc: string, recv: string): string | null => { let re = recvReCache.get(recv); if (!re) { @@ -1228,7 +1228,7 @@ export async function cFnPointerDispatchEdges( // structs (the base of a chained receiver needn't carry a fn pointer itself). // Falls back to a file-scope table variable (`cmdnames` in `cmdnames[i].fn()`). const escapeRe = (x: string): string => x.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); - const varReCache = new Map(); + const varReCache = new LRUCache(4096); const varTypeIn = (fnSrc: string, v: string): string | null => { let re = varReCache.get(v); if (!re) { diff --git a/src/resolution/callback-synthesizer.ts b/src/resolution/callback-synthesizer.ts index dc8333149..0d53829b7 100644 --- a/src/resolution/callback-synthesizer.ts +++ b/src/resolution/callback-synthesizer.ts @@ -1241,7 +1241,12 @@ async function reactJsxChildEdges(ctx: ResolutionContext, onYield: MaybeYield): if ((++scanned & 255) === 0) await onYield(); // #1091: yield mid-scan on huge graphs const content = ctx.readFile(file); if (!content || (!content.includes(''))) continue; // JSX-file gate - const parents = ctx.getNodesInFile(file).filter((n) => PARENT_KINDS.has(n.kind)); + // File-level language gate, not merely a project-level one: mixed C/JS + // monorepos must not interpret `""` inside C as JSX (#1560). + const parents = ctx.getNodesInFile(file).filter( + (n) => PARENT_KINDS.has(n.kind) && JS_FAMILY.includes(n.language) + ); + if (parents.length === 0) continue; for (const parent of parents) { const src = sliceLines(content, parent.startLine, parent.endLine); if (!src || (!src.includes(''))) continue; @@ -3003,6 +3008,10 @@ const ERLANG_BEHAVIOUR_FANOUT_CAP = 24; */ function erlangArityAt(src: string, openIdx: number): number { let depth = 1; + // `<<1,2,3>>` binary literals: commas inside are element separators, not + // argument separators. Tracked separately from bracket depth because the + // single-char `<`/`>` comparison operators must stay inert (#1358). + let binDepth = 0; let commas = 0; let sawArg = false; const limit = Math.min(src.length, openIdx + 4000); @@ -3023,13 +3032,15 @@ function erlangArityAt(src: string, openIdx: number): number { sawArg = true; continue; } + if (ch === '<' && src[i + 1] === '<') { binDepth++; i++; sawArg = true; continue; } + if (ch === '>' && src[i + 1] === '>' && binDepth > 0) { binDepth--; i++; continue; } if (ch === '(' || ch === '[' || ch === '{') { depth++; sawArg = true; continue; } if (ch === ')' || ch === ']' || ch === '}') { depth--; if (depth === 0) return sawArg ? commas + 1 : 0; continue; } - if (ch === ',' && depth === 1) { commas++; continue; } + if (ch === ',' && depth === 1 && binDepth === 0) { commas++; continue; } if (!/\s/.test(ch)) sawArg = true; } return -1; @@ -3260,12 +3271,18 @@ async function erlangBehaviourDispatchEdges(queries: QueryBuilder, ctx: Resoluti } if (declaringBehaviours.size === 0) return []; - // Implementer target lookup, lazy per (behaviour, fn): implementers come - // from the `implements` edges extraction resolved, and the target is the - // implementer module's own exported `fn` function node. + // Implementer target lookup, lazy per (behaviour, fn, arity): implementers + // come from the `implements` edges extraction resolved, and the target is + // the implementer module's own exported `fn` node OF THE SITE'S ARITY — + // function qualifiedNames carry arity (`mod::fn/2`, #1610), so the arity the + // dispatch site used selects among same-named definitions. const targetCache = new Map(); - const targetsOf = (behaviour: Node, fn: string): Node[] => { - const cacheKey = `${behaviour.id}#${fn}`; + const qnArity = (qn: string): number => { + const m = /\/(\d{1,3})$/.exec(qn); + return m ? Number(m[1]) : -1; + }; + const targetsOf = (behaviour: Node, fn: string, arity: number): Node[] => { + const cacheKey = `${behaviour.id}#${fn}/${arity}`; let targets = targetCache.get(cacheKey); if (targets) return targets; targets = []; @@ -3274,7 +3291,13 @@ async function erlangBehaviourDispatchEdges(queries: QueryBuilder, ctx: Resoluti if (!impl || impl.language !== 'erlang' || impl.kind !== 'namespace') continue; const fnNode = ctx .getNodesInFile(impl.filePath) - .find((n) => n.kind === 'function' && n.name === fn && n.isExported !== false); + .find( + (n) => + n.kind === 'function' && + n.name === fn && + qnArity(n.qualifiedName) === arity && + n.isExported !== false, + ); if (fnNode) targets.push(fnNode); } targetCache.set(cacheKey, targets); @@ -3303,7 +3326,7 @@ async function erlangBehaviourDispatchEdges(queries: QueryBuilder, ctx: Resoluti const behaviours = declaringBehaviours.get(`${fn}/${arity}`); if (!behaviours || behaviours.length !== 1) continue; // unknown or ambiguous const behaviour = behaviours[0]!; - const targets = targetsOf(behaviour, fn); + const targets = targetsOf(behaviour, fn, arity); if (targets.length === 0 || targets.length > ERLANG_BEHAVIOUR_FANOUT_CAP) continue; const line = safe.slice(0, m.index).split('\n').length; const disp = enclosingFn(nodesInFile, line); @@ -3533,7 +3556,7 @@ export const SYNTH_PASSES: SynthPassDef[] = [ { name: 'closureCollEdges', gate: ALWAYS, run: (q, c, y) => closureCollectionEdges(q, c, y) }, { name: 'emitterEdges', gate: ALWAYS, run: (_q, c, y) => eventEmitterEdges(c, y) }, { name: 'renderEdges', gate: ALWAYS, run: (q, c, y) => reactRenderEdges(q, c, y) }, - { name: 'jsxEdges', gate: ALWAYS, run: (_q, c, y) => reactJsxChildEdges(c, y) }, + { name: 'jsxEdges', gate: (has) => has(...JS_FAMILY), run: (_q, c, y) => reactJsxChildEdges(c, y) }, { name: 'vueEdges', gate: (has) => has('vue'), run: (_q, c, y) => vueTemplateEdges(c, y) }, { name: 'svelteKitEdges', gate: (has) => has('svelte'), run: (_q, c, y) => svelteKitLoadEdges(c, y) }, { name: 'pascalEdges', gate: ALWAYS, run: (_q, c, y) => pascalFormEdges(c, y) }, diff --git a/src/resolution/frameworks/swift.ts b/src/resolution/frameworks/swift.ts index 0dd1513aa..5e1bf42cd 100644 --- a/src/resolution/frameworks/swift.ts +++ b/src/resolution/frameworks/swift.ts @@ -367,7 +367,17 @@ export const vaporResolver: FrameworkResolver = { // (`BlogUser.parameter`, `:id`, a path constant) so accept any comma-separated // args before `use:` — the label keeps only the string parts. `use:` // discriminates a real route from Environment.get("X")/req.parameters.get("X"). - const routeRegex = /\b(\w+)\.(get|post|put|patch|delete|head|options)\s*\(\s*((?:[^,()]+,\s*)*)use:\s*([A-Za-z_][\w.]*)/g; + // Each arg repetition must end at a comma, and `,` is outside the char class, + // so the split is unique and matching stays linear. The earlier + // `(?:[^,()]+,\s*)*` was ambiguous — the trailing `\s*` and the next + // iteration's `[^,()]+` could both claim the same spaces — which backtracked + // exponentially on a long arg list that never reaches `use:`. + // The tail is `\s*` rather than a lazy `[^,()]*?` on purpose: both are + // linear, but the lazy form drops the "`use:` is preceded by a comma" + // requirement and widens the match set — `req.get(foo.use: bar)` would then + // be indexed as a route (groups `["req","get","foo.","bar"]`) where both + // this pattern and the original match nothing. + const routeRegex = /\b(\w+)\.(get|post|put|patch|delete|head|options)\s*\(\s*((?:[^,()]+,)*\s*)use:\s*([A-Za-z_][\w.]*)/g; let match: RegExpExecArray | null; while ((match = routeRegex.exec(safe)) !== null) { const [, receiver, method, segsStr, handlerExpr] = match; diff --git a/src/resolution/import-resolver.ts b/src/resolution/import-resolver.ts index a32a97916..60c7b3008 100644 --- a/src/resolution/import-resolver.ts +++ b/src/resolution/import-resolver.ts @@ -12,6 +12,7 @@ import { applyAliases } from './path-aliases'; import { resolveWorkspaceImport } from './workspace-packages'; import { resolveMethodOnType, + resolveObjectLiteralMember, localReceiverTypePatterns, normalizeInferredTypeName, } from './name-matcher'; @@ -26,7 +27,7 @@ const EXTENSION_RESOLUTION: Record = { // module-entry convention, hit when a bare workspace import ("data") is // rewritten to the member's directory; lowercase variants for safety. arkts: ['.ets', '.ts', '.d.ts', '.js', '/Index.ets', '/index.ets', '/index.ts', '/index.js'], - javascript: ['.js', '.jsx', '.mjs', '.cjs', '/index.js', '/index.jsx'], + javascript: ['.js', '.jsx', '.mjs', '.cjs', '.xsjs', '.xsjslib', '/index.js', '/index.jsx'], tsx: ['.tsx', '.ts', '.d.ts', '.js', '.jsx', '/index.tsx', '/index.ts', '/index.js'], jsx: ['.jsx', '.js', '/index.jsx', '/index.js'], // SFC consumers import plain TS/JS, sibling components, and barrels @@ -1545,6 +1546,20 @@ export function resolveViaImport( resolvedBy: 'import', }; } + // An imported object literal used as a namespace (#1573): + // `api.call()` after `import { api } from './api'` where `api` is + // `export const api = { call() {…} }`. Its members have bare + // qualified names inside the constant's extent, so the + // `Container::member` lookup above can't see them and the edge + // landed on the constant — every cross-file caller of the method + // went missing. Resolve the member by containment instead. + if (targetNode.kind === 'constant' || targetNode.kind === 'variable') { + const member = ref.referenceName.slice(imp.localName.length + 1).split('.')[0]; + if (member) { + const literalMember = resolveObjectLiteralMember(targetNode, member, ref, context, 0.9, 'import'); + if (literalMember) return literalMember; + } + } // An imported VALUE (singleton constant / shared instance) called // through a member: `reproStore.notifyJoinGuildStatus()` after // `import { reproStore } from './store'`. findExportedSymbol diff --git a/src/resolution/index.ts b/src/resolution/index.ts index 01f615b28..7b4bccc18 100644 --- a/src/resolution/index.ts +++ b/src/resolution/index.ts @@ -886,10 +886,13 @@ export class ReferenceResolver { // indexed under the bare name, so the existence check strips the dot. // Nix static path imports (`import ./x.nix`) name a FILE, not a symbol — // they bypass the symbol-existence check and resolve via resolveViaImport. - const existenceName = + let existenceName = ref.language === 'arkts' && ref.referenceName.startsWith('.') ? ref.referenceName.slice(1) : ref.referenceName; + // Erlang refs carry the call-site arity (`f/1`, `mod::f/2` — #1610); the + // name index stores bare names, so existence is checked arity-less. + if (ref.language === 'erlang') existenceName = existenceName.replace(/\/\d{1,3}$/, ''); const tPre = this.profileStages ? process.hrtime.bigint() : 0n; const preFilterPass = isNixPathImportRef(ref) || diff --git a/src/resolution/name-matcher.ts b/src/resolution/name-matcher.ts index 651051466..c74d8f272 100644 --- a/src/resolution/name-matcher.ts +++ b/src/resolution/name-matcher.ts @@ -503,6 +503,35 @@ export function matchByQualifiedName( } } + // Erlang qualified refs (#1610): every erlang function's qualifiedName + // carries its arity (`mod::f/2`), and refs carry the call-site arity when it + // is statically known. + if (ref.language === 'erlang' && ref.referenceName.includes('::')) { + // A ref WITH arity that missed the exact lookup names an arity that isn't + // defined (or a module out of repo). Never fall through to the partial + // match — its "last segment" would be the arity digits — and never settle + // for a sibling arity: silent beats wrong. + if (/\/\d{1,3}$/.test(ref.referenceName)) return null; + // An arity-LESS qualified ref (dynamic MFA whose args list wasn't a + // static literal): resolve only when the module defines exactly ONE arity + // of that function; several arities with no signal is a guess. + const base = ref.referenceName.slice(ref.referenceName.lastIndexOf('::') + 2); + const prefix = `${ref.referenceName}/`; + const arityCands = keepForRef(context.getNodesByName(base)).filter( + (n) => + n.qualifiedName.startsWith(prefix) && /^\d{1,3}$/.test(n.qualifiedName.slice(prefix.length)), + ); + if (arityCands.length === 1) { + return { + original: ref, + targetNodeId: arityCands[0]!.id, + confidence: 0.85, + resolvedBy: 'qualified-name', + }; + } + return null; + } + // Try partial qualified name match — again preferring the call site's own // file when more than one symbol's qualifiedName ends with the reference. const parts = ref.referenceName.split(/[:.]/); @@ -543,6 +572,93 @@ export function preferCallSiteFile(nodes: Node[], callSiteFile: string): Node[] return same.length ? [...same, ...other] : nodes; } +/** + * Languages whose object literals declare callable members — `export const + * api = { call() {…}, get: () => {…} }` used as a namespace (#1573). + */ +const OBJECT_LITERAL_LANGUAGES = new Set(['typescript', 'tsx', 'javascript', 'jsx', 'arkts']); + +/** True when `inner`'s source range lies within `outer`'s (lines, then columns on a shared line). */ +function rangeWithin(inner: Node, outer: Node): boolean { + const innerEnd = inner.endLine ?? inner.startLine; + const outerEnd = outer.endLine ?? outer.startLine; + if (inner.startLine < outer.startLine || innerEnd > outerEnd) return false; + if (inner.startLine === outer.startLine && inner.startColumn < outer.startColumn) return false; + if (innerEnd === outerEnd && inner.endColumn > outer.endColumn) return false; + return true; +} + +function sameRange(a: Node, b: Node): boolean { + return ( + a.startLine === b.startLine && + a.startColumn === b.startColumn && + (a.endLine ?? a.startLine) === (b.endLine ?? b.startLine) && + a.endColumn === b.endColumn + ); +} + +/** + * Resolve `container.member` where `container` is a VALUE holding an object + * literal — `export const api = { call() {…}, get: () => {…} }` used as the + * module's namespace (#1573). The members are extracted as plain functions + * with BARE qualified names inside the constant's source extent (there is no + * `api::call`), so neither the `Container::member` lookup the class-shaped + * kinds use (#825) nor the declared-type inference for singleton instances + * (#1292) can reach them, and every such call resolved to nothing — or, via + * an import, to the constant itself. This looks the member up by CONTAINMENT: + * a node named `member` whose range lies inside the container's, in the + * container's own file. A helper declared inside a member's body is not a + * member and is skipped; nothing else in the file can donate a match. Calls + * take callable kinds only; other references accept value members too. + */ +export function resolveObjectLiteralMember( + container: Node, + member: string, + ref: UnresolvedRef, + context: ResolutionContext, + confidence: number, + resolvedBy: ResolvedRef['resolvedBy'], +): ResolvedRef | null { + if (container.kind !== 'constant' && container.kind !== 'variable') return null; + if (!OBJECT_LITERAL_LANGUAGES.has(container.language)) return null; + if (!sameLanguageFamily(container.language, ref.language)) return null; + + const inFile = context.getNodesInFile(container.filePath); + const callable = (n: Node) => n.kind === 'function' || n.kind === 'method'; + const valueMember = (n: Node) => + callable(n) || n.kind === 'property' || n.kind === 'variable' || n.kind === 'constant'; + const accepts = ref.referenceKind === 'calls' ? callable : valueMember; + + const inside = inFile.filter((n) => n.id !== container.id && rangeWithin(n, container)); + let candidates = inside.filter((n) => n.name === member && accepts(n)); + if (candidates.length === 0) return null; + + // Drop a candidate nested inside ANOTHER callable's body within the literal + // (`{ run() { const call = () => {}; } }` — `call` is `run`'s local, not a + // member). Strict containment: an identically-ranged sibling node for the + // same member (a property node over an arrow function) is not a body. + const bodies = inside.filter(callable); + candidates = candidates.filter( + (c) => !bodies.some((b) => b.id !== c.id && !sameRange(b, c) && rangeWithin(c, b)) + ); + if (candidates.length === 0) return null; + + // Several survivors (a property AND a function for one arrow member, say): + // a callable first, then the earliest in source order. + candidates.sort((a, b) => { + const ca = callable(a) ? 0 : 1; + const cb = callable(b) ? 0 : 1; + if (ca !== cb) return ca - cb; + return a.startLine - b.startLine || a.startColumn - b.startColumn; + }); + return { + original: ref, + targetNodeId: candidates[0]!.id, + confidence, + resolvedBy, + }; +} + // Exported for the precedence unit tests (#1079): they assert the // preferredFqn → same-file → matches[0] ordering directly. export function resolveMethodOnType( @@ -1731,6 +1847,18 @@ export function matchMethodCall( return matchGoFieldChainCall(objectOrClass!, methodName!, ref, context); } + // Rust call through a field of the enclosing type — `self.inner.run()`, + // emitted as `self.inner.run` (#1585). Same discipline as the Go branch + // above, and EXCLUSIVE for the same reason: validated field-type inference + // or nothing. Letting this shape reach the bare-name strategies below is + // how `self.inner.run()` resolved to a same-named method on an unrelated + // type — or to the calling method itself, a self-edge the source doesn't + // contain — whenever the field's type was external or merely shared a + // method name with something nearby. + if (ref.language === 'rust' && dotMatch && objectOrClass!.startsWith('self.')) { + return matchRustSelfFieldCall(objectOrClass!.slice('self.'.length), methodName!, ref, context); + } + // Java/Kotlin: receiver may be a field whose name doesn't match the type by // Java naming convention (`userbo` → class `UserBO`, abbreviated). Look up // the field in the enclosing class to get its declared type, then resolve @@ -1759,6 +1887,27 @@ export function matchMethodCall( } } + // Object-literal namespace receiver (#1573): `api.call()` where `api` is a + // same-file `const api = { call() {…}, get: () => {…} }`. Its members are + // plain functions with bare names inside the constant's extent — no + // `Container::member` qualified name — so none of the class-shaped + // strategies below can see them (Strategy 3 only considers `method` + // kinds) and the call resolved to nothing at all. Same file only: a + // cross-file use reaches the same helper through the import path. + if (dotMatch && !objectOrClass!.includes('.') && OBJECT_LITERAL_LANGUAGES.has(ref.language)) { + const literalMatch = nmTimedT('mc-literal', ref, (): ResolvedRef | null => { + const holders = preferCallSiteFile(context.getNodesByName(objectOrClass!), ref.filePath).filter( + (n) => (n.kind === 'constant' || n.kind === 'variable') && n.filePath === ref.filePath + ); + for (const holder of holders) { + const hit = resolveObjectLiteralMember(holder, methodName!, ref, context, 0.85, 'instance-method'); + if (hit) return hit; + } + return null; + }); + if (literalMatch) return literalMatch; + } + // Strategy 1: Direct class name match (existing logic). When the receiver // names a class that exists in several files (`Logger.log()` / `Logger::log()` // with a `Logger` in both `a/` and `b/`), try the class in the call site's @@ -1992,6 +2141,110 @@ function matchGoFieldChainCall( return null; } +// Rust primitives and the prelude's own types: a field of one of these never +// names a project type, so a `self..()` on it stays unresolved. +const RUST_NON_PROJECT_FIELD_TYPES = new Set([ + 'bool', 'char', 'str', 'String', + 'i8', 'i16', 'i32', 'i64', 'i128', 'isize', + 'u8', 'u16', 'u32', 'u64', 'u128', 'usize', + 'f32', 'f64', + 'Self', 'self', +]); + +/** + * Reduce a Rust field's declared type text to the simple name of the type a + * method call on that field auto-derefs to, or null when there is none we can + * name. Only the layers Rust's method-call auto-deref looks through are + * unwrapped: references (`&`, `&'a mut`) and the owning smart pointers + * (`Box`, `Rc`, `Arc`) — `self.inner.run()` with `inner: Box` calls + * `Inner::run`. Containers that do NOT auto-deref to their parameter + * (`Option`, `Vec`, `Mutex`, `RefCell`) keep their + * own name and, having no project node, resolve to nothing — `self.items.push()` + * must never become `Inner::push`. A trait object (`Box`) yields + * the trait, whose method node the interface-impl synthesizer fans out. A + * generic parameter (`T`), a primitive, a tuple / array / raw pointer / fn + * type, or a non-identifier yields null. + */ +export function rustFieldTypeName(raw: string): string | null { + let t = raw.trim(); + for (;;) { + const before = t; + t = t.replace(/^&\s*(?:'\w+\s+)?(?:mut\s+)?/, ''); + t = t.replace(/^(?:Box|Rc|Arc)\s*<\s*/, ''); + t = t.replace(/^(?:dyn|impl)\s+/, ''); + if (t === before) break; + } + // Drop generic args, the closing `>`s of unwrapped pointers, and trait-object + // bounds (`dyn Source + Send`); keep the last path segment. + t = t.replace(/[<>+].*$/, '').trim(); + const seg = t.split('::').filter(Boolean).pop(); + if (!seg || !/^[A-Za-z_]\w*$/.test(seg)) return null; + if (RUST_NON_PROJECT_FIELD_TYPES.has(seg)) return null; + if (/^[A-Z]$/.test(seg)) return null; // bare single-letter generic parameter + return seg; +} + +/** + * Resolve a Rust call through a field of the enclosing type — + * `self.inner.run()`, emitted by the extractor as `self.inner.run` (#1585). + * Mirrors the Go 2-hop precedent above (#1276): the owner type is the calling + * method's qualified-name prefix (`Outer::run` → `Outer`), the field's declared + * type comes from the owner struct's OWN declaration lines, and the method is + * resolved AND VALIDATED on that type by resolveMethodOnType. The caller + * treats this branch as exclusive for `self.` receivers: a field whose + * type is external (`std::vec::IntoIter`, `regex::Regex`), a generic + * parameter, or not declared where we can see it yields null and the ref stays + * unresolved. Rust struct fields are not graph nodes, so the declaration text + * is the only place the type lives. + */ +function matchRustSelfFieldCall( + field: string, + methodName: string, + ref: UnresolvedRef, + context: ResolutionContext, +): ResolvedRef | null { + // The extractor only ever emits a single field hop; anything else is not ours. + if (!field || field.includes('.')) return null; + const caller = context.getNodeById?.(ref.fromNodeId); + if (!caller) return null; + const sep = caller.qualifiedName.lastIndexOf('::'); + if (sep <= 0) return null; // a free fn has no `self` + const owner = caller.qualifiedName.slice(0, sep).split('::').pop(); + if (!owner) return null; + + const owners = preferCallSiteFile(context.getNodesByName(owner), ref.filePath).filter( + (n) => + (n.kind === 'struct' || n.kind === 'union' || n.kind === 'class') && + n.language === 'rust' + ); + const fieldEsc = field.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); + // `pub inner: Inner,` / `inner: Box,` / `pub(crate) inner: T }` — + // the type text runs to the field separator. A comma inside generic args + // (`HashMap`) truncates the capture, which rustFieldTypeName then + // reduces to the container's own name — exactly the non-deref case it + // refuses anyway. + const fieldRe = new RegExp(`\\b${fieldEsc}\\s*:\\s*([^,{}]+)`); + for (const s of owners) { + const source = context.readFile(s.filePath); + if (!source) continue; + // Only the struct's own declaration lines, comment-stripped line by line — + // same discipline as the Go helper: prose or a same-named identifier + // elsewhere in the file can never donate a type. + const declLines = source.split('\n').slice(Math.max(0, s.startLine - 1), s.endLine); + for (const rawLine of declLines) { + const line = rawLine.replace(/\/\/.*$/, '').replace(/\/\*.*?\*\//g, ''); + const m = line.match(fieldRe); + if (!m || !m[1]) continue; + const fieldType = rustFieldTypeName(m[1]); + // The field is declared here; whether or not its type names a project + // symbol, this owner is the answer — no other same-named struct applies. + if (!fieldType) return null; + return resolveMethodOnType(fieldType, methodName, ref, context, 0.85, 'instance-method'); + } + } + return null; +} + /** * Split a camelCase or PascalCase string into words. */ @@ -2295,6 +2548,52 @@ export function matchReference( }; } + // Erlang call/fun refs carry the call-site arity (`f/1` — #1610) because + // arity is part of the function's identity and every erlang function's + // qualifiedName carries it (`mod::f/1`). Resolve ONLY to a definition of + // that exact arity: the call site's own file first (a local call targets its + // own module by language semantics; `-import`ed functions ride the + // cross-file branch), and when no definition of that arity exists anywhere, + // resolve to NOTHING rather than a sibling arity — the real target may be + // macro-generated or out of repo, and a wrong-arity edge is worse than none. + if ( + ref.language === 'erlang' && + !ref.referenceName.includes('::') && + (ref.referenceKind === 'calls' || ref.referenceKind === 'references') + ) { + const am = /^(.+)\/(\d{1,3})$/.exec(ref.referenceName); + if (am) { + // endsWith is length-anchored, so `/1` cannot match `…/11`. + const arityTail = `/${am[2]}`; + const candidates = context + .getNodesByName(am[1]!) + .filter( + (n) => + n.language === 'erlang' && n.kind === 'function' && n.qualifiedName.endsWith(arityTail), + ); + if (candidates.length > 0) { + const sameFile = candidates.find((n) => n.filePath === ref.filePath); + if (sameFile) { + return { original: ref, targetNodeId: sameFile.id, confidence: 0.95, resolvedBy: 'exact-match' }; + } + if (candidates.length === 1) { + return { original: ref, targetNodeId: candidates[0]!.id, confidence: 0.8, resolvedBy: 'exact-match' }; + } + const best = findBestMatch(ref, candidates, context); + if (best) { + const proximity = computePathProximity(ref.filePath, best.filePath); + return { + original: ref, + targetNodeId: best.id, + confidence: proximity >= 30 ? 0.7 : 0.4, + resolvedBy: 'exact-match', + }; + } + } + return null; + } + } + // Try strategies in order of confidence let result: ResolvedRef | null; diff --git a/src/resolution/path-aliases.ts b/src/resolution/path-aliases.ts index 362baac75..73e366e7d 100644 --- a/src/resolution/path-aliases.ts +++ b/src/resolution/path-aliases.ts @@ -11,11 +11,13 @@ * ignored — every import through an alias was treated as unresolvable * unless it happened to match the small hard-coded fallback list. * - * Scope deliberately small for v1: - * - reads tsconfig.json, then jsconfig.json - * - honours top-level `compilerOptions.baseUrl` and `compilerOptions.paths` + * Scope: + * - reads tsconfig.json, then jsconfig.json, then tsconfig.base.json + * - honours `compilerOptions.baseUrl` and `compilerOptions.paths` + * - follows `extends` chains, nearest config wins (#1534) — Nx-style + * monorepos keep every alias in a `tsconfig.base.json` the root + * config merely inherits, so without this they resolved nothing * - supports `*` wildcard (the only TS-supported wildcard) - * - does NOT follow `extends` chains yet (most projects don't need it) * - does NOT read Vite/webpack/Rollup configs (separate follow-up) * * The file is parsed as JSON-with-comments-tolerant — tsconfigs in the @@ -104,12 +106,118 @@ function stripJsonc(src: string): string { } interface RawTsconfig { + extends?: string | string[]; compilerOptions?: { baseUrl?: string; paths?: Record; }; } +/** + * The `baseUrl`/`paths` a config ends up with once its `extends` chain has + * been folded in. `pathsDir` is the directory of the config that actually + * declared `paths` — with no `baseUrl` anywhere, tsc anchors the targets + * there, not at the project root. + */ +interface EffectiveOptions { + baseUrl?: string; + paths?: Record; + pathsDir?: string; +} + +/** Guards against a pathological chain; real ones are 1-3 deep. */ +const MAX_EXTENDS_DEPTH = 32; + +/** + * Locate an `extends` target the way tsc does: `./x`-style values are + * relative to the referencing config, anything else is a node_modules + * package specifier resolved by walking up from that config. A missing + * `.json` extension is implied, and a bare package name means its + * `tsconfig.json`. + */ +function resolveExtendsTarget(spec: string, fromDir: string): string | null { + const isFile = (p: string): boolean => { + try { + return fs.statSync(p).isFile(); + } catch { + return false; + } + }; + + if (spec.startsWith('./') || spec.startsWith('../') || path.isAbsolute(spec)) { + const base = path.resolve(fromDir, spec); + for (const cand of [base, `${base}.json`, path.join(base, 'tsconfig.json')]) { + if (isFile(cand)) return cand; + } + return null; + } + + let dir = fromDir; + for (;;) { + const base = path.join(dir, 'node_modules', spec); + for (const cand of [base, `${base}.json`, path.join(base, 'tsconfig.json')]) { + if (isFile(cand)) return cand; + } + const parent = path.dirname(dir); + if (parent === dir) return null; + dir = parent; + } +} + +/** + * Read `filePath` and fold its `extends` chain into a single set of + * effective options. Parents are applied first and the nearest config + * wins — tsc replaces `paths` wholesale rather than merging it. + * + * `stack` holds the configs currently being resolved, so a cycle + * (`a extends b extends a`) stops instead of recursing forever. + */ +function loadEffectiveOptions( + filePath: string, + stack: Set, + depth: number +): EffectiveOptions | null { + const abs = path.resolve(filePath); + if (stack.has(abs) || depth > MAX_EXTENDS_DEPTH) { + logDebug('path-aliases: extends chain cycle or too deep', { filePath: abs, depth }); + return null; + } + const raw = readTsconfigLike(abs); + if (!raw) return null; + + stack.add(abs); + const dir = path.dirname(abs); + const effective: EffectiveOptions = {}; + + const parents = typeof raw.extends === 'string' ? [raw.extends] : (raw.extends ?? []); + for (const spec of parents) { + if (typeof spec !== 'string') continue; + const target = resolveExtendsTarget(spec, dir); + if (!target) { + logDebug('path-aliases: unresolved extends', { from: abs, spec }); + continue; + } + const inherited = loadEffectiveOptions(target, stack, depth + 1); + if (!inherited) continue; + if (inherited.baseUrl !== undefined) effective.baseUrl = inherited.baseUrl; + if (inherited.paths !== undefined) { + effective.paths = inherited.paths; + effective.pathsDir = inherited.pathsDir; + } + } + stack.delete(abs); + + const co = raw.compilerOptions ?? {}; + // Both are relative to the file that declared them, not to whichever + // config started the chain. + if (typeof co.baseUrl === 'string') effective.baseUrl = path.resolve(dir, co.baseUrl); + if (co.paths && typeof co.paths === 'object') { + effective.paths = co.paths; + effective.pathsDir = dir; + } + return effective; +} + function readTsconfigLike(filePath: string): RawTsconfig | null { try { const raw = fs.readFileSync(filePath, 'utf-8'); @@ -143,26 +251,40 @@ function splitWildcard(pattern: string): { * resolver does it via {@link aliasCache}). */ export function loadProjectAliases(projectRoot: string): AliasMap | null { - const candidates = ['tsconfig.json', 'jsconfig.json']; - let raw: RawTsconfig | null = null; + // `tsconfig.base.json` comes last on purpose: when a root `tsconfig.json` + // exists it stays authoritative and reaches the base through `extends`. + // The fallback is for the Nx layouts where that never happens — a + // solution-style root config (`references`, no `extends`, no `paths`), or + // no root `tsconfig.json` at all. + const candidates = ['tsconfig.json', 'jsconfig.json', 'tsconfig.base.json']; + let effective: EffectiveOptions | null = null; let usedFile: string | null = null; for (const name of candidates) { const p = path.join(projectRoot, name); - if (fs.existsSync(p)) { - raw = readTsconfigLike(p); - if (raw) { - usedFile = name; - break; - } + if (!fs.existsSync(p)) continue; + const opts = loadEffectiveOptions(p, new Set(), 0); + if (!opts) continue; + // Remember the first readable config so a `paths`-less project still + // logs the file it was judged on, but keep looking: a config that + // contributes no aliases must not shadow one that does. + if (!effective) { + effective = opts; + usedFile = name; + } + if (opts.paths) { + effective = opts; + usedFile = name; + break; } } - if (!raw) return null; + if (!effective) return null; - const co = raw.compilerOptions ?? {}; - const baseUrlRel = co.baseUrl ?? '.'; - const baseUrl = path.resolve(projectRoot, baseUrlRel); + // With no explicit baseUrl, `paths` targets are relative to the config that + // declared them — which is the project root only when that config is the + // root one (the pre-`extends` assumption). + const baseUrl = effective.baseUrl ?? effective.pathsDir ?? projectRoot; - const paths = co.paths; + const paths = effective.paths; if (!paths || typeof paths !== 'object') { // baseUrl alone isn't an "alias" per se; with no paths we'd just // be redirecting the whole tree. Skip — the existing resolver diff --git a/src/search/identifier-segments.ts b/src/search/identifier-segments.ts index 110a6b725..75bfde4e2 100644 --- a/src/search/identifier-segments.ts +++ b/src/search/identifier-segments.ts @@ -126,6 +126,33 @@ export function extractProseCandidates(prompt: string): string[] { return [...seen]; } +/** + * Words to look up in the segment vocabulary for a SEARCH query (as opposed + * to a prompt-hook gate): the query's prose candidates PLUS the segments of + * its identifier-shaped tokens. An agent's query names concepts both ways — + * "auto-scroll to bottom" (prose) and "atBottom tracking" (camel) — and the + * camel token must still reach the segment "bottom" even though the whole + * token matches no name. Same stopword/length rules as the hook path, since + * both feeds run through {@link extractProseCandidates}. + */ +export function extractSegmentSearchWords(query: string): string[] { + if (!query) return []; + const words = new Set(extractProseCandidates(query)); + const segments: string[] = []; + for (const run of query.match(/[\p{L}\p{N}]+/gu) ?? []) { + // Only camel-humped tokens contribute segments — a plain word's + // "segments" are itself (already covered above), and snake_case arrives + // as separate runs because `_` is not a letter. + if (/[\p{Ll}\p{N}]\p{Lu}/u.test(run)) { + segments.push(...splitIdentifierSegments(run)); + } + } + if (segments.length > 0) { + for (const w of extractProseCandidates(segments.join(' '))) words.add(w); + } + return [...words]; +} + /** * Lookup variants for a prose word: the word itself plus light plural folding * ("services" → service, "dependencies" → dependencie/dependency is NOT diff --git a/src/search/query-paths.ts b/src/search/query-paths.ts new file mode 100644 index 000000000..c91272ddb --- /dev/null +++ b/src/search/query-paths.ts @@ -0,0 +1,287 @@ +/** + * File-path recognition for explore queries. + * + * Agents routinely name files by path in a `codegraph_explore` query — + * "the scroll logic in src/routes/m/projects/[id]/runs/[runId]/+page.svelte" — + * and until this module existed those spans were SHREDDED by the downstream + * tokenizers instead of being read as file references: + * + * - the named-symbol seeder splits on `[\s,()[\]]+`, so SvelteKit/Next + * bracketed segments (`[id]`, `[runId]`) and route groups (`(protected)`) + * exploded the path into fragments; the identifier-shaped survivors + * (`runId`, `scope`) then seeded as "symbols the agent named" and + * headlined the blast radius; + * - FTS saw the fragments (`page`, `chat`, `runs`) and admitted every + * sibling `+page.svelte` in the repo, which ate the output envelope and + * truncated the files the agent actually asked for. + * + * `extractQueryPaths` finds path-like spans — slashed paths, dotted basenames, + * and extension-less kebab basenames (`background-image-table`, the spelling + * import paths and prose actually use) — resolves them against the INDEXED + * file list (resolution IS the detector — `and/or`, `gen_server:call/2`, + * `non-blocking` and other path-shaped non-paths match nothing and are left + * alone), and returns the matches as pinned files plus the query with those + * spans removed. + * Callers treat pinned files as first-class: guaranteed admission, top rank, + * funded first. Pure string work — no DB, no fs — so it is trivially testable + * and safe inside the query-pool workers. + */ + +export interface QueryPathExtraction { + /** The query with resolved/clearly-path spans removed, whitespace-joined. */ + strippedQuery: string; + /** Indexed file paths the query named, appearance-ordered, deduped. */ + pinnedFiles: string[]; + /** + * Spans that are unambiguously path-shaped but resolved to nothing (stale + * path, unindexed file) or to too many files (bare `+page.svelte`). Stripped + * from the query — their fragments could only mint junk matches — and + * surfaced to the agent so the miss is visible instead of silent. + */ + unresolvedPathSpans: string[]; +} + +/** + * Cheap pre-gate so callers only fetch the indexed file list when the query + * could possibly contain a path: a slash, a dot-extension-shaped tail + * (`chat-manager.ts`), or a hyphen-joined word (`background-image-table` — + * kebab files are named WITHOUT their extension more often than with, so the + * shape must open the gate on its own). Extensions cap at 8 chars, which + * keeps `Class.method` spans (`app.isPackaged`) from qualifying; the kebab + * alternative requires clean non-word boundaries, which keeps `--flags` and + * snake_case-with-a-dash hybrids from firing it. + */ +export function queryMightContainPaths(query: string): boolean { + return /[/\\]/.test(query) + || /\.[A-Za-z][A-Za-z0-9]{0,7}(?=[\s,;:)\]'"`]|$)/.test(query) + || /(?:^|[^-\w])[A-Za-z0-9]+(?:-[A-Za-z0-9]+)+(?=[^-\w]|$)/.test(query); +} + +/** + * Longest span→suffix walk tried per span. 8 covers an absolute macOS path + * (`/Users//dev//…`) over a deeply nested repo-relative file; + * deeper prefixes buy nothing. + */ +const MAX_SUFFIX_TRIES = 8; +/** Spans examined per query — a prose sentence is not 50 paths. */ +const MAX_CANDIDATE_SPANS = 8; + +/** `name.ext` shape with a plausible source extension (no slash required). */ +const DOTTED_BASENAME = /^[^\s/\\]+\.[A-Za-z][A-Za-z0-9]{0,7}$/; + +/** + * Extension-less kebab basename (`background-image-table`). Hyphens are + * illegal in identifiers, so consuming these tokens can never steal one from + * the named-symbol seeder; ≥2 segments keeps single words out. + */ +const KEBAB_BASENAME = /^[A-Za-z0-9]+(?:-[A-Za-z0-9]+)+$/; + +/** A basename's last dot-extension, same shape DOTTED_BASENAME accepts. */ +const LAST_EXTENSION = /\.[A-Za-z][A-Za-z0-9]{0,7}$/; + +/** + * Lowercased basename stems of the hyphen-named indexed files, stem → paths. + * A stem drops only the LAST extension (`a-b.module.scss` → `a-b.module`), so + * a bare kebab token can't accidentally pin a same-named stylesheet or + * `.d.ts` sibling of the source file it names; an extension-less basename + * (`pre-commit`) is its own stem. Hyphen-free basenames are skipped — a + * KEBAB_BASENAME token can never equal one, and the filter keeps the map + * near-empty in repos that don't name files this way. + */ +function buildBasenameStems(indexedPaths: readonly string[]): Map { + const stems = new Map(); + for (const p of indexedPaths) { + const basename = p.slice(Math.max(p.lastIndexOf('/'), p.lastIndexOf('\\')) + 1); + if (!basename.includes('-')) continue; + const stem = basename.replace(LAST_EXTENSION, '').toLowerCase(); + if (!stem) continue; + const existing = stems.get(stem); + if (existing) existing.push(p); + else stems.set(stem, [p]); + } + return stems; +} + +/** + * Strip prose punctuation wrapped around a token without eating punctuation + * that is PART of the path: quotes/backticks always strip; a trailing `)`/`]` + * strips only when the token has no matching opener (so `(protected)` and + * `[id]` segments survive, while "…(see src/foo.ts)" loses its parenthesis); + * a leading `(`/`[` mirrors that. Trailing sentence punctuation strips last, + * so "src/foo.ts." resolves. + */ +function stripWrapping(token: string): string { + let s = token; + for (;;) { + const first = s[0]; + if (!first) break; + if ('\'"`<'.includes(first)) { s = s.slice(1); continue; } + if (first === '(' && !s.includes(')')) { s = s.slice(1); continue; } + if (first === '[' && !s.includes(']')) { s = s.slice(1); continue; } + if (first === '{' && !s.includes('}')) { s = s.slice(1); continue; } + break; + } + for (;;) { + const last = s[s.length - 1]; + if (!last) break; + if ('\'"`>.,;!?'.includes(last)) { s = s.slice(0, -1); continue; } + if (last === ')' && !s.includes('(')) { s = s.slice(0, -1); continue; } + if (last === ']' && !s.includes('[')) { s = s.slice(0, -1); continue; } + if (last === '}' && !s.includes('{')) { s = s.slice(0, -1); continue; } + break; + } + // Line references ride along in agent-written paths: `foo.ts:123`, + // `foo.ts:12-40`, `foo.ts#L88`. The file is what gets pinned. + s = s.replace(/(?::\d+(?:-\d+)?|#L\d+(?:-L?\d+)?)$/, ''); + return s; +} + +/** Normalize a span into the repo-relative shape the files table stores. */ +function normalizeSpan(span: string): string { + return span + .replace(/\\/g, '/') + .replace(/^(?:\.\/)+/, '') + .replace(/\/{2,}/g, '/') + .replace(/\/+$/, ''); +} + +/** Path-shaped beyond doubt: ≥2 segments and a dot-extension on the last. */ +function isClearlyPathShaped(normalized: string): boolean { + const slash = normalized.lastIndexOf('/'); + if (slash <= 0) return false; + return DOTTED_BASENAME.test(normalized.slice(slash + 1)); +} + +/** + * Resolve one normalized span against the indexed paths: exact match first, + * then segment-aligned suffix matches, dropping leading segments one at a + * time (so an absolute path, or one prefixed with the repo directory name, + * still lands on the indexed repo-relative file). Suffixes only get shorter — + * and therefore only match MORE — so the walk stops at the first suffix that + * matches anything: within budget it resolves, over budget it is ambiguous. + */ +function resolveSpan( + normalizedLower: string, + lowerToOriginal: ReadonlyMap, + maxMatches: number, +): { matches: string[]; ambiguous: boolean } { + const exact = lowerToOriginal.get(normalizedLower); + if (exact) return { matches: [exact], ambiguous: false }; + + const segments = normalizedLower.split('/').filter(Boolean); + const tries = Math.min(segments.length, MAX_SUFFIX_TRIES); + for (let drop = 0; drop < tries; drop++) { + const suffix = segments.slice(drop).join('/'); + if (!suffix) break; + const withSlash = '/' + suffix; + const matches: string[] = []; + for (const [lower, original] of lowerToOriginal) { + if (lower === suffix || lower.endsWith(withSlash)) { + matches.push(original); + if (matches.length > maxMatches) return { matches: [], ambiguous: true }; + } + } + if (matches.length > 0) return { matches, ambiguous: false }; + } + return { matches: [], ambiguous: false }; +} + +export function extractQueryPaths( + query: string, + indexedPaths: readonly string[], + opts: { maxPins?: number; maxMatchesPerSpan?: number } = {}, +): QueryPathExtraction { + const maxPins = Math.max(1, opts.maxPins ?? 8); + const maxMatchesPerSpan = Math.max(1, opts.maxMatchesPerSpan ?? 3); + + const passthrough: QueryPathExtraction = { + strippedQuery: query, + pinnedFiles: [], + unresolvedPathSpans: [], + }; + if (!query.trim() || indexedPaths.length === 0) return passthrough; + + // Lowercase view of the index, built once per call. Last writer wins on a + // case-colliding pair, which is the existing file-view behavior too. + const lowerToOriginal = new Map(); + for (const p of indexedPaths) lowerToOriginal.set(p.toLowerCase(), p); + + const tokens = query.split(/\s+/).filter(Boolean); + const consumed = new Set(); + const pinned: string[] = []; + const pinnedSeen = new Set(); + const unresolved: string[] = []; + let candidatesExamined = 0; + + for (let i = 0; i < tokens.length; i++) { + if (pinned.length >= maxPins) break; + if (candidatesExamined >= MAX_CANDIDATE_SPANS) break; + const stripped = stripWrapping(tokens[i]!); + if (stripped.length < 4) continue; + const hasSlash = /[/\\]/.test(stripped); + if (!hasSlash && !DOTTED_BASENAME.test(stripped)) continue; + + const normalized = normalizeSpan(stripped); + if (!normalized) continue; + candidatesExamined++; + + const { matches, ambiguous } = resolveSpan( + normalized.toLowerCase(), lowerToOriginal, maxMatchesPerSpan, + ); + if (matches.length > 0) { + consumed.add(i); + for (const m of matches) { + if (pinnedSeen.has(m) || pinned.length >= maxPins) continue; + pinnedSeen.add(m); + pinned.push(m); + } + } else if (ambiguous || isClearlyPathShaped(normalized)) { + // A real path that didn't resolve to a usable set. Keeping it in the + // query is strictly worse — its fragments are what minted the junk + // matches this module exists to stop — so strip it and say so. + consumed.add(i); + if (unresolved.length < 4) unresolved.push(normalized); + } + // Anything else (`and/or`, `call/2`, `foo.Bar`) is not a path reference: + // leave the token for the normal matching pipeline. + } + + // Second pass — extension-less kebab basenames. `background-image-table` + // opens no door above (no slash, no dotted tail), the hyphens disqualify it + // from the named-symbol seeder downstream, and FTS shreds it into the most + // common words in a kebab-cased repo (`background`, `image`, `table`) — + // which admit look-alike SIBLINGS that crowd out the named file. Resolution + // stays the detector: a token pins only when its whole lowercased form is + // the stem of an indexed basename. Two deliberate asymmetries vs the first + // pass: prose that resolves to nothing (`non-blocking`, `cross-call`) is + // LEFT IN the query — unlike a slashed span it may be legitimate wording, + // so it keeps feeding FTS and is not reported as an unresolved path — and a + // stem hotter than maxMatchesPerSpan is likewise left alone (pinning half a + // monorepo off one hot name trades precision the wrong way; a directory + // segment, which the first pass handles, disambiguates). Runs after the + // slashed/dotted pass so explicit paths win the shared maxPins budget, and + // examines every remaining token: lookups are O(1) map hits, so the + // scan-cost rationale behind MAX_CANDIDATE_SPANS doesn't apply. + let basenameStems: Map | null = null; + for (let i = 0; i < tokens.length && pinned.length < maxPins; i++) { + if (consumed.has(i)) continue; + const stripped = stripWrapping(tokens[i]!); + if (stripped.length < 4 || !KEBAB_BASENAME.test(stripped)) continue; + basenameStems ??= buildBasenameStems(indexedPaths); + const matches = basenameStems.get(stripped.toLowerCase()); + if (!matches || matches.length > maxMatchesPerSpan) continue; + consumed.add(i); + for (const m of matches) { + if (pinnedSeen.has(m) || pinned.length >= maxPins) continue; + pinnedSeen.add(m); + pinned.push(m); + } + } + + if (consumed.size === 0) return passthrough; + return { + strippedQuery: tokens.filter((_, i) => !consumed.has(i)).join(' '), + pinnedFiles: pinned, + unresolvedPathSpans: unresolved, + }; +} diff --git a/src/search/query-utils.ts b/src/search/query-utils.ts index e3db25b1a..6600d1554 100644 --- a/src/search/query-utils.ts +++ b/src/search/query-utils.ts @@ -222,6 +222,7 @@ export function scorePathRelevance( filePath: string, query: string, projectNameTokens?: Set, + isDeprioritized?: boolean, ): number { const pathLower = filePath.toLowerCase(); const fileName = path.basename(filePath).toLowerCase(); @@ -264,10 +265,19 @@ export function scorePathRelevance( else if (subtokens.some((t) => pathLower.includes(t))) score += 3; } - // Deprioritize test files unless the query is explicitly about tests + // Deprioritize test files unless the query is explicitly about tests, and + // apply the same -15 to a path the project declared peripheral (#982). + // + // Two deliberate asymmetries, both pinned by tests: + // - the built-in test/fixture penalty is waived for a test-y query, because + // the tool inferred that classification; a `deprioritize` pattern is a + // standing statement by the project, so it is NOT waived. The name-bonus + // damping at the call site is what keeps such a tree findable. + // - a path that is both is docked ONCE, not twice. const queryLower = query.toLowerCase(); const isTestQuery = queryLower.includes('test') || queryLower.includes('spec'); - if (!isTestQuery && isTestFile(filePath)) { + const offTarget = (!isTestQuery && isTestFile(filePath)) || isDeprioritized === true; + if (offTarget) { score -= 15; } diff --git a/src/sync/watcher.ts b/src/sync/watcher.ts index fed6ea608..034be858a 100644 --- a/src/sync/watcher.ts +++ b/src/sync/watcher.ts @@ -34,7 +34,7 @@ import * as fs from 'fs'; import * as path from 'path'; import { isSourceFile, buildScopeIgnore, type ScopeIgnore } from '../extraction'; -import { loadExtensionOverrides } from '../project-config'; +import { loadExtensionOverrides, PROJECT_CONFIG_FILENAME } from '../project-config'; import { logDebug, logWarn } from '../errors'; import { normalizePath } from '../utils'; import { isCodeGraphDataDir } from '../directory'; @@ -328,11 +328,13 @@ export class FileWatcher { * deterministically gate on watcher readiness. */ private readyWaiters: Array<() => void> = []; - // The shared scope matcher (built-in defaults + project .gitignore, with - // embedded child repos matched by their OWN rules — #514), built once at - // start(). Same source of truth the indexer uses, so watcher scope can - // never diverge from index scope. An embedded repo created after start() - // joins the scope on the next watcher restart / re-index. + // The shared scope matcher (built-in defaults + project .gitignore + the + // `codegraph.json` exclude/include rules, with embedded child repos matched + // by their OWN rules — #514), built at start() and REBUILT whenever one of + // the files it is derived from changes (see `refreshScope`, #1590). Same + // source of truth the indexer uses, so watcher scope can never diverge from + // index scope. An embedded repo created after start() joins the scope on + // the next scope refresh / watcher restart / re-index. private ignoreMatcher: ScopeIgnore | null = null; private readonly projectRoot: string; @@ -573,7 +575,24 @@ export class FileWatcher { private handleChange(rel: string): void { if (!rel || rel === '.' || rel.startsWith('..')) return; if (this.isAlwaysIgnored(rel)) return; + // The two root files the scope matcher is derived from are handled BEFORE + // the matcher is consulted: a user `exclude` pattern that happens to cover + // them (`*.json`, `.*`) must not be able to hide their own edits (#1590). + if (rel === PROJECT_CONFIG_FILENAME || rel === '.gitignore') { + this.refreshScope(rel); + return; + } if (this.ignoreMatcher && this.ignoreMatcher.ignores(rel)) return; + // A nested `.gitignore` (an embedded child repo's own rules, #514, or a + // subdirectory rule the git-backed full scan honors) is only a scope + // change when it sits INSIDE the current scope — checked after the matcher + // on purpose, so the thousands of package-local `.gitignore`s an + // `npm install` writes under an ignored `node_modules/` never trigger a + // rebuild storm. + if (rel.endsWith('/.gitignore')) { + this.refreshScope(rel); + return; + } if (!isSourceFile(rel, loadExtensionOverrides(this.projectRoot))) { this.maybeScheduleForRemovedDir(rel); return; @@ -591,6 +610,34 @@ export class FileWatcher { this.scheduleSync(); } + /** + * A scope-defining file changed (`codegraph.json`, a `.gitignore`): rebuild + * the ignore matcher and make the next sync a FULL reconcile (#1590). + * + * The matcher used to be built once in `start()` and kept for the watcher's + * lifetime — in a long-lived MCP daemon that meant a `codegraph.json` + * created or edited after startup was invisible to the live watcher, while + * `codegraph sync` (a fresh process) honoured it immediately: the CLI + * removed a newly excluded file and the watcher re-added it seconds later. + * `loadExtensionOverrides()` on the same filter line was already read live + * (mtime-cached), so two fields of the same config file disagreed. + * + * Rebuilding costs one `git ls-files` pass (embedded-repo discovery), which + * is fine per config edit — never per event. Replacing the field is enough + * for both strategies: the recursive handler and the per-directory + * `shouldIgnoreDir` walk read `this.ignoreMatcher` on every call. The full + * scan is required because a scope change has no per-file events: newly + * excluded files must be REMOVED from the index and newly included ones + * added, and only the scan-diff (which builds its own fresh matcher) knows + * which those are. + */ + private refreshScope(rel: string): void { + logDebug('Scope config changed; rebuilding watcher scope', { file: rel }); + this.ignoreMatcher = buildScopeIgnore(this.projectRoot); + this.needsFullScan = true; + this.scheduleSync(); + } + /** * A deleted DIRECTORY arrives as one event on the directory's own path — * no source extension, so the source-file filter drops it, and the files diff --git a/src/types.ts b/src/types.ts index b0ebfe433..186f57adc 100644 --- a/src/types.ts +++ b/src/types.ts @@ -688,4 +688,13 @@ export interface FindRelevantContextOptions { /** Node types to include */ nodeKinds?: NodeKind[]; + + /** + * Extra symbol names to merge in as exact-name search candidates, at a + * dampened score. Fed by the segment-vocabulary supplement (CodeGraph. + * findRelevantContext): word-level query terms can't reach camelCase names + * through FTS — `pinFeedIfNearBottom` is one FTS token — so names whose + * SEGMENTS the query's words name are seeded here instead. + */ + seedNames?: string[]; }