gather: deterministic context-pack tool (aft_gather)

[iceteaSA]

gather: deterministic context-pack tool (aft_gather)

What

One new tool — aft_gather — assembles a bounded "context pack" (ranked, deduped, budgeted verbatim code evidence) in a single call. It replaces the multi-turn search → outline → zoom → callgraph read chain an agent otherwise runs to build context around a question or a symbol.

Two modes (mutually exclusive):

• question: "how does X work?" — seeds from handle_semantic_search (same pipeline as aft_search, all lanes/fallbacks)
• symbol + filePath — seeds from the callgraph (impact depth-1 callers + call_tree depth-1 callees)

Seeds expand one hop through the callgraph, dedupe by canonicalized (file, symbol) with seeds winning, and render via render_symbol_within_budget until a hard line budget (default 400, cap 800) is spent. Everything past the cut appears as one-line stubs under ## Beyond budget (zoom to expand) — nothing is silently dropped.

Why

Agents burn serial turns assembling context: search, then outline the hits, then zoom the symbols, then chase callers. Each turn round-trips through the model. A pack returns evidence (verbatim bodies with file:line headers), not conclusions — the agent reasons over it directly, ready to attach to a subagent dispatch.

Measured on a real config repo (3 questions, one call each vs. the manual chain):

• "how does the lane-verdict cache flow end-to-end" → complete 4-file chain (cache shapes, verdict logic, write/read, plugin deny hook) in one pack, used=283/400. Manual baseline: 5-6 tool calls.
• "how does score-tap decide when to write an executor row" → the full decision chain (event filter → dedup → context sources → verdict parse → DB insert) in one pack at budget=200.
• An unrehearsed cross-file question → correct core symbols plus their docstrings, first try.

Independently reproduced on the aft codebase itself: question: "how does bash output compression dispatch pick a compressor" → seeds=15, used=226/400, one pack assembling the full dispatch chain (compress → gate → compress_with_registry_exit_code 20-compressor array → Compressor trait → install path → subc mirror) that otherwise takes a 4–5-call search→zoom chain.

Honest degradation

The pack never lies about its own quality:

• While the semantic index is building, long NL queries degrade to lexical-only file-level hits. The pack renders them as visible file:line (no containing symbol) stubs and flags the header with degraded=semantic-index-building (partial results — retry when index ready) — detected via the response's semantic_status field, cleared as soon as one real seed resolves. No blocking or retry inside the tool.
• Grep-fallback hits ({file, line_text, line}, no symbol name) resolve to their containing symbol by line containment — definitions, call sites, and comment hits all upgrade to the enclosing symbol. Hits with no containing symbol stay visible as stubs.
• Unresolved external/stdlib callees collapse to one summary line ((N unresolved external calls omitted)) instead of drowning the stub list; unresolved seeds and callers are never suppressed.

Implementation

• crates/aft/src/commands/gather.rs — Rust-side composition: calls handle_semantic_search / impact_result / call_tree_result / render_symbol_within_budget directly (shared &AppContext, no bridge round-trips, no parallel reimplementation of search).
• Wiring: main.rs dispatch arm, subc_translate.rs mapping, TS factory packages/opencode-plugin/src/tools/gather.ts + registration (same tier as aft_callgraph — depends on the callgraph store).
• No new dependencies, no config surface beyond the tool args, no LLM calls, no caching.
• Follows the tri-state honest-reporting convention (protocol.rs Response doc-comment): success:false+code for un-performable calls (e.g. invalid_request on a bad mode combo), success:true with a visible degraded/stub pack for partial results — never a bare empty success.

Tests

22 unit tests in gather.rs, including red-checked regressions (each confirmed to fail against pre-fix code): mid-codepoint truncation panic, duplicate-symbol line-anchored resolution, abs/rel path dedupe, containing-symbol resolution via a real TreeSitterProvider, callee-only stub suppression driven through the production build_pack path, and degradation-flag presence/absence/mixed cases.

Limitations (deliberate scope)

• 1-hop expansion only — multi-hop was deliberately excluded: the budget math and ranking get harder and the packs get noisier.
• Callgraph-dependent by design: neighbor expansion quality follows the callgraph store's freshness, same as aft_callgraph.
• Budget counted in lines, not tokens — matches aft's existing budget idiom across zoom/outline.

---

^{Need help on this PR? Tag /codesmith with what you need. Autofix is disabled.}

---

Summary by cubic

Adds aft_gather_context, a single‑call, deterministic context‑pack builder that returns ranked, deduped, verbatim code within a fixed line budget, replacing the multi‑step search→outline→zoom→callgraph chain. Exposed as gather in Rust and aft_gather_context in @opencode on the “all” surface.

• New Features

  • Two modes (XOR): question (semantic seeds) or symbol+filePath (impact callers + call‑tree callees), 1‑hop expansion; dedupe by (file, symbol) with seeds winning.
  • Hard line budget (default 400, max 800) with per‑symbol balancing (min 10, cap 150); overflow listed under “Beyond budget”; unresolved external callees collapse to one summary; unresolved seeds/callers stay visible.
  • Grep‑fallback hits resolve to containing symbols; no‑symbol hits render as visible stubs.
  • Honest header flags: degraded=semantic-index-building and neighbors=skipped(callgraph-unavailable); exact used lines computed post‑render.
  • Robustness: repo‑relative path resolution, abs/rel normalization, and cross‑platform path‑separator normalization to dedupe absolute vs relative on Windows; a literal backslash in Unix filenames normalizes to / and yields a clean “file not found” stub. Unicode‑safe query truncation; best‑match selection for same‑name symbols by start_line. Docs and inventory updated; registered in @opencode with XOR schema validation.

• Migration

  • Call with { question } or { symbol, filePath }; optional budget 1–800 (default 400).
  • Neighbor expansion requires the callgraph store in symbol mode; in question mode, neighbors are skipped with a header notice if unavailable.

^{Written for commit b129cce. Summary will update on new commits.}

Greptile Summary

This PR introduces aft_gather — a single-call context-pack builder that replaces the multi-turn search→outline→zoom→callgraph chain agents previously needed to assemble code evidence. The implementation is thorough, with 22 unit tests covering regressions (mid-codepoint truncation, abs/rel path dedupe, containing-symbol resolution, callee suppression) and honest degradation reporting via header flags.

• New tool aft_gather_context: supports two mutually exclusive modes — question (semantic-search seeded) and symbol+filePath (impact/call-tree seeded) — with 1-hop callgraph neighbor expansion, score-ranked deduplication, and a hard line budget (default 400, cap 800) that relegates overflow to visible stubs.
• Budget accounting fixed: previous issues with separator-line undercounting and in-place used= patching are resolved; the header is now built last with exact counts.
• Wiring: Rust dispatch arm in main.rs, translate_gather in subc_translate.rs with its own XOR validation and filePath resolution, TS tool definition and registration in the all surface tier alongside aft_callgraph.

Confidence Score: 5/5

• The core budget accounting is correct, deduplication logic is sound, and all error paths return informative responses — no silent drops.
• All findings are non-blocking quality observations: select_symbol_match has a latent panic that is currently prevented by its caller's guard, hop_distance is dead code with a misleading comment, and handle_gather's XOR error message is imprecise for a case that's already caught by the translation layer. The logic, budget math, test coverage, and wiring are all solid.
• crates/aft/src/commands/gather.rs — the select_symbol_match caller invariant and hop_distance dead code are worth a second look before this grows further callers.

Important Files Changed

Filename	Overview
crates/aft/src/commands/gather.rs	New 1494-line implementation of aft_gather with 22 unit tests; core logic is sound, budget accounting is correct post-fix, but select_symbol_match has a latent panic on empty slice (caller prevents it) and hop_distance is documented as driving interleaving but is dead code in build_pack.
packages/opencode-plugin/src/tools/gather.ts	TS tool definition for aft_gather_context; XOR schema validation mirrors Rust-side logic, budget constraints are enforced, response error handling is consistent with sibling tools.
packages/opencode-plugin/src/tool-registration.ts	Adds aft_gather_context to ALL_ONLY_TOOLS and includes gatherTools in the map; registration is consistent with aft_callgraph (same "all"-surface tier).
crates/aft/src/subc_translate.rs	Adds translate_gather with proper XOR validation, path resolution for filePath, and passthrough of question/symbol/budget; validation is more precise than the Rust handler's own guard (separate "symbol without filePath" error).
crates/aft/src/commands/mod.rs	Single-line module declaration for gather.
crates/aft/src/main.rs	Single dispatch arm added for "gather" → handle_gather; no other changes.

Sequence Diagram

sequenceDiagram
    participant Agent
    participant gather.ts as gather.ts (TS)
    participant translate as subc_translate.rs
    participant handle as gather.rs (Rust)
    participant search as handle_semantic_search
    participant callgraph as callgraph_store
    participant render as render_symbol_within_budget

    Agent->>gather.ts: "aft_gather_context({ question | symbol+filePath, budget })"
    gather.ts->>gather.ts: XOR validate modes
    gather.ts->>translate: callToolCall("gather", rawArgs)
    translate->>translate: translate_gather() — validate + resolve filePath
    translate->>handle: handle_gather(req, ctx)

    alt question mode
        handle->>search: "handle_semantic_search(top_k=15)"
        search-->>handle: results + semantic_status
        handle->>handle: resolve grep-fallback hits via resolve_containing_symbol
        handle->>callgraph: callgraph_store_for_ops() — availability check
        handle->>callgraph: impact_result + call_tree_result per seed (1-hop)
        callgraph-->>handle: callers + callees (or empty if unavailable)
    else symbol mode
        handle->>callgraph: "impact_result(depth=1) — callers"
        handle->>callgraph: "call_tree_result(depth=1) — callees"
        callgraph-->>handle: neighbors
    end

    handle->>handle: dedupe by (file, name), seeds win
    handle->>handle: sort seeds by score, interleave neighbors per seed_ordinal

    loop each candidate until budget exhausted
        handle->>render: render_symbol_within_budget(per_symbol_budget)
        render-->>handle: "section text | truncated | stub"
        handle->>handle: "budget accounting (lines_used += section_lines + 1)"
    end

    handle->>handle: "build header with final used= count"
    handle-->>gather.ts: "{ text: pack_output }"
    gather.ts-->>Agent: pack_output string

Loading

_{Reviews (15): Last reviewed commit: "gather: deterministic context-pack tool ..." | Re-trigger Greptile}

[cubic-dev-ai]

4 issues found across 7 files

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[iceteaSA]

Follow-up on the two maintainability notes from the Greptile summary (they weren't separate review threads, so noting here) — both addressed in cf09b9b9:

• degraded expression flagged as always-false in the non-empty-seeds path — intentional: degraded only applies when zero symbol-level seeds resolve (a building index yields no seeds; a ready index with seeds is never degraded). Logic unchanged; added a comment on the why so it doesn't read as dead code.
• callee-suppression string-match coupling — extracted UNRESOLVED_MARKER ("(symbol not resolved)") and CALLEE_PROVENANCE_PREFIX ("callee-of-") consts, referenced by both the producer (render_symbol_section err arm / collect_callees_for_seed) and the consumer (build_pack suppression guard) so the match can't silently drift. Only test-site literals remain, intentionally — a test pointing at the const couldn't catch const drift.

Matched/rendered text is byte-identical; 23/23 gather tests green.

[iceteaSA]

Rebased onto current main (690bed59) and renamed the tool to aft_gather_context — the longer name reads more clearly at the call site.

The rename covers the advertised tool name and the agent-facing error strings only. The wire command stays "gather" (TS→Rust transport), as do the file and function names, matching how aft_callgraph maps to callers/call_tree/impact from navigation.ts. Both manifest entries were updated: the REG-V049-OC-ALL set and the HOSTONLY-V049-006 allowlist row.

Rebase conflicts were confined to ARCHITECTURE.md and STRUCTURE.md; all code auto-merged. tools/structure.ts and tools/lsp.ts are gone from main, so I kept your shorter tool lists and only re-inserted gather.ts.

Verified: registration-parity 10, tool-surface-transport-invariant 2, tools 10, cargo test gather 23, bun run build and cargo build clean. The built bundle contains aft_gather_context and zero occurrences of the old name.

Still one commit. Two notes on the full plugin suite, both pre-existing on main rather than from this branch: the two e2e outline command failures and the format_on_edit ones. main's own CI is currently red on 1d1a6968, 1da22bfe, and ef57355f, and 690bed59's run was cancelled — happy to hold this until that's sorted if you'd rather rebase onto a green base.