fix(backend): cap repo-index metric cardinality and add OOM diagnostics

[better-stack-hq]

💬 AI SRE · 👥 Sourcebot team · 👤 Requested by Brendan Kellam

Summary

• Add --heapsnapshot-near-heap-limit=3 (writing to a persisted DATA_CACHE_DIR/heap-snapshots directory) to the backend supervisord program, so if the process approaches its heap limit again, we get an actual heap snapshot to diagnose the retained object graph — instead of only the generic FATAL ERROR: JavaScript heap out of memory crash log.
• Fix an identified metric-cardinality leak: PromClient's repo-scoped metrics (activeRepoIndexJobs, pendingRepoIndexJobs, repoIndexJob{Reattempts,Fail,Success}Total) are keyed by repo name but are never cleared. Every distinct repo name ever observed over the process's lifetime stays resident in the prom-client registry forever, even after the repo is deleted. Add PromClient.removeRepoMetrics() and call it once a repo is actually deleted in RepoIndexManager's cleanup job.

Root cause

app.sourcebot.dev's backend process (node packages/backend/dist/index.js) hit FATAL ERROR: Ineffective mark-compacts near heap limit ... JavaScript heap out of memory after ~20.7h of uptime, causing a ~48s window of 503s while supervisord restarted the container.

Static review of the backend surfaced one confirmed correctness bug that contributes to unbounded heap growth over the process's lifetime (unbounded Prometheus label cardinality for repo names that are never released). Given the repo count in this deployment (~150-160, fairly stable), this alone may not fully account for the ~4GB heap exhausted, so this PR also adds heap-snapshot-on-near-limit diagnostics to get a definitive answer if this recurs, rather than guessing further from static analysis alone.

Test plan

• Could not run a full workspace build in this sandbox (no package registry network access) — please run yarn workspace @sourcebot/backend build and yarn workspace @sourcebot/backend test in CI to confirm the TypeScript changes compile.
• Verify supervisord status backend picks up the new NODE_OPTIONS and that $DATA_CACHE_DIR/heap-snapshots exists after container start.
• Confirm /metrics no longer shows stale repo="<deleted-repo>" series after a repo is deleted and its CLEANUP job runs.

---

🤖 Generated with Better Stack AI SRE

---

Note

Medium Risk
Changes long-running backend process behavior (metrics lifecycle and NODE_OPTIONS) in response to production OOMs; logic is narrow but affects observability and memory under load.

Overview
Addresses backend heap growth and OOM crashes by bounding Prometheus memory and adding post-mortem heap dumps.

Metric cleanup: Adds PromClient.removeRepoMetrics() to drop all repo-labeled index/cleanup time series when a repo is permanently removed, and invokes it from the CLEANUP job path right after the DB delete. This stops prom-client from retaining every historical repo label combination for the life of the process.

OOM diagnostics: Ensures $DATA_CACHE_DIR/heap-snapshots exists at container start and configures the backend supervisord program with NODE_OPTIONS (--heapsnapshot-near-heap-limit=3 and --diagnostic-dir pointing at that directory) so near-limit heap pressure writes snapshots for analysis instead of only a generic OOM log.

^{Reviewed by Cursor Bugbot for commit 92ff09a. Bugbot is set up for automated code reviews on this repo. Configure here.}

[github-actions]

@better-stack-hq[bot] your pull request is missing a changelog!

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

Want reviews to match your repository better? Bugbot Learning can learn team-specific rules from PR activity. A team admin can enable Learning in the Cursor dashboard.

^{Reviewed by Cursor Bugbot for commit 92ff09a. Configure here.}