Release-hardening follow-ups from #325 (unit/build CI on push to main) and the branch-protection change that enabled "require branches up to date before merging".
Goal: guarantee that released code has passing acceptance tests.
1. Fix residual acceptance-test flakiness
The harness already retries transient API 502/500s (runWithHTTP502Retry, acceptance502MaxAttempts = 4 in test/acceptance/helpers.go). But acceptance-telemetry still flakes: when a 502 triggers a CLI-command retry, the retried invocation emits a new invocation_id, and AssertTelemetryConsistent (test/acceptance/helpers.go) requires a single consistent invocation_id across all recorded requests — so the retry itself trips the assertion.
Options:
- Make the telemetry assertion tolerant of multiple
invocation_ids produced by retried commands; or
- Segment recorded requests per invocation and assert consistency within each; or
- Disable the 502-retry for the telemetry-consistency tests and assert per-invocation.
Needs validation against a live (or simulated) 502.
2. Add an acceptance gate to the release workflow
Once (1) is done: add an acceptance job to .github/workflows/release.yml (the 3-slice matrix + acceptance-telemetry, using the HOOKDECK_CLI_TESTING_API_KEY* secrets) and make build-mac / build-linux / build-windows (and therefore publish-npm) needs: it — so a release only publishes when acceptance passes. Rely on GitHub "re-run failed jobs" for any residual transient failure (no re-tag needed).
Optional: expand required checks
unit-test is currently the only required status check (the anchor for strict mode). Consider adding build-mac / build-linux / build-windows once confirmed they report cleanly on all PR types (note dependabot's conditional build-linux).
Related: #325.
Release-hardening follow-ups from #325 (unit/build CI on push to
main) and the branch-protection change that enabled "require branches up to date before merging".Goal: guarantee that released code has passing acceptance tests.
1. Fix residual acceptance-test flakiness
The harness already retries transient API 502/500s (
runWithHTTP502Retry,acceptance502MaxAttempts = 4intest/acceptance/helpers.go). Butacceptance-telemetrystill flakes: when a 502 triggers a CLI-command retry, the retried invocation emits a newinvocation_id, andAssertTelemetryConsistent(test/acceptance/helpers.go) requires a single consistentinvocation_idacross all recorded requests — so the retry itself trips the assertion.Options:
invocation_ids produced by retried commands; orNeeds validation against a live (or simulated) 502.
2. Add an acceptance gate to the release workflow
Once (1) is done: add an
acceptancejob to.github/workflows/release.yml(the 3-slice matrix +acceptance-telemetry, using theHOOKDECK_CLI_TESTING_API_KEY*secrets) and makebuild-mac/build-linux/build-windows(and thereforepublish-npm)needs:it — so a release only publishes when acceptance passes. Rely on GitHub "re-run failed jobs" for any residual transient failure (no re-tag needed).Optional: expand required checks
unit-testis currently the only required status check (the anchor for strict mode). Consider addingbuild-mac/build-linux/build-windowsonce confirmed they report cleanly on all PR types (note dependabot's conditionalbuild-linux).Related: #325.