Skip to content

CI: gate releases on acceptance tests (fix AT flakiness first) #326

Description

@leggetter

Release-hardening follow-ups from #325 (unit/build CI on push to main) and the branch-protection change that enabled "require branches up to date before merging".

Goal: guarantee that released code has passing acceptance tests.

1. Fix residual acceptance-test flakiness

The harness already retries transient API 502/500s (runWithHTTP502Retry, acceptance502MaxAttempts = 4 in test/acceptance/helpers.go). But acceptance-telemetry still flakes: when a 502 triggers a CLI-command retry, the retried invocation emits a new invocation_id, and AssertTelemetryConsistent (test/acceptance/helpers.go) requires a single consistent invocation_id across all recorded requests — so the retry itself trips the assertion.

Options:

  • Make the telemetry assertion tolerant of multiple invocation_ids produced by retried commands; or
  • Segment recorded requests per invocation and assert consistency within each; or
  • Disable the 502-retry for the telemetry-consistency tests and assert per-invocation.

Needs validation against a live (or simulated) 502.

2. Add an acceptance gate to the release workflow

Once (1) is done: add an acceptance job to .github/workflows/release.yml (the 3-slice matrix + acceptance-telemetry, using the HOOKDECK_CLI_TESTING_API_KEY* secrets) and make build-mac / build-linux / build-windows (and therefore publish-npm) needs: it — so a release only publishes when acceptance passes. Rely on GitHub "re-run failed jobs" for any residual transient failure (no re-tag needed).

Optional: expand required checks

unit-test is currently the only required status check (the anchor for strict mode). Consider adding build-mac / build-linux / build-windows once confirmed they report cleanly on all PR types (note dependabot's conditional build-linux).

Related: #325.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions