NightlyQAStatus
Guide · NightlyQA

Green CI, broken signup: why a passing deploy is not the end of the check

Your pipeline passed, the deploy went out and the homepage returns 200. A new user still cannot sign up. This guide covers why that gap exists and what checking the live workflow looks like.

Where the gap comes from

Unit and integration tests run against the code you changed, usually with test doubles in place of the real database, mail provider and third-party scripts. An uptime check asks one question: does the homepage answer? Neither one opens the live product the way a new user does and follows the signup path to the end.

So a deploy can be green while something only the live site shows is broken: a form that says "Success" but never saves the record, a page that now redirects to a login wall, an API call that returns a server error, or a section hidden behind a feature gate that was switched on for everyone.

What checking the live workflow means

NightlyQA opens the deployed product in a real Chromium browser on a schedule and checks it in two ways:

  • A golden map. A list of pages that must stay reachable and ungated. Each one is opened and fails on an HTTP error, a redirect onto a login path, an inline gate marker, a subresource returning 5xx, an unhandled console error or a failed xhr/fetch call. What a golden map checks lists the rules.
  • Declared workflows. Steps such as goto, fill, click and expect_text, plus an assert_persisted step that checks the record actually exists afterwards. Form fills use run-tagged test data (nw-<date>-<rand>) so test records are easy to find and clean up. So far these workflow steps have been tested against a local fixture app, not yet against a live signup.

What you get when it breaks

A failure becomes a GitHub issue with an evidence pack: screenshot, video, console log, network log, page DOM and repro steps. The same failure on the next run updates the existing issue instead of opening a new one. Evidence-backed bug reports covers the format, and the redacted sample finding shows a real one from a local fixture.

What this does not replace

A nightly walk is not a substitute for tests in CI. It runs after the deploy, against what users actually see, and it only checks the pages and workflows you declare. It does not claim full coverage.

NightlyQA is an early-stage product in portfolio dogfood. Public onboarding is not open yet. Request early access.