A failed release should never touch production

There are two ways a deploy pipeline can handle a broken release.

The first deploys it, notices it’s broken, and rolls back. Customers may see errors for a few minutes. Someone gets paged.

The second never deploys it at all. The checks fail, the pipeline stops, the team gets a notification, and production keeps serving the version it already had. Nobody outside the team ever knows.

I build the second kind. Rollbacks are a safety net, not a plan.

What “gated” means

A gated pipeline runs every check before anything reaches production, and a single failure stops the release. On the fintech platform I worked on, production only deploys after all of these pass:

1. Lint: code style and obvious mistakes
2. Vulnerability scan: known security issues in the code and its dependencies
3. System checks: the build and its runtime requirements are sound
4. Configuration checks: the config for the target environment is valid
5. Business-logic tests: the parts of the system that move money still behave correctly

If any step fails, the release is blocked. Nothing was deployed, so there’s nothing to roll back.

$ release → production
  lint ................. ok
  vulnerability scan ... ok
  system checks ........ ok
  config checks ........ ok
  business-logic tests . FAILED
✗ Release blocked. Nothing was deployed.
  team notified · production unchanged

Order the checks cheapest first

Put fast checks before slow ones. A formatting error should fail in seconds, not after a ten-minute test run. Developers get feedback quickly, and the pipeline only spends time on releases that have a chance of passing.

Separate environments, same pipeline

Dev, staging and production all run through the same pipeline, with production requiring every gate to pass. In GitHub Actions, environments make this straightforward:

jobs:
  checks:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: make lint
      - run: make vulncheck
      - run: make test

  deploy-production:
    needs: checks          # never runs unless every check passed
    environment: production
    runs-on: ubuntu-latest
    steps:
      - run: make deploy-prod

The needs: checks line is the gate. GitHub Actions also lets you add required reviewers to the production environment if you want a human approval on top.

Zero downtime for the releases that pass

Once a release passes, the deploy itself shouldn’t cause an outage either. Start the new version alongside the old one, check that it answers its health check, then switch traffic over. If the new version doesn’t come up healthy, the old one never stopped serving.

What changed for the team

Before this pipeline, developers built and deployed binaries by hand. After it:

• More releases, deployed more often each day
• Confidence that a broken build can’t reach customers
• No downtime during deploys
• A record of every release and why any failed

The surprising part is that stricter gates made the team deploy more, not less. When a bad release can’t hurt production, shipping stops being scary.

Start small

You don’t need all five gates on day one. Start with tests and a hard rule that production only deploys after they pass. Add the vulnerability scan next. Grow from there.

The rule that matters is simple: if a check fails, nothing ships.