Skip to content
#

Gate a merge into main

Run one RunStory check in your pipeline when a branch like dev is about to merge into main, and let its exit code decide.

#

Where the check fits

Work lands on a branch like dev all week: agent commits, small pull requests, quick fixes. You don't want a check on each one. You want one check before dev reaches main, covering everything that changed, running together.

One RunStory check gates the merge from dev into main main dev Your CI #208 #210 #212 #213 #214 pull request into main one check of everything new on dev A job on the pull request, signed in with an API key runstory run origin/main..HEAD check 12 exit 0: merge exit 1: the job fails

Five merged pull requests wait on dev. One check covers them all, and the merge into main waits on the job's exit code.

Each change may have passed on its own. This check runs them as one app, which is where two changes that break each other show up.

#

Set it up on GitHub Actions

  1. Create an API key on the keys page. A key made while an org is selected acts in that org, and the org pays for its checks.

  2. Save the key as a secret of the repository.

    gh secret set RUNSTORY_API_TOKEN --repo acme/store
    
  3. Run runstory init and runstory setup once on your own machine, and commit .runstory/runstory.yml. That file tells the job how to build and start your app. If the app needs secret values, pass each one in the job's env, next to the key.

  4. Add this workflow.

.github/workflows/runstory.yml

name: runstory
on:
  pull_request:
    branches: [main]
    types: [opened, ready_for_review, synchronize]
jobs:
  check:
    if: github.event.pull_request.draft == false
    runs-on: ubuntu-latest
    timeout-minutes: 80
    steps:
      - uses: actions/checkout@v6
        with:
          ref: ${{ github.event.pull_request.head.sha }}
          fetch-depth: 0
      - run: curl -fsSL https://runstory.specstory.com/install.sh | sh
      - run: echo "$HOME/.local/bin" >> "$GITHUB_PATH"
      - run: runstory run "origin/${{ github.base_ref }}..HEAD" --wait --timeout 75m
        env:
          RUNSTORY_API_TOKEN: ${{ secrets.RUNSTORY_API_TOKEN }}
LineWhat it does
branches: [main]Starts the job only for pull requests into main
fetch-depth: 0Fetches the whole history, so RunStory can find where the branch left main
ref: ...head.shaBuilds the pull request's newest commit, not GitHub's test merge
origin/${{ github.base_ref }}..HEADChecks every commit the branch has that its base, main, doesn't, as one change
--wait --timeout 75mWaits until the check ends. A check can run for 60 minutes

To make the job a gate, require its check, named check, in the branch rule for main. GitHub offers it there once it has run in the repository.

Two things to know. A pull request from a fork or from Dependabot doesn't get the secret, so the job fails there. And a merge queue needs its own merge_group trigger, which this workflow doesn't have.

#

What the exit code means

Exit codeThe checkThe job
0Found no confirmed problemsPasses
1Confirmed a new problemFails
2Ended without a clear resultFails. End the line with || [ $? -eq 2 ] to let it pass
3Didn't start, e.g. the key is missing or the daily limit is used upFails
4Couldn't build or start the appFails
5Got no answer from the serviceFails. Run the job again
8Was still running when the wait endedFails. The check goes on, so raise --timeout

When the check ends, RunStory adds the result to the job's summary: each finding's id and title, and a link to the check's page. Hand that link to your coding agent, and it can read the check and its findings with runstory show <link> --json.

If another account connected the GitHub App to this repository, add --personal to the command. The check then runs on the key's account, with no question asked.

#

Run the same check on another pipeline

Any pipeline can run this check. The job needs 4 things: the whole Git history, the runstory CLI, the key in RUNSTORY_API_TOKEN, and a range to check.

curl -fsSL https://runstory.specstory.com/install.sh | sh
export PATH="$HOME/.local/bin:$PATH"
runstory run origin/main..HEAD --wait --timeout 75m

Always name the range. A plain runstory run checks only the newest commit, and it's refused where the pipeline checked out no branch.

On GitLab CI/CD, store the key as a masked variable named RUNSTORY_API_TOKEN, with Protected off, and add a job for merge requests into main.

.gitlab-ci.yml

runstory:
  image: ubuntu:24.04
  rules:
    - if: $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == "main"
  variables:
    GIT_DEPTH: "0"
  timeout: 80 minutes
  before_script:
    - apt-get update && apt-get install -y curl git openssl
    - curl -fsSL https://runstory.specstory.com/install.sh | sh
    - export PATH="$HOME/.local/bin:$PATH"
  script:
    - git fetch origin "+refs/heads/main:refs/remotes/origin/main"
    - runstory run origin/main..HEAD --wait --timeout 75m

On Bitbucket Pipelines, set clone: depth: full, fetch the destination branch, and check the range origin/$BITBUCKET_PR_DESTINATION_BRANCH..$BITBUCKET_COMMIT.

#

Example: the Thursday merge

A team at Acme, an online furniture store, merges dev into main every Thursday. This week, pull request #220 holds 5 merged pull requests. A developer marks it ready for review, and the job starts.

Checking origin/main..HEAD, 5 commits from origin/main (9d2f0a1) to 4e7c1b8, with 23 files changed
  1. RunStory builds the store from 4e7c1b8 and starts it. Its agents buy a sofa with a gift card, then change the delivery address. The gift card is gone from the order.
  2. The script repeats the failure 3 more times, and a second agent confirms it. Gift cards came from #212 and the address change from #213, and each passed on its own.
  3. The job fails with exit code 1. Its summary lists finding f-7c41e2a9b0d3.
  4. A developer asks a coding agent to fix it on dev. The agent names the finding's id in its commit message, so the next check runs that finding's script again.
  5. The push starts the job again. It passes, and #220 merges.
#

What it costs

Each run of the job is one check. A default check uses 2 frontier models and costs 6 credits. With 1 model, it costs 3.

  • Keep the pull request a draft while it fills up. The job skips drafts, and starts when you mark the pull request ready.
  • Set a daily limit. runstory settings --daily-limit 30 caps this project's checks at 30 credits a day. For an org's key, add --org and the org's name.
  • Preview a check. runstory run origin/main.. --dry-run shows what would be checked and sends nothing.