Gate a merge into main
Run one RunStory check in your pipeline when a branch like dev is about to merge into main, and let its exit code decide.
Where the check fits
Work lands on a branch like dev all week: agent commits, small pull requests, quick fixes. You don't want a check on each one. You want one check before dev reaches main, covering everything that changed, running together.
Five merged pull requests wait on dev. One check covers them all, and the merge into main waits on the job's exit code.
Each change may have passed on its own. This check runs them as one app, which is where two changes that break each other show up.
Set it up on GitHub Actions
-
Create an API key on the keys page. A key made while an org is selected acts in that org, and the org pays for its checks.
-
Save the key as a secret of the repository.
gh secret set RUNSTORY_API_TOKEN --repo acme/store -
Run
runstory initandrunstory setuponce on your own machine, and commit.runstory/runstory.yml. That file tells the job how to build and start your app. If the app needs secret values, pass each one in the job'senv, next to the key. -
Add this workflow.
.github/workflows/runstory.yml
name: runstory
on:
pull_request:
branches: [main]
types: [opened, ready_for_review, synchronize]
jobs:
check:
if: github.event.pull_request.draft == false
runs-on: ubuntu-latest
timeout-minutes: 80
steps:
- uses: actions/checkout@v6
with:
ref: ${{ github.event.pull_request.head.sha }}
fetch-depth: 0
- run: curl -fsSL https://runstory.specstory.com/install.sh | sh
- run: echo "$HOME/.local/bin" >> "$GITHUB_PATH"
- run: runstory run "origin/${{ github.base_ref }}..HEAD" --wait --timeout 75m
env:
RUNSTORY_API_TOKEN: ${{ secrets.RUNSTORY_API_TOKEN }}
| Line | What it does |
|---|---|
branches: [main] | Starts the job only for pull requests into main |
fetch-depth: 0 | Fetches the whole history, so RunStory can find where the branch left main |
ref: ...head.sha | Builds the pull request's newest commit, not GitHub's test merge |
origin/${{ github.base_ref }}..HEAD | Checks every commit the branch has that its base, main, doesn't, as one change |
--wait --timeout 75m | Waits until the check ends. A check can run for 60 minutes |
To make the job a gate, require its check, named check, in the branch rule for main. GitHub offers it there once it has run in the repository.
Two things to know. A pull request from a fork or from Dependabot doesn't get the secret, so the job fails there. And a merge queue needs its own merge_group trigger, which this workflow doesn't have.
What the exit code means
| Exit code | The check | The job |
|---|---|---|
| 0 | Found no confirmed problems | Passes |
| 1 | Confirmed a new problem | Fails |
| 2 | Ended without a clear result | Fails. End the line with || [ $? -eq 2 ] to let it pass |
| 3 | Didn't start, e.g. the key is missing or the daily limit is used up | Fails |
| 4 | Couldn't build or start the app | Fails |
| 5 | Got no answer from the service | Fails. Run the job again |
| 8 | Was still running when the wait ended | Fails. The check goes on, so raise --timeout |
When the check ends, RunStory adds the result to the job's summary: each finding's id and title, and a link to the check's page. Hand that link to your coding agent, and it can read the check and its findings with runstory show <link> --json.
If another account connected the GitHub App to this repository, add --personal to the command. The check then runs on the key's account, with no question asked.
Run the same check on another pipeline
Any pipeline can run this check. The job needs 4 things: the whole Git history, the runstory CLI, the key in RUNSTORY_API_TOKEN, and a range to check.
curl -fsSL https://runstory.specstory.com/install.sh | sh
export PATH="$HOME/.local/bin:$PATH"
runstory run origin/main..HEAD --wait --timeout 75m
Always name the range. A plain runstory run checks only the newest commit, and it's refused where the pipeline checked out no branch.
On GitLab CI/CD, store the key as a masked variable named RUNSTORY_API_TOKEN, with Protected off, and add a job for merge requests into main.
.gitlab-ci.yml
runstory:
image: ubuntu:24.04
rules:
- if: $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == "main"
variables:
GIT_DEPTH: "0"
timeout: 80 minutes
before_script:
- apt-get update && apt-get install -y curl git openssl
- curl -fsSL https://runstory.specstory.com/install.sh | sh
- export PATH="$HOME/.local/bin:$PATH"
script:
- git fetch origin "+refs/heads/main:refs/remotes/origin/main"
- runstory run origin/main..HEAD --wait --timeout 75m
On Bitbucket Pipelines, set clone: depth: full, fetch the destination branch, and check the range origin/$BITBUCKET_PR_DESTINATION_BRANCH..$BITBUCKET_COMMIT.
Example: the Thursday merge
A team at Acme, an online furniture store, merges dev into main every Thursday. This week, pull request #220 holds 5 merged pull requests. A developer marks it ready for review, and the job starts.
Checking origin/main..HEAD, 5 commits from origin/main (9d2f0a1) to 4e7c1b8, with 23 files changed
- RunStory builds the store from
4e7c1b8and starts it. Its agents buy a sofa with a gift card, then change the delivery address. The gift card is gone from the order. - The script repeats the failure 3 more times, and a second agent confirms it. Gift cards came from #212 and the address change from #213, and each passed on its own.
- The job fails with exit code 1. Its summary lists finding
f-7c41e2a9b0d3. - A developer asks a coding agent to fix it on
dev. The agent names the finding's id in its commit message, so the next check runs that finding's script again. - The push starts the job again. It passes, and #220 merges.
What it costs
Each run of the job is one check. A default check uses 2 frontier models and costs 6 credits. With 1 model, it costs 3.
- Keep the pull request a draft while it fills up. The job skips drafts, and starts when you mark the pull request ready.
- Set a daily limit.
runstory settings --daily-limit 30caps this project's checks at 30 credits a day. For an org's key, add--organd the org's name. - Preview a check.
runstory run origin/main.. --dry-runshows what would be checked and sends nothing.