Skip to content
#

Check every pull request your agents open

Install the RunStory GitHub App once. Each push to a pull request is checked after CI passes, and the result shows up on the pull request.

#

Where the check fits

Your coding agents work on branches and open pull requests, often several at once. GitHub reports that by September 2026, more than 40% of its pull requests started with an agent's first draft. Reading every diff won't tell you whether the app still works. So the RunStory GitHub App checks each push by running it, and writes the result on the pull request as a Check named Runstory.

The RunStory GitHub App checks each push to a pull request Coding agent GitHub RunStory push once CI passes writes the Check each new push is checked again merge CI passes Pull request #213 Runstory: 1 problem found Pushes a branch and opens a pull request Pushes a fix for what the Check lists Checks the pushed commit builds it, starts it, uses it

One pull request, from its first push to its merge. The App does the same for every push to any branch by you, a teammate in your org, or a bot you've listed.

There's no workflow file to add and nothing to install on your laptop. The account that connected the repository pays for its checks.

#

Set it up

  1. Open your repositories in RunStory and press Connect GitHub.
  2. Install the Runstory app on the repository.
  3. Press Set up checks, then choose the app's folder, the models, and a daily limit in credits.
  4. If a coding agent pushes as a bot, like claude[bot], add it under Bots it checks on the GitHub App page. An agent that pushes with your own GitHub account needs no entry.
  5. Turn on Automatic checks and save. The next push to any branch is checked.

To make the check a gate, add Runstory as a required status check in the branch rule for main. A pull request whose check found problems then can't merge. A push that RunStory skipped, like a docs-only change, still passes the rule.

#

Whose pushes are checked

Who pushesChecked?
You, or a coding agent that pushes with your GitHub accountYes
A coding agent that pushes as a bot, like claude[bot]Yes, once the bot is under Bots it checks
A teammateYes, once the installation is in an org and the teammate is a member with their GitHub account linked
Anyone elseNo. The Check says why, and the push costs nothing
#

What happens to each push

The pushWhat RunStory doesThe Check says
CI passesBuilds, starts, and uses the pushed commit"No confirmed problems", or something like "2 problems found"
CI failsNothing, and it costs nothing"Not checked because CI did not pass"
Nothing that runs changed since the last checked commit, like a docs-only pushNothing, and it costs nothing"Not checked because nothing that runs changed"
No CI check shows up on the commitStarts the check after 60 secondsThe same as when CI passes

RunStory reads CI from the check runs on the commit, like a GitHub Actions workflow. It doesn't see CI that only posts a commit status.

A push with several commits gets one check, of its newest commit. Two markers in a commit message change what's checked:

  • [runstory full] checks the push whatever files it changes.
  • [skip runstory] in every commit message of a push skips it.
#

Get the result back to an agent

The Check's summary lists each finding by id and title, with a link to the check's page.

  • An agent in your terminal. Check out the pull request's branch, signed in to a RunStory account that can read the repository's checks. The next time RunStory reads results, like on runstory status, the App's results for that branch reach your checkout, and your coding agent gets the findings at the end of its turn.
  • An agent that reads GitHub. Once gh pr checks lists Runstory, the agent can read the check with runstory show <link> --json.
  • A fix. When a commit message names the finding's id, like "Keep the cart when the address changes (f-3a9c41d27b6e)", the next check runs that finding's script again.
#

Example: three pull requests before lunch

A team at Acme, an online furniture store, hands three tasks to coding agents in the morning. By noon, three pull requests are open.

Pull requestWhat happenedThe Runstory Check
#212 Gift cards at checkoutCI passed, and the check found nothingNo confirmed problems
#213 Edit the delivery addressCI passed, and the check found the cart empty after an address change1 problem found
#214 Search filtersA unit test failed in CINot checked because CI did not pass

The team merges #212. For #213, a developer asks the agent to fix finding f-3a9c41d27b6e, and the agent pushes a commit that names the id. CI passes, RunStory checks the new push, and the finding's script passes 3 times. The Check now reads "No confirmed problems", so #213 merges. #214 waits for a push whose CI passes, and it hasn't cost anything yet.

#

What it costs

A default check uses 2 frontier models and costs 6 credits.

ToDo this
Cap the repository's checksSet Daily limit on the repository's settings page. A push over the limit waits for room
Check with 1 modelTick 1 provider. Each model in a check costs 3 credits
Skip one pushPut [skip runstory] in its commit messages
Pause every checkTurn off Automatic checks. A push made while they're off isn't checked. To check one later, turn them back on and press Re-run on its Check
#

Not on GitHub.com?

The App works on GitHub.com. On GitLab or Bitbucket, start the same check from a pipeline job with an API key. Gate a merge into main shows the job.