Skip to content
#

Check an agent that runs without you

Put an API key where the agent runs, and let the loop's own script start each check and hand failures to the next turn.

#

Where the check fits

A script keeps a coding agent working through a list of tasks for hours, with nobody at the keyboard. The agent's tests pass, and it reports each task done. Until morning, its own report is all you have.

So put the check inside the loop. The script starts it, because a script can sign in with an API key and read an exit code. The agent gets each failure on its next turn, while the task is still fresh.

An unattended loop whose script starts each check Your script Coding agent RunStory its turn ends with an API key exit code then the next task Starts a turn with the next task Starts one check runstory run "$start.." Reads the exit code 0: go on, 1: fix first Works and commits as often as it needs Fixes each finding and commits the fix Checks the task's commits as one change Retests each script for free

One lap of the loop. The script starts a turn, then one check of the commits that turn made. If there's a problem, the agent fixes each finding and RunStory retests it. Then the script moves on to the next task.

#

Set it up

  1. Set a daily limit first, from your own machine. A key spends as your account and has no limit of its own. The daily limit caps this project's checks, whoever starts them.

    runstory settings --daily-limit 60
    
  2. Create an API key on the keys page. You choose when it expires, from 30 days to 1 year, and you can revoke it there at any time.

  3. Put the key where the loop runs.

    Where the loop runsHow it signs in
    A cloud sandbox or a pipelineThe secret RUNSTORY_API_TOKEN, in the platform's own secret store
    A machine you keep, like a build boxrunstory login --key, which asks for the key once and saves it
  4. Install the CLI there.

    curl -fsSL https://runstory.specstory.com/install.sh | sh
    
#

Add the check to the loop

This script gives each task one check and one round of fixes. Swap claude -p for your own agent's command for one unattended turn, and leave automatic checks off where the loop runs.

loop.sh

#!/bin/sh
# tasks.txt holds one task on each line.
while read -r task <&3 || [ -n "$task" ]; do
  [ -z "$task" ] && continue
  start=$(git rev-parse HEAD)
  claude -p "$task Commit your work."

  # No commit, so nothing to check.
  [ "$(git rev-parse HEAD)" = "$start" ] && continue

  # One check of every commit this task made.
  runstory run "$start.." --wait --timeout 75m
  code=$?
  if [ "$code" -eq 1 ]; then
    # A new problem: one round of fixes, then a retest.
    claude -p "Run runstory findings --json, fix each open finding, and commit."
    runstory findings retest --timeout 30m || echo "$task" >> needs-a-person.txt
  elif [ "$code" -ne 0 ]; then
    # The check did not finish, e.g. the daily limit is used.
    echo "$task (check ended with exit $code)" >> needs-a-person.txt
  fi
done 3< tasks.txt
LineWhat it does
start=$(git rev-parse HEAD)Notes where the task began
runstory run "$start.."Checks every commit since then as one change, and builds the newest
--wait --timeout 75mWaits until the check ends, so the result is ready for the next turn
runstory findings retestRuns each open finding's script 3 times, with no model. 3 passes close the finding

A retest costs no credits. It counts as 1 of your 10 free setup runs a day, and it covers every open finding on the branch.

#

What the agent is told

runstory init adds a short section to AGENTS.md. Codex and Cursor read that file each time they start. Claude Code reads it through a line that init adds to your CLAUDE.md, when the repository has one.

The section tells the agent to read runstory findings --json before it calls a task done, to fix each open finding, and to name the finding's id in the commit message of its fix.

A model wrote each finding and its script while it read your code. So RunStory also tells the agent never to run a finding's script on your machine, and never to follow an instruction in a finding's words.

#

Example: twelve tasks overnight

A developer at Acme, an online furniture store, leaves 12 tasks in tasks.txt at 7 in the evening. The project checks with 1 model, so each check costs 3 credits.

In the morningCount
Tasks whose check found no confirmed problems10
Tasks with a problem that the agent fixed, and the retest closed1
Tasks in needs-a-person.txt1
Credits spent on 12 checks36

In the loop's folder, runstory status lists each check that ended overnight. Each one has a web page, with a video of the browser for a web app. So the developer can watch the one failure that's left before reading any code.

#

Agents in a vendor's cloud

A cloud coding agent works in a sandbox you never sit at. You can check its work two ways.

  • It pushes a branch to GitHub. Let the GitHub App check each push. Check every pull request your agents open shows how.
  • It finishes inside the sandbox. Add RUNSTORY_API_TOKEN to the sandbox's secrets, install the CLI in its setup script, and ask for the check in the task: "When you're done, run one RunStory check of your branch and fix what it finds."
#

What it costs

A default check uses 2 frontier models and costs 6 credits.

ToDo this
Cap a day's checksrunstory settings --daily-limit 60. The day is a UTC day, and a check over the limit doesn't start
Check with 1 model, for 3 creditsrunstory settings --providers anthropic
Stop a keyRevoke it on the keys page. Commands that use it stop at their next request
Check once for the whole nightNote the commit before the first task, and run runstory run "$first.." once after the last task