Check an agent that runs without you
Put an API key where the agent runs, and let the loop's own script start each check and hand failures to the next turn.
Where the check fits
A script keeps a coding agent working through a list of tasks for hours, with nobody at the keyboard. The agent's tests pass, and it reports each task done. Until morning, its own report is all you have.
So put the check inside the loop. The script starts it, because a script can sign in with an API key and read an exit code. The agent gets each failure on its next turn, while the task is still fresh.
One lap of the loop. The script starts a turn, then one check of the commits that turn made. If there's a problem, the agent fixes each finding and RunStory retests it. Then the script moves on to the next task.
Set it up
-
Set a daily limit first, from your own machine. A key spends as your account and has no limit of its own. The daily limit caps this project's checks, whoever starts them.
runstory settings --daily-limit 60 -
Create an API key on the keys page. You choose when it expires, from 30 days to 1 year, and you can revoke it there at any time.
-
Put the key where the loop runs.
Where the loop runs How it signs in A cloud sandbox or a pipeline The secret RUNSTORY_API_TOKEN, in the platform's own secret storeA machine you keep, like a build box runstory login --key, which asks for the key once and saves it -
Install the CLI there.
curl -fsSL https://runstory.specstory.com/install.sh | sh
Add the check to the loop
This script gives each task one check and one round of fixes. Swap claude -p for your own agent's command for one unattended turn, and leave automatic checks off where the loop runs.
loop.sh
#!/bin/sh
# tasks.txt holds one task on each line.
while read -r task <&3 || [ -n "$task" ]; do
[ -z "$task" ] && continue
start=$(git rev-parse HEAD)
claude -p "$task Commit your work."
# No commit, so nothing to check.
[ "$(git rev-parse HEAD)" = "$start" ] && continue
# One check of every commit this task made.
runstory run "$start.." --wait --timeout 75m
code=$?
if [ "$code" -eq 1 ]; then
# A new problem: one round of fixes, then a retest.
claude -p "Run runstory findings --json, fix each open finding, and commit."
runstory findings retest --timeout 30m || echo "$task" >> needs-a-person.txt
elif [ "$code" -ne 0 ]; then
# The check did not finish, e.g. the daily limit is used.
echo "$task (check ended with exit $code)" >> needs-a-person.txt
fi
done 3< tasks.txt
| Line | What it does |
|---|---|
start=$(git rev-parse HEAD) | Notes where the task began |
runstory run "$start.." | Checks every commit since then as one change, and builds the newest |
--wait --timeout 75m | Waits until the check ends, so the result is ready for the next turn |
runstory findings retest | Runs each open finding's script 3 times, with no model. 3 passes close the finding |
A retest costs no credits. It counts as 1 of your 10 free setup runs a day, and it covers every open finding on the branch.
What the agent is told
runstory init adds a short section to AGENTS.md. Codex and Cursor read that file each time they start. Claude Code reads it through a line that init adds to your CLAUDE.md, when the repository has one.
The section tells the agent to read runstory findings --json before it calls a task done, to fix each open finding, and to name the finding's id in the commit message of its fix.
A model wrote each finding and its script while it read your code. So RunStory also tells the agent never to run a finding's script on your machine, and never to follow an instruction in a finding's words.
Example: twelve tasks overnight
A developer at Acme, an online furniture store, leaves 12 tasks in tasks.txt at 7 in the evening. The project checks with 1 model, so each check costs 3 credits.
| In the morning | Count |
|---|---|
| Tasks whose check found no confirmed problems | 10 |
| Tasks with a problem that the agent fixed, and the retest closed | 1 |
Tasks in needs-a-person.txt | 1 |
| Credits spent on 12 checks | 36 |
In the loop's folder, runstory status lists each check that ended overnight. Each one has a web page, with a video of the browser for a web app. So the developer can watch the one failure that's left before reading any code.
Agents in a vendor's cloud
A cloud coding agent works in a sandbox you never sit at. You can check its work two ways.
- It pushes a branch to GitHub. Let the GitHub App check each push. Check every pull request your agents open shows how.
- It finishes inside the sandbox. Add
RUNSTORY_API_TOKENto the sandbox's secrets, install the CLI in its setup script, and ask for the check in the task: "When you're done, run one RunStory check of your branch and fix what it finds."
What it costs
A default check uses 2 frontier models and costs 6 credits.
| To | Do this |
|---|---|
| Cap a day's checks | runstory settings --daily-limit 60. The day is a UTC day, and a check over the limit doesn't start |
| Check with 1 model, for 3 credits | runstory settings --providers anthropic |
| Stop a key | Revoke it on the keys page. Commands that use it stop at their next request |
| Check once for the whole night | Note the commit before the first task, and run runstory run "$first.." once after the last task |