docs · harnessbench-cli 0.1

Quick start

Five steps from nothing to your first verdict. Needs Node 20, git, and Claude Code signed in.

1. Set it up in your repoWrites the config, detects your test command, installs the recording hooks, and checks Claude Code is signed in.

   npx harnessbench-cli init

2. Work as usualUse Claude Code in that repo for a day or two. Each task is saved automatically. See them with npx harnessbench-cli tasks.
3. Change your harness on a branchA rule in CLAUDE.md, a hook, a skill, an MCP server. Commit it.
4. Run the comparisonEvery saved task runs twice, old harness and new.

   npx harnessbench-cli run --base main

5. Read the verdictShip, hold or revert, with the line responsible. Open the dashboard for detail.

   npx harnessbench-cli ui

docs · start

How it works

Record, replay, grade, decide.

The key idea: same prompt, same starting code, only the harness differs. Whatever changes in the agent's behaviour is caused by your change.

docs · use

Recording tasks

Automatic once init has installed the hooks.

A task is one prompt you typed plus everything until the agent stopped. Follow-ups you send in the same session are saved too and become the answers the replay uses if the agent asks again. Slash commands, system notifications and prompts sent from inside a replay are ignored.

By hand

npx harnessbench-cli tasks add "add a retry helper to the fetch client"
npx harnessbench-cli tasks label <id> "flaky webhook fix"

docs · use

Running a comparison

--base and --head are git refs. Only the harness files differ between the two runs.

npx harnessbench-cli run --base main                # main vs your branch
npx harnessbench-cli run --base main --head feat/x
npx harnessbench-cli run --base main --task webhook  # ids containing "webhook"
npx harnessbench-cli run --base main --parallel 2
npx harnessbench-cli run --base main --grader none   # checks only, no grader
npx harnessbench-cli run --base main --agent fake    # dry run, no Claude calls

Worktrees live under .hb/wt/ during the run and are removed after. --keep-worktrees keeps them. A run costs about what the tasks cost the first time, doubled. Start with --task on one task.

docs · use

Reading the result

A verdict, the reason, and the line that caused it.

Each task gets a score out of 10 for both runs. Under half a point of movement counts as unchanged. Flags appear when tokens or time exceed your budgets.

docs · use

Dashboard

A local page that reads your reports.

npx harnessbench-cli ui               # http://127.0.0.1:4321
npx harnessbench-cli ui --port 5000 --no-open

For each run: the verdict, a one-line explanation of how the agent behaved differently, which axes moved, and the tasks. Expand a task for the two runs side by side, the questions it asked, and the transcript. The harness diff is at the bottom.

docs · use

In CI

ci is run plus a markdown summary and an exit code.

on:
  pull_request:
    paths: [".claude/**", "CLAUDE.md", "AGENTS.md", ".mcp.json"]
jobs:
  regression:
    runs-on: self-hosted   # claude installed and signed in
    steps:
      - uses: actions/checkout@v4
        with: { fetch-depth: 0 }
      - run: npx harnessbench-cli ci --base origin/${{ github.base_ref }}

Exits 1 on hold or revert. --fail-on revert lets holds through. Writes to $GITHUB_STEP_SUMMARY when present. The recorded tasks must be available on the runner.

docs · configure

Config file

.harnessbench/config.yml, written by init with comments.

agent:
  command: claude              # claude | fake | your own adapter
  permissionMode: acceptEdits  # see Permissions
  timeoutMinutes: 12
  parallel: 4
harnessPaths: [CLAUDE.md, AGENTS.md, .claude, .mcp.json]
signals:
  - name: tests
    command: npm test --silent
replay:
  policy: replay-then-proceed  # proceed | stop
grader:
  enabled: true
  model: claude-opus-5
rubric:
  axes: [...]
  hardFail: ["rm -rf /", "git push --force", "DROP TABLE"]
  budget: { tokensPct: 25, wallPct: 50 }
  verdict: { holdIfRegressionGt: 2.0, revertIfRegressionsGte: 3 }

docs · configure

Grading

Picked automatically, in this order.

1. API key present: the Anthropic API with grader.model and schema-enforced JSON.
2. Claude Code signed in: the same prompts through claude -p on your login.
3. Neither: checks only. Correctness from your tests, safety from commands run, scope from files touched.

Override with --grader none or --grader claude-sonnet-5. Every score comes with the grader's one-line reason.

docs · configure

Permissions

What the agent may do in the throwaway worktree without asking.

docs · configure

All commands

docs · more

Plans and license keys

Free for one person. A key unlocks the team features.

Activating a key

The key arrives by email after purchase. On each machine that should use it:

npx harnessbench-cli login HB-XXXX-XXXX-XXXX
npx harnessbench-cli whoami        # shows the active plan
npx harnessbench-cli logout        # frees the seat for another machine

Per-seat keys activate on one machine at a time. Per-repo and Business keys have no activation limit. In CI, set HARNESSBENCH_KEY in the environment instead of running login.

Keys are checked once a day and keep working for a week without network. Cancelling disables the key at the end of the period and the CLI drops back to Free. Nothing else changes.

docs · more

Troubleshooting

"Claude Code is not signed in"

Run claude once and sign in.

Tasks aren't recorded

Hooks load at session start; open a new Claude Code session after init. Check for three record --hook entries in .claude/settings.json or settings.local.json. HARNESSBENCH_DEBUG=1 prints hook decisions.

Both runs fail my checks

Worktrees start without dependencies. Node projects get node_modules linked in; for others, make your check command install first.

Where things live

.harnessbench/config.yml   settings and rubric
.harnessbench/tasks/       task.json + transcript.jsonl
.harnessbench/reports/     one JSON per run
.harnessbench/runs/        raw agent events        (gitignored)
.hb/wt/                    worktrees during a run  (gitignored)

Stop recording

Remove the record --hook entries from your Claude settings. Delete .harnessbench/ for everything else.