The outcome gate and evidence

Coding agents are good at saying they are done. The outcome gate is how kanman checks that they are. The executor never declares a story finished: kanman runs the gates itself, on infrastructure the executor does not control, and moves the story only when the gates pass. Everything it checked ends up in the evidence pack on the pull request.

Proof that fits the story

A one-line CSS fix and a new export feature do not need the same ceremony. kanman scales the proof to the story’s size, so a small change ships at the speed of the coding agent alone, with the checks a careful reviewer would do anyway.

• Quick stories need no new acceptance spec. Their proof is your repository’s own build and tests, CI, kanman’s review of every acceptance criterion, and screenshots of the running app before and after the change, in light and dark mode. The reviewer compares before and after for each colour scheme: a visual criterion only passes when it holds in both. The story page shows Spec: not needed (quick story), and the evidence says Quick story instead of “not proven”.
• Routine and complex stories get an acceptance test, written by the coding agent together with the fix. The agent has the code and the running app at hand, so it uses the real names of buttons and fields instead of guessing them. The story page shows Spec: written with the fix. When the attempt ends, kanman checks the test: it must be honest (no mocks of your app), fail on the base branch where the change is missing, pass on the run’s branch, and cover every acceptance criterion with a test that actually performs it and checks the outcome. A test that only passes is not enough.
• If the coding agent writes no test, kanman writes one itself and the run continues with it. Rewrite the acceptance spec and Use my spec work as before.

Which stories get an acceptance spec is a team setting: Settings > Checks and delivery, Which stories get an acceptance spec, one choice with the options every story, routine and complex stories (the default) and complex stories only. Off is offered only on the Trial preset. See acceptance_spec_for.

Look before writing

Whoever writes a test first looks at the app. Before a story goes to Ready, kanman starts your app as your acceptance manifest describes, logs in the way your example spec does, opens the pages the story’s Demonstrate step names and records what is on them: which headings, buttons, fields and links there are and what they are called, in light and dark mode. The coding agent gets this as context, and so does kanman’s spec author when it writes a test. The screenshots are the “before” half of the evidence. If the app cannot be opened, the story still moves on; the evidence says why there are no screenshots.

Ready to PR ready

The run’s Cost & details tab shows how long a story took from Ready until its pull request was ready to merge, split into the time the coding agent worked, the time kanman waited for people (for example a plan approval) and kanman’s own checks. For quick stories the target is that kanman’s checks add at most 5 minutes to what the coding agent alone would take; the evidence says whether the run met it. The run timeline ends with PR ready after N min.

The gates

Each gated move of a story has its own checks:

New commits on the pull request during Review are checked again; they never move the story on their own.

kanman performs every gated move itself. When someone moves a story into Ready, on the team page or in the tracker, kanman places it in Refining first, looks at the running app and runs the Ready gate; when kanman writes the spec itself, the card shows Checking the acceptance spec until the spec fails the way it should. Moves back to Ready from later columns are direct, because those stories already passed.

On your git host, kanman reports the result as the commit status kanman/outcome-gate. Make it a required status check in your branch protection so a pull request cannot be merged before the gate passed.

Red before Ready

A spec that already passes before any code is written does not test the new behaviour. The coding agent’s own test is checked against the base branch after the attempt for the same reason, and goes back to the agent with Spec passes before implementation if it does not fail there. When kanman authors the spec from the story’s Demonstrate block, the story stays in Refining unless the spec runs and fails. The story page then shows Spec: red (expected), kanman commits the spec to the branch kanman/spec/<KEY>, and the story moves to Ready.

Red only counts when the spec’s checks fail, because the new behaviour is missing. A spec that fails on a mistake of its own, such as a misspelled function, an undefined variable, a selector that cannot be read, a locator that matches several elements or a setup step that crashes, would never pass, so it does not count as red. The story stays in Refining with The spec fails on an error of its own, not on its assertions, naming the test and the line, and kanman writes the spec again. A spec you provided yourself goes back to you with the same reason. When the app under test cannot be reached at all, the test environment failed, and kanman retries quietly.

If kanman cannot write a working spec, it tries once more with the gate output. If that fails too, a Check failed decision asks you to improve the Demonstrate step and retry, or to keep the story in Refining.

How kanman writes the spec

kanman writes the spec the way your team writes acceptance tests. Before it starts, it reads the example spec in .kanman/acceptance/EXAMPLE/ (or, without one, the newest spec of another story) and your test helpers, such as a login helper in a _support/ folder or the files you list under context in the acceptance manifest. The spec then imports those helpers instead of reinventing them, and reads test logins from the variables your manifest grants.

A spec only observes. It waits until the app is ready after every page change, including login and selection screens such as a tenant picker, and finds elements by what people see on the screen. It never changes the checkout it runs in: no git commands, no installing or removing packages, no deleting files. For stories about the command line or dependencies, it reads files and runs commands that only look.

Logging in is where specs break most often, so kanman copies the login and setup steps of your example spec unchanged into every new spec and writes the story’s own steps after them. It waits for every screen your app shows on the way, such as a tenant picker, until that screen is there, instead of clicking it only if it happens to be visible already. It never waits for the network to go quiet: apps that keep loading updates never do, and such a spec would only time out. It waits for something visible that shows the page is ready instead.

Your team knows its app best. Under Settings > Checks and delivery, Notes for acceptance specs, team admins write down what every spec of the team must respect, for example “never reload the page after picking the tenant” or “the app is ready when the search button shows”. kanman follows these notes in every spec it writes or rewrites for the team. See spec_notes.

Under Settings > Agent and models, in the spec writer fold of Which model works on what, team admins also choose the model that writes the specs: kanman’s assistant model (the default), or the model of the team’s coding agent. The coding agent’s model usually knows test frameworks and how to find elements on a page better, so its specs fail less often for the wrong reason; each spec then costs a call to that model. See spec_author_model.

Honest specs

A spec must test the real thing. A static check rejects specs that mock their own target, for example with route interception, MSW, nock or module mocks, before the spec ever runs. Mocks of third-party services are fine. The story stays in Refining with a reason such as Spec mocks its own target: route interception of /api/invoices.

The same check rejects specs that would change the checkout (for example The spec changes the checkout instead of only observing it: git clean -xfd), specs that wait for the network to go quiet (The spec waits for network idle), and specs that import a helper from a path where no file exists. kanman writes such a spec again right away, with the reason, before it ever runs.

Rewrite the acceptance spec

Sometimes the spec, not the code, is wrong: it checks a page before the app is ready, or it looks for a button that the story never asked for. You do not have to fix it yourself. Tell kanman what is wrong, and it rewrites the spec.

• On the story page or the run page, choose Rewrite the acceptance spec below the spec state, describe the problem in a sentence or two and choose Rewrite spec.
• When kanman asks you during a run whether the spec should be fixed, choose that answer and add your note.

A note can be up to 50,000 characters long. Close to the limit, the form shows how many characters you have used; a longer note is not sent, and the form tells you so instead of shortening it.

kanman writes the spec again with your note, checks once more that it fails before the change, and replaces the approved spec. If a run was waiting, the new spec goes to its branch and the run continues. While a run is actively working, kanman asks you to wait until it pauses or waits for you. Every request is recorded in the audit log.

One rewrite runs at a time per story. If you send another note while kanman is still rewriting, for example to correct your first note, the form says A rewrite is in progress: your note will be added to it. kanman finishes the current rewrite, applies your new note on top of it and only then lets the waiting run continue: the same run, without a new plan to approve. A note sent before the rewrite has started is combined with the earlier one; where the two disagree, the newest note wins. Sending the same note twice changes nothing.

To keep the number of rounds small, kanman writes specs that check only what the acceptance criteria ask for: no pixel positions or alignment, no exact lists or counts the criteria do not list, and no navigation the story does not describe. Independent checks are written so that one run reports every mismatch at once, not only the first one. When a run’s checks fail, the run and any rewrite get the full list of failing checks. If the run has already built something, kanman also tries the rewritten spec against the run’s branch before the run continues. When a check there asks for something the criteria do not require, kanman corrects the spec once by itself, so you do not have to write another note for it.

kanman remembers your note. If the story goes through Refining again later, kanman keeps the approved spec as long as the story itself did not change. If the story changed, kanman starts from the approved spec and your earlier notes instead of starting over, so a fix you asked for is not lost. Whenever a new spec is approved, kanman also puts it on the story’s existing branch, so the next run starts from the approved version.

Use your own spec

Sometimes you already have the right spec, for example one a tester wrote by hand. Owners and admins can put it in place directly instead of describing a fix:

1. On the story page or the run page, choose Use my spec below the spec state. If the rewrite form is already open, switch to Use my spec at its top.
2. Paste the spec (the file name is optional), upload one or more files, or enter a commit on the story’s spec branch kanman/spec/<KEY>. Leave the commit empty to use the newest one. Each file can be up to 120 KB, up to 10 files at once.
3. Choose Use this spec.

kanman checks your spec exactly as it is and never changes it: it must be honest, must not change the checkout, its imports must exist, and it must fail before the change. If it passes, kanman approves it, puts it on the spec branch and on the branch of a waiting run, and the run continues. Before anything uses it, kanman reads the spec branch back and makes sure it holds exactly the file that passed the checks; a spec that differs is not used, and the form tells you so. If a check fails, the form shows why, for example The spec waits for network idle; fix the file and choose Fix it and send again. kanman does not rewrite your file.

If you paste a complete spec file into a rewrite note, kanman uses it as it is, in the same way. If your note quotes code and asks kanman to use it unchanged, kanman only accepts a spec that contains that code exactly as written.

Every spec you provide is recorded in the audit log, and the evidence pack names who provided the spec.

The diff guard

The acceptance spec lives on a protected path, .kanman/acceptance/<KEY>/. Executors may never change an approved spec: the path is denied for every preset and cannot be allowed. The only exception is a story whose coding agent writes its own test with the fix: it may write in that story’s folder, and nowhere else under .kanman/acceptance/. If an implementation commit touches it anyway, or the spec folder no longer matches the version approved at Ready, the run stops with Diff guard: the acceptance spec was changed by the implementation, no pull request is marked mergeable, and the decision inbox offers Revert spec changes and retry or Reject. kanman also stamps commit authorship itself, so it knows which commits came from the executor.

Not every difference is the run’s doing. If the branch carries another version of the spec although none of the run’s commits changed it, for example because the branch was started before a rewritten spec was approved, kanman puts the approved spec on the branch itself and runs the check again. Only if that does not help does the decision inbox ask you, with Diff guard: the acceptance spec on the branch is not the approved one and the option Put the approved spec back and retry. kanman never claims that the run changed its spec in that case.

Changed test expectations

While it works, the coding agent may add tests but never change or delete the tests that already exist: they check its work. Sometimes the fix of a story really is a test, for example when a live test checks an external service’s answer and the vendor changed that answer on purpose. Then the story may change exactly those test files:

• Failing-test stories that kanman files from your CI may change the test file of the failing test. kanman finds the file in your repository by itself (on your runners when you run everything on your runners).
• Any other story: when kanman finds that a test’s expectation is wrong for the story, it asks once in the decision inbox, for example May I change the test expectation in e2e/placed_test.go? Choose Yes, adjust the expectation in e2e/placed_test.go to allow that file for the story, No, keep the test as it is and fix the code, or Stop the run. The same option appears on Verification failed and repeated failures when an attempt changed a test.

A permission holds for the story until the story ends: kanman never asks about the same file twice, and when you answer while the agent is still working, the next attempt starts with your answer. On autonomous teams kanman’s own reviewer answers such questions (see Autonomy).

Acceptance specs are never part of this: they stay protected. Every change to an existing test is marked: the pull request says Test expectation changed and names the files, the evidence shows the test changes with their diff first, and a person always approves the merge, whatever your merge policy or autonomy settings say. Who allowed which test files is recorded in the audit log.

Waiting for CI

When a run reaches Review, kanman reads the CI state of its branch. Code hosts that send kanman their CI events trigger the next check by themselves. kanman also checks on its own: every few minutes at first, less often the longer a run waits. So CI on a code host that cannot reach kanman, such as a self-managed GitLab behind a firewall, still moves the run on. With Run everything on your runners, your runner reads CI.

While a run waits, its run page shows Waiting for CI and when CI was last checked, for example “last checked 3 minutes ago”, together with what kanman found: CI is still running, the branch has no CI checks yet, or the last check could not reach your code host or runner. Team admins can choose Check CI now to have kanman look right away.

If a branch has no CI checks at all 10 minutes after the last push, kanman treats CI as not configured. The CI check passes with the note No CI configured in the evidence pack, so a repository without CI never blocks a story. CI that fails still fails the check.

What fails is named. When CI fails, kanman reads the failing job’s log and names the failing tests (Go, pytest, Jest and Vitest output), links the job and quotes the lines that show the failure. A decision about a run that keeps failing starts with what fails, so your hint can be specific.

Failures that were already there do not count. A test that also fails on your default branch is not caused by the story. When your local verify command fails, kanman runs it once more on the base branch; for CI it compares with the last CI run of your default branch. Failures the base branch shares do not count against the story, the coding agent is told not to fix or ask about them, and the evidence names them as not caused by this story. Only failures the story’s branch adds still fail the check.

kanman also remembers what it has already seen fail on the default branch, from earlier checks and recent CI runs, so a failure stays excused when the latest run on the base branch could not name it. It compares like with like: a CI failure is excused only by what fails in your own CI, and only recent results count. A test known to be flaky that failed only once is reported as flaky, not counted; when it fails again, it counts.

Checks that stop making progress

After an attempt ends, kanman starts its checks right away. Sometimes a check cannot start, for example because your runner restarts at that moment. kanman notices when the checks of a finished attempt show no progress and starts them again on its own: after 5 quiet minutes, then after 10, 20 and 40 minutes, then every hour, up to six times. A check that is still waiting for a runner or still running is never started twice.

If the checks still show no progress after 10 minutes, the run page says The checks show no progress, when kanman last saw activity and how often it has already restarted them. Team admins can choose Re-run checks to start them again right away, or Stop to end the run. Each restart appears in the audit log as Checks restarted, with the person who asked or kanman as the actor. If kanman restarts checks on its own after a longer quiet time, the team activity says so, for example “I restarted the checks of run-x223, they had shown no progress for 20 minutes.”

Checking the current code again

A failed verdict is about the code it judged. Sometimes that is no longer the code on the branch, or the failing check was the reviewer, which is a model’s judgement and not an executed test. Another attempt does not help then: the code may already be right, and the executor finds nothing to change.

• On a decision: when a new attempt ended without a change, the No change made decision offers Re-run the checks on the current code. kanman checks the existing commits again, from the acceptance spec to the review and the clean checkout, with the team’s current policy. No new attempt starts. kanman recommends this option when the check that failed last was the reviewer or judged an older commit.
• On its own: when the check that failed last was the reviewer and its verdict may come from older code, kanman checks the current code again right away instead of asking.
• On the run page: when a finished run’s latest verdict failed, team admins see The last verdict failed with Re-run checks. The run’s open questions close, because the new verdict answers them. While an open decision offers the same option, the run page leaves it to the decision.

Each of these appears in the audit log as Checks restarted, with the person who asked or kanman as the actor.

Merging

When every Review gate passed, the team’s merge policy decides what happens with the pull request:

• A person merges (the default): kanman asks in the decision inbox and on the run page with Ready to merge. Owners and admins of the workspace choose Merge, and kanman merges the pull request with the repository’s merge method and moves the story to Done. Not now leaves it open. Other members see the question, but only owners and admins can merge from kanman.
• Merge small changes automatically: within the line limit kanman merges by itself. Above it, or when the code host refuses the merge, the decision Merge approval asks you.
• A change that modifies existing tests always asks a person, with either policy and under any autonomy setting (see Changed test expectations).

kanman merges the way your repository is set up. On GitLab the project’s merge method and squash setting apply, and the source branch is deleted when the merge request says so. On GitHub kanman squashes when the repository allows it, otherwise it creates a merge commit, otherwise it rebases. With Run everything on your runners, your runner merges with its own token.

If the code host refuses the merge, the decision stays open and kanman says why, with what to do next: the branch has conflicts, the pipeline must pass first or is still running, approvals or resolved threads are missing, the merge request is a draft, the token may not merge into the target branch, or your runner did not answer. Fix it and choose Merge again. Every merge and every refused merge is recorded in the audit log, with who chose Merge.

You can also merge on the code host as before. kanman notices the merge and moves the story to Done, from the code host’s webhook or, for a code host that cannot reach kanman such as a self-managed GitLab behind a VPN, from its own regular check of the pull request (the same schedule as Waiting for CI). An open Ready to merge question then closes by itself. If someone closes the pull request without merging, kanman asks whether to Stop the run (the story goes back to Backlog) or Keep it in Review after you reopened it.

Clean-room check

The final check never runs in the executor’s workspace. kanman checks out the branch fresh, restores the approved spec, provisions a new environment as your acceptance manifest describes, and runs the spec there. Traces, screenshots, videos or apply logs are stored as proof. A green spec without stored artifacts does not count.

Gate reference

Each gate result has a verdict (pass, fail, error or skipped), a reason and links to its artifacts. Results are logged as gate_passed or gate_failed in the audit log.

The second signal: the reviewer run

Next to the spec, a reviewer run checks each acceptance criterion against the diff, with file and line references, and flags scope creep. A criterion without a code reference does not pass. The reviewer uses a model from the other vendor (a Codex model when Claude Code wrote the code, and the other way round), so the code is never reviewed by the model that wrote it. Its result per criterion is written back to the story.

The reviewer also sees what kanman has already run for the attempt: the acceptance spec that just passed on an environment installed and built from the branch (with the names of the passing tests), your local verify command and CI. Some criteria can only be shown by running the software, such as “the build succeeds” or “the app starts without errors”. When a passing acceptance run covers such a criterion, the criterion passes, and the story names the run and the test as its evidence. A criterion stays unproven only when neither the diff nor a check that ran covers it. A criterion the reviewer finds failed stays failed.

kanman’s own acceptance spec on the branch never counts as scope creep. Big files such as lockfiles reach the reviewer as a short summary: for a lockfile, the packages the change added, removed or updated; for other large files, the changed line counts and sections.

When a gate fails

kanman decides on its own whether another attempt is worth it, and it never asks you “retry or stop?”:

1. Something new failed: kanman tries again. The next attempt gets the gate output, the reviewer’s findings and the failing CI log. A clear failing check, for example “dark mode: a single 1px divider, expected 1, got 2”, hands the next attempt the full list of failing checks.
2. A test that is broken on its own (a typo in the test, a locator that matches several elements, a setup step that crashes) is not the change’s fault. kanman repairs it once before anyone is asked: the coding agent fixes its own test, or kanman’s spec author fixes a spec it wrote. A spec you provided is never changed for you.
3. The same failure twice: kanman finds out why before it tries again. It runs the same checks again on the same code. If they pass now, the story goes on. If they fail the same way, kanman looks for the cause and acts on it:
   • The test environment failed, not the change: the story waits for it once, then kanman works on the environment itself and tells you only if it needs you, once for every story it holds up (see Stopped work never just sits).
   • The main branch fails the same way: the story waits until the main branch is fixed, instead of trying again against it.
   • The acceptance spec contradicts the story: you get one question with what failed and kanman’s recommendation: Change the spec to match the story (recommended), Narrow it: leave out the contradicting part, or Drop it for now.
   • The change does not meet the criterion yet: kanman makes one attempt with a different approach, not a repeat of the last one, and hands it what it found.
4. Still the same failure, or the story’s attempts are used up: one question, in your brief. The run waits and you get one item with what was tried, how many more attempts the story would likely need, what that would cost and take, and the estimated chance of success, with kanman’s recommendation: Narrow it: leave out what keeps failing (recommended), Keep going: one more attempt, or Drop it for now: back to the backlog. It reaches you with your next brief, not as an interruption.
5. A failed diff guard is never reworked automatically; it always asks. The one exception is a branch that only lacks the approved spec: kanman puts it back and checks again once, without asking.

Problems of the test environment itself (clone, provisioning, ready check, setup) and short outages are retried quietly, and only within a small share of your team’s work, so a broken service never turns into a wave of retries. An attempt that failed before the agent could start, because of such a problem, does not count against the story’s attempts; once the problem is fixed, the story simply starts again.

While kanman confirms that this works better for your team, a failed story on some days still comes to you the earlier way: one automatic rework, then Verification failed, and Repeated gate failure once a gate’s failure budget (gate_failure_budget, 3 by default) is used up.

A failed gate never turns into an endless loop.

The evidence pack

When the Review gates pass, kanman writes the evidence pack. It is stored on the run, shown on the run page’s Evidence tab, and posted once as a pull request comment and once as a tracker comment. It contains:

• the plan and the approaches considered
• the acceptance criteria, each passed, failed or unknown, with file and line references
• the gate results with their reasons and their proof files: screenshots with their file name and the check that took them, and links to logs, reports, traces and videos by file name
• test counts and the CI link
• cost against budget, duration and number of attempts
• whether outcome proof exists, or that a quick story was proven without an acceptance spec
• screenshots of the running app before and after the change, in light and dark mode
• the time from Ready to PR ready, see Ready to PR ready
• who provided the acceptance spec, when a person wrote it instead of kanman

On the Evidence tab, a coverage table shows for every criterion what proves it (code, acceptance test, clean room and review), the screenshots appear as before and after pairs in light and dark mode, and the tab also lists the policy decisions taken during the run; select a screenshot to open it in full size. The plan, the approaches and the time from Ready to PR ready are on the Cost & details tab. Links on the run page always work, also weeks later.

On the pull request, the comment shows up to eight screenshots inline, each with its file name and check, and ends with a link to the evidence on kanman’s run page. The images in the comment stay visible for 30 days; after that, the link to the run page still opens every proof file.

A shortened example as it appears on a pull request. Every line starts with a plain status word (PASS, FAIL, ERROR, SKIPPED or UNKNOWN) instead of an icon, so the comment reads the same on GitHub, GitLab and in Jira, in e-mail notifications and with a screen reader:

Evidence: SBX-12 (run-7f3k)

Outcome: PROVEN. The acceptance spec passed in a clean environment.

Plan
Add GET /invoices/export with a date-range filter and a CSV serializer.

Acceptance criteria
- PASS ac-1: Given a customer with 3 invoices in March ... (src/invoices/export.ts:42)
- PASS ac-2: Given ... (src/invoices/export.ts:61, src/invoices/csv.ts:12)

Checks
- PASS spec_green_ephemeral: Spec passed in a fresh environment (kanman-verify.log, invoices-export.png)
- PASS diff_guard: The implementation did not touch the acceptance spec
- PASS policy_diff: The change stays within the team policy
- PASS ci: CI passed
- PASS reviewer: All 2 criteria passed review
- PASS clean_room: Spec passed on a clean checkout (kanman-verify.log, trace.zip, invoices-export.png)

- Tests: 12 passed, 0 failed
- CI: success
- Cost: €0.84 / €2.00
- Duration: 11 min
- Attempts: 1

Screenshots
invoices-export.png · Spec green on a clean checkout
[image]

Open the evidence in kanman

The run page shows the evidence pack in your language. The comments on the pull request and in the tracker are written in the team’s language: the language the team’s owner chose in kanman (English, German, Spanish or French), otherwise English.

Repositories without a manifest

Quick stories do not need a manifest: without one they are proven by CI and the review, just without screenshots. For other stories, without an acceptance manifest there is no spec to run. Only the Trial preset allows this, so you can try kanman on a repository before adding a manifest. kanman then falls back to CI, the test plan and the reviewer run, records the spec gates as skipped, and the evidence pack says plainly: No acceptance spec was run, so the outcome is not proven. Review the change carefully. With any other preset, a story in a repository without a manifest stays in Refining. Add a manifest to get the full gate: see Add an acceptance manifest to your repo.

Related

• .kanman/acceptance.json and .kanman/verify.json
• Runs
• Stories and acceptance criteria