Failing pipelines

Nightly test suites fail, and with many repositories the e-mails about them pile up faster than anyone reads them. kanman reads the pipelines of your teams’ default branches, works out why they failed and does something about it: one story per failure across all repositories it affects, a sentence about outages that are not your code, and merges held only where the branch is red.

What kanman watches

Every hour kanman reads the newest pipeline of each repository’s default branch (the branch the team works against, usually main or develop), scheduled nightly runs included.

• GitHub on kanman Cloud: kanman sees your checks as they finish.
• GitLab on kanman Cloud: kanman reads the pipelines once an hour with your team’s GitLab connection, in the same way your runner would, and the coding agent can read a failing job’s full log through it.
• A workspace that runs everything on its runners (for example a GitLab behind a VPN): your runner reads the pipelines with its own access and reports only their state and short, redacted log excerpts. A team with hundreds of repositories in one GitLab group costs a handful of requests per hour, not one per repository.

Credentials, tokens and passwords that appear in a log are removed before a log excerpt is stored: on your runner, or in kanman Cloud before it is kept.

Why it failed

For each failed pipeline kanman reads the end of the failed jobs’ logs and sorts the failure into one of these:

To tell a flaky test from a broken one, kanman runs a failed job once more when it is allowed to bring in work itself. It only retries pipelines from the last three days, and at most ten per hour, so your CI is never flooded.

One story per failure, not per repository

The same failure in many repositories is one piece of work: the same failing test, the same error or the same outage. kanman groups them and files one story that lists every affected repository with a link to its failing pipeline. The story contains what fails, the failing tests, a log excerpt, the proposed fix and acceptance criteria: the pipeline passes again and no test is skipped or weakened to get there.

• Stories go to Ready, so kanman starts on them like on any other Ready story (your merge rules still apply).
• A team where only people file stories gets a decision per failure instead.
• At most five new stories per check (the team’s maintenance cap); the rest follow with the next checks.
• A failure that already has an open story only adds its repository to that story.
• Once the fix is merged in the first repository, kanman rolls the same change out to the other affected repositories (see Maintenance mode).

The story’s card on the team board says Pipeline red in 12 repositories and links the newest failing pipeline.

Infrastructure kanman pursues

A pipeline that fails because of a CI variable, a vendor test account, a quota, a vendor or the CI runners is not a fix in the code, but it is still work. kanman does not just mention it and move on: it takes it on as its own work and stays on it until the pipelines are green.

• The vendor’s answer first. Before kanman asks anyone, it reads the failing job’s log and finds what the vendor itself answered, for example 403 TenantDisabled: “Tenant is disabled.” or 429: “You have reached the limit of 30 requests per hour allowed on a Free plan”, rather than a test’s summary line. That answer decides the cause and what kanman asks for: a new value in the CI variables only for a revoked, expired or missing credential; reactivating the account or tenant when the vendor disabled it; a plan or fewer requests for a plan’s limit. When the vendor changed its API or its answers, nobody needs to act: kanman fixes it in the code as its own story.
• One item per cause. kanman tracks each cause once, for example “Restore CI access: the CI variables VERINICE_E2E_USERNAME and VERINICE_E2E_PASSWORD”, with every repository that fails on it, also repositories that share the same vendor account or tenant. When the vendor’s answers show that one item has several causes, kanman splits it and says so in the chat. The items sit in the team’s Backlog, assigned to kanman, together in the outcome Restore CI access for . On the board the outcome’s card says who it waits on, for example Waiting on whoever manages the CI variables for VERINICE_E2E_USERNAME.
• The chat first. Each item asks the team one specific question, for example Who manages the CI variable VERINICE_E2E_USERNAME for the Integrations CI? It fails since Oct 5 in 1 repository. The question appears with the outcome’s other requests and offers:
  • I’ll ask : kanman asks that person itself, in kanman and in their Slack or Teams direct messages when they connected one. kanman suggests the person who fixed a similar problem before, or the owner of the repositories in your Backstage catalog.
  • Give me the message to send: a ready-to-send message with everything the expert needs: what fails since when, the vendor’s own answer per repository with a link to the job, the exact variable names, the repositories with their failing pipelines and where to set the value. Copy it into a direct message or an email.
  • Assign to or Assign it to me.
  • It’s someone else: tell me who: name the person; kanman asks them when they are in your workspace, or addresses the message to them.
  • Let kanman change the tests: offered when a change in the code helps too, for example live tests that send fewer requests to stay under a plan’s limit. It changes test files, so kanman only does it with your OK.
• Never a secret in the chat. kanman never asks for a token or password. When a credential is the cause, it asks the right person to set the value where the CI reads it, for example in GitLab under Settings > CI/CD > Variables.
• Follow-up. kanman checks every pipeline. When the cause is gone, it closes the item itself and says so, for example Good news on INT-170: the CI variable VERINICE_E2E_USERNAME works again, and 7 of 7 repositories are green. If nothing changes for two working days after the last step, kanman follows up once, politely, and keeps it among the stories that wait on people until it is green. Change the waiting time with Follow up on CI access after, in the fold of the When CI breaks card on Team > Maintenance and CI. The card also says in words what kanman does for your team when a pipeline breaks.

One message a day

In the team’s conversation kanman keeps one message per day up to date, for example:

14 of 534 repositories are red on their main branch: 9 test or build failures, 4 infrastructure problems and 1 flaky test. I filed 2 stories, one per failure across all affected repositories: INT-81 and INT-82. The service api.example.com answers with errors in 4 repositories since 02:00. That is not the code, so I filed no story for it. Affected repositories: acme/billing, acme/crm, acme/shop and 1 more. Who should act: nobody in the code. If it lasts, whoever owns the vendor account should contact the vendor.

Infrastructure problems kanman already pursues are named by their items instead (“I pursue 3 infrastructure problems as my own work until the pipelines are green: INT-170, INT-171 and INT-172.”); only a problem nobody pursues yet gets the sentence about who should act.

The weekly report shows how many repositories are red on main and how many turned red that week.

When someone else’s change turned the branch red

Sometimes the change that turned a default branch red is not kanman’s but a colleague’s. kanman does not leave that to whoever notices first, and it does not raise an alarm about the person either. Once a check confirmed which merge it was, kanman:

1. opens a fix for it, or a revert of that change when a fix would take longer. Like every change kanman makes, it is verified before it merges.

2. sends the author of the change one message, in kanman and in their Slack or Teams direct messages when they connected one, for example:

   acme/billing: the base branch has been red since your change !123. I opened !456 to fix it forward, and it is verified before it merges. Should I merge it, or do you take it from here?

The message offers three answers:

The message carries no blame words, and nobody else is copied: not the team’s owner, not the channel. It never interrupts either: it waits for the author together with their other questions and comes with their next summary, unless they open it sooner. kanman writes only to an author who is a member of the team in kanman.

Merges while a branch is red

A red default branch holds kanman’s merges in that repository only. The team’s other repositories merge as usual. The story that fixes the red pipeline may merge: it is how the branch turns green again.