Put kanman on call: services and signal sources

This guide puts kanman on call for a service: you describe the service, connect where alerts come from and where errors can be read, and decide what kanman may do. How incidents work is explained in Incidents and on-call.

You need to be an owner or admin of the workspace to add services, sources and connections; team heads map connected systems to their services. Every member can see them, test a connection and work on incidents.

1. Add a service

1. Open the team’s Mind tab, click the Memory pill next to the board filter and choose Services (/<workspace>/teams/<team>/mind/memory/services), or ask kanman “Show me the services” in the team’s chat.
2. Click New service.
3. Enter a name, for example “Checkout API”, and an optional description.
4. Tick the repositories the service is built from. Only repositories the team is allowed to work on are offered; add more under Team > Repositories of the team.
5. List the environments, for example production with its URL.
6. Click Save. The service opens on its own page (.../mind/memory/services/<service>).

Alert sources under Team > Connections connects your Backstage catalog and your monitoring (Checkmk, Prometheus, Grafana, InfluxDB), so kanman knows who owns what and how your systems are doing. See Backstage and Checkmk and Monitoring signals. kanman reviews ServiceNow tickets as described in ServiceNow.

The same line connects SonarQube. Choose SonarQube, enter the address you open it at and a user token with the Browse permission, and turn on Read through our runner when SonarQube is only reachable inside your network. kanman checks the connection every hour and shows whether it works. When kanman reviews a change, it reads the quality gate and the new issues of that change itself: a gate that fails on new code is a finding kanman fixes, not a question for you. To replace the token, edit the source and enter the new one; to remove it, delete the source and the secret under Settings › Secrets.

2. Choose what kanman may do

On the service page, under On-call settings:

Each setting is saved as soon as you change it (Saved appears next to it, Undo takes it back). All settings are described in On-call settings. What kanman may do in your alerting and paging tools (acknowledge, resolve, page) is chosen per connected system, see step 3.

Incident places in Slack or Microsoft Teams

kanman posts incidents to the places your team talks to it in. Connect Slack or Microsoft Teams and add the team’s channels under Team > Connections, Chats, as described in Slack and Microsoft Teams. Then, under Incident places in the on-call settings, tick the channels that should get this service’s incidents, for example a dedicated incidents channel. With nothing ticked, every channel of the team gets them.

Each incident is one thread per place: kanman opens it with the title, severity and service, adds every finding and action as a reply, and posts the resolution at the end. Incident posts appear in the team’s chat history in the app like every other message kanman sends, there with every finding in your language.

3. Connect your alerting, paging and log systems

Alerting tools, paging, error tracking, status pages and log systems are connected once for the workspace and then used by every team that needs them. Each team decides what reaches it.

Connect a system (owners and admins)

1. Open Settings › Connections, find the system (for example Datadog, PagerDuty or Sentry) and connect it: give the connection a name, enter its address where it has one, the credential and the settings the card asks for (for example the Datadog site, the Sentry organization or the PagerDuty region). kanman stores the credential as a workspace secret; you find it under Settings › Secrets.
2. Choose where kanman reaches it from: kanman’s cloud for systems on the internet, or through your runner for systems inside your network (for example a Checkmk, Elastic or Prometheus your company runs itself). Through the runner, kanman’s cloud never connects to the system; your self-hosted runner does, and it may use a plain http:// address inside your network.
3. For systems that send alerts, enter a Webhook signing secret: a long random value you also enter in the tool, or the signing secret the tool generates itself (PagerDuty, Sentry). The connection card then shows the address under Webhooks to, with a button to copy it.
4. Click Test connection. kanman checks, with read-only requests, what the credential may do and shows it per area (alerts, logs, on-call, …).

Choose what reaches the team

Open the team, click Team and choose Connections. The Connections card at the top lists every connected system of the workspace; open one. For each:

• What belongs to this team: add what the system sends that is the team’s, and point it at The whole team or one of your services. All alerts takes everything the system sends. Or map what the system calls a service, for example the Datadog service checkout, the Sentry project checkout-api, the Loki query {app="checkout"}, the log group /ecs/checkout or the Checkmk hosts pay-prod-. Alerts open incidents under the on-call settings of their service; for the whole team, kanman picks the service from the service name in the alert. When kanman investigates an incident of a service, it reads exactly the logs, traces and errors mapped to it.
• What kanman may do there: one row per action, each Off, Ask first or Automatic, within the limit an owner or admin set for the workspace.

Team heads change the mappings and modes; everyone else sees them. Sources your team connected earlier were moved over by kanman, with their URLs and secrets: they appear as Earlier webhook URL, and alerts sent to an old URL keep arriving.

What kanman may do in connected systems

With Ask first, kanman shows what it would do on the incident page with Approve and Decline; the incident’s timeline says when it waits for you and what happened. A service set to Watch only never gets any of these. Every action, also a declined or failed one, is in the team’s activity.

Set up the tools

Copy the address from the connection card (Webhooks to). It has this form:

https://api.kanman.ai/functions/v1/extension-webhook/<connection>

“The secret” below is the connection’s Webhook signing secret.

Prometheus Alertmanager: add a receiver and route your alerts to it.

receivers:
  - name: kanman
    webhook_configs:
      - url: https://api.kanman.ai/functions/v1/extension-webhook/<connection>
        send_resolved: true
        http_config:
          authorization:
            type: Bearer
            credentials: <secret>

kanman reads severity and service (or job, app) from the labels and summary and description from the annotations. To let kanman silence alerts, enter the Alertmanager’s address in the connection.

Grafana alerting: create a contact point of type Webhook with the URL and enter the secret as HMAC signature secret (or as Bearer credentials). Use the contact point in a notification policy.

Datadog: in Integrations › Webhooks, add a webhook named kanman with the URL, the header {"Authorization": "Bearer <secret>"} and this payload, then mention @webhook-kanman in the monitors that should reach kanman:

{"id":"$ID","alert_id":"$ALERT_ID","aggregate":"$AGGREG_KEY","title":"$EVENT_TITLE","transition":"$ALERT_TRANSITION","priority":"$ALERT_PRIORITY","alert_type":"$ALERT_TYPE","body":"$EVENT_MSG","link":"$LINK","tags":"$TAGS","date":"$DATE"}

For logs and APM traces, the connection’s credential is the API key and the application key as <api key>:<application key>.

New Relic: create a webhook destination with the URL and the secret as Bearer token, and add it to a workflow with the default payload. For logs, enter the account id in the connection.

PagerDuty: in Integrations › Generic Webhooks (v3), add a subscription with the URL and the events incident.triggered, incident.resolved, incident.reopened, incident.escalated and incident.priority_updated. PagerDuty shows its own signing secret: enter it as the connection’s Webhook signing secret. For acknowledging, resolving and paging, the connection needs a REST API key, the email of the PagerDuty user kanman acts as and, for paging, the service kanman pages through.

Opsgenie: add a Webhook integration with the URL and the header Authorization: Bearer <secret>. For acknowledging, closing and paging, the connection needs an API key, the region and, for paging, the responder team.

Sentry: create an internal integration with the URL as webhook URL, Alert Rule Action and the issue events on, and enter its Client Secret as the connection’s Webhook signing secret. Add “Send a notification via kanman” to the alert rules that should reach kanman. A new issue opens an incident, an issue that comes back opens it again, resolving the issue resolves it.

Amazon CloudWatch: create an HTTPS subscription on the SNS topic your alarms publish to, with the URL plus ?key=<secret>. kanman confirms the subscription itself and checks the AWS signature of every message; enter the topic ARN in the connection to accept only that topic. Tag the severity in the alarm description, for example [critical] Checkout 5xx.

Azure Monitor: in the action group, add a Webhook action with the URL plus ?key=<secret> and turn on the common alert schema.

Atlassian Statuspage: add a webhook subscriber with the URL plus ?key=<secret>. Component outages and incidents reach kanman.

Logs, errors and traces

Log, trace and error systems send nothing: kanman reads them while it investigates an incident, for the window around the first alert, through the mappings of the service. Connect them under Settings › Connections:

kanman keeps only the error lines it reports in the incident’s timeline. kanman can also read these systems when you ask, for example “what does Sentry say about issue 4504?” or “who is on call in PagerDuty?”.

A team’s own sources

On the service page, Add source adds what belongs to the team itself:

• Uptime check: kanman calls a URL on a schedule. Settings: URL, method, interval (60 seconds or more), timeout, expected status codes (for example 200 or 2xx), text the response must contain, request body, how many failures in a row open an incident (default 2), the severity and request headers for protected endpoints (their values are stored encrypted). kanman only checks public addresses. The first passing check after a failure resolves the incident. Check from several regions: tick up to five regions; a check counts as failed only when most of them fail, so a network problem in one region never pages anyone. Regions that must fail changes that number.
• Other tool: alerts in kanman’s generic format, signed with the source’s secret, described in On-call settings.
• Kubernetes logs: namespace, label selector and container; your self-hosted runner reads them inside your network.

Test connection on a source runs a check once, reads Kubernetes logs or shows whether alerts arrived; it never opens an incident.

4. Try it

Send a test alert from your tool, or let an uptime check point at a page that answers an error. Within a minute the incident appears in your conversation with kanman, in the team’s incidents (“1 incident open” in the team status) and in your channel, and kanman starts posting what it finds.