Put kanman on call: services and signal sources

This guide puts kanman on call for a service: you describe the service, connect where alerts come from and where errors can be read, and decide what kanman may do. How incidents work is explained in Incidents and on-call.

You need to be an owner or admin of the workspace to add services and sources. Every member can see them, test a connection and work on incidents.

1. Add a service

1. Open the team, click the Settings tab and choose Services and on-call (/<workspace>/teams/<team>/settings/services), or ask kanman “Show me the services” in the team’s chat.
2. Click New service.
3. Enter a name, for example “Checkout API”, and an optional description.
4. Tick the repositories the service is built from. Only repositories the team is allowed to work on are offered; add more under Settings > Repositories and tracker of the team.
5. List the environments, for example production with its URL.
6. Click Save. The service page opens (.../settings/services/<service>).

Below the services, the Catalog and monitoring card on the same page connects your Backstage catalog and your monitoring (Checkmk, Prometheus, Grafana, InfluxDB), so kanman knows who owns what and how your systems are doing. See Backstage and Checkmk and Monitoring signals.

2. Choose what kanman may do

On the service page, under On-call settings:

Each setting is saved as soon as you change it (Saved appears next to it, Undo takes it back). All settings are described in On-call settings.

Incident places in Slack or Microsoft Teams

kanman posts incidents to the places your team talks to it in. Connect Slack or Microsoft Teams and add the team’s channels under the team’s Settings > Chat and mentions, as described in Slack and Microsoft Teams. Then, under Incident places in the on-call settings, tick the channels that should get this service’s incidents, for example a dedicated incidents channel. With nothing ticked, every channel of the team gets them.

Each incident is one thread per place: kanman opens it with the title, severity and service, adds every finding and action as a reply, and posts the resolution at the end. Incident posts appear in the team’s chat history in the app like every other message kanman sends, there with every finding in your language.

3. Connect signal sources

On the service page, click Add source, choose the kind, give it a name and fill in the settings. After you save an alerting tool, kanman shows the webhook URL and the secret once. The URL has this form:

https://api.kanman.ai/functions/v1/signal-webhook/<source token>

Copy both into your tool right away. kanman stores the secret encrypted and never shows it again; Rotate secret creates a new one (the old one stops working at once). If a source feeds several services, tick them in the source’s settings.

Use Test connection on any source: an uptime check runs once, a log source reads the last 15 minutes, an alerting tool shows whether alerts have arrived. Testing never opens an incident.

Uptime check

Settings: URL, method, interval (60 seconds or more), timeout, expected status codes (for example 200 or 2xx), text the response must contain, request body, how many failures in a row open an incident (default 2), and the severity. Add request headers for protected endpoints, for example Authorization; their values are stored encrypted. kanman only checks public addresses. The first passing check after a failure resolves the incident.

Check from several regions: tick up to five regions under Check from these regions (Frankfurt, Ireland, London, N. Virginia, Oregon, Canada, São Paulo, Singapore, Tokyo, Sydney). kanman then runs every check from each of them and counts it as failed only when most of them fail, so a network problem in one region never pages anyone, while an outage your customers see everywhere does. Regions that must fail changes that number, for example 1 to hear about any region. The source shows each region’s last result, and an incident names the regions that failed (“status 503 in N. Virginia, Singapore”). Without regions, kanman checks from its own region in Europe.

Prometheus Alertmanager

Add a receiver and route your alerts to it:

receivers:
  - name: kanman
    webhook_configs:
      - url: https://api.kanman.ai/functions/v1/signal-webhook/<source token>
        send_resolved: true
        http_config:
          authorization:
            type: Bearer
            credentials: <secret>

kanman reads severity and service (or job, app) from the alert labels and summary and description from the annotations.

Grafana alerting

Create a contact point of type Webhook with the URL. Under Optional Webhook settings, set the authorization header scheme to Bearer and the credentials to the secret. Alternatively, enter the secret as HMAC signature secret. Use the contact point in a notification policy.

Datadog

1. In Integrations > Webhooks, add a webhook named kanman with the URL.
2. Set Custom Headers to {"X-Kanman-Token": "<secret>"}.
3. Use this payload:

{"id":"$ID","alert_id":"$ALERT_ID","aggregate":"$AGGREG_KEY","title":"$EVENT_TITLE","transition":"$ALERT_TRANSITION","priority":"$ALERT_PRIORITY","alert_type":"$ALERT_TYPE","body":"$EVENT_MSG","link":"$LINK","tags":"$TAGS","date":"$DATE"}

4. Mention @webhook-kanman in the monitors that should reach kanman. Recoveries resolve the incident.

New Relic

Create a webhook destination with the URL and the secret as Bearer token, then add it to a workflow with the default webhook payload. Closed issues resolve the incident.

PagerDuty

1. In Integrations > Generic Webhooks (v3), add a subscription for the service with the URL.
2. Select the events incident.triggered, incident.resolved, incident.reopened, incident.escalated and incident.priority_updated.
3. PagerDuty shows its own signing secret: edit the source in kanman and paste it into Signing secret from the tool.
4. Optional, to let kanman acknowledge and resolve incidents: add a REST API key and the email of the PagerDuty user kanman acts as, then turn on Acknowledge and resolve alerts in the source for the service.

Opsgenie

Add a Webhook integration with the URL and a custom header X-Kanman-Token with the secret. Created and closed alerts reach kanman. Optional: add an API key and choose the region to let kanman acknowledge and close alerts.

Sentry

1. In Settings > Developer Settings, create an internal integration with the URL as webhook URL, turn on Alert Rule Action and the issue events.
2. Paste the integration’s Client Secret into Signing secret from the tool of the source.
3. Add the action “Send a notification via kanman” to the alert rules that should reach kanman. Issue alerts and metric alerts are supported; resolving the issue in Sentry resolves the incident.

Amazon CloudWatch

1. Create an HTTPS subscription on the SNS topic your alarms publish to, with the URL plus ?key=<secret>: https://api.kanman.ai/functions/v1/signal-webhook/<source token>?key=<secret>
2. kanman confirms the subscription by itself and checks the AWS signature of every message. Optionally enter the topic ARN in the source to accept only that topic.
3. Set the severity in the alarm description, for example [critical] Checkout 5xx; untagged alarms count as high. ALARM opens, OK resolves.

Azure Monitor

In the action group, add a Webhook action with the URL plus ?key=<secret> and turn on the common alert schema. Fired alerts open, resolved alerts resolve. Sev0 is critical, Sev1 high, Sev2 medium, Sev3 low.

Atlassian Statuspage

Add a webhook subscriber with the URL plus ?key=<secret>. Component outages and incidents on your status page reach kanman; “operational” or a resolved status page incident resolves it.

Other tools

Send alerts in kanman’s generic format, signed with the secret. The format and the signature are described in On-call settings.

Logs, errors and traces

Log sources do not send anything; kanman reads them while it investigates an incident.

kanman only reads errors and warnings around the start of the incident and keeps the error lines it reports in the incident’s timeline. From trace sources it reads the failed traces and error spans, with their error message, stack and trace ID, and weighs them like error lines when it looks for the cause.

4. Try it

Send a test alert from your tool, or let an uptime check point at a page that answers an error. Within a minute the incident appears in your conversation with kanman, in the team’s incidents (“1 incident open” in the team status) and in your channel, and kanman starts posting what it finds.