Monitoring signals: Prometheus, Grafana and InfluxDB

kanman can read your monitoring to notice when a service stops behaving normally, find out what probably caused it, and do the right thing within its limits. It works with:

• Prometheus and compatible APIs (Thanos, Grafana Mimir, VictoriaMetrics)
• Grafana: alert rules, dashboards and metrics through one of its data sources
• InfluxDB: InfluxQL, Flux or SQL with a read-only token
• Checkmk, as described in Backstage and Checkmk

kanman has no dashboard of its own. Your monitoring stays where it is; kanman links to it in its findings and answers when you ask “How is payment-api doing?”.

kanman never acts in production

kanman reads your monitoring and acts only outside production:

• it writes a finding with the evidence, its assessment of cause and impact, and how sure it is
• it files a fix story, or adds the finding to the one that is already open
• it proposes a revert merge request, but only of a change kanman made itself
• it proposes a story for an alert rule kept as code in your repository
• it drafts the change request and the incident note for your own change and incident process
• it tells the person on call, once, what it found and what a person would do next

Restarting, rolling back, scaling, acknowledging alerts, silences and downtimes are always a person’s change. kanman cannot do them: it only uses the read parts of each system, refuses everything else before a request leaves, and says so in the incident. Stories and merge requests run through your team’s normal pipeline, approvals and merge policy.

1. Prepare read access

• Prometheus: a token with read access, or user and password. For Mimir, include /prometheus in the address.
• Grafana: a token of a service account with the Viewer role, and the data source kanman should query (its uid, for example your Prometheus).
• InfluxDB: a read-only token, and the database or bucket.

2. Add the source

1. Open the team, click Team, choose Services and on-call and find the Catalog and monitoring card below the services.
2. Click Add source and choose Prometheus, Grafana or InfluxDB.
3. Enter a name, the address and the token. kanman keeps the token encrypted as a workspace secret (Workspace settings > Secrets).
4. Under How I read it, choose Directly, read-only or Through your MCP server (see below).
5. Service label: the label or tag that names the service in your metrics, for example service or app. kanman watches the services of the team (Team > Services and on-call) with this label.
6. Optional: Alert rules as code, the repository and folder where your alert rules live.
7. Click Save and read. The card shows how many services, alerts and deviations kanman found, and that the source is read-only.

If the source is only reachable inside your network, turn on Read through our runner: your self-hosted runner reads it and sends kanman only the results. Workspaces that run everything on their runners always read this way.

3. Through your own MCP server

If your company only allows production signals to reach an AI through MCP servers it runs itself, choose Through your MCP server and enter its address. kanman then uses only the server’s read tools (metrics, alerts, dashboards, checks) with the token you saved. Tools that change something (silences, acknowledgements, edits) are never called. Checkmk can be read the same way.

Admins can make this a rule for the whole workspace: under Production signals on the same page, turn on Only through approved MCP servers and list the approved servers, one per line. kanman then refuses direct connections to monitoring, and a source that does not follow the rule says why on its card.

4. Logs and traces stay off

Metrics, alerts and check states are always allowed. Logs and traces may contain personal or communication data, so kanman does not read them: queries against a log or trace data source (such as Loki or Elasticsearch) are refused, and a dashboard panel that shows logs is only linked, not read. Turn on Logs and traces for a source only when you have a reason, and write it down; kanman keeps it with the source. Every reading is cleaned of credentials before kanman stores or uses it.

5. How kanman finds a problem

kanman does not react to red. It learns what is normal for each service:

• the usual range of latency, errors, traffic and saturation for every hour of the week, from the last three weeks, so a quiet Sunday night and a busy Monday morning are both normal
• permanently open problems: alerts and checks that have been open for a day or longer are part of the normal and never page anyone

A shift is a lasting move out of the usual range in the harmful direction: slower, more errors, more saturation or less traffic. For the error ratio, kanman also watches how fast the error budget burns. A short spike that recovers by itself is no shift. New alerts that fire go through your on-call rules too.

6. What happens then

For each shift kanman opens one investigation per service and cause (a second shift of the same kind joins it), and the team’s on-call rules decide about the incident: severity threshold, on-call hours, who is notified, and what kanman may do.

Within minutes the incident’s thread, in the team’s chat and in your incident channels, shows:

• the shift (“p95 latency of Payment API is 500 ms instead of about 125 ms since 13:20”)
• the evidence: the queries, the time range and links to your dashboards and panels
• the closest deploy or merge before the shift, and whether kanman made that change
• the owner from Backstage, the docs and runbooks from your team context
• kanman’s assessment of cause and impact, with its confidence
• what kanman did within its limits, and for the person on call: what a person would do next

With kanman may set to Watch only, kanman only writes the finding, the drafts and the message to the person on call. A fix story goes onto the team’s board like any other story; an alert rule always needs a person.

When the metrics are back in their usual range, the investigation closes and the incident recovers.