Functional Weave
Code in TypeScript

monitor.alert-mute@1.0.0

README.md

2,564 bytes · view raw

# monitor.alert-mute

Whether to hold an alert back right now. It answers the question every
alerting system has to ask before it pages someone, in the two ways
Alertmanager does:

- **Maintenance** (a silence): the alert's component is inside a scheduled
  maintenance window and the alert's kind is one maintenance mutes. Planned
  work should not page for the downtime, slowness or error-budget burn it
  was planned to cause.
- **Inhibition**: an alert that makes this one redundant is already firing on
  the same component. When a target is down, its latency alert adds nothing
  but a second page, so `down` inhibits `latency`.

Pass the result on as the condition of `monitor.alert-state`:
`condition and not muted`. A muted alert behaves as if its condition were
false: it never fires, and one already firing resolves after its
`clearForSeconds`, as the down alert does when a component goes into
maintenance.

## Why it is shaped this way

- **Kinds are strings the caller chooses** (`down`, `latency`, `burn-rate`,
  `cert-critical`), matched exactly. The policy is data, usually read from the
  project's config, so a new kind of alert needs no new version.
- **Maintenance lists what it mutes**, rather than muting everything: a
  certificate that is about to expire is not planned work, and should still
  alert during a database upgrade.
- **Maintenance wins over inhibition** and `by` stays null, since the
  maintenance window is the reason that holds for the whole period.
- **Inhibitions are checked in policy order**, not in the order the firing
  kinds arrive, so the answer does not depend on how the caller collected them.
- **An alert never inhibits itself**: `source` and `target` must differ, so a
  kind listed in `firingKinds` for itself changes nothing.
- **The whole policy is checked on every call**, even when the first
  maintenance kind already decides it. A mistake in the policy fails loudly
  instead of silently never muting.
- Inhibition is per component (Alertmanager's `equal: [component]`): the
  caller passes only the kinds firing on the alert's own component.

## Errors

- `kind must not be empty`
- `maintenance kinds must not be empty`
- `inhibition source must not be empty`
- `inhibition target must not be empty`
- `an alert kind cannot inhibit itself: X`

## Sources

Prometheus Alertmanager: silences and inhibition
(https://prometheus.io/docs/alerting/latest/alertmanager/#inhibition) and
`inhibit_rules` in the configuration reference
(https://prometheus.io/docs/alerting/latest/configuration/#inhibit_rule).