monitor.incidents

export function findIncidents(checks: readonly Check[], incidentStatus: CheckStatus, confirmChecks: number, now: number): readonly Incident[]

/** One run of bad checks, from the first bad check to the first good one after it. */
export interface Incident {
  /** at of the first bad check */
  readonly start: number;
  /** at of the first good check after the run; null while ongoing */
  readonly end: number | null;
  /** (end, or now while ongoing) - start */
  readonly durationSeconds: number;
  /** down if any check in the run was down, else degraded */
  readonly worst: CheckStatus;
  /** how many bad checks the run holds */
  readonly checks: number;
}

import { findIncidents } from "#fune/monitor.incidents@^1";

import { type Check, type CheckStatus } from "./monitor_check_status_types.ts";
import { type Incident } from "./monitor_incidents_types.ts";

/**
 * The incidents in a check series. A run of consecutive bad checks is one
 * incident, from its first bad check to the first good check after it, but
 * only when it holds at least confirmChecks checks: a single failed probe is
 * noise, not an outage, and would otherwise page and skew MTTR.
 */
export function findIncidents(checks: readonly Check[], incidentStatus: CheckStatus, confirmChecks: number, now: number): readonly Incident[] {
  if (!Number.isSafeInteger(now)) throw new RangeError("now must be a whole number of seconds");
  if (incidentStatus !== "down" && incidentStatus !== "degraded") {
    throw new RangeError(`incidentStatus must be down or degraded, received ${String(incidentStatus)}`);
  }
  if (!Number.isSafeInteger(confirmChecks) || confirmChecks < 1) {
    throw new RangeError(`confirmChecks must be at least 1, received ${confirmChecks}`);
  }
  for (let i = 0; i < checks.length; i++) {
    const s = checks[i].status;
    if (s !== "up" && s !== "degraded" && s !== "down") throw new RangeError(`unknown check status: ${String(s)}`);
    if (i > 0 && checks[i].at <= checks[i - 1].at) {
      throw new RangeError(`checks must be in strictly ascending time order: ${checks[i].at} follows ${checks[i - 1].at}`);
    }
  }
  if (checks.length > 0 && now < checks[checks.length - 1].at) {
    throw new RangeError(`now ${now} is before the last check at ${checks[checks.length - 1].at}`);
  }
  const bad = (s: CheckStatus): boolean => s === "down" || (incidentStatus === "degraded" && s === "degraded");
  const out: Incident[] = [];
  let i = 0;
  while (i < checks.length) {
    if (!bad(checks[i].status)) { i++; continue; }
    const start = checks[i].at;
    let count = 0;
    let worst: CheckStatus = "degraded";
    while (i < checks.length && bad(checks[i].status)) {
      if (checks[i].status === "down") worst = "down";
      count++;
      i++;
    }
    const end = i < checks.length ? checks[i].at : null;
    if (count >= confirmChecks) {
      out.push({ start, end, durationSeconds: (end ?? now) - start, worst, checks: count });
    }
  }
  return out;
}

Install

fune build

With that line in your source, in a TypeScript project (language typescript in fune.project), fune build resolves it and its 1 dependency, pins them in fune.lock, downloads only the TypeScript package of each, and builds the code above into your project’s .fune/build, one readable file per capability with a header linking back here. Or pin a range in fune.project and build in one step:

fune add monitor.incidents

The manifest, vectors and README with only the TypeScript implementation. Install it without the registry with fune add ./monitor.incidents-1.0.0-typescript.fune, or fetch it from a terminal with fune pull monitor.incidents@1.0.0:typescript.

The whole function, every language, is one file too: monitor.incidents-1.0.0.fune, 19,417 bytes, sha256 4664c6be7bf3e0139623ab73f3d74becc8b401bdee0de17e21fa7824a05124a8. It installs into a project of any language.

// fune: before monitor.incidents

// fune: after monitor.incidents

// fune: replace monitor.check-status in monitor.incidents

// fune: step monitor.incidents after <n|label>

More from the author

- **Bad** depends on `incidentStatus`: with `down`, only `down` checks are bad (a slow site is not an outage); with `degraded`, `degraded` and `down` both are, and one incident can mix them. `up` is refused: an "incident of being up" is a configuration mistake. - A run counts only when it has at least `confirmChecks` checks. This is flap suppression, the same idea as "alert after N consecutive failures" in most uptime monitors: one lost probe does not make an outage, and would drag MTTR down if it did. Shorter runs are dropped entirely, including an unconfirmed run still in progress at the last check. - `start` is the `at` of the first bad check; `end` is the `at` of the first check after the run that is not bad (with `incidentStatus: down`, a `degraded` check ends a down incident). An incident still bad at the last check is ongoing: `end` is null and `durationSeconds` runs to `now`. - The start is the first check that *saw* the failure, not a guess at when it really began between two checks; and the end is the first check that saw recovery. So durations are accurate to the check interval, and err long. - `worst` is `down` if any check in the run was down, else `degraded`. - `checks` counts the bad checks in the run.

Incidents come out in time order and never overlap, which is what `monitor.mttr` expects.

## Errors

- `checks must be in strictly ascending time order` - `unknown check status: X` - `incidentStatus must be down or degraded, received up` - `confirmChecks must be at least 1` - `now 200 is before the last check at 240`: now ends ongoing incidents, so it cannot be earlier than the data. - `now must be a whole number of seconds`

Files