Monitoring & alerting

Monitoring that has never fired is a configuration, not a safety net. This page states what is watched and then states the thing that is usually left out: whether it has been proven to fire.

What is watched

Five alert rules run against the platform database and API, wired to an action group that notifies CIQRA. They cover:

• the database not responding
• connection failures
• storage passing a high-usage threshold
• server errors from the API
• the API dropping to zero running instances

The drill

An alert was deliberately provoked and observed firing.

Step	Result
Failure condition generated	Connection failures registered on the database metric
Alert state	Fired, severity 1
Time from metric to alert	Under three minutes

How the drill was run, and why it matters

The obvious way to test a database alert is to break connectivity. We did not do that: the same platform serves live merchant sites, and an outage staged for a rehearsal is still an outage for whoever is shopping. The alert was raised instead by generating failed authentication attempts — the same metric, with no interruption to service.

A rehearsal that harms the thing it protects has misunderstood what it is rehearsing.

One alert was built and then removed

While setting this up, an alert was configured against a replication-delay metric — and then deleted, because that metric emits no data on this deployment. An alert that can never fire is worse than an absent one: it occupies a line on a dashboard and reports “no problems” forever.

We mention it because the useful part of a monitoring page is not the count of alerts.

What this page does not say

• It does not claim that the notification reached a human inbox. What was measured is that the alert rule fired; delivery to the recipient was not verified in the same drill, and we are not going to imply that it was.
• It does not publish response-time commitments. Nobody has measured them.