Fresh evidence supports a healthy state.
ENGINEERING STORY 001
Everything was green.
That was the problem.
A monitoring system can tell you everything is healthy. But what happens when the monitoring system itself stops producing current evidence?
HEALTH EVIDENCE
CURRENT
STALE
03:00
Reported condition
HEALTHY
Evidence validity
VALID
EXPIRED
The system looked healthy.
Every indicator was green. No alarms were showing. Nothing suggested that anything required attention.
There was just one problem.
The evidence behind those green indicators was getting older.
THE CLAIM
A green light is a claim.
Monitoring systems make claims about the systems they watch. Healthy is a claim. Available is a claim. Secure is a claim.
Like any engineering claim, it ought to be supported by evidence.
If a system was checked at 10:00 and found healthy, that tells us something useful about its state at 10:00.
It does not necessarily tell us anything about its state at 14:00.
THE PROBLEM
When monitoring stops, yesterday's green can survive.
No new evidence is produced.
The last successful result still looks reassuring.
The system can no longer justify the health claim.
A lack of current evidence is not evidence of failure.
But neither is it evidence of health.
THE ENGINEERING DECISION
So we stopped trusting green forever.
The answer was not another alarm. It was to change what green meant.
A health indication would only remain valid while the evidence supporting it was sufficiently recent.
Different evidence could have different validity periods because different conditions change at different rates.
But every important health claim needed one thing: an expiry date.
FAILURE INJECTION
Then we broke the assumption deliberately.
The system was given healthy evidence. The evidence was then aged beyond its permitted freshness limit without changing the underlying reported condition.
Reported condition
HEALTHY
Evidence condition
STALE
Expected appliance state
AMBER
Observed appliance state
AMBER
NEGATIVE ASSURANCE
PASS
The system had learned to distinguish between “we know something is wrong” and “we can no longer prove that everything is right”.
THE PRINCIPLE SPREAD
Health evidence now has a shelf life.
SMART evidence is rejected when it exceeds its permitted age.
Valid but stale security evidence deliberately degrades appliance health.
Host package evidence expires rather than silently remaining current.
Mission Control's own persistence and notification capability contribute to the overall health claim.
THE LESSON
Monitoring is not really about coloured indicators.
It is about evidence.
A green dashboard is useful only if you can answer three questions:
What evidence made it green?
When was that evidence collected?
How long are we prepared to trust it?
If the system cannot answer those questions, the green light may still be reassuring.
Reassurance and assurance are not the same thing.
That is why green has to be earned.
ENGINEERING STORIES