When a drill fails

The single most useful thing a drill does when it goes wrong is tell you which kind of wrong it is. A backup that did not restore is a finding about your recovery position. A Docker daemon that was down is a finding about the machine the drill ran on. They arrive through the same command and they are not the same event.

The exit code is the contract

Code	Meaning
0	backup proven restorable
1	recoverability failure (backup, restore, or check)
2	infrastructure error or cancelled
3	usage or setup error
5	evidence record could not be written

An alerting rule that cannot tell 1 from 2 will be muted within a month, because the second kind is common and boring and the first kind is neither. Set them up as separate alerts from the start, with different urgency and different owners.

Reading each one

0 — proven restorable. The backup restored and every check passed. This is a claim with a timestamp, not a green light on a dashboard: the record says what was restored, how long it took, and what was verified.

1 — a recoverability failure. The restore failed, or it succeeded and a check did not. Treat this as an incident about your backups, at the same severity you would treat losing them, because in the case that matters you have. Work from the checks in the record: a failing healthcheck points at the archive or the engine version, a failing row count points at the source — most often a job that has been backing up something other than what you assumed.

2 — an infrastructure error. The drill could not be completed: the container runtime was unreachable, the sandbox timed out, the run was cancelled. This says nothing about the backup. It is still worth alerting on, at a lower urgency, because a drill that errors every night is a drill that has stopped proving anything — the dangerous failure here is the silence, not the error.

3 — usage or setup. The configuration is wrong, a flag is missing, an adapter is not on the path. Almost always a change that has just been made.

5 — the record could not be written. The rarest and the most uncomfortable: the drill may have run correctly and the proof was lost. Treat it as an incident about the evidence log — a full disk, a permission change, a path that moved — because a drill whose record cannot be written is a drill that did not happen, as far as anyone reading the history later is concerned.

A failure is evidence

Every outcome is recorded and signed, including the failures. That is deliberate, and it is worth understanding before someone asks why the log is not all green.

A log containing only successes is not evidence of a working recovery position. It is evidence of either a working recovery position or a drill that was quietly switched off after it started failing, and from the outside the two are indistinguishable. The published example log ships as a pass, a failure and an error for exactly this reason.

So do not clean the log. A record of a failure, followed some days later by a record of a pass on the same target, is a stronger artefact than an unbroken run of passes: it shows a problem was found and fixed, with dates.

When the fix is in the backup, not in Probavi

Most 1 outcomes are resolved somewhere else — in the backup tool, the retention policy, the credentials, the source path. Probavi takes no backups and repairs none; it tells you which of them is not what you thought, and records what it saw at the time.

The record is the handover: it names the source kind, the backup’s checksum and size, the adapter and engine version, and each check with its result. That is usually enough to reproduce the failure by hand against the same archive without needing to reconstruct the conditions from memory.