Automating repetitive audit procedures without automating judgment

The boundary worth drawing is not between simple and complex work. It is between work that has a single correct answer, which can be computed and checked by anyone, and work that requires a view, where two competent auditors may reasonably differ. Matching, recomputation, reconciliation and pattern detection are on the first side and should be run over the whole population. Deciding what an exception means, whether evidence is sufficient, and what to conclude are on the second, and automating them removes the part of the audit the signature is attached to.

A test for placing any task

Three questions settle most cases. A task that passes all three belongs on the mechanical side.

• If two competent auditors performed this task on the same data, would they necessarily get the same answer? If they might legitimately differ, it is judgement.
• Can the answer be checked by someone who was not there, using only the data and the rule? If checking requires knowing the client, it is judgement.
• Does the value of the output lie in its completeness and consistency, or in the insight of the person producing it? If the latter, it is judgement.

The pattern in every row is the same: the automation produces a fact, and the auditor decides what the fact means. The moment a tool starts producing the second column, the file stops recording anybody's judgement.

What full-population testing actually changes

Sampling exists because testing everything by hand is unaffordable. When the mechanical part is automated, that constraint loosens, and three things change in the audit rather than just in the cost.

1. 01Detection of rare items improves substantiallyA duplicate pair, a single unauthorised payment or one round-sum series is unlikely to be caught by a sample of forty from twelve thousand. It is certain to be caught by a rule applied to all twelve thousand. This is the clearest quality gain available, and it is not an efficiency argument at all.
2. 02The sample changes purpose rather than disappearingOnce everything has been matched, a sample is no longer the basis for the conclusion on accuracy. It becomes a check on whether the matching worked, which is a smaller sample with a different rationale. The working paper has to say so rather than carrying forward last year's basis.
3. 03Exception volume rises, and that is not a problem unless it is mismanagedTesting everything surfaces more exceptions than testing forty items, most of them immaterial. The risk is an exception queue nobody works through, which converts a quality gain into a documentation failure. Every exception still needs a disposition.

How the boundary erodes

Nobody decides to automate judgement. It happens through four specific slippages, each of which looks reasonable at the time.

The common feature is that each slippage removes a person from a decision without removing the decision. The counter to all four is the same: require a name and, where it matters, a reason.

Keeping the boundary in place

• Label what produced every piece of content: computed by a rule, drafted and unreviewed, or written by a named auditor. A reader who cannot tell which is which cannot evaluate the file.
• Require a written conclusion from a named person before a paper can be approved. Not a generated one, and not an empty field.
• Require a reason when an exception is dismissed. The reason belongs in the audit file, because the decision to dismiss is audit evidence.
• Do not display confidence percentages. A poorly calibrated number invites misplaced trust, and coarse bands with stated reasons route work just as well without pretending to a precision nobody has.
• Keep the arithmetic deterministic. A language model should never compute a total, a variance or a reconciliation, because it may be right most of the time and there is no way to tell which time this is.
• Make the trail append-only, so a decision cannot be quietly revised after the fact.

What the working paper must say

An automated procedure needs more description in the file than a manual one, not less, because the reviewer cannot infer what was done from watching someone do it.

• The population the routine ran over, and evidence that it was complete.
• The rules applied, in enough detail that another auditor could evaluate whether they address the assertion. “System-matched” is not a procedure description.
• Any parameters or thresholds used, and why those values.
• What was done to establish that the routine works, which usually means running it against data with a known answer.
• The exceptions produced, and each one's disposition by a named person.
• The conclusion, in the auditor's own words, addressing the assertion.