The four risks that matter most are hallucination on numeric and calculation tasks, where a model produces a fluent but wrong figure with no visible sign of error; the confidentiality exposure of sending client data to a third-party model, which is a disclosure decision whether or not anyone treats it as one; over-reliance by junior staff who accept fluent output without checking it, because fluency reads as confidence; and the general preference for extraction that a person can verify against a source document over a conclusion generated by a model that nobody can independently check.
Hallucination is worst exactly where it looks safest
A language model generates the next most likely sequence of words or tokens given what came before. On a numeric task — totalling a column, reconciling two figures, computing a variance — this means the model is not calculating in any sense a calculator would recognise. It is producing a plausible-looking number. Plausible and correct are not the same thing, and the gap between them does not announce itself in the output.
A model asked to total a column of figures can produce a wrong total that is formatted identically to a right one, in the same font, with the same confident tone. There is no visual or textual signal that distinguishes a hallucinated total from a correct one. The only reliable way to know is to recompute it deterministically and compare.
This is not a criticism of any particular model or vendor. It is how the underlying technology works, and it does not improve by using a newer or larger model for the same task. Arithmetic, matching and reconciliation belong to rule-based code that produces the same answer from the same input every time, not to a model that produces a statistically likely answer.
Where client data actually goes
Sending a document or a figure to an AI tool means that data leaves the firm's own systems and is processed by, at minimum, the model provider, and often by one or more sub-processors behind that provider. This is true of consumer chat tools and of audit software with an AI feature built in. The confidentiality question is the same in both cases: who can see this, where is it processed, and for how long is it kept.
| Question | Why it matters |
|---|---|
| Is this data used to train the provider's models? | If yes, information from one client's file can shape outputs seen by other users, including other firms' clients |
| Where is the data processed, and does it leave India? | Determines which law governs it and what recourse exists if something goes wrong |
| How long is it retained after the engagement, and can it be deleted? | A retention policy with no deletion mechanism is a retention policy in name only |
| Who are the named sub-processors? | A model provider behind a product is a sub-processor whether or not the vendor calls it one |
Regulatory requirement
Confidentiality obligations under the ICAI Code of Ethics attach to client information regardless of the medium used to process it. Separately, the handling of personal data is governed by Indian data protection law, which has been evolving. Confirm the current position and take advice before any client data reaches a third-party AI tool.
Over-reliance is a training problem before it is a tool problem
The failure mode that shows up most often in practice is not a dramatic hallucinated figure that someone catches on review. It is a junior staff member reading fluent, well-structured AI output and treating fluency as a proxy for correctness. A badly written wrong answer gets questioned. A well-written wrong answer gets filed.
- Staff new to a cycle are least equipped to spot a wrong figure, and are the group most likely to be handed AI output to speed up their work.
- A model's confident, consistent tone does not vary between a correct extraction and an incorrect one, so tone cannot be used as a quality signal.
- Reviewing AI-assisted work takes real discipline precisely because the output looks finished. A half-finished draft invites scrutiny; a polished one does not.
- 01Name what the tool is used for, specifically“AI-assisted” is not a description. “Extracted invoice fields from scanned PDFs” is. Specificity is what makes the control reviewable later.
- 02Require every extracted figure to be traceable to its source documentIf a figure cannot be checked against the page it came from, it cannot be verified, and should not be relied on without independent recomputation.
- 03Keep arithmetic out of the model's hands entirelyTotals, reconciliations, variances and matching should be computed by code the firm or vendor can show behaves the same way every time, not generated.
- 04Train staff on the specific failure mode, not AI in general“The output can be confidently wrong and look identical to correct output” is a trainable, checkable instruction. “Be careful with AI” is not.
Extraction over generated conclusions
The distinction that holds up under scrutiny is between a model reading something that already exists — a figure on an invoice, a date on a bank statement — and a model producing a conclusion about what something means. The first is checkable in seconds: look at the source, look at the extraction, confirm they match. The second is not checkable at all without redoing the underlying work, because the conclusion is the thing that was supposed to be produced by judgement in the first place.
A firm that lets a model draft a description of a transaction, which a person then reviews and rewrites, is using the model for something a person can verify. A firm that lets a model decide whether a related-party transaction was conducted at arm's length is handing over the judgement itself, with a fluent paragraph standing in for a person's name on the conclusion.
How we think about this
Every risk above traces back to one distinction: a model's output is either checkable against a source in a bounded amount of time, or it is not. Lekhya restricts model use to extraction, classification and drafting — tasks that fall on the checkable side — computes anything numeric with deterministic code, and leaves every conclusion to a named auditor. The risks in this article are not solved by a better model. They are managed by keeping the model away from the tasks where its errors are invisible.
This is practical guidance written for audit teams, not an authoritative statement of any standard or statute. Standards, forms and timelines change. Check the current text issued by the ICAI and the relevant authority before relying on anything here in an audit file.
Related resources
- Technology, data & AI in auditData analytics in audit: what full-population testing changesThe rare but material exceptions full-population testing surfaces that sampling cannot, and the judgement about what an exception means that no tool replaces.
- Technology, data & AI in auditCloud audit software and data residency: what to askWhere client data physically sits, who inside a vendor can see it, and the difference between an architectural guarantee and a policy that could change tomorrow.
- Technology, data & AI in auditChoosing an audit automation vendor: a checklistThe questions that separate a vendor worth trusting with client data from one that merely demos well: security, what's computed versus generated, exit terms, pricing, and India-specific fit.
- Audit workflow & automationAI in audit: what it is good at, and what it must not touchThe three tasks language models genuinely do well in audit, the three they must never be given, and the questions to ask a vendor before any client data moves.
Free tools, no signup
Each one reads a file entirely in your browser — or, for two of them, just answers a few questions — and nothing is ever uploaded.
- Purchase register checkDuplicate payments, round-sum amounts, weekend-dated invoices, incomplete records.
- GSTR-2B reconcilerMatches your purchase register against GSTR-2B — missing invoices either side, amount mismatches.
- Bank reconciliation checkerMatches your cash book against a bank statement — what's outstanding on each side.
- Fixed asset register checkDuplicate asset codes, missing location or useful life, negative net book value.
- Payroll red-flag scannerBank accounts shared across employees, and records missing an ID, PAN or salary.
- CARO applicability checkerAnswer a few questions about the company and see which CARO 2020 clauses likely apply.
- MSME payment delay checkerMSME-flagged invoices paid — or still unpaid — beyond the 45-day Section 22 limit.
- Related-party transaction screenerTransactions above a threshold you set with no approval reference, grouped by counterparty.
- Depreciation useful life checkerCompares a useful life in use against the Schedule II, Part C indicative figure.
- Materiality calculatorOverall and performance materiality against PBT, revenue, assets or equity, with reasoning shown.
- Sample size calculatorStatistical sample size for a test of controls or a substantive test of detail.