Can AI check financial figures? Tests with thresholds fixed in advance
cashflow-lab
Verification experiments on public company filings, two of them on UK accounts. Each had a pass or fail line set before it ran, and the results are reported against that line, including the one that missed.
Problem
Anyone using AI on financial documents has to answer one question first: when it reports a figure, is the figure right? I wanted measured answers, and a way to stop a weak idea early rather than tune it until it looked good.
Approach
Each experiment had a pass or fail line written down before it ran, and the result is read against that line as it was set. Agents ran the experiments unattended and wrote the reports. Every result below is stated as the line, the measured value and the decision.
Architecture
Three small tools built on public company filings. A distress flag scores companies from their accounts and filing history, and is tested against companies that later entered insolvency, with random companies as controls. A number checker reads the figures quoted in an analyst's memo and checks each against filed facts, sorting errors into kinds such as scale, period and sign. A filing-facts service returns figures from companies' tagged accounts, and declines to answer rather than guess when it cannot be sure.
What I built
- Distress flag. Line set in advance: flagged companies fail at 5 times the normal rate or more. Measured: 3 times. Decision: misses the line. Recommended: stop, or allow one pre-registered re-test.
- Number checker. Line set in advance: catch at least 90% of planted errors, with at most 5% false alarms. Measured: every planted error caught, no false alarms. Decision: passes its line.
- Filing facts. Line set in advance: at least 95% of figures right. Measured: the first version served wrong figures. After the fix every figure it returned was right, but it returned figures for under half the fields. Decision: accuracy passes after the fix, coverage does not.
Results
The distress flag missed its line, and the report says so in its first sentence. The one re-test the report allowed, on a fresh window, has had its window, weights and kill line written down on 8 October, before any request went out. Every wrong figure in the first filing-facts version came from group accounts, where the tagged figures belonged to the parent company alone. The fix declines to give any figure from group accounts. The public claim the report allows states accuracy and coverage separately, with the sample size, and does not support any claim of perfect accuracy.
| Check | Expected or threshold | Actual | Result |
|---|---|---|---|
| Distress flag | Flagged companies fail at 5 times the base rate or more | 3 times | Missed |
| Number checker, seeded errors caught | At least 90% | Every one | Pass |
| Number checker, false alarms | At most 5% | None on clean memos | Pass |
| Filing facts, first version | Field accuracy of at least 95% | Several wrong figures | Missed |
| Filing facts, after the fix | Field accuracy of at least 95% | Every returned figure right | Pass |
What didn't work
Three caveats sit beside these results. The distress flag's line was not set long in advance: a smaller first run had already been read against it. The filing-facts line came from a research note rather than the project itself. The number checker's perfect score was measured on a synthetic set built from the same facts and vocabulary the checker was written against, so it proves the matching logic, not that it copes with real analysts' writing.
Stack
- TypeScript
- Node.js
- iXBRL
- Claude Code
How these figures are sourced
The outcome figure comes from an experiment: measured once and reported against criteria fixed before it ran. The status was read from the code repository at commit b874161. Every figure and claim on this page is tied to a source file or a CV line in the site's content.