Skip to content
Cameron Mills

All projects

Can AI check financial figures? Tests with thresholds fixed in advance

cashflow-lab

Verification experiments on public company filings, two of them on UK accounts. Each had a pass or fail line set before it ran, and the results are reported against that line, including the one that missed.

Problem

Anyone using AI on financial documents has to answer one question first: when it reports a figure, is the figure right? I wanted measured answers, and a way to stop a weak idea early rather than tune it until it looked good.

Approach

Each experiment had a pass or fail line written down before it ran, and the result is read against that line as it was set. Agents ran the experiments unattended and wrote the reports. Every result below is stated as the line, the measured value and the decision.

Architecture

Three small tools built on public company filings. A distress flag scores companies from their accounts and filing history, and is tested against companies that later entered insolvency, with random companies as controls. A number checker reads the figures quoted in an analyst's memo and checks each against filed facts, sorting errors into kinds such as scale, period and sign. A filing-facts service returns figures from companies' tagged accounts, and declines to answer rather than guess when it cannot be sure.

What I built

  • Distress flag. Line set in advance: flagged companies fail at 5 times the normal rate or more. Measured: 3 times. Decision: misses the line. Recommended: stop, or allow one pre-registered re-test.
  • Number checker. Line set in advance: catch at least 90% of planted errors, with at most 5% false alarms. Measured: every planted error caught, no false alarms. Decision: passes its line.
  • Filing facts. Line set in advance: at least 95% of figures right. Measured: the first version served wrong figures. After the fix every figure it returned was right, but it returned figures for under half the fields. Decision: accuracy passes after the fix, coverage does not.

Results

The distress flag missed its line, and the report says so in its first sentence. The one re-test the report allowed, on a fresh window, has had its window, weights and kill line written down on 8 October, before any request went out. Every wrong figure in the first filing-facts version came from group accounts, where the tagged figures belonged to the parent company alone. The fix declines to give any figure from group accounts. The public claim the report allows states accuracy and coverage separately, with the sample size, and does not support any claim of perfect accuracy.

cashflow-lab: results against their thresholds
CheckExpected or thresholdActualResult
Distress flagFlagged companies fail at 5 times the base rate or more3 timesMissed
Number checker, seeded errors caughtAt least 90%Every onePass
Number checker, false alarmsAt most 5%None on clean memosPass
Filing facts, first versionField accuracy of at least 95%Several wrong figuresMissed
Filing facts, after the fixField accuracy of at least 95%Every returned figure rightPass
Each line was set before its experiment ran. Filing facts also returned figures for under half of the fields, a coverage figure with no line set in advance.

What didn't work

Three caveats sit beside these results. The distress flag's line was not set long in advance: a smaller first run had already been read against it. The filing-facts line came from a research note rather than the project itself. The number checker's perfect score was measured on a synthetic set built from the same facts and vocabulary the checker was written against, so it proves the matching logic, not that it copes with real analysts' writing.

Stack

  • TypeScript
  • Node.js
  • iXBRL
  • Claude Code

How these figures are sourced

The outcome figure comes from an experiment: measured once and reported against criteria fixed before it ran. The status was read from the code repository at commit b874161. Every figure and claim on this page is tied to a source file or a CV line in the site's content.