Skip to content
Cameron Mills

All projects

Finding the AI calls that do not need a frontier model

Model substitution

Part of Chief of Staff

A harness, built but not yet merged, that reads an agent system's own records, finds the narrow and repetitive calls that could run on a small open model, swaps one in behind the same interface, and measures what changes in cost and quality.

This build is part of the chief of staff's test management system. It is the part that tests which of an agent system's calls could run on a smaller model.

Problem

Most agent systems send narrow, repetitive jobs through the most expensive model available. Sorting, extracting and checking a fixed list are typical, and a small open model would do. I wanted to find those jobs in a real system's own records, rather than guess at them.

Approach

Use my own Life Tracker's agents as the first subject, because every run already leaves a full record: several hundred runs over a month, and still growing. Rank the candidate jobs by how often they happen and how predictable they are, swap a small open model in for the best one, and compare cost and agreement on examples held back from development. Research and review come first, through my /build pipeline.

Architecture

The substitute model runs on the Mac itself, so trying it costs nothing per call. It sits behind the same interface as the original call in a separate tool, rather than inside the live agent system, so the experiment cannot disturb the system it is measuring. Because the records contain personal material, the proposed design keeps their content inside the tracker and gives the new tool code only, which still needs my sign-off.

What I built

The research and its review came first. The first round measured where the spend actually goes, and found that some of the work had already moved to a cheaper model, concentrated in a few large research runs. It proposed the editing pass every report goes through as the first candidate: a clean proof, though not the biggest saving. A six-stage build then produced a working harness on a separate branch: a reader of the records, a ranker, a local substitute behind the original call's shape, a scorer and a report writer. Its 70 tests pass, but it is not merged, and the real measurement waits on my sign-off for reading the private records.

Results

About nine-tenths of the spend still goes to frontier models, against about two-thirds of the tokens, so how the opportunity is measured changes its size. The review held the plan back until the method is sound.

What didn't work

The review found the first candidate was neither frequent enough nor easy to score: the proposed measure of agreement could not tell a real model from one that always gave the same answer. It also pointed out that nothing planned yet was distillation in the strict sense, training a small model on the large one's outputs, so the next round has to either do that or call the result substitution. The final review's three judges each scored the finished build 4 out of 10, because the substitute is prompted rather than trained and the measurement so far rests on two synthetic items.

Stack

  • Python
  • Ollama
  • Claude Code

How these figures are sourced

The outcome figure is agent-reported: it is taken from a test log or report written by the agents that built and ran the system. The status was read from the build's final review. Every figure and claim on this page is tied to a source file or a CV line in the site's content.