Gateway-reported cost for each of 418 model calls.
Signal experiments
Can models reading our own production chats find problems worth fixing, and at what cost?
Chats 139
Model calls 418
Total spend $3.32
Outcome
Reading every production chat with a strong model costs about two cents per chat, or roughly a dollar a day at today's volume. Over three days it wrote 43 problem notes, which one grouping call turned into 26 distinct findings, 11 of them bugs. The daily Insights digest flagged 9 chats in the same days; the reading found problems in 7 of them, plus 18 findings with no flagged chat behind them.
Jev, our cheap decision model, answers 13 questions about one chat for a fiftieth of a cent. On short text with a concrete question it is a good tool. On long transcripts it does not separate good chats from bad ones: as a filter it passed 88% of them, so it saves nothing at our volume. The clearest bugs did not come from reading at all, but from grouping tool errors by message in plain code.
The limits are real. A model (Claude Sonnet 5) set the reference labels; no person reviewed them. Thresholds were tuned on one day and checked on two others. No finding has been fixed or proven in code yet.
Setup
Three readers answered the same 13 questions about each chat
We took every chat from three days through the ops tools, rendered each one as a transcript with every tool call and its result, and gave the same questions to three readers. The strong reader also wrote one sentence naming the problem, the way an engineer would file it.
flowchart TB Ops[Production chats<br/>via pnpm ops] --> T[Transcript with<br/>every tool call] T --> J[Jev<br/>one call, 13 answers] T --> S[Flash Lite<br/>labels] T --> R[Sonnet 5<br/>labels + note] R --> G[Group notes<br/>into findings] G --> F[26 findings]
Something is brokenWrong resultNo resultUser repeats the requestAgent asks againA value was revertedCredits block the userMissing capabilityClaims done without checkingEscalated to supportUser is frustratedNot a website taskSafety risk
Cost
The cost per chat spans a factor of one hundred
Costs are what the AI gateway reported for each call, about 4 to 6 thousand input tokens per chat. At 50 to 70 chats a day, even the strong reader stays near a dollar. So at today's volume, cost is not the reason to filter; it becomes one only when volume grows ten times.
Jev$0.0002
per chat · 0.4 s
$0.03 for all 139Gemini 3.5 Flash Lite$0.0017
per chat · 1.3 s
$0.24 for all 139Claude Sonnet 5$0.021
per chat · 8.3 s
$2.92 for all 139Grouping$0.14
43 notes → 26 findings
one Sonnet 5 callJev
Jev is cheap, but it cannot be the filter on long transcripts
The idea was a funnel: Jev reads everything, and only the chats it flags go to the expensive reader. On full transcripts Jev's probabilities are high for almost every chat, so the funnel keeps almost everything. Flash Lite goes the other way: what it flags is right, but it misses more than half.
Thresholds tuned on 21 Oct did not hold on 19–20 Oct. For example, "a value was reverted" was perfect on the first day and gave 5 false alarms and no hits on the next two.
Method
Hide the tool calls and the model sees problems that are not there
The first run gave the readers only the text of each message and the names of failed tools. The strong reader then said the agent "claimed it was done without checking" in 20 of 65 chats on 21 Oct. With every tool call and its result in the transcript, the same day gave 9. The checks had run; the reader could not see them.
chats labelled "claimed done without checking"
33 problem notes
chats labelled "claimed done without checking"
24 problem notes
Insights digest
The digest hears users; reading every chat finds more, and some digest items are noise
Fimo Insights posts a daily summary to Slack, with a "What users said" section. Most of what it flags, the reading also found. The reading also found problems in chats where the user never complained.
Some digest lines repeat for weeks with no owner, for example credits charged for errored turns on its 26th observed day. A line with no action is a chart, not a signal.
9chats flagged by the digest
across 19–21 Oct
7 of 9also flagged by reading
no problem seen in the other 2
18findings with no flagged chat
8 of them bugs
Findings
Three days of chats grouped into 26 findings, half of them checkable by code
One grouping call merged notes that one fix would remove. It also said, for each finding, whether a code check could catch it again or whether it needs reading. These are candidates, not confirmed bugs.
Code signal
One code check over tool errors gives many findings for free
Of 2,387 tool calls in three days, 75 failed. Grouping them by tool and normalized error text needs no model, and each new error text is its own finding. This is the answer to "can one source feed many signals".
The memory tool fails one call in ten, almost always a write over a file the agent did not read first or that changed underneath it. The skill loader fails with a message that names no cause. Content queries reject AND.
Top error texts, normalizedops tool results · 19–21 Octmemory 15× <file> changed since you last read it / already exists and you have not read it
fimo 5× entry write: expected record, received undefined
fimo 4× The session is closed.
skillLoad 5× An error occurred.
contentSql 2× SQL function "and" is not available in entry queriesRules
A signal must earn its place; explorations find the next ones
Reading every chat is an exploration: it finds candidates. A candidate becomes an always-on signal only when it keeps coming back and someone owns the action. Without these rules the factory ends up like the digest's oldest lines: visible every day, acted on never.
- It comes from evidenceA real incident or an exploration found the problem at least once.
- It names its actionNo action means it is a chart, not a signal.
- It is quiet when healthyTrack how many of its findings led to a fix.
- Noisy means reviewThree dismissed findings in a row, or 14 days open with no action, and an agent proposes a tune or a delete.
- Silent is fineAs long as it proves it ran.
flowchart TB E[Exploration] --> N[Nothing:<br/>pause after 4 runs] E --> B[Bug:<br/>finding and fix PR] E --> I[Improvement:<br/>PR and watch] E --> P[Keeps coming back:<br/>new signal]
Next
What a daily run would look like, and what is still unproven
Code checks first, reading second, Jev where the question is short. Each finding goes to a coding agent as a draft PR with a failing test, and a person reviews it. The first candidates are the memory write conflicts, the opaque skill loader error, and content queries that reject AND.
Code signals$0
tool error texts, reverted settings, run outcomes
Read every chat~$1.5
per day, with a daily cap, then one grouping call
Jevpennies
short, concrete questions only
Counted from production tool results, 19–21 Oct.
Set by Claude Sonnet 5. No person reviewed them.
Tuned on 1 day, checked on 2. Too few positives to trust.
None investigated in code yet.