Essay
What I learned trying to reduce my LLM costs
Justin Ahinon
I've asked my coding agent some version of "can I make this cheaper?" so many times that I recently needed another session just to find the previous attempts.
My main OpenRouter key cost about $77 in August. I'd already spent months trying to make it cheaper. Nobody was calling an emergency meeting about the bill, but I kept finding another experiment I wanted to try.
I run WhatIsThatBook, where people describe a book they half remember and the app tries to name it. The model suggests candidates, then the app checks them against a book catalog. A cheaper answer is only useful if the reader still finds their book.
The conversations go back to June. I started by comparing models, then wondered whether I needed one for every search. At one point I was considering training something tiny to do this one job.
What eventually helped was a shorter prompt and a different pricing tier for the model I was already using. I hadn't expected to spend months getting to those two changes.
The cheaper models usually came with something missing
My first instinct was to compare model prices. The app returns a handful of book titles and authors. Surely one of the many cheap models could handle that.
The response format makes the task look simpler than it is. To identify a particular children's book from a vague description, the model has to know something about that book. It can return perfectly valid JSON with five real titles and still miss the one the reader is looking for.
I used the eval set I'd already built. The main score was whether the correct book appeared among the first five results. I also checked response failures, latency, and cost.
Even the comparison needed checking. Mistral initially looked promising while live OpenLibrary requests were timing out. Once I used a local catalog to make that part repeatable, Gemini was substantially better. I'd partly been measuring how well an external API happened to behave during each run.
By August, I had a comparison using the same questions and catalog lookup for each model. Try choosing on price before revealing how many books each one found.
Try the comparison
Which model would you pick?
Start with the price. Pick a model to see how many books it found.
The prices are only half the comparison.
See all benchmark results
| Model | First result | Within five |
|---|---|---|
| Gemini 3.7 Flash | 69/100 | 71/100 |
| GPT-5.6 Terra | 64/100 | 66/100 |
| Qwen 3.7 Plus | 56/100 | 60/100 |
| Nemotron 3.5 Lightning | 47/100 | 50/100 |
All four completed the pipeline on 100% of cases. Costs are token-based estimates at the prices used in that run, excluding catalog costs. These are historical results, not current model rankings. Read the benchmark.
The discounts were real. So were the missing books. Getting a model to return valid JSON was sometimes a job in itself, and fixing that didn't mean it knew the answer.
These results describe that particular experiment. The prompts, prices, and test sets changed over the months, so the savings later in this article aren't percentages I can add together.
Sometimes the experiment hadn't happened yet
Gemini Flash Lite got close enough offline to deserve a production test. On one hundred cases, it found the right first result on 66%, compared with 67% for the control, at roughly half the projected cost.
I implemented a 50/50 feature-flag rollout. When I came back to review it, PostHog showed 3,454 logical model attempts on the control and zero on Flash Lite.
The flag was enabled, but when the app evaluated it, the missing value sent everyone back to the control model. Later, timeouts interfered with assignment too.
Once treatment traffic existed, the feedback still didn't give me enough confidence to switch everyone to Flash Lite. The flag alone had told me very little about which path people actually took.
Maybe I could call the expensive model less often
The next idea was to start with GLM and escalate only when it looked uncertain. This seemed more reasonable than asking a cheap model to handle every query.
One rule looked good: use Gemini when GLM failed, returned nothing, or produced only a small number of low-confidence recommendations. On the development fixture, the hybrid matched Gemini's top-five score, escalated 19% of cases, and promised roughly 69% savings.
I wanted to follow that up, but I'd selected the rule by looking at those same cases. I still needed to find out whether it worked on searches I hadn't used to choose it.
On a separate fifty-case validation set, GLM missed fourteen cases that Gemini solved. My 20% escalation cap allowed ten Gemini calls. Even a router that somehow knew exactly which answers were wrong couldn't recover all the losses. The actual router did worse than that imaginary perfect one.
Calling Gemini more often might recover more answers, but it would also eat into the saving. Under the limit I'd set, one Gemini call for every five searches, the hybrid couldn't keep up.
I also tried finding books directly from their descriptions, without asking a model to generate candidates. On an untouched production-like sample, that cut variable cost by 99.96%. It dropped the top-five score from 26% to 6%. Some books weren't in the searchable catalog; others never made it into the candidate list. Paying a model to reorder that list couldn't bring back a book it never received.
Web evidence still helped as a fallback. Replacing the normal identification path with retrieval was another matter. I explored specialist models and caching too, but the next useful saving came from the prompt I already had.
The prompt got smaller. Then the price changed
I'd also been shortening the prompt I already used with Gemini, which turned out to be more useful than most of the model comparisons.
In August, I cut average input from about 2,683 tokens to 901. Output grew a little, from 361 to 416 tokens. At the prices used in that evaluation, estimated generation cost fell from $0.1682 to $0.1118 per hundred calls, a 33.5% reduction.
First-result accuracy stayed at 69%. The top-five score moved from 71% to 73%. That was one hundred cases per prompt, so I wouldn't claim I'd proved an accuracy improvement. It was enough to keep the shorter version.
Then the Gemini 3.7 promotion ended.
The model had been discounted when I adopted it. At regular pricing, its token rates were higher than the older Gemini model's. I'd discussed that with my agent at the time, but it becomes much easier to appreciate once it shows up in the export.
For Gemini 3.7 calls on the main key, the bill averaged about $1.68 per thousand calls from August 20 through 27. From August 29 through September 4, it was about $3.27. Those were different queries, but comparing billed amounts with token counts supported the pricing explanation: the earlier calls cost roughly half the standard-rate estimate; the later calls cost almost exactly the standard rate.
Removing tokens had helped, but it didn't protect me from paying more for each remaining token.
That changed where I wanted the next investigation to start. I had spent a lot of time comparing models. I wanted to look closely at what I was actually being charged for.
The seventy-percent saving I didn't keep
On September 5, I gave Codex the previous experiments, an OpenRouter export, and a $5 evaluation budget. The goal was to find a substantial saving while keeping match quality.
The audit gave me more things to test, including reasoning tokens, unused explanations, and extra calls. Two cheap models agreeing on an answer sounded useful until paying for both, then escalating uncertain cases, made it more expensive. Asking for a second opinion on every "no matches" response added twenty-nine calls for one extra correct match.
The promising option was an OpenRouter Flex endpoint for the same Gemini model. Combined with another shorter prompt, it appeared to cut cost by about 70% and improve title-and-author matches.
Then I ran the saved responses through the book catalog.
Look one step further
Did the cheaper prompt find more books?
The same saved answers look different depending on the scoring method.
The shorter prompt plus Flex looks better when comparing generated titles and authors: seven more successes.
Top-five successes, including correct abstentions where applicable. September 5's first shortest-prompt experiment saved 70.4% on primary generation. It was rejected; the final Flex change kept the original prompt. These views use different scoring rules or subsets, so compare the two prompts within each view.
See all three views and the source
| View | Original | Shorter |
|---|---|---|
| Generated titles | 107/171 | 114/171 |
| Catalog results | 123/171 | 122/171 |
| Additional cases | 48/71 | 45/71 |
Some apparent gains disappeared during resolution. The shorter prompt missed constraints, abstained on difficult queries the original had answered, or dropped a useful candidate. A revised version fixed the constraint checks but still lost top-five matches on the additional cases.
I couldn't call that preserved quality just because the aggregate title score looked better.
The final change kept the existing production prompt, model, and low-reasoning setting. It preferred Flex and allowed provider fallback. If generation still failed to return usable output, the app made one additional attempt through standard routing. A valid "no matches" response didn't trigger that extra call. I'd just measured how expensive it could be to keep asking for a second opinion.
Two runs over the same 171 queries measured roughly 53% to 58% lower primary-call cost. The range accounts for missing billing information on failed calls. It excludes additional retry and fallback spending.
Quality was close, with a small observed loss: across the repeated evaluations of answerable cases, Flex found the correct book within five results 231 times, versus 234 for standard routing. Those are repeated observations, not hundreds of independent queries, and they don't prove equivalent quality.
That's the result I kept. The public experiment notes include the less flattering parts too.
Try your own numbers
What would it save on the whole bill?
A discount on identification only changes that part of your spending.
$33.00 saved per month
Estimated bill: $67.00. Overall reduction: 33.0%.
Illustration, not a forecast. The starting 55% discount is within the benchmark range; the $100 bill and 60% share are example assumptions. Extra retries, fallback overhead, and changes in usage are excluded.
Where Astra helped
Astra ran the September investigation in Codex, using the experiments and tools I'd accumulated over the previous months. It reused saved responses for catalog checks and caught differences between the test requests and the production settings. That round cost about $3.61 in evaluation calls, separate from Codex usage and my time.
It also reported the promising 70% saving before the catalog checks were finished. The useful part was continuing past that result and finding the lost matches.
I can't separate Astra's contribution from the better starting point it had. But it reinforced something I keep writing about: leaving usable experiments behind makes the next investigation easier.
What the first production numbers say
The Flex PR merged on September 7. In that evening's data, spending was about 47% lower than the previous evening, with almost the same number of identification outcomes. Usable responses were faster too.
That was encouraging. The next morning, I checked the OpenRouter export against PostHog, following my investigation runbook. I compared completed evening hours because an afternoon failure period would distort a whole-day comparison.
All three rows cover 17:00 to midnight UTC. Latency is for identification calls that produced usable responses.
| Evening | Logical identification outcomes | Main-key API spending | Median identification latency |
|---|---|---|---|
| August 31 | 323 | $1.60 | 8.97s |
| September 6 | 206 | $1.20 | 7.57s |
| September 7 | 209 | $0.64 | 4.18s |
People searched for different books each evening. I know when the PR merged, but haven't verified exactly when the deployment reached users. I also can't connect each charge in the export to a particular search or see every provider fallback. Slightly fewer searches returned results, and I still need to know whether those results were right. That will take more than an evening of data.
There is also spending outside the changed stage. Around one in five identification outcomes invoked the web-grounded lane during the preceding week. Its extraction calls weren't changed by this PR. A discount on the first call can't simply be applied to the whole bill.
I started this expecting to find a cheaper model and move on. I ended up learning much more about the evaluation, the resolver, and how providers charge for the same workload.
I'll probably try another cheap model. I'm still interested in finding one that works for this. Next time, though, I want the comparison to include catalog resolution and the extra calls from the start, priced at what I'm actually paying.
For now, the smaller change is merged. I'm watching what happens to the bill and whether people still find their books.