On September 23, I exported 47 hours of OpenRouter activity for WhatIsThatBook. It held 2,104 calls, all to Gemini 3.7 Flash, for a total of $4.13.

That's not a scary bill. But I'd just been reading Cloudflare's Workers AI pricing, and I wanted to know if one of the cheap models in their catalog could do the same job. By the end of the evening I'd screened seven cheaper models, and one looked close enough on my eval to put in front of 70% of my users. I turned that flag off fourteen minutes after turning it on.

I've written about the months I spent trying to reduce these costs and about the eval set behind them. This one is the practical part: the harness I'd run before switching models, rebuilt from what that week taught me as a small TypeScript project you can clone. It's the llm-eval-harness folder in my segbedji-tutorials repo.

It runs without an API key. A mock provider stands in for OpenRouter, so you can watch every check pass and fail before you spend anything.

The harness has to answer five questions before a switch

WhatIsThatBook takes a half-remembered description ("a boy who lives in a cupboard under the stairs gets a letter from a wizard school") and asks a model for up to five candidate books as JSON. The app then checks each candidate against a book catalog. A model is only useful if the reader's book ends up in that list.

So before I swap models, I want one run to tell me:

  1. How often is the right book first? How often is it in the top five?
  2. How many calls failed outright?
  3. How many times did the model return nothing at all?
  4. What did the run cost, and could it have cost more than I meant to spend?
  5. Should CI block this change?

My first version answered all of them except the third. That's the one that sank the rollout, and I'll get to it.

The companion repo is about 1,100 lines of TypeScript with two dev dependencies. Every code block below is a trimmed excerpt; the full files are in the repo.

cases/books.jsonl                  16 cases, OpenLibrary work IDs
catalog/openlibrary-snapshot.json  title + author → work ID, committed
src/schema.ts                      strict schema, strict-mode checker, parser
src/budget.ts                      spend cap
src/runner.ts                      the eval loop
src/score.ts                       P@1, P@5, MRR, empty answers
src/gate.ts                        baseline comparison, exit codes
src/providers/                     OpenRouter + mock

Cases need IDs, not titles

Each case is one JSONL line. Here's one from the companion fixture, which uses public-domain books:

{
  "id": "case-013",
  "query": "weird old book narrated by a square who lives in a two dimensional world and gets visited by a sphere, it's also making fun of victorian society",
  "expected": { "title": "Flatland", "author": "Edwin A. Abbott" },
  "acceptable_work_ids": ["OL118388W", "OL118393W"],
  "difficulty": "hard",
  "answerable": true
}

Two fields carry most of the weight.

acceptable_work_ids is a list because OpenLibrary has duplicate work records for the same book. Flatland has at least two. My first real baseline scored 25% because the scorer expected exactly one ID, and many of those "misses" were the right book under another ID. The fixture says up front which IDs count.

answerable: false marks cases like "the one about the guy". The right answer there is no books and a clarifying question. A model that guesses Great Expectations on that query is wrong, even though Great Expectations is a real book.

The fixture gets validated before any run. bun run validate checks the shape. bun run validate:online fetches every work ID from OpenLibrary and fails if one redirects or isn't a work. All 18 IDs in the companion fixture passed when I ran it.

Send the exact schema production sends

This one skewed my first results for GPT-6 Luna.

The app has two versions of its response schema. The one tuned for Gemini marks the filter fields as optional. OpenAI's strict structured outputs don't allow optional fields: every property has to be listed in required, and "maybe absent" has to be written as nullable. The app already had a strict version for that reason. My eval script didn't use it. It sent the Gemini-shaped schema to every model.

With strict mode off, Luna mostly coped. On a 20-case screen it got 13 books right at rank 1, and one response failed the app's schema because a required field was missing. With strict mode on and the loose schema, OpenAI rejected the request with an HTTP 400 before generating anything. Once the eval sent the app's real strict schema, Luna on Flex routing got 14 of 20, matching Gemini on the same cases, with no format errors. That run also moved from standard routing to Flex, so the schema wasn't the only thing that changed.

Nothing changed between those runs except the schema the eval sent. So the harness has one schema, the strict one, and it checks that schema against strict-mode rules before the first call:

// Trimmed from src/schema.ts
export const strictModeProblems = (schema: JsonSchema, path = '$'): string[] => {
  const problems: string[] = [];
  if (schema.properties) {
    const required = new Set(schema.required ?? []);
    for (const key of Object.keys(schema.properties)) {
      if (!required.has(key)) problems.push(`${path}.${key} is not in "required"`);
    }
    if (schema.additionalProperties !== false) {
      problems.push(`${path} must set additionalProperties: false`);
    }
    for (const [key, child] of Object.entries(schema.properties)) {
      problems.push(...strictModeProblems(child, `${path}.${key}`));
    }
  }
  if (schema.items) problems.push(...strictModeProblems(schema.items, `${path}[]`));
  return problems;
};

The repo keeps the loose schema around so you can see the failure:

bun src/cli.ts --mock --model mock/steady --schema loose

✗ The "loose" schema is not strict-mode compatible:
  - $ must set additionalProperties: false
  - $.recommendations[].published_before is not in "required"
  - $.recommendations[] must set additionalProperties: false

The request itself uses strict: true and asks OpenRouter to route only to providers that support it:

// Trimmed from src/providers/openrouter.ts
response_format: {
  type: 'json_schema',
  json_schema: { name: request.schemaName, strict: true, schema: request.schema }
},
provider: { require_parameters: true },
usage: { include: true }

require_parameters matters more than it looks. Without it, a provider that ignores response_format can still serve the request, and you end up measuring how well a model guesses your JSON shape.

Resolve answers the same way every run

The model returns titles and authors, but the scorer compares work IDs, so something has to turn one into the other.

In the cost essay, I mentioned that Mistral once looked promising while live OpenLibrary requests were timing out. I had been partly measuring the network. The harness avoids that by resolving against a committed snapshot by default. scripts/build-catalog.ts builds it from OpenLibrary search, and --resolver openlibrary switches to the live API with a disk cache when you need it.

Scoring then works on IDs:

// Trimmed from src/score.ts
export const firstHitRank = (resolvedWorkIds: (string | null)[], acceptable: string[]) => {
  const accepted = new Set(acceptable);
  const index = resolvedWorkIds.findIndex((id) => id !== null && accepted.has(id));
  return index === -1 ? 0 : index + 1;
};

P@1 is the share of answerable cases with rank 1. P@5 is rank 1 to 5. MRR averages 1 / rank, so a book at rank 2 earns half a point. Failed calls stay in the denominator as misses. Dropping them is the easiest way to make a flaky model look accurate.

Cap spend before the first call

Cheap models make it easy to stop paying attention to cost. The danger in an eval isn't the expected bill. It's the run where a reasoning model writes thousands of tokens per case, or a loop retries forever.

The harness reserves each call's worst case before making it:

// Trimmed from src/budget.ts
export const worstCaseCost = (prices: Prices, promptTokens: number, maxTokens: number) =>
  promptTokens * prices.prompt + maxTokens * prices.completion;

reserve(amountUsd: number) {
  if (this.spent + this.reserved + amountUsd > this.capUsd) {
    throw new BudgetExceededError(/* ... */);
  }
  this.reserved += amountUsd;
  // settle(actual) swaps the reservation for the real cost when the call returns
}

Three details make the cap hold:

  • Prices come from OpenRouter's /models/{id}/endpoints, and the harness takes the most expensive endpoint. The same model can cost different amounts depending on the provider that serves it.
  • Reasoning tokens bill as completion tokens and count against max_tokens, so max_tokens × completion price is still a real ceiling.
  • A failed call keeps its whole reservation as spent. You don't always know what a failure was billed, so the harness assumes the worst.

Live runs refuse to start without --max-cost. When the cap is hit, the run exits with code 2 and says how many cases were skipped. A run that stopped halfway isn't a score, and the harness doesn't treat it like one.

My real eval script works the same way. The 20-case screen that compared Gemini, Gemma 4 and GPT-OSS ran under a $1 cap and accounted for about four cents.

Make it fail CI

The last step compares the run against a saved baseline and turns the result into an exit code:

Code Meaning
0 passed
1 regressed: failures, empty answers, or P@1 dropped too far
2 stopped at the spend cap
3 configuration error, including a schema strict mode would reject

The baseline stores a hash of the fixture. If someone edits a case, the gate refuses to compare old and new scores and asks for a new baseline. Otherwise you can "improve" a model by quietly deleting the cases it misses.

The companion's GitHub workflow runs the mock eval on every pull request that touches the harness. It costs nothing and proves the gate still works. It runs the live eval only if the repo has an OpenRouter key and a committed live baseline.

Running it

bun install
bun run validate
bun test
bun src/cli.ts --mock --model mock/steady --save-baseline
bun src/cli.ts --mock --model mock/bargain --baseline runs/baseline.json

mock/steady plays the expensive model. mock/bargain plays the cheap one: it gets the easy classics, misses the rest, guesses on vague queries and truncates one response. Trimmed output from the second run:

P@1 50.0%  P@5 50.0%  MRR 0.500  (14 answerable, 1 failed)
No books on 0/14 answerable cases
Abstained correctly on 0/2 vague queries
Spent $0.000752 (2 calls estimated, not billed)

✗ Gate failed:
  - 1 failed calls (allowed: 0)
  - P@1 50.0% is 35.7% below mock/steady's 85.7% (allowed drop: 5.0%)

For a live run, set OPENROUTER_API_KEY, save a baseline with your current production model, then point --model at the candidate. Keep --max-cost small. Sixteen cases on a cheap model should cost cents.

Where my first version fell short

After the strict-schema fix, the paired 100-case run came back close. Luna Flex found the right book first on 64 of the 95 answerable cases. Gemini found it on 66. Luna had zero failures, and its billed cost for all 100 cases was $0.0202, against $0.0885 for Gemini. It was slower: a median of 5.2 seconds per generation against 2.8 for Gemini, and a much longer tail. I accepted that, put Luna behind a PostHog flag at 70%, and deployed.

Four of the first five production searches that reached Luna came back with no books.

The responses were valid JSON and parsed fine; they just held clarifying questions instead of candidates. I turned the flag off 14 minutes after enabling it, then replayed ten real searches that Gemini had answered with books. Luna returned books on three of them.

The prompt tells the model to abstain when the description is too vague to identify anything. Luna followed that rule much more strictly than Gemini did, and real searches are vaguer than most of my fixture.

Later that night I relaunched it in a guarded form, with Gemini filling in whenever Luna came back empty. That didn't hold either: all 8 searches routed to Luna came back without Luna candidates, and I switched it off at 22:54 UTC. By September 25 a different experiment held that 70% slot.

The evidence was already in the paired report. On the 95 answerable cases, Luna returned no books 21 times. Gemini did it twice, plus one call that failed outright. My summary printed P@1, P@5 and MRR, and none of those can tell "guessed wrong" from "didn't guess". In the app they're very different: a wrong guess still gives the reader something to click, and an empty page gives them nothing.

So the companion counts empty answers on answerable cases and gates on them:

// Trimmed from src/score.ts
emptyAnswers: answerable.filter((r) => r.status === 'ok' && r.abstained).length,

// Trimmed from src/gate.ts
const rate = summary.emptyAnswers / summary.answerable;
if (rate > options.maxEmptyRate) {
  reasons.push(`Returned no books on ${pct(rate)} of answerable cases ...`);
}

mock/cautious reproduces the problem. It matches the baseline's P@1 exactly and still fails:

P@1 85.7%  P@5 85.7%  MRR 0.857  (14 answerable, 0 failed)
No books on 2/14 answerable cases

✗ Gate failed:
  - Returned no books on 14.3% of answerable cases (allowed: 10.0%)

The default threshold is 10%. Luna was at 22% on my real fixture, and Gemini at 2%. Choose yours from your own production model's rate, not mine.

The cheap model wasn't always the cheap one

Two days later I ran the same eval on OpenAI models. GPT-5.4 Mini at low reasoning scored 66% at rank 1 against 69% for Gemini on Flex routing. It also returned no books on 18 of 100 queries, so this time I checked that number first. A Mini-specific prompt brought empty answers on answerable cases down to zero (the 4 left were correct abstentions) and rank-1 accuracy up to 68%.

The cost column surprised me. At OpenAI's list price, Mini with the prompt I shipped would have cost $0.231 for the 100 cases. Gemini on Flex had been billed $0.098 on an earlier run of the same cases. The "cheap" switch only made sense because OpenAI gives complimentary daily tokens to accounts that share data, and OpenRouter's bring-your-own-key routing appears to use them. A test Mini call through my production key came back with is_byok: true and no OpenRouter charge. On the OpenAI usage page, the week to October 27 shows 4,610 requests and 4.8 million input tokens, with $2.07 billed, all of it on October 22. The other six days show nothing billed. That page covers every project on my account, not just WhatIsThatBook, so I can't say what the $2.07 was for. It does suggest that about 650 requests a day mostly fit inside the free allowance.

That's not something the harness measures. It reports what OpenRouter bills. If a model is free up to some daily allowance, the eval can't tell you where your traffic will land against it.

What a 70/30 flag actually measures

I put Mini behind a 70% PostHog flag, with Gemini on the other 30%. About three and a half hours in, 81 first searches had gone to Mini and 24 to Gemini. That's 77/23, not 70/30.

Nothing was broken. The flag assigns visitors, not searches, so a few people who search a lot can tilt the split, and small samples wobble anyway. If you want an exact share of requests, you have to assign per request, and then the same person can see both models.

The flag ran for about 47 hours, from 15:15 UTC on September 25 to 14:38 UTC on September 27. Over that whole window, counting first searches and leaving out refinements:

GPT-5.4 Mini Gemini 3.7 Flash
Searches 770 261
People 510 183
Searches that returned books 688 (89%) 198 (76%)
Median / p95 model time 7.4s / 13.2s 4.4s / 9.1s
Searches marked "none of these match" 201 of 769 (26%) 56 of 256 (22%)
Searches with a confirmed match 11 10

The split settled near 75/25 by searches and 74/26 by people. Mini returned books far more often, which is what the prompt change was for. It was also slower, and readers rejected its lists a bit more often. Feedback is voluntary, so those last two rows describe the people who clicked a button, not every search. They also come from a separate query, which is why their totals are slightly lower than the search counts above.

On September 27, I moved every search to Mini, with Gemini as a fallback when Mini fails.

What I still haven't verified

The companion's numbers prove the plumbing, not any model. Every harness output in this post comes from the mock provider; the Gemini, Luna and Mini numbers come from my real eval. I haven't run the companion against live OpenRouter, because no key was configured where I built it. The pricing lookup does parse OpenRouter's real endpoint data.

Sixteen cases is enough to show the mechanics. It isn't enough to pick a model: one case moves P@1 by about seven points. My real fixture has 100, and even there, a two-point gap like 64 against 66 is two books.

The harness records latency per case but doesn't gate on it. Luna's slow tail was visible in the eval and I accepted it. I'd add a p95 check before trusting this for a user-facing switch.

And the biggest one: whether Mini's books are as right as Gemini's. Readers said "none match" on 26% of Mini searches against 22% for Gemini, and confirmed a match less often. The eval says the models are close. Voluntary feedback from the people who bothered to click can't settle it either way.

For now, every search goes to Mini. I'm watching that "none match" rate and the OpenAI usage page, and the next cheap model I try goes through the empty-answer check before it gets anywhere near a feature flag.