The last two things I wrote were about an eval set on whatisthatbook.com: how it caught a bug in itself, and the two upgrades it killed. Both of those upgrades failed for the same reason. They asked the model to do more of what it was already doing.

Re-ranking candidates by plot-summary similarity was the model's own confidence in a new outfit. Multi-query expansion was the model rephrasing its own guess and searching on the rephrasings. Neither brought in a single fact the model didn't already have. The eval killed both, and the lesson under both was the same: you can't fix a recall problem by asking the thing with the bad memory to try harder.

So I stopped asking it to remember.

The task was never "know every book"

whatisthatbook takes a fuzzy description and names the book. For months I treated that as a memory test for the LLM. Describe the book, the model recalls the title, done. And for famous books it works, because famous books are all over the training data.

The long tail is where it falls apart. Obscure books, foreign originals, recent releases, the half-remembered childhood book with a blue cover and a horse on it. The model has never seen those, or saw them once, and it either whiffs or confidently invents a title that doesn't exist.

I kept walking past something obvious. Somebody has already answered almost every one of these questions. There's a subreddit, r/whatsthatbook, where people post exactly these descriptions and other people reply with the answer. Thousands of solved threads. It is, structurally, a database of (fuzzy description → correct book) pairs. The exact thing my product does, already done, by humans, for free.

That was the idea. The model doesn't need to remember the book. It needs to find the thread where someone else remembered it.

What I built

A fallback lane. When the normal pipeline returns fewer than three verified results (the long-tail case, which turns out to be about half of all searches), I run the query through a Google search via DataForSEO, keep only the results from an allowlist of book sources (the relevant subreddits, Goodreads, Wikipedia, a few more), pull the top comments off any Reddit thread that comes back, and hand all of that text to the model with a much smaller job: read these snippets and pull out the book titles. Those candidates run through the same OpenLibrary verification as everything else.

The model is still in the loop. But its job changed completely. It went from "recall the right book out of everything you've ever read" to "extract the title from this Reddit thread that already contains the answer." The first is a memory test it fails on the long tail. The second is reading comprehension, which it's extremely good at.

There's a guard worth mentioning, because I thought it was the boring part that made the whole thing safe. After the model extracts a title, I check that the title actually appears, verbatim, in the snippet text. If it doesn't, I drop it. That kills the failure mode where the model, handed a Reddit thread, ignores it and reaches for a famous title from memory anyway.

It also turned out to kill real answers. More on that in a minute.

The three queries that sold me

Before building it out, I tested the idea by hand on three production queries the LLM-only path had lost.

"Girls raised in a compound, dystopian, on an island." The model had nothing. The search returned the r/whatsthatbook thread plus an Amazon page: Gather the Daughters.

"Big brother watches ppl." Four words. The search returned the Wikipedia article for Nineteen Eighty-Four; the title was right there in the result.

"Boy wakes up fifty years early on a spaceship, programming glitch." This is the one that sold me. The search returned the exact Reddit thread, and the top comment, 160 upvotes, named it: Sleepers, Wake. I care about this case more than the other two because months earlier I'd built an 862,000-book vector search index specifically to catch queries like this, and it had missed this one. A weekend of embedding the better part of a million books, beaten by one Reddit comment a stranger wrote years ago.

Three for three on the cases the model couldn't do alone.

Why not Exa, and what it costs

I looked at Exa first, which does neural search over the web and is built for roughly this. It works. It's also about six times more expensive per query, and its curated index doesn't preferentially route to r/whatsthatbook, which for this product is the single most valuable source on the internet. I didn't want clever retrieval over a smaller corpus. I wanted plain Google results that happen to surface the subreddit where the answer already lives.

So it's a Google SERP through DataForSEO at $0.002 a query, wrapped through my own naps.sh, firing on maybe half of searches. Order of ten to twenty dollars a month at my volume. The whole thing is cheaper than the embedding experiment it replaced, and it doesn't need a single GPU.

What production did to this story

Then I turned the feature flag up to 50%.

On the first full day, the fallback ran 223 times across roughly 78 users. It returned books 27% of the time. A little over half of the runs, 122 of 223, ended with no_candidates.

There were 40 confirmed results in that cohort. Every one came from the normal LLM path. The web-grounded lane contributed zero.

Three for three was enough to make me build it. It wasn't enough to prove it worked.

The Google search itself wasn't the problem. Successful and failed runs both returned around ten results. They took about the same time to reach extraction. Query length was almost identical.

What changed? Reddit.

When the lane had a fetched Reddit thread, 47% of runs returned candidates. Without Reddit text, that fell to 6%. The other allowlisted sources, Goodreads, Wikipedia, Amazon, contributed only the short description from the Google result. Then my safety guard checked whether the extracted title appeared verbatim in that thin pile of text. Usually it didn't, so I dropped it.

The boring guard I was proud of had become the recall bottleneck. It worked when I had a full Reddit thread. It deleted almost everything when I had two sentences from Google.

While tracing that, I found another failure one layer down. In one production search the LLM returned four candidates. OpenLibrary answered with a mix of 403s, timeouts, and connection resets. Four books went into verification. Zero came out.

I added a persistent cache around OpenLibrary searches and Google Books as a fallback. I also let the app render candidate cards when OpenLibrary was unavailable. I had spent weeks trying to improve what the model remembered while a flaky HTTP request was quietly throwing its answers away.

Even the bad rollout numbers were flattering. When the LLM returned nothing and the grounded lane also recovered nothing, the code returned early before sending the PostHog event. The complete failures were the ones I couldn't see.

The exact Reddit thread was still useful. The pipeline around it wasn't. I had proved that Google could find the right source. I had not proved that my code could carry the answer all the way to the user.

Then the model beat all my retrieval systems

A few weeks later I built the opposite version: no model memory, just local retrieval over 1.86 million OpenLibrary descriptions.

Plain full-text search found the right book in its first five results on 21% of the 95-case eval. Dense vector retrieval got 20%. Combining both candidate sets reached 29%. The production pipeline, with the LLM naming candidates and a local copy of OpenLibrary resolving them, reached 68% on the same cases.

Sixty-eight percent. More than twice the best retrieval-only result.

An awkward number for a post called "I stopped asking the model to remember."

The model's memory was carrying the product for well-known books. The web lane still made sense for the long tail, where the model had nothing useful to recall. But it was a fallback. I had been treating it like a replacement.

The local OpenLibrary run removed the flaky API from the measurement too. Once the resolver stopped timing out, the normal pipeline looked much better. Some of the failures I had blamed on the model were HTTP failures several steps later.

The lesson

For a year, my instinct with every LLM weakness was to improve the LLM. Better prompt, bigger model, more few-shots, embeddings, re-ranking. All of it aimed at making the model know more or judge better. The eval killed most of those attempts, because most of them were the model grading its own homework.

The model is good at remembering famous books. Reddit is good at remembering the weird ones somebody asked about eight years ago. I need both.

The mistake was treating every empty result as the same failure. Sometimes the model didn't know. Sometimes Google found the answer and my guard deleted it. Sometimes OpenLibrary fell over after the model had already done its job.

Find where the answer already lives.

Then make sure your own code doesn't lose it.