At 17:29 UTC on July 14, my coding agent told me this:

This was a false-positive alert, not a search outage.

I run WhatIsThatMovie, a search engine for when you remember the plot but not the title. A search alert had fired. In PostHog, the last completed UTC hour had zero movie_show_search events.

The agent checked the application code, PostHog, and Cloudflare. It found two calls to the search function during that hour. Both returned HTTP 200, as did the corresponding LLM generations. Searches started completing again during the next hour.

The agent called it a false positive, and the answer made enough sense that I almost stopped there.

I stared at the answer for a minute and typed this:

hmm
but compared to other days, same time range, i feel like there's less searches today?
also today in general seems like there's less searches?
is it related to traffic?

Thirty seconds later, the agent came back with a different answer:

You're right: today is dramatically lower.

The two HTTP 200s were still true. The mistake was deciding that they explained the alert.

The event was not a search attempt

movie_show_search sounds like it should fire whenever somebody searches for a movie. That is how I had been reading it. The code said otherwise.

Trimmed down to the branches that matter here, the search function looked like this:

const tmdbMatches = merged.slice(0, 10);

if (tmdbMatches.length === 0) {
	return { error: 'No matches found' };
}

const existing = await getSearchByQuerySlug({ db: mainDb, querySlug });
if (existing?.slug) {
	redirect(302, `/search/${existing.slug}`);
}

const [search] = await time(() =>
	createNewSearch({
		db: mainDb,
		data: {
			query: description,
			slug: `${generateId()}_${querySlug}`,
			querySlug,
			aiMatches: matches ?? [],
			tmdbMatches
		}
	})
);

emitMovieShowSearch({
	distinctId,
	data: {
		// Other timing and result properties omitted.
		searchId: search.id
	}
});

By the time the event fired, the title-matching and embedding work had finished, at least one TMDB match existed, the query had not reused an existing permalink, and the new search was already in the database.

So this event counted new persisted searches. It did not count every attempt.

Zero could mean the endpoint was unavailable. It could also mean there was no traffic, the searches returned no matches, every query reused an existing permalink, something failed before the database write, or the PostHog event stopped being captured.

Looking at the emission point was useful. The agent did not immediately treat zero as proof of an outage. Its first update said it would treat the number as "a symptom to validate, not yet proof of an outage." It understood what the event counted, then stopped one comparison too early.

Why the first diagnosis looked right

There were two $ai_generation events in the alert hour. Both had an AI HTTP status of 200. Cloudflare recorded two matching POST requests to searchMoviesOrShows, and both returned HTTP 200.

The requests happened at 15:19:31 and 15:20:03 UTC. The PostHog AI events followed at 15:19:33 and 15:20:04. The timestamps lined up almost exactly.

Those facts ruled out a complete search outage for the two requests we could see. They also made an LLM provider failure unlikely for those requests.

The two attempts were repeats of the same vague quote. Neither produced a new persisted search. No result or a reused permalink was a plausible explanation.

That should have ended as a limited conclusion: the two observed search requests did not fail. Instead, the agent jumped from there to "the alert was a false positive." It had ruled out one explanation without finding the reason completed searches reached zero.

I had handed the investigation to the agent, and I was about to accept its answer. That part is on me too.

The comparison that changed the answer

My "hmm" added no new production data. It asked the agent to compare the hour with a useful baseline.

The agent compared the incident hour with the previous day, the same weekday one week earlier, and the following day after traffic recovered.

UTC window Pageviews Unique visitors AI generations New persisted searches
July 7, 15:00 307 173 129 110
July 13, 15:00 365 203 118 108
July 14, 15:00 22 18 2 0
July 15, 15:00 285 164 98 96

The incident did not look like a normal hour with two unlucky searches. Pageviews had fallen from 365 to 22 compared with the previous day. AI generations had fallen from 118 to 2. The same weekday comparison said the same thing, and the following day returned close to the earlier range.

Almost everything upstream of the conversion event had collapsed too. At that point, calling the alert a false positive no longer made sense.

PostHog gave me the product view. I still wanted an independent signal, because an analytics instrumentation problem can make several events disappear together.

Cloudflare's zone analytics showed 13,903 edge requests during the incident hour. The previous day's equivalent hour had 27,629. Cloudflare collects traffic independently of PostHog, and it saw the drop too.

The absolute Cloudflare and PostHog numbers should not match. Cloudflare includes bots, static assets, and requests that never become a product event. I only cared that both systems saw the drop.

Starting with the search endpoint would have sent me in the wrong direction. The next question was where the traffic had gone.

That was as far as the data went. Cloudflare did not prove it was Google, the ads, ConvertBox, or a deployment. I had evidence of a traffic incident, but not its cause.

A second event tried to mislead me

There were 238 PostHog http_request events during the incident hour. At first, that number made the traffic explanation look shaky.

My instrumentation only emitted http_request for non-2xx responses. Successful requests were skipped:

if (opts.status >= 200 && opts.status < 300) return false;

The 238 events were redirects and errors, not a count of healthy Worker traffic.

I needed Cloudflare logs for the two successful search responses and zone analytics for total edge traffic. PostHog was useful for product behavior, but it was not the source of truth for whether the Worker received a request.

I have made this mistake before: reading the event name as the metric definition. The definition lives where the event is emitted. If I skip that code, I end up debugging the name instead of the behavior.

The matrix I wish the agent had used

The alert hour had several plausible explanations before the comparison. I wish the agent had kept a table like this while it worked:

Hypothesis What zero completions says What the other evidence says
Search endpoint outage Compatible Weakened. The two observed search POSTs returned 200.
LLM provider failure Compatible Weakened. Both observed AI generations returned 200.
No match or reused permalink Compatible Still possible for the two attempts, but it does not explain the missing pageviews.
PostHog instrumentation failure Compatible Weakened. Product events fell together, and Cloudflare independently saw less traffic.
Traffic collapse Compatible Best fit. Pageviews, AI generations, completed searches, and edge requests all fell.
Cause of the traffic collapse Says nothing Still underdetermined. The data did not identify the source.

The last row is where I want the agent to stop. A traffic collapse was the best fit. Its cause was still unknown. A useful answer should be able to say both without filling in the missing part.

The rule I added to the process

I keep PostHog and Cloudflare investigation runbooks in a public knowledge base now. I wrote recently about why the process has to survive the session. This incident is one of the reasons.

After this incident, I added an order for this kind of investigation:

  1. Find the event emission point before calling it an attempt, success, or failure.
  2. Compare the incident with the previous day and the same weekday before classifying it.
  3. Add an upstream product signal and an independent infrastructure signal.
  4. For every new signal, say which hypotheses it weakens and which ones remain.
  5. If more than one explanation still fits, say the incident is underdetermined.

Everything in the first answer came from real data. The agent found the right code, queried the right PostHog project, and matched two Cloudflare requests to two AI generations. The failure was the jump from two successful requests to a false-positive alert.

When I asked for the same-hour comparison, the answer changed in thirty seconds. I want that comparison to happen before I have to type "hmm" next time.