profile

Doug Turnbull

Don’t measure search with end-of-funnel conversions (daily search tip)

Published 5 days ago • 1 min read

When I worked at Shopify, the gold standard was GMV (the dollar amount in revenue). Naturally, that’s what we wanted to move with search A/B tests.

Seems sane. BUT it doesn’t actually isolate search per-se. A lot can happen:

  • Someone buys a $50K rolex on control randomly, destroying an A/B test
  • User checkouts happen rarely, add to carts occasionally, and search clicks frequently

For these experiments, You’d have to wait for very sparse confounders to even out. That might take months? Or maybe never as inventory + indices shift.

A better way:

  • Focus on signals in the search interaction (clicks + dwell)
  • Focus on events that give users value - but adjacent to search (add to cart from search UI, clear signals of interest, hovers, reading, scrolling, “read more”, subscribing, etc)
  • Study the relationship between search value and lagging metrics: retention and conversion

As metrics move away from search, danger of confusing a final sale with poor/good search quality only increases.

-Doug

​

​Events · Consulting · Training (use code search-tips)

You're subscribed to Doug Turnbull's daily search tips where I share tips, blog articles, events, and more. You can always manage your profile:

Doug Turnbull

Search, RAG, AI expert

I share search tips, blog articles, and free events I'm hosting about the search+retreval industry, vector databases, information retrieval and more.

Read more from Doug Turnbull

Have you been to a conversion-crazy site? It’s nuts. Their site screams at you. They probably have the modern version of the HTML blink tag. Popups everywhere just won't go away. Buy buy buy! It’s fun to go to a physical store when you can browse the shelves, talk to customer service, and get help. People avoid stores lacking information and only high pressure salespeople in your face. If your search stinks of pressure, users will retreat. They’ll stay on Google. They win precisely because...

1 day ago • 1 min read

My buddy John Berryman and I were nicely hosted by Hugo Browne-Anderson on the Vanishing Gradients podcast. We talked about how agentic search stands poised to be more disruptive to the Information Retrieval space than RAG. Check it out! Other upcoming events Tomorrow (free) Cheat at Search Essentials, Vector Search (free) ReasoningLayer.ai - symbolic reasoning with LLMs Tuesday (free) Cheat at Search Essentials - Search Evaluation (WTF is an NDCG?) Friday (free) Doug Turnbull + Daniel...

3 days ago • 1 min read

Youtube masterminded how to turn engagement into insights. Whether search or their feed, you can learn from how they learn from you! They: Create low-friction, sticky, addictive little interactions you do subconsciously on the surface. How often have you found yourself hovering over a video on your feed? Give you many actions to take on a video, even from search results themselves (bookmark, share, etc) Treat their monetization (ads) more as a guardrail. They know engaging you with a sticky,...

3 days ago • 1 min read