profile

Doug Turnbull

What AI teams screw up

Published about 2 months ago • 1 min read

My work is split between mature search teams and new AI teams.

Search teams are often farther along and manage a mature, traditional search product. AI teams, however, often don't know what they don't know yet. They've just been cobbled together, have built a few agent demos, and are often in the process of discovering the three big mistakes I blog here:

  1. Evals+measurement need to dominate a lot of your product thinking
  2. Retrieval isn't "one thing" (ie classic RAG) - its extremely custom to your domain, it's the main engineering problem to solve
  3. Context isn't about giving chunks to LLMs, it's about metadata needed to help the LLM make better decisions (in classic search its the humans processing what comes back, but the same idea)

More in my blog - https://softwaredoug.com/blog/2026/08/29/ai-team-mistakes​

-Doug

Starts Monday: Building Production Agents course w/ me and Hugo Bowne-Anderson

As a reminder, next week Hugo Bowne Anderson and I are teaching a class aimed at bridging the AI -- Search gap. A single course for AI builders to learn how to create batteries-included agents that search. Similarly a course for search people to get an in-depth point of view on agents.

Sign up here - https://maven.com/softwaredoug/build-enterprise-agents​

​Slack Community * Events · Consulting · Training (use code searchtips)

You're subscribed to Doug Turnbull's daily search tips where I share tips, blog articles, events, and more. You can always manage your profile:

Doug Turnbull

Search, RAG, AI expert

I share search tips, blog articles, and free events I'm hosting about the search+retreval industry, vector databases, information retrieval and more.

Read more from Doug Turnbull

Over the next two weeks, Trey Grainger and I are hosting a free online conference, The Future of AI Search w/ talks from some of your favorite search people. The lineup Unreasonable Effectiveness of BM25 for Agentic Search by Hornet.dev founder Jo Kristian-Bergum Introducing Luxir: Can we build the fastest search engine? by Solr creator Yonik Seeley Building topk-embed-v1: frontier multi-vector embeddings by Marek Galovic, CEO Top K Can a Decision Model Think like a Search Engineer by David...

8 days ago • 1 min read

In a Jev reranker you ask Jev to give P(Yes) of the answer to a question like Is this document relevant to the query? And you rank by P(yes) It works! But... that's kinda boring. I think we can do better: we could also have an LLM generate the questions per query. IE query "desk for kids" has us evaluate products with criteria like Is it less than 36 inches? Is it a playful design? Does it explicitly say its designed for children? Then rerank this query using the sum of P(yes) of those...

15 days ago • 1 min read

Hey all, Just a note tomorrow I'll be giving a talk on High-Scale Agent Memory in Turbopuffer Its a first step into a larger world! Agent memory helps new sessions feel like they're continuing a conversation not restarting one. Come, have fun, see the mistakes I've made, and get an intro to Turbopuffer. Event link - https://maven.com/p/e48f16/high-scale-agentic-memory-with-turbopuffer Cheers, -Doug Slack Community * Events · Consulting · Training (use code search-tips) You're subscribed to...

23 days ago • 1 min read