What AI teams screw up

Published about 1 month ago • 1 min read

My work is split between mature search teams and new AI teams.

Search teams are often farther along and manage a mature, traditional search product. AI teams, however, often don't know what they don't know yet. They've just been cobbled together, have built a few agent demos, and are often in the process of discovering the three big mistakes I blog here:

Evals+measurement need to dominate a lot of your product thinking
Retrieval isn't "one thing" (ie classic RAG) - its extremely custom to your domain, it's the main engineering problem to solve
Context isn't about giving chunks to LLMs, it's about metadata needed to help the LLM make better decisions (in classic search its the humans processing what comes back, but the same idea)

More in my blog - https://softwaredoug.com/blog/2026/08/29/ai-team-mistakes

-Doug

Starts Monday: Building Production Agents course w/ me and Hugo Bowne-Anderson

As a reminder, next week Hugo Bowne Anderson and I are teaching a class aimed at bridging the AI -- Search gap. A single course for AI builders to learn how to create batteries-included agents that search. Similarly a course for search people to get an in-depth point of view on agents.

Sign up here - https://maven.com/softwaredoug/build-enterprise-agents

Slack Community * Events · Consulting · Training (use code searchtips)

You're subscribed to Doug Turnbull's daily search tips where I share tips, blog articles, events, and more. You can always manage your profile:

Tomorrow: Agent Memory in Turbopuffer

Hey all, Just a note tomorrow I'll be giving a talk on High-Scale Agent Memory in Turbopuffer Its a first step into a larger world! Agent memory helps new sessions feel like they're continuing a conversation not restarting one. Come, have fun, see the mistakes I've made, and get an intro to Turbopuffer. Event link - https://maven.com/p/e48f16/high-scale-agentic-memory-with-turbopuffer Cheers, -Doug Slack Community * Events · Consulting · Training (use code search-tips) You're subscribed to...

7 days ago • 1 min read

This week: Embeddings don't solve RAG

This week the Retrieval Augmented Gathering online conference continues! Last week we explored index-only SPLADE, got a demo of tensors in the search engine, and how to think of agentic memory as telemetry. This week we've got talks below: Embeddings don't solve RAG (today) Solving retrieval for agents isn't as simple as chunk + embed. For one, similarity is not relevance. An decides what's relevant depending on not just raw text, but everything that 'surrounds' it. Places it in context....

9 days ago • 1 min read

Cheating at Search w/ Jev

I've spent the last several years building query classifiers with LLMs. Now Jev has arrived, making this process simpler, faster, and more accurate. I decided to sit down with a simple query understanding task with Jev. It let me make better tradeoffs in recall / precision, while being an order of magnitude faster / cheaper Enjoy: https://softwaredoug.com/blog/2026/09/22/jev-query-understanding.html -Doug Slack Community * Events · Consulting · Training (use code search-tips) You're...

14 days ago • 1 min read