profile

Doug Turnbull

I share search tips, blog articles, and free events I'm hosting about the search+retreval industry, vector databases, information retrieval and more.

Featured Post

What is agentic search? Nobody knows

Agentic search gets interesting when agents do not know how to find the right answer. Oh, the agent might think it knows. It might confidently BS us. But the agent’s poor domain intuition steers itself astray. Agents make false assumptions about what our users think is relevant. Our fashionista users think “red shoes” should return high-heels. When I worked at one company ABE wasn’t a president, it was an A/B testing tool. Agents need context to know these things - and context engineering...

Upcoming events in the next week or so Show us your skills w/ Hugo Bowne-Anderson Thursday May 28th - https://luma.com/ltpzpqgw Pray to the demo gods! I'll be joining Hugo Bowne-Anderson's "Show us your skills" event on Luma - highlighting using a coding agent to optimize search rankers.. Come hang out if you want to see how others in the industry leverage agentic AI to build in their domain. User search trends in 2026 Monday June 1 -...

At Haystack I spoke about autoresearch: Code generation to optimize search rankers. Can we use it to improve on BM25? This article represents my lab notes. My agent starts with a BM25 implementation, proposes changes, and accepts those that improve NDCG. We’ll zero-in on passage retrieval dataset MSMarco. I won’t claim I’ve found a “better BM25” but I’ve iterated towards a decent tuning regime. All while learning valuable lessons about how validation data can leak. Let’s walk through what...

Search events this week! Bag of Documents Model w/ Daniel Tunkelang Tuesday, 1PM ET - https://maven.com/p/7270ba/a-bag-of-documents-model-for-query-understanding-retrieval Tomorrow, Trey Grainger and I will host Daniel Tunkelang as he introduces his "Bag of Documents" technique for vector search. Bag of Documents is a form of pseudo-relevance feedback using document embeddings. Search week in Berlin June 7-12 2026 - https://berlinsearchweek.com/index.html A reminder in June: Berlin Buzzwords...

It's the final week before my Cheat at Search with Agents course. But we've got a fun tailgaite party before the big event 😀. Agent harnesses are dead - long live harnesses Today, 7PM ET - https://maven.com/p/6dc9ef/building-effective-agent-harnesses Search backend design means agent design. Hugo Bowne Anderson lives by a KISS ethos - Keep it Simple Smartypants. He talks real-life production LLM + harness design. You don't need to chase the latest OpenClaw -> Hermes -> Claude Code design...

Give an agent a set of search tools, it finds relevant products and improves result ranking. So should we throw away our traditional search stack and just let an agent drive some retrievers? Will the future of search APIs just be an agent, not query understanding or reranking? Here's the rub - finding things with agent's help differs from helping agents find information. In one case, the agent helps us. In the other, we must help the agent find what it doesn't know. This last case can't work...

Search community happenings for this week! 2026 is the year of agentic search w/ Jo Kristian Bergum Thursday April 30th - https://maven.com/p/a4f265/2026-will-be-the-year-of-agentic-search What's happening in Information Retrieval in 2026? Agentic search! This is THE topic everyone is focused on. Agents searching for us. Agents performing deep research. Agentic models like SID-1 focused on fast search (replacing your search API?). And so on and on. I'll be hosting a conversation with Jo...

Talks this week + other events. Hope you can make it and help keep this community awesome 😎 First - Leonie Monigatti will share how Context Engineering IS Agentic Search. Tuesday, 10:30AM ET What everyone is missing: when people talk aobut "context engineering", they really ought to be improving search. Leonie will cast aside myths about context engineering to show how agents build their own context via retrieval. And THAT, not prompting magic, decides whether your AI app is successful....

Late interaction is having a moment. The team at LightOn - including superstar developer Antoine Chaffin - has demonstrated how a 150M(!) late interaction model beats much larger models - some up to 8B parameters. David beats Goliath! Better search only cost you less! Tested on what dataset? BrowseComp. BrowseComp asks difficult questions requiring detailed, complex research. Tasks you can imagine agents chugging away, searching, getting frustrated and lost. Here’s an example prompt / answer...

How do teams choose vector databases / search engines? People wrack their brains between Elasticsearch/OpenSearch/Solr/Vespa/Pinecone/Turbopuffer/Weaviate/…? First things first - DO NOT start with a feature matrix. Start with the simple question: What is my team most comfortable with? That’s the default. If everyone can go deep in one system, don’t overcomplicate the decision. It might be good enough to stop here. NEXT - consider the high-level characteristics of the project. Use these as...