I share search tips, blog articles, and free events I'm hosting about the search+retreval industry, vector databases, information retrieval and more.
Remove term frequency from title fields (daily search tip)
|
Users want to know if a document is about the searched for terms. Search tech news articles for “iPhone” - relevance statistics like BM25 infer more occurrences in the article body means the article is more about iPhone. Human authors write more than paragraphs. They add titles and create section headings. Repeated occurrences of a term here rarely matter. There’s no difference between:
So today’s tip, remove the influence of term frequency when searching title fields. you can
-Doug Services: Training (use code search-tips) · Consulting You're subscribed to Doug Turnbull's daily search tips where I share tips, blog articles, events, and more. You can always manage your profile: |
Doug Turnbull
I share search tips, blog articles, and free events I'm hosting about the search+retreval industry, vector databases, information retrieval and more.
Over the next two weeks, Trey Grainger and I are hosting a free online conference, The Future of AI Search w/ talks from some of your favorite search people. The lineup Unreasonable Effectiveness of BM25 for Agentic Search by Hornet.dev founder Jo Kristian-Bergum Introducing Luxir: Can we build the fastest search engine? by Solr creator Yonik Seeley Building topk-embed-v1: frontier multi-vector embeddings by Marek Galovic, CEO Top K Can a Decision Model Think like a Search Engineer by David...
In a Jev reranker you ask Jev to give P(Yes) of the answer to a question like Is this document relevant to the query? And you rank by P(yes) It works! But... that's kinda boring. I think we can do better: we could also have an LLM generate the questions per query. IE query "desk for kids" has us evaluate products with criteria like Is it less than 36 inches? Is it a playful design? Does it explicitly say its designed for children? Then rerank this query using the sum of P(yes) of those...
Hey all, Just a note tomorrow I'll be giving a talk on High-Scale Agent Memory in Turbopuffer Its a first step into a larger world! Agent memory helps new sessions feel like they're continuing a conversation not restarting one. Come, have fun, see the mistakes I've made, and get an intro to Turbopuffer. Event link - https://maven.com/p/e48f16/high-scale-agentic-memory-with-turbopuffer Cheers, -Doug Slack Community * Events · Consulting · Training (use code search-tips) You're subscribed to...