BM25: probabilistic but not a probability (daily search tip)

Published 6 days ago • 1 min read

BM25 models the odds a term would be observed in a relevant document (vs the term occurring in an irrelevant doc).

It’s based on probabilistic relevance, capturing:

t - a query term match occurs
R - the doc is relevant

Queries of course contain multiple terms. How do we combine those odds? The odds of BOTH terms being in a relevant doc, we’d need to multiply Odds(t1) * Odds(t2).

If we take the log of these multiplied odds, we can take advantage of a property of logarithms: log(Odds(t1) * Odds(t2)) = log(Odds(t1)) + log(Odds(t2)). Or taken together across each term:

BM25 tries to model this!

How?

We can imagine cases where a term match would occur in a relevant doc (the numerator). Why might a term match occur an in an irrelevant doc?

The term’s ambiguous, not specific to the user’s intent. luke skywalker - luke occurs all over the place. It’s common. The rare term skywalker occurs only in only Star Wars → thats how we get Inverse Document Frequency.
The document is not ABOUT that term - A term occurring spuriously (low term frequency) would not mean the document is relevant. Or a term not being a large enough proportion of the text: Skywalker being mentioned a few times in a book length tome may just happen accidentally.

Through trial, error, and experimentation with open datasets we arrived at the exact BM25 formula.

That’s how we get a BM25 that’s probabilistic, but not a probability

My AI Powered Search training with Trey Grainger starts THURSDAY - signup here

-Doug

Events · Consulting · Training (use code search-tips)

You're subscribed to Doug Turnbull's daily search tips where I share tips, blog articles, events, and more. You can always manage your profile:

Blog post - can BM25 be a probability?

Reviewing Bayesian BM25 - a new approach to creating calibrated BM25 probabilities for hybrid search. I talk about this vs naive approaches I've used to do similar things. Enjoy! https://softwaredoug.com/blog/2026/03/06/probabilistic-bm25-utopia -Doug Events · Consulting · Training (use code search-tips) You're subscribed to Doug Turnbull's daily search tips where I share tips, blog articles, events, and more. You can always manage your profile:

1 day ago • 1 min read

BM25’s other parameter - WTF is k3? (daily search tip)

You may know BM25 lets you tune two parameters: k1: how quickly to saturate document term frequency’s contribution b: how much to bias towards below average length docs What you may NOT know is there is another parameter k3 What does k3 do? It handles repeated query terms. Old papers suggest k3=100 to 1000, which immediately saturates. That’s why Lucene ignores k3. It just uses the query term frequency. Some other search engines like Terrier set it to 8. So for the query, “Best dog toys for...

1 day ago • 1 min read

High IDF doesn't always mean relevant search (daily search tips)

Rare terms have high inverse document frequency (IDF). BM25 scoring treats high IDF terms as more relevant. Why? We assume if a term occurs rarely in the corpus, it must unambiguously point to what the user wants. It’s specific. But that’s not always true. Not all text is created equal. Corpuses violate this assumption frequently. Why? No need to use a common term - Book titles may rarely mention the word “book”, but clearly “book” in a book index has low specificity. Language gaps between...

4 days ago • 1 min read