I share search tips, blog articles, and free events I'm hosting about the search+retreval industry, vector databases, information retrieval and more.
How to shrink an embedding
|
Almost all my clients face this issue. They want to build a RAG system. They have:
That's the recipe of nearly every speculative RAG project I've worked on. A mature search team stands up a vector database and can't tame the sheer scale of data they're facing. That drags project timelines out months while costs skyrocket. It's important to have tools to manage the scale. One often overlooked method: dimensionality reduction In this blog article, I share how to use PCA as one important, easy-to-use tool: https://softwaredoug.com/blog/2026/07/24/pca-shrink-ray -Doug PS I'm teaching a "Build your Own Vector DB" one-day, small workshop for anyone facing vector scaling conundrum. There's only 8 seats left. Hope to see you there. --- Slack Community * Events · Consulting · Training (use code search-tips) You're subscribed to Doug Turnbull's daily search tips where I share tips, blog articles, events, and more. You can always manage your profile: |
Doug Turnbull
I share search tips, blog articles, and free events I'm hosting about the search+retreval industry, vector databases, information retrieval and more.
Over the next two weeks, Trey Grainger and I are hosting a free online conference, The Future of AI Search w/ talks from some of your favorite search people. The lineup Unreasonable Effectiveness of BM25 for Agentic Search by Hornet.dev founder Jo Kristian-Bergum Introducing Luxir: Can we build the fastest search engine? by Solr creator Yonik Seeley Building topk-embed-v1: frontier multi-vector embeddings by Marek Galovic, CEO Top K Can a Decision Model Think like a Search Engineer by David...
In a Jev reranker you ask Jev to give P(Yes) of the answer to a question like Is this document relevant to the query? And you rank by P(yes) It works! But... that's kinda boring. I think we can do better: we could also have an LLM generate the questions per query. IE query "desk for kids" has us evaluate products with criteria like Is it less than 36 inches? Is it a playful design? Does it explicitly say its designed for children? Then rerank this query using the sum of P(yes) of those...
Hey all, Just a note tomorrow I'll be giving a talk on High-Scale Agent Memory in Turbopuffer Its a first step into a larger world! Agent memory helps new sessions feel like they're continuing a conversation not restarting one. Come, have fun, see the mistakes I've made, and get an intro to Turbopuffer. Event link - https://maven.com/p/e48f16/high-scale-agentic-memory-with-turbopuffer Cheers, -Doug Slack Community * Events · Consulting · Training (use code search-tips) You're subscribed to...