- LLMs with a grep or full-text search tool turn out to be great at fuzzy search already - they throw a bunch of OR conditions together and run further searches if they don't find what they want
- ChatGPT web search and Claude Code code search are my favorite AI-assisted search tools and neither bother with vectors
- Building and maintaining a large vector speech index is a pain. The vector are usually pretty big and you need to keep them in memory to get truly great performance. FTS and grep are way less hassle.
- Vector matches are weird. So you get back the top twenty results... those might be super relevant or they might be total garbage, it's on you to do a second pass to figure out if they're actually useful results or not.
I expected to spend much of 2025 building vector search engines, but ended up not finding them as valuable as I had thought.
On the other hand, generating and regenerating embeddings for all your documents can be time consuming and costly, depending on how often you need to reindex
Roughly, first there is the query analysis/manipulation phase where you might have NER, spell check, query expansion/relaxation etc
Then there is the selection phase, where you retrieve all items that are relevant. Sometimes people will bring in results from both text and vector based indices. Perhaps and additional layer to group results
Then finally you have the reranking layer using a cross encoder model which might even have some personalisation in the mix
Also, with vector search you might not need query expansion necessarily since semantic similarity does loose association. But every domain is unique and there’s only one way to find out