I've spent the last few years watching embedding technologies transform from Big Tech's "secret sauce" into everyday developer tools. What happened next—the vector database gold rush, the RAG hype cycle, and the eventual correction—taught us valuable lessons about how new technologies find their place in the broader ecosystem.
Embeddings go mainstream
What was once the domain of tech giants is becoming standard practice for developers everywhere. For over a decade, companies like Google, Meta, and Amazon have used embedding techniques to power recommendation systems and search features at a massive scale. These techniques are now accessible to developers through pre-trained models and improved tooling.
Through deep learning, we can transform virtually any type of content—text, images, video, audio, code—into vector representations that capture patterns and relationships in the data. While these techniques have deep roots in representation learning research, what's revolutionary today is their accessibility. Powerful pre-trained models and intuitive APIs have transformed what was once complex research into practical tools for everyday developers.
These embeddings enable developers to build features that were previously restricted to tech giants with massive R&D budgets. The embedding ecosystem is thriving on multiple fronts. Commercial providers like OpenAI, Cohere, Jina, and Voyager offer high-quality hosted models (mainly text), while the open-source community has produced powerful alternatives like Sentence Transformers. HuggingFace has become the de facto platform for sharing and discovering these models, with thousands of embedding models available for different languages, modalities, domains, and tasks. Their transformers library and model hub make experimenting with different embedding approaches trivial.
The result? Methods that were once the exclusive domain of advanced ML teams in large tech companies are now being integrated into everyday applications. Whether using commercial APIs or open-source models, developers can choose the approach that best suits their needs regarding cost, customization, and deployment requirements. Content that was previously difficult to search effectively—videos, podcast transcripts, technical diagrams, scientific papers—can now be represented in ways that enable more sophisticated (and more effective) retrieval systems.
The rise and fall of vector databases
The explosion of embedding-based applications created a new challenge: efficiently storing, indexing, and searching these high-dimensional vectors at scale. This gap gave rise to the vector database category, with companies like Pinecone leading the charge in 2022-2023 by defining specialized infrastructure for vector operations. The category saw explosive growth following ChatGPT's launch in late 2022, as developers rushed to build AI applications using Retrieval-Augmented Generation (RAG). This surge was partly driven by a widespread misconception that embedding-based similarity search was the only viable method for retrieving context for LLMs. The resulting "vector database gold rush" saw massive investment and attention directed toward vector search infrastructure, even though traditional information retrieval techniques remained equally valuable for many RAG applications.
However, the landscape has evolved rapidly. What started as pure vector search engines now expand their capabilities to match traditional search functionality. Vector database providers have recognized that real-world applications often require more than just similarity search. They're increasingly adding features like filtering, faceting, and text search capabilities that were previously the domain of conventional search engines.
This convergence reflects a more profound understanding that vector search alone is insufficient for practical, real-world use cases. No example better illustrates this convergence than Elasticsearch's evolution. As a leader in the search market, Elastic recognized the potential of embedding-based search and moved to incorporate these capabilities. In 2024, they repositioned Elasticsearch as "a search engine with a fully integrated vector database," acknowledging that modern search applications require traditional text-based and vector operations. This shift wasn't just marketing - Elasticsearch enhanced its vector search implementation with optimized indexing structures and added support for efficient approximate nearest neighbor (ANN) search algorithms. By bringing vector search capabilities directly into their search platform, Elastic demonstrated that the future of search lies in unified solutions that blend traditional search features like relevance scoring and text analysis with embeddings. Their approach validated what many in the industry had begun to recognize: vector search isn't a separate category but an essential capability in the modern search toolkit.
Meanwhile, established database vendors watched this gold rush with barely concealed amusement. Rather than treating vector search as a separate category, many shrugged and integrated vector capabilities directly into their existing products. PostgreSQL, MongoDB, Redis, and others have added native support for vector data types and similarity search operations - treating it with all the excitement of adding a new index type. From their perspective, vectors are just another data type to index and query, similar to text, numbers, or geospatial coordinates. This integration simplifies architecture for many applications, allowing developers to manage vector operations alongside traditional data workloads in familiar systems without convincing their CTO to adopt another specialized database.
However, adding a vector type to an existing database isn't quite the silver bullet it might appear. While storing and retrieving vectors is one thing, building a sophisticated retrieval system is another. Many of these databases lack refined ranking mechanisms, relevance tuning, and battle-tested text-matching algorithms (like BM25), which dedicated search engines have spent decades perfecting. This is why companies that care about search quality still turn to dedicated search engines rather than treating their database as a one-size-fits-all solution. The reality is that high-quality information retrieval requires a deep toolbox of ranking strategies and relevance optimization features that go beyond simple vector similarity calculations. Just as nobody who cares about search quality would use a regular database to power their e-commerce search, adding vector capabilities doesn't suddenly transform an SQL database into a complete retrieval engine.
Conclusion
Looking back, it's clear we overcomplicated things. While embeddings fundamentally changed how we can represent and compare content, they didn't need an entirely new infrastructure category. What we label as "vector databases" are, in reality, search engines with vector capabilities. The market is already correcting this categorization—vector search providers rapidly add traditional search features while established search engines incorporate vector search capabilities. This category convergence isn't surprising: building a good retrieval engine has always been about combining multiple retrieval and ranking strategies. Vector search is just another powerful tool in that toolbox, not a category of its own.