How we selected the next vector database at Booking.com

This work was done in collaboration with Klaus Schaefers.

Over the past few years, embeddings and vector search have become an important capability in many of our machine learning and GenAI systems at Booking.com. We initially started with a handful of use cases and experiments, and later this capability has grown into shared infrastructure that powers similarity search, semantic filtering, and retrieval-augmented generation (RAG).

We used to treat vector search as a backend implementation detail, but today it directly drives the user experience. The real win isn’t only speed but also the context. Expanding the variety of domain data we can retrieve efficiently gives our system the depth of context it needs to deliver accurate, and personalized experiences across the platform.

This makes selecting the underlying vector database an infrastructure decision similar to choosing a primary datastore or message queue. It has to be predictable and scalable.

As more teams started using our vector store, we began seeing highly diverse functional and operational requirements across different use cases. Some teams needed advanced capabilities like hybrid search or multi-vector support, while others demanded larger vector capacities and higher RPS metrics. These architectural needs ultimately brought us to a point where we needed to reassess whether our current setup could support this next phase of growth.

Context: how embeddings fit into our stack

Embeddings are vectors: fixed-length arrays of numbers produced by a model to represent an item (text, image, etc.). Each vector can be seen as a point in a high-dimensional space, where distance (or similarity) between points approximates semantic relatedness. By searching for the nearest vectors to a query vector, we retrieve items that are semantically “similar”.

This simple mechanism enables a wide range of use cases for us due its ability to do semantic similarity search. RAG-based use cases are the most well known examples. Another example is our Partner to guest messaging agent use case which can be found in this article.

Across most workloads, the requirement is the same: fast k-nearest-neighbour search over large collections of vectors, often combined with metadata filtering and high concurrency.

In other words, the storage layer matters as much as the model.

Our starting point: OpenSearch

When we first launched our internal Embedding Service as a vector database, we chose OpenSearch with vector support as the backend.

This was largely a pragmatic decision. OpenSearch was already adopted inside the company, available in AWS, and easy to provision with Terraform configurations. It allowed us to move quickly and onboard the first embedding-based use cases without having to evaluate, not introduce new technologies or operational overhead.

While this setup provided the agility we needed to validate our platform’s value early on, it eventually became clear that we were outgrowing OpenSearch’s capabilities as our workloads intensified.

The pain at scale

As more teams adopted the service, the workload profile changed significantly. Datasets grew to hundreds of millions of embeddings. Queries became more complex, often combining vector search with filtering. Concurrency increased. At the same time, latency expectations tightened, particularly for user-facing applications.

Under these conditions, we encountered the limitations of using a general-purpose search engine for vector-heavy workloads. Achieving acceptable performance required constant tuning and increasingly large clusters. Operational overhead grew, and costs spiked as we scaled out hardware to compensate for the engine’s inherent limitations with vector search at scale.

While nothing was fundamentally broken, it was getting harder to operate, slower to evolve, and more expensive. We were having to spend more time managing the AWS OpenSearch infrastructure than focusing on enabling the high-value use cases built on top of it.

This is when we began seriously considering specialized vector databases.

Verifying if generic benchmarks apply

Once we started exploring alternatives, we found plenty of public benchmarks and comparison posts. At first glance they seemed helpful, but most of them used small datasets, synthetic workloads, or lower levels of concurrency. Our reality looks very different.

We operate on large embedding datasets, combine search with filtering, and serve highly concurrent workloads. At this scale, small differences in latency or efficiency (the CPU, RAM, and cluster size required to meet our target recall and tail-latency) have a significant impact on both user experience and cost.

Because of this, we decided not to rely on external benchmarks. Instead, we chose to design an evaluation that mirrors our production workloads as closely as possible. The goal was not to identify the system with the best marketing numbers, but the one that behaves most reliably under our conditions. Your mileage may vary

How we conducted the evaluation

We defined the requirements based on lessons learned from operating our existing system and supporting production workloads at scale. Beyond raw performance, we considered operational maturity, deployment flexibility, cost predictability, and integration with our ML ecosystem. In this article, we detail that evaluation process, specifically contrasting the capabilities of our selected vendor against our legacy OpenSearch implementation.

Dataset

For performance testing, we constructed a dataset containing 100M embeddings based on Amazon Reviews 2023 dataset. We processed millions of records across categories like Electronics, Baby Products, and Home & Kitchen.

To transform this text into vectors, we used the all-MiniLM-L6-v2 model. We chose it because it was already available within our internal model catalog, making it representative of models used by existing production workloads. It also provides a good balance between embedding quality and inference speed, allowing us to generate a dataset of this scale efficiently.

In addition to the vectors, we stored metadata for each record, such as ratings and custom tags, to evaluate filtered KNN search performance. This allowed us to replicate the complex, real-world queries our AI applications face every day, ensuring our chosen vendor could handle high-dimensionality data alongside complex metadata filtering.

Workloads

Full a-KNN search

Each request performs an approximate k‑nearest‑neighbour (ANN / a‑KNN) search across the entire dataset (~100M embeddings). Because ANN methods trade retrieval accuracy for speed, we tuned each system to a comparable target recall (~0.99) so latency and throughput results reflect system efficiency rather than lower-quality retrieval.

Filtered a-KNN search

Queries combine vector similarity with metadata filtering (e.g., rating ranges, category filters, or high-cardinality tags). This is closer to real usage, where retrieval is rarely unconstrained and often requires prefiltering the search space restricting the candidate set by metadata before running ANN and ranking by vector similarity.

Mixed read/write workload

In addition to search queries, we introduced a continuous ingestion stream (~20% of total operations) to simulate real-time updates. The benchmark used a closed-loop workload, where each client issued the next request immediately after receiving a response. This allowed us to evaluate how query latency behaves under concurrent indexing pressure.

To understand how systems behave under load, we scaled the workloads along multiple dimensions:

Concurrency: we increased the number of parallel clients (5 → 50 → 100 threads) issuing requests

Request rate (RPS): higher concurrency naturally drove higher throughput, pushing the systems toward saturation

Dataset size: all workloads were executed on the full dataset (~100M vectors) to avoid small-scale bias

Across all scenarios, we measured latency per request, overall throughput, resource utilization, and stability. Particular attention was paid to tail latency (P99), as this most closely reflects user experience.

Measuring retrieval quality (recall)

Since ANN is approximate, we validated that performance differences were not caused by returning lower-quality results. We randomly sampled approximately 2 million vectors already present in the index and used each as a query. We then measured Recall@k by checking whether the query vector itself appeared in its expected top-k nearest neighbours. For each system, we tuned the ANN search parameters (e.g., search breadth) to achieve a comparable target recall of approximately 99%, ensuring that the comparison primarily reflected differences in latency, throughput, and operational characteristics rather than retrieval quality.

Our existing OpenSearch setup served as the baseline for comparison, allowing us to quantify improvements relative to what we were already running.

During this phase, we evaluated several specialized vendors and ran the same workloads against each of them under comparable conditions. We will only share the results from OpenSearch and our chosen vendor Weaviate.

The Feature Comparison: Beyond Raw Speed

Performance was our primary driver, but a database is only as good as its operational “survivability.” We compared our existing OpenSearch setup against the specialized vendors on several critical dimensions.

The Performance Showdown

Results

The following results compare the systems across our production-inspired workloads. We focus on P99 latency under comparable recall targets, as well as resource utilization and cost, to understand how each system behaves under realistic operating conditions.

P99 Latency

Across our tests, a consistent pattern emerged. Dedicated vector databases handled our workloads more efficiently than OpenSearch. They delivered lower tail latency, better throughput under concurrency, and more predictable scaling characteristics, often with fewer resources. That efficiency translated directly into cost: at comparable recall and SLO targets, we observed roughly a 40% reduction in usage cost ( (compute/memory footprint) compared to our OpenSearch baseline.

This confirmed something intuitive: systems built specifically for vector search behave differently from general-purpose search engines with vector capabilities added on.
Among the evaluated options, Weaviate showed the most consistent performance across our scenarios, so we selected it as the new backend for the Embedding Service.

An architectural decision we had made earlier significantly simplified the migration. All database access was already abstracted behind our internal service, so clients never interacted directly with the underlying storage. This allowed us to run multiple databases in parallel and switch backends transparently. For most teams, migration required little more than a configuration change.

Lessons learned

Looking back, the most important takeaway from this process is not about any specific vendor or feature. It is about ownership of the evaluation itself.

At large scale, workloads are highly specific, and generic benchmarks rarely reflect real-world behavior. Running our own production-shaped tests gave us confidence that the system we chose would hold up under the conditions that actually matter to us.

Vector search has become foundational infrastructure for modern ML and AI systems. Treating it with the same rigor as any other critical datastore, testing realistically, abstracting the storage layer, and optimizing for stability rather than peak numbers, made a significant difference for us.