VDBBench Leaderboards | Zilliz

2 min read Original article ↗

Dataset Note:

  1. For "Single tenant LAION 100M" below, we refer to the "Laion 100m" (Click to copy S3 URI) Dataset (100M * 768 float dense vectors).
  2. For "Multitenant Cohere 10M" Dataset below, we refer to the "Cohere Large" (Click to copy S3 URI) Dataset (10M * 768 float dense vectors). All data are randomly splitted across 1000 tenants.
Workload
Cost basis

Price order

Cost vs. QPS ParetoQuery only · lower is better · USD / hour

Notes:All tests are conducted in AWS us-west-2. All costs shown here are based on each product's charges in this region.

InsertedSearchableFully IndexedWrite Cost

Zilliz Cloud Capacity 12CU

Notes:For exactly how we defined "inserted", "searchable", "fully indexed", please check VectorDBBench source code for each client. For basic intuition:

Mode
Filter
Payload
Latency

Max concurrency P99 latency

Vector Search Latency and QPSunfiltered · ids only · Max concurrency P99

Zilliz Cloud Capacity 32CU

2,000 bytes/query

recall@10 0.9728

Turbopuffer

2,000 bytes/query

recall@10 0.9321

Zilliz Cloud Capacity 12CU

2,000 bytes/query

recall@10 0.9723

Turbopuffer Pinned

2,000 bytes/query

recall@10 0.9321

Zilliz Cloud Tiered 4CU

2,000 bytes/query

recall@10 0.9510

Pinecone Serverless

2,000 bytes/query

recall@10 0.9609

Mode

Cold / Warm Latencyunfiltered

Zilliz Cloud Capacity 12CU

Cold / Warm Ratiolower is better

Zilliz Cloud Capacity 12CU

1.01×

Zilliz Cloud Tiered 4CU

2.16×

Notes:

  1. We note that while certain products may have a more dramatic cold/warm ratio at p99 percentile, this usually indicates a network shaking problem in later queries and cannot be fully reproduced. Thus we stick with the more faithful definition of cold/warm latency, i.e. the first query for each round.
  2. The timing for when a product's collection becomes cold is rather ambiguous since most products don't offer public APIs to provide such info. In order to simulate real world production settings, for cold latency benchmarking, we ensure to wait at least 24 hours since the last operations on the products for the collections to become as cold as possible.