On the Theoretical Limitations of Embedding-Based Retrieval

AI Overview

Problem Statement and Motivation

Information retrieval has increasingly shifted from traditional sparse methods like BM25 toward dense retrieval systems that use neural language models to encode queries and documents into fixed-dimension vector embeddings. These single-vector embedding models have demonstrated impressive performance across various benchmarks and are now being deployed for increasingly complex tasks involving instruction-following, reasoning, and multi-step inference. However, despite their empirical success, fundamental questions remain about the theoretical limits of what these models can represent.

This work addresses a critical gap in understanding: while the community has focused extensively on how to improve embedding models through better training data, larger architectures, and more sophisticated learning objectives, little attention has been paid to what cannot be achieved by single-vector embeddings due to inherent mathematical constraints. The authors tackle this fundamental question by connecting information retrieval to established results in communication complexity theory and geometric algebra.

Theoretical Framework

The authors formalize information retrieval as the problem of constructing a score matrix $B = U^T V$, where $U$ contains $d$-dimensional query embeddings and $V$ contains $d$-dimensional document embeddings. The goal is for $B$ to accurately reflect a ground-truth binary relevance matrix $A$, where $A_{ij} = 1$ if document $j$ is relevant to query $i$.

They introduce three increasingly strict definitions of representational capacity:

• Row-wise Order-Preserving Rank: The minimum dimension $d$ such that $B$ preserves the relative ordering of relevant versus irrelevant documents for each query
• Row-wise Thresholdable Rank: The minimum $d$ such that each query has a threshold separating its relevant from irrelevant documents
• Globally Thresholdable Rank: The minimum $d$ such that a single global threshold separates all relevant from irrelevant pairs

The key theoretical insight connects these measures to the sign rank of matrices. For a matrix $M \in \{-1, 1\}^{m \times n}$, the sign rank is the smallest dimension $d$ such that there exists a rank-$d$ matrix whose entries have the same sign pattern as $M$. The authors prove:

This establishes that the required embedding dimension is fundamentally bounded by the sign rank of the transformed relevance matrix. Since matrices can have arbitrarily high sign rank, this proves that for any fixed dimension $d$, there exist retrieval tasks that cannot be perfectly represented by $d$-dimensional embeddings.

Empirical Validation Through "Free Embeddings"

To validate these theoretical limits under ideal conditions, the authors conduct experiments with "free embeddings" - directly optimizable query and document vectors unconstrained by natural language. They create synthetic datasets where queries represent all possible top-$k$ combinations of documents, then optimize embeddings using standard contrastive learning objectives.

The experiments reveal a "critical-n point" - the maximum number of documents for which perfect accuracy can be achieved at a given dimension. The relationship follows a polynomial curve ($y = -10.53 + 4.03d + 0.05d^2 + 0.0037d^3$), demonstrating that even under optimal conditions, representational capacity grows polynomially with dimension while combinatorial complexity grows exponentially.

The LIMIT Dataset: A Real-World Stress Test

To bridge theory and practice, the authors develop LIMIT (LImitations of eMbedding-based reTireval), a dataset that instantiates theoretical limitations in natural language. The dataset uses simple "person likes attribute" statements with queries asking "who likes X?". Despite this apparent simplicity, LIMIT is designed to test all possible pairwise combinations among a small set of relevant documents within a larger corpus.

The dataset's key properties include:

• Simple, unambiguous natural language
• Dense relevance patterns that maximize combinatorial complexity
• Comprehensive coverage of document combinations rather than sparse sampling

Experimental Results on State-of-the-Art Models

Testing reveals that current state-of-the-art single-vector embedding models perform poorly on LIMIT despite the simple language involved. On the full 50,000 document version, models struggle to achieve even 20% Recall@100. Even on the smaller 46-document version containing only relevant documents, the best models achieve less than 60% Recall@20.

Several key patterns emerge:

Dimension Dependency: Performance consistently improves with higher embedding dimensions, supporting the theoretical predictions. Models using Matryoshka Representation Learning show significant degradation when truncated to lower dimensions.

Architecture Matters: Alternative architectures significantly outperform single-vector embeddings:

• Cross-encoders (Gemini 2.5 Pro) achieve perfect performance
• Sparse methods (BM25) perform nearly perfectly
• Multi-vector models (ColBERT variants) substantially outperform single-vector approaches

Not a Domain Shift Issue: Fine-tuning experiments show that the difficulty persists even when models are adapted to the LIMIT domain, indicating that the problem stems from fundamental representational limitations rather than unfamiliarity with the task domain.

Impact of Relevance Pattern Density

Ablation studies using different relevance patterns provide crucial insights into what makes certain retrieval tasks fundamentally difficult for single-vector models.

The "dense" pattern, which maximizes the number of document combinations that must be represented, proves significantly more challenging than random, cycle, or disjoint patterns. This confirms that combinatorial complexity - the number of different relevance relationships that must be simultaneously encoded - is the critical factor driving the observed limitations.

Limited Correlation with Existing Benchmarks

A striking finding is the lack of correlation between model performance on LIMIT and established benchmarks like BEIR. This suggests that current evaluation practices may not capture the fundamental representational constraints that become apparent when testing comprehensive combinatorial coverage.

Models that perform well on standard benchmarks show no systematic advantage on LIMIT, indicating that existing evaluation frameworks may inadvertently mask these theoretical limitations by sampling only sparse subsets of possible query-document relationships.

Implications for Future Research

This work establishes several important implications for the information retrieval community:

Theoretical Boundaries: The results demonstrate that single-vector embedding models face fundamental mathematical constraints that cannot be overcome through scaling model size or training data alone. As retrieval tasks become more complex - particularly those involving instruction-following and reasoning - these limitations become increasingly relevant.

Architectural Considerations: The superior performance of cross-encoders, multi-vector models, and sparse methods suggests that future advances in complex retrieval may require moving beyond the single-vector paradigm. This represents a significant shift from current trends focused on scaling dense retrievers.

Evaluation Framework Gaps: Current benchmarks may systematically underestimate the difficulty of comprehensive retrieval tasks by sampling only small fractions of possible query-document relationships. This has implications for how the community evaluates progress and allocates research resources.

Resource Allocation: Understanding these theoretical limits can guide more efficient research investment, suggesting that efforts to make single-vector models solve inherently impossible problems may be better directed toward architectures that can overcome these fundamental constraints.

The work provides both a theoretical foundation for understanding embedding limitations and practical evidence that these constraints manifest in realistic scenarios, even with simple natural language. This dual contribution makes it a significant reference point for future research in information retrieval architecture and evaluation methodology.