NEEDLE — search engine benchmarks

NEEDLE: Which search API moves the needle for AI agents?

Last run: Oct 24, 11:52 UTC. News runs each hour; all other suites run each day. Times are UTC. Results table.

NEEDLE (News, Everyday, Expert, Deep-tail, and Legal Evaluation) is a live open-source search benchmark. It compares search engines with queries that model agent behavior. The five verticals match the use cases we see each day in Keenable's own search traffic:

Scholar: expert literature searches: find a specific paper from a degraded title, a full-text detail, or a tip-of-the-tongue description
News: breaking and developing stories; new queries each hour from curated RSS feeds and Google Trends
Finance: everyday company and SEC-filing lookups with known answers from public registries
AgenticRare: obscure, long-tail entities; rare-word queries sampled from the DeepResearchGym, OpenResearcher, and LRAT agentic search logs
Legal: find a specific court opinion or section of the Code of Federal Regulations

Methodology. Each engine in a run receives the same query text. We score each engine on the ranking, titles, and snippets it returned. We do not fetch pages and we do not re-rank. The judge does not see the engine name. We make new query sets from public sources, News each hour and the rest each day, so there is no fixed set to overfit. Appendix C gives the full comparison protocol. The benchmark is open source: the keenableai/needle repo is public, every run executes in plain GitHub Actions, and we welcome contributions.

Metrics. News and AgenticRare queries have no single correct result, so we measure ranking quality. An LLM judge rates each result from 0 to 4. We report nDCG@5, normalized so that the pooled ultimate engine scores exactly 1, with a penalty for duplicate URLs. Finance measures if the answer text reaches the agent: we scan the top-5 titles and snippets and report answer-recall@5 with MRR@5. Scholar and Legal are known-item tasks: we match identifiers and report recall@10 and recall@5 with MRR. Appendix D gives the exact definitions.

The ultimate ceiling. The italicized row in each leaderboard is ultimate, a synthetic engine. It pools all engines' results per query and ranks them by their relevance labels (Appendix C). The standings radar plots each engine as a share of ultimate.

Quality results

Standings — all verticals

share of ultimate (all engines pooled, the outer ring) per vertical, farther out is better; click a table row to toggle an engine · hatched band?

engine	overall	overall 95% CI	Scholar	News	Finance	AgenticRare	Legal
ultimate (raw)	100%		0.901	1.000	0.990	1.000	0.971
gap to ultimate	−27%		−17%	−36%	−5%	−50%	−18%
Keenable	73%	[72%, 74%]	83%	59%	90%	50%	82%
exa	69%	[67%, 70%]	64%	58%	95%	48%	78%
perplexity	66%	[65%, 68%]	67%	64%	92%	46%	63%
brave-llmcontext	66%	[65%, 67%]	60%	64%	89%	45%	70%
perplexity-fast	65%	[63%, 67%]	66%	63%	92%	45%	61%
parallel	63%	[62%, 65%]	65%	50%	91%	36%	74%
exa-instant	60%	[59%, 61%]	45%	48%	93%	44%	70%
linkup	60%	[58%, 61%]	54%	57%	85%	40%	63%
brave	59%	[58%, 60%]	55%	56%	82%	35%	67%
you	58%	[56%, 59%]	53%	56%	81%	34%	64%
deepseek	57%	[55%, 58%]	28%	63%	84%	38%	70%
context.dev	57%	[55%, 58%]	61%	45%	83%	33%	61%
parallel-turbo	54%	[53%, 56%]	42%	44%	88%	33%	65%
kagi	53%	[51%, 54%]	63%	32%	78%	29%	62%
firecrawl	52%	[50%, 53%]	56%	28%	85%	25%	65%
octen	51%	[49%, 53%]	34%	51%	78%	31%	60%
google/serper	49%	[48%, 51%]	54%	26%	78%	25%	62%
bing/searchapi	48%	[47%, 50%]	50%	32%	76%	18%	64%
chatgpt-search	47%	[41%, 52%]	27%	32%	73%	35%	70%
jina	44%	[42%, 46%]	52%	21%	78%	16%	55%
tavily	43%	[42%, 45%]	35%	29%	87%	28%	38%
claude-search	39%	[32%, 46%]	43%	32%	51%	21%	49%
tinyfish	39%	[38%, 41%]	14%	25%	79%	21%	56%

7-day leaderboards

Query-weighted 7-day means with a 95% CI; suite columns split the score.

Overall — 7-day mean

mean share of ultimate across 5 verticals, last 7 days

engine	of ultimate	95% CI
Keenable	73%	[72.0%, 74.1%]
exa	69%	[67.4%, 69.7%]
perplexity	66%	[64.5%, 68.2%]
brave-llmcontext	66%	[64.7%, 66.8%]
perplexity-fast	65%	[63.4%, 67.4%]
parallel	63%	[61.9%, 64.6%]
exa-instant	60%	[58.9%, 60.7%]
linkup	60%	[58.2%, 60.9%]
brave	59%	[58.0%, 60.5%]
you	58%	[56.3%, 59.0%]
deepseek	57%	[55.1%, 58.1%]
context.dev	57%	[54.9%, 58.2%]
parallel-turbo	54%	[53.1%, 55.9%]
kagi	53%	[51.2%, 54.1%]
firecrawl	52%	[50.4%, 53.2%]
octen	51%	[49.5%, 52.6%]
google/serper	49%	[47.7%, 50.7%]
bing/searchapi	48%	[46.6%, 49.6%]
chatgpt-search	47%	[41.2%, 51.9%]
jina	44%	[41.9%, 46.4%]
tavily	43%	[42.1%, 44.6%]
claude-search	39%	[32.1%, 45.9%]
tinyfish	39%	[37.6%, 40.6%]

Scholar — 7-day mean

mean of 9 runs, last 7 days — latest Oct 23, 15:21 UTC

engine	recall@10	95% CI	title	body	clue	tot
ultimate	0.901	[0.890, 0.910]	1.000	0.900	0.942	0.760
Keenable	0.747	[0.732, 0.761]	0.913	0.754	0.822	0.498
perplexity	0.606	[0.557, 0.658]	0.889	0.609	0.648	0.281
perplexity-fast	0.597	[0.547, 0.649]	0.883	0.584	0.643	0.278
parallel	0.584	[0.568, 0.600]	0.996	0.429	0.654	0.258
exa	0.575	[0.551, 0.599]	0.909	0.588	0.498	0.305
kagi	0.564	[0.548, 0.581]	0.972	0.660	0.575	0.050
context.dev	0.546	[0.504, 0.590]	0.877	0.671	0.514	0.120
brave-llmcontext	0.539	[0.522, 0.556]	0.913	0.444	0.624	0.176
firecrawl	0.502	[0.484, 0.520]	0.865	0.667	0.457	0.018
brave	0.497	[0.479, 0.515]	0.904	0.359	0.602	0.123
google/serper	0.490	[0.472, 0.509]	0.862	0.630	0.448	0.020
linkup	0.483	[0.466, 0.500]	0.893	0.356	0.580	0.105
you	0.482	[0.465, 0.498]	0.909	0.341	0.585	0.091
jina	0.464	[0.370, 0.521]	0.826	0.602	0.408	0.020
bing/searchapi	0.451	[0.434, 0.468]	0.948	0.381	0.433	0.041
exa-instant	0.402	[0.379, 0.427]	0.712	0.401	0.358	0.137
claude-search	0.391	[0.222, 0.514]	0.688	0.315	0.460	0.099
parallel-turbo	0.378	[0.362, 0.395]	0.923	0.184	0.326	0.080
tavily	0.312	[0.296, 0.328]	0.836	0.241	0.143	0.028
octen	0.304	[0.282, 0.324]	0.715	0.061	0.222	0.217
deepseek	0.254	[0.228, 0.279]	0.581	0.176	0.176	0.081
chatgpt-search	0.241	[0.162, 0.310]	0.671	0.049	0.166	0.076
tinyfish	0.126	[0.105, 0.149]	0.235	0.151	0.117	0.002

News — 7-day mean

mean of 161 runs, last 7 days — latest Oct 24, 11:52 UTC

engine	nDCG@5	95% CI
ultimate	1.000	[1.000, 1.000]
brave-llmcontext	0.644	[0.635, 0.654]
perplexity	0.635	[0.624, 0.646]
deepseek	0.634	[0.623, 0.645]
perplexity-fast	0.633	[0.623, 0.644]
Keenable	0.595	[0.583, 0.606]
exa	0.580	[0.564, 0.594]
linkup	0.569	[0.559, 0.580]
brave	0.562	[0.552, 0.572]
you	0.556	[0.546, 0.566]
octen	0.513	[0.500, 0.526]
parallel	0.497	[0.478, 0.516]
exa-instant	0.481	[0.466, 0.495]
context.dev	0.455	[0.442, 0.468]
parallel-turbo	0.436	[0.420, 0.451]
chatgpt-search	0.320	[0.305, 0.335]
claude-search	0.319	[0.283, 0.353]
kagi	0.319	[0.310, 0.327]
bing/searchapi	0.316	[0.308, 0.324]
tavily	0.295	[0.285, 0.305]
firecrawl	0.283	[0.274, 0.292]
google/serper	0.262	[0.254, 0.270]
tinyfish	0.254	[0.246, 0.263]
jina	0.207	[0.200, 0.215]

Finance — 7-day mean

mean of 7 runs, last 7 days — latest Oct 24, 03:41 UTC

engine	recall@5	95% CI	filings	filingdoc
ultimate	0.990	[0.983, 0.996]	1.000	0.945
exa	0.938	[0.920, 0.959]	0.957	0.754
exa-instant	0.921	[0.893, 0.954]	0.947	0.664
perplexity	0.914	[0.883, 0.945]	0.936	0.625
perplexity-fast	0.909	[0.877, 0.943]	0.926	0.613
parallel	0.905	[0.884, 0.936]	0.951	0.525
Keenable	0.894	[0.870, 0.929]	0.893	0.513
brave-llmcontext	0.885	[0.861, 0.922]	0.968	0.333
parallel-turbo	0.875	[0.844, 0.917]	0.906	0.545
tavily	0.857	[0.830, 0.900]	0.848	0.361
firecrawl	0.842	[0.813, 0.889]	0.692	0.467
linkup	0.839	[0.794, 0.896]	0.778	0.410
deepseek	0.831	[0.801, 0.881]	0.925	0.400
context.dev	0.821	[0.783, 0.877]	0.654	0.445
brave	0.814	[0.776, 0.872]	0.665	0.370
you	0.801	[0.764, 0.860]	0.633	0.347
tinyfish	0.781	[0.742, 0.839]	0.526	0.459
google/serper	0.777	[0.741, 0.839]	0.500	0.433
jina	0.776	[0.735, 0.839]	0.631	0.471
octen	0.773	[0.731, 0.842]	0.773	0.167
kagi	0.768	[0.724, 0.834]	0.503	0.342
bing/searchapi	0.755	[0.710, 0.827]	0.423	0.390
chatgpt-search	0.719	[0.466, 0.879]	0.650	0.555
claude-search	0.509	[0.282, 0.752]	0.324	0.387

AgenticRare — 7-day mean

mean of 8 runs, last 7 days — latest Oct 24, 08:38 UTC

engine	nDCG@5	95% CI
ultimate	1.000	[1.000, 1.000]
Keenable	0.505	[0.481, 0.529]
exa	0.479	[0.454, 0.505]
perplexity	0.459	[0.434, 0.484]
brave-llmcontext	0.454	[0.430, 0.478]
perplexity-fast	0.454	[0.429, 0.478]
exa-instant	0.439	[0.414, 0.462]
linkup	0.396	[0.372, 0.420]
deepseek	0.377	[0.353, 0.402]
parallel	0.363	[0.336, 0.396]
brave	0.352	[0.331, 0.373]
chatgpt-search	0.350	[0.244, 0.412]
you	0.340	[0.319, 0.361]
parallel-turbo	0.332	[0.308, 0.356]
context.dev	0.329	[0.307, 0.350]
octen	0.314	[0.291, 0.336]
kagi	0.291	[0.272, 0.310]
tavily	0.277	[0.256, 0.297]
firecrawl	0.252	[0.234, 0.271]
google/serper	0.247	[0.229, 0.265]
claude-search	0.210	[0.106, 0.301]
tinyfish	0.206	[0.188, 0.224]
bing/searchapi	0.181	[0.165, 0.197]
jina	0.165	[0.149, 0.180]

Legal — 7-day mean

mean of 7 runs, last 7 days — latest Oct 23, 20:37 UTC

engine	recall@5	95% CI	caselaw	code
ultimate	0.971	[0.952, 0.987]	1.000	0.961
Keenable	0.798	[0.757, 0.839]	0.958	0.743
exa	0.761	[0.716, 0.805]	0.885	0.718
parallel	0.718	[0.672, 0.765]	0.927	0.646
chatgpt-search	0.684	[0.624, 0.740]	0.781	0.650
brave-llmcontext	0.678	[0.631, 0.725]	0.927	0.593
deepseek	0.678	[0.626, 0.729]	0.863	0.617
exa-instant	0.676	[0.628, 0.723]	0.885	0.604
brave	0.652	[0.604, 0.699]	0.885	0.571
parallel-turbo	0.633	[0.577, 0.690]	0.875	0.550
firecrawl	0.628	[0.569, 0.676]	0.740	0.589
bing/searchapi	0.622	[0.573, 0.671]	0.740	0.582
you	0.620	[0.571, 0.668]	0.896	0.525
perplexity	0.612	[0.545, 0.678]	0.948	0.496
linkup	0.612	[0.563, 0.660]	0.844	0.532
kagi	0.601	[0.550, 0.651]	0.708	0.564
google/serper	0.598	[0.549, 0.649]	0.667	0.575
context.dev	0.590	[0.541, 0.640]	0.729	0.543
perplexity-fast	0.588	[0.512, 0.657]	0.948	0.464
octen	0.585	[0.535, 0.635]	0.594	0.582
tinyfish	0.548	[0.497, 0.597]	0.656	0.511
jina	0.532	[0.481, 0.583]	0.563	0.521
claude-search	0.476	[0.307, 0.583]	0.740	0.386
tavily	0.370	[0.321, 0.419]	0.698	0.257

Slices over time

Slice trends — all verticals

The same slices as the leaderboards, plotted over the full history: each engine's score as % of ultimate, smoothed with a 7-day rolling mean. Click legend names to highlight engines.

Scholar — recall@10 over time

recall@10, gold papers per run across all query types

Scholar — title-query recall@10 over time

recall@10 on papers found by their (degraded) title

Scholar — body-query recall@10 over time

recall@10 on papers found by a full-text-only detail

Scholar — clue-query recall@10 over time

recall@10 on papers found by a natural-language question over full-text-only facts

Scholar — tip-of-the-tongue recall@10 over time

recall@10 on papers found from a hedged, half-remembered description

News — nDCG@5 over time

20 fresh queries per run, mean nDCG over all queries in each 6-hour window

Finance — answer-recall@5 over time

answer-recall@5, gold queries per run across all suites

Finance — MRR@5 over time

MRR@5, same runs as recall

Finance — filings recall@5 over time

recall@5 on quarterly facts (net income, operating income, diluted EPS) in top-5 snippets

Finance — filingdoc recall@5 over time

recall@5 on SEC filings found by a distinctive detail (accession identity)

AgenticRare — nDCG@5 over time

100 rare-entity queries per daily run

Legal — recall@5 over time

recall@5, gold documents per run across both suites

Legal — caselaw recall@5 over time

recall@5 on federal court opinions found by caption-style queries

Legal — code recall@5 over time

recall@5 on CFR sections found from their regulatory substance

Quality vs price

Overall quality vs price

7-day mean share of ultimate vs lowest public price per 1,000 queries; Oct 17, 12:13 – Oct 24, 11:52 UTC

engine	$ / 1k queries	overall
Keenable	$1.00	73%
exa	$7.00	69%
perplexity	$5.00	66%
brave-llmcontext	$5.00	66%
perplexity-fast	$1.00	65%
parallel	$5.00	63%
exa-instant	$7.00	60%
linkup	$5.00	60%
brave	$5.00	59%
you	$5.00	58%
deepseek	$0.75	57%
context.dev	$0.50	57%
parallel-turbo	$1.00	54%
kagi	$12.00	53%
firecrawl	$1.20	52%
octen	$1.00	51%
google/serper	$0.30	49%
bing/searchapi	$1.00	48%
chatgpt-search	$10.00	47%
jina	$3.40	44%
tavily	$5.00	43%
claude-search	$10.00	39%
tinyfish	$0.00	39%

Price per 1,000 queries

biggest self-serve tier, no enterprise quotes; collected 2026-08-21

engine	$ / 1k queries	plan
tinyfish	$0.00	free
google/serper	$0.30	Serper Ultimate — $3,750 / 12.5M credits
context.dev	$0.50	Scale — $499/mo / 1M credits, 1 credit per 10 results
deepseek	$0.75	pay-as-you-go — $0.75 / 1k requests
bing/searchapi	$1.00	SearchApi.io Octo 5M — $5,000/mo
Keenable	$1.00	Frontier Tier — at 100 RPS+
octen	$1.00	pay-as-you-go
parallel-turbo	$1.00	pay-as-you-go
perplexity-fast	$1.00	fast search — $1 / 1k requests
firecrawl	$1.20	Scale — $599/mo / 1M credits, 2 credits per search
jina	$3.40	$0.05 / 1M tokens, ≈68k tokens per search
brave	$5.00	flat rate
brave-llmcontext	$5.00	flat rate
linkup	$5.00	pay-as-you-go — standard depth, searchResults
parallel	$5.00	pay-as-you-go
perplexity	$5.00	flat rate
tavily	$5.00	Growth — $500/mo / 100k credits
you	$5.00	Web Search API plan
exa	$7.00	pay-as-you-go
exa-instant	$7.00	pay-as-you-go
chatgpt-search	$10.00	web_search tool — $10 / 1k calls, plus tokens
claude-search	$10.00	web search tool — $10 / 1k searches, plus tokens
kagi	$12.00	pay-as-you-go

Search latency

Agents wait on each search call, so latency adds directly to task time.

Search latency — last 7 days

p50 (filled) to p95 (open), successful calls, log scale; hover for values (Appendix C)

engine	p50	p95	errors
Keenable	143 ms	351 ms	0.0%
perplexity-fast	161 ms	232 ms	0.0%
octen	206 ms	412 ms	0.0%
perplexity	262 ms	1.6 s	0.0%
parallel-turbo	348 ms	604 ms	1.2%
exa-instant	498 ms	783 ms	0.5%
brave	509 ms	737 ms	0.0%
you	515 ms	697 ms	0.0%
deepseek	566 ms	689 ms	0.0%
firecrawl	776 ms	4.5 s	0.0%
context.dev	1.1 s	4.6 s	0.0%
kagi	1.3 s	2.1 s	0.0%
brave-llmcontext	1.3 s	2.5 s	0.0%
linkup	1.3 s	1.9 s	0.0%
google/serper	1.5 s	3.4 s	0.0%
tinyfish	2.0 s	3.6 s	30.2%
tavily	2.0 s	3.7 s	0.1%
bing/searchapi	2.4 s	8.7 s	0.3%
exa	2.5 s	6.2 s	0.5%
parallel	3.2 s	5.0 s	1.2%
jina	17 s	25 s	5.5%
claude-search	34 s	51 s	27.6%
chatgpt-search	42 s	55 s	13.4%

Index independence

Repeated overlap on junk or near-identical lists suggests a shared upstream index (all benchmarks).

Definition: suspicious sharing

the pairwise test behind the matrix and the borrowing graph (src/needle/shared/overlap.py)

Let U_A(q) be the normalized URLs engine A returned for query q, and L_A(q) ⊆ U_A(q) those judged below HM. Engines A and B share suspiciously on q when

|L_A(q) ∩ L_B(q)| ≥ 1 or |U_A(q) ∩ U_B(q)| ≥ 3.

Each matrix cell shows the share of comparable queries (both searches succeeded and at least one returned a result) where the pair shares suspiciously, pooled over the selected runs. Engines that share good results show normal ranking behavior. Engines that share junk, or 3+ URLs at once, possibly share an upstream index. Results on the known-answer verticals have no judgements, so only the second clause can apply.

Index independence — four views

suspicious-sharing matrix, URL-intersection matrix, borrowing graph, and per-engine result uniqueness

Result uniqueness (all runs)

share of an engine's URLs that no other engine family returned for the same query and that are relevant (judged HM/FullyM on the judged verticals, the known-item gold hit on the known-answer verticals); the URLs column counts unique relevant URLs / all URLs the engine returned; same-family engines (Exa/Exa-instant, Parallel/Parallel-turbo) do not count against each other

Table view — all runs in range

Appendix — methodology & configuration

We pin everything below in keenableai/needle; you can compute each number on this page again from the archived artifacts (Appendix G).

Appendix A — judgement browser

pick a benchmark, then a run, to inspect each query; full runs on the HF dataset

Appendix B — query generation

how we build each slice's queries (per-bench generate.py); the exact rows every run used are archived (Appendix G)

We make each query set fresh from public sources: News each hour, the other verticals each day.

For Scholar we sample papers from arXiv (computer, physical, life, and social sciences) and Europe PMC (health sciences). Four age bands reach from the last week back to approximately five years. Each paper gives one query per slice, so the four Scholar slices rank engines on the same papers. The gaps between the slices are the point: title vs body isolates full-text indexing, body vs clue isolates keyword-vs-prose handling, and tot isolates semantic retrieval.

We leak-check each LLM-written Scholar query: at least two content words must not appear in the title or the abstract. We keep a paper only if each slice produced a valid query. The known-item and tip-of-the-tongue framing follows Exa's "SOTA Search Over Academic Publications" (Jul 2026).

The AgenticRare stream merges search queries from three public agent-trajectory releases: the anonymized DeepResearchGym logs (14M+ search requests from real agentic clients; "Agentic Search in the Wild", SIGIR '26), the OpenResearcher deep-research trajectories (TIGER-AI-Lab, 2026), and LRAT-Train ("Learning to Retrieve from Agent Trajectories", SIGIR '26).

vertical	slice	query construction
Scholar	title	the paper's title, lowercased with LaTeX and punctuation stripped; plain known-item lookup, answerable from metadata
Scholar	body	a keyword query (3–8 words) anchored on a globally distinctive detail that appears in the full text but not in the title or abstract (a named method, cohort, instrument, or an unusual measured value); answerable only by engines that index the paper body
Scholar	clue	a natural-language question built from 2–3 precise factual clues taken from the body, the way a researcher re-finds a paper from remembered specifics
Scholar	tot	a hedged, half-remembered description with every specific blurred: approximate quantities, no names the paper introduces, at most two content words reused from the title
News	fresh	the newest item per feed from ~124 curated RSS feeds within a freshness window, merged with Google Trends RSS across all 52 US geos (fuzzy-deduped, capped by traffic); an LLM projects each item into a keyword query and refuses evergreen content
Finance	finance / finance_nl	registry facts per company (CEO, founding year, HQ country, website, employees, LEI) from Wikidata and GLEIF, ticker from SEC, asked as a keyword query and, where a template exists, as a natural-language variant
Finance	filings	single-quarter 10-Q facts (net income, operating income, diluted EPS) from SEC XBRL, companies stratified by market cap; half the queries are quoted phrases
Finance	filingdoc	known-item SEC-filing retrieval; the gold answer is the filing's accession number; half the queries carry operator syntax (site:, "quoted", after:/before:)
AgenticRare	rare-entity	sampled from the merged DeepResearchGym + OpenResearcher + LRAT-Train query stream, keeping queries where BERT WordPiece splits a word into 5+ subwords or maps it to [UNK]; fastText language ID gates for English, an n-gram cap drops near-copies, and each run samples stratified by query length
Legal	caselaw	recent published opinions from the CourtListener API across 14 federal courts, one caption-style query per case; gold is the reporter citation, docket number with party tokens, and cluster id
Legal	code	eCFR sections; an LLM projects each section into a query about its regulatory substance with one verbatim quoted span, rejected if it leaks the citation; gold is the "title CFR section" citation

Appendix C — comparison protocol

the fairness rules behind every number on this page

We send each engine in a run the same query text. We pin the per-engine request shapes in git (Appendix F). The runner sends requests in sequence, one call at a time across the whole run. Thus the latency percentiles are comparable across engines, and no engine gets concurrent load. Engines see the live web minutes apart in a run, not at the same instant.

We judge an engine on its own ranking, titles, and snippets. We do not fetch pages and we do not re-rank. We cap the evidence at the same snippet length for each engine, so engines that return more content get no extra signal to the judge or the answer matchers. We judge each distinct combination of URL, title, snippet, and date once per query. We use that rating again where the identical result occurs for other engines (Appendix E gives the full judging setup).

We remove failed searches and judgements from the mean and report them as search_errors and judge_errors. An outage decreases num_scored and increases the error rate; it does not decrease the quality score.

We make fresh News queries each hour and new known-answer queries each day from public sources (Appendix B), so there is no fixed benchmark set to overfit. We archive the exact queries with each run (Appendix G).

Each vertical also reports ultimate, a synthetic engine. It pools all engines' results per query and ranks them oracle-style: best judge rating first on the judged verticals, and the gold item first on the known-answer verticals. Its score shows what one engine could get if it always returned the best results that the full field found. This separates "the engine ranked poorly" from "no engine found better results". We do not include ultimate in the overlap and uniqueness statistics.

Appendix D — metric definitions

src/needle/shared/metrics.py and per-bench score.py

metric	benches	definition
recall@10, MRR@10	Scholar	known-item: the gold paper identified in the top 10 by arXiv ID / DOI / PMID match.
nDCG@5	News, AgenticRare	DCG over the top 5 results (Σ gain(rᵢ) / log₂(i + 1), with judge ratings mapped to gain 4 → 1.0, 3 → 0.667, 2 → 0.117, 1/0 → 0), divided by the DCG of the ideal ranking: the pooled ultimate ordering of every engine's deduplicated results, so ultimate scores exactly 1. A query is scored only if every one of its results was judged (otherwise it is dropped from the mean and counted as judge errors) and only if the pooled ideal has any gain; queries where no engine found anything relevant are excluded.
redundancy penalty	News, AgenticRare	applied to engine ratings before scoring: a duplicate URL drops the rating to 0. The pooled ideal is already deduplicated, so no penalty applies there. Skipped when the query itself carries a site: operator. Raw and penalized ratings are both shown in the judgement browser.
answer-recall@5, MRR@5	Finance	a query is a hit when the gold answer appears in a top-5 title or snippet (deterministic matcher, with an LLM-judge backstop for paraphrased answers). MRR is the mean reciprocal rank of the first hit.
recall@5, MRR@5	Legal	known-item: the gold opinion or CFR section identified by citation, docket (with a party-token check), CFR citation, or URL match.
latency p50 / p95	all	percentiles over per-call durations of successful search API calls, pooled across queries in the window; runs published without per-call samples contribute their per-run percentiles instead. Failed calls carry no latency sample and show up in the error rate.

Appendix E — LLM judge

rates News and AgenticRare results; Finance fallback matcher only; Scholar and Legal are deterministic

We judge with openai/gpt-6-luna via OpenRouter, high reasoning effort, temperature 0, one pointwise call per document. We record the model for each run as judge_model in its archived report. The judgement browser shows it when you select a run, so judge changes stay visible in the history.

The judge does not see the engine identity. The prompt contains the query, the document title, the URL, the published date, and the snippet — not the engine name. We judge each distinct combination of URL, title, snippet, and date once per query and use that rating again where the identical result occurs for other engines. We judge the same URL with a different snippet separately, with the same blind judge and rubric.

A judge preference for a snippet style would be a shared bias across engines. We publish the per-result reasoning with each run, so you can audit this.

Rubric: 0–4 Needs Met scale (FailsM / SM / HM / FullyM) with hard caps for a wrong entity, stale coverage of dated events, and search-operator compliance ("quoted phrases", site:). The full prompt is one template in the repo: src/needle/shared/prompts/judgement.jinja.

Known limitation: we operate NEEDLE ourselves, and we did not yet do a systematic human-agreement audit of the judge. But you can check the numbers: the judge is blind and shared across engines, each rating includes its reasoning, and we publish the queries, prompts, and raw results. Thus anyone can score any rating again.

Appendix F — engine configuration

request shapes pinned in git (src/needle/shared/search/); k = 5 results per query except Scholar (10). Engines are live, unversioned APIs; provider-side changes surface as trend shifts.

Every run builds each client with the same snippet budget of 2,000 characters. Keenable, Exa and Ceramic accept a request-side length parameter, so they get it in the request (snippet_max_length=2000, highlights.maxCharacters=2000, maxDescriptionLength=2000). The other engines have no such parameter; we clip whatever they return to the same 2,000 characters before judging and answer matching (Appendix C). Requests time out at 30 s (60 s for Perplexity, Parallel and Jina, 180 s for claude-search and chatgpt-search) and retry up to 3 times with exponential backoff on transport errors, 429, and 5xx.

engine	API	pinned request
Keenable	Keenable POST /v1/search	mode=realtime, snippet_max_length=2000; query verbatim with site: kept in the text, after:/before: → published_after:/published_before: tokens; no count parameter — we keep the first k results
google/serper	Serper.dev (Google SERP) POST /search	q = query verbatim (operators kept as text), num=min(k,100), gl=us, hl=en; knowledge-graph and answer-box entries fold into the ranking
bing/searchapi	SearchApi.io GET /search	engine=bing, q = query verbatim (operators kept as text), num=min(k,50); knowledge-graph and answer-box fold in
brave	Brave GET /web/search	q = query text with site: kept, clipped to 75 words / 600 chars; count=min(k,20), country=us, search_lang=en, result_filter=web, text_decorations=false; after:/before: → freshness=YYYY-MM-DDtoYYYY-MM-DD; extra_snippets joined with newlines, falling back to description when absent
brave-llmcontext	Brave POST /llm/context	q = query text clipped to 75 words / 600 chars; country=us, search_lang=en; after:/before: → freshness=YYYY-MM-DDtoYYYY-MM-DD; per-URL snippets joined with newlines and clipped client-side to the 2,000-char budget, published date from the sources map
exa	Exa POST /search	type=auto, numResults=k, contents.highlights.maxCharacters=2000; site: → includeDomains, after:/before: → startPublishedDate/endPublishedDate (full-day UTC bounds)
exa-instant	Exa POST /search	same as exa but type=instant
parallel	Parallel POST /v1/search	mode=basic, search_queries=[query text], advanced_settings.max_results=k; site: → source_policy.include_domains, after: → source_policy.after_date (before: has no mapping and is dropped)
parallel-turbo	Parallel POST /v1/search	same as parallel but mode=turbo
perplexity	Perplexity POST /search	max_results=min(k,20), search_context_size=low; site: → search_domain_filter, after:/before: → search_after_date_filter/search_before_date_filter (MM/DD/YYYY)
perplexity-fast	Perplexity POST /search	same as perplexity but search_type=fast (perplexity leaves the field unset, so it takes the API default web)
tavily	Tavily POST /search	search_depth=basic, max_results=min(k,20); site: → include_domains, after:/before: → start_date/end_date (ISO dates)
linkup	Linkup POST /v1/search	depth=standard, outputType=searchResults, maxResults=k; site: → includeDomains, after:/before: → fromDate/toDate (ISO dates); HTML entities in name/content unescaped, image results dropped; the API returns no published date
tinyfish	TinyFish GET api.search.tinyfish.ai	query = query text clipped to 2,000 chars; no count parameter — we keep the first k results; site: → include_domains (comma-separated), after:/before: → after_date/before_date (ISO dates)
you	You.com POST /v1/search	count=min(k,100); site: → include_domains, after:/before: → freshness=YYYY-MM-DDtoYYYY-MM-DD; web and news sections merge, deduplicated by URL
firecrawl	Firecrawl POST /v2/search	limit=min(k,100), sources=[web]; site: kept in the query text, after:/before: → tbs=cdr:1,cd_min:M/D/YYYY,cd_max:M/D/YYYY
kagi	Kagi POST /api/v1/search	query = query text, limit=min(k,1024); site: → lens.sites_included, after:/before: → filters.after/filters.before (ISO dates); search and news sections merge, deduplicated by URL; HTML markup stripped from titles and snippets
octen	Octen POST /search	query = query text clipped to 500 chars, count=k, language=[en], safesearch=off, highlight.enable=true with max_tokens=512; site: → include_domains, after:/before: → start_time/end_time (full-day UTC bounds); highlight is the snippet
jina	Jina GET s.jina.ai	q = full query with site:/after:/before: kept as Google operators, num=min(k,20); default mode fetches each result page, so the snippet is the page markdown (content) clipped client-side to the 2,000-char budget, falling back to the SERP description when the fetch returns nothing; published date from publishedTime, else the SERP date string
context.dev	Context.dev POST /v1/web/search	query = query text clipped to 500 chars, numResults=min(max(k,10),100) (API floor is 10), country=us, queryFanout=false; site: → includeDomains, after: (with no before:) → freshness last_24_hours/last_week/last_month/last_year covering that date, before: dropped; snippet is the SERP description, markdown scrape left off
deepseek	DeepSeek POST /v1/tools/search	query = query text clipped to fit 128 chars with site: (several joined by OR), lang:en and after:/before: as native operators, count=min(k,50), need_content=false; snippet is the long query-aware summary, falling back to the short snippet; published date from date
claude-search	Anthropic POST /v1/messages	model=claude-sonnet-5, max_tokens=32000, tools=[web_search_20250305 with max_uses=1], system prompt asks for a JSON array of every search result with the snippet copied verbatim (≤500 chars); rank = web_search_tool_result order, title/snippet from the JSON row with the same URL, published_date = page_age; the prompt tells the model to pass the query to the tool verbatim (operators kept as text), site: also → allowed_domains
chatgpt-search	OpenAI POST /v1/responses	model=gpt-5.5, reasoning.effort=low, max_output_tokens=32000, tools=[web_search with search_context_size=low], tool_choice forces the search, include=web_search_call.action.sources, same system prompt; rank = action.sources order, title/snippet from the JSON row with the same URL (utm_ parameters stripped); the prompt tells the model to pass the query to the tool verbatim (operators kept as text), site: also → filters.allowed_domains

Appendix G — raw artifacts

everything you need to re-score a run from scratch

artifact	contents	where
per-run reports	every query and result (title, URL, snippet), judge rating + penalized rating + reasoning, per-call latencies, and the run's config (judge_model, num_results)	HF: keenable-ai/needle-results, runs/<run-id>/
query files	the exact queries each run issued, including gold answers (fresh.jsonl, gold.jsonl, scholar.jsonl, legal.jsonl)	same run folder
dashboard aggregates	history.jsonl, overlap.jsonl, uniqueness.jsonl, runs.json, ci_scores.jsonl, ci.json	data/ on this site (gh-pages)
prompts & scoring code	judge prompt (shared/prompts/judgement.jinja), metric implementations, engine clients	GitHub: keenableai/needle

Citation

if you use NEEDLE in your work, please cite

@misc{needle2026,
  author = {Styskin, Andrey and Petri, Matthias and Gusev, Ilya},
  title  = {{NEEDLE}: A Live Open-Source Search Benchmark for {AI} Agents},
  year   = {2026},
  url    = {https://keenableai.github.io/needle/},
  note   = {Contact: andrey@keenable.ai, matthias@keenable.ai, ilya.gusev@keenable.ai}
}