GitHub - brcrusoe72/agent-search: Self-hosted search API + MCP server for AI agents. Bundles SearXNG. Zero API keys, one-command deploy. Open-source alternative to Tavily, Exa, and Serper.

⚡ Quick Start

Three commands. One endpoint. No API keys, no quotas, no vendor lock-in.

git clone https://github.com/brcrusoe72/agent-search.git
cd agent-search
docker compose up -d

curl "http://localhost:3939/search?q=distributed+consensus+algorithms&count=5"

That's it. You now have a deduplicated, multi-engine, LLM-ready search API running at http://localhost:3939.

Terminal Demo

For a reproducible terminal GIF workflow using the real AgentSearch quick-start commands, see docs/TERMINAL_GIF_GUIDE.md and the tapes in docs/demo/.

🎯 Why AgentSearch?

You could call SearXNG directly. Most people building serious agent infrastructure end up writing this layer anyway. AgentSearch is that layer, already built.

📊 How It Compares

Translation: if you're running more than ~10K queries/month against Tavily or Exa, AgentSearch pays for itself the first month. If you're processing sensitive queries, it's the only option that doesn't leak them to a third party.

🧩 Core Features

Deduplication & Cross-Engine Scoring

Every query hits multiple engines. Results are fingerprinted, deduplicated, and scored by cross-engine agreement. A result that surfaces on Google and Bing and DuckDuckGo ranks higher than one that only appears on one.

curl "http://localhost:3939/search?q=python+async+patterns&count=5"

{
  "results": [
    {
      "title": "Async IO in Python: A Complete Walkthrough",
      "url": "https://realpython.com/async-io-python/",
      "snippet": "A comprehensive guide to async/await in Python 3...",
      "engines": ["google", "bing", "duckduckgo"],
      "score": 1.0,
      "position": 1
    }
  ],
  "meta": {
    "query": "python async patterns",
    "total": 5,
    "engines_used": ["google", "bing", "duckduckgo"],
    "cached": false,
    "response_time_ms": 842.3
  }
}

The 9-Strategy Kill Chain

Content extraction is the silent killer of most RAG pipelines. /read doesn't give up on the first failure — it cascades through nine strategies, each tuned for a different class of stubborn URL.

flowchart TD
    Start([URL Request]) --> S1{1. Direct Fetch}
    S1 -->|✓| Done([Return Content])
    S1 -->|✗| S2{2. Readability Parse}
    S2 -->|✓| Done
    S2 -->|✗| S3{3. UA Rotation}
    S3 -->|✓| Done
    S3 -->|✗| S4{4. JS-Rendered Fallback}
    S4 -->|✓| Done
    S4 -->|✗| S5{5. Wayback Machine}
    S5 -->|✓| Done
    S5 -->|✗| S6{6. Google Cache}
    S6 -->|✓| Done
    S6 -->|✗| S7[7–9. Additional Fallbacks]
    S7 -->|✓| Done
    S7 -->|✗| Report[Report to /adapt/report]
    Report --> Loop[Self-improvement loop<br/>re-orders the chain]

    style Done fill:#10b981,stroke:#065f46,color:#fff
    style Report fill:#f59e0b,stroke:#92400e,color:#fff
    style Loop fill:#8b5cf6,stroke:#5b21b6,color:#fff

Most URLs resolve on strategies 1–3. The chain exists for the rest.

Deep Search with Query Expansion

Ambiguous or underspecified queries are the norm in agent workflows. Deep search generates 3–5 variations, runs them all, deduplicates across result sets, and returns a fused ranking.

curl "http://localhost:3939/search/deep?q=best+practices+for+llm+caching&count=10"

Expands to: "LLM response caching strategies", "semantic cache for language models", "prompt caching best practices", and similar — then merges the top results.

MCP: Claude Desktop, Cursor, Windsurf

Six tools exposed over stdio: search, deep_search, read_url, read_batch, news, search_jobs. Plug it into any MCP-compatible client and your agent can reach the open web without custom tool code.

{
  "mcpServers": {
    "agent-search": {
      "command": "python",
      "args": ["/path/to/agent-search/mcp-server/server.py"]
    }
  }
}

Production Essentials, Built In

📡 API Reference

curl "http://localhost:3939/search?q=rust+error+handling&count=3&fetch=true"

curl "http://localhost:3939/read?url=https://example.com/some-article"

curl "http://localhost:3939/news?q=ai+regulation&count=5"

curl "http://localhost:3939/search/jobs?q=senior+python+engineer&location=remote"

curl -X POST "http://localhost:3939/read/batch" \
  -H "Content-Type: application/json" \
  -d '{"urls": ["https://example.com/a", "https://example.com/b"]}'

🐍 Clients & Integrations

Python Client

pip install agentsearch-client

from agentsearch import AgentSearch

client = AgentSearch()  # defaults to localhost:3939
results = client.search("distributed systems consensus algorithms")

for r in results:
    print(f"{r.title} — {r.url}")

LangChain

from langchain.tools import tool
import requests

@tool
def web_search(query: str) -> str:
    """Search the web using AgentSearch."""
    resp = requests.get(
        "http://localhost:3939/search",
        params={"q": query, "count": 5}
    )
    results = resp.json()["results"]
    return "\n".join(
        f"- {r['title']}: {r['url']}\n  {r['snippet']}"
        for r in results
    )

MCP Server

pip install mcp httpx
python mcp-server/server.py

See mcp-server/README.md for remote setup, custom ports, and troubleshooting.

🏗️ Architecture

flowchart LR
    subgraph Clients["🧑‍💻 Clients"]
        Agent["Your Agent<br/>any LLM"]
        MCP["MCP Clients<br/>Claude · Cursor · Windsurf"]
    end

    subgraph Core["⚙️ AgentSearch — :3939"]
        API["FastAPI<br/>Dedup · Scoring · Cache<br/>Rate limits · Auth · Kill chain"]
        MCPServer["MCP Server<br/>(stdio)"]
    end

    subgraph Upstream["🔎 Search Layer"]
        SXNG["SearXNG<br/>:8080"]
        Engines["Google · Bing · DuckDuckGo<br/>Brave · Startpage · Wikipedia<br/>70+ engines"]
    end

    Agent <-->|HTTP/JSON| API
    MCP <-->|stdio| MCPServer
    MCPServer <--> API
    API <-->|HTTP| SXNG
    SXNG <--> Engines

    style API fill:#3b82f6,stroke:#1e40af,color:#fff
    style SXNG fill:#8b5cf6,stroke:#5b21b6,color:#fff
    style MCPServer fill:#10b981,stroke:#065f46,color:#fff

The heavy lifting — deduplication, cross-engine scoring, kill-chain extraction, query expansion, caching, auth, self-improvement — happens in the middle layer. SearXNG handles engine rotation and upstream rate limiting. Your agent just gets clean JSON.

🔧 Configuration

Environment Variables

Search Engines

Edit searxng/settings.yml to enable/disable engines, then restart:

docker compose restart searxng

Running Without Docker

pip install -r requirements.txt
SEARXNG_URL=http://localhost:8080 uvicorn app.main:app --reload --port 3939

Requires a SearXNG instance running separately.

🚀 Running in Production

Things that are easy to forget until they bite:

1. Set AGENT_SEARCH_TOKEN. The default docker-compose binds to 127.0.0.1:3939, but the moment you put a reverse proxy in front, you need auth.
2. Tune rate limits for your traffic shape. RATE_LIMIT=60 per IP is conservative. Bump GLOBAL_RATE_LIMIT first — it protects upstream engines.
3. Enable more engines. More engines = better cross-engine scoring and better rate-limit headroom. SearXNG rotates automatically.
4. Watch /adapt/stats. If a site consistently fails the kill chain, the self-improvement loop will re-rank strategies. Let it cook.
5. Cache aggressively. Default TTL is 1 hour. For research-style workloads, 6–24 hours is reasonable. For news, drop it to 5 minutes.

❓ FAQ

🗺️ Roadmap

• Semantic re-ranking layer (optional, BYO embedding model)
• Redis-backed cache for multi-instance deployments
• Additional kill chain strategies (headless browser pool, Archive.today)
• Prometheus metrics endpoint
• Async Python client
• Postgres-backed adaptation store (currently in-memory)

Have ideas? Open an issue or drop a PR.

🤝 Contributing

# 1. Fork it
# 2. Create your branch
git checkout -b feature/better-dedup

# 3. Commit
git commit -am 'Improve dedup algorithm'

# 4. Push
git push origin feature/better-dedup

# 5. Open a PR

Bug reports, feature requests, and documentation improvements are all welcome. For larger changes, open an issue first so we can discuss scope.

🙏 Acknowledgments

Built on the shoulders of SearXNG — a privacy-respecting metasearch engine that does the hard work of engine rotation and result federation. AgentSearch wouldn't exist without it.

Also inspired by the broader ecosystem of agent infrastructure tooling: LangChain, Model Context Protocol, Ollama, and the many others proving that self-hosted is not only viable, but often better.

📄 License

MIT — do whatever you want with it. See LICENSE for details.