TinySearch: Let Small Local LLMs Search the Web Without Burning Context

I have been playing around with local LLM agents a lot lately, as I use them as part of my setup with my home lab. We aren't talking the big boys here, mostly smaller models, the likes of Qwen3.5–9B, Gemma-4-E4B, local setups, Cline/Roo style workflows, MCP tools, that whole world.

And honestly, one thing kept progressively annoying me as the months went by..

Web search technically works, sure. But a lot of the time it just dumps way too much random page text into the model’s context. You ask a simple question, the tool grabs half the internet, and suddenly your tiny local model is sitting there trying to reason through scraped menus, cookie banners, broken markdown, irrelevant paragraphs, and five pages worth of SEO junk.

Of course, you can go paid, and have a go at the likes of Perplexity and Exa, and while these tools are admittedly good, they still suffer from some of these same issues while costing real money once you add them into agentic flows that run 10s to 100s of queries per day. Btw, I do actually recommend the API from Perplexity, if you have the money for it, you get very high quality results for your buck. But for many of us, this is not an investment thats worthwhile.

So, for smaller local models, running in home labs, we can agree this is genuinely painful.

They do not just need access to the whole web and huge, unfiltered page dumps, they need access to a very specific slice, relating to a very specific query.

So I built TinySearch, my version of the solution.

GitHub: https://github.com/MarcellM01/TinySearch

The basic idea

TinySearch was created (and is being continuously developed every time I have some free moments) as a small open-source MCP research tool that searches the web, crawls pages, reranks the content, and returns a compact source-grounded prompt that your model can answer from.

Its not a giant search backend (that's handled by duckduckgo and more to come).

Nor is it some massive crawler that wants to become your whole infrastructure layer.

Its simply just:

search -> crawl -> rerank -> return a grounded prompt

That's it, that's the whole flow.

What's important to understand conceptually is that TinySearch does not directly answer the question itself. It prepares the evidence. Your actual client model then uses that evidence to answer.

That matters because I do not want another LLM layer doing summaries on top of summaries. I want the agent to receive clean, ranked, source-attached context and then reason from there.

Why I built it

The original pain, as I was talking about above was simple: I wanted local agents to have web research without the insane context overhead.

When a tool dumps entire pages into context, it creates three problems.

First, it burns tokens for no reason. Even with cloud models, that means cost. With local models, it means slower prompt eval and worse performance.

Second, it confuses smaller models. They are already more sensitive to noisy context. Give them too much low-signal text and they start missing the actual answer.

Third, it makes agent workflows feel heavier than they need to be. Sometimes you do not need a whole search product. You just need a lightweight research layer that grabs a few good sources and gives the model the useful parts.

TinySearch is built exactly for that kind of workflow.

What it does under the hood

The current flow, in more (but still digestable) detail, looks roughly like this:

user question
   ↓
DuckDuckGo HTML search
   ↓
rank search results with dense + BM25-style scoring
   ↓
crawl selected pages with Crawl4AI
   ↓
chunk the extracted markdown
   ↓
rerank chunks globally
   ↓
dedupe and apply source quotas
   ↓
build a source-grounded answer prompt

The important part is that the final output is not just “some scraped text.”

It is a structured prompt with the original question, today’s date, instructions, URLs, search previews, and the most relevant extracted chunks.

Basically, TinySearch shrinks the web into something a smaller model can actually use. Below you can see a demo of a model searching for a classic “Basel III Updates” question, something one of my local agents actually searched while compiling a financial analysis report.

Why not just use a search API?

You can, and fully disclosure, for many serious production cases, you probably should.

But that's not really where TinySearch lives, its not trying to replace commercial search APIs, persistent crawlers, enterprise search systems, or production SLA-backed infrastructure. That would be cope lol.

It is more for the annoying middle ground where you want web research, but setting up an entire stack stack feels like overkill.

Stuff like:

• local agents
• MCP workflows
• small RAG experiments
• personal research tools
• prototypes
• coding agents that occasionally need current docs
• smaller models that cannot handle context bloat

That is where TinySearch makes sense.

It gives your agent a small research layer, just as simply as you would hook yourself up to the perplexity api, the other major difference being that everything you, ore your agents search stays entirely local. I mean duckduckgo still sees those naughty naughty agentic searches, but they don't go through a third provider, thus one less pair of eyes ;).

MCP first, but not MCP only

TinySearch runs as an MCP server, so you can plug it into MCP-compatible clients.

The quick Docker version looks like this (I really recommend this way of usage, its been made as simple as possible, if you have docker installed just go with this):

docker run --rm -p 8000:8000 \
  -e MCP_TRANSPORT=streamable-http \
  -e MCP_HOST=0.0.0.0 \
  marcellm01/tinysearch:latest

Then you connect your MCP client to:

{
  "mcpServers": {
    "tinysearch": {
      "url": "http://localhost:8000/mcp"
    }
  }
}

TinySearch exposes one simple tool:

research(query)

You pass the user’s question as-is, and TinySearch handles the search/crawl/rerank/prompt-building flow.

There is also an optional FastAPI server if you want to use it over HTTP instead of MCP.

Local embeddings or API embeddings

One thing I wanted from the start was flexibility.

TinySearch can use local ONNX embeddings, or an OpenAI-compatible embedding API.

ONNX because I am biased against their format and not afraid to admit it lol. For CPU usage, you cant really beat their efficieny and cross system compatibly. You dear reader do NOT have to worry about this, its all handled internally.

So you can keep things fully local, or you can point it at an embedding provider if that fits your setup better.

The repo currently includes local presets like:

fast      -> all-MiniLM-L6-v2 ONNX
balanced  -> bge-small-en-v1.5
quality   -> bge-base-en-v1.5

The nice thing is that you can start simple and then tune it later.

Search depth, rerank weights, chunk limits, crawl concurrency, tokenizer settings, embedding backend — all of that can be configured.

But the default idea stays the same: return a useful research prompt, not a landfill of raw page text.

What the output looks like

The output is designed to be directly usable by an LLM.

Roughly:

================================================================================
SEARCH-GROUNDED ANSWER PROMPT
================================================================================

QUESTION
What are the latest Basel III updates?TODAY
2026-05-18CRITICAL INSTRUCTIONS
Use only the text under RESULTS.
If the answer is not supported, say the results are not enough.
Cite source URLs after factual claims.RESULTSRESULT 1
TITLE
...URL
...SEARCH PREVIEW
...RELEVANT TEXT
...RESULT 2
...

That format is intentional.

I want the final model to know what it is allowed to use, where the evidence came from, and when the search happened.

This matters especially for questions with “latest,” “today,” “this year,” or anything where dates can mess up the answer.

What TinySearch is not

• TinySearch is not magic (duh).
• It will not guarantee perfect search coverage (but I firmly believe no one can).
• It does not build a long-term index (speed and privacy!!).
• It does not replace a serious production search system (for that my friend you need to pay up).

And honestly, that is the point.

The goal is not to be everything.

The goal is to be small, inspectable, and useful enough that you can throw it into a local agent workflow and immediately get better web research without context-window abuse.

Why I care about this problem

I think a lot of local agent work is blocked less by model intelligence and more by the surrounding harness.

People put too much focus on the model, but then the model gets wrapped in tools that were clearly designed as if context was free and every model had infinite attention.

That works badly for small models, but even large models benefit from this, as I have hooked this thing up to multiple models on open router, very large ones at that such as Kimi K2.6 with good results.

LLMs needs cleaner inputs. they need less junk. They need tools that understand that context is not a trash can.

That is the actual thing TinySearch is trying to solve.

Not “web search for AI” in some huge abstract way.

More like:

Can we give a local model just enough high-quality web context to answer properly, without burying it alive?

That is the name of game here.

Try it

If you have made it this far, I thank you, and with a subtle nudge urge you to try this tool:

https://github.com/MarcellM01/TinySearch

It is still early, but it is already useful in my own workflows.

Feedback is very welcome, especially from people building with:

local LLMs
MCP
Cline / Roo-style coding agents
RAG systems
small model workflows
personal research agents

And yeah, roast it too.

That is usually where the useful feedback is anyway.