Building a Search Engine for 6,000 Personal Blogs

2026-08-03

TL;DR: Semsearch is an embedding-based search engine whose corpus is bootstrapped from indieblog.page's blog list. It aims to make indie blogs more discoverable. Results are ranked by semantic relevance alone, without link popularity or SEO-derived signals. At the time of writing, the live instance is up at semsearch.blog with roughly 6,000 unique blog sites, 154k pages, and 1.4B tokens. The source code is at github.com/yikerman/semantic-search.

Motivation

The idea for this project emerged when I read about The Great Blogging Collapse (HN discussion). While the post itself has a couple of problems, its broader point resonated with me as a blogger: there are fewer ways to promote small and individual blogs. In the past, many visits began when Google led users to your blog. But Google, as shown in The Great Blogging Collapse, is doing so less and less for various reasons, such as the rise of AI summarization that completely eliminates visits to other sites.

This aligns with other sources: people (including me) prefer searching on Reddit since Google's top results are cluttered by ads and SEO-optimized content farms. For example, a Google search for the old Postgres vs MySQL debate returns mostly AWS and IBM articles that are of little use. (Interestingly, at the time of writing, it seems that Google has boosted Reddit's ranking and put it among the top results.)

While the debate has largely settled, this would have been the perfect scenario to display some old-n-wise engineer writing about his experience with both in production, even if his webpage is styled uglier than The Motherfucking Website.

Recent embedding models have made it practical to search by meaning rather than exact keywords (which also requires non-trivial tricks to work effectively), and vector databases have also proven to be practical and easy to use. While most existing embedding-based search systems are used to provide agentic LLMs with retrieval capabilities, I want to explore the possibility of using purely embedding-based search to help people discover indie blogs.

Implementation

I used FastAPI for the web app and Postgres + VectorChord for the database. The set of blogs is bootstrapped from indieblog.page's blog list. For each page discovered through either an RSS feed or a sitemap, I used Trafilatura to extract the text content, then chunked it into overlapping 384-token segments. For the embedding task, I chose Qwen3-Embedding-4B as the embedding model, since it topped the MTEB leaderboard at the time of writing, is open source, and is relatively cheap to run. Every chunk is stored in the database as a 2,560-dimensional halfvec.

At query time, Semsearch embeds the query once and runs designated retrievers. By default, only the dense retriever, based on the cosine similarity between the query embedding and chunk embeddings, is used, and the top 64 entries are returned. There is also an optional BM25^[1] retriever for both evaluation and combining results through RRF^[2] (Anthropic claims that the hybrid approach performs better^[3]). For details, see the search pipeline and the SQL doing the heavy lifting.

Deployment

Deployment is a bit tricky, since the HNSW index takes about 30 GB of RAM to load at the current scale, and VPS costs scale a lot with RAM. For reference, an r6a.xlarge instance on AWS costs $0.252/hr. I ended up getting a dedicated i7-6700 box from Hetzner's server auction for €55/mo. While the CPU is crappy by modern standards, it comes with 64 GB RAM and 2 × 512 GB NVMe SSDs, an absolute bargain in the RAMageddon era.

I used OpenRouter's API to run the embedding model, since it is relatively cheap, namely $0.02/M tokens. Indexing the current corpus cost a modest $28.60, which is not bad. The latency and throughput are not ideal, though: I get roughly 6,000 tokens/sec on OpenRouter, and it bottlenecked the initial indexing process.

Evaluation

So how do the results look? Going back to the earlier query of Postgres vs MySQL (archived), the results are mixed and sit somewhere between useful, fun, and nonsense.

• The first result is a funny not-quite-a-failure. It is a cached Hacker News comment (why are you caching all the content you bookmark on your blog, Nitsikopoulos?) from a submission about a company migrating from MongoDB to PostgreSQL. The comment compares operating Postgres and MySQL at scale (while the submission itself is, indeed, about MongoDB), which explains the high similarity score. The search works and provides deep insights, but the presentation can be confusing to users in certain cases.
• The 2nd and 3rd results are indeed very relevant, but the writing is thin and of mediocre quality.
• The 4th result is an old blog post dating back to 2002 and is the kind of fun content I would like to see. It is obscure and based on firsthand experience, and much of the technical advice is now historical.
• The 5th result is probably the strongest answer overall: Roztropiński has a detailed comparison of the two, with some deep dives into their storage models and some benchmark code.

After fiddling around, I found that it gives more insightful results when my query is an assertion a blogger might make in a post, and that longer queries resembling a sentence work better. Some extra interesting examples I tried:

• what homelab brings to a people (archived) yields some funny anecdotes about Nicole's overpowered homelab, and also some low-quality content farms.
• the camera I kept for years (archived) introduced me to Henry's Canon PowerShot A510 and the beautiful pictures taken with it.
• For working remotely made me lonely (archived), I expected to find some COVID-19-era blogs, but was surprised to find Jasdev Singh's 2015 writing at the top.
• why you should use Linux Desktop (archived) returns loads of convenient links to send to your friends and lure them into using Linux. Also obligatory: Windows 11 sucks.

In conclusion, the search definitely works and offers a more personal side of the Internet from time to time. As expected, quality heavily depends on the corpus, while the search can only suggest relevance. The frontend can be confusing sometimes, since it only displays the title and the first few lines of a post, and the user may have a hard time locating the relevant chunk in the original post again.

Limitations & Future Work

This is still an early proof of concept, and some quirks exist. A notable one is that the embedding model is not very robust to slightly nonstandard phrasing. For example, the query "framework laptop review" (archived) returns gibberish results, while the properly capitalized query "Framework Laptop review" (archived) returns relevant results. The website would also benefit from displaying relevant chunks in the search results as discussed above, and some community voting mechanism may be useful to filter out low-quality content (though it will introduce other problems).

I plan to build a homelab and get a 3090 for self-hosting and further experiments, namely:

• Assemble a standardized evaluation dataset and test the following approaches against it.
• Test more chunking strategies, such as the late chunking method^[4].
• Test other embedding models and address the quirk mentioned above.
• More blogs! A potential source is Kagi Small Web.