Show HN: Stork: A customizable, WASM-powered full-text search plugin for the web
stork-search.netI've been working on Stork for just over a year! I recently wrote a blog post about what building it has been like -- if you're interested in the project, definitely check the blog post out!
I have been following and using this for a while and it works great for small to medium sites.
I built from source and self-host.
Congratulations on gettiing it launched James!
Interesting project!
The fact that I have to fetch the stork.js from your CDN, and a random wasm blob gets pulled in as well, is unfortunately a deal killer for adoption, at least for me. A pointer to how to self-host might help address this.
Self-hosted Stork is definitely possible, and definitely something I want to build! I've been focusing on a different type of user for the 1.0.0 (the user who wants to paste in a single script tag), and there are still some last experience improvements I want to make for those folks, but I completely recognize the value of self-hosting code like this and want to build something into Stork that will fill that niche, as well.
For now (and I haven't tried this for self-hosting, so use at your own risk) -- the whole project (incl. the final JS file and the WASM blob) can be built from source. Tweaking the build a little bit would probably get you most of the way there.
I'm visiting my parents on a low speed connection and it didn't work at first. I had to clear the cache and restart. I don't know if it's related to this, but I suspect so.
I hate it when stuff is marketed as magic, like in this case "impossibly fast web search". It's like marketeers aren't even trying to cover up their lies anymore.
If you look elsewhere in the thread it seems like a solo individual worked on this for years and posted it here today. Your comment is needlessly negative and mean.
IIUC, it's "impossibly fast" if you assume the search -> see results flow requires a network round-trip (a very common assumption).
It's actually impossibly slow for me, as it didn't work at all on Firefox or Chrome. Chrome showed a progress bar but no results when searching for the recommended "liberty".
Similar project: https://endler.dev/2019/tinysearch/
Like the idea but nothing happens when I type in the search box using ff85 (beta 4).
I get nothing either, on stable Firefox (84).
Oh no! If it's any consolation, I think it might be the site itself, not the search plugin... the site has been acting up in unrelated ways. The site desperately needs a rewrite, but I didn't have time before this launch date.
Refreshing might work, but no promises? Or you could try my personal site's search bar: https://jameslittle.me/search/
It takes some time to download the index though.
That maybe ignoble in normal use case but I tried your website solely for searching, just to say.
Same for me, Brave on Android. Your personal site works though.
That other site works, your suspicion is probably correct.
Same here- Firefox 84, search doesn't work on the OP but does work on the link above
Apologies for this! Someone's very kindly put up a PR[0] that I suspect will fix this, but it's past my bedtime so I won't get a chance to review/merge/relaunch it until tomorrow.
How much does WASM matter for performance, or is it more about the convenience of only having to write Rust?
Does the query engine support any boolean operations, or is it just finding documents that contain every (stemmed) input word?
Nice. I wonder how it works with around 8000 pages of content, will give it a whirl in a few weeks.
I'll keep my eye on this. It sounds like this runs in the browser itself.
Right now I'm using meilisearch for a project and would prefer to not use up a http port for the search engine. One thing I'll need is fuzzy search though.
Fuse.js (https://fusejs.io/ ) has been super simple for client side search for me, I’ve used it in a couple projects now.
Fuzzy search is definitely on my list! Keep an eye out in the near future...
Can stork.js be served locally rather from the domain https://files.stork-search.net ?
How does it compare to lunr.js?
OP - congratulations on shipping! WASM-powered search caught my eye.
Looks like the search index is downloaded and used locally in the browser, so this is as fast as search can get. One trade-off though is that you're limited to relatively small datasets. While this shouldn't be an issue for small-medium static sites, an index that needs to be downloaded over the wire will affect your page performance for larger sites / datasets.
In those cases, you'd want to use a dedicated search server like Algolia or its open source alternative Typesense [1]. Both of them store the index server-side and only return the relevant records that are searched for.
For eg: you'd probably not want to download 2M records over the wire to search through them locally [2]. You'd be better off storing the index server-side.
[1] https://github.com/typesense/typesense (shameless plug)
Just a quick non-thought-through idea but would it be possible to build an index in a way that allows clients to download only parts of it based on what they search? I.e. the search client normalizes the query in some way and then requests only the relevant index pages. The index would probably be a lot larger on the server but if disk space is not an issue...?
(Though at some point you have to ask yourself what the benefits of such an approach are compared to a search server.)
Merkle Search Trees: Efficient State-Based CRDTs in Open Networks https://hal.inria.fr/hal-02303490/document
Peer-to-Peer Ordered Search Indexes https://0fps.net/2020/12/19/peer-to-peer-ordered-search-inde... (which adds useful context about the above)
Sorry for just dropping these links, I should already be asleep :)
I keep wanting to try that! Sharding the index would be really cool, and would definitely allow for a seemingly infinite index size. The barrier, as you were getting to, would be making sure index shard downloads could keep up with typing speed.
Definitely something I'll try experimenting with in the future, so... watch this space, perhaps?
You as well as your business partner are so creative at hijacking threads that it is almost cringeworthy :(
Typesense itself might be a great product but but please stop devaluating it in trying to hijack every search related post.
As an introvert and an engineer it’s definitely not the easiest for me to talk about something I’m working on openly. So I do cringe every time I mention Typesense in contexts like this thread.
But it seems like every time I mention it, new people discover it and there continues to be interest in the community for an oss search product. I’m sorry if you saw me mention Typesense too often.
IMO your post was fine and informative, I certainly appreciated it. As someone developing something similar, you would have thought about the problem a lot more than the casual reader and be able to provide insight others might not. I can relate to that.
However, it wasn't clear to me that you were involved with Typesense. Re-reading it now I spot one reference at the very end. If it was instead tied in with your first reference it would come across more honest and make your observations more appreciated.