Sloth search for Ruby Weekly – a 100 minute hack turned 20h open sauce project
slothfinder.apishop.ioHey there,
Took me really long to understand what this is... let me see if I get it right:
* You scrape rubyweekly.com into Sqlite entries.
* You provide a simple search engine on top of that.
1) Less is more, you can simply state that on your page ("Search an index of scraped rubyweekly articles).
2) If I get it right you are doing regex match for keywords, but maybe you could rely on FTS5 since you are using Sqlite, you may be able to get better results faster, maybe with less code? ([1] random article I found about rails+FTS5).
Cheers!
--
1: https://mariochavez.io/desarrollo/2023/09/01/full-text-searc...
P.S. it just occurred to me that if the aggregated links are less than 1mb of data or something like that you could try to bundle all data in a single JSON object and search in the frontend with a library like flexsearch [1]. That could be even less work than running a minimal rails backend.
--
They don't even need to output to JSON - https://phiresky.github.io/blog/2021/hosting-sqlite-database...
The entire thing could be a static site using the above method -- even if it's many hundreds of megabytes in size!
Sloth Finder helps you encounter the most amazing weekly Ruby articles around your favorite Ruby and Rails topics for the past decade sourced from [Ruby Weekly](https://rubyweekly.com/). This tool was made because the creator, a Sloth in human form, was interested in all the greatest articles around his favorite weird Ruby niche, so he built a primitive search and looked for:
``` api, openapi, automation, rest, graphql, rpc, soap, webhook, scrap, event-driven, serializ ```
What followed was a wealth of articles full of knowledge, understanding of his niche, and potential for connecting to potential leads and partners. Instead of calling it a day, he decided to share the tool with us, the people (which basically means his 100 minute hack turned into a 20 hour project if you include all the side quests and write-ups related to it).
https://slothfinder.apishop.io/
He has also open-sourced it on GitHub. Here are some reasons for you to look at the code:
- You want to see how to deploy a Rails project on the cheapest [render.com](http://render.com) non-free plan running on SQLite (making it cheaper than usual). After I already embarked on render.com, I found that [someone else documented how they experimented with something similar and ran Python on SQLite on fly.io for basically free](https://github.com/irskep/cheapo_website), so this might be another solution if you are interested in hosting your hobby projects. - You want a simple Ruby scraper template to scrape simply structured sites like xlcd.com/1, xkcd.com/2, etc. - You wanna see a logger that writes to STDOUT and logs (useful for local scraping if you want to watch progress but also write to logs for search purposes later). - You want to see a Turbo stream in action. - You want to see how a „first thing that came to mind search over a small ActiveRecord collection“ was implemented.
Ok writing first-person again. If you find the search useful, let me know if you would like something similar for the other publishings of cooperpress(https://cooperpress.com/publications/). They have top notch stuff on JavaScript, Postgres, and much more! I’ll also take suggestions for other scrapeable publishings that don’t have a search.
Enjoy