Linkok.com – modern broken link checker I've been working on for 3 years
linkok.comHi HN! I'm Bruno, the founder of https://linkok.com. linkok.com is a modern broken link checker.
I got the idea before the Covid pandemic: at the time I was looking for a better senior developer job. I sent a couple dozen job applications with my resume website, and I only got a couple job interviews, and no offers. I gave up and stayed at my old job. A couple months later I was optimizing my website. I checked for broken links and the results were bad: from 12 links total, 4 were broken. That's more than 30% broken links - no wonder I couldn't find a new job!
Since then I realized broken links are a problem for pretty much every website out there. If you have a website, you likely have broken links too. And the existing solutions don't cut it. Most online broken link checkers are old, and have outdated interface. Advanced SEO tools (ahrefs, semrush etc) can do broken links checking, but are too complex and expensive for regular people.
Under the hood linkok.com crawls sites using Async Ruby, which has proven great for this type of work. I love how easy it is to work with (when compared to threads). It also made some advanced features easy, like request retries. Async is a somewhat underused in the Ruby community. If someone has more questions about it, I'd be glad to answer.
My goal for linkok.com is that it should be easy to use (think: blogger moms), and it should be accurate and powerful enough for enterprises. linkok.com is free forever for small sites (up to 100 internal pages) and charges a fee for bigger sites. It's also completely free for open source, educational and non-profit websites. This is not automated yet, reach out at bruno.sutic@linkok.com and I'll set you up.
You may find this interesting: I worked on this project for 3 years! By far the hardest part was billing - integrating with Stripe properly took me almost 1 year (Stripe is very hard). I also shaved a couple of yaks, highlights: implemented postgresql and redis high availability, I run things bare metal servers (Hetzner).
I'd love to hear your thoughts, feedback and ideas! Looking forward to your comments.
Congratulations on shipping, the site looks great. I've been working on a desktop app in the same space (website health/administration). It's been a similarly long journey. I've found marketing to be an uphill battle. It's a crowded marketplace, which in many ways indicates a healthy ecosystem, but difficult to crack nonetheless. Having a focused audience (e.g. bloggers) will help there. If you ever want to discuss marketing or talk shop, definitely reach out (ben at hn-username dot com)... maybe we could both learn something.
Quick question: how do you handle bot protection (eg Cloudflare) with your app?
I'm not sure I've the best antibot solution, but it's handled through some crawler options exposed to the user. Out of the box, I use a (fast) http crawler with my app's user-agent. It is not at all resilient to antibot. I direct users who are encountering issues to first try a user-agent override, and if that doesn't work, to next enable javascript crawling (think headless chrome), which is slower and heavier, but clears up a lot of issues. I don't have a strategy for aggressive antibot (captcha/etc.) other than to tell the customer to dial it back on their website.
Edit: I've seen antibot SAAS providers, which claim to provide workarounds at a cost. You route traffic through their network, and they have teams that are constantly tweaking things to keep the requests working, much like scrapers adapting to website redesigns. It would be a treadmill to do on your own. There more info on this at https://substack.thewebscraping.club/ In my case, selling single-user perpetual licenses, it doesn't make sense.
Yes, I thought you played with those apps and their proxies to get around anti-bot protection.
I also don't have anti-bot implemented right now, but that's my next step. I mean, my app "has one job" and it's not doing it well because of anti-bot protection...
I think with the distributed nature of a desktop app (running headless chrome), outside of cloud infrastructure/IPs, the internet is generally less defensive. It has bought some leeway, but I definitely should look into proxy support.