Show HN: Nerd Crawler – monitoring original comic art sites so you don't have to
nerdcrawler.comI've been a fan of comics since I watched the X-Men Animated Series in the 90s, and I fell in love with collecting original comic art when I got my first Jim Lee sketch in high school.
But, after missing out on some original comic art pieces because I didn't know when they were added for sale on websites, I decided to take it upon myself to make an app that monitors original comic art sites and emails/texts you when new art drops.
It's called Nerd Crawler and I'm building it myself so there might be some bugs but I'm hoping it helps comic art collectors. It works with over 40 original comic art websites like Albert Moy (Jim Lee's art dealer), Cadence Comic Art, Artcoholics, a bunch of Big Cartel sites like Jim Cheung / Jason Fabok / Dustin Nguyen, Greg Capullo Art, Skottie Young, and more.
It's free to try @ https://www.nerdcrawler.com/, and you can upgrade to a paid plan if you want text messages alerts or want to check sites every 10 minutes or 1 minute.
From a technical standpoint, my tech stack is:
- Ruby on Rails
- Hosted on Heroku
- Emails sent by Mailgun
- Texts sent by Twilio
- Images hosted on Cloudinary
- Credit card charging handled by Stripe and the new, low-code Stripe Checkout
The minimum viable product was built in about a week with minor bug fixes and new features added weekly.
If you have any feedback, have art sites you wanted added, or questions, let me know! Definitely interesting idea! Congrats on shipping. My two cents: Switch up the pricing plans. I certainly would never pay $50 per month (and my gosh $100?!?) for this service, but I may want to watch more than 10 pages your $10 plan has. My suggestion, is to just simplify all the pricing. Get rid of all the tiers. Keep the free and $10/month tiers. Make the $10 tier do unlimited pages. No one needs 10 min or 1 min updates. I think do hourly updates on the $10 tier. I think you'll find that more people are willing to subscribe if you bring the price down while providing more value for that. Thanks for checking it out and for the feedback! I will probably keep the minute and 10 minute checks because some pieces drop and are sold in under 2 minutes of dropping (Greg Capullo, Jim Cheung, and Paul Smith in particular sell sketches insanely fast)! I think I slightly misunderstood the point of this site, people probably do want fast updates. But I still think the price should be brought down. You could probably save costs by pooling the crawling, eg. if multiple subscribers request the same page, do that crawl once for all of them. You may already be doing that. RE: pooling - yup, pooling for the win! As a self taught programmer, this was one block of code I was particularly proud of :) Crawling too fast from the same IP addy can get that IP blocked. Per 10 mins prob ok, per 1 minute may be too much depending on the volume being downloaded and the sensitivity (you mention saleable stuff in another comment so this might well make a difference). FYI. I did a reasonable amount of scraping myself. (err, crawling = scraping? or maybe they different things) Edit: tidy + clarify Having higher priced tiers works the other way, more people subscribe because it seems like a good value There is an enterprise plan? Would be amused to hear if that ever gets picked :-) Haha current number of enterprise users is a big 0, but a guy can dream! Interesting idea but as an original art collector, I form relationships with the dealer and try to get an in with new art before it hits the web site. Yeah I think the dealer relationship is super important to get pieces and commissions, but so many artists drop new sketches and pages randomly on their site so I’m hoping this will help people like it’s helped me! Are comic drops valuable? So professionals benefit? Or is it purely serving hobbyists and comic artists? I love the concept and tech. So many applications Comic drops can be valuable if there's a rare print run, but what Nerd Crawler focuses on is the original art for comics. For example: there might be 100,000 copies of a 22 page comic called X-Men #50. Drops for the 100,000 are less valuable, but when 1 of the 22 pages goes on sale then it's REALLY valuable because once that 1 page is done you can't get it again until the new owner sells it. Nerd Crawler exists to let you know when the artist puts that 1 page on sale so you can get it. Isn't this a more complicated RSS? Not exactly. I don't store anything; instead, I just send an email and/or text alert to a user if an update is detected on a page they've selected. The main gap it's solving for is that great original comic art is usually sold within minutes of dropping so a passive RSS feed isn't great. You need the alert quickly to decide if you're buying or not so the email/text alerts help with that. The website design is damn so good!!! Pure love to navigate such website! This looks like RSS with text alerts… Personally, I toss RSS feeds into newsboat and check it once a day. I want to get away from “always on alert” and take control of where and when I consume media.