Ask HN: Best Framework to build Reddit in 2021?
I am planning to build something like Reddit for our Region (it will mostly be about regional topics).
I found out Reddit is getting 250M comments per month and though we will never be this big, I am wondering what technology stack we should use to build Reddit to overcome scaling challenges.
I believe the DB will be Mongo but what can be the right web stack? Maybe spez can chime in, but I believe the first Python re-write of Reddit 1.0 was in Pylons. The key was pre-render every page, no ajax, just pagination, and sacrifice page "freshness" for stability. Modern equivalent is something like static NuxtJS deployed to Netlify, but with tight re-generation loops. At <1M writes per day, its an interesting challenge if the site is regionally limited, and you are trying to self host and avoid cloud costs. And no one really does old school caching any more as its all React / infinite scrolling. Best of Luck ;) Ah, infinite scrolling, really hate it... There are existing open-source clones with sites that are active. What are they doing wrong? https://gitlab.com/postmill/Postmill As far as I know, reddit uses postgresql with heavy partitioning. I personally would stay far from MongoDB because of personal preferences and try something else, like Citus Hyperscale for Postgresql. Then, I would make a prototype using hasura.io and something like next.js My backup plan would be to go with Ruby On Rails because it's actually still very good. You need to work out where you sit on the efficiency vs development speed and your architecture. If your focusing on development speed to be able to add features and trying to grow quickly, then your should lean towards where existing skills are, but design it in a scale out fashion. i.e. break out your state from your http layer so you can run as many front end instances as needed to handle the load. Work on getting the data layer optimized as far as data access patterns go, and serve most of your data from a caching layer. Keep in mind that a web app like reddit spends most of its time dealing with your data layer. A Higher performance language won't fix a slow data source. If your optimizing for efficiency (hosting and cost), then it might make more sense to start with a higher performance language, but at that point you likely want to optimize for your hosting environment. Can you make your application highly static? Can you make the pages be generated on change? i.e. new comments / posts go into a worker queue and a static HTML page is generated (possibly with some small dynamic content being pulled in from another microservice). In this architecture the web stack is dictated by your hosting environment. For AWS this would be whatever runs best in lambdas. Something like Cloudflare workers enforce JS or Web Assembly. You also need to ask yourself if you want to develop the code or the site. If your more interested in the application then you should look for something that already exists that is close and see if you can make it work for your needs. The site likely will need a lot of work to attract a userbase, and your likely better focusing on that unless the goal is to develop the software itself. Use the framework you know. Don't use MongoDB. Use PostgreSQL or something. Don't worry about scaling yet, just get an MVP out. You should not be thinking about scaling at this point. Just build the thing and go from there. The chances of you getting enough users for 1000s of comments per minute are low. Worry about that problem when you get there, it's a good one to have. what do you know? use that. Also you could start with reddit's latest opensource version. I can use Ruby on Rails but I don't think an RoR app can handle thousands of comments per minute in an optimised way. I run a midsize app in Rails 5 and already having lot of scaling issues. Iirc Reddit stopped open sourcing their code from 2017 and I guess understanding and running that 12 year old spaghetti code wasn't worth it anyway. I don't see anything useful on their github page too https://github.com/reddit Rails will handle "thousands of comments" just fine. If you are hitting scaling issues with a midsize Rails app you probably could look into how to optimize. Throwing Mongo at the problem is not really a solution. GitHub and shopify are written in RoR afaik, many millions of requests go through them just fine Its not like reddit improved since 2017... they added silver, copper and a sucky new interface but I dont think that helps anyone. Worst possible advice, if you pardon my borderline rudeness. That's what most programmers do, following famous saying that for someone with a hammer every problem looks like a nail. Ops problem is more architecture and scale, IMO, as opposed to what coding language or JS framework. I’d start with Postgres on the back end and then work with scale as needed over time. The service can be written in pretty much any language like Kotlin, Java, Python, or even Rust if you want. Personally I’ve never used Swift server side. IMO the worse option is to try to over optimize and use some shiny technology that you don’t understand how to use. This is how we end up writing iOS apps in Kotlin despite Swift being very similar and built for iOS development.