Is Your Site Hacker News Ready?
brandonsavage.netIf you're hosting some sort of blog post, the best way to ensure that all goes well with high traffic is to make it static. Something like jekyll works great for this (publishing especially), and you can host any static site with a node server in like 3 lines of code on nodejitsu for $3/month.
Nodejitsu can handle hacker news traffic without even the slightest blink, with one dyno - I had a post up on the homepage for hours with over 200 concurrent visitors and like 13,000 total for the day and never had a single issue. Again, $3/month for that.
No need to over-complicate things : )
Really, I don't know why dynamic languages became so popular for blog platforms.
It's an overkill all you need is static content.
Non-programmers find it much easier to log into a site, type in a text box, and click 'submit' than to run some sort of static site generator.
If you're at the stage where you're maintaining your own server anyway, it doesn't seem like too much to ask.
Static websites are glorious. Much more secure, much faster to load, far fewer things that can go wrong. Cheaper, too, since they require fewer resources. And, as luck would have it, there's a perfect Hacker News thread to get one started:
Or, if you really want or need a dynamic website, throw CloudFront or similar in front of the site.
Then lock down your CMS at a secret, SSL-only URL.
It's about 2 lines of Ruby or PHP to ping CloudFront and request an invalidation when you change a page (which only happens rarely anyway.)
Secure, fast, dynamic, only goes down if CloudFront does.
I agree, but we weren't talking about maintaining your own server, just static languages and blog platforms.
I use Jekyll/Octopress all the time, I totally agree, static sites are great. But non-programmers are never going to use it.
To add to that. You can add a whole lot these days via Javascript. You want to get some comments? Just embed a Disqus widget. Want some feedback? Embed a getsatisfaction widget.
I'm constantly amazed at how poorly Wordpress is able to perform. 2.5GB virtual server with a cache in front of it and it went down?
My blog is a Django app running on a 512MB Rackspace VM with a basic Apache, mod_wsgi, and Postgres setup. I have a dozen low traffic applications all running on the same server alongside it. I have zero caching set up so every hit is doing the full process of querying the database and pushing the results out through a template.
I was on the front page of HN last year and the server load was negligable. I have graphs: http://thraxil.org/users/anders/posts/2012/06/02/How-Django-...
I bet if you switched to nginx, your server would use even less resources to handle the same amount of traffic.
Having used nginx on other projects, I'm sure you're right. My blog setup is pretty old and hasn't really needed any improvements.
On newer deployments, I'm still using Apache+mod_wsgi to manage the Django processes (it's a setup that has been reliable for me and that I know how to debug), but I'm sticking an nginx proxy in front as a matter of course. Not for performance reasons, but just to leave myself a hook for handling later changes. I can trivially spin up additional application backend servers and load balance across them, or quickly flick a switch to have nginx cache a particular resource in an emergency.
He didn't have a cache in front of it - he had W3 total cache installed, but not enabled for object caching.
If he'd had W3 total cache, or varnish (my preferred option) in front of it, he wouldn't of noticed the impact.
There's a bigger question of why Wordpress still doesn't ship with sensible caching enabled by default, backwards compatibility with some plugins just doesn't cut it for me.
Mostly as an educational exercise (perhaps in futility) I decided to try writing my own small php framework specifically to build and serve static files, load external libraries only when necessary and automatically send caching headers. The point being, for instance, not to even load Dwoo except if compiling the static pages (as opposed to fetching data from a view, building them and serving them with each request.)
Now i'm wondering if i'm not both overoptimizing and missing the point entirely.
And yes I do realize 'custom php framework' is missing the point of frameworks and that php is an abomination. Getting Laravel to do what I wanted (awesome as it is, and it is) started to seem like trying to tweak a cannon into a flyswatter.
It's still fun, but I dread it ever getting posted here. Assuming it gets finished.
Building a static site generator is always a good learning exercise.
NginX + php-fpm + APC did the trick for me, at one time even in an EC2 Micro instance under heavy load.
There is a very nice hack I applied to my WordPress installs, but it's not for the faint hearted: hack the WP index.php to serve pages directly from Memcache for all non-authenticated GET requests. The beauty of this is that for most requests, the WP environment (which is horribly bloated) does not even get loaded.
It works about an order of magnitude better than all those cache plugins, but the downside is of course that software updates require some special care.
You could just pop Varnish cache in front of Wordpress to do the same thing - any unauthenticated GET's will just hit Varnish, and come nowhere near Wordpress itself.
That way you wouldn't need to worry about software updates.
Totally. I guess I could also use some advanced Nginx directive/plugin to do that. The hack is just something that occurred to me spontaneously, it took about 2 minutes to implement, and I didn't even have to install anything :)
You could give these 2 config files a try sometime:
https://github.com/ewanleith/Wordpress-Server-Configuration-... https://github.com/ewanleith/Wordpress-Server-Configuration-...
Basically they make nginx cache itself, which combined with this Wordpress plugin:
http://wordpress.org/extend/plugins/nginx-champuru/
Makes things very minimal, no need for even a dedicated cache :)
First time I reached the front page I had ~250 concurrent users and increasing, then the server crashed.
After that I wrote a cronjob that runs a script which automatically increases my virtual machines RAM. When I have little or no visits it uses 1GB RAM the most it went up to was 9GB RAM (increasing with 1GB at a time).
It also sends me an e-mail when increasing/decreasing the RAM so I can ensure that it doesn't crash.
I also use Wordpress with caching enabled and the above helps me keep the costs down, don't want the server to run on 8GB RAM all the time nor 1GB RAM.
Which provider is that? AWS? I feel like it's only a matter of time before auto-scaling works well enough that it's just a standard feature.
Glesys http://www.glesys.com/ they have server centers in Sweden, New York and Amsterdam (from what I can remember...)
200 hits per minute? Is that all it takes?
(I guess the answer is "No, only if you've misconfigured", but still...)
A misconfigured site that serves all its own assets will perform terribly because 200 hits per minute x 20 assets = 4,000 hits per minute. Wordpress has tons of assets too.
The approach I've been taking with my recent projects is to serve all static assets from Amazon S3/CloudFront. That way 200 hits per minute is only 200 requests. 19 of those 20 assets in your example would get served from the CDN. I don't have a lot of experience building robust apps (I'm a self-taught hobbiest developer) but intuitively this configuration makes sense and it has been incredibly easy to implement. I did some stress testing for the first time on one of my apps using jmeter and was able to achieve much higher performance that 200 hits per minute on my free micro instance.
Never used WP, but aren't those static assets? The bottleneck would be bandwidth then, should not be a huge load on the server. If it's compiling stuff or fetching them from DB though, then that could cause some problems.
It has to do with the number of requests plus the number of allowed processes plus the requests each process was allowed to serve. At a limit of 300 requests per process, the processes fell over at about 3 seconds time. With only 20 processes, dying every three seconds, the server was overwhelmed. So it's really a combination of issues.
Looking at the records for MySQL data, there wasn't that much of an issue there with MySQL connections. This is probably due to the fact that the page cache was working correctly.
I was expecting a lot more that that given the number of servers going down as soon as they hit the front page.
I think that most people leave it configured for default and never bother to do any kind of caching. My server was down less than five minutes because I knew what to look for.
My blog (elweb.co) has been on the HN home page a few times. I use Wordpress with WP Super Cache, hosted on cheapest available shared plan from Dreamhost with no problems.
We had a blog article on the front page of HN recently ( https://news.ycombinator.com/item?id=5133906 ) and I can’t say we had much trouble. Pushed the Heroku dynos up to two for an evening then dropped them back down to one at the end of the day. The site assets are hosted on S3 and we didn’t have to touch that. Ended up with about 16k hits.
We recently launched a simple website monitoring app for the iPhone where we ping sites from our servers, and send push notifications when we detect something is wrong (a poor man's pagerduty or pingdom where you do everything from your phone and can subscribe using IAP).
Probably the biggest lesson we've learned from collecting data on latency and timeouts from our backend is that there are a LOT of very poor hosts out there. Trying to strike a balance between intermittent failures, and actual "Your site is down, you need to do something" is pretty difficult, unless you are willing to simply wait long enough. We'll regularly see timeouts from places like Tumblr and Github Pages--they never go down, but things can get spotty. We can also tell when certain hosting providers are doing their weekly backups (or something - just guessing) because things will just go to pot at the same time every weekend for about an hour.
I can second the S3/CloudFront recommendation. Pretty rock solid (modulo downtime in specific regions).
For anyone curious about the app: http://vigil-app.com
I've read so many times on HN that people find it hard to scale WordPress. Really? Running WordPress on 128MB VPS and theoretically I can handle more than 250 request per second without problem[1]. Granted that number is just theoretical value but please, if you have gigabytes of RAM and your WordPress site fail after some hundreds of request per second, you certainly have done something terribly wrong. Especially for site with text and very few media content.
Forget any other setup you've ever used, go with this for high performance WordPress
1) Nginx [set so that request to static content to completely bypass PHP]
2) PHP-FPM
3) APC [Object & Database cache]
4) W3 Total Cache or WP Super Cache
5) Varnish
There many other high performance WordPress setup you could find out there but in term of simplicity and manageability, this is the one I fancy most. The bottleneck in this setup is RAM, not PHP, WordPress or MySQL read/write operation as I've seen in many server setup.
Disclaimer : I run small operation of managed WordPress hosting called KittySensei[2]
Good tips for Wordpress users. Another great option we went for is hosting your entire site/blog on S3/cloudfront. This can be done using tools like Jekyll (https://github.com/mojombo/jekyll) or Middleman (http://middlemanapp.com/).
If your site is entirely static content and mostly text, you don't really need a CDN. I've had a static site survive the HN frontpage without any problem on cheap-ass shared hosting.
Correct. But availability of cheap-ass shared hosting will be nowhere near that of S3+Cloudfront.
Additionaly, S3 and Cloudfront are extremely cheap themselves. And you get a few nice bonusses (like low latency, everywhere).
If all you need is static hosting then S3 is great. The benefits of a CDN, and it's incredibly cheap too.
If you want the benefits of a CDN I think you'd have to link S3 to CloudFront. S3 alone does not provide CDN capabilities AFAIK.
If you are self hosting Wordpress, the w3 Super Cache plugin is solid: http://wordpress.org/extend/plugins/w3-total-cache/
My otherwise poorly maintained blog didn't have a problem with a simultaneous HN/Slashdot hit, and I'm not even sure I'm using all the settings correctly. But the basic page cache seems to do what it promises.
Edit: oops, I see the OP is using Super Cache as well but still flopped. Then he is definitely doing someone wrong. My blog is hosted on the cheapest hosting plan that Dreamhost allows.
My solution was to move my WP blog from being hosted locally to Github pages using Jekyll. They'll do a much better job keeping it up than I will and more importantly it's less for me to worry about.
I wrote about my experience related to this in detail - Post mortem of a failed HackerNews launch [1]. This blog post provides insights into the caching, concurrency and stability issues I’ve faced.
http://www.gigpeppers.com/post-mortem-of-a-failed-hackernews...
For http://pineapple.io I had 250 simultaneous users at once, and I believe 16,000 over the course of the day.
Server cut through it like butter :) Just lots of caching, optimization, sprites, etc. That traffic isn't even that bad, but I was happy my rails code performed nicely.
Tip: if you have an image-heavy blog, don't host the images somewhere that charges extra for exceeding a network traffic limit. Getting an overage charge email was how I discovered that my fractal article full of pictures was on HN.
As an aside, I found that being on Reddit can give much, much more traffic than HN.
I'd also like to suggest, as a matter of politeness, don't just hotlink to images on other people's sites either. Then you're just screwing someone else over. I'm still getting requests for an image I posted to an old wordpress blog years ago because random j. user liked it and reposted it to a forum. It's a popular forum, judging from my logs.
2.5 GB virtual server? You don't need that at all. I use shared hosting for my blog, and yet I survived several hours as #1 on the HN front page. The trick? Static HTML pages (made with a homemade PHP blog system, but I could just as easily have used one of the many static site generators).
Is Wordpress.com itself capable of handling a slashdotting? If so, why not use it?
You can just use free services to host your wordpress blog without any trouble. Set it up on AppFog or OpenShift, use cloudflare too, and use a cache plugin if needed. I don't think any HN-effect could take that down.
I am preparing a new blog, that I do have the intention to have it featured on HN sometimes.
For now what I will do is use a very lightweight Wordpress theme, and cloudflare.
Is that enough?
If you just want to walk through a basic setup which will cope easily with HN and other front pages, just follow this post (disclosure, it's my blog!):
http://www.ewanleith.com/blog/900/10-million-hits-a-day-with...
It works, has been on HN front page a couple of times, and other similar sites, sometimes getting > 50k hits per day without an issue.
If you're just blogging, is there a reason you want to do it all yourself? I just set up a Blogger account with a custom domain and let it be...
I used to have a wordpress.com blog, but after they charged to everything I wanted to do, I got annoyed with them.
Also, I like to sometimes fiddle with stuff, and not let my "webmastering" get too rusty.
It is good to sometimes ssh to something that you own and wreck it at your will :)
To me, part of the fun of setting up a blog, is not only writing, but also taking care of it. I don't know really why (I have no intention of doing that for other people)
That's fair. I have a similar thought process around other servers and sites that I have, so I can appreciate the need for fiddling.
Funnily enough, I think the reason I went with Blogger (alongside not needing to worry too much about traffic etc.) was specifically to take away my complete ability to fiddle... otherwise I tend to do all that instead of actually writing.