Keeping node.js servers up forever
blog.nodejitsu.comThis reminds me of the bad old days of Rails, when every man and his dog had some random hacky and myopic solution to keep Rails and Mongrel running. Just say no to Node/Rails/whatever specific-stuff, kids--use system level tools for running processes.
Edit: removed references to "proprietary", as indexzero is correct about meaning of that.
That's a strange usage for "proprietary" since it's all free and open source (MIT). Either way, the nodejs child_process module is really just a light weight wrapper to execvp():
http://linux.die.net/man/3/execvp
I haven't dug through the source for tools like monit, daemontools, but I'm sure near the metal they're using POSIX or similar system APIs.
EDIT: There's really no reason Forever has to be node specific, I actually say that in the article:
"Honestly, it's a one line fix here (https://github.com/indexzero/forever/blob/master/lib/forever...), but I'm not sure if users want to put 'node' in-front of every command."
I'll file your comment as a +1 for that feature ;)
I just have to wonder - after listing all those projects which already do this job and do it well... Why create / use `forever` which didn't get the same exposure to the production environments yet?
Well, we can call native node.js code from our process monitor now.
We can extend our process monitor with functionality that would be difficult to implement in those other tools.
Any examples of when this would be desirable? Seems to me that monitoring and running tools should just, you know, monitor and run things. Separation of concerns.
How would you setup alerts for your monitoring?
I'd much rather have access to a well structured library (node.js) where sockets and http are first class citizens, opposed to being forced into a solution or having to write C or bash.
@intranation Just to toss out a scenario: Lets say under high traffic your application hits an edge case and starts to crash very frequently. Suppose you want to receive an email, SMS, or IM when something like this happens as a devops person.
Would you consider that a valid concern of process monitoring?
Planned features to Forever include this from the command line, but if you use it from node directly one could implement that feature now:
Bit by bit... monit restarts the server a defined number of times per second. Then it stops if the service fails to start. If the service is not running, nagios dispatches the snmp traps, sms-es, ims and all the rest.
I want my monitoring process to be rock solid. Extensions such as notifications should be done in another process, otherwise you're at the mercy of whatever email/SMS/IM library you include to do the notification.
This leads to a very bad practice though, which is to cause bugs to be left laying around in production systems. You should really form the habit of analyzing such crashes, getting to the root cause of the problem and making sure that it never ever happens again.
It's good to have a system like this in place, but it should be used as an insurance policy against unknown bugs, not as a way to work around known and reproducible ones.
Off topic, but I like to read blogs on my ipad while in bed and I hate it when authors fuck up with the meta viewport tag.
If I want to double tap to zoom in, because my eyes are not the same as 20 years ago, just let me do that, ok?
Sounded snob? maybe. Sounded reasonable? hell yes.
Btw, is there a bookmarklet to remove the meta viewport tag available somewhere?
Just for you: https://gist.github.com/725354
I was thinking about something like:
Thanks anyway for your quick response!meta=document.querySelector('meta[name=viewport]') meta.parentNode.removeChild(meta)Obviously that's fine for your use-case where the querySelector function is available, but it wouldn't work e.g. on Internet Explorer. My version is also more robust when dealing with markup errors, i.e. when there's more than one viewport meta element.
When I see a blog post like this, it serves more to raise doubts about Node.js than anything else.
Before today, I had been considering Node for an upcoming product. Now I read that people are actually building and releasing software to work around the fact that Node.js servers regularly incapacitate themselves. Really? Like it just goes down and doesn't know how to cycle itself? Ouch. That translates to me as "Don't go anywhere near Node.js until they get it stable."
So yeah, I'm sure this is a great way to shore up your system. Buy I'm going to think twice before investing time in a system that needs shoring up.
More of a safety mechanism than anything else. I use daemontools to run qmail, and qmail has never crashed. But if it did, I would still want to receive my email, even though it's "horrible" that qmail could crash.
Sometimes the safest way to handle an error is to kill the process and start a new one. Starting recovery from a known-good state is better than starting from a known-bad one.
Seems reasonable enough. From the tone of the post, it sounded like crashing was a fairly common thing to happen with Node.js, and that it didn't cycle itself.
Coming from the context of IIS/ASP.NET, which hasn't yet crashed on me (at least not without cycling itself harmlessly) in the 10 years I've been running sites on it, the possibility that you'd need to worry about such things seemed a bit novel.
So basically what you're saying is that it just follows the Unix philosophy and doesn't run its own daemon to cycle it if it falls down. That doesn't sound anywhere near as unreasonable.
Yeah, exactly. I have not looked at the app described in the article, but daemontools is a set of programs that let you manage persistent processes. "supervise <directory>" will run a script called "run" in <directory>, restarting it if it fails. It also keeps status information around, so you can run "svstat <directory>" to check to status, "svc -d <directory>" to kill it, etc.
On top of that is svscan, which will look for service directories in a directory, and will start a supervise instance for each. It will also start a supervise instance for <directory>/log if it exists, piping the output of the supervised script to the logging process.
In this way, you just need to write a service that writes log messages to stdout, and it will handle log rotation and process management. It's really a nice system, although apparently rather unpopular. I guess it's more fun to write your own logging and daemonization code, rather than let a very small C program that has existed unchanged for 10 years do it. Or something.
Erlang has heart (http://www.erlang.org/doc/man/heart.html) for similar reasons. Of course, it also has distribution, for when the whole computer it's running on dies.
I use runit (http://smarden.org/runit/) instead of daemontools (same sort of thing, but a bit less opinionated about e.g. where it gets installed). Rather than expecting every daemon to implement its own supervisor, logging system, etc. correctly, just use one of those.
Does daemontools care anymore? I use "supervise ~/.dotfiles/service | readproctitle ............." to start my user-local services when I log in. Works like a charm.
It's been a while since I installed daemontools. IIRC it created a bunch of root directories (e.g. /commands). There was also an OpenBSD port for runit already.
Criticizing node.js for not knowing how to restart itself is like criticizing Python for not knowing how to restart itself. node.js is an interpreter, it doesn't implement every part of a production stack.
I don't use Forever, but I use upstart with respawn which is similar, and I use it to monitor not just node.js daemons but also ones written in python, bash, and whatever else. Monitoring software is a reasonable part of a stable production system; it indicates a healthy ecosystem rather than a deficient one.
By using the 'http' module we can run a stand-alone web server in node.js without the need for a separate server like Apache or nginx.
Last I checked Ryan was saying this was a very bad idea. Has that changed? It's certainly enticing.
When did you last check with him? I've been running w/o a separate server (effectively) for ~3 months.
Oh, I just meant last I checked on blogs and such (not with him personally). IIRC, his main reason for advising against it was that there were too many security holes. This was in a video talk from several months ago if not longer. I'm curious to know if conditions have changed.
I have been using supervisord to solve this problem. It doesn't care what language your app was written in and works very well.
daemontools is a beautiful thing, personally I think its a much better idea to have a general purpose tool for this kind of thing. I'm not sure why everyone feels the need to reinvent the wheel when it comes to managing background services.
I like how there's a command line interface plus an application interface, so I can tie forever process management into my web interface and backend control interface. Right now the processes are just sitting in a `while true` bash loop.
Thanks. If you dig in a little deeper you can see I use optimist for the CLI parsing :)
In the next version I'd like to expose a web service and also have hooks for doing things like sending emails when forever restarts a process.
Just deployed this now. Took all of roughly 10 seconds to setup and working beautifully. I've used supervisor in the past, and this was definitely easier.
Oddly enough, the site is down.
How is this directly related to node and not servers in general?
The application described in the article is specifically written to interface with Node.js servers. It keeps those processes running, and restarts them as needed. It is not a general process monitor.
What's wrong with good ol' inittab?
bookmarked. thank you.
Use Upstart.
Forever a node.