Daemon showdown: Upstart vs. Runit vs. systemd vs. Circus vs. God - Cue Engineering Blog

We write a lot of daemons: programs which run on servers in the background, like an HTTP server, or a database. Once we've written the programs, though, we have to run them, and running programs as daemons is surprisingly heavy on details; it's fraught with perils for the unwary. If you go the traditional Unix way, you do some magic with double-forking and pid files and init scripts, and it's horribly tedious. If you run in a detached session of screen or tmux, you end up doing everything manually. In both cases, there's nothing in place to restart your program if it crashes. Steve Huffman, one of Reddit's founders, will tell you from experience just how bad an idea this is:

To reclaim their ability to sleep, they had the computer keep watch for them, resurrecting dead processes. This also let them automate health checks: when Reddit's front page became unresponsive, they could detect this automatically and simply kill the offending process, secure in the knowledge that the supervisor program would quickly bring it back up again. They could sleep right through this fleeting death and rebirth, and only a few people would notice.

There are some good tools for keeping programs running as daemons, and all of them make it easy for you. If you're on Ubuntu, Upstart is the default, and it's a good choice. If you want something less monolithic, Runit is featureful and appealingly Unixy. Several Linux distributions use systemd as their init, and for our purposes it's comparable to Upstart. Other interesting options include Supervisord, God, and Circus. Unfortunately there is no clear best answer here, so which one you want depends on your needs. Let's compare.

Upstart

Used as the default init replacement on Ubuntu, Upstart is very good at starting things up. The Upstart Manual can be kind of overwhelming, but the config files themselves are pretty simple. For example:

Let's walk through this. The comments at the top customarily include a one-line description and an optional longer description below – a lot like a git commit message. The description line is, again, for your own use. The start on runlevel [2345] part means to start the service when the machine is started. The respawn and respawn limit commands tell Upstart to restart the server if it dies, but stop trying to restart it after 10 failures in 5 seconds. Finally, Upstart sets the uid to myuser and runs the server. Make sure to setuid, because if you don't include it, your server will be run as root by default. This is a profoundly weird default – it's usually not what you want, and it's very dangerous. So, be careful with this.

Upstart expects myserver not to fork. Any output from myserver on stdout or stderr gets sent, by default, to a log file in /var/log/upstart/. Both of these things can be changed; Upstart can handle the classic forking style of daemons, and you can plug in other options for logging.

Upstart is one of the best options for running traditional forking daemons, since it actually uses ptrace to make sure it's watching the correct process ids. The system for this is very slick.

Runit

Upstart, while admirably simple to use, does have some shortcomings. Its default logging, for example, does not support log rotation and is only compatible with logrotate if you use the copytruncate hack. Upstart's options for running servers sometimes don't include all that you might want; for instance, it currently has no way to adjust the maximum number of file descriptors, or limit memory usage. Runit is an interesting alternative, inspired by D. J. Bernstein's daemontools: it's a collection of programs which each do one thing, and together, they have a lot of features that Upstart lacks. If you want a gentle introduction to using Runit, the blog post Runit for Ruby (And Everything Else) is very well-written.

The two best things about Runit are the chpst and svlogd programs. With chpst you can start a program with a changed process state:

chpst -u myuser -/ /var/roots/mychroot -o 10000 -n 3 /usr/local/bin/myserver --port 8080

This runs the command /usr/local/bin/myserver --port 8080 as user myuser, chrooted to /var/roots/mychroot, with a maximum of 10000 open file descriptors, at a niceness level of 3. Some of these options are available with Upstart, and some aren't. The beauty of chpst, though, is that you can use it with anything. You can use chpst with Upstart, or Supervisord, or whatever else strikes your fancy. And because Runit relies entirely on chpst for this sort of thing, you end up with completely reproducible execution environments: if you run the Runit script for a service by hand, you get the same environment as you would get if Runit had run it. This lack of magic is invaluable when debugging.

Similar to chpst, Runit's svlogd can be used with anything to provide log-handling. It reads text-based logs on stdin, and either writes that to rotated log files, or sends it to syslog, or some combination thereof.

Runit is a decent way to run daemons, and it has useful pieces that work with just about anything. We use it for most of our services. The one caveat to be aware of is that Runit expects daemons not to fork; this is great for servers that you write yourself, but it can be inconvenient if you try to use Runit with some other programs out there.

systemd

The always-lowercase systemd is used as the default init on a fair number of Linuxes, most notably the Redhat/Fedora family of distributions. If you're using one of these and would like to go with the already-installed option, this blog post is a good introduction. Since it's pretty similar to Upstart, the easiest way to describe it is to talk about some of the differences. First, obviously, the configuration files have a different syntax, but that's more of a superficial difference – the concepts are the same. More noteworthy are the ways that systemd handles logging, resource limiting, and daemons that fork.

Systemd is serious about logging (though I suspect they may have over-engineered it a bit). By default, all processes have their stdout and stderr connected to systemd's logging facility (described here and compared with syslog here), but it can also go to syslog, the kernel log buffer, a socket, a tty, or any combination of these. Docs for the logging are here.

Systemd has a very clever solution to the problem of tracking daemons that fork, which coincidentally happens to handle resource limiting at the same time. Where Upstart uses ptrace to watch the forking, systemd runs each daemon in a control group (requires Linux 2.6.24 or newer) from which it can not escape with any amount of forking. This allows easy resource limiting, both for forking and non-forking daemons, since control groups were made for this sort of thing.

More documentation can be found on the systemd web site. Or, for a reference, the man pages are here.

Other contenders: Supervisord, God, Circus

Supervisord is pretty convenient to set up, and seems to get a fair amount of use in Pythonland. Its main advantage over Upstart, Runit, and systemd is that Supervisord can easily start pools of several instances of a program. This is useful if you want a pool of worker processes, e.g. as FastCGI workers. Supervisord expects programs to not daemonize, though it can handle daemonized programs with a pid file workaround. This workaround does not handle edge cases as well as Upstart or systemd, but it should work fine most of the time.

The hard-to-google God hails from the Ruby world. Its great virtue is programmability: you decide what programs to start, how to run them, etc., and God will monitor them and keep them up and running. Suppose you have three back-end web servers, which you want to run on three different ports, so that a reverse proxy can load-balance across all of them. The config file is just a Ruby program, so a simple loop can bring up all three servers. The God web site has examples with code. If you're comfortable writing Ruby and you have interesting needs, God might be what you're looking for.

Mozilla's Circus is perhaps the most interesting of the three. It's programmable in Python, it can handle pools of workers, and it can bind sockets for you. This last point is especially neat: Circus can act as an HTTP server container by binding to a socket, starting up a pool of worker processes, and passing them the file descriptor of the socket. It acts a lot like Gunicorn, but more powerful. Circus also handles logging and log rotation, has a web-based console for watching the supervised processes, and sends a stream of all events over a ZeroMQ PUB socket. It's more actively developed than Supervisord or God, but already stable enough for production – and it has the best docs. Here are some examples of how to use it. If I wanted to serve up a web site with a WSGI stack, Circus is the option I would look at first.

Any of these three can be run under, say, Upstart. They don't want to be an init replacement, so they play nice with everything else.

Pick something

Hopefully this infodump is a useful comparison, but the most important thing isn't which of these options you use; what's important is that you use something. Automating server administration a bit more is almost always a win. The more things you can solve once and then stop thinking about, the better.

It's also worth mentioning that a simple crash is far from the only reason why a daemon might need to be restarted. Several of our server health checks can kill unresponsive processes, if something has frozen up or gone into a weird state. This can also be a temporary stopgap against hard-to-debug intermittent memory leaks: if the server is using too much memory, kill it! We have an ever-growing number of health checks keeping our servers in working order, because it's easy to add more. We're not quite writing crash-only software, but it turns out that you can get a lot of its advantages by making crashes cheap.

(If you find this kind of thing interesting, we're hiring. curl -L cueup.com/jobs)