How I migrated from Heroku to Digital Ocean with Chef and Capistrano - Matteo Depalo's Blog

I’ve always loved deploying to Heroku. The simplicity of a git push let me focus on developing my applications which is what I really care about. However, both because of the scandal about the routing system and because I wanted to expand my skill set by entering the sysadmin land, at Responsa I decided to migrate to a VPS solution.

At this point I had three choices to make:

Hosting provider Technology stack Deploy strategy

Provider

Many hackers I follow were recommending Digital Ocean so I decided to give it a try. I must say I was very impressed with the simplicity and power of their dashboard, so I decided to use it.

Technology

The decision of the web server was also quick. I wanted to achieve 0 downtime deployments so Github use of Unicorn + Nginx jumped to my mind.

Deploy strategy

This is where things got a little bit complicated. Disclaimer: I’m not a Linux/Unix pro, so many system administration practices where unknown to me prior to this week. Having said that, It was clear to me that the community is very fragmented. There were so many solutions to the same problems and so many scripts! After digging, trying and failing miserably I settled on the stack that caused me the least suffering:

Chef solo and Knife for the machine provisioning Capistrano for the deployment

Chef

Chef is a provisioning tool written in Ruby. Its DSL is very expressive and powerful. The community is full of useful cookbooks that ease the setup of common services, however it seemed to lack a way to handle community cookbooks. This is where Librarian comes in. I just had to write a Cheffile with all the dependencies and I was done.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 # Cheffile #!/usr/bin/env ruby #^syntax detection site 'http://community.opscode.com/api/v1' cookbook 'libqt4', :git => 'https://github.com/phlipper/chef-libqt4' cookbook 'nodejs' cookbook 'nginx' cookbook 'runit' cookbook 'java' cookbook 'imagemagick' cookbook 'vim' cookbook 'ruby_build', :git => 'git://github.com/fnichol/chef-ruby_build.git' cookbook 'rbenv', :git => 'git://github.com/fnichol/chef-rbenv.git' cookbook 'redis', :git => 'git://github.com/cassianoleal/chef-redis.git' cookbook 'mongodb', :git => 'git://github.com/untoldone/chef-mongodb.git', :ref => 'patch-1' cookbook 'memcached' cookbook 'elasticsearch', :git => 'git://github.com/elasticsearch/cookbook-elasticsearch.git'

To bootstrap the machine with Chef and Ruby many people where using custom Knife templates that were not working for me. Some installed ruby with RVM, others with rbenv. In the end I found Knife Solo that solved all my problems. With one command I could install Chef AND run all my recipes to install Ruby and every other service I needed.

1 knife solo bootstrap root@$IP node.json

Librarian and Knife Solo forced me to use a specific project structure:

1 2 3 4 5 6 mychefrepo/ ├── cookbooks ├── site-cookbooks ├── Cheffile ├── Cheffile.lock └── node.json

The node.json contains the run list of recipes:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 { "user": { "name": "deployer", "password": $PASSWORD }, "deploy_to": "/var/www/responsa", "ruby-version": "1.9.3-p286", "nodejs" : { "install_method": "package" }, "elasticsearch": { "allocated_memory": "200m" }, "run_list": [ "recipe[vim]", "recipe[libqt4]", "recipe[imagemagick]", "recipe[java]", "recipe[elasticsearch]", "recipe[mongodb::10gen_repo]", "recipe[mongodb]", "recipe[redis::source]", "recipe[memcached]", "recipe[nodejs]", "recipe[ruby_build]", "recipe[rbenv::system]", "recipe[runit]", "recipe[nginx]", "recipe[main]" ] }

All recipes except the “main” one are taken from community cookbooks.

The main recipe contains machine/application specific setup:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 rbenv_ruby node['ruby-version'] rbenv_global node['ruby-version'] rbenv_gem 'bundler' do rbenv_version node['ruby-version'] end rbenv_rehash "rehashing" group 'admin' do gid 420 end user node[:user][:name] do password node[:user][:password] gid 'admin' home "/home/#{node[:user][:name]}" supports :manage_home => true end directory "#{node[:deploy_to]}/tmp/sockets" do owner node[:user][:name] group 'admin' recursive true end template '/etc/nginx/sites-enabled/default' do source 'nginx.erb' owner 'root' group 'root' mode 0644 notifies :restart, 'service[nginx]' end ["sv", "service"].each do |dir| directory "/home/#{node[:user][:name]}/#{dir}" do owner node[:user][:name] group 'admin' recursive true end end runit_service "runsvdir-#{node[:user][:name]}" do log false end runit_service 'responsa' do sv_dir "/home/#{node[:user][:name]}/sv" service_dir "/home/#{node[:user][:name]}/service" owner node[:user][:name] group 'admin' restart_command '2' restart_on_update false default_logger true end service 'nginx' service 'elasticsearch' do supports :status => true, :restart => true start_command "service elasticsearch start" stop_command "service elasticsearch stop" restart_command "service elasticsearch restart" action [:enable, :start] end

I’m using runit to manage the unicorn service that is declared in a template file:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 #!/bin/bash exec 2>&1<% unicorn_command = @options[:unicorn_command] || 'unicorn_rails' -%> # # Since unicorn creates a new pid on restart/reload, it needs a little extra love to # manage with runit. Instead of managing unicorn directly, we simply trap signal calls # to the service and redirect them to unicorn directly. function is_unicorn_alive { set +e if [ -n $1 ] && kill -0 $1 >/dev/null 2>&1; then echo "yes" fi set -e } echo "Service PID: $$" CUR_PID_FILE=/var/www/responsa/shared/pids/unicorn.pid OLD_PID_FILE=$CUR_PID_FILE.oldbin if [ -e $OLD_PID_FILE ]; then OLD_PID=$(cat $OLD_PID_FILE) echo "Waiting for existing master ($OLD_PID) to exit" while [ -n "$(is_unicorn_alive $OLD_PID)" ]; do /bin/echo -n '.' sleep 2 done fi if [ -e $CUR_PID_FILE ]; then CUR_PID=$(cat $CUR_PID_FILE) if [ -n "$(is_unicorn_alive $CUR_PID)" ]; then echo "Unicorn master already running. PID: $CUR_PID" RUNNING=true fi fi if [ ! $RUNNING ]; then echo "Starting unicorn" cd /var/www/responsa/current chmod +x /etc/profile.d/rbenv.sh source /etc/profile.d/rbenv.sh # You need to daemonize the unicorn process, http://unicorn.bogomips.org/unicorn_rails_1.html bundle exec <%= unicorn_command %> -c config/unicorn.rb -E <%= @options[:environment] || 'staging' %> -D sleep 3 CUR_PID=$(cat $CUR_PID_FILE) fi function restart { echo "Initialize new master with USR2" kill -USR2 $CUR_PID # Make runit restart to pick up new unicorn pid sleep 2 echo "Restarting service to capture new pid" exit } function graceful_shutdown { echo "Initializing graceful shutdown" kill -QUIT $CUR_PID } function unicorn_interrupted { echo "Unicorn process interrupted. Possibly a runit thing?" } trap restart HUP QUIT USR2 INT trap graceful_shutdown TERM KILL trap unicorn_interrupted ALRM echo "Waiting for current master to die. PID: ($CUR_PID)" while [ -n "$(is_unicorn_alive $CUR_PID)" ]; do /bin/echo -n '.' sleep 2 done echo "You've killed a unicorn!"

Nginx is used as a reverse proxy:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 upstream unicorn { server unix:/var/www/responsa/tmp/sockets/responsa.sock fail_timeout=0; } server { listen 80; # server_name responsa.domain.com; root /var/www/responsa/current/public; # set far-future expiration headers on static content expires max; server_tokens off; # set up the rails servers as a virtual location for use later location @unicorn { proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_intercept_errors on; proxy_redirect off; proxy_pass http://unicorn; expires off; } location / { try_files $uri @unicorn; } error_page 500 502 503 504 /500.html; }

Capistrano

After setting up the machine I created a snapshot on Digital Ocean, in case I had to restart from scratch.

Time to deploy! Capistrano was an easy choice.

Using Capistrano multistage I set up the staging script

1 2 3 4 5 # staging.rb set :server_ip, $MY_IP server server_ip, :app, :web, :primary => true set :rails_env, 'staging'

This is used in combo with the deploy script:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 # deploy.rb require 'bundler/capistrano' require 'sidekiq/capistrano' require 'capistrano/ext/multistage' set :stages, %w(production staging) set :default_stage, 'staging' default_run_options[:pty] = true ssh_options[:forward_agent] = true set :application, 'responsa' set :repository, $PATH_TO_GITHUB_REPO set :deploy_to, "/var/www/#{application}" set :branch, 'development' set :scm, :git set :scm_verbose, true set :deploy_via, :remote_cache set :use_sudo, true set :keep_releases, 3 set :user, 'deployer' set :bundle_without, [:development, :test, :acceptance] set :rake, "#{rake} --trace" set :default_environment, { 'PATH' => '/usr/local/rbenv:/usr/local/rbenv/shims:/usr/local/rbenv/bin:$PATH' } after 'deploy:update_code', :upload_env_vars after 'deploy:setup' do sudo "chown -R #{user} #{deploy_to} && chmod -R g+s #{deploy_to}" end namespace :deploy do desc <<-DESC Send a USR2 to the unicorn process to restart for zero downtime deploys. runit expects 2 to tell it to send the USR2 signal to the process. DESC task :restart, :roles => :app, :except => { :no_release => true } do run "sv 2 /home/#{user}/service/#{application}" end end task :upload_env_vars do upload(".env.#{rails_env}", "#{release_path}/.env.#{rails_env}", :via => :scp) end

Now with two simple commands I can deploy with 0 downtime!

1 2 cap deploy:setup cap deploy

I must thank czarneckid for sharing his setup on Github from which I stole some useful portions.