Why you should avoid using capistrano-sidekiq
I found, that developers like quick and simple solutions when it comes to deploying applications. One of them is Sidekiq integration for Capistrano called capistrano-sidekiq.
Capistrano is a gem for the Ruby on Rails application which nicely handles the deployment process of the app to the target environment (usually a staging or production server). It creates directories, downloads code, links configuration files, precompiles assets and, most importantly, starts the application.
Sidekiq is a gem, as well and it can be used for background processing of certain jobs (like sending emails). Because of that, Sidekiq needs to be started as a separate process to work independently from Rails. The problem is that you need to manage this process during deployment, i.e., stop and start it when the code is updated.
Although Sidekiq’s wiki warns what can go wrong if you use “capistrano-sidekiq”, I’ve seen that it is the first choice of oblivious developers – you just add a gem to your “Gemfile”, add one line to your “Capfile” and Capistrano will take care of Sidekiq during deployments – great, right?
No, and here’s why.
Let’s analyze what happens during deployment when you use “capistrano-sidekiq”:
- Capistrano performs its standard tasks (obtaining code, precompiling assets and so on).
- Capistrano checks whether the Sidekiq process is running by inspecting its PID file; if there is a running process, it is stopped with a “SIGTERM” signal (or with a “SIGKILL” signal if it does not end within a given time); if there’s no Sidekiq process running, nothing happens.
- Capistrano rotates the release (i.e., symlinks the newest release to the “current” one)
- Capistrano starts Sidekiq with a “–daemon” argument.
- Sidekiq process forks (creates a child process) and shuts down, leaving the children process running.
- The child Sidekiq process is quickly inherited by the operating system’s init process (PID 1)
- The child Sidekiq process starts operating.
- Capistrano restarts the Rails process (Rails, Unicorn, Puma or another) – this is usually done by calling the operating system’s process manager (systemd, Upstart, SysV), provided by Operations.
Notice that the orphaned Sidekiq process is now adopted by PID 1, which is the init process of the operating system. In modern Linux distributions, it is systemd, but it can also be Upstart or SysV – it doesn’t matter really. The problem is that the init system will not monitor this process – and this includes restarting it when something goes wrong or starting it after a reboot. You may think that it’s unlikely to happen – but imagine how your client would react on a Monday morning if Sidekiq would go down on Friday evening, with queues growing rapidly and no one to fix it- because no one would even notice.
Here’s a list of things that can go wrong:
- Sidekiq unexpectedly shuts down and is not restarted (because this process is not monitored).
- Sidekiq slowly leaks memory and is killed by an OOM killer, but is not restarted (for the same reason as above).
- The server is rebooted and Sidekiq is not started (because it is Capistrano’s job).
- Some minor configuration changes are made on the server and the application (together with Sidekiq) has to be restarted, but it’s not clear how to restart Sidekiq (because the arguments passed when starting Sidekiq are configured within the application and the README is missing).
What to do, then? Control your processes with process managers – you can use the process manager of your operating system (system, Upstart, SysV) or an additional one (e.g., supervisor). It is described, with examples, in Sidekiq’s wiki.
Software development is not only about writing code anymore. You need to fully understand how your application runs on the target system.