Delayed Job on OpsWorks: A Chef Recipe Debugging Story

One of my current projects’ Rails application is hosted on AWS OpsWorks. OpsWorks is a lower-cost alternative to Heroku and EngineYard that still provides a full-suite of features, from deployment to scalability and fail-over. As with most Rails applications, this application requires background tasks. Initially, I had configured the Rails application to use Sidekiq. Sidekiq uses Redis for queue management. AWS provides first class Redis support with its ElasticCache service. But the use of ElasticCache for a background job queue is overkill and not worth the (albeit minimal) AWS charges. I decided to switch to Delayed Job because it uses good old ActiveRecord and an application’s relational database for the queue. The code modifications required to switch from Sidekiq to delayed_job were minimal and soon my local tests told me it was time to deploy my changes to OpsWorks.

A Recipe for Success

Delayed Job, like other background job managers, requires that you launch a service that polls and then runs jobs listed in a queue. In a default installation of delayed_job, that service is script/delayed_job. You can run script/delayed_job to view a list of the command’s options, which include: run, start, stop, and status. Running script/delayed_job run is easy enough locally but how do you coerce that service to run automatically on OpsWorks? Certainly, I could quickly SSH into EC2 and launch the background service — but OpsWorks applications should be turnkey. You should be able to automate the addition of new server instances and it should be easy to start and stop each EC2 instance. What my application’s OpsWorks implementation needed was a custom Chef cookbook. Chef cookbooks contain Ruby files — known as recipes — that are coded to run operating system specific commands at various life-cycle points of application deployment. All I needed was to launch script/delayed_job upon application restart.

You don’t have to be a master chef to write your own cookbook and code your own recipe as, most often, there already is a recipe available on git. A quick search and I found opsworks_delayed_job. The author’s README is excellent and I was quickly able to modify my OpsWorks configuration:

Take note of the Repository URL specified in the Custom Chef Recipes section above and then the recipe names listed in the 5 AWS Lifecycle events of Setup, Configure, Deploy, Undeploy, and Shutdown.

It Don’t Work

Unfortunately, the above recipes didn’t work. As usual in these situations — for about few seconds —, I think maybe I’m in the wrong profession as I’m obviously not smart enough to figure stuff out. I pondered, for a moment, if maybe I should go back to being a prison guard (which I was for about six months back in the early 80s), but I quickly snapped out of my fit of despair, calmed down, and began the process of debugging.

A Recipe Walk Through

Chef cookbooks contain, at a minimum, the following three directories:

1. attributes
2. recipes
3. templates/default

default[:delayed_job][application][:restart_command] =
  "sudo monit restart -g delayed_job_#{application}_group"

Obviously, that’s where the definition of the delayed_job command is crafted.

The recipes directory’s deploy.rb does the following:

execute "restart delayed_job" do
  command node[:delayed_job][application][:restart_command]
end

So that must be where the delayed_job is actually launched.

What is Monit?

But what is the monit command that was invoked with the above sudo monit command definition?

Wikipedia tells us that “Monit is a open source process supervision tool for Unix and Linux.” The OpsWorks Rails stack (as well as Heroku and EngineYard) uses Monit. And Chef cookbook recipes rely heavily on the availability and use of Monit. To find out a bit more about Monit you can run the monit command with the -h option to view command help. The most interesting option for me was -v, which lists Monit-managed processes. Below lists the output of one of those processes:

Process Name = delayed_job.incidenttracker-0
 Group       = delayed_job_incidenttracker_group
 Pid file    = /srv/www/incidenttracker/shared/pids/delayed_job.incidenttracker-0.pid
 Monitoring mode = active
 Start program = '/bin/su - deploy -c cd /srv/www/incidenttracker/current &&
   RAILS_ENV=production bundle exec bin/delayed_job start
         --identifier=incidenttracker-0 '
         timeout 30 second(s)
 Stop program  = '/bin/su - deploy -c cd /srv/www/incidenttracker/current &&
   RAILS_ENV=production bundle exec bin/delayed_job stop
          --identifier=incidenttracker-0'
          timeout 30 second(s)
 Existence     = if does not exist 1 times within 1 cycle(s) then restart
     else if succeeded 1 times within 1 cycle(s) then alert
 Pid           = if changed 1 times within 1 cycle(s) then alert
 Ppid          = if changed 1 times within 1 cycle(s) then alert

Hey, there’s my delayed_job.incidenttracer_group job! The Monit output also listed the Start and Stop programs. The Start program command shown above was built from hash values defined in attributes.rb:

default[:delayed_job][application][:restart_command] = "sudo monit restart -g delayed_job_#{application}_group"

Where the application, in my case, was incidenttracker. To see if there was an issue with the command, I decided to run the Start program manually:

[ec2-user@rails-app1 current]$ ./bin/delayed_job
-bash: ./bin/delayed_job: No such file or directory

Oh! The Delayed Job rails generate delayed_job installation command had placed the delayed_job utility in the script directory. Note that Delayed Job does have instructions in the README to replace script/ with bin/ in Rails 4. But the opsworks_delayed_job recipe is clearly looking for the delayed_job utility to be in the bin directory. I could have modified the recipe, but I opted to simply move script/delayed_job to bin as the application will be moving from Rails 3 to 4 in the next month.

I pushed my change to git and clicked the Deploy button on OpsWorks and now — IT WORKS! My background jobs now process smoothly. It turns out, modifying Chef recipes isn’t difficult at all. And now that I have a better of Chef recipes, my next step is to write my own. Thinking about writing your own Chef recipe for OpsWorks? Add a comment below — Corgibytes and other readers would be interested to hear about it.

{::comment} The following lists images referenced {:/comment}