Chef wrapper cookbooks allow reuse of chef code without having to copy the whole recipe. The focus of this post is using this pattern on Engine Yard Cloud.
Stable-v5, the latest stack on Engine Yard Cloud, includes what we call main chef recipes and custom chef recipes. Main chef recipes run by default depending on your environment settings (ruby version, database, app server, etc). Custom chef recipes are optional and will only run when you enable them. We can use custom chef recipes for Sidekiq, Resque, Thinking Sphinx, Elastic Search and a lot more. Not all applications use these so we don't run them all the time.
The cookbook wrapper pattern is our way of enabling custom chef recipes. Instead of copying the Sidekiq recipe that Engine Yard wrote, you will copy the wrapper recipe custom-sidekiq. Typically, you copy the wrapper recipe to the cookbooks/ directory on the root of your Rails application. Create the directory if it doesn't exist.
The advantage of the wrapper pattern might not be obvious at first. You still had to copy a recipe after all! The big difference is the wrapper recipe starts out almost empty. The custom-sidekiq recipe if not modified only includes the main sidekiq recipe.
Because you're not copying the actual sidekiq recipe, you'll get fixes automatically when Engine Yard updates the main recipe.
The wrapper pattern provides a way to make customizations. Some customers would want to run more workers depending on the EC2 instance size. Some customers want to have multiple instances dedicated to Sidekiq while some would run Sidekiq on the same instance as Redis. There are a number of different scenarios and the wrapper pattern allows Engine Yard to support all these without requiring customers to copy the actual Sidekiq recipe.
The customizations happen on the attributes file and they're documented on each of the custom chef recipes on Github.
Here are some of the ways you can customize the Sidekiq recipe.Specify the Sidekiq instances
The default setting is to run sidekiq on all instances so most users will have to edit this. You can choose instances based on their role or their name.
Specify the number of Sidekiq processes
default['sidekiq']['is_sidekiq_instance'] = (node['dna']['instance_role'] == 'util' && node['dna']['name'] == 'background_workers') # or default['sidekiq']['is_sidekiq_instance'] = (node['dna']['instance_role'] == 'solo')
The default adjusts the number of workers depending on the instance size. If you want to specify a different value you can do something like
worker_count = if node['dna']['instance_role'] == 'solo' 1 else case node['ec2']['instance_type'] when 'm4.large' then 4 when 'm4.xlarge' then 8 when 'm4.2xlarge' then 8 else # default 2 end end default['sidekiq']['workers'] = worker_count
You can also hard-code the number of workers.
Specify the worker memory limit
default['sidekiq']['workers'] = 4
We run sidekiq under monit which monitors its memory usage. If the Sidekiq process exceeds the memory limit, monit will restart it. The default is 400mb. Adjust it according to your app's needs.
default['sidekiq']['worker_memory'] = 500
After you make the modifications, you need to create the ey-custom recipe. This is the entry point of ALL of your custom chef recipes. You need to include the custom-sidekiq (not sidekiq) recipe on after-main.rb and add it to the dependencies on metadata.rb.
If this is your first recipe, you can also copy ey-custom from Github.
Note: the Sidekiq recipe requires an initializer on your Rails app and Redis running on your environment. We skipped these steps on this post and you can find the full documentation here.
Lastly, install the ey-core gem on your local machine, and upload the recipes with
ey-core recipes upload --environment <nameofenvironment> --apply
You can find a list of the available custom chef recipes on the ey-cookbooks-stable-v5 repo. If you want to use a recipe that is not listed you can still use them on stable-v5. Create the recipe on the cookbooks/ directory, include it from after-main.rb and add it to the dependencies on metadata.rb like we did above for custom-sidekiq.
If you have a recipe that works on a previous stack, you can convert the recipe by following the instructions here.
If you want us to write the chef recipes for you, our Platinum Support includes 100 hours of Application and Development Consulting Services on top of twice a year Database Analysis, 24/7 Support, Custom Runbook, and more. Contact us for more details.