Monday, February 1, 2016

Packer, Puppet and Jenkins: Automating VM Images


VM templates.  We love to have them.  Hate to update them.  Worse is the problem of "template sprawl", a web server template, file server template, DNS server template, this one runs Elasticsearch and this one is for Mongo.

You can script the software update process but what if your Elasticsearch config changes slightly?  What if your webserver template uses apache and you want to change over to nginx?

Here's a process and series of tools that automate a VM template process and link to configuration management.  First we briefly look at the individual tools and what they do.  Then we identify our VM template creation process.  Lastly, we connect the tools together in a way that mirrors our process.

Process Overview

If you have ever created a VM template you have a list of steps.  Here are the steps I use to create a VM template with tools to automate the process.
  1. Jenkins - Start the image build process.
  2. Packer - Create a VM from specified configuration.
  3. Packer, VMware Fusion, VirtualBox - Boot the VM from installation media.
  4. Packer - Install the OS.
  5. Puppet - Configure VM using production Puppet code.
  6. Packer - Perform templating tasks to configure a "generic" OS install.
  7. Packer - Shutdown the VM.
  8. Packer - Mark as template or export to OVF.
  9. Jenkins - Archive build artifacts, deploy to Vagrant repository in Artifactory.
Now that we have a basic overview of the process I'll discuss the tools in detail.


Packer is developed by HashiCorp.  Packer is a tool to describe how to build a VM or container and the resulting output.  Possible outputs include VM formats VirtualBox, VMware, and AMI; or containers.

Packer is a way to describe how to build the same VM with control over the output format and user defined variables.  We can use the same set of scripts and configuration file to build a web server or a mysql server by changing how we call packer.  Our packer configuration can install VirtualBox tools or VMware tools depending on what output format was specified.

I really like the Chef bento repository.  It has a number of great packer configuration files and scripts for Linux.  I use the bento repository and added a few modifications to install my preferred configuration management tool, Puppet.


Puppet is a configuration management tool,  Puppet allows us to describe in code the system state.  There are many benefits to describing your infrastructure as code.  In this case, we are able to use the same code we use after a VM is deployed to also build a VM from scratch.

Packer describes how to build the VM by calling a provisioner.  There are many types of provisioners can be a script, shell or Powershell, Puppet, or Chef or other.  I modified the Packer scripts in the bento repository to install Puppet Enterprise in the VM.

I configured Puppet Enterprise to use my puppet-control repository.  If you are not familiar with this concept, read this post by Gary Larizza.  In short, I configure the Puppet Enterprise install in the VM to grab a copy of all my production Puppet code.  I use a role/profiles method so each server type corresponds to one role.

Then I run puppet apply and specify which role I want to configure.  Since the node has full Puppet Enterprise installed it also has my hieradata and all my code "just works".

At this point I have a fully provisioned node using my production code.  Packer takes over again when provisioning is complete and exports the VM image into my specified format.


Jenkins is primarily a continuous integration tool,  In this case it acts as job scheduler, worker and archiver for my VM template process.  I schedule the packer script to run once a week.  The Jenkins nodes have VMware and VirtualBox hypervisors installed and run the packer script in headless mode, no GUI required.  I modified the Jenkins VM settings to allow for nested virtualization.

Jenkins runs the entire provisioning process and archive the resulting packer image.  If the image is a Vagrant box, Jenkins uploads it to a Vagrant repository on the Artifactory server.  This requires setting a three artifact properties for Vagrant to detect the box.  These properties should be passed from the job to Artifactory.

  • box_provider
    • Hypervisor used to create VM.
  • box_name
    • Must match output .box file.
  • box_version
    • Must be a number.


Using Packer, Puppet, and Jenkins we can schedule and automate the VM template build process.  Puppet also allows us to create VM templates based on the types of nodes we use in our production environment.  We can have one VM template for each Puppet role.

The VM templates can be used in a number of ways and provide benefits:
  1. Developers have recent production VM templates to develop on.
  2. Provisioning VMs from templates is faster because Puppet has already installed and configured most of the software.
  3. Deploying to multiple cloud providers is more manageable with consistency between VMs.
  4. VM template library can grow in size as manual management and update of VM templates is eliminated.
There are some things I would like to work on.
  1. Resulting VM template is over 1GB.
    1. Some of my VM templates are over 1GB.  This may be OK for deploying to vSphere but not good if a developer needs to download a 1GB Vagrant box.  The Puppet Enterprise install is consuming some space.  It may be that our images are just large.  Maybe I can enable some compression?
  2. Jenkins, VirtualBox and VMWare Workstation don't always play nice.
    1. Sometimes the hypervisors on the Jenkins nodes crap out.  When that happens I need to run the packer scripts in GUI mode and try to debug the process on the Jenkins node.  This is a pain in the ass.  I think this is due to kernel updates, then the VirtualBox and VMware kernel drivers need to be rebuilt.  Until VMware Workstation 12 I had a problem with the USB Arbitrator service.
  3. Use Puppet in a way that doesn't require a full Puppet Enterprise install.
    1. I would like to use Puppet but not install Puppet Enterprise on the VM template.  I think there must be a way to install just the Puppet agent and still run the code I want.  I just haven't given much thought to how this would work.