Tuesday, September 30, 2014

Practical Puppet Development Using Vagrant and Jenkins


I don't consider myself a software developer and I have no practical experience, training or formal education in software development (save one C class I took late 1990's).

Recently I started to write puppet code to provision applications.  I wasn't writing and deploying code fast enough and making a lot of mistakes.  I wanted to go faster and make fewer mistakes.  To accomplish those goals I needed to automate the testing, deployment and development environment processes.

I adopted the tools (Jenkins, Vagrant and Git) of the other developers around me, incorporated other tools (r10k, puppet-lint) specific to puppet, and developed a workflow.

This post is not a technology deep dive nor will I touch on the deployment process.  This is about developing a workflow with which you can write good code and a build consistent development environment.

Everyone wants to write good code but why care about the development environment?  A consistent development environment allows your peer(s) to review the code before pushing it into qa or production.  Your peers have 100% assurance their development environment is exactly the same as yours.  It is important to have identical environments to ensure all tests are repeatable and have the same outcome.  In my opinion, it is even better if your development environment is identical or nearly identical to production.


My workflow for writing and testing code consists of these steps: 
  1. Developer writes code on local workstation.
  2. Developer commits code to centralized repository via version control system.
  3. Developer's version control tool runs a pre-commit hook for syntax and whitespace check. Commit is ignored if these tests exit with an error.
  4. Successful commit is received by centralized version control repository.
  5. Centralized version control repository notifies orchestration tool of commit.
  6. Orchestration tool performs additional testing.
  7. Developer provisions a new or re-provisions an existing virtual machine.
  8. The automated provisioner starts a new virtual machine using the local hypervisor.
  9. The automated provisioner runs the automated code installer.
  10. The automated code installer downloads any required code for the virtual machine.
  11. The provisioner provisions the virtual machine with the required code.
  12. The virtual machine is ready for testing.


The specific technologies don't matter.  You could weave together other technologies that serve the same purpose and still use the same workflow.  Here are the technologies I use and what they provide:
  • VirtualBox: local virtualization, on the developer workstation
  • Vagrant: automated provisioning for virtual machines, bootstraps provisioner
  • Puppet: provisioning for the virtual machines
  • r10k: automated Puppet code installation
  • Jenkins: automation/orchestration tool to automate testing of Puppet code
  • Gitlab: centralized version control repository
  • git: version control system
  • puppet-lint: syntax check for code
  • puppet parser validate: checks for valid code.
I've re-written my workflow, identifying each technology at each step.
  1. I write code on my workstation.
  2. I commit the code to Gitlab using git.
  3. I have a pre-commit hook that runs puppet-lint and a whitespace check.
  4. My commit is ignored if those tests fail.
  5. On a successful commit, Gitlab notifies Jenkins via a webhook to run a job.
  6. Jenkins runs the job and performs additional testing.
  7. I provision a new or re-provision an existing virtual machine on my workstation.
  8. Vagrant handles the virtual machine provisioning and spins up a virtual machine using VirtualBox.
  9. Vagrant r10k module downloads any required Puppet code from either Puppet Forge or Gitlab.
  10. Vagrant runs the Puppet provisioner after the virtual machine has started.
  11. Puppet provisions the machine according to the virtual machine's manifest and using the code r10k downloaded.

Put It All Together

So how to glue all those technologies together?  Let's walk through each step and outline what features are used to bring them together.

Step 1 - Writing code.

Be comfortable writing code on your workstation.  I chose vim to write the majority of my code because I'm usually in a command line all day.  I installed a vim module specifically for puppet.  It helps enforce syntax and does some autocompletion.

For Vim lovers:
Download and install https://github.com/rodjek/vim-puppet.

Do it your way:
Find software they helps enforce good coding habits.  You don't need to use a full blown integrated development environment.

Step 2 - Commit the code.

Use a version control system and commit your code to it often.  I use a shared git repository, Gitlab.  More on Gitlab and how it's configured later.

For Git users:
Download and install git on a Mac.
Download and install Gitlab on a server for private repositories.

Do it your way:
Whatever repository you choose ensure it has the ability to run an action on every commit aka webhook in Gitlab parlance.  This important feature is used later on when we set up automated testing.

Step 3, 4 - Use a pre-commit hook.

I use a git pre-commit hook to check my code before it is committed.  This simple automation runs puppet-lint and a whitespace check.  I added the following to .git/hooks/pre-commit
puppet-lint --with-filename . 
if [ $? != 0 ]; then
  exit 1
It will not commit unless it exits without error.  This helps to enforce good code when it counts: during the development process. 

For Git users:
Read more about git hooks.
Install puppet-lint.

Do it your way:
Find a way to run a script before a commit to the version control system.

Step 5, 6 - Automate testing.

I use Jenkins to automate testing by using a Gitlab webhook to call a Jenkins job URL that will run puppet-lint and "puppet parser validate".

Why run puppet-lint again?  Because Git is a shared repository I can't guarantee that every developer will run puppet-lint as a pre-commit hook.  Running puppet-lint again in the Jenkins job ensures puppet-lint is run at least once.

I could write an entire post on just Jenkins but there are some basic steps you need to accomplish.

For Jenkins users:
  1. Install and configure Jenkins on a server you control.
  2. Create a new job that uses your puppet code repository that can be triggered remotely.
  3. Add executable shell code to run puppet-lint and puppet parser validate on your code.
  4. Install puppet-lint and puppet wherever Jenkins executor is running the job.
  5. Configure to job to succeed only if the tests pass.
Do it your way:
Download and install an orchestration engine that will run whenever code is pushed to the repository.

Step 7 - Test locally.

Testing locally on your workstation may not be suitable for every situation.  It has worked for me in almost all situations.  To test my puppet code I spin up a virtual machine and apply the puppet code to it.  I really like VMware but I find VirtualBox suits my needs and is free.

VirtualBox users:
Download and install VirtualBox.

Do it your way:
Download and install VMware Fusion or whatever virtual machine provider you want. Just make sure it has a Vagrant plugin.
Download and install Vagrant Fusion plugin if you're using VMware.

Step 8 - Use Vagrant.

Testing all my code locally would be a royal pain if it weren't for Vagrant.  Vagrant allows me to spin up a consistent development environment on my local workstation.  I then use the Vagrant provisioners to download the puppet modules and apply the code.

There is no substitute download and install Vagrant.

Step 9 - Use r10k.

If you've never heard of r10k before, go read this article.

There is no substitute download and install the Vagrant r10k plugin.

Step 10, 11 - Putting it all together.

Lastly, you'll need to create the directory structure to support all the configuration files, puppet code, etc.

I've written this shell script which will build a basic directory structure and configuration files supporting Vagrant, Puppet, r10k and hiera.

  1. Download the shell script.
  2. Run as: ./createVagrant.sh linux-server
  3. Add any puppet modules you want available to puppet/Puppetfile. See syntax for Puppetfile.
  4. Apply those modules to your node by creating a puppet node definition in manifests/default.pp.
  5. Define any configuration in hiera and store the config in puppet/hiera.
  6. Run: vagrant up


This is my first attempt at putting my workflow out there and I hope it helps you get started.  Feel free to switch up the tools.  I happened to use what developers around me are using, that helps whenever I have a question about a particular tool.  Take the workflow a bit at a time or take it all at once.