I recently gave a talk at PuppetCamp Toronto 2015 focusing on how we use puppet at work. Slides are available here.
One of the more interesting topics I covered was how to integrate the building of puppet manifests into the software development life-cycle.
I talk about three different approaches that we have tried.
Operations building the manifest as part of a SCUM sprint
When we first tried to integrate puppet into the earlier stages of our processes we tried using SCRUM. We said that operations folks would be assigned to the SCRUM team and would be responsible for developing the puppet manifest during the same sprint that the software was being developed. The idea was that the sprint deliverable would be a piece of software + puppet manifests to install and configure that software all of which were tested by QA.
This sounded great in theory, the folks with operational experience and puppet expertise would be able to work hand-in-hand with the developers as they build the product. The problem was that while developers and QA staff were dedicated to the sprint, meaning that the spring deliverables had 100% (or even 90%) of their focus, we couldn’t dedicate an operations person to the sprint. Our operational teams (system-administrators, DBA’s etc..) support many products and projects and the teams aren’t structured such that they can take a single person and dedicate them to a particular project. Operational staff need to keep the lights on and deal with operational problems. If a series operational problem happens in the middle of a sprint they need to deal with the problem sprint work will have to wait.
Having Developers build the manifest
A puppet manifest is like code(or an installation script), shouldn’t the developers build the manifest as part of the deliverable artifacts? The argument sounds good at first but the problem is that many developers don’t have the operational experience or mindset to build a good manifest. Writing a good manifest not only requires good puppet knowledge but an understanding of how to properly setup a production server. Developers who have spent some time troubleshooting or maintaining production systems might have these skills but many don’t. The skill set to setup, configure and maintain a production service isn’t the same as the skill set needed to build great web front-ends or write code that implements business rules. Some developers might be able to build an acceptable manifest but others won’t. Our experience asking developers to build manifests from scratch without also having them spend time an operational teams has mostly resulted in unusable manifests.
Call a Puppet Superhero
Process frameworks like CMM argue against heroics but in the quest towards standardization you have to remember that people aren’t widgets, no two people in your company of the exact same background,skills or experience. Writing an effective puppet manifest from scratch requires the following
- Sufficient operational experience to understand what a production environment should look like. This covers everything from file permissions to useful logging levels (and locations) along with monitoring and higher availability concerns. Most importantly they need to understand the differences between environments (production, qa, staging etc..) so they can might the right configuration options parameters
- A schedule that allows someone to focus on building the initial manifest at the right point in time. The right point in time is when enough of the software is developed so that what needs to be installed and how it should be configured is known. This also needs to happen before QA starts testing because you want QA to be testing in an environment that was setup with the manifest
- Reasonably strong knowledge of puppet. Building a puppet manifest obviously requires knowledge of puppet and the better the puppet skills this person has the better the manifest your likely to end up with.
You don’t need to dedicate a puppet superhero to all your products or projects. You just need to have people who aren’t part of (or pulled from) the daily operational routines so that they can focus on building the manifests at the right time. Once the manifest is complete they can hand it off to the project team, to either developers or operations people who can adjust the manifests as required. Adding a few extra configuration options or files to an existing well written manifest is a lot easier than building one from scratch. The puppet superhero needs to fly into the project to build the manifest and then fly out once a working manifest is available.
I have started to see a lot of talk at conferences and job postings about dedicated ‘DevOps positions’ this is compatible (or perhaps the same) as what I describe above. Someone with the skills to build the components required to do deployment automation from development to production. This is the primary part of their job description versus keeping the lights on that makes up a large part of a traditional system administrators responsibilities