Building Your Toolbox

| Comments

I know it’s been since June we’ve worked on the Puppet Development series, but as work goes, so go I, and as I go, so delays the blog. :)


We started our journey with a reintroduction to Vim. As strange as it sounds, often times we techno-guys take for granted that people coming into the DEVOPS space are well versed in all these things, and overlook remediating the basics. We covered Vim basics and switching between command and insert mode as well as linking you to some good cheat sheets to help you beef up your vim-fu.

Next, we covered Vim plugins for syntax highlighting and just general code visualization so we can see visually when our code editing has issues and needs to be fixed.

The next article centered on revision control in general, but Git in particular. In addition to using Git, we also covered that amazing tool GitHub and how to get registered for it, create your own repos, and how to work with repos from your local command line as well and I linked you some excellent resources on Git to expand your knowledge of the Git world and become proficient and fluent in its use.

Finally, we got around to Vagrant and I gave you a simple tour of Vagrant to be familiar with the tool and what it does for you. We stood up a Vagrant instance and saw how we could destroy and re-provision that precise same instance with only a shell command, and saw the power of automated provisioning at work right on our own node.


Like the social media world will tell you, I did it because reasons. In short, I wanted you familiar with all these separate tools as we start to coalesce them into an integrated whole we can use as our development toolbox.

So, since we’ve got Vim, Git, and Vagrant as needed components, what else might we need to continue pressing forward?

We need to understand Puppet itself.

Puppet - The Product

Puppet, as I’m sure many of you are aware, is simply “configuration management software” produced by Puppet Labs, Inc. Puppet Labs was founded in 2005 by then Systems Administrator/Engineer Luke Kaines to help automate common repetetive tasks encountered in his regular work duties encoutered on a day-to-day basis.

After a few rounds of venture funding and explosive growth of the market segment known as “configuration management”, Puppet Labs has become a market leader in the space, and continues to develop and improve upon the product at a rather aggressive rate.

Configuration Management

If Puppet is “configuration management” software, what is this thing called “configuration management”?

Configuration management as a systems engineering process covers a lot of landscape in its purview. It can mean, speaking generally, a process for maintaining consistency of a product’s performance and can become considerably complex, such as the methodology used to manage miliary weapons systems, IT service management, and other domain models covering civil and industrial engineering.

For the purposes of the technical field of Systems Administration, Engineering, and Automation, however, Configuration Management as a discipline is very well defined. Specifically, the model that covers these areas is Operating System Configuration Management. Certainly, when automating your site you step over into additional disciplines, but at its core, Configuration Management almost always implies that you are working with the primary target of Operating Systems Configuration Management.

IT Automation

Often times, configuraiton management as it is expressed in the realm of operating systems more specifically begins to take on automation as the primary characteristic of the work performed. From provisioning to deployment, automation saves the most time and effort through modeling systems design in a modular fashion, whereby allowing you to apply systems configurations against classes of machines tooled to perform specific types of work.

This paradigm allows for many idioms to be used in the description of the destination machines, and is the primary domain within which products like Puppet operate.

Puppet – the Product

As you peruse the main website for Puppet Labs, you start to see a much larger emphasis and prevalence of “IT Automation” throughout the website–specifically in the area of data center automation. It is important to note that Puppet prefers to work within this space, although it has abilities that stretch into the entire IT lifecycle workflow.

Puppet is comprised of two distinct products: “Puppet Enterprise” and “Open Source Puppet”. As is common in the Open Source space, the distinction between the two is defined primarily by way of the support options available for each.

Puppet Enterprise

The Puppet Labs flagship product is Puppet Enterprise. Puppet enterprise is a fully supported, maintained, and actively developed data-center ready software package designed to enable you to model configurations for your site out-of-the-box. It has an integrated installer, smoothing the installation process, a series of Puppet apps available for use in the Enterprise Console (the Puppet GUI), and for-pay support and licensing options to meet the needs of your enterprise, regardless of size.

Open Source Puppet

Open Source Puppet is the core product found within Puppet Enterprise. It is the engine which drives the Puppet product and does the job of configuration modeling against your environment. It has several components, individually installed, and no shipped console. It generally leads the Enterprise version by several revisions, allowing early access to new features and benefits but lacks the additional features afforded by the Enterprise offering.

Make no mistake. Both products are the same software. However, the Enterprise product has much of the legwork done for you in the integrated installer, additional functionality in independently released “Puppet Apps”, and of course, enterprise-level 24x7 support availability. Add to that a vibrant community of development and expansion, the Puppet Forge, an annual conference, and a respected certification program, and Puppet Labs’ offerings, while similar, certainly shine on the Enterprise side of things.


The main concept upon which Puppet’s operation is founded is that of idempotency. Idempotence is the property of certain operations in mathematics and computer science that can be applied multiple times without changing the result beyond the initial application.

As you can see, this characteristic only applies an effect on the target if and only if it has not already been applied. Subsequent applications will have no effect, as the desired state of the target is as it should be.

This is good news in that it allows us to think of our enviromnet in new ways. Instead of thinking of all the changes we’re making or going to make to our environment, we begin to considered the desired state of our site and the maintenance of that target at all times. Then, “events” are no longer misconfigurations, but instances of deviation from the desired state, the “norm”.


Think of the ramifications of the shift in mindset this represents. Audit reports are no longer a series of weeks of review of the site. Your site’s state is described in code, and is a public record within your organization for all to see. Instead, those conversations become “We are ALWAYS compliant. Here is a report of the few times we weren’t compliant in the last year.”

Considerably different conversation to have.

Now, as you model more configuration state in code, your entire way you think of your site evolves. More things hapen automatically, are automatically reported on, and are automatically remediated. Your reporting changes. Your audits change. Your compliance reporting changes. In fact, your entire internal culture chages, and all involved teams need to make adjustments in the way they think about the site and how to mold the way they’ve traditionally done their job (whether administration, audit, or compliance) into this new world of DEVOPS, automation, and the like.


Puppet Enterprise and Open Source promise to not only change how you view your systems and site, but how your entire organization functions. From automated configuration expression within the site, to how you originally model your configuration target, the tools provided by Puppet have changed the face of IT. Automation and configuration management bring culture change and ideological evolution to the enterprise, and step you into the next level of efficiency, compliance, and security management.

Some Coding Work of Late…

| Comments


I know that sounds a little silly with all the Coderwall links over there in the sidebar, but I’m not.

I got into this business as a lowly PC building guy, and worked my way into systems administration through light consulting. A necessary evil of the day was tweaking an autoexec.bat & config.sys to release as much memory to the user as was possible for applications. (This was long before Win95, FYI)

As progress and learning would have it, I landed myself a systems administration job and began to grow. Here, 22 years later, back to consulting (but on a much larger scale), I look back on my professional career and see and realize that there’s a LOT of code behind me. Perl, BASH, ksh, HTML, PHP, light Ruby, old DOS debug scripts, Puppet DSL, Expect scripting… tons of it. All encountered and fleshed out in the context of systems engineering and/or management over the years as situations and needs arose.

Fast forward to today. The juxtapostion of Development, QA, and Operations into one big hairy hard-to-define (but getting clearer) term known as “DEVOPS” is the landscape a new admin comes into, and he or she learns from the very beginning the principles of placing infrastructure definition into code, and working as a developer to enhance and automate the hard infrastructure of the operations world.

I say all that to say this… I’ve got some new releases on my GitHub I’d like to share with you to help you out while navigating in this world of DEVOPS. If you have to call me a coder because of it, I may frown, but it is what it is.

Vagrant/Virtualbox Fun

As I’ve been slowly revealing through my series in past months, there’s a lot of tools out there for working with Puppet and there’s a ton of the same to prototype for your company’s environment. One of these is Vagrant, and it has the ability, in a huge way, to help you automate the setup and teardown of sample infrastructures to work with your Puppet code in. I’ve just updated and released a few of these, and I want to tell you about them.

Vagrant with CentOS 6.5 and PE 3.7.1

If you look here, you’ll find my current project I use with customers. This is a Vagrant instance that turns up a 4-VM environment including a PE Master, a DEV, TEST, and PROD VM running the PE agent, Enterpise Console, Directory Environments pre-configured, r10k configured, and a simple set of Puppet Modules to get you started.

Most commonly, I share those with customers, coworkers, and community folks to get them started coding right away, and to have a platform with which to teach them how to deploy, merge, and promote code through an environment in a smaller version of wat they might already have in their company. This is the enviroment I spoke about at the Atlanta Puppet User’s Group last year in its current iteration.

Vagrant with CentOS7 and PE 3.7.2

Similarly, you can find a CentOS7 + PE 3.7.2 project here. Much like the above, you get the latest of PE with CentOS7 to help your prototyping over a more current OS.

Vagrant with CentOS7 and Puppet OSS 4

If you look up the word “experiemental” in the dictionary, this project right here is linked as an example.

I’ve gotten a very rudimentary working setup of a Master and one agent to install completely and autosign, and haven’t even scratched the surface of all the new goodies in Puppet 4. As Puppet 4 is still in Beta, this is not recommended in any way for any reason at any time for you to use for any purpose. :)

My hope here is to prepare myself for the PE4 features long before they’re released. I hope to work on getting directory environments and r10k working for this only to have a base from which to rapidly develop for PE4 when it’s released. EXPECT THIS ONE TO GO AWAY IN FAVOR OF THE NEW PROJECT.


I hope these projects assist you in rapidly creating a platform and developing for Puppet. If you have any questions, don’t hesitate to contact me via,, or one of the many other social media nexii you have available to you.

As always, these are in active, deep development. If you’ve got some Vagrant chops and/or want to contribute in any way, feel free to do pull requests, and I’ll integrate changes as soon as I’m able between customer engagements and/or other duties I may have here at Shadow-Soft.

Management May Be Missing an Important Component of DEVOPS

| Comments

As I travel around the country installing and training people in Puppet Enterprise, I’m noticing some characteristics of management perspective on the DEVOPS movement that has a disjoint with implementation and reality. In short: Managers are now hiring personnel in the field of DEVOPS for positions they may be highly qualified for, but have no institutional ability to execute on the tasks that will be assigned them in a modern infrastructure especially one that has to meet governance criteria such as ITIL, PCI, SOX, HIPAA, and various STIG requirements we see in governmental circles.

First a story, then an observation…

Early Rumblings of DEVOPS

A number of years ago I was a senior engineer in a large TV/Web property. The team was probably one of the best I’d ever worked with from an operational perspective. What I mean by that, is they not only knew how to do what they knew, but when confronted with requirements on something that did not exist, or had not been built yet, they just built it themselves. (handy to have in the days before ubiquitous workflow engines, automation tools, and deployment mechanisms!)

At the same time, the devlopment team was mostly tiered… Entry level personnel were basic coders, seniors were considerably more integrated into the nuts & bolts of the site and the leads & managers could actually commit. Quite a well organized protectionist strategy to keep the codebase clean and mostly devoid of errors. It was a great setup for a 2000-2002 era development shop. Problem was, it was 2005.

As business goes, eventually newer and more well rounded developers with experience in a new subset of tools and techniques began to be hired, and from their background they might have had elevated privileges in their past environments, the ability to commit at will (or continuously integrate?) and felt as though this somewhat “experienced” development model was archaic and slow. And it was.

Inevitably, one of these nice folks would make their way over to the operations side of the house, usually in despair, looking for ways to make their lives easier, which usually ended up in some sort of altercation over “root” level access to systems throughout the environment they had to touch. One could assume how that conversation would go, ultimately operations could not find a business justification for such a level of access, and the request was denied. This would engender a certain amount of tension between teams, and life would roll on in much the same way.

Finally, one day, one of the best developers I personally have ever had the privilege of working with came on as a contractor. (he would ultimately come on board full time and then become the Sr. Architect for the team) Everything he did would turn to gold. his development models and abilities were changing the way developers would think about what they could do, and methods and procedures were changing, deployment techologies were being tried, workflow engines were making the development side of the house quite modern by all measures.

However, the existing operational model continued along at the early-century norms, and would not/could not budge. Now, this wasn’t due to the fact that there were jerks in the department, no quite the contrary. In scearios where a team is so competent in what they do, they look for ways to script and automate away mundanity. The better the team, the stronger this backbone. The stronger the backbone, the more tendrils get attached the the core until automation and development reaches each and every part of the infrastructure. When the team builds to that level, each part hands off to the other. Centralized data stores provide the API for the site, and to touch any particular part of the infrastructure at the design level affects the entire system. So goes Infrastructure architecture.

Before there was a “DEVOPS”

As you can see, in very real terms, this was a “pre-incarnate” DEVOPS infrastructure. A little more OPS than DEV, but nonetheless automated as was possible.

But these new upstart tools were going to ruin this! Yes, they had promise and could certainly replace large portions of the existing workflow, but it could take months or years to “undo” what had already been done to supplant existing mechanisms with newer, better tools.

And therein lies the road to DEVOPS.

The Climate Today

I tell the above story to illustrate the tensions existent before the rise of DEVOPS and the subsequent automation revolution we’re currently experiencing.

Many times one would love to implement their new tools, but the operational infrastructure would prevent it. Or, lesser-informed development teams would accept no less than the highes level of access into the environment, but modern compliance standards prevent that from happening as well.

The manager that has to navigate this particular problem when hiring or resourcing a need in her infrastructure has quite the task ahead of them. Why? DEVOPS has integrated the two fields at a vector point to a degree whereby it is incredibly difficult to determine where the DEV ends and the OPS begins. Sure, there are considerably more well-defined responsibilities on the extremeties of the respective disciplines, but that joining point threatens to cause dischord in the world of the IT infrastructure and many sleepless nights for the IT manager in trying to nail down his talent needs.

Take the requirements for the “DEVOPS Application Operations Engineer” found on one of the major online employment sites posted just a few days ago for a major metro in the U.S.:

Minimum 4 years’ experience in scripting and or any development languages like C#,.NET, Python, Java, Shell, Ruby or any other open source languages.

Experience with HTML/XML and Java Script

Familiarity with Microsoft SCOM, SolarWinds Orion, Keynote, Nagios, Puppet, Chef or other monitoring, SaaS management solutions is desired

Proven experience debugging and troubleshooting software-related issues in a software development or advanced application support position

As you can see, this is a development-heavy position (that, IMO, is all over the map from a requirements perspective), but so goes job descriptions today. Read between the lines, though…

Someone needs a competent developer that isn’t completely freaked out when someone says “Puppet Environment”, “Monitoring”, or “SaaS”, that knows their way around deployment and automation and can get things done. That’s fine. Problem is, this assumes full lifecycle respnsibility when the actuality is that the future employee has a hard-line stopping point beyond which he or she will never be “allowed” to tread due to compliance alone, and that is the breakpoint between DEV and OPS in the DEVOPS world. Consider this:

There are three clearly defined worlds here, all converging on a singluar point known as DEVOPS. From the development skill and expertise of the developer to the testing and assurance retrospect of the QA Engineer, to the Security and Compliance purview of the Operational Architect, DEVOPS is not a “one trick pony” with a singularity view. It is a methodology that brings together the three worlds in a clear developmental workflow to speed safe and secure deployment with minimal errors into serving infrastructure. As often as people try and push DEVOPS into a development position or into an Operations or QA postion when hiring, success will be limited, and frustration will be the result.

What, Then, Is the Manager Missing?

As has been heavily implied thus far, the manager may be missing the fact that DEVOPS is not a position but a way of doing things. DEVOPS is a methodology, not a granting of rights or abilities. And, if we’re talking about lines of demarcation within groups, DEVOPS is a superstructure of tools built, implemented and designed by the Operational Architectural team to move, implement, and regression test code and associated objects provided by the Development Architectural team with tests, regressions, and automated mechanisms specified by the Quality Assurance Architectural team in a specific fashion and after a specific methodology that has commonly become referred to as DEVOPS.

The manager has to realize that this is not a subset of bullet points on a resume, but a wholistic approach to all of their environmental considerations that requires all teams to cooperate through to the end result: QUality Software Products delivered in as short a cycle as possible in an automated fashion with as few errors and bugs as reasonably can be remediated before going “live”.

The manager who is looking for this methodology in a single position has already lost the battle before ever posting the position.

Vagrant and Docker Love

| Comments

Not a full-on post, but more a note for myself to both investigate and test this… It would seem that Vagrant has now added provisioner support for Docker here. Be looking for a new post on this in the near future!

I’m working steadily to start off the new year, so my posting may be somewhat sporadic, but I will continue to blog Puppet fundamentals, supporting tools, and related items as I have the chance.

And in that vein, I encountered something this week I wanted to share with you.

It would seem that the new feature whereby you can include facter facts in a module for pluginsync to distribute them, but using the new mechanism of:


is not 100% reliable when distributing those facts. By this, I mean the following observed behavior.

  1. You create your fact and place it in the above directory. Say, a shell script that gives you a value.
  2. You make your fact executable, and running it natively at the shell works perfectly.
  3. You do a puppet agent run, and the fact syncs to the agent machine, but never becomes available in the facter table.
  4. You find the sync’ed location of the fact (in the losgs from the sync) and run it manually, and it works perfectly.

I spoke with some folks at Puppet just in a conversation describing what I was seeing and they suggested the following workaround:

Make the fact a file resource and place the fact in /etc/puppetlabs/facter/facts.d. This makes the fact available to the facter system, and displays correctly in the facter table and responds to the facter -p as expected.

That’s it for now! Look for more to come on workflows and tools in the very near future.

Moving My Tech Content to GitHub

| Comments

Happy New Year, all!

Well, you may notice a bit of a change in format & layout. I got tired of fighting the foibles of WordPress. Every few weeks, WordPress decided quite on its own it no longer wished to display my blog for what appeared to be no reason at all.

As such, I’ve moved to Octopress, hosted it at GitHub, and am doing a permanent redirect at it from my site until I can work out both Web & Mail while having my MX stay where it is and having my Web address move to GitHub directly.

In the meantime, all tech posts have been duplicated here, and can be found by navigating the menus.


Over the next few weeks and months, you’ll see things move around, features being added and removed, themes and plugins changing and/or disappearing as I learn Octopress and figure out all its ins and outs. Please bear with my dist during this time.

PuppetConf 2014

| Comments

Glad to be at PuppetConf with #ShadowSoft exploring all the latest and greatest in PuppetLabs.


Southeast Puppet User’s Group September

| Comments


John Ray is bringing the Puppet + Docker goodness in his talk tonight: “Deploying Docker Containers with Puppet”.  Join us each month at the Shadow Soft offices for the latest in DEVOPS topics and information.  Always fun, lots of discussion and information surrounding Puppet topics and associated technologies.  There’s always pizza and beverages of all kinds, and we’ve finally moved into our new meeting/class rooms, so come on out.

The Toolbox Grows…

| Comments

So far we’ve gotten our heads around some important things.  First and foremost, vim.  Our editor and companion for creating great code and ways to see our code in action and be able to determine at a glance whether our syntax is correct.  Also, we’ve looked at revision control.  The single largest “CYA” ohmygodimgladivegotanoldercopytorestoreto sort of paradigm where you can roll yourself back to previously “known good” revisions to save that day…besides that, it’s just darned good practice to keep your code externally saved, revision controlled, and accessible.

I’ve also talked about importance of workflow clarity and quality.  If you implement a poor workflow, you just have an automated poor workflow. Key word here is “poor”.

Next up on our browse through the “toolbox” is “Vagrant”.  What is this Vagrant, you ask?

Virtualization is paramount in today’s world in a number of ways and for a number of reasons.  For extending your server farms to handle even more application expression, to expand your own desktop machines to test/try different operating systems, and even just rolling up an ad-hoc VM so you can try something without touching a “real” machine in your environment.

Some may disagree, but I’ve found virtualization to be one of the most powerful tools added to the toolbox in years.  Not only can you prototype systems or applications, but you can prototype entire environments.  This is where Vagrant shines, and especially in the context of Puppet (master + clients), allows you to create a fully functioning Puppet environment upon which to develop, prototype, and test without ever jeopardizing even the least important system of your infrastructure.  I count that as a “win”.  Let’s see what this tool can do.

What is Vagrant?

According to its website:

Vagrant provides easy to configure, reproducible, and portable work environments built on top of industry-standard technology and controlled by a single consistent workflow to help maximize the productivity and flexibility of you and your team.

To achieve its magic, Vagrant stands on the shoulders of giants. Machines are provisioned on top of VirtualBox, VMware, AWS, or any other provider. Then, industry-standard provisioning tools such as shell scripts, Chef, or Puppet, can be used to automatically install and configure software on the machine.

There’s a lot there, but it’s just a fancy way of saying exactly what I said before.  Vagrant is essentially a framework system that wraps your virtualization engine to manage environments of VMs.  Here is where Vagrant will hold the power for us.


If Vagrant is the framework, then Virtualization is the foundation.  Now, I’ve chosen to use “Virtualbox” for my virtualization technology, but VMWare works every bit as well.  I am doing all my testing over Virtualbox, however, so YMMV.  Virtualbox is freely available from oracle, and you can download the appropriate version from Virtualbox at  I am running the latest version at 4.3.12 (as of this writing) and it serves the Vagrant system extremely well.


Next, you’ll need to install Vagrant on your system.  You can find all the right packages at  I am currently running version 1.6.3 without errors.

[warning]I want to make a disclaimer here since I’ve had an issue or two with Vagrant on a platform I don’t use-Windows.  I am a Mac & Linux user, and have had no issues using the Vagrant/Virtualbox combo on either of these.  However, literally every time I’ve used Vagrant over Windows, it’s just been a mess.  I’ve known one person (ONE!) who has gotten Vagrant to work over Windows, and it required his getting into the product, editing code, etc.  As such, I wouldn’t recommend it for those new to the platform.[/warning]

On the Mac platform, you get a .dmg file and can extract it run the installer.  Linux versions are available as RPM installs and Debian Packages.  Once you’re installed, let’s mess around a bit with Vagrant to see what we can do.

Getting Started

Vagrant is a unique tool in that it allows you to manage all these varied VMs, but adds a twist.  The big twist is that you don’t have to have the source materials for the VMs you’re installing.  In fact, the simplicity of turning up a new VM is astounding.  Take the following series of commands:

cd mkdir precise32 cd precise32 vagrant init hashicorp/precise32 vagrant up

If your Vagrant is installed correctly, a number of things start to happen.  First, Vagrant places a file in your cwd called “Vagrantfile”.  Your vagrant file (indie) looks like this:

Note that this is a long file with a lot of explanatory documentation.  In actuality, the most important part of your Vagrantfile can be summed up here:

These are the lines that are uncommented plus the top two declaratives that tell Vagrant what to do.  It’s a very simple file that does some very powerful things.  First, it checks your home directory in the ~/.vagrant.d location to see if you already have the “precise32” Vagrant source “box”.  (more on boxes later).  Next, if you do have this, it will simply start up a VM in your virtualization of choice with a randomized name.  For instance, mine is called “precise32_default_1402504453444_30545”.  Vagrant takes away the selection of an .iso image, connecting it to the virtual CD/DVD Rom, starting an installer, etc.  It simply sends you a pre-rolled image, places it in your .vagrant.d directory, and provisions the VM to respond to Vagrant commands, and starts it up within Virtualbox.  Precise32 is simply a test scenario, as Vagrant’s site has quite a number of varied and specially configured “box” files that you can use to prototype on at their “ready-made” box discovery site:  You can install boxes with too many variations and differentiations to enumerate here, and that’s not really the point for our purposes… you may find these of great assistance in your own workplace, but let’s continue.

When you run your “vagrant init” command listed above, it places a Vagrantfile, and when you do a “vagrant up”, it automatically retrieves your box file, provisions, and starts the VM.  Now, by simply running “vagrant ssh default”, you are now logged into this virtual machine!  You also have full sudo to become root and do any sort of damage you may wish to do.  If you logout (“exit” or CTRL-D), and type “vagrant destroy”, the VM goes away and you have nothing in Virtualbox.

Were we to just stop here, the power inherent in being able to just have these “Vagrantfiles” (sort of like a “Makefile” for boxes) to spin up and down test scenarios at will is incredible.  But, let’s look at this in light of the Vagrantfile, what it can do and how you can customize it.  There is an entire descriptive language surrounding Vagrant PLUS Vagrant has a plugin infrastructure whereby developers can extend Vagrant’s capabilities.  We will capitalize on these later.

So, imagine a scenario where you can create a directory, copy a text file into it, run a single command, and it automatically provisions a 4-node Puppet Enterprise infrastructure, fully installed with a master and three agents, MCollective fully installed, PuppetDB installed and in use…  literally a full installation just like you would use for your infrastructure…  Now we get powerful.  NOW we have the ability to do some cool things.

Next time, that’s exactly what we’re going to do.

“Do’s and Dont’s” for Your Puppet Environment

| Comments

IT Automation, like the features and functions offered by Puppet, is riddled with a number of pitfalls. Nothing dangerous or site-threatening in the near term, however evolving a bad plan can lead you down a painful path to re-trek when you ultimately need to demolish what you’ve done and re-tool, re-work, or even re-start from scratch.  Some simple suggestions can help smooth your integration, and also provide tools and methodologies that make changes in philosophy easy to test and implement as well as make the long road back from a disaster easy(-ier?) to navigate.

Here are some simple guidelines that can provide that foundation and framework:

DO Always Use Revision Control

It would seem this would be a foregone conclusion in this day and age, but you would be surprised just how many shops don’t have revision control of any kind in place.  A series of manifests or configurations might be tarred up and sent to the backup system, but aside from dated backups, there’s no real versioning…just monolithic archives to weed through in a time of disaster.

Revision control puts you one command away from restoring those configurations and manifests (and even your data vis-a-vis “Hiera”) to their original locations in the most recent state.

DO Rethink Your Environments

If you automate a bad workflow, you still have a bad workflow.  (albeit an automated one!)

Rethink how you do things and why.  Why do you promote code the way you do, and is there a better way to do it?  Why do you still have a manual portion to your procedure, and is it entirely necessary, or can this be remanded to Puppet to do for you as well.  What things are you doing well?  How can they improve?

Try to think through all your procedures.  There are more than you think, and they’re often less optimized than they can be.  If you’re going to implement Puppet automation, it’s time to retool.

DO Implement Slowly and Methodically

Another pitfall a lot of shops wander into is they try to do too much all at once, and do none of it well.  Either they implement too quickly and migrate a huge environment it took years to build (sometimes as much as a decade!) through a single professional services engagement or at an unrealistic pace.  Automation is complex, but if you take the time to implement correctly, piece-by-piece and hand-in-hand with your rethinking of your environment referred to above, you can revolutionize the way you work and make the environment considerably more powerful, considerably easier to work with, and ultimately release yourself to work on much more interesting problems in your environment.  Take your time to build the environment you want.

DO Engage the Community

By using Puppet, you are the beneficiary of the greatest software development paradigm in history – the Open Source movement.  People all over the world have taken part in crafting the powerful tool you have before you.  If you are able to help in like manner, by all means contribute your code to the community. (With your data in Hiera, this is easier than ever!)  Join a Puppet Users Group.  Share your clever solutions to unique problems with the community via GitHub, the Puppet Forge, your website… give back.  The more you pour in, the more you get out, and something you solve may end up baked into the final product one day in the future.

DON’T Pit Teams Against Each Other

DON’T make this a DEV vs OPS paradigm.  This is a marriage of the best tools of both worlds.  Depending on how your culture breaks down, this could be an OPS-aware way of doing development, or a DEV-informed way of doing operations.  You need to remember one thing in all of it.  The marriage of these worlds is a teamwork effort.

I was averse to the term DEVOPS when it first started being used, as it was a tool of the development world I was engaged with to cede root level access to developers.  In a properly managed, secure environment, this is always a no-no.  Development personnel are not trained systems people and rarely are.  By the same token, never ask your systems people to delve into core development, or to troubleshoot your developers’ code.  They are not tooled for that work.

This does not say that one is better than the other, nor does it say they do not share a certain amount of core skills at the basest levels. Much like the differences between civil and mechanical engineers, each has a base level of knowledge that ties them together, but each is highly specialized.  You don’t want your civil engineer building machine tools just as much as you don’t want your mechanical engineer building bridges.  Each discipline is highly specialized and carries with it nuance and knowledge you only gain through experience…experience on the job.

Instead, find a culture and a paradigm that joins the forces of these two disciplines to build something unique and special rather than wasting time with dissension and argument.

DON’T Expect Automation to Solve Everything

I know, that sounds like a sacrilege at this point, but its true.  No matter how automated your site becomes, how detailed your configuration elements are, or how much you’ve detailed your entire workflow, you still can never replace the element of human consideration and decision-making.

Automation, as I’ve said before, automates away the mundane to make time for you, DEVOPS person, to work on really interesting and curious work.  You can now write that entire new whiz bang gadget you’ve been conceptualizing for the last several years, but have never quite gotten there because you were too busy “putting out fires”.  Puppet automation is definitely a watershed in modern administration and development, but people are still needed.

Another “intangible” you may not readily think about when considering a DEVOPS infrastructure is one of culture.  The best places to work are always the best cultures brought about by the right collection of people, ideas, personalities, and management styles.  When you find that right mix of people and ideas, the workplace becomes a, forgive me, magical place to be.  Automation can never make that happen.

DON’T Starve Your Automation Environment

Automation solves a lot of things, but one thing it cannot do is feed itself.  This particular animal has a ton of needs over time.  From appropriate hardware to personnel, the environment needs time, attention, and consideration.  Remember that this is the “machine tool” of your whole company.  It is the thing that builds and maintains other things.  As such, its priority rises above that of the next web server or DNS system.

Always allocate enough resources (read: money, personnel, and time) to your environment.  If that means engineer time to work on a specially project and to do the job right, that’s what it means.  And, yes, it’s more important than meeting an arbitrarily assigned “live date” to your new widget or site or application.  The environment comes first, and all else follows.  If you give the resources and time to your automation initiatives it deserves, a number of years down the road you will look back and be amazed at the sheer amount of work your team was able to accomplish just by keeping this simple precept.

DON’T Stop Evolving

Never stop learning.  Never stop bettering yourself or your environment.  Always keep refactoring your code.  (i.e., if you wrote that Apache module 4 years ago, chances are good that what you’ve learned in the interim can go back into making that module even better.)  Always keep your people trained and engaged on the latest developments in Puppet and all the associated tools.  Never stop striving to be better and never stop reaching.  I may sound lil your coach from high school in this, but those principles he was trying to impart hold true.  If you continue to drive forward and reinvent yourself as a regular part of your forward pursuits, the endpoint of that evolution will benefit you personally, your team both vocationally and culturally, your company’s efficiency, and your environment’s impact on your bottom line.


If we keep a stronger eye on our environment and tools that rises above the simple concept of “that software I bought” and “fit it in between all the other things you have to do” and give Puppet its proper place in our company, it can truly revolutionize your workflow.  However, when properly placed culturally and from a design, implementation, and workflow perspective, it can transform any shop on levels not readily observable when looking at the price tag or the resource requirements list.  DO let Puppet transform your environment and workflow and DON’T be afraid to take the plunge.  It’s exciting, challenging, and can easily take your company to the “next level”.