Building a highly available ELK solution with Puppet, Part II: The base profile

by | Mar 28, 2016 | blogs

NOTE: this series is a work-in-progress that will be finished in due course.  I leave it up here because the code and approach is likely to be nonetheless useful to people engaged in building a similar ELK / Puppet solution.

Introduction

In this second part of our series, ‘Building a highly available ELK solution with Puppet’, we look at our example base profile.

The example is artificially simple, and provides an illustration of the concepts for readers who are new to the roles and profiles pattern, and allows me to make some general points about writing roles and profiles that inform my decisions in subsequent ELK-related posts. Most readers will already have their own base profile, and if not, will probably have requirements that go beyond this contrived example.

If you are already familiar with the roles and profiles pattern, you may want to read the section about the yum configuration, and you may be interested in the firewall_multi module.

As mentioned in part I, the source code is available at Github. It is licensed under the MIT license.

Configuring the Puppetfile

Before we can write the base profile we need to install the Puppet Forge modules that it will depend upon, and to do that, we must set up our Puppetfile. If you have not used a Puppetfile before, have a look at the documentation here.

Why Librarian-puppet

It should be noted that I am using Tim Sharpe’s Librarian-puppet, the original Puppetfile processor and, in my opinion, still the best. Many users will be using r10k both to install modules as specified in Puppetfile, and also to deploy their code into target environments. Meanwhile, Puppet Enterprise users may be using Code Manager, which uses r10k under the hood.

The advantage of Librarian-puppet is that it is simpler, and it resolves dependencies. And if you, like me, intend to deploy your code onto Puppet Masters using the deployment capabilities of your CI/CD system, you may find that Librarian-puppet is still the right choice.

Puppetfile config for base profile

Our base profile requires the following modules:

We therefore add the following lines to our Puppetfile (source code):

forge 'https://forgeapi.puppetlabs.com' mod 'puppetlabs/stdlib' mod 'puppetlabs/ntp' mod 'alexharvey/firewall_multi' read more

Dumping the catalog in rspec-puppet

by | Mar 13, 2016 | blogs

I always knew there was a better way.

In a previous post I documented a procedure for compiling a catalog without logging onto a Puppet Master.  The procedure is useful but also complicated.

Some would ask, why can’t Rspec-puppet just dump the catalogs it compiles during unit testing?

It can; I just found a bit of the answer here, and the rest of it inside a debugger.

How to dump the catalog in Rspec-puppet

I assume we have Rspec-puppet set up already.  If not, try the Rspec-puppet tutorial.

Imagine we have a simple test as follows:

require 'spec_helper' describe 'myclass' do it { is_expected.to compile.with_all_deps } end read more

Parallelising rspec-puppet

by | Jan 3, 2016 | blogs

I recently migrated a client away from Andrew Cunningham’s puppet-validator – an open source project that simply compiles catalogs based on configurable fact values – to rspec-puppet.

The advantages of rspec-puppet are many and, obviously, being able to do more than just compile catalogs is one advantage.  However, Andrew’s project also had some advantages; in particular it used threads to parallelise catalog compilation.  This meant that some 100 catalogs could be compiled and tested on my 8 CPU-core laptop in less than 3 minutes.  After setting up rspec-puppet, however, I found that the same tests were now taking over 20 minutes.

It seems to me that the Puppet community has thus far tolerated rspec-puppet’s slowness.  To illustrate, I’ll focus in this post on the very mature Puppet Labs Apache module and show how parallelising its rspec-puppet tests by setting up Michael Grosser’s parallel_tests would get the current execution time of about 30 minutes (on my laptop) down to under three.

Running the rspec tests in puppetlabs/apache

To get started, let’s clone the puppetlabs/apache module:

$ cd /tmp $ git clone https://github.com/puppetlabs/puppetlabs-apache.git read more

Using catalog-diff while refactoring Puppet code

by | Jan 2, 2016 | blogs

In yesterday’s post I showed how you can compile a Puppet catalog from a bundle on a laptop.  Today I’m going to show how you can use Zack Smith’s catalog diff tool to assist with complex refactoring changes.

Code examples

For the purpose of describing how to use the catalog diff tool, it will be better to use an artificially simple code example.  Imagine we have all of our code in site.pp as follows:

node 'myhost.example.com' { file { '/tmp/myfile': ensure => file, content => "My hostname is ${::hostname}\n", } } read more

Compiling a puppet catalog – on a laptop

by | Dec 31, 2015 | blogs

From time to time I have wished that I could easily compile a Puppet catalog from my laptop. Use cases that spring to mind include, "hey, wouldn't it be great if I could see what this super-complicated Puppet Forge module is actually doing without having to spin up a VM and 'puppet apply' it". Othertimes it has just been curiosity.