A short 2017 review

It’s time for a little 2017 navel gazing. Prepare for a little self-congratulation and a touch of gushing. You’ve been warned. In general my 2017 was a decent one in terms of tech. I was fortunate to be presented a number of opportunities to get involved in projects and chat to people that I’m immensely thankful for and I’m going to mention some of them here to remind myself how lucky you can be.

Let’s start with conferences, I was fortunate enough to attend a handful of them in 2017. Scale Summit was, as always, a great place to chat about our industry. In addition to the usual band of rascals I met Sarah Wells in person for the first time and was blown away by the breadth and depth of her knowledge. She gave a number of excellent talks over 2017 and they’re well worth watching. The inaugural Jeffcon filled in for a lack of Serverless London (fingers crossed for 2018) and was inspiring throughout, from the astounding keynote by Simon Wardley keynote all the way to the after conference chats.

I attended two DevopsDays, London, more about which later, and Stockholm. It was the first in Sweden and the organisers did the community proud. In a moment of annual leave burning I also attended Google Cloud and AWS Summits at the Excel centre. It’s nice to see tech events so close to where I’m from. I finished the year off with the GDS tech away day, DockerCon Europe and Velocity EU.

DevopsDays holds a special place in my heart as the conference and community that introduced me to so many of my peers that I heartily respect. The biggest, lasting contribution, of Patricks for me is building those bridges. When the last “definition of Devops” post is made I’ll still cherish the people I met from that group of very talented folk. That’s one of the reasons I was happy to be involved in the organisation of my second London DevOps. You’d be amazed at the time, energy and passion the organisers, speakers and audience invest in to a DevopsDays event. But it really does show on the day(s).

I was also honoured to be included in the Velocity Europe Program Committee. Velocity has always been one of the important events of industry and to go from budgeting most of a year in advance to attend to being asked to help select from the submitted papers, and even more than that, be a session chair, was something I’m immensely proud of and thankful to James Turnbull for even thinking of me. The speakers, some of who were old hands at large events and some giving their first conference talk (in their second language no less!), were a pleasure to work with and made a nerve wracking day so much better than I could have hoped. It was also a stark reminder of how much I hate speaking in front of a room full of people.

Moving away from gushing over conferences, I published a book. It was a small experiment and it’s been very educational. It’s sold a few copies, made enough to pay for the domain for a few years and led to some interesting conversations with readers. I also wrote a few Alexa skills. While they’re not the more complicated or interesting bits of code from last year they have a bit of a special significance to me. I’m from a very non-technical background so it’s nice for my family to actually see, or in this case hear, something I’ve built.

Other things that helped keep me sane were tech reviewing a couple of books, hopefully soon to be published, and reviewing talk submissions. Some for conferences I was heavily involved in and some for events I wasn’t able to attend. It’s a significant investment of time but nearly every one of them taught me something. Even about technology I consider myself competent in.

I still maintain a small quarterly Pragmatic Investment Plan (PiP), which I started a few years ago, and while it’s more motion than progress these days it does keep me honest and ensure I do at least a little bit of non-work technology each month. Apart from Q1 2017 I surprisingly managed to read a tech book each month, post a handful of articles on my blog, and attend a few user groups here and there. I’ve kept the basics of the PiP for 2018 and I’m hoping it keeps me moving.

My general reading for the year was the worst it’s been for five years. I managed to read, from start to finish, 51 books. Totalling under 15,000 pages. I did have quite a few false starts and unfinished books at the end which didn’t help.

Oddly, my most popular blog post of the year was Non-intuitive downtime and possibly not lost sales. It was mentioned in a lot of weekly newsletters and resulted in quite a bit of traffic. SRE weekly also included it, which was a lovely change of pace from my employer being mentioned in the “Outages” section.

All in all 2017 was a good year for me personally and contained at least one career highlight. In closing I’d like to thank you for reading UnixDaemon, especially if you made it this far down, and let’s hope we both have an awesome 2018.

Terraform testing thoughts

As your terraform code grows in both size and complexity you should invest in tests and other ways to ensure everything is doing exactly what you intended. Although there are existing ways to exercise parts of your code I think Terraform is currently missing an important part of testing functionality, and I hope by the end of this post you’ll agree.

I want puppet catalog compile testing in terraform

Our current terraform testing process looks a lot like this:

  • precommit hooks to ensure the code is formatted and valid before it’s checked in
  • run terraform plan and apply to ensure the code actually works
  • execute a sparse collection of AWSSpec / InSpec tests against the created resources
  • Visually check the AWS Console to ensure everything “looks correct”

We ensure the code is all syntactically validate (and pretty) before it’s checked in. We then run a plan, which often finds issues with module paths, names and such, and then the slow, all encompassing, and cost increasing apply happens. And then you spot an unexpanded variable. Or that something didn’t get included correctly with a count.

I think there is a missed opportunity to add a separate phase, between plan and apply above, to expose the compiled plan in a easy to integrate format such as JSON or YAML. This would allow existing testing tools, and things like custom rspec matchers and cucumber test cases, to verify your code before progressing to the often slow, and cash consuming, apply phase. There are a number of things you could usefully test in a serialised plan output. Are your “fake if” counts doing what you expect? Are those nested data structures translating to all the tags you expect? How about the stringified splats and local composite variables? And what are the actual values hidden behind those computed properties? All of this would be visible at this stage. Having these tests would allow you to catch a lot of more subtle logic issues before you invoke the big hammer of actually creating resources.

I’m far from the first person to request this and upstream have been fair and considerate but it’s not something that’s on the short term road map. Work arounds do exist but they all have expensive limitations. The current plan file is in a binary format that isn’t guaranteed to be backwards compatible to external clients. Writing a plan output parser is possible but “a tool like this is very likely to be broken by future Terraform releases, since we don’t consider the human-oriented plan output to be a compatibility constraint” and hooking the plan generation code, an approach taken by palantir/tfjson will be a constant re-investment as terraforms core rapidly changes.

Adding a way to publish the plan in an easy to process way would allow many other testing tools and approaches to bloom and I hope I’ve managed to convince you that it’d be a great addition to terraform.

Show server side response timings in chrome developer tools

While trying to add additional performance annotations to one of my side projects I recently stumbled over the exceptionally promising Server-Timing HTTP header and specification. It’s a simple way to add semi-structured values describing aspects of the response generation and how long they each took. These can then be processed and displayed in your normal web development tools.

In this post I’ll show a simplified example, using Flask, to add timings to a single page response and display them using Google Chrome developer tools. The sample python flask application below returns a web page consisting of a single string and some fake information detailing all the actions assembling the page could have required.

# cat hello.py

from flask import Flask, make_response
app = Flask(__name__)


@app.route("/")
def hello():

    # Collect all the timings you want to expose
    # each string is how long it took in microseconds
    # and the human readable name to display
    sub_requests = [
        'redis=0.1; "Redis"',
        'mysql=2.1; "MySQL"',
        'elasticsearch=1.2; "ElasticSearch"'
    ]

    # Convert timings to a single string
    timings = ', '.join(sub_requests)

    resp.headers.set('Server-Timing', timings)

    return resp

Once you’ve started the application, with FLASK_APP=hello.py flask run, you can request this page via curl to confirm the header and values are present.

    $ curl -sI http://127.0.0.1:5000/ | grep Timing
    ...
    Server-Timing: redis=0.1; "Redis", mysql=2.1; "MySQL", elasticsearch=1.2; "ElasticSearch"
    ...

Now we’ve added the header, and some sample data, to our tiny Flask application let’s view it in Chrome devtools. Open the developer tools with Ctrl-Shift-I and then click on the network tab. If you hover the mouse pointer over the coloured section in “Waterfall” you should see an overlay like this:

Chrome devtools response performance graph

The values provided by our header are at the bottom under “Server Timing”.

Support for displaying the values provided with this header isn’t yet wide spread. The example, and screenshot, presented here are from Chrome 62.0.3202.75 (Official Build) (64-bit) and may require changes as the spec progresses from its current draft status. The full potential of the Server-Timing header won’t be obvious for a while but even with only a few supporting tools it’s still a great way to add some extra visibility to your projects.

Use your GitHub SSH key with AWS EC2 (via Terraform)

Like most people I have too many credentials in my life. Passwords, passphrases and key files seem to grow in number almost without bound. So, in an act of laziness, I decided to try and remove one of them. In this case it’s my AWS EC2 SSH key and instead reuse my GitHub public key when setting up my base AWS infrastructure.

Once you start using EC2 on Amazon Web Services you’ll need to create, or supply an existing, SSH key pair to allow you to log in to the Linux hosts. While this is an easy enough process to click through I decided to automate the whole thing and use an existing key, one of those I use for GitHub. One of its lesser known features is that GitHub exposes a users SSH public keys. This is available from everywhere, without authenticating against anything and so seemed like a prime candidate for reuse.

The terraform code to do this was a lot quicker to write than the README. As this is for my own use I could use a newer version of 0.10.* and harness the locals functionality to keep the actual resources simpler to read by hiding all the variable composing in a single place. You can find the results of this, the terraform-aws-github-ssh-keys module on GitHub, and see an example of its usage here:

module "github-ssh-keys" {
  source = "deanwilson/github-ssh-keys/aws"

  # fetch the ssh key from this user name
  github_user = "deanwilson"

  # create the key with a specific name in AWS
  aws_key_pair_name = "deanwilson-from-github"
}

I currently use this for my own test AWS accounts. The common baseline setup of these doesn’t get run that often in comparison to the services running in the environment so I’m only tied to GitHub being available occasionally. Once the key’s created it has a long life span and has no external network dependencies.

After the module was (quite fittingly) available on GitHub I decided to go a step further and publish it to the Terraform Module Registry. I’ve never used it before so after a brief read about the module format requirements, which all seem quite sensible, I decided to blunder my way through and see how easy it was. The Answer? Very.

Screen shot of my module on the Terraform registry

The process was pleasantly straight forward. You sign in using your GitHub account, select your Terraform modules from a drop down and then you’re live. You can see how github-ssh-keys looks as an example. Adding a module was quick, easy to follow, and well worth finishing off your modules with.

Prevent commits to the local git master branch

I’ve been a fan of Yelps pre-commit git hook manager ever since I started using it to Prevent AWS credential leaks. After a recent near miss involving a push to master I decided to take another look and see if it could provide a safety net that would only allow commits on non-master branches. It turns out it can, and it’s actually quite simple to enable if you follow the instructions below.

Firstly we’ll install pre-commit globally.

pip install pre-commit

Before we enable the plugin we’ll make a commit to an unprotected local master branch to ensure everything’s working the way we think it is.

# confirm we're on master
$ git branch
* master

# create a local change we can work with
$ echo "Text" >> text

$ git add text

# successfully commit the change
$ git commit -v -m "Add text"
[master e1b84e5] Add text
 1 file changed, 1 insertion(+)
 create mode 100644 text

Now we’ve confirmed we can commit to master normally we’ll add the pre-commit config to prevent it.

$ cat .pre-commit-config.yaml

- repo: https://github.com/pre-commit/pre-commit-hooks.git
  sha: v0.9.5
  hooks:
  -  id: no-commit-to-branch

and then we activate the config.

$ pre-commit install
pre-commit installed at ~/protected-branch-test/.git/hooks/pre-commit

If anything fails then you’ll probably need to read through ~/.pre-commit/pre- commit.log to find the issue. Now we’ve installed the pre-commit pip, added its config, and then enabled it we should be protected. No more accidental committing to the master branch for us! But let’s verify.

# make a change to the checkout
echo "More text" >> text

git commit -m "Added more text"
... snip ...
Don't commit to branch.............Failed
... snip ...

# and the change is not committed.

By default this plugin protects the master branch. If you have other branches you want to deny commits on you can add the args key to the config as shown in this snippet.

  hooks:
  -  id: no-commit-to-branch
     args:
     - --branch=release

If you need to commit to master while this plugin is enabled you can use the --no-verify argument to git commit to disable all pre-commit hooks.

It’s worth noting you can also prevent inadvertent pushes to master at the remote end by enabling branch protection on a number of the popular git providers, both GitHub and BitBucket support this. This approach has the advantage of not needing client side configuration but does require that all your providers support it, and that you actually enable it on each of them and their repositories. While you can of course do that by hand there are also a few tools that will manage this for you, but that’s a something for a different post.

I wrote a book

A few months ago while stunningly bored I decided, in a massive fit of hubris, that I was going to write and publish a technical book. I wrote a pile of notes and todo items and after a good nights sleep decided it’d be a lot more work than I had time for. So I decided to repurpose Puppet CookBook and try going through the publication process with that instead. But (disclaimer) with a different title as there is already an excellent real book called Puppet Cookbook that goes in to a lot more depth than my site does.

My Puppet CookBook has always been a hand built site. It’s ruby, erb templates and currently uses blueprint for the CSS. My hope was to just add another small wrapper and some slightly modified templates and pass that to a tool that actually knows how to format ebooks. Which ended up being very close to what actually happened. I did some research on both asciidoctor and pandoc and ended up using the latter, mostly because its desired input format was closest to how I already produce the site.

Completely skipping the monotonous updating and rewording of a number of recipes part of the process we soon get to the interesting part, tooling. I generated markdown from my own tooling (but as one massive file instead of many pages) and then ran pandoc against it and some supporting information, such as a cover image and book specific material. The actual commands look a lot like this:

#!/bin/bash
set -xue

book=$foo.epub

# generate the raw markdown text from my custom template
bundle exec ruby -I lib bin/build-markdown.rb > ebook/book-source.md


# Generate the ebook itself
pandoc 
  -t epub3 
  -o ebook/$book ebook/metadata.yaml ebook/introduction.md ebook/book-source.md ebook/changelog.md 
  --table-of-contents --toc-depth 2 
  --epub-cover-image ebook/${book}-cover.png

# open the reader to see if it looks... passable
FBReader ebook/$book

Once the book was generating, and looked readable, it was time to attempt submitting to see what else I was missing. I’m not going to detail all the text fields you have to complete when submitting, as it’s dull, self-explanatory and will probably change over time. It is worth noting that uploading the epub here is the first time I actually received any feedback on how valid my submission was. And it very much wasn’t. While Amazon does offer their tool chain on Linux it’s all pre-built 32 bit binaries. Even when I managed to make them run they didn’t return any errors and seem to be validating off old versions of the ebook specs. After a few iterations of blind fix and upload I had to swallow my pride and ask a friend to use his Apple laptop to run some of the Amazon publishing tool chain, such as Kindle Previewer, for me so I could see the reasons my submission was being rejected. This was worth the shame as it gave me actionable errors messages and cut the cycle time down significantly.

Once the previewer ran clean I re-submitted the book and it went through on the first try. I then went and did other stuff for a few hours and then returned, search for the books name and ‘lo there was a page for it on Amazon.co.uk

There are still a few oddities I have no clue how royalties work when publishing via the Amazon Kindle publisher. I think I’m now due about 3 cans of coke worth but it’d cost me more in time to figure out how to get that sweet 2.750 than I’ll ever make from it. You don’t get a free copy. As an author if I want to see how the book looks to customers I have to buy a copy. You also can’t submit a book as ‘free’. If you’re in the UK then the minimum you can sell it for it 99p. There is a way around this, as I’d like to offer mine for free, but you need to set the book up on another site and then have Amazon price match. Which is a massive pain to set up. I also had to do a lot of googling for things like “how to insert a hard page break” in a kindle book (<mbp:pagebreak /> for the record) but that might be an indication of how unprepared I was.

I’m not going to link to the book here, and I’m also not going to recommend that anyone actually buy it. All the material contained inside it can be seen, with actual coloured syntax, on Puppet CookBook. Which is something I state very clearly in the foreword.

I’ve found the sales numbers interesting. Firstly I’ve done zero promotion of it, and I’ve not linked to it from anywhere. The google results for it are amazingly terrible. I have problems finding it with a search engine and I wrote it. There’s also no way to see how many copies you’ve sold since release. In terms of raw numbers I think I’m still in double rather than treble digits, you can also see that I won’t be retiring on my royalties.

Picture of very low royalties

Writing a tech book is never going to be a retirement plan. In my case it’s even slightly worse as the material is online and free, and viewed by about 1500 unique visitors a day for the last 6 or so years. This was never intended to be anything beyond an experiment but I’ve had some nice feedback from readers and it’s something I always wanted to do so I think it was worth investing a weekend into it. The technology is freely available, the process is actually quite easy (if you have something other than Linux available) and I’d heartily recommend it anyone interested.

Managing multiple puppet modules with modulesync

With the exception of children, puppies and medical compliance frameworks managing one of something is normally much easier than managing a lot of them. If you have a lot of puppet modules, and you’ll eventually always have a lot of puppet modules, you’ll get bitten by this and find yourself spending as much time managing supporting functionality as the puppet code itself.

Luckily you’re not the first person to have a horde of puppet modules that share a lot of common scaffolding. The fine people at Vox Pupuli had the same issue and maintain an excellent tool, modulesync that solves this very problem. With modulesync and a little YAML you’ll soon have a consistent, easy to iterate, on set of modules.

To get started with module sync you need three things, well four if you count the puppet module horde you want to manage.

I’ve been using modulesync for some of my projects for a while but we recently adopted it for the GDS Operations Puppet Modules so there’s now a full, but nascent, example we can look at. You can find all the modulesync code in our public repo.

First we set up the basic module sync config in modulesync.yml -

---
git_base: 'git@github.com:'
namespace: gds-operations
branch: modulesync
...
# vim: syntax=yaml

This YAML mostly controls how we interact with our upstream. git_base is the base of the URL to run git operations against. In our case we explicitly specify GitHub (which is also the default) but this is easy to change if you use bitbucket, gitlab or a local server. We treat namespace as the GitHub organisation modules are under. As we never push directly to master we specify a branch our changes should be pushed to for later processing as a pull request.

The second config file, managed_modules.yml, contains a list of all the modules we want to manage:

---
- puppet-aptly
- puppet-auditd
- puppet-goenv

By default modulesync will perform any operations against every module in this file. It’s possible to filter this down to specific modules but there’s only really value in doing that as a simple test. After all keeping the modules in sync is pretty core to the tools purpose.

The last thing to configure is a little more abstract. Any files you want to manage across the modules should be placed in the moduleroot directory and given a .erb extension. At the moment we’re treating all the files in this directory as basic, static, files modulesync does expand them provides a @configs hash, which contains any values you specify in the base config_defaults.yml file. These values can also be overridden with more specific values stored along side the module itself in the remote repository.

Once you’ve created the config files and added at least a basic file to moduleroot, a LICENSE file is often a safe place to start, you can run modulesync to see what will be changed. In this case I’m going to be working with the gds-operations/puppet_modulesync_config repo.

bundle install

# run the module sync against a single module and show potential changes
bundle exec msync update -f puppet-rbenv --noop

This command will filter the managed modules (using the -f flag to select them) clone the remote git repo(s), placing them under modules/, change the branch to either master or the one specified in modulesync.yml and then present a diff of changes from the expanded templates contained in moduleroot against the cloned remote repo. None of the changes are actually made thanks to the --noop flag. If you’re happy with the diff you can add a commit message (with -m message), remove --noop and then run the command again to push the amended branch.

bundle exec msync update -m "Add LICENSE file" -f puppet-rbenv

Once the branch is pushed you can review and create a pull request as usual.

Screen shot of GitHub pull request from modulesync change

We’re at a very early stage of adoption so there is a large swathe of functionality we’re not using so I’ve not mentioned. If you’re actually using the moduleroot templates as actual templates you can have a local override, in each remote module/github repo, that can localise the configuration and be correctly merged with the main configuration. This allows you to push settings out to where they’re needed while still keeping most modules baselined. You can also customise the syncing workflow to specify bumping the minor version, updating the CHANGELOG and a number of other helpful shortcuts provided by modulesync.

Once you get above half-a-dozen modules it’s a good time to take a step back and think about how you’re going to manage dependencies, versions, spec_helpers and such in an ongoing, iterative way and modulesync presents one very helpful possible solution.

Job applications and GitHub profile oddities

I sift through a surprising amount, to me at least, of curricula vitae / resumes each month and one pattern I’ve started to notice is the ‘fork only’ GitHub profile.

There’s been a lot written over the last few years about using your GitHub profile as an integral part of your job application. Some in favour, some very much not. While each side has valid points when recruiting I like to have all the information I can to hand, so if you include a link to your profile I will probably have a rummage around. When it comes to what I’m looking for there are a lot of different things to consider. Which languages do you use? Is the usage idiomatic? Do you have docs or tests? How do you respond to people in issues and pull requests? Which projects do you have an interest in? Have you solved any of the same problems we have?

Recently however I’ve started seeing a small but growing percentage of people that have an essentially fork only profile. Often of the bigger, trendier projects, Docker, Kubernetes, Terraform for example, and there will be no contributed code. In the most blatant case there were a few amended CONTRIBUTORS files with the applicants name and email but no actual changes to the code base.

Although you shouldn’t place undue weight on an applicants GitHub profile in most cases, and in the Government we deliberately don’t consider it in any phase past the initial CV screen, it can be quite illuminating. In the past it provided an insight towards peoples attitude, aptitudes and areas of interest and now as a warning sign that someone may be more of a system gamer than a system administrator.

AWS Trivia – Broken user data and instance tag timing

Have you ever noticed in the AWS console, when new instances are created, the “Tags” tab doesn’t have any content for the first few seconds? A second or two before values are added may not seem like much but it can lead to elusive provisioning issues, especially if you’re autoscaling and have easily blamed network dependencies in your user data scripts.

A lot of people use Tag values in their user data scripts to help ‘inflate’ AMIs and defer some configuration, such as which config management classes to apply, to run time when the instance is started, rather than embedding them at build time when the AMI itself is created. In a stupendous amount of cases everything will work exactly as you expect. Instances will start, tags will be applied and user data will determine how to configure the instance based on their values. However, very rarely, the user data script will begin before the tags are applied to the instance.

If your script requires these tag values then you need to consider this rare but occasional issue and decide how to handle it. You can ignore it, as it’s very rare. If you’re using tags to assign config management roles or similar provide sensible defaults such as applying the base class. It’s possible to ensure that instances that don’t detect their tags fail their health checks and are marked as defective and terminated before they come into service. You can also stack the odds a little more in your favour by having tags reading happen a little later in your user data, run that apt-get update or AWS agent installing curl before fetching the tags for instance to give the tags more time to be applied.

Tagging is often a simple after thought but in the cloud you need a very firm understanding of which things are atomic units and which are separate services and can fail independently. Although tags may seem like a direct property of the instance they are actually handled (I think) by a completely different service, which can always fail. Understanding this split also explains why you can’t read tags and their values from the local metadata service. Which as an aside can, even more rarely, be unavailable. That was a fun afternoon.

I’ll leave you with a closing comment from the days when you could only have 10 tags. Tag values can be complex strings, for example, JSON objects. Possibly even compressed and base64 encoded JSON objects. Just putting that out there.

Over engineering a badly thought out terraform data provider

All the well managed AWS accounts I have access to include some form of security group control over which IP addresses can connect to them. I have a home broadband connection that provides a dynamic IP address. These two things do not play well together.

Every now and again my commands will annoyingly fail with ‘access denied’. I’ll run a curl icanhazip.org, raise a new PR against the isolated bootstrap project that controls my access, get it reviewed and after running terraform, restore my access. This process has to be improvable right? I know, more code will fix it!

As an experiment in writing a custom data provider for Terraform, the real reason I did any of this, I decided to try and remove the IP address from the code base completely and instead make it a run time determined value. The, never to be merged, Icanhazip data source pull request that implements this is still available and shows how to add a simple data source to terraform. Becoming a little more familiar with the code base, and how to test it properly, thanks to Richard Clamp of the Terraform GitLab provider for lots of pointers on testing, were worth the time invested even with the rejected PR.

Was this data provider a good idea? No, not really. The HTTP data source solution proposed by Martin Atkins is a much better approach and requires no changes to terraform itself. The code is easy to follow:

# main.tf

# use the swiss army knife http data source to get your IP
data "http" "my_local_ip" {
    url = "https://ipv4.icanhazip.com"
}

# write it to a local file to prove everything's fine
resource "local_file" "my_ip" {
    content  = "${chomp(data.http.my_local_ip.body)}"
    filename = "/tmp/my_ip"
}

and it does exactly what my pile of Golang does -

$ terraform apply
Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

$ cat /tmp/my_ip
312.533.143.224

The more time that passes since this little experiment, the more I think the whole idea was a terrible one. My use case, bootstrapping AWS access with security groups, is at best a very niche one. It assumes your bootstrap tool isn’t restricted and only works if everyone executes terraform from the same location. Was it a complete waste of time? Not really. I learned a lot about how data sources work and how I’d implement a sensible one in the future. I also know the Terraform PR reviewers are quick, courteous and good at spotting well meaning mistakes, which as a user of the tool itself is quite reassuring.