AWS Support and leaked credentials

Once you have enough people each working in multiple accounts it becomes a waiting game until you’ll eventually get the dreaded “Your AWS account 666 is compromised.” email. As someone who’s been using AWS since S3 arrived this is the first time I’ve encountered this so I thought I’d write up some notes about what actually happens.

First comes the easy, but not recommended, part of the whole experience; push some credentials to GitHub. You get bonus points, well faster discovery anyway, if you use the perfect combination of export AWS_ACCESS_KEY and export AWS_SECRET_ACCESS_KEY or a literal add of your .aws/credentials file. As all AWS_ACCESS_KEYs begin ‘AKIA’ I assume they scan the fire hose for commits containing that string.

Once the AWS scanning systems detect an access key in the wild, which in our case took them mere minutes, You’ll receive an email to your accounts contact address, and if you’ve got enterprise support a phone call too. The email looks like this:

Amazon Web Services has opened case 111111111 on your behalf.

The details of the case are as follows:

Case ID: 111111111
Subject: Your AWS account 6666666666666 is compromised
Severity: Urgent
Correspondence: Dear AWS Customer,

Your AWS Account is compromised! Please review the following notice and
take immediate action to secure your account.

Your security is important to us. We have become aware that the AWS
Access Key AKIATHEEVILHASESCAPED (belonging to IAM user
"ohgodpleasenotme") along with the corresponding Secret Key is publicly
available online at

If, like me, it’s the first time you’ve seen one of these you might experience a physical flinch. Once the initial “Oh god, think of the Lambdas!” has passed and you read on you’ll find the email is both clear and comprehensive. It guides you through checking your accounts activity, deleting the root accounts keys (which you don’t have, right?) and all AWS access keys in that account that were created before the breach happened. Because we have quite good role and account isolation we made the repository private for future investigation, confirmed we don’t have root access keys and forced a rotation (re: deleted all existing keys in the account.)

A little while later we received a followup phone call and email from AWS support to check in and ensure we were OK and had actioned the advice given. The second email reiterates the recommendation to rotate both the root password / access key and all AWS access keys created before the breach happened. You’ll also get some handy recommendations for aws-labs git-secrets and a Trusted Advisor / Cloudwatch Events based monitoring mechanism to help detect this kind of leak.

All in all this is actually a well handled process for what could be an amazingly painful and stress filled event. While some of this communication may vary for people without Enterprise Support I’d assume at the very least you’d still get the emails. If you have good key rotation practices this kind of event becomes just another trello card rather than an outage and now we’ve seen one of these it’s an easy scenario to mockup and add to our internal game day scenarios.

Quarterly SRE Health check

At $WORK I’m one of the people responsible for our SRE community and in addition to the day to day mechanics of ensuring everyone is willing and able to meaningfully contribute I’ve been looking at ways to gain a higher level, people focused, view of how they’re feeling about their role. With our move to quarterly missions now department wide it seemed like the perfect time to try our first “Quarterly SRE Health check”.

As someone more comfortable asking about system implementation than personal feelings I heavily ‘borrowed’ the great questions from How to Tell If You’re a Great Manager and reworded some of them a little. For our initial survey I ended up with fourteen questions, all except the last were multiple choice with the canned answers of ‘Strongly disagree, Disagree, Neutral, Agree and Strongly Agree.

  • I know what is expected of me at work
  • I have the materials and equipment I need to do my work
  • At work, I have the opportunity to do what I do best every day
  • In the last seven days, I received recognition or praise for doing good work
  • My line manager, or someone at work, cares about me as a person
  • There is someone at work who encourages my development
  • At work, my opinions seem to count
  • The mission/purpose of my program makes me feel my job is important
  • My co-workers are committed to doing high-quality work
  • I work with people I consider close friends
  • In the last three months someone at work spoke to me about my progress
  • In the last six months, I had opportunities at work to learn and grow
  • It’s OK to make mistakes in my role
  • Any comments or feedback you would like to leave (free form response)

In order to keep the implementation as simple as possible, the answers being the only thing that actually mattered, I created a Google form, added a disclaimer explaining what the questions were for, who would see them and that it was completely anonymous and then emailed it to the SRE teams. Each of the finished questions looked like this:

A question with 5 checkbox answers

I then closed the tab and waited for a few days to see what would happen. I’m fortunate to work somewhere that’s very open and willing to participate in this kind of engagement attempt and within a few days 75% of our SREs had responded. A quick slack reminder and a few days later the poll was closed and a meeting was scheduled to go over the results. After the initial read through and presentation to the department heads we’ve spotted positive trends to encourage, and a couple of early warning signs to address. Considering the information it brings, and the small time investment required it seems it’d be worth doing again, with a couple of question changes, at the end of the next quarter.

Pi-hole – the first two weeks

It all started with someone trying to show me an article on their mobile phone. As an adblock user I’d forgotten how bad the world was with all the screen space being stolen by pop-unders, pop overs and I was soon done. After mulling it over for a little while I decided to use it as a flimsy excuse to buy another Raspberry Pi and trial running Pi-hole - ‘A black hole for internet advertisements’.

With the joy of Amazon Prime ensuring I’d have all the required pieces (Pi, SD Card, case and power supply) the very next day I downloaded the newest Raspian Lite image and went off to do something else. From the hardware arriving it took less than 30 minutes from getting the very pretty to burn the image to having Pi-hole installed (wrapped up with flu I didn’t trust myself to use dd correctly) and running. The documentation, initial user experience, and web admin are all very well done and provide a very smooth first encounter. I moved my iPad over to using it, manually as Virgin media routers don’t allow you to set the DNS servers via their built-in DHCP functionality(!), and instantly noticed empty spaces on a few of the tabs I reloaded.

Number of DNS queries graph

I’ve been running pi-hole for a few weeks now and on a normal day it’s blocking between 7-12% of DNS queries, which is a nice little saving on bandwidth. The number rises significantly when family visits with all their horrible malware covered devices and other than adding it to my apt-get wrappers for security patching it’s been a great, zero touch, little service so far.

Swapping Pragmatic Investment Plans for OKRs

For a number of years I’ve maintained what I call my Pragmatic Investment Plan (PiP). It’s a collection of things that ensure I have to invest at least a little time each quarter into my career and industry. While it’s always been somewhat aspirational, in that I don’t often complete everything, it does give me a little prod every now and again and stops me becoming too stagnant. My first few PiPs were done annually, but after I started seeing ever decreasing completed items I moved to quarterly and had a lot more success.

While I’ve been laid up with flu I’ve had time to think about some stuff and a brief review of the items on my recent PiPs shows a distinct lack of cohesion. It’s always been more focused on motion rather than progress, so I’ve decided to make a change for a quarter and try a different approach, using the Objectives and Key Results (OKR) format.

If you’re looking for a concise introduction to OKRs, Introduction to OKRs from O’Reilly is an excellent little read. In essence Objectives and Key Results is a framework / process for defining and tracking objectives and their outcomes. It’s supposed to help with setting, communicating and monitoring quarterly goals and results. They seem quite handy for keeping the scope focused on the work that will provide the most value, which is what I’m looking for. So instead of my old PiPs, which have looked like this for the last 15 years:

A list of objectives from a previous PiP

I’ve decided to do 3 separate OKRs, each with their own objective, running for a month each. This should provide enough time to trial the approach, and allow a little tweaking and refinement, while being short enough that if it’s not working for me I can try something else. I’ve mocked up a small, slightly too vague example, based on the seemingly most common OKR format.

A 4 section OKR template with mock values

It’s been a decade and a half since I started using PiPs to keep myself honest and as my other commitments have grown I don’t have the time I used to for adhoc, exploratory, experimentation. I’m hoping this new approach will help focus my limited spare time a little more into providing more worth and progress over just motion. And if it doesn’t work? I can always go back.

Respect can be a local currency

In the IT industry we are reputed to be serial job hoppers. While this may seem a little unfair, if it applies to you then you should consider where you’re spending your limited additional time and effort. First, a disclaimer: you need to invest enough time and effort into your current job to stay employed.

Now that’s out of the way let’s look at our normal days. All those extra hours and hard work you put in everyday? That’s all local currency. In the best case your current employer and co-workers will hopefully appreciate it, you’ll be recognised for your outcome and ability, and hopefully be considered an integral part of the team. But that’s almost as far as it goes. No one outside your employer, and depending on the size and organisation, maybe not even throughout it, will ever know that you pulled a 70 hour week to get that one release done and dusted or stepped up and handled a Sunday emergency. It’s possible that a small part will transfer into the wider work. This typically appears as references, LinkedIn praise etc but most of it won’t go with you when you change roles.

If you’re someone who likes to change jobs, short permanent roles or contracting, you should carefully consider the balance between local and remote respect. Writing blog posts and articles, releasing open source projects, giving presentations, these things have value in the wider world as well as hopefully at work and may serve you better in reaching your career goals. Some companies are wonderful at unifying these two threads but at the end of the day it’s your career and you need to deliberately weigh the options.

All of those possible career value adds eat at your time, and not everyone is in a position to do all or even some of them, but where you can it helps to build up a portfolio of subjects larger than your day job and makes future interviews more about culture than demonstrating technical minutia. Nothing beats a pre-warmed audience, especially one that already uses your code or reads your blog.

This has been on my mind recently as my working hours creep up and my personal projects wither and I think it’s something worth taking a moment to deliberately consider every few quarters.

Accessing an iPads file system from Linux

Despite using Linux on pretty much every computer I’ve owned for the last 20 years I’ve made an exception when it comes to tablet devices and adopted an iPad into my life as commute friendly “source of all books.” Overtime it’s been occasionally pressed into service as a camera and I recently realised I’ve never backed any of those photos up. “That’s something easy to remedy” I naively thought as I plugged my iPad into a laptop and watched as it didn’t appear as a block device.

While there are many pages on the internet that explain parts of the process of accessing your iPad file system from Linux it was awkward enough to piece together that I decided to summarise my own commands in this post for future me. I used the following commands on a Fedora 28 install to access an iPad Air 2.

First add the software needed to make the connection work:

    # install the required packages (on fedora)
    sudo dnf install ifuse libimobiledevice-utils

Once this is installed unlock the iPad and run idevicepair pair to pair the iPad with your Linux host. You should see a message saying that pairing was successful. Now we have access to the device let’s access its file system. Create the mount point and make the current user the owner:

    sudo install -d /mnt/ipad -o $USER

Finally, mount the iPad so we can access its file system:

    ifuse /mnt/ipad

    ls -alh /mnt/ipad/

If this fails ensure the ifuse module is loaded by running lsmod, and run modprobe ifuse if it isn’t. Once you’ve finished exploring don’t forget to umount /mnt/ipad to release the iPad.

Slightly Shorter Meetings

A few jobs ago, as the number of daily meetings increased, I picked up a tiny meeting tweak that I’ve carried with me and deployed at each place I’ve worked since. End all meetings five minutes early. Instead of half past, end it at 25 and instead of on the hour (complex maths ahead) end at 55.

My reasoning is simple and selfish, I hate being late for things. This approach gives people time to get to their next meeting.

Rediscovering Age of Kings

About a year ago, I decided it’d been long enough since I last wasted significant amounts of time playing computer games that I could buy a gaming machine and play for a sensible amount of time and not impact other demands for my time. I looked at all of the current generation consoles and to be honest I was put off by the price of the games. I’m aware of the Steam sale and considering it’s been a decade since I played anything seriously (I still miss you, Left 4 Dead 2) my plan was to quickly recoup the extra cost of a gaming PC by sticking to the best games of a few years ago.

Other than obsessively 100 percenting a handful of Lego games (Lego Spider- man! Lego Quasar!) I’ve not really played anything new, instead I have an overly powerful, at least for its current usage, machine that is essentially now for Age of Empires II. I had fond memories of the game from when I was a kid and after looking a few things about it up I discovered that there’s actually a seriously skilled community keeping the game alive.

I’m only a passive observer, I played one online game and my god was it embarrassing, but there’s an amazing amount of Twitch content from a number of community games and even sponsored competitions. Most of the material I’ve seen has been cast by Zero Empires or T-90Official and there is currently the Escape Champions League tournament (with a 60k USD prize) that’s show casing some amazing team play. It’s great to see such an awesome old game still going strong.

The 4PM stand-up

I’m not a morning person. I never have been and I doubt it’ll suddenly become one of my defining characteristics. In light of this I’ve always had a dislike of the daily stand-up happening first thing in the morning, instead over the years I’ve become to much prefer having it at about 4PM.

A late afternoon stand-up isn’t a common thing. Some people absolutely hate the idea and with no scientific studies to back me up I’m essentially just stating an opinion but I do have a few reasons.

People are sometimes late in the mornings. Having the stand-up at the very start of the day means that anyone having issues getting to work, dropping the kids off at school or dealing with tube delays for example, will probably miss it. When things are slightly off having the added pressure of your team all standing there with trello up as you stumble in, soaked and stressed, doesn’t exactly give the best of openings for the day.

My second main point is the lack of situational awareness. A lot of my day will change based on the rest of the departments. To understand what impact that will have it’s helpful to have some time for other people to start disseminating information. Did we have a small on-call issue last night? Is anyone off sick on my team or the ones they deal with? Is there a security alert for Nginx?

By having my stand-ups at a much later point, such as 4PM, all the urgent issues have normally been raised at the start of the day, and hopefully dealt with. People know about unusual circumstances and who’s not in. At the stand-up itself people are less aspirational and actually get to talk about what they’ve done, not what they intended to, and I’ve still got some time to try and get anything blocked sorted before the next day. Later sessions can also work better if you’re dealing with Americans. It’s not too early to have to deal with them, and the time zones sync up more. But then you can rightly say, “They are having their stand-ups first thing in the morning!” and you’d be right, but they have bear claws available (thanks ckolos!), so it all balances out in the end.

There are downsides to a later time. Team wide issues might lay hidden for longer in the morning. People might leave early to pick the kids up and some people will find the later slot more disruptive to their afternoon flow. It’s not going to be for everyone but if a morning slot isn’t working for the team then maybe it’s time to shake things up a little and try a later time. Maybe 4PM.

Some talk submission thoughts

The summer conference submission season is slowly subsiding and after reading through a combined total of a few thousand submissions I’ve got some hastily compiled thoughts. But before we get started, a disclaimer: I don’t publicly present. My views on this are from the perspective of a submission reviewer and audience member. And remember, we want to say yes. We have slots to fill and there’s nothing more satisfying than giving a new speaker a chance and seeing the feedback consist of nothing but 10’s. Hopefully some of these points will help me say yes to you in the future.

Firstly I’ll address one of the most common issues, even when it’s not a completely fair one - people submitting on the subject their employer focuses on. As an organiser you want your speakers to have solid and wide experience on their chosen topic and it’s often easier to find those depths in people that live and breath a certain thing day in and day out. However it’s easy to submit what appears to be a vendor sales pitch. In non-anonymised submissions there will always be a moment of “This company has product / service in that area. Is this a sales pitch?” Audiences have paid for their tickets and being trapped for a 45 minute white paper spiel is a sure way to fill the twitter stream with complaints.

To balance that I’m much more careful when dealing with those kind of submissions. Despite my defensive watchfulness there are things you can do to make it easier to say yes. If it’s unrelated to your paid work but in the same industry, say so. You should also state how the talk relates to your product. Is it a feature overview for enterprise customers or all generic theory anyone can use? Be explicit about how much of the talk is product specific, “20 minutes on the principles, 10 on the open source offerings and 10 on the enterprise product additions” might not be exactly what I want to see but it’s better than my assumption. I should also note no matter how much it hurts your chances you should be honest. Event organisers chat. A lot of the Velocity program chairs know each other outside of work, there’s a lot of cross over between DevOpsDays events and London isn’t that big. If you’re given the benefit of the doubt and you were less than honest then good luck in the future. As an aside this also applies to sponsors. We know who’s a joy to deal with and who’s going to keep us dangling for 8 months.

Onto my next bugbear. Submissions that include things like “8 solutions to solving pipeline problems.” If you have a number in your submission topic or introduction and don’t tell me what they are in the body of the submission I’ll assume you don’t know either. Context and content are immensely important in submissions and it’s so hard to highly rate a talk with no actual explanation of what it’s covering. If you say “The six deadly secrets of the monkey king” then you better list those six secrets with a little context on each or expect to be dropped a point or three. The organisers probably won’t be in the session and without enough context to know what the audience will be seeing, neither will you.

My third personal catch is introducing a new tool at a big event. Unless you’re someone like Hashicorp or AWS then you need to be realistic about your impact. I will google technologies and programs I don’t recognise in submissions and if the entire result set is your GitHub page and some google group issues then it’s probably not ready for one of the bigger events. Instead start at a user group or two. A couple of blog posts and then maybe something bigger at a site like dzone or the new stack. Build some buzz and presence so I can tell that people are adopting it and finding merit. There’s often an inadvertent benefit to this, a lot of user groups record and upload their sessions and this is great benefit for after the anonymised stage of the reviews. Being able to see someone present and know that they can manage an audience and don’t look like they are about to break into tears for 25 minutes is reassuring and a great presentation style can help boost your submission.

Other than my personal idiosyncrasies there are a few things you should always consider. What’s the audience getting out of this? Why are you the person to give it to them? What’s the actionable outcome from this session? You don’t have to be a senior Google employee but you do need to have an angle on the material. This is especially true on subjects like career paths or health issues where it’s easy to confuse personal anecdotes with data. Does your employer have evangelists or advocates that spend a large amount of their time presenting or reviewing submissions? If so reach out and ask them for a read through. It’s in their interests to not see 10 submissions from the same company that all get rejected for not having enough information to be progressed. I wouldn’t normally single someone out but if, as an example, you’re working for Microsoft and submitting to a conference, especially a DevOpsDays, and you’ve not asked Bridget Kromhout to review your submission then you’re missing a massive opportunity. She’s seen everything get submitted at least once and can nearly always find something constructive to improve. There’s probably a similar person at many large tech companies and getting their opinion will almost always help the process.

In general it’s a pleasure to read so many thoughtful submissions but with just a little bit more effort in the right places it becomes a lot easier to get the reviewers to say yes. And then comes the really difficult part for us.