Once you have enough people each working in multiple accounts it becomes a
waiting game until you’ll eventually get the dreaded “Your AWS account 666
is compromised.” email. As someone who’s been using AWS since S3 arrived this is
the first time I’ve encountered this so I thought I’d write up some notes about
what actually happens.
First comes the easy, but not recommended, part of the whole experience; push some
credentials to GitHub. You get bonus points, well faster discovery
anyway, if you use the perfect combination of
export AWS_SECRET_ACCESS_KEY or a literal add of your
.aws/credentials file. As all
AWS_ACCESS_KEYs begin ‘AKIA’ I
assume they scan the fire hose for commits containing that string.
Once the AWS scanning systems detect an access key in the wild, which in our case
took them mere minutes, You’ll receive an email to your accounts contact
address, and if you’ve got enterprise support a phone call too. The email looks
Amazon Web Services has opened case 111111111 on your behalf.
The details of the case are as follows:
Case ID: 111111111
Subject: Your AWS account 6666666666666 is compromised
Correspondence: Dear AWS Customer,
Your AWS Account is compromised! Please review the following notice and
take immediate action to secure your account.
Your security is important to us. We have become aware that the AWS
Access Key AKIATHEEVILHASESCAPED (belonging to IAM user
"ohgodpleasenotme") along with the corresponding Secret Key is publicly
available online at
If, like me, it’s the first time you’ve seen one of these you might experience
a physical flinch. Once the initial “Oh god, think of the Lambdas!” has passed
and you read on you’ll find the email is both clear and comprehensive.
It guides you through checking your accounts activity, deleting the root
accounts keys (which you don’t have, right?) and all AWS access keys in
that account that were created before the breach happened. Because we have quite
good role and account isolation we made the repository private for future
investigation, confirmed we don’t have root access keys and forced a rotation
(re: deleted all existing keys in the account.)
A little while later we received a followup phone call and email from AWS
support to check in and ensure we were OK and had actioned the advice given. The second email reiterates
the recommendation to rotate both the root password / access key
and all AWS access keys created before the breach happened. You’ll also get some
handy recommendations for
and a Trusted Advisor / Cloudwatch Events based monitoring mechanism to
help detect this kind of leak.
All in all this is actually a well handled process for what could be an
amazingly painful and stress filled event. While some of this
communication may vary for people without Enterprise Support I’d assume
at the very least you’d still get the emails. If you have good key
rotation practices this kind of event becomes just another trello card
rather than an outage and now we’ve seen one of these it’s an easy
scenario to mockup and add to our internal game day scenarios.
At $WORK I’m one of the people responsible for our SRE community and in
addition to the day to day mechanics of ensuring everyone is willing and able to
meaningfully contribute I’ve been looking at ways to gain a higher level, people
focused, view of how they’re feeling about their role. With our move to
quarterly missions now department wide it seemed like the perfect time to try our first “Quarterly SRE Health check”.
As someone more comfortable asking about system implementation than personal
feelings I heavily ‘borrowed’ the great questions from
How to Tell If You’re a Great Manager
and reworded some of them a little. For our initial survey I ended up
with fourteen questions, all except the last were multiple choice with
the canned answers of ‘Strongly disagree, Disagree, Neutral, Agree and
- I know what is expected of me at work
- I have the materials and equipment I need to do my work
- At work, I have the opportunity to do what I do best every day
- In the last seven days, I received recognition or praise for doing good work
- My line manager, or someone at work, cares about me as a person
- There is someone at work who encourages my development
- At work, my opinions seem to count
- The mission/purpose of my program makes me feel my job is important
- My co-workers are committed to doing high-quality work
- I work with people I consider close friends
- In the last three months someone at work spoke to me about my progress
- In the last six months, I had opportunities at work to learn and grow
- It’s OK to make mistakes in my role
- Any comments or feedback you would like to leave (free form response)
In order to keep the implementation as simple as possible, the answers being the
only thing that actually mattered, I created a Google form, added a disclaimer
explaining what the questions were for, who would see them and that it was
completely anonymous and then emailed it to the SRE teams. Each of the finished
questions looked like this:
I then closed the tab and waited for a few days to see what would
happen. I’m fortunate to work somewhere that’s very open and willing to
participate in this kind of engagement attempt and within a few days 75% of our
SREs had responded. A quick slack reminder and a few days later the poll was
closed and a meeting was scheduled to go over the results. After the
initial read through and presentation to the department heads we’ve
spotted positive trends to encourage, and a couple of early warning
signs to address. Considering the information it brings, and the small
time investment required it seems it’d be worth doing again, with a
couple of question changes, at the end of the next quarter.
It all started with someone trying to show me an article on their mobile
phone. As an adblock user I’d forgotten how bad the world was with all
the screen space being stolen by pop-unders, pop overs and I was soon
done. After mulling it over for a little while I decided to use it as a
flimsy excuse to buy another Raspberry Pi and trial running Pi-hole - ‘A black hole for internet advertisements’.
With the joy of Amazon Prime ensuring I’d have all the required pieces (Pi, SD
Card, case and power supply) the very next day I downloaded the newest Raspian
Lite image and went off to do something else. From the hardware arriving it took
less than 30 minutes from getting the very pretty https://etcher.io/
to burn the image to having Pi-hole installed (wrapped up with flu I
didn’t trust myself to use
dd correctly) and running. The documentation,
initial user experience, and web admin are all very well done and provide a very
smooth first encounter. I moved my iPad over to using it, manually as
Virgin media routers don’t allow you to set the DNS servers via their
built-in DHCP functionality(!), and instantly noticed empty spaces on a few of
the tabs I reloaded.
I’ve been running pi-hole for a few weeks now and on a normal day it’s blocking
between 7-12% of DNS queries, which is a nice little saving on bandwidth. The
number rises significantly when family visits with all their horrible malware
covered devices and other than adding it to my
apt-get wrappers for security
patching it’s been a great, zero touch, little service so far.
For a number of years I’ve maintained what I call my
Plan (PiP). It’s a collection
of things that ensure I have to invest at least a little time each quarter into
my career and industry. While it’s always been somewhat aspirational, in
that I don’t often complete everything, it does give me a little prod
every now and again and stops me becoming too stagnant. My first few
PiPs were done annually, but after I started seeing ever decreasing
completed items I moved to quarterly and had a lot more success.
While I’ve been laid up with flu I’ve had time to think about some stuff
and a brief review of the items on my recent PiPs shows a distinct lack
of cohesion. It’s always been more focused on motion rather than
progress, so I’ve decided to make a change for a quarter and try a
different approach, using the Objectives and Key Results (OKR) format.
If you’re looking for a concise introduction to OKRs,
Introduction to OKRs
from O’Reilly is an excellent little read. In essence Objectives and Key
Results is a framework / process for defining and tracking objectives
and their outcomes. It’s supposed to help with setting, communicating
and monitoring quarterly goals and results. They seem quite handy for
keeping the scope focused on the work that will provide the most value,
which is what I’m looking for. So instead of my old PiPs, which have
looked like this for the last 15 years:
I’ve decided to do 3 separate OKRs, each with their own objective,
running for a month each. This should provide enough time to trial the
approach, and allow a little tweaking and refinement, while being short
enough that if it’s not working for me I can try something else. I’ve mocked up
a small, slightly too vague example, based on the seemingly most common OKR
It’s been a decade and a half since I started using PiPs to keep myself honest
and as my other commitments have grown I don’t have the time I used to for
adhoc, exploratory, experimentation. I’m hoping this new approach will help
focus my limited spare time a little more into providing more worth and progress
over just motion. And if it doesn’t work? I can always go back.
In the IT industry we are reputed to be serial job hoppers. While this
may seem a little unfair, if it applies to you then you should consider
where you’re spending your limited additional time and effort. First, a
disclaimer: you need to invest enough time and effort into your current
job to stay employed.
Now that’s out of the way let’s look at our normal days. All those extra
hours and hard work you put in everyday? That’s all local currency. In the best
case your current employer and co-workers will hopefully appreciate it,
you’ll be recognised for your outcome and ability, and hopefully be
considered an integral part of the team. But that’s almost as far as it
goes. No one outside your employer, and depending on the size and
organisation, maybe not even throughout it, will ever know that you
pulled a 70 hour week to get that one release done and dusted or
stepped up and handled a Sunday emergency. It’s possible that a small
part will transfer into the wider work. This typically appears as
references, LinkedIn praise etc but most of it won’t go with you when
you change roles.
If you’re someone who likes to change jobs, short permanent roles or
contracting, you should carefully consider the balance between local and
remote respect. Writing blog posts and articles, releasing open source
projects, giving presentations, these things have value in the wider
world as well as hopefully at work and may serve you better in reaching
your career goals. Some companies are wonderful at unifying these two
threads but at the end of the day it’s your career and you need to
deliberately weigh the options.
All of those possible career value adds eat at your time, and not
everyone is in a position to do all or even some of them, but where you
can it helps to build up a portfolio of subjects larger than your day job
and makes future interviews more about culture than demonstrating
technical minutia. Nothing beats a pre-warmed audience, especially one that
already uses your code or reads your blog.
This has been on my mind recently as my working hours creep up and my
personal projects wither and I think it’s something worth taking a
moment to deliberately consider every few quarters.
Despite using Linux on pretty much every computer I’ve owned for the
last 20 years I’ve made an exception when it comes to tablet devices and
adopted an iPad into my life as commute friendly “source of all books.”
Overtime it’s been occasionally pressed into service as a camera and I
recently realised I’ve never backed any of those photos up. “That’s
something easy to remedy” I naively thought as I plugged my iPad into a
laptop and watched as it didn’t appear as a block device.
While there are many pages on the internet that explain parts of the
process of accessing your iPad file system from Linux it was awkward
enough to piece together that I decided to summarise my own commands in
this post for future me. I used the following commands on a Fedora 28
install to access an iPad Air 2.
First add the software needed to make the connection work:
# install the required packages (on fedora)
sudo dnf install ifuse libimobiledevice-utils
Once this is installed unlock the iPad and run
idevicepair pair to
pair the iPad with your Linux host. You should see a message saying
that pairing was successful. Now we have access to the device let’s
access its file system. Create the mount point and make the current
user the owner:
sudo install -d /mnt/ipad -o $USER
Finally, mount the iPad so we can access its file system:
ls -alh /mnt/ipad/
If this fails ensure the
ifuse module is loaded by running
modprobe ifuse if it isn’t. Once you’ve finished exploring don’t
umount /mnt/ipad to release the iPad.
A few jobs ago, as the number of daily meetings increased, I picked up
a tiny meeting tweak that I’ve carried with me and deployed at each
place I’ve worked since. End all meetings five minutes early. Instead of
half past, end it at 25 and instead of on the hour (complex maths
ahead) end at 55.
My reasoning is simple and selfish, I hate being late for things.
This approach gives people time to get to their next meeting.
About a year ago, I decided it’d been long enough since I last wasted
significant amounts of time playing computer games that I could buy a
gaming machine and play for a sensible amount of time and not impact
other demands for my time. I looked at all of the current generation
consoles and to be honest I was put off by the price of the games. I’m
aware of the Steam sale and considering it’s been a decade since I
played anything seriously (I still miss you, Left 4 Dead 2) my plan was
to quickly recoup the extra cost of a gaming PC by sticking to the best
games of a few years ago.
Other than obsessively 100 percenting a handful of Lego games (Lego Spider-
man! Lego Quasar!) I’ve not really played anything new, instead I have
an overly powerful, at least for its current usage, machine that is
essentially now for Age of Empires
II. I had fond
memories of the game from when I was a kid and after looking a few
things about it up I discovered that there’s actually a seriously
skilled community keeping the game alive.
I’m only a passive observer, I played one online game and my god was it
embarrassing, but there’s an amazing amount of Twitch content from a number of
community games and even sponsored competitions. Most of the material I’ve seen
has been cast by Zero Empires
or T-90Official and
there is currently the Escape Champions League tournament (with a 60k USD prize) that’s show casing
some amazing team play. It’s great to see such an awesome old game still going
I’m not a morning person. I never have been and I doubt it’ll suddenly
become one of my defining characteristics. In light of this I’ve always had a
dislike of the daily stand-up happening first thing in the morning, instead over
the years I’ve become to much prefer having it at about 4PM.
A late afternoon stand-up isn’t a common thing. Some people absolutely hate the
idea and with no scientific studies to back me up I’m essentially just stating
an opinion but I do have a few reasons.
People are sometimes late in the mornings. Having the stand-up at the
very start of the day means that anyone having issues getting to work,
dropping the kids off at school or dealing with tube delays for example,
will probably miss it. When things are slightly off having the added pressure of
your team all standing there with trello up as you stumble in, soaked and
stressed, doesn’t exactly give the best of openings for the day.
My second main point is the lack of situational awareness. A lot of my
day will change based on the rest of the departments. To understand what
impact that will have it’s helpful to have some time for other people to
start disseminating information. Did we have a small on-call issue last
night? Is anyone off sick on my team or the ones they deal with? Is
there a security alert for Nginx?
By having my stand-ups at a much later point, such as 4PM, all the urgent
issues have normally been raised at the start of the day, and hopefully
dealt with. People know about unusual circumstances and who’s not in. At
the stand-up itself people are less aspirational and actually get to talk
about what they’ve done, not what they intended to, and I’ve still got
some time to try and get anything blocked sorted before the next day.
Later sessions can also work better if you’re dealing with Americans.
It’s not too early to have to deal with them, and the time zones sync up
more. But then you can rightly say, “They are having their stand-ups
first thing in the morning!” and you’d be right, but they have bear claws
available (thanks ckolos!), so it all balances out in the end.
There are downsides to a later time. Team wide issues might lay hidden
for longer in the morning. People might leave early to pick the kids up
and some people will find the later slot more disruptive to their
afternoon flow. It’s not going to be for everyone but if a morning slot isn’t
working for the team then maybe it’s time to shake things up a little and try a
later time. Maybe 4PM.
The summer conference submission season is slowly subsiding and after
reading through a combined total of a few thousand submissions I’ve got
some hastily compiled thoughts. But before we get started, a
disclaimer: I don’t publicly present. My views on this are from the
perspective of a submission reviewer and audience member. And remember, we
want to say yes. We have slots to fill and there’s nothing more
satisfying than giving a new speaker a chance and seeing the feedback
consist of nothing but 10’s. Hopefully some of these points will help me say yes
to you in the future.
Firstly I’ll address one of the most common issues, even when it’s not a
completely fair one - people submitting on the subject their employer
focuses on. As an organiser you want your speakers to have solid and
wide experience on their chosen topic and it’s often easier to find
those depths in people that live and breath a certain thing day in and
day out. However it’s easy to submit what appears to be a vendor sales
pitch. In non-anonymised submissions there will always be a moment of
“This company has product / service in that area. Is this a sales
pitch?” Audiences have paid for their tickets and being trapped for a 45
minute white paper spiel is a sure way to fill the twitter stream with
To balance that I’m much more careful when dealing with those kind of
submissions. Despite my defensive watchfulness there are things you can
do to make it easier to say yes. If it’s unrelated to your paid work but
in the same industry, say so. You should also state how the talk relates
to your product. Is it a feature overview for enterprise customers or
all generic theory anyone can use? Be explicit about how much of the
talk is product specific, “20 minutes on the principles, 10 on the open
source offerings and 10 on the enterprise product additions” might not
be exactly what I want to see but it’s better than my assumption. I
should also note no matter how much it hurts your chances you should be
honest. Event organisers chat. A lot of the Velocity program chairs know
each other outside of work, there’s a lot of cross over between
DevOpsDays events and London isn’t that big. If you’re given the benefit
of the doubt and you were less than honest then good luck in the future.
As an aside this also applies to sponsors. We know who’s a joy to deal
with and who’s going to keep us dangling for 8 months.
Onto my next bugbear. Submissions that include things like “8
solutions to solving pipeline problems.” If you have a number in your submission
topic or introduction and don’t tell me what they are in the body of the
submission I’ll assume you don’t know either. Context and content are
immensely important in submissions and it’s so hard to highly rate a
talk with no actual explanation of what it’s covering. If you say
“The six deadly secrets of the monkey king” then you better list
those six secrets with a little context on each or expect to be
dropped a point or three. The organisers probably won’t be in the
session and without enough context to know what the audience will be
seeing, neither will you.
My third personal catch is introducing a new tool at a big event. Unless you’re
someone like Hashicorp or AWS then you need to be realistic about your impact. I
will google technologies and programs I don’t recognise in submissions and if
the entire result set is your GitHub page and some google group issues then it’s
probably not ready for one of the bigger events. Instead start at a user group or
two. A couple of blog posts and then maybe something bigger at a site like dzone
or the new stack. Build some buzz and presence so I can tell that people are
adopting it and finding merit. There’s often an inadvertent benefit to this, a
lot of user groups record and upload their sessions and this is great benefit
for after the anonymised stage of the reviews. Being able to see someone present
and know that they can manage an audience and don’t look like they are about to
break into tears for 25 minutes is reassuring and a great presentation style can
help boost your submission.
Other than my personal idiosyncrasies there are a few things you should
always consider. What’s the audience getting out of this? Why are you
the person to give it to them? What’s the actionable outcome from this
session? You don’t have to be a senior Google employee but you do need
to have an angle on the material. This is especially true on subjects
like career paths or health issues where it’s easy to confuse personal
anecdotes with data. Does your employer have evangelists or advocates that spend
a large amount of their time presenting or reviewing submissions? If so reach
out and ask them for a read through. It’s in their interests to not see 10
submissions from the same company that all get rejected for not having enough
information to be progressed. I wouldn’t normally single someone out but
if, as an example, you’re working for Microsoft and submitting to a
conference, especially a DevOpsDays, and you’ve not asked
Bridget Kromhout to review your
submission then you’re missing a massive opportunity. She’s seen everything
get submitted at least once and can nearly always find something constructive to
improve. There’s probably a similar person at many large tech companies and
getting their opinion will almost always help the process.
In general it’s a pleasure to read so many thoughtful submissions but with just
a little bit more effort in the right places it becomes a lot easier to get the
reviewers to say yes. And then comes the really difficult part for us.