Today at work, I needed to prepare the patching of a two-nodes cluster. Not only I should patch this cluster, but I should also mimic the same patching level as the currently used prod server. Well Well. Instead of making a diff of the showrev and try to sort out which patch is installed on this node, not on the other and so on, I tried to define a new way of doing this kind of things. I'll describe my method here, don't hesitate to comment, suggest or criticize something :)

The Idea

The idea is to use PCA to achieve everything. As you may (or may not) know, PCA is based upon patchdiag.xref files, which are provided by SUN Oracle once a day.

They contain the list of latest released patches as well as dependencies. The Idea of my solution is to generate a patchdiag.xref based on the patching level I should match. I could then use PCA together with this patchdiag.xref on the two nodes I need to patch. Child-Game!

WeSunSolve to the rescue

Again, you may (or not) know that WeSunSolve allow you to register yourself and use the Panel as a little server dashboard. You can enter some server name and link some patching level to them. This is being done using a simple "showrev -p" output that you can paste on the website to add patch level to a server.

Once you got two (or more) server, you can use the newly added feature Server Upgrade. You can choose two patching level there to generate a patchdiag.xref file:

• Source patching level, is the patch level of the server you'll need to patch.
• Destination patching level, is the patch level of the server you want to mimic.

Use PCA, and voila!

Next step is known by you all! Just use pca together with the patchdiag.xref file and see the output:

 $ ./pca -X . -f explorer.XXXXXXXXX.YYYYYYYYY-2011.12.17.02.00 -l
 Using /home/wildcat/YYYYYYYYYYY/./patchdiag.xref from Feb/03/12
 Host: XXXXXXXXXXX (SunOS 5.9/Generic_122300-05/sparc/sun4u)
 List: missing (133/210205)
 
 Patch  IR   CR RSB Age Synopsis
   -  - - -
 112951 13 < 14 RS- 999 SunOS 5.9: patchadd and patchrm Patch
 111711 16 < 18 R-- 999 SunOS 5.9: 32-bit Shared library patch for C++
 111712 16 < 18 R-- 999 SunOS 5.9: 64-Bit Shared library patch for C++
 **SNIPPED**

It seems to me that these are people who interact with the world in a fundamentally different way from the way that I do. That doesn’t make them wrong – just different (and maybe a little sadomasochistic).

I have often thought it would be cool to have some putative automater person whose sole job is go round and improve all these sort of silly processes and do things like making the coffee machine report outages to IRC (and other such little things that bring joy and happiness) but somehow I doubt this idea will catch on any time soon.

james@lucinda:/var/log$ ls -l dmesg*
-rw-r----- 1 root adm 45822 Feb  4 08:41 dmesg
-rw-r----- 1 root adm 45822 Jan 30 00:58 dmesg.0
-rw-r----- 1 root adm 12264 Jan 30 00:28 dmesg.1.gz
-rw-r----- 1 root adm 12266 Jan 30 00:19 dmesg.2.gz
-rw-r----- 1 root adm 12271 Jan 29 20:02 dmesg.3.gz
-rw-r----- 1 root adm 12130 Jan 29 19:48 dmesg.4.gz

Old logs are the in same directory as current logs which is just noisy but more importantly there is also a whole slew of problems with log rotation, compression, keeping arcane syslog configuration updated as things change, running out of space if you mess up etc. And let’s face it, having logs sitting on a server like this is next to useless – you want them pushed to a central location so that you can keep track of what’s going on.

With this mind, I finally got round to trying out an idea I had a while ago about how to clean this stuff up. I now have this:

james@lucinda:/var/log$ ls -l dmesg*
prw-r--r-- 1 root root 0 Feb  5 14:05 dmesg

My logs are now named pipes! From an application’s perspective nothing has changed – the log file is the same place and looks and acts like a file. Of course, this isn’t very useful if nothing is connected to the other end of the pipe, so I wrote a little ruby application called Monsoon that sits and watches named pipes in directory that you configure and every time something is sent down one of these pipes, it packages it up into a json fragment as follows:

    {
        "time": "Sun Feb 05 14:05:41 +0000 2012",
        "message": "test message",
        "origin": {
            "route": "/var/log/dmesg",
            "hostname": "lucinda"
        }
    }

and sends off to a TCP server socket that you configure – but this of course could be sent off via stomp or HTTP instead – where you can do something useful with it (put into a database, logstash, SOLR, and so on) It’s then just a case of ensuring that whatever is storing the logs has enough disk space instead of your whole web cluster or whatever (though obviously you should still monitor that too).

I’m quite pleased with the results so far. I’m going to test some more and see how things go.

james@lucinda:/var/log$ ls -l dmesg*
-rw-r----- 1 root adm 45822 Feb  4 08:41 dmesg
-rw-r----- 1 root adm 45822 Jan 30 00:58 dmesg.0
-rw-r----- 1 root adm 12264 Jan 30 00:28 dmesg.1.gz
-rw-r----- 1 root adm 12266 Jan 30 00:19 dmesg.2.gz
-rw-r----- 1 root adm 12271 Jan 29 20:02 dmesg.3.gz
-rw-r----- 1 root adm 12130 Jan 29 19:48 dmesg.4.gz

Old logs are the in same directory as current logs which is just noisy but more importantly there is also a whole slew of problems with log rotation, compression, keeping arcane syslog configuration updated as things change, running out of space if you mess up etc. And let’s face it, having logs sitting on a server like this is next to useless – you want them pushed to a central location so that you can keep track of what’s going on.

With this mind, I finally got round to trying out an idea I had a while ago about how to clean this stuff up. I now have this:

james@lucinda:/var/log$ ls -l dmesg*
prw-r--r-- 1 root root 0 Feb  5 14:05 dmesg

My logs are now named pipes! From an application’s perspective nothing has changed – the log file is the same place and looks and acts like a file. Of course, this isn’t very useful if nothing is connected to the other end of the pipe, so I wrote a little ruby application called Monsoon that sits and watches named pipes in directory that you configure and every time something is sent down one of these pipes, it packages it up into a json fragment as follows:

    {
        "time": "Sun Feb 05 14:05:41 +0000 2012",
        "message": "test message",
        "origin": {
            "route": "/var/log/dmesg",
            "hostname": "lucinda"
        }
    }

and sends off to a TCP server socket that you configure – but this of course could be sent off via stomp or HTTP instead – where you can do something useful with it (put into a database, logstash, SOLR, and so on) It’s then just a case of ensuring that whatever is storing the logs has enough disk space instead of your whole web cluster or whatever (though obviously you should still monitor that too).

I’m quite pleased with the results so far. I’m going to test some more and see how things go.

Most of these teams had to fight with some very basic things. They lacked a decent monitoring system or monitoring at all. They didn’t deploy systems, they installed it by hand. Systems where not documented etc.

So here are some guidelines, I try to aspire with my team. This is by far not a complete list of things you need to run successful operations but it should give you a fair hint about what it takes.

Also please note that you might want to adapt your own policy a bit to fit your needs. I’m coming from the web industry, but we still run our own hardware, so this might especially not fit a typical cloud based infrastructure.

Systems

A System is considered the lowest part of our infrastructure and services. All rules defined here, should be considered in all other policies.

A system….

• is documented at a central location.
• is monitored and being graphed.
• is being backuped.
• is updated regularly.
• has a defined production level. (spare, pre-production, production)
• has a defined owner and maintainer.
• has a predefined maintenance level.
• has a predefined availability.
• has a physical location.
• has a unique name, which is resolvable by DNS.
• has only required software installed.
• was installed with all currently available updates.
• was inspected and approved by a second man before being released to production.
• All parts are functional at any time. All Faults get documented RFN and repaired as soon as possible.
• There are always 2+ people informed about it.
• Network access vectors are defined.
• Configurations are not only available locally (including scripts).
• Sensible data gets protected.

Hardware

A piece of hardware can be anything from a big server to a small temperature sensor in your server room.

A piece of hardware…

• has a maintenance contract or spare hardware available.
• has got an inventory number.
• is labeled (hostname + inventory).
• is physically secure (environmental! and mechanical access control).
• has got a bill, which is documented at a central location.
• should have redundant power supplies.
• should have some kind of out of band management solution (OOB).
• has at least one power circuit connected to an electronic circuit protected by an uninterruptible power supply (USV).

All tools needed to open and repair any part of the system are available.

Servers

A server…

• has at least two disks configured with RAID >= 1.
• has at least two separate network interface cards (NICs).
• has all RAID controllers backed with battery backed write caches (BBWC).
• was dimensioned with adequate future-proof hardware.
• has a lifetime of 2+ years.

Switches

A switch…

• is manage- or configurable.
• is supported by the configuration backup software in use (e.g. RANCID)
• provides the following protocols: STP, SNMP, IPv6 support (mgmt+multicast), RADIUS for AAA
• does not forward the default VLAN (1) on it’s uplink/trunk ports.
• does have a description for every port in use (including hostname and interface, e.g.: server01#eth0, server01#oob, switch03#24)
• does not have any enabled, unused ports: set them to disabled and remove any other configuration for this port.
• blocks or does not forward any discovery protocols on it’s user ports.
• is using AAA for authenticating users.
• logs to a central syslog server.

Operating Systems

An operating system (OS) is considered as everything running on a server or instance, to support a service or an application.

An Operating System…

• uses OS-CHOICE-HERE/stable as default distribution on servers.
• uses OS-CHOICE-HERE as default on clients.
• is rebooting without any manual interventions.
• provides access by SSH.
• does not permit root login via SSH.
• has a root password set.
• has the current time, synchronized with a time server and uses TIMEZONE-CHOICE-HERE as time zone.
• can resolve internal and internet names via DNS.
• installs software by packages.
• installs packages from a central internal repository and the official distribution repositories.
• software installed by packages should conform to the FHS.
• software not installed by packages should be installed by a reproducible deployment process.
• has sane defaults set, for user and process environments (locales, shells, screen, got some handy tools, etc.).
• should not provide typical compiler tools (gcc, build-essential).
• provides a manageable AAA concept (e.g. automated provisioning and de-provisioning of staff users).
• sends mails destinated for root to a central location.
• provides a local mailer.

Hostnames

Hostnames exist to identify every part of your infrastructure uniquely. They are used to refer to systems in your configurations and in discussions. You should think about a naming convention, but here are some rough guidelines.

Hostnames …

• have to be unique.
• have to end with a number, which should never be reused and always be incremented.

Services

A service is considered as everything running on a server’s operating system, to provide continuous functionality (e.g. a script or an application).

A service…

• does only log errors and auditing information. Application services may as well log more information (e.g. Apache access log).
• has defined log retention times.
• logs to syslog unless it’s not possible.
• is authenticating only on secure connections.
• has an adequate and future-proof dimensioned datastore.
• was deployed in a reproducible way.

Networks

A network is considered any part of infrastructure, which is used to interconnect servers or systems. (Layer 1,2,3,4,…)

A Network…

• has clear entry and routing points.
• has a diagram which describes access vectors, the logical and physical setup.
• is deployed in adequate and future-proof dimensions (vlans, ip addresses, bandwidth).
• uses structured cabling.
• there is no cross-cabling, except for very rare situations (e.g. HA cabling).
• should not be used for multiple purposes at least not share one of the following classifications.

  Class	Description
  net	Internet/upstream network
  mgmt	Management network (monitoring, remote access)
  traffic	Site local traffic network
  backup	Traffic network for backups
  voip	Voip Telephony network
  clients	A network with client workstations.
  devel	A network with development machines.
  staging	A network with staging equipment.

  
  
  
• OOBs are easy to reach, even in case of an outage.
• VLAN-IDs are considered global, create a list.
• All VLAN-IDs below 99 are switch-local.
• VLANs have a name and a location.
• All address space is considered global (vlans, ip- and mac addresses, including RFC1918)

To round up my article, here is a example checklist we use to peer review new systems:

Example Review Checklist

Every newly deployed host or instance should undergo a peer-review process. The checklist below will provide you with a couple of base acceptance criteria and is going to ensure a certain level of quality. Give it to any other sysadmin and ask him or her to check the system, before it’s put into production.

* DNS works (including reverse dns)               :
* SSH login works                                 :
* Host+services monitored                         :
* Host+services graphed                           :
* All Filesystems backuped                        :
* Database dumps                                  :
* All Updates installed                           :
* Host in HostDoc                                 : 
* Puppet works                                    :
* Time is accurate                                :
* Root mails are being delivered                  :
* Firewall is active                              :
* No unneeded services are reachable (nmap)       :
* Network configuration works (+ipv6)             :
* Syslog/dmesg/oob logs are clean of errors       :

-- Physical Host --

* Root password documented                        :
* Root login works                                :
* OOB password documented                         :
* OOB login works                                 :
* OOB monitored                                   :
* Switch ports are labeled (+ documented)         :
* Hardware is labeled (+ documented in rack docu) :
* Firmware up to date                             :
* RAID level is > 1 and all disks OK              :

There's already a good number of people that have confirmed their presence and some people have asked

As for practical details .. the plan is simple.
I`m going to be at the place somewhere between 8:30 and 9:00 on monday. ( Hey .. it's the day after Fosdem you know :))

The only thing I've planned is to do a get to know eachother round around 10:30 after that I`m expecting the hackathon to be self organising,

There will be water, coffee , etc , IP connectivity, and electricity.

The location is still Duboisstraat 50, Antwerp

Free parking is on the Hardenvoort or Kempenstraat ( 3minutes walk) , paid parking right in front of the door.

<group name="netsnmp-memory-nonlinux" rrdRepository="/opt/opennms/share/rrd/snmp/">
        <expression type="low" expression="memAvailReal / memTotalReal * 100.0" ds-type="node" ds-label="" value="5.0" rearm="10.0" trigger="2"/>
</group>

ie. if free memory drops below 5% then an event will be created. The alert will be cancelled automatically if free memory subsequently rises above 10%

I wanted to configure some specific nodes with a different threshold, eg. generate an event when free memory drops below 2.5%.

Here's what I did.

Add new Surveillance Category

Add nodes to new Surveillance Category

Add a new group to thresholds.xml:

    <group name="netsnmp-memory-linux-2.5" rrdRepository="/opt/opennms/share/rrd/snmp/">
        <expression type="low" ds-type="node" value="2.5" rearm="5.0"
            trigger="2" filterOperator="or" expression="(memAvailReal + memCached) / memTotalReal * 100.0"/>
    </group>

Update threshd-configuration.xml to modify the existing netsnmp-memory-linux package and add a new netsnmp-memory-linux-2.5 package:

    <package name="netsnmp-memory-linux">
        <filter>IPADDR != '0.0.0.0' &amp; nodeSysOID == '.1.3.6.1.4.1.8072.3.2.10' &amp; ! ( catincNetSNMP-Mem-2_5 )</filter>
        <include-range begin="1.1.1.1" end="254.254.254.254"/>
        <include-range begin="::1" end="ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff" />
        <service name="SNMP" interval="300000" user-defined="false" status="on">
            <parameter key="thresholding-group" value="netsnmp-memory-linux"/>
        </service>
    </package>

    <package name="netsnmp-memory-linux-2.5">
        <filter>IPADDR != '0.0.0.0' &amp; nodeSysOID == '.1.3.6.1.4.1.8072.3.2.10' &amp; catincNetSNMP-Mem-2_5</filter>
        <include-range begin="1.1.1.1" end="254.254.254.254"/>
        <include-range begin="::1" end="ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff" />
        <service name="SNMP" interval="300000" user-defined="false" status="on">
            <parameter key="thresholding-group" value="netsnmp-memory-linux-2.5"/>
        </service>
    </package>

The tricky bit is the "catincNetSNMP-Mem-2_5". This is a function "catinc" which matches all nodes in the specified category ie. "NetSNMP-Mem-2_5" in this example. The first use of it is in the netsnmp-memory-linux category to exclude nodes in the NetSNMP-Mem-2_5 category. The second use is to include nodes in the NetSNMP-Mem-2_5 category.

In the post, I described automatically adding the DNS entries with the dnsimple cookbook. I did this as a proof of concept, but I didn’t add it to all my nodes, instead using my existing data bag-driven solution.

That said, this post serves as a brief document on how you can mimic this behavior with your own environment.

Encrypted Data Bag

I put my DNSimple credentials in an encrypted data bag. Since I have to decrypt and read the entire thing anyway, I also store the relevant data there. I keep my encrypted data bags in a bag called secrets. The structure looks like this:

1
2
3
4
5
6
7

{
  "id": "dnsimple",
  "api_token": "DNSimple API Token Here",
  "domain": "your-domain.example.com",
  "username": "DNSimple username",
  "password": "DNSimple password"
}

Replace the values with your values. Encrypting the data is optional, but requires that you create a secret key or key file. Read my previous post on the topic for more information.

1
2

% knife data bag from file secrets dnsimple.json
% knife data bag from file secrets dnsimple.json –secret-file /etc/chef/encrypted_data_bag_secret

The first command just uploads the data bag item, the second encrypts it. Note that I manage my workstation with Chef, so I will use the same secret file as the Chef default. The secret file needs to be copied to each system that will need it.

Recipe

The recipe looks like this:

1
2
3
4
5
6
7
8
9

dnsimple = encrypted_data_bag_item("secrets", "dnsimple")
dnsimple_record "#{node['hostname']}.int.#{dnsimple['domain']}" do
  content node['ipaddress']
  type "A"
  action :create
  username dnsimple['username']
  password dnsimple['password']
  domain dnsimple['domain']
end

As mentioned in the previous blog post, the encrypted_data_bag_item method is in a library. Either add that library to your cookbook, or use the class directly.

1

dnsimple = Chef::EncryptedDataBagItem.load("secrets", "dnsimple")

If you’re not using an encrypted data bag, then the item can be accessed with the normal method:

1

dnsimple = data_bag_item("bagname", "dnsimple")

The real work happens in the dnsimple_record LWRP, which will add an “A” record for the system running the recipe. Note that the actual entry is going to use the int subdomain, and it will use the domain stored in the data bag item. It also will use the default IP address of the node, which means the IP for the interface with the default route.

At my day job, I worked with a customer that uses tmux for remote pairing between developers. At the time, tmux had better customizability, and better split-pane support (screen didn’t yet have vertical split). I stuck with tmux ever since, and was very pleased when an iTerm2 update announced integration with tmux.

iTerm2

For those who aren’t aware, iTerm2 is an alternate terminal program for Mac OS X. It is actually an updated codebase from the original, iTerm, which is effectively unmaintained. iTerm2 offers a lot of excellent features like split panes, Growl support, and many more.

One of the excellent new features is integration with tmux.

iTerm2’s tmux integration

If you already have iTerm2 installed, you may have seen the update check prompt you to update. You also need to install a special version of tmux that has the integration patched. The iTerm2 author is working with the tmux author to get this into the latest tmux codebase, so hopefully the custom compiled version won’t be necessary soon.

Using the new feature is relatively straightforward. Start up iTerm2 like normal. Then run tmux -C to open a new iTerm2 window that works like tmux.

Use the tmux menu in iTerm2 to open new windows in tmux. Note that there are keyboard shortcuts for each of these, and they are not the same as the tmux window commands.

You can also attach to a tmux session running in iTerm2. In the screenshot, this is running on the same system, for example purposes. However, since OS X has SSH, this can be useful if you want to SSH to another system in the local network and connect to the running session. For example, the system shown below is my wife’s iMac over screensharing, but I wouldn’t need to use screenshare (or participate in its lag) to connect to this anymore. The same holds true for connecting to my work laptop if necessary.

In this final example screenshot, you can see that I have multiple panes split in one iTerm2 tab. These correspond to the split windows in the attached tmux in the other window. Also, the two tabs in the iTerm2 window are separate tmux windows (0:zsh and 1:zsh).

And now, I can SSH to that system and attach to the tmux session started by iTerm2.

Automating Installation with Chef

Installing OS X apps is quite easy, but I automate them with Chef anyway. While it is a simple “install update and restart”, with a couple commands to install the update, I do have three systems I want this on. I updated my iterm2 cookbook to support installing the tmux integration for iTerm2. This is disabled by default, so it needs to be enabled via a node attribute. For example, I have this in my workstation role applied to my OS X workstations.

1
2
3
4
5
6
7
8

name "workstation"
description "Mac OS X workstations"
run_list("recipe[tmux]")
default_attributes(
  "iterm2" => {
    "tmux_enabled" => true
  }
)

Check out the iterm2 cookbook’s README for more information.