Linux Utilities for Diagnostics

I spend a fair amount of time troubleshooting issues on Linux and other Unix and Unix-like systems. While there are dozens of utilities I use for diagnosing and resolving issues, I consistently employ a small set of tools to do quick, high-level checks of system health. These checks are in the categories of disk utilization, memory and CPU utilization, and networking and connectivity. Triaging the health of the system in each of these categories allows me to quickly hone in on where a problem may exist.

These utilities are usually available on all Linux systems. Most are available, or have analogues, on other Unix and Unix-like systems.

Disk Utilization

Generally, disk utilization is the first thing I check as a lack of free disk space spells certain doom for most user and kernel processes. I have seen more strange behavior from a lack of free disk space than anything else.

  • df reports filesystem disk space usage. This quickly allows me to see how much free space remains on each filesystem.

  • df -h displays, in human-readable format, the free space available on all mounted filesystems.

    $ df -h
    Filesystem      Size  Used Avail Use% Mounted on
    /dev/xvda        47G   26G   19G  58% /
    devtmpfs        4.0G   12K  4.0G   1% /dev
    none            802M  184K  802M   1% /run
    none            5.0M     0  5.0M   0% /run/lock
    none            4.0G     0  4.0G   0% /run/shm

  • du estimates file space usage. This allows me to pinpoint which fields are taking up large amounts of disk space so I can investigate further.
  • du -sh * summarizes, in human-readable format, the space utilized by all files/folders in the current directory.
    $ du -sh *
    18M     bundle
    8.6M    cached-copy
    444M    log
    4.0K    pids
    4.0K    system

Memory, CPU Utilization, and I/O

Running out of available memory is also a major cause of performance problems and strange behavior on systems. CPU utilization and I/O rates can quickly provide clues as to whether performance problems are due to bottlenecks internal to a given system, or from external sources.

  • free reports the amount of free and used memory on the system. This provides immediate feedback on whether a system lacks free memory.
  • free -m displays, in megabytes, the amount of used and free physical and swap memory, and the amount of memory used for buffers/caching.
    $ free -m
                    total       used       free     shared    buffers     cached
    Mem:          8014       6339       1674          0        136       3887
    -/+ buffers/cache:       2314       5699
    Swap:          511        153        358
  • vmstat reports on memory, swap, I/O, system activity, and CPU activity. This provides averages of various metrics since boot and can report continuously on current metrics. Analyzing the metrics can provide insight into what the system is doing at a given time (e.g. frequently swapping, waiting on I/O, etc.).
  • vmstat 1 will print out the metrics once every second until halted, using megabytes instead of bytes.
    $ vmstat 1
    procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
    r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
    1  0     80     59     43   1791    0    0     0     0 1131 1057 15  2 83  0
    0  0     80     57     43   1791    0    0     8    96 1031  936 19  2 79  0
    0  0     80     60     43   1791    0    0    40    64 1666 1444  9  2 89  0
    0  0     80     60     43   1791    0    0     8     0  667  553  0  0 100  0
    1  0     80     57     43   1791   16    0    16   104  808  748 12  2 86  0
    0  0     80     59     43   1791    0    0    12  3028 1813 1723 44  5 50  0
    0  0     80     59     43   1791    0    0     0    56 1119 1066 17  1 81  0
    1  0     80     50     43   1791    0    0    68     0 1219 1024 25  4 71  0
    0  0     80     60     43   1791    0    0    52    68 1725 1435 12  1 86  0
    0  0     80     60     43   1791    0    0     8     0 2236 1699 35  5 60  0
    0  0     80     60     43   1791    0    0     0    68  163  209  0  0 99  0
    1  0     80     60     43   1791    0    0     0   140 1456 1379 22  3 74  0
    1  0     80     61     43   1791    0    0     0    56 1481 1242 24  4 72  0
    0  0     80     60     43   1791    0    0   356     0 1359  930 11  3 86  0
    0  0     80     60     43   1792    0    0   428     0 1619  992  2  1 97  0
    0  0     80     60     43   1792    0    0     8  2196  313  396  0  0 100  0
    0  0     80     60     43   1792    0    0     0     0  144  181  0  0 100  0

Networking and Connectivity

Network connectivity and routing issues are usually apparent. However, trying to determine the exact nature of or reason for the issue can be a bit more difficult.

  • ping sends an ICMP echo request to a host. This provides immediate confirmation of whether or not a remote host is accessible.
  • ping will ping Google’s DNS servers, which usually indicates with a high degree of certainty whether or not Internet connectivity is available.
    $ ping
    PING ( 56(84) bytes of data.
    64 bytes from icmpreq=1 ttl=54 time=0.681 ms
    64 bytes from icmpreq=2 ttl=54 time=0.679 ms
    64 bytes from icmpreq=3 ttl=54 time=0.703 ms
    64 bytes from icmpreq=4 ttl=54 time=0.703 ms
    64 bytes from icmp_req=5 ttl=54 time=0.677 ms
  • mtr combines ping with traceroute and prints the route packet trace to a remote host, along with packet response times and loss percentages.

  • mtr -c 5 -r will send five packets to Google’s DNS servers and report back the intermediate routers, with details about response times and packet loss along the way.

    $ mtr -c 5 -r
    HOST: localhost                 Loss%   Snt   Last   Avg  Best  Wrst StDev
    1.|--     0.0%     5    0.9   0.7   0.6   0.9   0.2
    2.|--  0.0%     5    0.3   6.3   0.3  30.5  13.5
    3.|-- po102.dsr01.dllstx2.netwo  0.0%     5    1.1   0.6   0.5   1.1   0.3
    4.|-- po21.dsr01.dllstx3.networ  0.0%     5    1.3   2.5   0.6   8.0   3.1
    5.|--  0.0%     5    0.5   0.6   0.5   0.8   0.1
    6.|-- ae7.bbr01.eq01.dal03.netw  0.0%     5    0.5   0.6   0.5   0.7   0.1
    7.|--  0.0%     5    0.6   0.9   0.6   2.2   0.7
    8.|--              0.0%     5    0.5   0.6   0.5   0.8   0.1
    9.|--              0.0%     5    1.0   0.8   0.8   1.0   0.1
    10.|-- google-public-dns-a.googl  0.0%     5    0.8   0.8   0.7   0.8   0.0

  • netstat displays information about network connections, routing tables, and interfaces. While it is a very sophisticated tool which has many different possible applications, it provides an easy way to display a few important bits of data:
  • netstat -nlp displays information about processes that are currently listening on a socket.
    $ sudo netstat -nlp
    Active Internet connections (only servers)
    Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
    tcp        0      0*               LISTEN      2858/mysqld
    tcp        0      0    *               LISTEN      2665/sshd
    tcp        0      0    *               LISTEN      3133/master
    tcp6       0      0 :::8080                 :::*                    LISTEN      3160/apache2
    tcp6       0      0 :::22                   :::*                    LISTEN      2665/sshd
    tcp6       0      0 :::25                   :::*                    LISTEN      3133/master
    tcp6       0      0 :::443                  :::*                    LISTEN      3160/apache2
    udp        0      0    *                           2633/dhclient3
  • netstat -rn displays the current routing table.
    $ netstat -rn
    Kernel IP routing table
    Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface         UG        0 0          0 eth0   U         0 0          0 eth0


The examples above show some of the most common ways these utilities can be used to perform diagnostics on systems based on disk utilization, memory and CPU utilization, and network activity and connectivity. Some of these utilities (particularly netstat) are quite powerful, and could be used to display or diagnose much more than shown in the examples above. Past troubleshooting experience, and the specific histories of given systems, guide the particular ways that I deploy these tools to assist in the investigation and resolution of system issues.

The post Linux Utilities for Diagnostics appeared first on Atomic Spin.

sudo, pipelines, and complex commands with quotes

We've all run into problems like this:

$ echo 12000 > /proc/sys/vm/dirty_writeback_centisecs
-bash: /proc/sys/vm/dirty_writeback_centisecs: Permission denied

The command fails because the target file is only writeable by root. The fix seems obvious and easy:

$ sudo echo 12000 > /proc/sys/vm/dirty_writeback_centisecs -bash: /proc/sys/vm/dirty_writeback_centisecs: Permission denied

Huh? It still fails. What gives? The reason it fails is that it is the shell that sets up the re-direction before running the command under sudo. The solution is to run the whole pipeline under sudo. There are several ways to do this:

echo 'echo 12000 > /proc/sys/vm/dirty_writeback_centisecs' | sudo sh
sudo sh -c 'echo 12000 > /proc/sys/vm/dirty_writeback_centisecs'

This is fine for simple commands, but what if you have a complex command that already includes quotes and shell meta-characters?

Here's what I use for that:

sudo su <<EOF
echo 12000 > /proc/sys/vm/dirty_writeback_centisecs

Note that the backslash before EOF is important to ensure meta-characters are not expanded.

Finally, here's an example of a command for which I needed to use this technique:

sudo sh  << EOF
perl -n -e '
use strict;
use warnings;
if (/^([^=]*=)([^$]*)(.*)/) {
  my $pre = $1;
  my $path = $2;
  my $post = $3;
  (my $newpath = $path) =~ s/usr/usr/local/;
  $newpath =~ s/://g;
  print "$pre$newpath:$path$postn"
else {
' < /opt/rh/ruby193/enable > /opt/rh/ruby193/

Protecting the Root Filesystem on Ubuntu with Overlayroot

In certain situations, it is desirable to have a read-only root filesystem. This prevents any changes from occurring on the root filesystem that may alter system behavior, and it allows a simple reboot to restore a system to its pristine state. Examples of such applications include kiosks and embedded devices. Using overlayroot on Ubuntu makes creating a read-only root filesystem quick and easy.


Let us imagine that we have an embedded Linux device that needs to run a particular program. The program does not need to store any data (except perhaps some logs), and we want to protect the system against any changes. If the program or any system processes make filesystem modifications, we want to toss them out and return to the original state. We want a robust system that will run our program, but be restored to it’s ‘factory’ state with a power-cycle.


Overlayroot is a package for Ubuntu which utilizes OverlayFS, a union filesystem implementation. OverlayFS presents a unified view of two different filesystems; the presented filesystem is the result of overlaying one filesystem over another.

In OverlayFS, there is an ‘upper’ filesystem and a ‘lower’ filesystem. If a particular object exists in both the upper filesystem and the lower filesystem, the object from the upper filesystem is presented, and the object from the lower filesystem is hidden. If the object is a directory, the contents of the directory on the upper and lower filesystems are merged and presented.

With overlayroot, the lower filesystem is a read-only mount of the root filesystem, and the upper filesystem is a read-write mount of another block device. That block device can be tmpfs, a standard block device, or an encrypted block device (à la dmcrypt).

All changes made on top of the root filesystem are stored elsewhere and do not affect the root filesystem. Depending on the block device chosen, those changes can persist across reboots. While the root filesystem will not be affected, the view presented by OverlayFS will include any changes stored in the upper filesystem.


For our embedded Linux device, we are not interested in preserving any changes to the root filesystem. Any data that we want to preserve will be stored on a separate filesystem dedicated to data storage.

Here is our filesystem structure:

  • /dev/sda1 on /
  • /dev/sda2 on /data

We want to mount / using overlayroot so that we have a read-write filesystem available but not persist any changes to /, and we want to mount /data normally so that our program can write out and persist log files.


Overlayroot has been available since Ubuntu 12.10, and has been back-ported to 12.04 LTS. It is quick to install:

apt-get install overlayroot

The configuration file is stored at /etc/overlayroot.conf, and contains a wealth of in-line documention.

The only item to change is the overlayroot variable. By default, it is blank:


Start Simple

Since we want to use tmpfs mode for overlayroot, we set the variable accordingly.


The other modes (specifying a device, or crypt) are documented in overlayroot.conf, with some nice diagrams on the post introducing overlayroot.

Add Swap

By default, overlayroot disables swap. This can be re-enabled by providing another option to the configuration:


(The default value for swap is 0).

Don’t Recurse

By default, overlayroot will mount all filesystems under / in the specified mode. This can be prevented by adding another option to the configuration:


Note that swap=1 and recurse=0 are separated by a comma, not a colon.

(The default value for recurse is 1).

This will prevent overlayroot from overlaying a read-write tmpfs on top of a read-only /data filesystem. All of / will be protected from any modifications, except /data, which is what we want.

The Result

After making changes to overlayroot.conf and rebooting, the root filesystem should now be mounted as a read-write tmpfs overlaying the read-only / filesystem.

Running mount, we should see something like:

1 2 3 4 5 6 7 8 9 10 11 12 13 
overlayroot on / type overlayfs (rw,lowerdir=/media/root-ro/,upperdir=/media/root-rw)
proc on /proc type proc (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,noexec,nosuid,nodev)
none on /sys type sysfs (rw,noexec,nosuid,nodev)
none on /sys/fs/fuse/connections type fusectl (rw)
devtmpfs on /dev type devtmpfs (rw,mode=0755)
none on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
none on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)
none on /run/shm type tmpfs (rw,nosuid,nodev)
/dev/sda1 on /media/root-ro type ext4 (ro)
tmpfs-root on /media/root-rw type tmpfs (rw,relatime)
/dev/sda2 on /data type ext4 (rw)

Note how the root filesystem is of type overlayfs with specified upper and lower filesystems that correspond to the read-write tmpfs and read-only mounts listed.

Disabling Overlayroot

Once overlayroot is enabled, it is no longer possible to make changes to the root filesystem. This introduces problems if we legitimately need to make changes (such as for updates, tweaks, etc.). Fortunately, we can override any overlayroot configuration by passing overlayroot=disabled to the kernel at boot. This can be done as a one-off operation as needed, or can be set up as an alternate boot configuration in a boot loader such as GRUB.

For example, with GRUB2, something like the following could be used:

1 2 3 4 5 6 7 8 9 10 11 
menuentry 'Ubuntu, with Linux 3.5.0-54-generic (Writable)' --class ubuntu --class gnu-linux --class gnu --class os {
	gfxmode $linux_gfx_mode
	insmod gzio
	insmod part_msdos
	insmod ext2
	set root='(hd0,msdos1)'
	search --no-floppy --fs-uuid --set=root 28adfe9d-c122-479a-ab81-de57d16516dc
	linux	/vmlinuz-3.5.0-54-generic root=/dev/mapper/faramir-root ro overlayroot=disabled
	initrd	/initrd.img-3.5.0-54-generic


Overlayroot makes the process of mounting the root filesystem as read-only on Ubuntu very easy. Prior to the availability of overlayroot packages, custom scripts were added to the initramfs configuration. These scripts were often fragile, and not always compatible across versions of Ubuntu. With the availability of OverlayFS and overlayroot, creating robust kiosks and embedded Linux devices with rollback capabilities is now extremely easy.

The post Protecting the Root Filesystem on Ubuntu with Overlayroot appeared first on Atomic Spin.

Command-line cookbook dependency solving with knife exec

Note: This article was originally published in 2011. In response to demand, I've updated it for 2014! Enjoy! SNS

Imagine you have a fairly complicated infrastructure with a large number of nodes and roles. Suppose you have a requirement to take one of the nodes and rebuild it in an entirely new network, perhaps even for a completely different organization. This should be easy, right? We have our infrastructure in the form of code. However, our current infrastructure has hundreds of uploaded cookbooks - how do we know the minimum ones to download and move over? We need to find out from a node exactly what cookbooks are needed for that node to be built.

The obvious place to start is with the node itself:

$ knife node show controller
Node Name:   controller
Environment: _default
FQDN:        controller
Run List:    role[base], recipe[apt::cacher], role[pxe_server]
Roles:       pxe_server, base
Recipes      apt::cacher, pxe_dust::server, dhcp, dhcp::config
Platform:    ubuntu 10.04

OK, this tells us we need the apt, pxe_dust and dhcp cookbooks. But what about them - do they have any dependencies? How could we find out? Well, dependencies are specified in two places - in the cookbook metadata, and in the individual recipes. Here's a primitive way to illustrate this:

bash-3.2$ for c in apt pxe_dust dhcp
> do
> grep -iER 'include_recipe|^depends' $c/* | cut -d '"' -f 2 | sort | uniq
> done

As I said - primitive. However the problem doesn't end here. In order to be sure, we now need to repeat this for each dependency, recursively. And of course it would be nice to present them more attractively. Thinking about it, it would be rather useful to know what cookbook versions are in use too. This is definitely not a job for a shell one liner - is there a better way?

As it happens, there is. Think about it - the Chef server already needs to solve these dependencies to know what cookbooks to push to API clients. Can we access this logic? Of course we can - clients carry out all their interactions with the Chef server via the API. This means we can let the server solve the dependencies and query it via the API ourselves.

Chef provides two powerful ways to access the API without having to write a RESTful client. The first, Shef, is an interactive REPL based on IRB, which when launched gives access to the Chef server. This isn't trivial to use. The second, much simpler way is the knife exec subcommand. This allows you to write Ruby scripts or simple one-liners that are executed in the context of a fully configured Chef API Client using the knife configuration file.

Now, since I wrote this article, back in summer 2011, the API has changed, which means that my original method no longer works. Additionally, we are now served by at least two local dependency solvers, in the form of Berkshelf (whose dependency solver, 'solve' is now available as an individual Gem), and Librarian-chef. In this updated version, I'll show how to use the new Chef server API to perform the same function. Berkshelf and Librarian solve a slightly different problem, in that in this instance we're trying to solve dependencies for a node, so for the purposes of this article I'll consider them out of scope.

For historical purposes, here's the original solution:

knife exec -E '(api.get "nodes/controller/cookbooks").each { |cb| pp cb[0] => cb[1].version }'

The /nodes/NODE_NAME/cookbooks endpoint returns the cookbook attributes, definitions, libraries and recipes that are required for this node. The response is a hash of cookbook name and Chef::CookbookVersion object. We simply iterate over each one, and pretty print the cookbook name and the version.

Let's give it a try:

$ knife exec -E '(api.get "nodes/controller/cookbooks").each { |cb| pp cb[0] => cb[1].version }'

The current way to solve dependencies using the Chef server API resides under the environments end point. This makes sense, if you think of environments as a way to define and constrain version numbers for a given set of nodes. This means that constructing the API call, and handling the results is slightly more than can easily be comprehended in a one-liner, which gives us the opportunity to demonstrate the use of knife exec with a script on the filesystem.

First let's create the script:

USAGE = "knife exec script.rb NODE_NAME"

def usage_and_exit
  exit 1

node_name = ARGV[2]

usage_and_exit unless node_name

node = api.get("nodes/#{node_name}")
run_list_expansion = node.expand!("server")

cookbook_solution ="environments/#{node.chef_environment}/cookbook_versions",
                            :run_list =>

cookbook_solution.each do |name, cb|
  puts name + " => " + cb.version


The way knife exec scripts work is to pass the arguments following knife to Ruby as the ARGV special variable, which is an array of each space-separated argument. This allows us to produce a slightly more general solution, to which we can pass the name of the node for which we want to solve. The usage handling is obvious - we print the usage to stderr if the command is called without a node name. The meat of the script is the API call. First we get the node object (from ARGV[2], i.e. the node we passed to the script) from the Chef server. Next we expand the run list - this means check for and expand any run lists in roles. Finally we call the API to provide us with cookbook versions for the specified node in the environment in which the node currently resides, passing in the recipes from the expanded run list. Finally we iterate over the cookbooks we get back, and print the name and version. Note that this script could easily be modified to solve for a different environment, which would be handy if we wanted to confirm what versions we'd get were we to move the node to a different environment. Let's give in a whirl:

$ knife exec src/knife-cookbook-solve/solve.rb asl-dev-1
chef_handler => 1.1.4
minitest-handler => 0.1.3
base => 0.0.2
hosts => 0.0.1
yum => 2.3.0
tmux => 1.1.1
ssh => 0.0.6
fail2ban => 1.2.2
users => 2.0.6
security => 0.1.0
sudo => 2.0.4
atalanta-users => 0.0.2
community_users => 1.5.1
sudoersd => 0.0.2
build-essential => 1.4.2

To conclude as did the original article....Nifty! :)

Acquiring a Modern Ruby (Part One)

Last month marked the 21st anniversary of the programming language Ruby. I use Ruby pretty much all the time. If I need to write a command line tool, a web app, pretty much anything, I'll probably start with Ruby. This means I've always got Ruby to hand on whatever machine I'm using. However, in practice, getting a recent Ruby on any platform isn't actually as simple as it sounds. Ruby is a fast-moving language, with frequent releases. Mainstream distributions often lag behind the current release. Let's have a quick look at the history of the releases of Ruby over the last year:

require 'nokogiri'
require 'open-uri'
news = Nokogiri::HTML(open(""))
news.xpath("//div[@class='post']/following-sibling::*").each do |item|
  match = item.text.match /Ruby (\S+) is released.*Posted by.*on ((?:\d{1,2} [a-zA-z]{3} \d{4}))/m
  if match
    puts "Ruby #{match[1]} was announced on #{match[2]}"

Ruby 2.1.0-rc1 was announced on 20 Dec 2013
Ruby 2.1.0-preview2 was announced on 22 Nov 2013
Ruby 1.9.3-p484 was announced on 22 Nov 2013
Ruby 2.0.0-p353 was announced on 22 Nov 2013
Ruby 2.1.0-preview1 was announced on 23 Sep 2013
Ruby 2.0.0-p247 was announced on 27 Jun 2013
Ruby 1.9.3-p448 was announced on 27 Jun 2013
Ruby 1.8.7-p374 was announced on 27 Jun 2013
Ruby 1.9.3-p429 was announced on 14 May 2013
Ruby 2.0.0-p195 was announced on 14 May 2013
Ruby 2.0.0-p0 was announced on 24 Feb 2013
Ruby 1.9.3-p392 was announced on 22 Feb 2013
Ruby 2.0.0-rc2 was announced on 8 Feb 2013
Ruby 1.9.3-p385 was announced on 6 Feb 2013
Ruby 1.9.3-p374 was announced on 17 Jan 2013

So 15 releases in 2013, including a major version (2.0.0) in February, and a release candidate of 2.1 shortly before Christmas. For a Ruby developer today, the current releases are:

  • Ruby 2.1.1
  • 2.0.0-p451
  • 1.9.3-p545

Let's compare to what's available in Debian stable:

$ apt-cache show ruby
Package: ruby
Source: ruby-defaults
Version: 1:1.9.3
Installed-Size: 31
Maintainer: akira yamada <>
Architecture: all
Replaces: irb, rdoc
Provides: irb, rdoc
Depends: ruby1.9.1 (>=

So the current version in Debian is older than January 2013? What about Ubuntu? The latest 'saucy' offers us 2.0.0. That's fractionally better, but really, it's pretty old. What about my trusty Fedora? At the time of writing, that gives me 2.0.0p353. Still not exactly current, and not a sniff of a 2.1 package. Archlinux offers a Ruby 2.1, but even that's not right up to date:

$ ruby --version
ruby 2.1.0p0 (2013-12-25 revision 44422) [x86_64-linux]

Now, I will grant that there might be third-party repositories, or backports or other providers of packages which might be more up-to-date, but my experience of this hasn't always been that positive. On the whole, it looks like we need to look somewhere else. On a Unix-derived system, this typically means building from source.

Building From Source

Building Ruby from source isn't difficult, not if you have some idea what you're doing, and can read documentation. However, equally, it's not entirely trivial. It does require you to know how to provide libraries for things like ffi, ncurses, readline, openssl, yaml etc. And, if you're using different systems, they have different names, and you might be working with different compilers and versions of make. In recognition of this, several tools have emerged to shield the complexity, and make it easier to get a modern Ruby on your system. The most popular are RVM, ruby-build, and ruby-install. Let's review each of them, briefly. I'm going to use a CentOS 6 machine as my example Linux machine, but each of these tools will work on all popular Linux distributions and Mac OSX. I've had success on FreeBSD and Solaris too, but I've not tested this recently, so YMMV.


RVM, the Ruby Version Manager, is the father of the tools designed to make managing modern Ruby versions easier. It's what one might call a monolithic tool - it does a huge range of things all in one place. We'll discuss it several times in this series, as it provides functionality beyond that of simply installing a modern Ruby, but for the purposes of this series, we're only going to use it to install Ruby.

RVM is a shell script. The most popular way to install it is via the fashionable 'curl pipe through bash' approach:

$ curl -sSL | bash
Creating group 'rvm'

Installing RVM to /usr/local/rvm/
Installation of RVM in /usr/local/rvm/ is almost complete:

  * First you need to add all users that will be using rvm to 'rvm' group,
    and logout - login again, anyone using rvm will be operating with `umask u=rwx,g=rwx,o=rx`.

  * To start using RVM you need to run `source /etc/profile.d/`
    in all your open shell windows, in rare cases you need to reopen all shell windows.

# Administrator,
#   Thank you for using RVM!
#   We sincerely hope that RVM helps to make your life easier and more enjoyable!!!
# ~Wayne, Michal & team.

In case of problems: and

Hmm, ok, let's try that.

# gpasswd -a sns rvm
Adding user sns to group rvm

I also added:

source /etc/profile.d/

to the end of ~/.bash_profile

$ rvm --version

rvm 1.25.19 (master) by Wayne E. Seguin <>, Michal Papis <> []

First let's see what Rubies it knows about:

$ rvm list known
# MRI Rubies

# GoRuby

# Topaz

# TheCodeShop - MRI experimental patches

# jamesgolick - All around gangster

# Minimalistic ruby implementation - ISO 30170:2012

# JRuby

# Rubinius

# Ruby Enterprise Edition

# Kiji

# MagLev

# Mac OS X Snow Leopard Or Newer

# Opal

# IronRuby

Wow - that's a lot of Rubies. Let's just constrain ourselves to MRI, and install the latest stable 2.1:

$ rvm install ruby
Searching for binary rubies, this might take some time.
No binary rubies available for: centos/6/x86_64/ruby-2.1.1.
Continuing with compilation. Please read 'rvm help mount' to get more information on binary rubies.
Checking requirements for centos.
Installing requirements for centos.
Updating system.
Installing required packages: patch, libyaml-devel, libffi-devel, glibc-headers, gcc-c++, glibc-devel, patch, readline-devel, zlib-devel, openssl-devel, autoconf, automake, libtool, bisonsns password required for 'yum install -y patch libyaml-devel libffi-devel glibc-headers gcc-c++ glibc-devel patch readline-devel zlib-devel openssl-devel autoconf automake libtool bison':

The first thing to notice, and this is pretty cool, is that RVM will try to locate a precompiled binary. Unfortunately there isn't one for our platform, so it's going to go ahead and install one from source. It's going to install the various required packages, and then crack on.

This assumes that I've got sudo set up for my user. As it happens, I don't, but we can fix that. For a mac or an ubuntu machine this would be in place anyway. Please hold, caller... ...ok done. My sns user now has sudo privileges, and we can continue:

$ rvm install ruby
Searching for binary rubies, this might take some time.
No binary rubies available for: centos/6/x86_64/ruby-2.1.1.
Continuing with compilation. Please read 'rvm help mount' to get more information on binary rubies.
Checking requirements for centos.
Installing requirements for centos.
Updating system.
Installing required packages: patch, libyaml-devel, libffi-devel, glibc-headers, gcc-c++, glibc-devel, patch, readline-devel, zlib-devel, openssl-devel, autoconf, automake, libtool, bisonsns password required for 'yum install -y patch libyaml-devel libffi-devel glibc-headers gcc-c++ glibc-devel patch readline-devel zlib-devel openssl-devel autoconf automake libtool bison':
Requirements installation successful.
Installing Ruby from source to: /usr/local/rvm/rubies/ruby-2.1.1, this may take a while depending on your cpu(s)...
ruby-2.1.1 - #downloading ruby-2.1.1, this may take a while depending on your connection...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 11.4M  100 11.4M    0     0  61.0M      0 --:--:-- --:--:-- --:--:-- 64.9M
ruby-2.1.1 - #extracting ruby-2.1.1 to /usr/local/rvm/src/ruby-2.1.1.
ruby-2.1.1 - #configuring....................................................
ruby-2.1.1 - #post-configuration.
ruby-2.1.1 - #compiling...................................................................................
ruby-2.1.1 - #installing................................
ruby-2.1.1 - #making binaries executable.
Rubygems 2.2.2 already installed, skipping installation, use --force to reinstall.
ruby-2.1.1 - #gemset created /usr/local/rvm/gems/ruby-2.1.1@global
ruby-2.1.1 - #importing gemset /usr/local/rvm/gemsets/global.gems.....
ruby-2.1.1 - #generating global wrappers.
ruby-2.1.1 - #gemset created /usr/local/rvm/gems/ruby-2.1.1
ruby-2.1.1 - #importing gemsetfile /usr/local/rvm/gemsets/default.gems evaluated to empty gem list
ruby-2.1.1 - #generating default wrappers.
ruby-2.1.1 - #adjusting #shebangs for (gem irb erb ri rdoc testrb rake).
Install of ruby-2.1.1 - #complete
Ruby was built without documentation, to build it run: rvm docs generate-ri

Great - let's see what we have:

$ ruby --version
ruby 2.1.1p76 (2014-02-24 revision 45161) [x86_64-linux]

In order to verify our installation, we're going to test OpenSSL and Nokogiri. If these two work, we can be pretty confident that we have a functional Ruby:

$ gem install nokogiri --no-ri --no-rdoc
Fetching: mini_portile-0.5.2.gem (100%)
Successfully installed mini_portile-0.5.2
Fetching: nokogiri-1.6.1.gem (100%)
Building native extensions.  This could take a while...
Successfully installed nokogiri-1.6.1
2 gems installed

Let's test this now. Here's a simple test to prove that both SSL and Nokogiri are working:

require 'nokogiri'
require 'open-uri'
require 'openssl'
https_url = ''
puts Nokogiri::HTML(open(https_url)).css('input')

If this works, we should see a PEM file printed to screen, and the HTML of various inputs on the Google homepage:

$ ruby ruby_test.rb
<input name="ie" value="ISO-8859-1" type="hidden">
<input value="en-GB" name="hl" type="hidden">
<input name="source" type="hidden" value="hp">
<input autocomplete="off" class="lst" value="" title="Google Search" maxlength="2048" name="q" size="57" style="color:#000;margin:0;padding:5px 8px 0 6px;vertical-align:top">
<input class="lsb" value="Google Search" name="btnG" type="submit">
<input class="lsb" value="I'm Feeling Lucky" name="btnI" type="submit" onclick="if(this.form.q.value)this.checked=1; else top.location='/doodles/'">
<input type="hidden" id="gbv" name="gbv" value="1">

So, RVM was fairly painless. We had to fiddle about with users and sudo, and add a line to our shell profile, but once that was done, we were easily able to install Ruby.

Ruby Build

Ruby-build is a dedicated tool, also written in shell, designed specifically to install Ruby, and provide fine-grained control over the configuration, build and installation. In order to get it we need to install Git. As with RVM, I'll set up a user with sudo access in order to permit this. Once we have Git installed, we can clone the project and install it.

$ git clone
Initialized empty Git repository in /home/sns/ruby-build/.git/
remote: Reusing existing pack: 3077, done.
remote: Total 3077 (delta 0), reused 0 (delta 0)
Receiving objects: 100% (3077/3077), 510.82 KiB | 386 KiB/s, done.
Resolving deltas: 100% (1404/1404), done.
$ cd ruby-build
$ sudo ./

Like RVM, we can find out what Rubies ruby-build knows about:

$ ruby-build --definitions

Let's opt for ruby 2.1 again. Now, unlike RVM, ruby-build doesn't make any attempt to install development tools suitable to allow Ruby to be built from source. It does provide a wiki with information on what is needed, but if you expect ruby-build to magically install your toolkit, you're going to be disappointed. Additionally, ruby-build requires you to state exactly where you want Ruby to be installed. It doesn't have a default, so you need to work out where it should go. Realistically this boils down to two choices - do you want to try to install it somewhere global, where everyone can see it, or just locally for your own use. The former adds some more complications around permissions and so forth, so in this instance we'll install to a local directory.

First, let's check the wiki for the build dependencies:

$ yum install gcc-c++ glibc-headers openssl-devel readline libyaml-devel readline-devel zlib zlib-devel

Now that's all installed, we can go ahead with building Ruby:

$ ruby-build 2.1.1 ~/local/mri-2.1.1
Downloading ruby-2.1.1.tar.gz...
Installing ruby-2.1.1...
Installed ruby-2.1.1 to /home/sns/local/mri-2.1.1

Not a very verbose output... in fact one might be forgiven for wondering at times if anything's happening at all! But, now we have a Ruby, let's add it to our shell path, and do our nokogiri/openssl test.

$ export PATH=$PATH:/home/sns/local/mri-2.1.1/bin/
$ ruby --version
ruby 2.1.1p76 (2014-02-24 revision 45161) [x86_64-linux]

$ gem install nokogiri --no-rdoc --no-ri
$ vi ruby_test.rb
$ ruby ruby_test.rb
<input name="ie" value="ISO-8859-1" type="hidden">
<input value="en-GB" name="hl" type="hidden">
<input name="source" type="hidden" value="hp">
<input autocomplete="off" class="lst" value="" title="Google Search" maxlength="2048" name="q" size="57" style="color:#000;margin:0;padding:5px 8px 0 6px;vertical-align:top">
<input class="lsb" value="Google Search" name="btnG" type="submit">
<input class="lsb" value="I'm Feeling Lucky" name="btnI" type="submit" onclick="if(this.form.q.value)this.checked=1; else top.location='/doodles/'">
<input type="hidden" id="gbv" name="gbv" value="1">

So, ruby-build did what it said on the tin. It didn't need us to create a group, or source any files in our profile. All we needed to do was update our shell path, which, of course, we'd add to our shell profile to make permanent. We did have to install the build dependencies manually, but this is documented on the wiki. It's a lighter-weight solution than RVM, optimised for local users.

Ruby Install

The third of our 'install from source' options is 'Ruby Install'. Another shell utility, it is more aligned with 'Ruby Build' than RVM, as it only really has one purpose in life - to install Ruby. It doesn't have all the extra features of RVM, but it does install build dependencies for most platforms.

To install, we obtain a tarball of the latest release:

$ wget -O ruby-install-0.4.0.tar.gz
$tar xzvf ruby-install-0.4.0.tar.gz
$ cd ruby-install-0.4.0
$ sudo make install
[sudo] password for sns:
for dir in `find etc lib bin sbin share -type d 2>/dev/null`; do mkdir -p /usr/local/$dir; done
for file in `find etc lib bin sbin share -type f 2>/dev/null`; do cp $file /usr/local/$file; done
mkdir -p /usr/local/share/doc/ruby-install-0.4.0
cp -r *.md *.txt /usr/local/share/doc/ruby-install-0.4.0/

This will make the ruby-install tool available. Again, we can see which Rubies are available:

$ ruby-install
Known ruby versions:
    1:      1.9.3-p484
    1.9:    1.9.3-p484
    1.9.1:  1.9.1-p431
    1.9.2:  1.9.2-p320
    1.9.3:  1.9.3-p484
    2.0:    2.0.0-p353
    2.0.0:  2.0.0-p353
    2:      2.1.0
    2.1:    2.1.0
    stable: 2.1.0
    1.7:    1.7.10
    stable: 1.7.10
    2.1:    2.1.1
    2.2:    2.2.5
    stable: 2.2.5
    1.0:    1.0.0
    1.1:    1.1RC1
    stable: 1.0.0
    1.0:    1.0.0
    stable: 1.0.0

Because we obtained the stable release of the ruby-install tool, we don't have the very latest version available. We could simply do as we did with ruby-build, and get the latest version straight from Git, but the process would be the same. For now, we're going to opt for the latest stable release the tool offers us, which is 2.1.0:

$ ruby-install ruby
>>> Installing ruby 2.1.0 into /home/sns/.rubies/ruby-2.1.0 ...
>>> Installing dependencies for ruby 2.1.0 ...
[sudo] password for sns:
... grisly details ...

The install is substantially more verbose than either ruby-build or RVM, showing the grisly details of the compiling and linking. Once installed, let's check the version and run our test:

$ export PATH=$PATH:/home/sns/.rubies/ruby-2.1.0/bin/
$ ruby --version
ruby 2.1.0p0 (2013-12-25 revision 44422) [x86_64-linux]

$ gem install nokogiri --no-ri --no-rdoc
Building native extensions.  This could take a while...
Successfully installed nokogiri-1.6.1
1 gem installed
$ vi ruby_test.rb
$ ruby ruby_test.rb
<input name="ie" value="ISO-8859-1" type="hidden">
<input value="en-GB" name="hl" type="hidden">
<input name="source" type="hidden" value="hp">
<input autocomplete="off" class="lst" value="" title="Google Search" maxlength="2048" name="q" size="57" style="color:#000;margin:0;padding:5px 8px 0 6px;vertical-align:top">
<input class="lsb" value="Google Search" name="btnG" type="submit">
<input class="lsb" value="I'm Feeling Lucky" name="btnI" type="submit" onclick="if(this.form.q.value)this.checked=1; else top.location='/doodles/'">
<input type="hidden" id="gbv" name="gbv" value="1">

So, ruby-install was pretty easy to use. We didn't need to mess about with groups or sourcing extra files in our shell profile. There was a default, local path for the Rubies, the dependencies were identified and installed for us, and everything worked. The only downside was that, on account of using the stable version of the tool, we didn't get access to the very latest Ruby. That's easily fixed by simply getting the tool straight from Git - which we'll look at when we come to discuss managing multiple versions of Ruby.


So at this stage in our journey, we've explored three approaches to simplifying the process of installing a Ruby from source - RVM, ruby-build and ruby-install. Given that the stated objective was to install a recent version of Ruby, from source, as simply as possible, in my view the order of preference goes:

  1. Ruby Install: This does the simplest thing that could possibly work. It has sane defaults, solves and installs dependencies, and requires nothing more than a path setting.

  2. RVM: This requires only slightly more setup than Ruby Install, and certainly meets the requirements. It's much more heavyweight, as it does many more things than just install Ruby, and these capabilities will come into consideration later in the series.

  3. Ruby Build: This brings up the rear. It works, but it doesn't have sane defaults, and needs extra manual steps to install build dependencies.

As we continue the series, we'll look into ways to manage multiple versions of Ruby on a workstation, strategies for getting the relevant versions of Ruby onto servers, running Ruby under Microsoft Windows, and automating the whole process. Next time we'll talk about Ruby on Windows. Until then, bye for now.

Heroku log drains into Logstash

The first and obvious option for shipping logs from a heroku app to Logstash is the heroku input plugin. However, this requires installing the Heroku gem and deploying the login + password of a Heroku user to your Logstash server(s). At this time it seems that any user given permissions to an app on Heroku has full control. Not good when you just want to fetch logs. Heroku has added more granular permissions via OAuth but the Heroku gem does not support OAuth tokens yet.

Fortunately there’s another option using Heroku’s log drain. Setting up a log drain from Heroku to Logstash is almost as simple as the Heroku input plugin but has the major advantage of not requiring any new users or passwords to be deployed on the Logstash serer.

Hooking up your Heroku-deployed apps to your Logstash/Kibana/Elasticsearch infrastructure is straightforward using the syslog drains provided by Heroku. Here is a recipe for putting the pieces together:

Install Heroku toolbelt

In order to configure a log drain on Heroku you need to install the Heroku toolbelt (or using the API directly). At this time I don't think there's a way to configure log drains from the web UI.

The heroku toolbelt can also be used to tail an apps' logs on the command line which is great for debugging since the logs output by this command are identical to the logs that should be sent to the Logstash if everything is configured correctly:

List apps:

$ heroku apps
myapp1  email@dom.tld
myapp2  email@dom.tld

Tail app's logs:

$ heroku logs --app myapp1 -t
2014-01-31T14:26:11.801629+00:00 app[web.1]: 2014-01-31T14:26:11.801Z - debug: FETCHING TICKET: 15846
2014-01-31T14:26:26.753977+00:00 app[web.1]: 2014-01-31T14:26:26.752Z - debug: FETCHING TICKET: 15851
2014-01-31T14:26:27.457415+00:00 heroku[router]: at=info method=GET path=/ping? host=myapp1 request_id=6cb9b9eb-388c-4364-9278-a81179067f21 fwd="" dyno=web.2 connect=7ms service=6ms status=200 bytes=2

Configure log drain

Use the Heroku toolbelt to configure a log drain:

$ heroku drains:add --app myapp1 syslog://logstash.dom.tld:1514

Heroku's Logplex system will now send all logs generated from myapp1 to logstash.dom.tld:1514 via TCP in syslog RFC-5424 format.

Configure Logstash

Next, configure a Logstash input to receive these logs:

input {
    tcp {
        port => "1514"
        tags => ["input_heroku_syslog"]

Heroku uses the syslog format as defined in RFC 5424. Logstash ships with a grok rules that parse out most syslog formats including RFC 5424 but I found that they were not quite perfect for Heroku logs.

For more details on Heroku log drains:

Here are examples of raw log lines sent by Heroku. These are exactly what Logstash will receive and suitable for testing with the grok debugger.

The first is an example of a log message sent from a heroku component, in this case the Heroku router:

231 <158>1 2014-01-08T01:05:27.967180+00:00 d.9bc44987-ff40-40ac-a248-ff4ec4d71d7c heroku router - - at=info method=POST path=
/ fwd="" dyno=web.1 connect=2ms service=81ms status=200 bytes=2

Next, is an example of a log line generated from the application's stdout:

158 <13>1 2014-01-08T17:49:14.822585+00:00 d.9bc44987-ff40-40ac-a248-ff4ec4d71d7c app web.1 - - 2014-01-08T17:49:14.822Z -
minfo: FETCHING TICKET: 15103

Here is the grok pattern we use to parse Heroku's syslog RFC 5424-(ish) log messages:

filter {
  if "input_heroku_syslog" in [tags] {
    grok {
      match => ["message", "%{SYSLOG5424PRI}%{NONNEGINT:syslog5424_ver} +(?:%{TIMESTAMP_ISO8601:timestamp}|-) +(?:%{HOSTNAME:heroku
_drain_id}|-) +(?:%{WORD:heroku_source}|-) +(?:%{DATA:heroku_dyno}|-) +(?:%{WORD:syslog5424_msgid}|-) +(?:%{SYSLOG5424SD:syslog5424
_sd}|-|) +%{GREEDYDATA:heroku_message}"]
    mutate { rename => ["heroku_message", "message"] }
    kv { source => "message" }
    syslog_pri { syslog_pri_field_name => "syslog5424_pri" }

A few notes about this filter config:

  • The log message will be stored in the message field.
  • Key/value pairs matching key=value in the message will be parsed and added to the Logstash event. Many of the internal Heroku components's logs include useful key/vals.
  • Special fields heroku_drain_id, heroku_source, and heroku_dyno will be extracted from each event.

Dell OpenManage on Fedora 19

Here's how to get Dell OpenManage running on Fedora 19. It is a simple process but I could not find any up to date information on how to do it elsewhere.

This Dell OpenManage wiki page is the primary source of information for running OMSA on Linux. Go there and run the script installer, or install the yum repo files and RPM signing keys manually.

yum repo

The installer will probably install the /etc/yum.repos.d/dell-omsa-repository.repo with URL's such as:


Change the osname=f$releasevar param and force it to el6:


Next, run yum search srvadmin and you should see the list of OpenManage packages:

srvadmin-all.x86_64 : Meta package for installing all Server Administrator features, 7.3.0
srvadmin-argtable2.x86_64 : A library for parsing GNU style command line arguments, 7.3.0
srvadmin-base.x86_64 : Meta package for installing the Server Agent, 7.3.0
srvadmin-cm.i386 : OpenManage Inventory Collector, 7.3.0

Install the ones you need. For the basic CLI tools you can install srvadmin-omcommon and srvadmin-omacore.

canary file to force dataeng.service to start

It's not obvious from the error output, unfortunately, but the cli tools omreport and omconfig will not work until the dataeng.service is started, but it will refuse to start when it detects Fedora – and "unsupported" platform.

Trick the startup scripts into attempting to start anyway by creating this file:

touch /opt/dell/srvadmin/lib64/openmanage/IGNORE_GENERATION

The presence of this file skips the checks performed in /opt/dell/srvadmin/sbin/CheckSystemType during dataeng startup.

test things

run sudo /opt/dell/srvadmin/bin/omreport chassis info, you should see:

Chassis Information
Index                                    : 0
Chassis Name                              : Main System Chassis
Host Name                                :
iDRAC7 Version                           : 1.40.40 (Build 17)
Lifecycle Controller 2 Version           :
Chassis Model                            : PowerEdge R720

Creating an optimally-aligned full-disk partition with parted

I wanted to create a full-disk partition, with optimal alignment, on a 4TB disk under CentOS 6.4 and use it as an LVM PV.

fdisk doesn’t work on disks larger than 2TB so I used parted:

parted -a optimal /dev/sda
(parted) mklabel
Warning: The existing disk label on /dev/sda will be destroyed and all data on this disk will be lost. Do you want to continue?
Yes/No? Yes
(parted) mkpart primary ext2 0% 100%
(parted) set 1 lvm on

Find files matching criteria, exclude NFS mounts

We have app servers with smallish local file systems and application data mounted over NFS.

Sometimes I want to find all files matching a particular set of criteria but don't want to traverse the NFS mounts.

Here's how to do it:

find / -group sophosav -print -o -fstype nfs -prune

Ordering is important, as is the explict inclusion of -print. If you omit this, it will print the name(s) of the NFS mounts as well.

Change start location (/) and criteria (-group sophosav) to suit your own purposes.