Overriding yum variables

If you work with rpm-based systems you will probably have seen content like this in the repo config files:

[base]
name=CentOS-$releasever - Base
mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=os&infra=$infra
#baseurl=http://mirror.centos.org/centos/$releasever/os/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7

The items in bold are yum variables.

Today, I needed to install i386 packages on a system running an x86_64 kernel (don't ask!).

Here's how I did it:

echo i386 > /etc/yum/vars/basearch

Documentation here: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/deployment_guide/sec-using_yum_variables

The power of packaging software, package all the things

Software delivery is hard, plenty of people all over this planet are struggling with delivering software in their own controlled environment. They have invented great patterns that will build an artifact, then do some magic and the application is up and running.

When talking about continuous delivery, people invariably discus their delivery pipeline and the different components that need to be in that pipeline.
Often, the focus on getting the application deployed or upgraded from that pipeline is so strong that teams
forget how to deploy their environment from scratch.

After running a number of tests on the code , compiling it where needed, people want to move forward quickly and deploy their release artifact on an actual platform.
This deployment is typically via a file upload or a checkout from a source-control tool from the dedicated computer on which the application resides.
Sometimes, dedicated tools are integrated to simulate what a developer would do manually on a computer to get the application running. Copy three files left, one right, and make sure you restart the service. Although this is obviously already a large improvement over people manually pasting commands from a 42 page run book, it doesn’t solve all problems.

Like the guy who quickly makes a change on the production server, never to commit the change, (say goodbye to git pull for your upgrade process)
If you package your software there are a couple of things you get for free from your packaging system.
Questions like, has this file been modified since I deployed it, where did this file come from, when was it deployed,
what version of software X do I have running on all my servers, are easily answered by the same
tools we use already for every other package on the system. Not only can you use existing tools you are also using tools that are well known by your ops team and that they
already use for every other piece of software on your system.

If your build process creates a package and uploads it to a package repository which is available for the hosts in the environment you want to deploy to, there is no need anymore for
a script that copies the artifact from a 3rd party location , and even less for that 42 page text document which never gets updated and still tells you to download yaja.3.1.9.war from a location where you can only find
3.2 and 3.1.8 and the developer that knows if you can use 3.2 or why 3.1.9 got removed just left for the long weekend.

Another, and maybe even more important thing, is the current sadly growing practice of having yet another tool in place that translates that 42 page text document to a bunch of shell scripts created from a drag and drop interface, typically that "deploy tool" is even triggered from within the pipeline. Apart from the fact that it usually stimulates a pattern of non reusable code, distributing even more ssh keys , or adding yet another agent on all systems. it doesn’t take into account that you want to think of your servers as cattle and be able to deploy new instances of your application fast.
Do you really want to deploy your five new nodes on AWS with a full Apache stack ready for production, then reconfigure your load balancers only to figure out that someone needs to go click in your continuous integration tool or deployment to deploy the application to the new hosts? That one manual action someone forgets?
Imvho Deployment tools are a phase in the maturity process of a product team.. yes it's a step up from manually deploying software but it creates more and other problems , once your team grows in maturity refactoring out that tool is trivial.

The obvious and trivial approach to this problem, and it comes with even more benefits. is called packaging. When you package your artifacts as operating system (e.g., .deb or .rpm) packages,
you can include that package in the list of packages to be deployed at installation time (via Kickstart or debootstrap). Similarly, when your configuration management tool
(e.g., Puppet or Chef) provisions the computer, you can specify which version of the application you want to have deployed by default.

So, when you’re designing how you want to deploy your application, think about deploying new instances or deploying to existing setups (or rather, upgrading your application).
Doing so will make life so much easier when you want to deploy a new batch of servers.

The power of packaging software, package all the things

Software delivery is hard, plenty of people all over this planet are struggling with delivering software in their own controlled environment. They have invented great patterns that will build an artifact, then do some magic and the application is up and running.

When talking about continuous delivery, people invariably discus their delivery pipeline and the different components that need to be in that pipeline.
Often, the focus on getting the application deployed or upgraded from that pipeline is so strong that teams
forget how to deploy their environment from scratch.

After running a number of tests on the code , compiling it where needed, people want to move forward quickly and deploy their release artifact on an actual platform.
This deployment is typically via a file upload or a checkout from a source-control tool from the dedicated computer on which the application resides.
Sometimes, dedicated tools are integrated to simulate what a developer would do manually on a computer to get the application running. Copy three files left, one right, and make sure you restart the service. Although this is obviously already a large improvement over people manually pasting commands from a 42 page run book, it doesn’t solve all problems.

Like the guy who quickly makes a change on the production server, never to commit the change, (say goodbye to git pull for your upgrade process)
If you package your software there are a couple of things you get for free from your packaging system.
Questions like, has this file been modified since I deployed it, where did this file come from, when was it deployed,
what version of software X do I have running on all my servers, are easily answered by the same
tools we use already for every other package on the system. Not only can you use existing tools you are also using tools that are well known by your ops team and that they
already use for every other piece of software on your system.

If your build process creates a package and uploads it to a package repository which is available for the hosts in the environment you want to deploy to, there is no need anymore for
a script that copies the artifact from a 3rd party location , and even less for that 42 page text document which never gets updated and still tells you to download yaja.3.1.9.war from a location where you can only find
3.2 and 3.1.8 and the developer that knows if you can use 3.2 or why 3.1.9 got removed just left for the long weekend.

Another, and maybe even more important thing, is the current sadly growing practice of having yet another tool in place that translates that 42 page text document to a bunch of shell scripts created from a drag and drop interface, typically that "deploy tool" is even triggered from within the pipeline. Apart from the fact that it usually stimulates a pattern of non reusable code, distributing even more ssh keys , or adding yet another agent on all systems. it doesn’t take into account that you want to think of your servers as cattle and be able to deploy new instances of your application fast.
Do you really want to deploy your five new nodes on AWS with a full Apache stack ready for production, then reconfigure your load balancers only to figure out that someone needs to go click in your continuous integration tool or deployment to deploy the application to the new hosts? That one manual action someone forgets?
Imvho Deployment tools are a phase in the maturity process of a product team.. yes it's a step up from manually deploying software but it creates more and other problems , once your team grows in maturity refactoring out that tool is trivial.

The obvious and trivial approach to this problem, and it comes with even more benefits. is called packaging. When you package your artifacts as operating system (e.g., .deb or .rpm) packages,
you can include that package in the list of packages to be deployed at installation time (via Kickstart or debootstrap). Similarly, when your configuration management tool
(e.g., Puppet or Chef) provisions the computer, you can specify which version of the application you want to have deployed by default.

So, when you’re designing how you want to deploy your application, think about deploying new instances or deploying to existing setups (or rather, upgrading your application).
Doing so will make life so much easier when you want to deploy a new batch of servers.

The power of packaging software, package all the things

Software delivery is hard, plenty of people all over this planet are struggling with delivering software in their own controlled environment. They have invented great patterns that will build an artifact, then do some magic and the application is up and running.

When talking about continuous delivery, people invariably discus their delivery pipeline and the different components that need to be in that pipeline.
Often, the focus on getting the application deployed or upgraded from that pipeline is so strong that teams
forget how to deploy their environment from scratch.

After running a number of tests on the code , compiling it where needed, people want to move forward quickly and deploy their release artifact on an actual platform.
This deployment is typically via a file upload or a checkout from a source-control tool from the dedicated computer on which the application resides.
Sometimes, dedicated tools are integrated to simulate what a developer would do manually on a computer to get the application running. Copy three files left, one right, and make sure you restart the service. Although this is obviously already a large improvement over people manually pasting commands from a 42 page run book, it doesn’t solve all problems.

Like the guy who quickly makes a change on the production server, never to commit the change, (say goodbye to git pull for your upgrade process)
If you package your software there are a couple of things you get for free from your packaging system.
Questions like, has this file been modified since I deployed it, where did this file come from, when was it deployed,
what version of software X do I have running on all my servers, are easily answered by the same
tools we use already for every other package on the system. Not only can you use existing tools you are also using tools that are well known by your ops team and that they
already use for every other piece of software on your system.

If your build process creates a package and uploads it to a package repository which is available for the hosts in the environment you want to deploy to, there is no need anymore for
a script that copies the artifact from a 3rd party location , and even less for that 42 page text document which never gets updated and still tells you to download yaja.3.1.9.war from a location where you can only find
3.2 and 3.1.8 and the developer that knows if you can use 3.2 or why 3.1.9 got removed just left for the long weekend.

Another, and maybe even more important thing, is the current sadly growing practice of having yet another tool in place that translates that 42 page text document to a bunch of shell scripts created from a drag and drop interface, typically that "deploy tool" is even triggered from within the pipeline. Apart from the fact that it usually stimulates a pattern of non reusable code, distributing even more ssh keys , or adding yet another agent on all systems. it doesn’t take into account that you want to think of your servers as cattle and be able to deploy new instances of your application fast.
Do you really want to deploy your five new nodes on AWS with a full Apache stack ready for production, then reconfigure your load balancers only to figure out that someone needs to go click in your continuous integration tool or deployment to deploy the application to the new hosts? That one manual action someone forgets?
Imvho Deployment tools are a phase in the maturity process of a product team.. yes it's a step up from manually deploying software but it creates more and other problems , once your team grows in maturity refactoring out that tool is trivial.

The obvious and trivial approach to this problem, and it comes with even more benefits. is called packaging. When you package your artifacts as operating system (e.g., .deb or .rpm) packages,
you can include that package in the list of packages to be deployed at installation time (via Kickstart or debootstrap). Similarly, when your configuration management tool
(e.g., Puppet or Chef) provisions the computer, you can specify which version of the application you want to have deployed by default.

So, when you’re designing how you want to deploy your application, think about deploying new instances or deploying to existing setups (or rather, upgrading your application).
Doing so will make life so much easier when you want to deploy a new batch of servers.

Building mod_proxy_wstunnel for CentOS 6

I had a need to be able to put an Apache-based reverse proxy in front of an install of Uchiwa which is a Node.js-based dashboard for Sensu. The only problem is that it uses WebSockets which means it doesn’t work with the regular mod_proxy_http module. In version 2.4.5 onwards there is mod_proxy_wstunnel which fills in the gap however CentOS 6 only has a 2.2.15 (albeit heavily patched) package.

There are various instructions on how to backport the module for 2.2.x (mostly for Ubuntu) but these involve compiling the whole of Apache from source again with the module added via an additional patch. I don’t want to maintain my own Apache packages but more importantly Apache has provided apxs a.k.a the APache eXtenSion tool to compile external modules without requiring the whole source tree available.

So, I have created a standalone RPM package for CentOS 6 that just installs the mod_proxy_wstunnel module alongside the standard httpd RPM package. In order to do this I took the original patch and removed the alterations to the various build files and also flattened the source into a single file, (the code changes were basically adding whole new functions so they were fine to just inline together). The revised source file and accompanying RPM spec file are available in this Github gist.

Building RPMs from ruby gems with fpm

Some time ago, I wrote up how I created RPMs for ruby gems to simplify installation on EL-flavoured distributions. In the comments for that article, Jordan Sissel pointed me at his fpm tool which I said I’d check out if I ever needed to build any more rubygem RPMs.

Well, that time has come. I wanted to deploy a later version of capistrano across a client’s infrastructure and my previous approach didn’t work so I grabbed fpm and did this:

mkdir ~/tmp/gems
cd ~/tmp/gems
gem install --no-ri --no-rdoc --install-dir . capistrano
find ./cache -name '*.gem' | xargs -rn1 fpm -s gem -t rpm
ls *.rpm
rubygem-capistrano-2.15.4-1.noarch.rpm	rubygem-net-scp-1.1.0-1.noarch.rpm   rubygem-net-ssh-2.6.7-1.noarch.rpm
rubygem-highline-1.6.19-1.noarch.rpm	rubygem-net-sftp-2.1.2-1.noarch.rpm  rubygem-net-ssh-gateway-1.2.0-1.noarch.rpm

Nice and easy. Kudos whack!

hbase lzo compression on CentOS 6.3

The installation of hbase on CentOS is fairly painless thanks to those generous folks at Cloudera. Add their CDH4 repository and you're there: yum install hbase.

However, adding lzo compression for hbase is a little more tricky. There are a few guides describing how to checkout from github, build the extension, and copy the resulting libraries into the right place, but I want a nice, simple RPM package to deploy.

Enter the hadoop-lzo-packager project on github. Let's try and use this to build an RPM I can use to install lzo support for hbase.

Get the source code:

git clone git://github.com/toddlipcon/hadoop-lzo-packager.git

Install the deps:

yum install lzo-devel ant ant-nodeps gcc-c++ rpmbuild java-devel

Build the RPMs:

cd hadoop-lzo-packager
export JAVA_HOME=/usr/lib/jvm/java
./run.sh --no-debs

Et voila – cloudera-hadoop-lzo RPMS ready for installation. But wait… The libs get installed to /usr/lib/hadoop-0.20… That's no good, I want them in /usr/lib/hbase.

So I went ahead & hacked run.sh and template.spec to allow the install dir on the target machine to be specified on the command-line. I can now use a command line something like this:

./run.sh --name hbase-lzo --install-dir /usr/lib/hbase --no-deb

That produces a set of RPMs (binary, source, and debuginfo) with the base name hbase-lzo and libraries installed to /usr/lib/hbase

My changes (plus another small change adding necessary BuildRequires to the RPM spec template) are in my fork of the project on github

Listing all files with their .rpmnew updates

When you install an update to a package on an RPM-based system, any configuration files which were originally distributed by the package and that have been changed are not replaced when the package is updated. Instead, rpm creates a new file with the extention ".rpmnew". It is then necessary to manually compare the existing file and the new one and decide how to deal with the new file.

So, the pattern I often need to use is "list all .rpmnew files and the corresponding original file". This is not a particularly complex issue, but is one that I don't use often enough to have at my fingertips. The trick is bash variable substitution; specifically pattern matching: ${variable%pattern}. This deletes the shortest possible match for pattern from the right of the contents of $variable.

An example using .rpmnew files created when upgading OpenNMS:

# ls -1 *.rpmnew| while read f ; do ls -l ${f%\.rpmnew}* ; done
-rw-rw-r-- 1 root root 7285 Jun  3 12:43 database-reports.xml
-rw-rw-r-- 1 root root 8232 Aug 10 19:01 database-reports.xml.rpmnew
-rw-rw-r-- 1 root root 333092 Aug 25 09:25 datacollection-config.xml
-rw-rw-r-- 1 root root 314141 May 11 01:31 datacollection-config.xml.rpmnew
-rw-rw-r-- 1 root root 3395 Jun  3 12:43 jasper-reports.xml
-rw-rw-r-- 1 root root 3664 Aug 10 19:01 jasper-reports.xml.rpmnew
-rw-rw-r-- 1 root root 25887 Aug 10 19:01 log4j.properties
-rw-rw-r-- 1 root root 25887 May 11 01:31 log4j.properties.rpmnew
-rw-rw-r-- 1 root root 19319 Apr 22 16:56 opennms.properties
-rw-rw-r-- 1 root root 19635 Aug 10 19:01 opennms.properties.rpmnew
-rw-rw-r-- 1 root root 9252 Apr 22 16:59 rrd-configuration.properties
-rw-rw-r-- 1 root root 9604 May 11 01:31 rrd-configuration.properties.rpmnew
-rw-r--r-- 1 root root 669444 Jun  3 13:51 snmp-graph.properties
-rw-rw-r-- 1 root root 604457 Aug 10 19:01 snmp-graph.properties.rpmnew

To Package, and what to package

One of the open sessions last week (corr: last month) at Devopsdays 2010 Hamburg was the one on packaging software. It's always a big question on wether you package the software that runs in your infrastructure or not. And if you package it .. what do you package ..

The general consensus of the open space was pretty much that you always package the software you deploy, unless you have some very good reasons not to. Pretty much the way I've been doing for ages ..

Good reasons that were mentionned were the use of scripting languages that update extremely frequently, but certainly not for compiled code, compiling code on a production machine also is a big nono.

There also was a consensus that you DO NOT PUT CONFIGURATION inside a package. You can put in default templates, but you don't put in config files that should change frequently .. There's plenty of configuration mgmt systems out there do that kind of stuff for you.

The naysayers claimed that packaging brings way to much overhead ... and others claim it takes to much time... however I feel it
should just be a 1 time effort that brings devs and ops closer to eachother and from there on it should automated
New versions of software don't mean that the packaging effort needs to be done again..

Another topic that gathered lots of questions was if you should be capable of installing multiple versions of the same package , lots of people mentionned they didn't like fiddling with symlinks however the best comment in that discussion was that there is already a system out there , the alternatives setup .. provide by most operating systems that allow you to do so in a pretty clean way. I must admit I should look into alternatives more in depth too ..

The ever recurring question is wether one should package war files ? Sure as you then can also use the dependency models a package mgmt system brings to deploy the dependent libraries.

However when people ship products, rather than a live service they seem to package everything , mainly because the code in the product isn't changing as quickly as a live website, or internally used application.

The biggest problem however is the frustration people have with GEM or CPAN packages .. they add yet another layer of management to a system, most lots of CPAN packages are already packaged.. but when it comes to GEM's disaster strikes. There's a lot of work left for distributions to integrate GEM and CPAN style packages.

Technorati Tags:Technorati Tags:

Trackback URL for this post:

http://www.krisbuytaert.be/blog/trackback/1020

RPM packages for Toggl Desktop client

I use the toggl time-tracking service to keep track of the hours I work for my various clients.

toggl make available desktop clients for Windows, Mac, & Linux, but the Linux packages are in .deb format for Ubuntu and, until recently, they did not provide x86_64 packages.

toggl recently released the desktop client as open source so I grabbed it and have built an RPM.

SRPM: TogglDesktop-2.5.1-1.fc12.src.rpm

RPM (Fedora 12, x86_64): TogglDesktop-2.5.1-1.fc12.x86_64.rpm