WordPress with php-fpm under Apache 2.4

I recently migrated this blog to a new server running CentOS 7 and decided to use php-fpm and mod_proxy_fcgi instead of mod_php. I also like to install WordPress in its own directory and had problems getting the wp-admin sections of the site to work. I figured it out eventually with help from this page: https://wiki.apache.org/httpd/PHPFPMWordpress

This is the complete apache config fragment that defines the vhost, including SSL:

<VirtualHost *:443>
  ServerName blog.yo61.com

  ## Vhost docroot
  DocumentRoot "/var/www/sites/blog.yo61.com/html"

  ## Directories, there should at least be a declaration for /var/www/sites/blog.yo61.com/html

  <Directory "/var/www/sites/blog.yo61.com/html">
    AllowOverride FileInfo
    Require all granted
    DirectoryIndex index.php
    FallbackResource /index.php
  </Directory>

  <Directory "/var/www/sites/blog.yo61.com/html/wordpress/wp-admin">
    AllowOverride None
    Require all granted
    FallbackResource disabled
  </Directory>

  ## Logging
  ErrorLog "/var/log/httpd/blog.yo61.com_https_error_ssl.log"
  ServerSignature Off
  CustomLog "/var/log/httpd/blog.yo61.com_https_access_ssl.log" combined

  ## SSL directives
  SSLEngine on
  SSLCertificateFile      "/etc/letsencrypt/live/blog.yo61.com/cert.pem"
  SSLCertificateKeyFile   "/etc/letsencrypt/live/blog.yo61.com/privkey.pem"
  SSLCertificateChainFile "/etc/letsencrypt/live/blog.yo61.com/chain.pem"
  SSLCACertificatePath    "/etc/pki/tls/certs"

  ## Custom fragment
  ProxyPassMatch ^/(.*.php(/.*)?)$ fcgi://127.0.0.1:9000/var/www/sites/blog.yo61.com/html/$1
  Header always set Strict-Transport-Security "max-age=63072000; includeSubdomains; preload"
  Header always set X-Frame-Options DENY
  Header always set X-Content-Type-Options nosniff
</VirtualHost>

passenger native libs on CentOS 7

I'm setting up a new puppet master running under passenger on CentOS 7 using packages from the puppetlabs and foreman repos. I used a fork of Stephen Johnson's puppet module to set everything up (with puppet apply). All went swimmingly, except I would see this error in the logs the first time the puppet master app loaded (ie. the first time it got a request):

[ 2014-11-07 23:22:13.2600 2603/7f1a0660e700 Pool2/Spawner.h:159 ]: [App 2643 stderr] *** Phusion Passenger: no passenger_native_support.so found for the current Ruby interpreter. Compiling one (set PASSENGER_COMPILE_NATIVE_SUPPORT_BINARY=0 to disable)...
[ 2014-11-07 23:22:13.2600 2603/7f1a0660e700 Pool2/Spawner.h:159 ]: [App 2643 stderr] # mkdir -p /usr/share/gems/gems/passenger-4.0.18/lib/phusion_passenger/locations.ini/buildout/ruby/ruby-2.0.0-x86_64-linux
[ 2014-11-07 23:22:13.2600 2603/7f1a0660e700 Pool2/Spawner.h:159 ]: [App 2643 stderr] Not a valid directory. Trying a different one...
[ 2014-11-07 23:22:13.2600 2603/7f1a0660e700 Pool2/Spawner.h:159 ]: [App 2643 stderr] -------------------------------
[ 2014-11-07 23:22:13.2600 2603/7f1a0660e700 Pool2/Spawner.h:159 ]: [App 2643 stderr] # mkdir -p /var/lib/puppet/.passenger/native_support/4.0.18/ruby-2.0.0-x86_64-linux
[ 2014-11-07 23:22:13.2600 2603/7f1a0660e700 Pool2/Spawner.h:159 ]: [App 2643 stderr] # cd /var/lib/puppet/.passenger/native_support/4.0.18/ruby-2.0.0-x86_64-linux
[ 2014-11-07 23:22:13.2600 2603/7f1a0660e700 Pool2/Spawner.h:159 ]: [App 2643 stderr] # /usr/bin/ruby '/usr/share/gems/gems/passenger-4.0.18/ruby_extension_source/extconf.rb'
[ 2014-11-07 23:22:13.3048 2603/7f1a0660e700 Pool2/Spawner.h:159 ]: [App 2643 stderr] /usr/bin/ruby: No such file or directory -- /usr/share/gems/gems/passenger-4.0.18/ruby_extension_source/extconf.rb (LoadError)
[ 2014-11-07 23:22:13.3156 2603/7f1a0660e700 Pool2/Spawner.h:159 ]: [App 2643 stderr] Compilation failed.
[ 2014-11-07 23:22:13.3156 2603/7f1a0660e700 Pool2/Spawner.h:159 ]: [App 2643 stderr] -------------------------------
[ 2014-11-07 23:22:13.3157 2603/7f1a0660e700 Pool2/Spawner.h:159 ]: [App 2643 stderr] Ruby native_support extension not loaded. Continuing without native_support.

I double checked, and I do have the native libs installed – they're in the rubygem-passenger-native-libs rpm – the main library is in /usr/lib64/gems/ruby/passenger-4.0.18/native/passenger_native_support.so.

Digging in the passenger code, it tries to load the native libs by doing:

require 'native/passenger_native_support'

If I hacked this to:

require '/usr/lib64/gems/ruby/passenger-4.0.18/native/passenger_native_support'

then it loaded correctly.

It seems that /usr/lib64/gems/ruby/passenger-4.0.18 is not in the ruby load path.

Additional directories can be added to the ruby load path by setting an environment variable, RUBYLIB.

To set RUBYLIB for the apache process, I added the following line to /etc/sysconfig/httpd and restarted apache:

RUBYLIB=/usr/lib64/gems/ruby/passenger-4.0.18

The passenger native libraries now load correctly.

Building mod_proxy_wstunnel for CentOS 6

I had a need to be able to put an Apache-based reverse proxy in front of an install of Uchiwa which is a Node.js-based dashboard for Sensu. The only problem is that it uses WebSockets which means it doesn’t work with the regular mod_proxy_http module. In version 2.4.5 onwards there is mod_proxy_wstunnel which fills in the gap however CentOS 6 only has a 2.2.15 (albeit heavily patched) package.

There are various instructions on how to backport the module for 2.2.x (mostly for Ubuntu) but these involve compiling the whole of Apache from source again with the module added via an additional patch. I don’t want to maintain my own Apache packages but more importantly Apache has provided apxs a.k.a the APache eXtenSion tool to compile external modules without requiring the whole source tree available.

So, I have created a standalone RPM package for CentOS 6 that just installs the mod_proxy_wstunnel module alongside the standard httpd RPM package. In order to do this I took the original patch and removed the alterations to the various build files and also flattened the source into a single file, (the code changes were basically adding whole new functions so they were fine to just inline together). The revised source file and accompanying RPM spec file are available in this Github gist.

Apache reverse-proxying and the REMOTE_USER variable

I spent an alarming amount of time yesterday attempting to make the most of Apache's ridiculously easy mod_auth_kerb module for SSO Kerberos authentication with a little in-house Sinatra app I've been working on. Apparently Kerberos within nginx or ruby is a bit of an unofficial ballache, so I decided to take the easy route out. However, it transpires that only one person on the whole internet knew of the existence of the ProxyPassInterpolateEnv boolean.

To put this in context, for my app I only want kerberos to validate the user and then pass on the username to the app. It's a git deploy frontend, and I like blaming people.

You'll find a lot of stuff about doing a complicated rewrite so that REMOTE_USER actually evaluates before a reverse proxy. I couldn't get any of this stuff to work - not only that but it's a horrible solution anyway requiring about three lines of rewrite - and I'll be honest, I'm not up together on my apache rewrites anyway.

So the following is the solution I ended up with. It simply makes Apache forward on the REMOTE_USER variable, created by your auth module, to whatever you're reverse proxying - in my case a Sinatra app. It actually appears as REMOTE_USER as opposed to the specified REMOTE-USER as well. I neither know why nor care.

Excuse the formatting.

<Virtualhost *:443>
SSLEngine on
SSLCipherSuite ...
              HA HA SSL BUSINESS
        ...
ServerName yer-mum.com
ProxyPassInterpolateEnv On
ProxyPass / http://localhost:4567/
RequestHeader set REMOTE-USER %{REMOTE_USER}s
    <Location />
        AuthType Kerberos
        AuthName "AD Login"
        KrbMethodNegotiate On
        KrbMethodK5Passwd On
        KrbAuthRealms MUMS.COM
        Krb5KeyTab /etc/krb5.keytab
        Require valid-user
    </Location>
</Virtualhost *:443>

Apache reverse-proxying and the REMOTE_USER variable

I spent an alarming amount of time yesterday attempting to make the most of Apache's ridiculously easy mod_auth_kerb module for SSO Kerberos authentication with a little in-house Sinatra app I've been working on. Apparently Kerberos within nginx or ruby is a bit of an unofficial ballache, so I decided to take the easy route out. However, it transpires that only one person on the whole internet knew of the existence of the ProxyPassInterpolateEnv boolean.

To put this in context, for my app I only want kerberos to validate the user and then pass on the username to the app. It's a git deploy frontend, and I like blaming people.

You'll find a lot of stuff about doing a complicated rewrite so that REMOTE_USER actually evaluates before a reverse proxy. I couldn't get any of this stuff to work - not only that but it's a horrible solution anyway requiring about three lines of rewrite - and I'll be honest, I'm not up together on my apache rewrites anyway.

So the following is the solution I ended up with. It simply makes Apache forward on the REMOTE_USER variable, created by your auth module, to whatever you're reverse proxying - in my case a Sinatra app. It actually appears as REMOTE_USER as opposed to the specified REMOTE-USER as well. I neither know why nor care.

Excuse the formatting.

<Virtualhost *:443>
SSLEngine on
SSLCipherSuite ...
              HA HA SSL BUSINESS
        ...
ServerName yer-mum.com
ProxyPassInterpolateEnv On
ProxyPass / http://localhost:4567/
RequestHeader set REMOTE-USER %{REMOTE_USER}s
    <Location />
        AuthType Kerberos
        AuthName "AD Login"
        KrbMethodNegotiate On
        KrbMethodK5Passwd On
        KrbAuthRealms MUMS.COM
        Krb5KeyTab /etc/krb5.keytab
        Require valid-user
    </Location>
</Virtualhost *:443>

Issues with iptables stateful filtering

I hit a weird issue today, I have Apache configured as a reverse proxy using mod_proxy_balancer which is a welcome addition in Apache 2.2.x. This is forwarding selected requests to more Apache instances running mod_perl, although this could be any sort of application layer, pretty standard stuff.

With ProxyStatus enabled and pushing a reasonable amount of traffic through the proxy, I started to notice that the application layer Apache instances would consistently get marked in error state every so often, removing them from the pool of available servers until their cooldown period expired and the proxy enabled them again.

Investigating the logs showed up this error:

[Tue Sep 28 00:24:39 2010] [error] (113)No route to host: proxy: HTTP: attempt to connect to 192.0.2.1:80 (192.0.2.1) failed
[Tue Sep 28 00:24:39 2010] [error] ap_proxy_connect_backend disabling worker for (192.0.2.1)

A spot of the ol’ google-fu turned up descriptions of similar problems, some not even related to Apache. The problem looked related to iptables.

All the servers are running CentOS 5 and anyone who runs this is probably aware of the stock Red Hat iptables ruleset. With HTTP access enabled, it looks something similar to this:

1
2
3
4
5
6
7
8
9
10
11
12
13
-A INPUT -j RH-Firewall-1-INPUT
-A FORWARD -j RH-Firewall-1-INPUT
-A RH-Firewall-1-INPUT -i lo -j ACCEPT
-A RH-Firewall-1-INPUT -p icmp --icmp-type any -j ACCEPT
-A RH-Firewall-1-INPUT -p 50 -j ACCEPT
-A RH-Firewall-1-INPUT -p 51 -j ACCEPT
-A RH-Firewall-1-INPUT -p udp --dport 5353 -d 224.0.0.251 -j ACCEPT
-A RH-Firewall-1-INPUT -p udp -m udp --dport 631 -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m tcp --dport 631 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 80 -j ACCEPT
-A RH-Firewall-1-INPUT -j REJECT --reject-with icmp-host-prohibited

Almost all of it is boilerplate apart from line 12, which I added and is identical to the line above it granting access to SSH. Analysing the live hit counts against each of these rules showed a large number hitting that last catch-all rule on line 13 and indeed, this is what is causing the Apache errors.

Analysing the traffic with tcpdump/wireshark showed that the frontend Apache server is only getting as far as sending the initial SYN packet and it’s failing to match either the dedicated rule on line 12 for HTTP traffic, or even the rule on line 10 to match any related or previously established traffic, although I wouldn’t really expect it to match that.

Adding a rule before the last one to match and log any HTTP packets that are considered to be in the state INVALID showed that indeed for some strange reason, iptables is deciding that an initial SYN is somehow invalid.

More information about why it might be invalid can be coaxed from the kernel by issuing the following:

# echo 255 > /proc/sys/net/ipv4/netfilter/ip_conntrack_log_invalid

Although all this gave me was some extra text saying “invalid state” and the same output you get from the standard logging target:

ip_ct_tcp: invalid state IN= OUT= SRC=192.0.2.2 DST=192.0.2.1 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=21158 DF PROTO=TCP SPT=57351 DPT=80 SEQ=1402246598 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A291DC7600000000001030307)

From searching the netfilter bugzilla for any matching bug reports I found a knob that relaxes the connection tracking, enabled with the following:

# echo 1 > /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_be_liberal

This didn’t fix things, plus it’s recommended to only use this in extreme cases with a broken router or firewall. As all of these machines are on the same network segment with just a switch connecting them there shouldn’t be anything mangling the packets.

With those avenues exhausted the suggested workaround of adding rules similar to the following:

1
-A RH-Firewall-1-INPUT -m tcp -p tcp --dport 80 --syn -j REJECT --reject-with tcp-reset

before your last rule doesn’t really work as the proxy still drops the servers from the pool as before, but just logs a different reason instead:

[Tue Sep 28 13:59:23 2010] [error] (111)Connection refused: proxy: HTTP: attempt to connect to 192.0.2.1:80 (192.0.2.1) failed
[Tue Sep 28 13:59:23 2010] [error] ap_proxy_connect_backend disabling worker for (192.0.2.1)

This isn’t really acceptable to me anyway as it requires your proxy to retry in response to a server politely telling it to go away and it’s going to be a performance hit. The whole point of using mod_proxy_balancer for me was so it could legitimately remove servers that are dead or administratively stopped, how is it supposed to tell the difference between that and a dodgy firewall?

The only solution that worked and was deemed acceptable was to simply remove the state requirement on matching the HTTP traffic, like so:

-A RH-Firewall-1-INPUT -m tcp -p tcp --dport 80 -j ACCEPT

This will match both new and invalid packets, however it’s left me with a pretty low opinion of iptables now. Give me pf any day.

Apache 2.2 finally hits debian

Yup, the long wait is finally over and thanks to a cast of thousands 2.2.3 is now in experimental.

I’d like to extend thanks to Mark and Canonical for sponsoring much of the original work, and also the sprint at the start of this year that got most of the remaining work done.

What we really need now is lots of upgrade reports so we can figure out how much automated help a 2.0->2.2 upgrade can reliably provide, and also where. I’ve been running these packages in production for some time so I’m not that concerned about overall stability, but I’ve not been using some of the weirder modules. We also need to get third-party module packages to stage updated packages into experimental built against 2.2