I’ve recently started using Ansible to manage Elastic Compute Cloud (EC2) hosts on Amazon Web Services (AWS). While it is possible to have public IP addresses for EC2 instances on an AWS Virtual Private Cloud (VPC), I opted to place the EC2 instances on a private VPC subnet which does not allow direct access from the Internet. This makes communicating with the EC2 instances a little more complicated.
While I could create a VPN connection to the VPC, this is rather cumbersome without a compatible hardware router. Instead, I opted to create a bastion host which allows me to connect to the VPC, and communicate securely with EC2 instances over SSH.
I run a fairly simple VPC architecture with four subnets, two public and two private, with one of each type paired in separate availability zones. The public subnets have direct Internet access, whereas the private subnets cannot be addressed directly, and must communicate with the Internet via a NAT gateway.
In the diagram, my computer at
188.8.131.52 wants to run Ansible against an EC2 instance at
172.31.50.5 in “Private Subnet 2.”
172.31.0.0/16 is a Class B private network; its addresses cannot be routed over the Internet. Furthermore, as “Private Subnet 2” does not have direct access to the Internet (it is via the NAT gateway at
172.31.32.2), there is no way to assign a public IP address.
On this network, in order to communicate with
172.31.50.5, my computer must either be connected to the VPC with a VPN connection, or by forwarding traffic via the bastion host. In my case, a VPN connection is not feasible, so I made use of the bastion host, which has both a publicly routable IP address (
184.108.40.206), and a private address on the
172.16.0.0/16 network at
SSH Jump Hosts
A common practice to reach hosts on an internal network which are not directly accessible is to use an SSH jump host. Once an SSH connection is made to the jump hosts, additional connections can be made to hosts on the internal network from the jump host.
Generally, this looks something like:
jk@localhost:~$ ssh firstname.lastname@example.org
email@example.com:~$ ssh firstname.lastname@example.org
This could also be simplified as one command invocation:
jk@localhost:~$ ssh -t email@example.com 'ssh firstname.lastname@example.org'
-t to force pseudo-TTY allocation.)
The connections from the jump host to other hosts do not necessarily need to be SSH connections. For example, a socket connection can be opened:
jk@localhost:~$ ssh email@example.com 'nc 192.168.0.20 22'
Ansible makes use of SSH to connect to remote hosts. However, it does not support configuration of an explicit SSH jump host. This would make it impossible for Ansible to connect to a private IP address without other networking (e.g. VPN) magic. Fortunately, Ansible takes common SSH configuration options, and will respect the contents of a system SSH configuration file.
ProxyCommand option for SSH allows specifying a command to execute to connect to a remote host when connecting via SSH. This allows us to abstract the specifics of connecting to the remote host to SSH; we can get SSH to provide a jump host connection transparently to Ansible.
ProxyCommand works by substituting the standard SSH socket connection with what is specified in the
ssh -o ProxyCommand="ssh firstname.lastname@example.org 'nc 192.168.0.20 22'" ubuntu@nothing
The above command will, for example, first connect to
220.127.116.11 via SSH, and then open a socket to
192.168.0.20 on port 22. The socket connection (which is connected to the remote SSH server) is then passed to the original SSH client command invocation to utilize.
ProxyCommand allows the interpolation of the original host and port to connect to with the
ssh -o ProxyCommand="ssh email@example.com 'nc %h %p'" firstname.lastname@example.org
Is equivalent to running:
ssh -o ProxyCommand="ssh email@example.com 'nc 192.168.0.20 22'" firstname.lastname@example.org
SSH Configuration File
ProxyCommand in conjunction with an SSH configuration file, we can make SSH connections to a private IP address appear seamless to whichever application is executing SSH.
For my VPC architecture described above, I could add the following to an SSH configuration file:
ProxyCommand ssh email@example.com nc %h %p
This makes all SSH connections to the private IP address
ssh -F ./mysshconfig_file firstname.lastname@example.org
And, if using the default
.ssh/config for storing your SSH configuration options, you don’t even need to specify the
All Together Now
ProxyCommand option, it is simple to abstract away the details of the underlying connection to the EC2 instances on the private VPC subnet and allow Ansible to connect to those hosts normally. Any hosts on the private VPC subnet can be added explicitly to an SSH configuration file, or the pattern can be expanded. For example, we can apply the
ProxyCommand option to all hosts on the
172.31.0.0/16 VPC subnet:
ProxyCommand ssh email@example.com nc %h %p
When running Ansible, the host’s inventory can simply specify the private IP address (such as
172.31.2.5) as the connection hostname/address, and SSH will handle the necessary underlying connections to the bastion host.
Generally, the system or user SSH configuration file (
~/.ssh/config)can be used, but Ansible-specific SSH configuration options can also be included in the
This is particularly convenient when using dynamic host inventories with EC2, which can automatically return the private IP addresses of new EC2 instances from the AWS APIs.
Additional SSH and
nc flags can be addded to the
ProxyCommand option to enhance flexibility.
For example, adding in
-A to enable SSH agent forwarding,
-q to suppress extra SSH messages,
-w to adjust the timeout for
nc, and any other standard SSH configuration options:
ProxyCommand ssh -q -A firstname.lastname@example.org nc -w 300 %h %p
The post Ansible Communication with AWS EC2 Instances on a VPC appeared first on Atomic Spin.