---
date: "2021-02-10T22:11:00Z"
title: "SSH Jumping and Bastion Hosts"
description: "Working with secure network architectures with bastion hosts and SSH jumping."
tags: [100daystooffload, technology, security]
---

For many small or personal services running on a VPS in the cloud, administration is often done by connecting directly to the server via SSH. Such servers should be hardened with firewalls, employ an SSHd config that denies root and password-based login, run [fail2ban](https://www.fail2ban.org), and other services and practices.

Linode has some [great getting-started guides](https://www.linode.com/docs/guides/securing-your-server) on the essentials of securing your server.

## Protecting sensitive servers

In more complex production scenarios heightened security can be achieved by isolating application (webapp, API, database, etc.) servers from external internet traffic. This is usually done by placing these "sensitive/protected" servers in a private [subnet](https://en.wikipedia.org/wiki/Subnetwork), without direct internet-facing network interfaces. This means that the server is not reachable from the outside world.

In this type of scenario, outbound traffic from the sensitive server can be routed through a [NAT gateway](https://en.wikipedia.org/wiki/Network_address_translation) and inbound traffic can be funnelled through a [load-balancer](https://en.wikipedia.org/wiki/Load_balancing_(computing)) or reverse proxy server. In both these cases the NAT gateway and load-balancer would exist in public subnets (with internet-facing network interfaces) and can reach the sensitive server through private network interfaces in order to forward requests (e.g. web traffic).

![Diagram of public and private subnets, with a NAT gateway and load balancer](/media/blog/ssh1.png)

Now the question is around how one _does_ manage the services running on the protected server, since it is no longer available to connect to. Traditionally this is done by introducing _bastion hosts_ into your network.

## Bastion hosts

[Bastion hosts](https://en.wikipedia.org/wiki/Bastion_host) - like the NAT gateway and load balancers - sit in the public subnet and so they are available to the outside world. They often accept SSH connections, from which one can "jump" through to the protected servers through the bastion's private networking interface.

![Adding a bastion host to the cloud infrastructure](/media/blog/ssh2.png)

Bastion hosts should be hardened as much as possible (with firewalls and other network rules), and should run a limited set of services - in many cases simply SSHd.

This server then enables administrators to connect through to the protected servers in order to carry out maintenance, upgrades, or other tasks.

## Connecting through a bastion host

SSH port-forwarding is a widely-used concept, in which a secure tunnel to a service running on the protected server is opened via a port on the local machine (using `ssh`'s `-L` option).

Another option is to use proxy _jumping_ (with the `-J` option):

```shell
ssh -J bastion.company.com protected.company-internal.com
```

In this example the user connects via the bastion through to the protected server at `protected.company-internal.com`. Since you should be using key-based authentication to connect, you may also need to specify the private key path (with the `-i` option), and also tell SSH to forward your agent to the bastion (using `-A`) so it can continue the connection.

This can make things hard to remember each time. You could write the command in a script, however it's probably easier to use SSH's own local configuration. To do so, you can add the following to your local `~/.ssh/config` file:

```
Host *.company-internal.com
  ProxyJump bastion.company.com
  User username
  IdentityFile /home/username/.ssh/identity
```

With that in place you can now simply run the following when you want to connect to the protected server:

```shell
ssh protected.company-internal.com
```

_Note: depending on your system you may need to add the key to your local agent first, but you just need to do this once per login (`ssh-add ~/.ssh/identity`)_.

## A note on DNS

Generally I would probably avoid assigning public domain names to a bastion host, as this may invite unwanted attention and traffic (even if the host is secured). Instead you can just include the IP address directly in the `ProxyJump` line of `.ssh/config`. _I used domain names in the examples above to make the process clearer_.

Also, in the above example I refer to the `company-internal.com` domain for use within the private network. This domain should only be resolvable within members of the private network - either by using an internal DNS server or by simply modifying `/etc/hosts` on the bastion. Alternatively you can just use the private IP address for the protected server on the `Host` line of `.ssh/config`.

## Additional notes

In setups like this you may also want to consider the following:

### Private keys

Don't provision these on your bastion host. Instead use agent forwarding (as described above). You'll need to add your public keys to both the bastion and protected servers.

### Restrict source network

For extra security you can restrict SSH connections to your bastion only from trusted networks (e.g. your office network or a VPN).

Similarly, restrict protected servers such that they only accept SSH traffic from the bastion, and not from other servers on the network.

### Make use of managed services if/when possible

For extra security you can use managed services when they are available. For example, if you use AWS then you can make use of a combination of VPCs, subnets, NAT gateways, elastic load balancing, security groups, Route 53, and other services to secure your hosts and control your network. You can of course set this up on your own servers without relying on managed services.

Either way, I hope this post has helped shed light on some simple ways to improve network security for your applications and services.