OpenSSH/Tunelling

Different types of tunnelling:

1. Local port forwarding(outbound tunnelling): connections from the SSH client are forwarded via the SSH server, then to a destination server
2. Remote port forwarding(inbound tunnelling) (reverse SSH tunnel): connections from the SSH server are forwarded via the SSH client, then to a destination server
3. Dynamic port forwarding: connections from various programs are forwarded via the SSH client, then via the SSH server, and finally to several destination servers

Local port forwarding - outbound tunneling

Used when only gateway/bastion can talk directly to databases (or other servers). The connection is made through gateway so the database see connection coming from the gateway and allows it.

-L <local-port-to-listen>:<remote-host>:<remote-port> <gateway>

Example

in_front_of_firewall@localhost:$ ssh -fNT -L8000:user@remotehost.com:3306 jumpbox.com

The above command sets up an ssh tunnel between your machine and the server, and forwards all traffic from localhost:3306 (in the context of server, localhost is the server itself. 3306 is the default port of MySQL) to localhost:8000 (on your machine). This is useful when you want to expose services running on your server which isn’t accessible to the outside world directly (for good reason).

Eg. connecting to Azure MySQL

# Create a tunnel, localhost will be listening on port 3306
ssh -fNT -L3306:database-eu.mysql.database.azure.com:3306 db@bastion

# Connect, to localhost, by default that port 3306, will forward connection over 
# the tunnel to database database-eu.mysql.database.azure.com:3306 using username 'admin@database-eu'
mysql -h 127.0.0.1 -u admin@database-eu -p

Remote port forwarding (inbound tunnelling)

This is also known as reverse SSH tunnel.

-R <sourcePort><forwardToHost>:<onPort> <user@gate>

Listen on sourcePort, then forward traffic to forwardToHost on port onPort via user@gate server.

From the firewalled host:

behind-firewall@server2:$ ssh -fNT -R2222:localhost:22 in-front-of-firewall@server1.com

Steps

in-front-of-firewall@server1           //NAT//           behind-firewall@server2       
                 <--------------------connection_init--------
2. the tunnel listens on             //FW //             1. this server initiates a -R reverse tunnel 
   localhost:2222 port and forwards                         (because is allowed out through FW/NAT)
   any data down via tunnel to port 22 of the other end
                 --------------------------------------------
       data >>> :2222      > > >   tunnel       > > >       :22
                 --------------------------------------------

This tells your client to establish a tunnel with a -Remote entry point. Anything that attaches to port 2222 on the far end of the tunnel will actually reach "localhost port 22" (computer that you execute the command).

Then at the in-front-of-firewall@server1 you can connect over ssh to localhost:2222 will send all traffic through the tunnel to behind-firewall@server2. This resolves issue for a remote users that needs a temporary access from theirs machines that are in-front-of-firewall.

ssh -p 2222 behind-firewall@localhost

The other options are:

• -f tells ssh to background itself after it authenticates, so you don't have to sit around running something on the remote server for the tunnel to remain alive.
• -N says that you want an SSH connection, but you don't actually want to run any remote commands. If all you're creating is a tunnel, then including this option saves resources.
• -T disables pseudo-tty allocation, which is appropriate because you're not trying to create an interactive shell.

Local port forwarding diagram

Example of ~/.ssh/config entry

Host redshift1
  HostName 10.1.1.1              # connect to this host
  User ubuntu                    # with this user
  IdentityFile ~/.ssh/id_rsa.pem # and the private key
  LocalForward 5439 redshift1.x.x.redshift.amazonaws.com:5439 # listen locally on this port, and forward via HostName to redshift.*:5439

All commands are executed from Host-A.

Local port forwarding keeping all traffic local

Local port forwarding

• -g it's Allow reomote connections (Gateway Ports), so Host-C can connect via Host-A

Remote port forwarding diagram

Remote port forwarding

Show current tunnels

Show -L local forwarding tunnels

netstat -tpln | grep ssh    #t: TCP, p: show process, l: listening, n: numeric values
(header added, tested on Debian wheezy)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.1:1443          0.0.0.0:*               LISTEN      4036/ssh

Which can be read as: SSH (not SSHd) is listening to local TCP port 1443

if you only want to list tunnels created by ssh

$ sudo lsof -i -n | egrep '\<ssh\>'
ssh  19749  user  3u  IPv4 148088244   TCP x.x.x.x:39689->y.y.y.y:22 (ESTABLISHED)
ssh  19749  user  4u  IPv6 148088282   TCP [::1]:9090 (LISTEN)
ssh  19749  user  5u  IPv4 148088283   TCP 127.0.0.1:9090 (LISTEN)

(that would be a -L 9090:localhost:80 tunnel)

Shows -R reverse tunnels

if you want to see the tunnels / connections made to a sshd:

$ sudo lsof -i -n | egrep '\<sshd\>'
sshd  15767  root  3u  IPv4 147401205   TCP x.x.x.x:22->y.y.y.y:27479 (ESTABLISHED)
sshd  15842  user  3u  IPv4 147401205   TCP x.x.x.x:22->y.y.y.y:27479 (ESTABLISHED)
sshd  15842  user  9u  IPv4 148002889   TCP 127.0.0.1:33999->127.0.0.1:www (ESTABLISHED)
sshd  1396   user  9u  IPv4 148056581   TCP 127.0.0.1:5000 (LISTEN)
sshd  25936  root  3u  IPv4 143971728   TCP *:22 (LISTEN)

the ssh-daemon listens on port 22 (last line), 2 subprocesses are spawned (first 2 lines, login of 'user'), a -R tunnel created on port 5000, and a -L tunnel which forwards a port from my (local) machine to localhost:80 (www).

sudo lsof -i -n | egrep '\<sshd\>' | grep -v ":ssh" | grep LISTEN | sed 1~2d | awk '{ print $2}' | while read line; do sudo lsof -i -n | egrep $line | sed 3~3d | sed 's/.*->//' | sed 's/:......*(ESTABLISHED)//' | sed 's/.*://' | sed 's/(.*//' | sed 'N;s/\n/:/' 2>&1 ;done

Windows Putty ssh tunnelling

The all above can also be achieved using Putty.

Putty Local port forwarding tunnel

Once connected the tunnel is established and from the local PC you can eg. go http://127.0.0.1:8888 that in effect pull data from server2:4444. If you don't want to have open CLI session you can disable Pseudo TTY terminal in Putty>Connection>SSH>TTY, tick Don't allocate a pseudo-terminal

Advanced use cases

#                  tunnel in beetween          via gateway
                ___________|______________         |
               /                          \        |
$ ssh -nNT -R 0.0.0.0:4000:192.168.1.111:631 user@gateway.com
                /                  \ 
#      available to          explicit bind addresss
#       all ifaces             (default localhost)
#    (default localhost)

• Instead of using the default bind address, I explicitly use 0.0.0.0. This makes the service available on the server on port 4000, will be accessible internally to the server network across all networks interfaces, including: bridge networks & virtual networks such as used by container environments such as docker.
• Instead of using localhost as the bind address for the local service, I have explicitly used the 192.168.1.111 IP Address, which can be the IP Address of an internal machine, such as a network printer. This allows to expose internal network printer directly from the server.

NOTE

From the ssh manual regarding bind address usage

By default, the local port is bound in accordance with the GatewayPorts setting. However, an explicit bind_address may be used to bind the connection to a specific address. The bind_address of localhost indicates that the listening port be bound for local use only, while an empty address or * indicates that the port should be available from all interfaces.

References

• The Black Magic Of SSH vimeo
• ssh-tunneling-work unix.stackexchange.com, diagrams
• ssh tunnels 21 usages of openssh
• Ssh tunnel - SOCKS proxy