It took me half the afternoon to figure out how to do this with all the bells and whistles (mostly a learning experience on systemd), so I’d better write it down! Meanwhile, it took me at least a dozen reference docs to piece it together, because autossh has pretty brief documentation.

Edit 2021-03-27: One thing in the original version of this article that didn’t work as intended was keeping the systemd unit file in a different location and linking it where it needed to be. That turns out not to work right under reload. Further info below.

I had a remote server running a mail forwarder which is password protected but exposed, yet I was only using it from devices within my LAN. That’s not only not ideal, but a smidge less reliable due to various odd properties of my LAN (with regard to WAN failover behaviors), so I wanted to move this traffic into an SSH tunnel. My local tunnel endpoint would be a Debian 10 (Buster) machine that is fully within the LAN, not a perimeter device. I want this connection to come up fully automatically on boot/reboot, and give me simple control as a service (hence systemd).

Remote: an unprivileged, high port PPPPP on server RRR.RRR.RRR.RRR, whose ssh server listens on port xxxxx.

Local: the same unprivileged, high port PPPPP on server LLL.LLL.LLL.LLL

My remote machine was already appropriately set up, all I needed to do was add an unprivileged user account to add this connection. In the code below, you’ll see the account name (on both the local and remote servers) is raul, which is also the local server’s name on the network. Substitute your account name of choice wherever you see this. Before beginning the real work, you need this account set up on both machines, with key based authentication (with no pass phrase). Log into the remote machine from the local account at least once, to verify it works with the certificate, and to store the host key.

Install autossh on your local Debian machine with sudo apt install autossh.

Since everything will be run as your unprivileged user, it’s actually a bit easier to do all your initial editing from that account so that you don’t have to play with permissions later. That’s not how I started, but it would’ve saved me some steps. So, switch to your new account with su raul or equivalent. We’ll be keeping the three files necessary to run this in that account’s home directory at /home/raul/, but one of them is created automatically by your script, so we really only need to write the startup script and the systemd unit.

One thing to note beforehand – the formatting and word wrapping on this site can make it less than obvious what’s an actual newline in code snippets, and what’s just word wrap. Because of this, I’ve linked a copy of the maillink.sh script where you can get to it directly, and just change out your account name, addresses, and ports.

Startup Script

First, we’ll create the startup script, which is the meat of the work. Without further ado, create /home/raul/maillink.sh:

#!/bin/bash

# This script starts an ssh tunnel to matter to locally expose the mail port, PPPPP.                                                                           

logger -t maillink.sh "Begin autossh setup script for mail link."
S_TIME=`awk '{print int($1)}' /proc/uptime`
MAX_TIME="300"

# First, verify connection to outside world is working. This bit is optional.

while true ; do
        M_RESPONSE=`ping -c 5 -A RRR.RRR.RRR.RRR | grep -c "64 bytes from"`
        C_TIME=`awk '{print int($1)}' /proc/uptime`
        E_TIME=`expr $C_TIME - $S_TIME`
        [[ $M_RESPONSE == "5" ]] && break
        if  [ $E_TIME -gt $MAX_TIME ]
        then
                logger -t maillink.sh "Waiting for network, timed out after $E_TIME seconds."                                                                  
                exit 1
        fi
        sleep 10
done

logger -t maillink.sh "Network detected up after $E_TIME seconds."

# Now, start the tunnel in the background.

export AUTOSSH_PIDFILE="/home/raul/maillink.pid"

autossh -f -M 0 raul@RRR.RRR.RRR.RRR -p xxxxx -N \
        -o "ServerAliveInterval 60" -o "ServerAliveCountMax 3" \
        -L LLL.LLL.LLL.LLL:PPPPP:127.0.0.1:PPPPP

MAILPID=`cat /home/raul/maillink.pid`

logger -t maillink.sh "Autostart command run, pid is $MAILPID."

As you can see, this script has a few more bells and whistles than just the most basic. While systemd should take care of making sure we wait until the network is up, I want the script to verify that it can reach my actual remote server before it starts the tunnel. It will try every ten seconds for up to five minutes, until it gets 100% of its test pings back in one go.

I also like logs … putting a few log lines into your script sure makes it easier to figure out why things are going wrong.

The “AUTOSSH_PIDFILE” line is necessary for making this setup play pretty with systemd and give us a way to stop the tunnel the nice way (systemctl stop maillink) instead of manually killing it. That environment variable will cause autossh to store its pid once it starts up. Autossh responds to a nice, default kill by neatly shutting down the ssh tunnel and itself, so it makes that control easy. Of course, finding that and figuring out how to do it was … less easy, but it’s simple once you know. That pid file is the second important file in making this work, but this line creates it automatically whenever the tunnel is working.

Now, the meat of this file is the autossh command. Some of these are for autossh itself, and most get passed to ssh. Here’s a breakdown of each part of this command:

• -f moves the process to background on startup (this is an autossh flag)
• -M 0 turns off the built in autossh connection checking system – we’re relying on newer monitoring built into OpenSSH instead.
• raul@RRR.RRR.RRR.RRR -p xxxxx are the username, address, and port number of the remote server, to be passed to ssh (along with all that follows). If your remote server uses the default port of 22, you can leave the port flag out. If your local and remote accounts for this service will be the same, you can also leave out the account name, but I find it clearer left in.
• -N tells ssh not to execute any remote commands. The manpage for ssh mentions this is useful for just forwarding ports. What nothing tells you, is that in my experience this autossh tunnel simply won’t work without the -N flag. It failed silently until I added it in.
• -o “ServerAliveInterval 60” -o “ServerAliveCountMax 3” are the flags for the built in connection monitoring we’ll be using. ServerAliveInterval causes the client to send a keep-alive null packet to the server at the specified interval in seconds, expecting a response. ServerAliveCountMax sets the limit of how many times in a row the response can fail to arrive before the client automatically disconnects. When it does disconnect, it will exit with a non-zero status (i.e. “something went wrong”), and autossh will restart the tunnel – that’s the main function of autossh. If you kill the ssh process intentionally, it returns 0 and autossh assumes that’s intentional, so it will end itself.
• -L LLL.LLL.LLL.LLL:PPPPP:127.0.0.1:PPPPP is the real meat of the command, as this is the actual port forward. It translates to, “Take port PPPPP of the remote server’s loopback (127.0.0.1) interface, and expose it on port PPPPP of this local client’s interface at address LLL.LLL.LLL.LLL.” That local address is optional, but if you don’t put it in, it will default to exposing that port on the local client’s loopback interface, too. That’s great if you just need to access it from the client computer, but I needed this port exposed to the rest of my LAN.

One handy thing to note, is that you can forward multiple ports through this single tunnel. You can just keep repeating that -L line to forward however many ports you need. Or, if you’re forwarding for various different services that you might not want all active at the same time, it’s easy to duplicate the startup script and service file, tweak the names and contents, and have a second separate service to control.

Before you test this the first time, it’s important to make sure it’s executable!

chmod +x /home/raul/maillink.sh

At this point, if you aren’t looking to make this a systemd service, you’re done – you can actually just start your connection manually using “. /home/raul/maillink.sh” (note the . and space up front) and stop it manually using kill <pid>, where the pid is the number saved in the maillink.pid file. (If you’re planning to do this, it’s actually easiest to keep these in your main user’s home directory and modify the script for the pid location accordingly.) At this point, you should manually test the script to ensure everything is working the way you expected. You should see some helpful output in your syslog, and you should also see that port listening on your local machine if you run netstat -tunlp4.

Systemd Unit

However, with just a little more work, making this controllable turns out to be pretty simple. It took way longer to corral the info on how to do it, than it would’ve taken to do if I’d already known how.

Edit 2021-03-27: I originally tried placing this unit file in /home/raul/ and sym linking it into /etc/systemd/system/. That … well, doesn’t work. It works fine the first time, when you run systemctl daemon-reload to pull the unit into the system. The problem is, for whatever reason systemd will not find that file on reboot, even though the link is still there. You’d have to reload manually every time, which just defeats the purpose. Edits have been made accordingly below.

First, create the systemd unit file, which must be located in /etc/systemd/system/maillink.service:

[Unit]
Description=Keeps a tunnel to remote mailserver open
Wants=network-online.target
After=network-online.target

[Service]
Type=forking
RemainAfterExit=yes
User=raul
Group=raul
ExecStart=/home/raul/maillink.sh                                                                                                                               
ExecStop=kill `cat /home/raul/maillink.pid`

[Install]
WantedBy=multi-user.target

This file contains:

• A description
• Two lines that ensure it won’t run until after networking is up
• Two lines that instruct the system to run it under your “raul” account
• A command to be run to start the service
• A command to be run to stop the service – this shuts it down clean using the pid file we saved on startup

Next, we need to update systemd to see the new service:

sudo systemctl daemon-reload

Now your systemd unit is loaded. You should be able to verify this by running systemctl status maillink, which should give you a bit of information on your service and its description.

Next, we can test the systemd setup out. First start the service once using sudo systemctl start maillink, and make sure it starts without error messages. Check it as well with systemctl status maillink, and verify the port is there using netstat -tnlp4.

If all went well, that status command should give you some nice output, including the process tree for the service, and an excerpt of relevant parts of the syslog.

Make sure you also verify the stop command, with systemctl stop maillink. This should turn off the port in netstat, and you should also no longer see autossh or ssh when you run ps -e.

If all looks good, you’re good to set this up and leave it! Enable the service to autostart using systemctl enable maillink, and if it’s not started already, start it back up with systemctl start maillink.

And, here’s hoping this was clear and helpful. If you catch any bugs, please let me know!