On-demand SSH tunnels with systemd

My homelab includes a small VPS hosted on the internet; let's call it myserver.example.com. It provides me with a static IP address and it mainly runs an application that gathers data for me. For security reasons, the only open port is the SSH one (22) with the password disabled, I can only access with an RSA certificate stored in my ./ssh directory. The application running there has a control panel which I can access that is on port 8080, but I have no intention of exposing an HTTP port on the internet just to access it. To reach the control panel, the better solution is to create a ssh tunnel : ssh -L 58080:localhost:8080 myserver Enter fullscreen mode Exit fullscreen mode and close it when I'm done. That works, but it gets old fast. I have to remember the incantation, leave a terminal open while I use the panel, and remember to close it afterwards. If I forget, an SSH connection sits open for days. If I close it too eagerly, I'm typing it again five minutes later. And it only helps the machine I typed it on, from my phone or the laptop in the other room, no tunnel. What I actually wanted: Open http://:58080 in a browser and reach http://myserver.example.com:8080 through an ssh tunnel. If the tunnel isn't up, it comes up. When I stop using it, it goes away by itself after, say, 10 minutes. It turns out systemd can do exactly this, and you don't have to write a single line of code. This article walks through the whole thing from zero. I'm assuming you know how to SSH into a machine and edit a file. I'll explain the systemd parts and the networking parts as we go. The cast Three machines, or rather three roles. Substitute your own names: The remote server myserver.example.com. Only port 22 open, RSA certificate-only. Runs the application's control panel on port 8080, listening on its own loopback interface only. The gateway, a machine on my home LAN at 192.168.1.104 that is always on. This is where everything in this article gets installed. In my case it's a little always-on box; a Raspberry Pi works fine. Any laptop/phone on my LAN, wants to open the web app in a browser. The goal: 192.168.1.104:58080 → (SSH) → myserver.example.com:8080 (actually localhost:8080). Everything below is done as root on the gateway. Background: the two ideas we're combining Idea 1, SSH local port forwarding ssh -L A:host:B server means: "listen on port A on my machine; anything that connects there, tunnel it through the SSH connection and hand it to host:B as seen from the server." So ssh -L 58080:localhost:8080 myserver.example.com makes port 58080 on my machine behave like port 8080 on the server. The word localhost in the middle is resolved by the server, not by me, it means the server's own loopback. That's the trick that lets us reach a port the server never exposes to the internet: as far as the app is concerned, the connection is arriving from the server itself. Run like that, SSH would also open an interactive shell on the server, which we don't want. Two flags fix that, and both show up in the unit file later: ssh -N -f -L 58080:localhost:8080 myserver.example.com Enter fullscreen mode Exit fullscreen mode -N, "don't run a command, just forward". No shell, no prompt; the connection exists only to carry the tunnel. (The command at the top of the article omits it, which is why it drops you into a shell as well as opening the tunnel.) -f, "go to the background, but only after the forward is established". The ssh you typed returns immediately, while a child process keeps the tunnel up. The "only after" part is what we'll lean on to let systemd know when the tunnel is actually ready. Before anything else: check that the server allows forwarding Port forwarding is a feature the server grants, and it can be switched off. Check this first, because if it's disabled nothing in this article will work and the failure looks like a client problem. On myserver.example.com, look at the SSH daemon's configuration: sudo sshd -T | grep -i 'allowtcpforwarding\|permitopen' Enter fullscreen mode Exit fullscreen mode sshd -T prints the effective configuration, everything actually in force, including defaults and anything pulled in from /etc/ssh/sshd_config.d/*.conf, which is more reliable than grepping sshd_config by hand. You want to see: allowtcpforwarding yes permitopen any Enter fullscreen mode Exit fullscreen mode AllowTcpForwarding defaults to yes in stock OpenSSH, so on most systems this is already fine. But hardened images, some VPS provider templates, and CIS-benchmark hardening scripts routinely set it to no, it's a common item on security checklists, since forwarding lets an authenticated user reach things the server can reach. If yours says no, edit /etc/ssh/sshd_config (or the file under sshd_config.d/ that sets it): AllowTcpForwarding yes Enter fullscreen mode Exit fullscreen mode then reload the daemon, sudo systemctl reload ssh on Debian/Ubuntu, sudo systemctl reload sshd on RHEL-family systems. Reloading does not drop existing sessions, so you won't lock yourself out. Two related settings worth knowing about: PermitOpen restricts where forwards may point. If it's set to anything other than any, your destination must be listed. Tightening it is actually a good idea here, PermitOpen localhost:8080 on the server allows exactly the tunnel we're building and nothing else. AllowTcpForwarding local is enough for us. -L is a local forward; the remote and yes values additionally permit -R, which we don't use. The quickest end-to-end test is simply to try it by hand: ssh -N -L 58080:localhost:8080 myserver.example.com Enter fullscreen mode Exit fullscreen mode and then, from another terminal, curl -I http://127.0.0.1:58080/. If forwarding is disabled you'll see this in the first terminal: channel 1: open failed: administratively prohibited: open failed Enter fullscreen mode Exit fullscreen mode That message means the server refused, not that your app is down, the distinction is worth remembering, because it's the one failure mode that no amount of fiddling on the client side will fix. Idea 2, systemd socket activation This is the part that makes it on-demand, and it's a genuinely elegant piece of design that not enough people know about. Normally a service listens on its own port: it starts, it binds the port, it waits. Socket activation inverts that. systemd binds the port and waits. Nothing else is running. The first time somebody connects, systemd starts the service and hands it the already-connected socket. The service does its job. When it has been idle for a while, it exits, and systemd goes back to holding the port, ready to start it again. From the outside, the port is always open. Behind it, the process only exists while it's being used. That's precisely the behaviour we want from a tunnel. We need one more piece: something to sit between the socket systemd holds and the SSH tunnel. That's systemd-socket-proxyd, a tiny program shipped with systemd whose entire job is "take connections from a socket-activated socket and forward them to some other address". It also has the flag that makes all of this worthwhile: --exit-idle-time=10min Enter fullscreen mode Exit fullscreen mode "Exit after 10 minutes with no connections." The architecture Put together, a connection travels three hops: browser on the LAN │ ▼ 192.168.1.104:58080 ← a systemd .socket unit holds this port open │ (nothing is running yet) ▼ systemd-socket-proxyd ← started on the first connection, │ exits after 10 min idle ▼ 127.0.0.2:58080 ← the local end of the SSH tunnel │ ▼ (SSH, over port 22) │ myserver.example.com, its own localhost:8080 ← the actual web app Enter fullscreen mode Exit fullscreen mode Three systemd units implement this: Unit Job myserver-tunnel@.socket holds the LAN port open, starts the proxy on demand myserver-tunnel@.service runs systemd-socket-proxyd; exits after 10 min idle myserver-ssh.service runs the actual ssh -L; starts when the proxy needs it, stops when it doesn't (The myserver prefix is just a label so the three units sort together. Rename freely, just rename it consistently in all three files.) The @ in two of those names makes them templates. One template file can be instantiated many times, once per port: myserver-tunnel@58080, myserver-tunnel@11000, and so on. Inside the file, %i expands to whatever comes after the @. That's what makes adding a second tunnel a two-line job later. Why 127.0.0.2? The proxy has to hand the connection to the SSH tunnel somewhere, and that somewhere needs an address and a port. It cannot be 192.168.1.104:58080, because systemd is already holding that, if you point the proxy at its own socket it connects to itself in an infinite loop and dies spectacularly (more on that in troubleshooting; I did exactly this). The usual fix is to pick a second, different port number for the middle hop. But then every tunnel needs two numbers you have to keep in sync, which is one more thing to get wrong. Nicer trick: on Linux, the entire 127.0.0.0/8 range is loopback, not just 127.0.0.1. So 127.0.0.2:58080 is a completely different socket from 192.168.1.104:58080, even though the port number is identical. Use 127.0.0.2 for the middle hop and one port number per tunnel is all you ever need. It's still loopback, so nothing outside the gateway can reach it. Step 1, SSH must work non-interactively, as root This is the step people skip and then spend an hour debugging. systemd services run with no terminal. If SSH asks anything, a passphrase, a host-key confirmation, there is nobody to answer and the unit just fails. So: root needs a key with no passphrase (or an agent, but let's keep it simple), and the server's host key must already be in known_hosts. Create /root/.ssh/config: Host myserver HostName myserver.example.com Port 22 User youruser IdentityFile /root/.ssh/id_rsa IdentitiesOnly yes StrictHostKeyChecking yes UserKnownHostsFile /root/.ssh/known_hosts Enter fullscreen mode Exit fullscreen mode sudo chmod 600 /root/.ssh/config Enter fullscreen mode Exit fullscreen mode Host myserver defines a nickname. From now on ssh myserver means all of the above. That's why the unit files below say myserver and not myserver.example.com, the real hostname lives in this one file. Now prove it works with no human present: sudo HOME=/root ssh -F /root/.ssh/config -o BatchMode=yes -o StrictHostKeyChecking=accept-new myserver echo ok Enter fullscreen mode Exit fullscreen mode BatchMode=yes disables every interactive prompt, so this command cannot succeed by accidentally prompting you. It must print ok and nothing else. The accept-new on this first run records the host key in /root/.ssh/known_hosts; afterwards you can drop it. If it asks for a passphrase or a password, stop here and fix that first, nothing downstream will work. Step 2, the SSH tunnel unit /etc/systemd/system/myserver-ssh.service: [Unit] Description=SSH tunnels to myserver.example.com StopWhenUnneeded=yes Wants=network-online.target After=network-online.target [Service] Type=forking User=root Environment=HOME=/root ExecStart=/usr/bin/ssh -F /root/.ssh/config -f -N -o ExitOnForwardFailure=yes -o ServerAliveInterval=30 -o ServerAliveCountMax=3 -L 127.0.0.2:58080:localhost:8080 myserver Restart=no Enter fullscreen mode Exit fullscreen mode Line by line, the parts that matter: StopWhenUnneeded=yes, the whole point. This unit stops automatically as soon as nothing else depends on it. When the proxy exits after its idle timeout, this goes down with it. You never stop it by hand. Type=forking, pairs with ssh -f. SSH backgrounds itself only after the forward is established, so systemd knows the tunnel is genuinely ready before it starts the proxy. Ordering for free. Environment=HOME=/root, services get a bare-bones environment. Without this, SSH doesn't know where ~/.ssh is. -o ExitOnForwardFailure=yes, if the port can't be bound, fail loudly instead of silently connecting with no forward. -o ServerAliveInterval=30 -o ServerAliveCountMax=3, notice a dead connection within ~90 seconds instead of hanging forever. Important for a tunnel over the open internet. -L 127.0.0.2:58080:localhost:8080, the forward itself. Local address 127.0.0.2, local port 58080, and localhost:8080 interpreted on the server. This is the only file that mentions the remote port. Remember that for the "adding a tunnel" section. Step 3, the socket /etc/systemd/system/myserver-tunnel@.socket: [Unit] Description=Listening socket for tunnel on LAN port %i [Socket] ListenStream=192.168.1.104:%i FreeBind=yes [Install] WantedBy=sockets.target Enter fullscreen mode Exit fullscreen mode ListenStream=192.168.1.104:%i, the address other machines connect to. Binding the explicit LAN IP rather than 0.0.0.0 keeps it off any other interface. If you only want it reachable from the gateway itself, use 127.0.0.1:%i. FreeBind=yes, allows binding the address at boot even before the network interface has finished coming up. Without it, a reboot can leave the socket failed. WantedBy=sockets.target, makes it start at boot when enabled. Step 4, the proxy /etc/systemd/system/myserver-tunnel@.service: [Unit] Description=On-demand proxy for tunnel on LAN port %i Requires=myserver-ssh.service After=myserver-ssh.service [Service] Type=notify User=root LimitNOFILE=4096 ExecStart=/usr/lib/systemd/systemd-socket-proxyd --exit-idle-time=10min --connections-max=64 127.0.0.2:%i Enter fullscreen mode Exit fullscreen mode The name matters: a socket unit automatically starts the service with the same name. myserver-tunnel@58080.socket starts myserver-tunnel@58080.service. Don't rename one without the other. Requires= / After=myserver-ssh.service, starting the proxy pulls up the SSH tunnel first. Combined with StopWhenUnneeded in that unit, this is the entire lifecycle management. There is no script anywhere. --exit-idle-time=10min, the auto-close. Requires systemd ≥ 246; check with systemctl --version. --connections-max=64 and LimitNOFILE=4096, guard rails. If you ever misconfigure the target address, these turn a runaway into a bounded error. The argument 127.0.0.2:%i is the destination, i.e. where the SSH tunnel is listening. On some distributions the binary is at /lib/systemd/systemd-socket-proxyd. Check and adjust: ls -l /usr/lib/systemd/systemd-socket-proxyd /lib/systemd/systemd-socket-proxyd Enter fullscreen mode Exit fullscreen mode Step 5, turn it on sudo systemctl daemon-reload sudo systemctl enable --now myserver-tunnel@58080.socket Enter fullscreen mode Exit fullscreen mode Note we enable the socket, never the service. The service is started for us. Verify in two stages. First the SSH leg on its own, which tells you whether the problem (if any) is SSH or systemd: sudo systemctl start myserver-ssh.service sudo ss -ltnp | grep 58080 curl -sS -o /dev/null -w '%{http_code}\n' http://127.0.0.2:58080/ sudo systemctl stop myserver-ssh.service Enter fullscreen mode Exit fullscreen mode Then the whole chain: curl -sS -o /dev/null -w '%{http_code}\n' http://192.168.1.104:58080/ sudo ss -ltnp | grep 58080 Enter fullscreen mode Exit fullscreen mode You should see two listeners with the same port number: LISTEN 192.168.1.104:58080 users:(("systemd",...)) LISTEN 127.0.0.2:58080 users:(("ssh",...)) Enter fullscreen mode Exit fullscreen mode Different addresses, same port, that's correct, and that's the 127.0.0.2 trick doing its job. Now open http://192.168.1.104:58080 from any machine on the LAN. Watch it work: journalctl -fu myserver-ssh.service -u 'myserver-tunnel@*' Enter fullscreen mode Exit fullscreen mode Close the browser tab, wait ten minutes, and systemctl status myserver-ssh.service shows inactive (dead). The port is still open. Reload the page and it all comes back in about a second. Adding another port forwarding Say you now also want 192.168.1.104:11000 → the server's localhost:10000. 1. Add one -L to the SSH unit. Edit ExecStart in /etc/systemd/system/myserver-ssh.service and append: -L 127.0.0.2:11000:localhost:10000 Enter fullscreen mode Exit fullscreen mode so it reads: ExecStart=/usr/bin/ssh -F /root/.ssh/config -f -N -o ExitOnForwardFailure=yes -o ServerAliveInterval=30 -o ServerAliveCountMax=3 -L 127.0.0.2:58080:localhost:8080 -L 127.0.0.2:11000:localhost:10000 myserver Enter fullscreen mode Exit fullscreen mode The pattern is always -L 127.0.0.2::localhost:. Keeping the middle number equal to the LAN port is what lets the templates work untouched. 2. Instantiate the template for the new port: sudo systemctl daemon-reload sudo systemctl restart myserver-ssh.service sudo systemctl enable --now myserver-tunnel@11000.socket Enter fullscreen mode Exit fullscreen mode 3. Open the port in the gateway's firewall, if you run one: sudo ufw allow from 192.168.1.0/24 to any port 11000 proto tcp Enter fullscreen mode Exit fullscreen mode That's it. No new files, ever. All tunnels share one SSH connection and one authentication, nice for latency and for the server's logs. The trade-off: StopWhenUnneeded now tears down that shared connection ten minutes after the last tunnel goes idle, so a page left open on 58080 keeps the 11000 forward alive too. If you need strictly independent lifetimes, make a second copy of the SSH unit with its own -L and point the second proxy at it. To remove a tunnel, reverse it: sudo systemctl disable --now myserver-tunnel@11000.socket, delete the -L, reload, restart. Troubleshooting Failed to allocate pipe buffer: Too many open files followed by Failed to connect to remote host: Cannot assign requested address. This is the mistake I made, and it's worth knowing because the error messages point nowhere near the cause. It means the proxy's destination is the same address:port as its own listening socket. Every incoming connection makes the proxy dial itself, which arrives as another incoming connection, and so on until it runs out of file descriptors and then out of ephemeral ports. Check that ExecStart in the @.service says 127.0.0.2:%i and not 192.168.1.104:%i. The socket file holds the address people connect to; the service file holds the address the proxy connects to; they must never be equal. The unit fails immediately and the journal shows nothing useful. Almost always SSH asking for something. Re-run the BatchMode=yes test from step 1. administratively prohibited: open failed. The server has AllowTcpForwarding no, or a PermitOpen that doesn't cover your destination. See the prerequisite check in Idea 1, this one can only be fixed on the server. Host check error instead of the web UI. Some apps validate the HTTP Host header. Syncthing, for example, refuses requests whose Host isn't localhost when its GUI is bound to loopback, so it works via curl http://127.0.0.2:58080 but not from a browser pointed at 192.168.1.104:58080. Fix it in the app's config (for Syncthing: true inside ), not in the tunnel. Nothing reachable from other machines. Check the firewall on the gateway, and check ss -ltnp really shows 192.168.1.104 and not 127.0.0.1. Editing a unit seems to change nothing. daemon-reload only reloads definitions; running processes keep their old configuration. Restart the unit. "Idle" is not what you think. --exit-idle-time counts open connections, not traffic. An app that holds a long-poll or WebSocket open (Syncthing's event stream does this) keeps the tunnel alive as long as the browser tab is open. The countdown starts when the last connection closes. In practice this is the behaviour you want; just don't expect the tunnel to drop while a tab is sitting there. A word on security The tunnel adds no authentication of its own. Once port 58080 is open on the LAN, anything on the LAN can reach the remote app through it, with no password prompt from SSH, because the gateway holds the key. So: make sure the app behind it has its own authentication, and think about whether you really want ListenStream on the LAN address rather than 127.0.0.1. The gateway's private key is now a credential that reaches your server, so treat that machine accordingly. On the plus side, nothing about the server changed. Port 22 is still the only thing exposed, still key-only. The app never gets a public port, and the connection to it exists only in the minutes you're actually using it. Summary Three small files, no code: a .socket that holds the port open, a .service running systemd-socket-proxyd --exit-idle-time=10min, a .service running ssh -L, marked StopWhenUnneeded=yes. systemd handles start-on-demand, ordering, and teardown. Adding a tunnel is one -L flag and one systemctl enable. And the SSH connection to your server exists exactly when you're using it, and not one minute longer. Disclaimer: The text above has been written with the help of AI. All the ideas, checks and minimalistic approch are mine.