For those that don’t know, sslh is a TCP port multiplexer. This basically means that you can serve both https and ssh traffic from the same port. It’s most useful for circumventing corporate firewalls that block TCP port 22 (i.e. ssh), but allow TCP port 443 (i.e. https) by serving both on TCP port 443.

In the default configuration, however, all connections that go through sslh look to ssh or apache as if they came from localhost. This isn’t ideal if you want to run something like denyhosts or fail2ban to block malicious ssh login attempts.

sslh does have an option to do “transparent” proxying so ssh and apache think that the connections have come from the right place. In this post, I’ll describe how I set this up on my machine.

Initial Setup

In Ubuntu, sslh can be installed by running this:

> sudo apt-get install sslh

When installed, the default options (in /etc/default/sslh) look like this:

DAEMON_OPTS="--user sslh --listen <change-me>:443 --ssh 127.0.0.1:22 --ssl 127.0.0.1:443 --pidfile /var/run/sslh/sslh.pid"

To get it running at all, you need to switch apache (or whatever web server you’re using) to serve https on a different port. I’ve selected 8443. Once this change is made (in /etc/apache2/ports.conf and /etc/apache2/sites-enabled/default-ssl.conf for me), then you’ll need to update the sslh config too.

sslh config (/etc/default/sslh) without transparent proxying:

DAEMON=/usr/sbin/sslh
DAEMON_OPTS="--user sslh --listen 0.0.0.0:443 --ssh 127.0.0.1:22 --ssl 127.0.0.1:8443 --pidfile /var/run/sslh/sslh.pid"

However, all https and ssh traffic (on port 443) looks like it comes from localhost, since we haven’t enabled transparent proxying yet.

Getting Transparent Proxying working

As will become obvious, I had some difficulty getting transparent proxying working. This section consists mostly of the notes I took while trying various things, and steps I took to debug why things aren’t working. If you’re only interested in copying my working configuration, skip to the next section.

First step, is to have ssh listen on another port. From the github page:

Note that these rules will prevent from connecting directly to ssh on the port 22, as packets coming out of sshd will be tagged. If you need to retain direct access to ssh on port 22 as well as through sslh, you can make sshd listen to 22 AND another port (e.g. 2222), and change the above rules accordingly.

/etc/ssh/sshd_config before:

Port 22

after:

Port 22
Port 2222

And corresponding change to /etc/default/sslh:

DAEMON_OPTS="--user sslh --listen 0.0.0.0:443 --ssh 127.0.0.1:2222 --ssl 127.0.0.1:8443 --pidfile /var/run/sslh/sslh.pid"

I tested port 22 from my offsite webhosting server, and it still works.  I couldn’t test port 2222 (because didn’t set up port forwarding rules), but I confirmed that sslh is using port 2222:

> pgrep sslh |xargs ps
PID TTY      STAT   TIME COMMAND
20397 ?        Ss     0:00 /usr/sbin/sslh --foreground --user sslh --listen 0.0.0.0 443 --ssh 127.0.0.1 2222 --ssl 12
20398 ?        S      0:00 /usr/sbin/sslh --foreground --user sslh --listen 0.0.0.0 443 --ssh 127.0.0.1 2222 --ssl 12

Next change, don’t use the loopback address (my machine’s hostname is “bruce”…I use characters from Finding Nemo for my machines):

DAEMON_OPTS="--user sslh --listen 0.0.0.0:443 --ssh bruce:2222 --ssl bruce:8443 --pidfile /var/run/sslh/sslh.pid"

…still works…

ok, moment of truth…let’s add the --transparent switch and set up the iptables rules.

# iptables -t mangle -N SSLH
# iptables -t mangle -A OUTPUT --protocol tcp --out-interface eth0 --sport 2222 --jump SSLH
# iptables -t mangle -A OUTPUT --protocol tcp --out-interface eth0 --sport 8443 --jump SSLH
# iptables -t mangle -A SSLH --jump MARK --set-mark 0x1
# iptables -t mangle -A SSLH --jump ACCEPT
# ip rule add fwmark 0x1 lookup 100
# ip route add local 0.0.0.0/0 dev lo table 100

…and it doesn’t work.

Maybe it’s a capabilities issue?

> getcap /usr/sbin/sslh

…strong possibility

> sudo setcap cap_net_admin+ep /usr/sbin/sslh
> getcap /usr/sbin/sslh
/usr/sbin/sslh = cap_net_admin+ep

…still doesn’t work.

Turns out that “bruce” doesn’t resolve the way I expect…need “bruce.” like this:

DAEMON_OPTS="--transparent --user sslh --listen 0.0.0.0:443 --ssh bruce.:2222 --ssl bruce.:8443 --pidfile /var/run/sslh/sslh.pid"

works now!

Fails after a reboot…I guess iptables loses the settings?  Works again after re-running the iptables commands and restarting sslh and apache2

Hmmm…getting this on reboot (from /var/log/syslog):

Sep 3 18:02:00 bruce sslh[1050]: Temporary failure in name resolution `bruce.:2222′
Sep 3 18:02:00 bruce sslh[1050]: Temporary failure in name resolution `bruce.’

Switching “bruce.” to “127.0.0.1” makes sslh not work for some reason, so I guess this just has to start after some other service.

Here’s how we’ll do it, I think.

The file lives here: lib/systemd/system/sslh.service

According to this, it sounds like we might want After=network-online.target?

Yes, making that change seems to fix it!

Final working configuration

First, let’s recap what we’ve done up to this point:

1. We’ve set Apache/nginx/whatever to listen on port 8443 for https
2. sslh is installed

Next, change ssh to listen on port 22 and 2222 by editing /etc/ssh/sshd_config:

Port 22
Port 2222

Run sudo service ssh restart to make that change take effect.

Modify the command line options in /etc/default/sslh (substituting your hostname instead of bruce):

DAEMON_OPTS="--transparent --user sslh --listen 0.0.0.0:443 --ssh bruce.:2222 --ssl bruce.:8443 --pidfile /var/run/sslh/sslh.pid"

Also, since that command wants to resolve the “bruce.” domain name, we need to wait until the network comes online, not just after the networking stack comes up. So, modify this line in lib/systemd/system/sslh.service:

After=network-online.target

Finally, the sslh executable needs some additional permissions, so add those:

> sudo setcap cap_net_admin+ep /usr/sbin/sslh

Put the iptables rules in place (as root):

# iptables -t mangle -N SSLH
# iptables -t mangle -A OUTPUT --protocol tcp --out-interface eth0 --sport 2222 --jump SSLH
# iptables -t mangle -A OUTPUT --protocol tcp --out-interface eth0 --sport 8443 --jump SSLH
# iptables -t mangle -A SSLH --jump MARK --set-mark 0x1
# iptables -t mangle -A SSLH --jump ACCEPT


Install software to load our firewall rules at boot-up:

sudo apt-get install iptables-persistent

At installation time, it prompts to save current iptables rules.

Alternatively, if iptables-persistent is already installed, run this as root:

# iptables-save > /etc/iptables/rules.v4

If you’re using IPv6, you’ll also need to repeat these steps with ip6tables.

Get the ip commands to run at boot-up…(similar to this)

I created the following file as /etc/network/if-up.d/sslh:

#!/bin/sh
ip rule add fwmark 0x1 lookup 100
ip route add local 0.0.0.0/0 dev lo table 100

It should be owned by root and be executable.

Set sslh to start by default:

> sudo systemctl enable sslh

Ok, that should be all the configuration! Reboot and verify:

> sudo iptables-save
# Generated by iptables-save v1.4.21 on Sat Sep 3 17:44:08 2016
*mangle
:PREROUTING ACCEPT [203520132:32153396803]
:INPUT ACCEPT [203513474:32153036883]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [309094595:519342799063]
:POSTROUTING ACCEPT [310985125:519865479412]
:SSLH - [0:0]
-A OUTPUT -o eth0 -p tcp -m tcp --sport 2222 -j SSLH
-A OUTPUT -o eth0 -p tcp -m tcp --sport 8443 -j SSLH
-A SSLH -j MARK --set-xmark 0x1/0xffffffff
-A SSLH -j ACCEPT
COMMIT
# Completed on Sat Sep 3 17:44:08 2016
> ip rule list
0: from all lookup local
32765: from all fwmark 0x1 lookup 100
32766: from all lookup main
32767: from all lookup default

I’m not sure how to check the ip route command, but if that fwmark rule is there, then we can be confident that the ip route command ran as well.

Note:On my system, it looks like the ip rule command is run several times during boot-up (I guess I have several if-up events?) This doesn’t seem to have any negative impact.

---

Update Feb 6 2018:

I’ve tried several times to tweak things so that it all works properly on reboot, but it seem that I never was able to get sslh to start after any domain resolution stuff, so bruce still resolves to 127.0.0.1 or something.  I always have to run service sslh restart after reboot to get things working.

Also, now that I’m using haproxy running on a different host (FIXME add link) to do this ssl/ssh multiplexing, I don’t actually need this anymore.  So I’m removing it.

---

 

---

---

---

This entry was posted on Monday, April 11th, 2016 at 8:10 pm and is filed under How To, Linux, Trial and Error. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.