{"id":332,"date":"2016-04-11T20:10:23","date_gmt":"2016-04-12T03:10:23","guid":{"rendered":"http:\/\/www.mjblythe.com\/hacks\/?p=332"},"modified":"2021-12-22T00:01:01","modified_gmt":"2021-12-22T07:01:01","slug":"better-sslh","status":"publish","type":"post","link":"http:\/\/www.mjblythe.com\/hacks\/2016\/04\/better-sslh\/","title":{"rendered":"Better sslh"},"content":{"rendered":"

For those that don’t know, sslh<\/a> is a TCP port multiplexer. This basically means that you can serve both https<\/code> and ssh<\/code> traffic from the same port. It’s most useful for circumventing corporate firewalls that block TCP port 22 (i.e. ssh<\/code>), but allow TCP port 443 (i.e. https<\/code>) by serving both on TCP port 443.<\/p>\n

In the default configuration, however, all connections that go through sslh<\/code> look to ssh<\/code> or apache<\/code> as if they came from localhost<\/code>. This isn’t ideal if you want to run something like denyhosts<\/code> or fail2ban<\/code> to block malicious ssh<\/code> login attempts.<\/p>\n

sslh<\/code> does have an option to do “transparent” proxying so ssh<\/code> and apache<\/code> think that the connections have come from the right place. In this post, I’ll describe how I set this up on my machine.<\/p>\n

<\/p>\n\n

Initial Setup<\/h2>\n

In Ubuntu, sslh<\/code> can be installed by running this:<\/p>\n

> sudo apt-get install sslh<\/code><\/p><\/blockquote>\n

When installed, the default options (in \/etc\/default\/sslh<\/code>) look like this:<\/p>\n

DAEMON_OPTS=\"--user sslh --listen <change-me>:443 --ssh 127.0.0.1:22 --ssl 127.0.0.1:443 --pidfile \/var\/run\/sslh\/sslh.pid\"<\/code><\/p><\/blockquote>\n

To get it running at all, you need to switch apache<\/code> (or whatever web server you’re using) to serve https on a different port. I’ve selected 8443. Once this change is made (in \/etc\/apache2\/ports.conf<\/code> and \/etc\/apache2\/sites-enabled\/default-ssl.conf<\/code> for me), then you’ll need to update the sslh config too.<\/p>\n

sslh<\/code> config (\/etc\/default\/sslh<\/code>) without transparent proxying:<\/p>\n

DAEMON=\/usr\/sbin\/sslh
\nDAEMON_OPTS=\"--user sslh --listen 0.0.0.0:443 --ssh 127.0.0.1:22 --ssl 127.0.0.1:8443 --pidfile \/var\/run\/sslh\/sslh.pid\"<\/code><\/p><\/blockquote>\n

However, all https<\/code> and ssh<\/code> traffic (on port 443) looks like it comes from localhost<\/code>, since we haven’t enabled transparent proxying yet.<\/p>\n

Getting Transparent Proxying working<\/h2>\n

As will become obvious, I had some difficulty getting transparent proxying working. This section consists mostly of the notes I took while trying various things, and steps I took to debug why things aren’t working. If you’re only interested in copying my working configuration, skip to the next section<\/a>.<\/p>\n

First step, is to have ssh listen on another port. From the github page<\/a>:<\/p>\n

Note that these rules will prevent from connecting directly to ssh on the port 22, as packets coming out of sshd will be tagged. If you need to retain direct access to ssh on port 22 as well as through sslh, you can make sshd listen to 22 AND another port (e.g. 2222), and change the above rules accordingly.<\/p><\/blockquote>\n

\/etc\/ssh\/sshd_config<\/code> before:<\/p>\n

Port 22<\/code><\/p><\/blockquote>\n

after:<\/p>\n

Port 22
\nPort 2222<\/code><\/p><\/blockquote>\n

And corresponding change to \/etc\/default\/sslh<\/code>:<\/p>\n

DAEMON_OPTS=\"--user sslh --listen 0.0.0.0:443 --ssh 127.0.0.1:2222 --ssl 127.0.0.1:8443 --pidfile \/var\/run\/sslh\/sslh.pid\"<\/code><\/p><\/blockquote>\n

I tested port 22 from my offsite webhosting server, and it still works.\u00a0 I couldn’t test port 2222 (because didn’t set up port forwarding rules), but I confirmed that sslh is using port 2222:<\/p>\n

> pgrep sslh |xargs ps
\nPID TTY\u00a0\u00a0\u00a0\u00a0\u00a0 STAT\u00a0\u00a0 TIME COMMAND
\n20397 ?\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 Ss\u00a0\u00a0\u00a0\u00a0 0:00 \/usr\/sbin\/sslh --foreground --user sslh --listen 0.0.0.0 443 --ssh 127.0.0.1 2222 --ssl 12
\n20398 ?\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 S\u00a0\u00a0\u00a0\u00a0\u00a0 0:00 \/usr\/sbin\/sslh --foreground --user sslh --listen 0.0.0.0 443 --ssh 127.0.0.1 2222 --ssl 12<\/code><\/p><\/blockquote>\n

Next change, don’t use the loopback address (my machine’s hostname is “bruce”…I use characters from Finding Nemo<\/a> for my machines):<\/p>\n

DAEMON_OPTS=\"--user sslh --listen 0.0.0.0:443 --ssh bruce:2222 --ssl bruce:8443 --pidfile \/var\/run\/sslh\/sslh.pid\"<\/code><\/p><\/blockquote>\n

…still works…<\/p>\n

ok, moment of truth…let’s add the --transparent<\/code> switch and set up the iptables<\/code> rules.<\/p>\n

# iptables -t mangle -N SSLH
\n# iptables -t mangle -A OUTPUT --protocol tcp --out-interface eth0 --sport 2222 --jump SSLH
\n# iptables -t mangle -A OUTPUT --protocol tcp --out-interface eth0 --sport 8443 --jump SSLH
\n# iptables -t mangle -A SSLH --jump MARK --set-mark 0x1
\n# iptables -t mangle -A SSLH --jump ACCEPT
\n# ip rule add fwmark 0x1 lookup 100
\n# ip route add local 0.0.0.0\/0 dev lo table 100<\/code><\/p><\/blockquote>\n

…and it doesn’t work.<\/p>\n

Maybe it’s a capabilities issue?<\/p>\n

> getcap \/usr\/sbin\/sslh<\/code><\/p><\/blockquote>\n

…strong possibility<\/p>\n

> sudo setcap cap_net_admin+ep \/usr\/sbin\/sslh
\n> getcap \/usr\/sbin\/sslh
\n\/usr\/sbin\/sslh = cap_net_admin+ep<\/code><\/p><\/blockquote>\n

…still doesn’t work.<\/p>\n

Turns out that “bruce” doesn’t resolve the way I expect…need “bruce.” like this:<\/p>\n

DAEMON_OPTS=\"--transparent --user sslh --listen 0.0.0.0:443 --ssh bruce.:2222 --ssl bruce.:8443 --pidfile \/var\/run\/sslh\/sslh.pid\"<\/code><\/p><\/blockquote>\n

works now!<\/p>\n

Fails after a reboot…I guess iptables<\/code> loses the settings?\u00a0 Works again after re-running the iptables<\/code> commands and restarting sslh<\/code> and apache2<\/code><\/p>\n

Hmmm…getting this on reboot (from \/var\/log\/syslog<\/code>):<\/p>\n

Sep 3 18:02:00 bruce sslh[1050]: Temporary failure in name resolution `bruce.:2222′
\nSep 3 18:02:00 bruce sslh[1050]: Temporary failure in name resolution `bruce.’<\/p><\/blockquote>\n

Switching “bruce.” to “127.0.0.1” makes sslh not work for some reason, so I guess this just has to start after some other service.<\/p>\n

Here’s<\/a> how we’ll do it, I think.<\/p>\n

The file lives here: lib\/systemd\/system\/sslh.service<\/code><\/p>\n

According to this<\/a>, it sounds like we might want After=network-online.target<\/code>?<\/p>\n

Yes, making that change seems to fix it!<\/p>\n

Final working configuration<\/h2>\n

First, let’s recap what we’ve done up to this point:<\/p>\n

    \n
  1. We’ve set Apache\/nginx\/whatever to listen on port 8443 for https<\/li>\n
  2. sslh is installed<\/li>\n<\/ol>\n

    Next, change ssh<\/code> to listen on port 22 and 2222 by editing \/etc\/ssh\/sshd_config<\/code>:<\/p>\n

    Port 22
    \nPort 2222<\/code><\/p><\/blockquote>\n

    Run sudo service ssh restart<\/code> to make that change take effect.<\/p>\n

    Modify the command line options in \/etc\/default\/sslh<\/code> (substituting your hostname instead of bruce<\/code>):<\/p>\n

    DAEMON_OPTS=\"--transparent --user sslh --listen 0.0.0.0:443 --ssh bruce.:2222 --ssl bruce.:8443 --pidfile \/var\/run\/sslh\/sslh.pid\"<\/code><\/p><\/blockquote>\n

    Also, since that command wants to resolve the “bruce.” domain name, we need to wait until the network comes online, not just after the networking stack comes up. So, modify this line in lib\/systemd\/system\/sslh.service<\/code>:<\/p>\n

    After=network-online.target<\/code><\/p><\/blockquote>\n

    Finally, the sslh<\/code> executable needs some additional permissions, so add those:<\/p>\n

    > sudo setcap cap_net_admin+ep \/usr\/sbin\/sslh<\/code><\/p><\/blockquote>\n

    Put the iptables<\/code> rules in place (as root):<\/p>\n

    # iptables -t mangle -N SSLH
    \n# iptables -t mangle -A OUTPUT --protocol tcp --out-interface eth0 --sport 2222 --jump SSLH
    \n# iptables -t mangle -A OUTPUT --protocol tcp --out-interface eth0 --sport 8443 --jump SSLH
    \n# iptables -t mangle -A SSLH --jump MARK --set-mark 0x1
    \n# iptables -t mangle -A SSLH --jump ACCEPT
    \n<\/code><\/p><\/blockquote>\n

    Install software to load our firewall rules at boot-up<\/a>:<\/p>\n

    sudo apt-get install iptables-persistent<\/code><\/p><\/blockquote>\n

    At installation time, it prompts to save current iptables<\/code> rules.<\/p>\n

    Alternatively, if iptables-persistent<\/code> is already installed, run this as root:<\/p>\n

    # iptables-save > \/etc\/iptables\/rules.v4<\/code><\/p><\/blockquote>\n

    If you’re using IPv6, you’ll also need to repeat these steps with ip6tables<\/code>.<\/p>\n

    Get the ip<\/code> commands to run at boot-up…(similar to this<\/a>)<\/p>\n

    I created the following file as \/etc\/network\/if-up.d\/sslh<\/code>:<\/p>\n

    #!\/bin\/sh
    \nip rule add fwmark 0x1 lookup 100
    \nip route add local 0.0.0.0\/0 dev lo table 100<\/code><\/p><\/blockquote>\n

    It should be owned by root and be executable.<\/p>\n

    Set sslh<\/code> to start by default:<\/p>\n

    > sudo systemctl enable sslh<\/code><\/p><\/blockquote>\n

    Ok, that should be all the configuration! Reboot and verify:<\/p>\n

    > sudo iptables-save
    \n# Generated by iptables-save v1.4.21 on Sat Sep 3 17:44:08 2016
    \n*mangle
    \n:PREROUTING ACCEPT [203520132:32153396803]
    \n:INPUT ACCEPT [203513474:32153036883]
    \n:FORWARD ACCEPT [0:0]
    \n:OUTPUT ACCEPT [309094595:519342799063]
    \n:POSTROUTING ACCEPT [310985125:519865479412]
    \n:SSLH - [0:0]
    \n-A OUTPUT -o eth0 -p tcp -m tcp --sport 2222 -j SSLH
    \n-A OUTPUT -o eth0 -p tcp -m tcp --sport 8443 -j SSLH
    \n-A SSLH -j MARK --set-xmark 0x1\/0xffffffff
    \n-A SSLH -j ACCEPT
    \nCOMMIT
    \n# Completed on Sat Sep 3 17:44:08 2016
    \n> ip rule list
    \n0: from all lookup local
    \n32765: from all fwmark 0x1 lookup 100
    \n32766: from all lookup main
    \n32767: from all lookup default<\/code><\/p><\/blockquote>\n

    I’m not sure how to check the ip route<\/code> command, but if that fwmark<\/code> rule is there, then we can be confident that the ip route<\/code> command ran as well.<\/p>\n

    Note:On my system, it looks like the ip rule<\/code> command is run several times during boot-up (I guess I have several if-up events?) This doesn’t seem to have any negative impact.<\/p>\n

    \n

    Update Feb 6 2018:<\/p>\n

    I’ve tried several times to tweak things so that it all works properly on reboot, but it seem that I never was able to get sslh to start after any domain resolution stuff, so bruce<\/code> still resolves to 127.0.0.1<\/code> or something.\u00a0 I always have to run service sslh restart<\/code> after reboot to get things working.<\/p>\n

    Also, now that I’m using haproxy running on a different host (FIXME add link) to do this ssl\/ssh multiplexing, I don’t actually need this anymore.\u00a0 So I’m removing it.<\/p>\n

    \n

     <\/p>\n","protected":false},"excerpt":{"rendered":"

    For those that don’t know, sslh is a TCP port multiplexer. This basically means that you can serve both https and ssh traffic from the same port. It’s most useful for circumventing corporate firewalls that block TCP port 22 (i.e. ssh), but allow TCP port 443 (i.e. https) by serving both on TCP port 443. […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"footnotes":""},"categories":[7,11,15],"tags":[],"class_list":["post-332","post","type-post","status-publish","format-standard","hentry","category-howto","category-linux","category-trial-and-error"],"_links":{"self":[{"href":"http:\/\/www.mjblythe.com\/hacks\/wp-json\/wp\/v2\/posts\/332","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/www.mjblythe.com\/hacks\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/www.mjblythe.com\/hacks\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/www.mjblythe.com\/hacks\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/www.mjblythe.com\/hacks\/wp-json\/wp\/v2\/comments?post=332"}],"version-history":[{"count":23,"href":"http:\/\/www.mjblythe.com\/hacks\/wp-json\/wp\/v2\/posts\/332\/revisions"}],"predecessor-version":[{"id":835,"href":"http:\/\/www.mjblythe.com\/hacks\/wp-json\/wp\/v2\/posts\/332\/revisions\/835"}],"wp:attachment":[{"href":"http:\/\/www.mjblythe.com\/hacks\/wp-json\/wp\/v2\/media?parent=332"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/www.mjblythe.com\/hacks\/wp-json\/wp\/v2\/categories?post=332"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/www.mjblythe.com\/hacks\/wp-json\/wp\/v2\/tags?post=332"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}