Canonical recently dropped official support for Ubuntu 15.10 “Wily Werewolf”, so I decided to upgrade. I also don’t like being stuck on Long-Term-Support releases, so I did 2 upgrades in sequence: 15.10 “Wily Werewolf” to 16.04 “Xenial Xerus” to 16.10 “Yakkety Yak”. In the past, I’ve just done a clean install, but the trouble with that is having to re-do all my various customizations. This time I figured I’d just try an in-place upgrade. Of course, this meant that my customizations conflicted with packages changes. I’m still not sure which method is better.<\/p>\n
<\/p>\n\n
\/etc\/phpmyadmin\/config.inc.php<\/code><\/h2>\nI had added this line:
\n$cfg['ForcsSSL'] = true;<\/code><\/p>\nWhen I added it back, I found that phpmyadmin<\/code> no longer did the https redirect. According to this<\/a>, the option has been removed, so I need to do it with .htaccess or something now.<\/p>\nTaking inspiration from here<\/a>, I ended up commenting out the “Alias” line in \/etc\/phpmyadmin\/apache.conf<\/code> and added it in my https vhost ( in \/etc\/apache2\/sites-enabled\/default-ssl.conf<\/code>)<\/p>\nAdditionally, phpmyadmin<\/code> was complaining about “blowfish secret too short”, so I used this trick<\/a> to generate more random characters to add to the secret:
\nbase64 \/dev\/urandom | head -c 100 >> \/var\/lib\/phpmyadmin\/blowfish_secret.inc.php<\/code><\/p>\nConflict in \/etc\/sudoers<\/code><\/h2>\nThe package maintainer added \/snap\/bin<\/code> to the secure_path. It was trivial to add that to my customized \/etc\/sudoers<\/code>.<\/p>\nMythTV and MySQL<\/h2>\n
MythTV was prompting me for the MySQL admin username & password…not sure why. Once I got MySQL running again, MythTV started up just fine…seems to be working OK.<\/p>\n
Conflict in \/etc\/mysql\/mysql.conf.d\/mysqld.cnf<\/code><\/h2>\nNeed to comment out bind-address = 127.0.0.1<\/code> (This allows MySQL to be available to remote machines for MythTV).<\/p>\nPrograms uninstalled during upgrade<\/h2>\n\n- digikam5\n
\n- haven’t decided whether I want to re-install this<\/li>\n<\/ul>\n<\/li>\n
- scribus-ng\n
\n- I can’t find anywhere that this exists anymore. Maybe the new version has finally been released to Ubuntu?<\/li>\n<\/ul>\n<\/li>\n
- php5-mysqlnd\n
\n- need native driver for a custom PHP application<\/li>\n
- non-issue<\/strong> – looks like mysqlnd is now distributed with the main mysql php plugin<\/li>\n<\/ul>\n<\/li>\n
- wine\n
\n- non-issue<\/strong> – the meta-package is just called wine1.6 now<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n
Unnecessary stuff that I can remove<\/h2>\n
When I originally brought up my system, I was having lots of stability problems. I installed kdump<\/code> and related utilities to try to debug the kernel panics I was getting. However, ever since I updated the BIOS (i.e. the one that actually added support for my CPU), things have been pretty stable, so I removed the kdump<\/code> stuff<\/p>\ncron errors<\/h2>\n
I got cron errors for a php5 session cleanup script. Just needed to “completely remove” all removed packages, and it solved the problem. (I did it in synaptic package manager…easy.)<\/p>\n
Postfix not delivering mail<\/h2>\n
Lots of errors in syslog from postfix…can’t resolve smtp.gmail.com. Turns out \/var\/spool\/postfix\/etc\/resolv.conf<\/code> was empty, so it didn’t know how to look up hostnames. Originally, I found this forum post<\/a>, so I “fixed” with this:
\ncp \/etc\/resolv.conf \/var\/spool\/postfix\/etc\/resolv.conf<\/code>
\nBut I had no idea why it broke in the first place, and when I rebooted, it was empty again.<\/p>\nThis seems to have been resolved at some point when I was messing with the systemd-networkd<\/code> and systemd-resolved<\/code> stuff (see below).<\/p>\nMore errors from a cron script<\/h2>\n
These errors were from a custom cron script I was running, and it was failing because the file started with #!\/usr\/bin\/php5<\/code>. php5<\/code> was removed and replaced with php7.0<\/code>, so it coudn’t find the interpreter to execute the script. I just changed it to #!\/usr\/bin\/php<\/code>, and it’s working OK.<\/p>\ngpg-agent<\/code> warning<\/h2>\nWhenever I opened a shell, I’d get a warning from gpg-agent<\/code> about --write-env-file<\/code>. This post<\/a> lead me to this “what’s new” page<\/a>.<\/p>\nI removed the following from my ~\/.bashrc<\/code>:<\/p>\n
\n# to use the gpg-agent:
\nenvfile=\"$HOME\/.gnupg\/gpg-agent.env\"
\nif [[ -e \"$envfile\" ]] && kill -0 $(grep GPG_AGENT_INFO \"$envfile\" | cut -d: -f 2) 2>\/dev\/null; then
\n eval \"$(cat \"$envfile\")\"
\nelse
\n eval \"$(gpg-agent --daemon --enable-ssh-support --write-env-file \"$envfile\")\"
\nfi
\nexport GPG_AGENT_INFO # the env file does not contain the export statement
\nexport SSH_AUTH_SOCK # enable gpg-agent for ssh
\n<\/code>
\nand replaced it with this:
\n
\nexport SSH_AUTH_SOCK=~\/.gnupg\/S.gpg-agent.ssh
\n<\/code><\/p>\nsslh<\/code> not working<\/h2>\nFor those who don’t know, sslh<\/code> is a TCP port multiplexer that allows you to serve both ssh<\/code> and https<\/code> from the same TCP port. This is useful for circumventing workplace firewall rules that block the default ssh<\/code> TCP port, 22, but allow https<\/code> traffic on TCP port 443. Setting up sslh really isn’t that hard, but I’m trying to use the transparent proxy<\/a> feature. If this isn’t used, then all traffic looks like it’s coming from localhost, not the appropriate remote IP. I have a separate post<\/a> from when I initially set this up<\/p>\nOk, this is where I spent most of my time, so I’ll break this into 2 sections, so folks can skip my debugging process.<\/p>\n
Debugging<\/h3>\n
sslh<\/code> is properly passing along ssh connections, but not https<\/code>. (Note: this is actually wrong. At this time, I was just connecting via port 22, not port 443.) Connecting from localhost<\/code> seems to work fine. When connecting from another host, it times out, and I see this in syslog:
\n
\nNov 19 22:45:17 bruce sslh[30040]: forward to ssl failed:connect: Connection timed out
\nNov 19 22:45:17 bruce sslh[30040]: connect: Connection timed out
\n<\/code>
\nTo me, this sounds like the sslh<\/code>-to-apache<\/code> connection isn’t happening for some reason.
\nI checked the output of ip rules list<\/code> and iptables-save<\/code>, and it looks OK.<\/p>\nAround this time, I realized that nmap<\/code> reported the following when run from a different machine on the same network:
\n
\n2222\/tcp filtered EtherNetIP-1
\n8443\/tcp filtered https-alt
\n<\/code><\/p>\nSo, I moved both apache<\/code> and ssh<\/code> to different ports, 9443 and 9022 respectively, and they both worked there. I updated the iptables<\/code> rules to use those ports, and bam! nmap<\/code> says filtered again! So, it has to have something to do with the iptables<\/code> rules.
\nBack to the previous ports (8443 and 2222), and I get this from nmap<\/code>:
\n
\n2222\/tcp open EtherNetIP-1
\n8443\/tcp open https-alt
\n<\/code>
\nperfect!
\nNow, what’s wrong with the iptables<\/code> rules?
\nTurns out that getcap \/usr\/sbin\/sslh<\/code> returned nothing…had to set the capabilities again.
\nDon’t think it’s fixed it, though.
\nok, so once again it’s a DNS name resolution problem. I don’t know what it’s resolving bruce.<\/code> to, but when I run this command line, it works as expected:
\nsudo \/usr\/sbin\/sslh --user sslh --transparent --listen 0.0.0.0:443 --ssh 192.168.1.102:9022 --ssl 192.168.1.102:9443 --pidfile \/var\/run\/sslh\/sslh.pid --foreground -v<\/code>
\nSo, let’s figure out what’s up with name resolution since that seems to be a wider issue.
\nLooks like other people<\/a> are having similar problems<\/a>.<\/p>\nI ran sudo dpkg-reconfigure resolvconf<\/code>…let’s reboot & see what happens.
\nNo change.
\nAha! This news<\/a> looks intriguing. In 16.10, Ubuntu has changed how they handle DNS resolving<\/a>.
\nI wonder if I’m still launching dnsmasq when I shouldn’t be? Would that cause problems?
\nIt’s possible that I need to change \/etc\/nsswitch.conf<\/code><\/a>, too.
\nTried commenting out dnsmasq<\/code> in NetworkManager<\/code>…no change.
\ntried uninstalling resolvconf<\/code>…no change.
\nAfter both of those, dig<\/code> doesn’t even work anymore! Great!<\/p>\nSolution<\/h3>\n
Ok, so here’s what I’ve learned. Ubuntu uses systemd-resolved<\/code> now for DNS stuff. However, if you want the DNS server from DHCP negotiation to be passed along to this service, you also need to use systemd-networkd<\/code> to configure your interfaces. This means no NetworkManager<\/code>, no resolvconf<\/code>, no dnsmasq<\/code>. Link<\/a> Link<\/a> Link<\/a><\/p>\nOk, so let’s switch to systemd-networkd<\/code>:
\n
\nsudo systemctl enable systemd-networkd
\nsudo systemctl disable NetworkManager
\nsudo systemctl stop NetworkManager
\nsudo systemctl start systemd-networkd
\n<\/code><\/p>\nI also need to create \/etc\/systemd\/network\/wired.network<\/code> so that systemd-networkd<\/code> knows what to do:
\n
\n[Match]
\nName=eth*
\n[Network]
\nDHCP=yes
\n<\/code><\/p>\nFinally, we need to set up this softlink so that systemd-resolved<\/code> controls which DNS servers are used:
\n
\nsudo rm \/etc\/resolv.conf
\nsudo ln -s \/run\/systemd\/resolve\/resolv.conf \/etc\/resolv.conf
\n<\/code><\/p>\nOk, so. For whatever reason, systemd-resolved<\/code> doesn’t properly resolve unqualified domains. I thought it might be because it was trying to do DNSSEC verification (which my router’s DNS server doesn’t support), but even when I allowed fallback, it still didn’t work. No idea what’s going on, but I have to assume it’s a bug in systemd-resolved<\/code>.<\/p>\nHaving said that, I can still use systemd-networkd<\/code> and systemd-resolved<\/code> to control \/etc\/resolv.conf<\/code> (i.e. have it point to my router’s DNS server). I just want things to skip trying to use systemd-resolved<\/code>, so that means removing resolve<\/code> from \/etc\/nsswitch.conf<\/code>, so everything always falls back to DNS, which properly resolves my unqualified domains names:<\/p>\nBefore:
\n
\nhosts: files mdns4_minimal [NOTFOUND=return] resolve [!UNAVAIL=return] dns
\n<\/code>
\nAfter:
\n
\nhosts: files mdns4_minimal [NOTFOUND=return] dns
\n<\/code><\/p>\nWhen nsswitch<\/code> falls back to DNS, it properly resolves bruce.<\/code> to its IP on my router’s subnet, not 127.0.0.1, so that’s good enough to fix this problem.<\/p>\nAnother alternative might be to have my router’s DNS server put all the names in a LAN domain, and configure systemd-resolved<\/code> to try to append that domain…but I’m fed up.<\/p>\nssh doesn’t resolve my local machine names<\/h2>\n
This was actually the same root cause as the sslh<\/code> issue above, so once I got that solution in place, it worked.<\/p>\nThird-party repos<\/h2>\n
During the upgrade, third-party apt repositories are disabled. This was just an exercise in replacing “wily” with “yakkety” in the repo configs, then running sudo apt-get update && sudo apt-get upgrade<\/code><\/p>\nMost of the repos support yakkety, but a couple (handbrake, mediaelch, dropbox) only had xenial.<\/p>\n
PHP short_open_tag<\/code><\/h2>\nphp7.0<\/code> is configured with this by default:
\nshort_open_tag = Off<\/code>
\nWith php5<\/code>, I had changed it to this (but php7.0<\/code> uses different config files, so there was no conflict at package install time):
\nshort_open_tag = On<\/code>
\nI modified both of these files to turn the option On<\/code>:
\n
\n\/etc\/php\/7.0\/apache2\/php.ini
\n\/etc\/php\/7.0\/cli\/php.ini
\n<\/code><\/p>\nMySQL connection errors<\/h2>\n
Some of my php scripts give this:
\nUnable to connect to database [No such file or directory]<\/code>
\nLooks like a MySQL problem…phpmyadmin won’t connect either.
\nI found this thread<\/a> and realized that the solution was trivial (and so dumb that the setting wouldn’t carry over from before):
\n
\n> sudo systemctl is-enabled mysql.service
\ndisabled
\n> sudo systemctl enable mysql.service
\nSynchronizing state of mysql.service with SysV service script with \/lib\/systemd\/systemd-sysv-install.
\nExecuting: \/lib\/systemd\/systemd-sysv-install enable mysql
\n> sudo systemctl is-enabled mysql.service
\nenabled
\n> sudo systemctl is-active mysql.service
\ninactive
\n> sudo systemctl start mysql.service
\n> sudo systemctl is-active mysql.service
\nactive
\n<\/code><\/p>\nPHP mysql<\/code> module deprecated<\/h2>\nSome of my other PHP sites were still broken…because they removed the mysql<\/code> component in php7.0 in favor of mysqli<\/code>. (Yes, I know I should have switched ages ago…) I found a converter script<\/a> that seems to have done a good job converting my code automatically.<\/p>\nMore PHP issues<\/h2>\n
Apparently, in a piece of code, I was using break<\/code> to try to jump to the end of an if<\/code> block. php7.0<\/code> gives an error for this. Since my code was just trying to handle an error case that I don’t think I’ve ever seen, I changed it to die<\/code>.<\/p>\nMore MySQL issues<\/h2>\n