Canonical recently dropped official support for Ubuntu 15.10 “Wily Werewolf”, so I decided to upgrade. I also don’t like being stuck on Long-Term-Support releases, so I did 2 upgrades in sequence: 15.10 “Wily Werewolf” to 16.04 “Xenial Xerus” to 16.10 “Yakkety Yak”. In the past, I’ve just done a clean install, but the trouble with that is having to re-do all my various customizations. This time I figured I’d just try an in-place upgrade. Of course, this meant that my customizations conflicted with packages changes. I’m still not sure which method is better.
Contents
Conflict in /etc/phpmyadmin/config.inc.php
I had added this line:
$cfg['ForcsSSL'] = true;
When I added it back, I found that phpmyadmin no longer did the https redirect. According to this, the option has been removed, so I need to do it with .htaccess or something now.
Taking inspiration from here, I ended up commenting out the “Alias” line in /etc/phpmyadmin/apache.conf and added it in my https vhost ( in /etc/apache2/sites-enabled/default-ssl.conf)
Additionally, phpmyadmin was complaining about “blowfish secret too short”, so I used this trick to generate more random characters to add to the secret:
base64 /dev/urandom | head -c 100 >> /var/lib/phpmyadmin/blowfish_secret.inc.php
Conflict in /etc/sudoers
The package maintainer added /snap/bin to the secure_path. It was trivial to add that to my customized /etc/sudoers.
MythTV and MySQL
MythTV was prompting me for the MySQL admin username & password…not sure why. Once I got MySQL running again, MythTV started up just fine…seems to be working OK.
Conflict in /etc/mysql/mysql.conf.d/mysqld.cnf
Need to comment out bind-address = 127.0.0.1 (This allows MySQL to be available to remote machines for MythTV).
Programs uninstalled during upgrade
- digikam5
- haven’t decided whether I want to re-install this
- scribus-ng
- I can’t find anywhere that this exists anymore. Maybe the new version has finally been released to Ubuntu?
- php5-mysqlnd
- need native driver for a custom PHP application
- non-issue – looks like mysqlnd is now distributed with the main mysql php plugin
- wine
- non-issue – the meta-package is just called wine1.6 now
Unnecessary stuff that I can remove
When I originally brought up my system, I was having lots of stability problems. I installed kdump and related utilities to try to debug the kernel panics I was getting. However, ever since I updated the BIOS (i.e. the one that actually added support for my CPU), things have been pretty stable, so I removed the kdump stuff
cron errors
I got cron errors for a php5 session cleanup script. Just needed to “completely remove” all removed packages, and it solved the problem. (I did it in synaptic package manager…easy.)
Postfix not delivering mail
Lots of errors in syslog from postfix…can’t resolve smtp.gmail.com. Turns out /var/spool/postfix/etc/resolv.conf was empty, so it didn’t know how to look up hostnames. Originally, I found this forum post, so I “fixed” with this:
cp /etc/resolv.conf /var/spool/postfix/etc/resolv.conf
But I had no idea why it broke in the first place, and when I rebooted, it was empty again.
This seems to have been resolved at some point when I was messing with the systemd-networkd and systemd-resolved stuff (see below).
More errors from a cron script
These errors were from a custom cron script I was running, and it was failing because the file started with #!/usr/bin/php5. php5 was removed and replaced with php7.0, so it coudn’t find the interpreter to execute the script. I just changed it to #!/usr/bin/php, and it’s working OK.
gpg-agent warning
Whenever I opened a shell, I’d get a warning from gpg-agent about --write-env-file. This post lead me to this “what’s new” page.
I removed the following from my ~/.bashrc:
# to use the gpg-agent:
envfile="$HOME/.gnupg/gpg-agent.env"
if [[ -e "$envfile" ]] && kill -0 $(grep GPG_AGENT_INFO "$envfile" | cut -d: -f 2) 2>/dev/null; then
eval "$(cat "$envfile")"
else
eval "$(gpg-agent --daemon --enable-ssh-support --write-env-file "$envfile")"
fi
export GPG_AGENT_INFO # the env file does not contain the export statement
export SSH_AUTH_SOCK # enable gpg-agent for ssh
and replaced it with this:
export SSH_AUTH_SOCK=~/.gnupg/S.gpg-agent.ssh
sslh not working
For those who don’t know, sslh is a TCP port multiplexer that allows you to serve both ssh and https from the same TCP port. This is useful for circumventing workplace firewall rules that block the default ssh TCP port, 22, but allow https traffic on TCP port 443. Setting up sslh really isn’t that hard, but I’m trying to use the transparent proxy feature. If this isn’t used, then all traffic looks like it’s coming from localhost, not the appropriate remote IP. I have a separate post from when I initially set this up
Ok, this is where I spent most of my time, so I’ll break this into 2 sections, so folks can skip my debugging process.
Debugging
sslh is properly passing along ssh connections, but not https. (Note: this is actually wrong. At this time, I was just connecting via port 22, not port 443.) Connecting from localhost seems to work fine. When connecting from another host, it times out, and I see this in syslog:
Nov 19 22:45:17 bruce sslh[30040]: forward to ssl failed:connect: Connection timed out
Nov 19 22:45:17 bruce sslh[30040]: connect: Connection timed out
To me, this sounds like the sslh-to-apache connection isn’t happening for some reason.
I checked the output of ip rules list and iptables-save, and it looks OK.
Around this time, I realized that nmap reported the following when run from a different machine on the same network:
2222/tcp filtered EtherNetIP-1
8443/tcp filtered https-alt
So, I moved both apache and ssh to different ports, 9443 and 9022 respectively, and they both worked there. I updated the iptables rules to use those ports, and bam! nmap says filtered again! So, it has to have something to do with the iptables rules.
Back to the previous ports (8443 and 2222), and I get this from nmap:
2222/tcp open EtherNetIP-1
8443/tcp open https-alt
perfect!
Now, what’s wrong with the iptables rules?
Turns out that getcap /usr/sbin/sslh returned nothing…had to set the capabilities again.
Don’t think it’s fixed it, though.
ok, so once again it’s a DNS name resolution problem. I don’t know what it’s resolving bruce. to, but when I run this command line, it works as expected:
sudo /usr/sbin/sslh --user sslh --transparent --listen 0.0.0.0:443 --ssh 192.168.1.102:9022 --ssl 192.168.1.102:9443 --pidfile /var/run/sslh/sslh.pid --foreground -v
So, let’s figure out what’s up with name resolution since that seems to be a wider issue.
Looks like other people are having similar problems.
I ran sudo dpkg-reconfigure resolvconf…let’s reboot & see what happens.
No change.
Aha! This news looks intriguing. In 16.10, Ubuntu has changed how they handle DNS resolving.
I wonder if I’m still launching dnsmasq when I shouldn’t be? Would that cause problems?
It’s possible that I need to change /etc/nsswitch.conf, too.
Tried commenting out dnsmasq in NetworkManager…no change.
tried uninstalling resolvconf…no change.
After both of those, dig doesn’t even work anymore! Great!
Solution
Ok, so here’s what I’ve learned. Ubuntu uses systemd-resolved now for DNS stuff. However, if you want the DNS server from DHCP negotiation to be passed along to this service, you also need to use systemd-networkd to configure your interfaces. This means no NetworkManager, no resolvconf, no dnsmasq. Link Link Link
Ok, so let’s switch to systemd-networkd:
sudo systemctl enable systemd-networkd
sudo systemctl disable NetworkManager
sudo systemctl stop NetworkManager
sudo systemctl start systemd-networkd
I also need to create /etc/systemd/network/wired.network so that systemd-networkd knows what to do:
[Match]
Name=eth*
[Network]
DHCP=yes
Finally, we need to set up this softlink so that systemd-resolved controls which DNS servers are used:
sudo rm /etc/resolv.conf
sudo ln -s /run/systemd/resolve/resolv.conf /etc/resolv.conf
Ok, so. For whatever reason, systemd-resolved doesn’t properly resolve unqualified domains. I thought it might be because it was trying to do DNSSEC verification (which my router’s DNS server doesn’t support), but even when I allowed fallback, it still didn’t work. No idea what’s going on, but I have to assume it’s a bug in systemd-resolved.
Having said that, I can still use systemd-networkd and systemd-resolved to control /etc/resolv.conf (i.e. have it point to my router’s DNS server). I just want things to skip trying to use systemd-resolved, so that means removing resolve from /etc/nsswitch.conf, so everything always falls back to DNS, which properly resolves my unqualified domains names:
Before:
hosts: files mdns4_minimal [NOTFOUND=return] resolve [!UNAVAIL=return] dns
After:
hosts: files mdns4_minimal [NOTFOUND=return] dns
When nsswitch falls back to DNS, it properly resolves bruce. to its IP on my router’s subnet, not 127.0.0.1, so that’s good enough to fix this problem.
Another alternative might be to have my router’s DNS server put all the names in a LAN domain, and configure systemd-resolved to try to append that domain…but I’m fed up.
ssh doesn’t resolve my local machine names
This was actually the same root cause as the sslh issue above, so once I got that solution in place, it worked.
Third-party repos
During the upgrade, third-party apt repositories are disabled. This was just an exercise in replacing “wily” with “yakkety” in the repo configs, then running sudo apt-get update && sudo apt-get upgrade
Most of the repos support yakkety, but a couple (handbrake, mediaelch, dropbox) only had xenial.
PHP short_open_tag
php7.0 is configured with this by default:
short_open_tag = Off
With php5, I had changed it to this (but php7.0 uses different config files, so there was no conflict at package install time):
short_open_tag = On
I modified both of these files to turn the option On:
/etc/php/7.0/apache2/php.ini
/etc/php/7.0/cli/php.ini
MySQL connection errors
Some of my php scripts give this:
Unable to connect to database [No such file or directory]
Looks like a MySQL problem…phpmyadmin won’t connect either.
I found this thread and realized that the solution was trivial (and so dumb that the setting wouldn’t carry over from before):
> sudo systemctl is-enabled mysql.service
disabled
> sudo systemctl enable mysql.service
Synchronizing state of mysql.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable mysql
> sudo systemctl is-enabled mysql.service
enabled
> sudo systemctl is-active mysql.service
inactive
> sudo systemctl start mysql.service
> sudo systemctl is-active mysql.service
active
PHP mysql module deprecated
Some of my other PHP sites were still broken…because they removed the mysql component in php7.0 in favor of mysqli. (Yes, I know I should have switched ages ago…) I found a converter script that seems to have done a good job converting my code automatically.
More PHP issues
Apparently, in a piece of code, I was using break to try to jump to the end of an if block. php7.0 gives an error for this. Since my code was just trying to handle an error case that I don’t think I’ve ever seen, I changed it to die.
More MySQL issues
MySQL now sets the ONLY_FULL_GROUP_BY option, which broke some of my queries. Basically, I had a query where I knew that all of the values for a particular column would be the same within a group. Older versions of MySQL would let you put that column in the SELECT list, and it would just arbitrarily pick one of the values. This is no longer allowed. Again, since I know all the values are the same, I just substituted SELECT column_name with SELECT MAX(column_name) AS column_name, and it worked fine.
