mysql will not start after 8.0.15 to 8.0.16 update on Ubuntu 16.04 LTS

======================================
mysql 8.0.15 -> 8.0.16 upgrade error 
======================================

1) The mysql server refused to start when we upgraded from 8.0.15 to 8.0.16
on Ubuntu 16.04 

Apr 25 09:08:09 yuktix-apiv11devm1 systemd[1]: mysql.service: Main process exited, code=exited, status=1/FAILURE
Apr 25 09:08:09 yuktix-apiv11devm1 systemd[1]: Failed to start MySQL Community Server


$sudo systemctl status mysql 
just shows BOOT_ERROR
and ERROR 2 (file not found)

$sudo journalctl -xe | less 

- so the guess is that either 
    (a) apparmor settings are not right 
    (b) or mysqld is not finding some file. 
    
To see if dmesg could not do something because of apparmor,
$dmesg -T | grep mysql

2) /var/log/mysql/error.log has following entry 

[Server] Could not create server upgrade info file at '/var/lib/mysql/'.
2019-04-25T09:08:08.106750Z 0 [ERROR] [MY-013380] [Server] Failed to upgrade server.

so the guess is that 
(c) /var/lib/mysql may not be writable 

3) To see what is happenning, we can try starting the debug version,

$sudo /usr/sbin/mysqld-debug --user=mysql  --log-error-verbosity=3
This shows details in /var/log/mysql/error.log,

Server upgrade from '80015' to '80016' started.
Running queries to upgrade MySQL server.
Upgrading system table data.
The sys schema is already up to date (version 2.0.0).
Upgrade of help tables started.
Upgrade of help tables completed.
Checking 'mysql' schema.
Checking 'sys' schema.
Could not create server upgrade info file at '/var/lib/mysql/'.
Failed to upgrade server.
Aborting

4) mysql is not able to write to /var/lib/mysql? why? 
- all permissions look ok. 
- however there is a file mysql_upgrade_info in /var/lib/mysql that is owned
by root. 

we change the permission for that file, 
$ sudo chowm mysql:mysql /var/lib/mysql/mysql_upgrade_info

Then start the debug version again.
$sudo /usr/sbin/mysqld-debug --user=mysql  --log-error-verbosity=3

5)next complain is that /var/run/mysqld does not exists! 
lock file /var/run/mysqld/mysqld.sock.lock

Server hostname (bind-address): '*'; port: 3306
IPv6 is available.
   - '::' resolves to '::';
 Server socket created on IP: '::'.
 Could not create unix socket lock file /var/run/mysqld/mysqld.sock.lock.
 Unable to setup unix socket lock file.
 Aborting

The fix is then to create /var/run/mysqld and /run/mysqld folders and 
give the ownership to mysql user 

$sudo chown mysql:mysql /var/run/mysqld 


# start again 
success! 

# systemd startup file 

is /lib/systemd/system/mysql.service

ExecStartPre=/usr/share/mysql-8.0/mysql-systemd-start pre
ExecStart=/usr/sbin/mysqld


# quickly enable/disable apparmor 

$sudo aa-status
$sudo aa-complain /usr/sbin/mysqld (will not enforce rules, only complain in log)
$sudo aa-enforce /usr/sbin/mysqld

Thanks man! In my case I had to crate /var/lib/mysql/data/mysql_upgrade_info and give it mysql:mysql ownership, so the server could write "8.0.16" into it although all the permissions were OK. Such problems are really annoying...

Same here. In my case the file was there but owned by root. The chown alone solved the issue.
Note: chown not chowm :)

@rjha I see some more comments for the same problem, so I think it is a bug, which should be reported at bugs.mysql.com and you were the first to report it I think, because last Thursday it was the only result that popped up in Google :-)

@sibienl Since 8.0.16 there's now the option --validate-config, so you could check your configuration changes before restarting the server ;-)

Terje Røsten from mysql team logged a bug.
https://bugs.mysql.com/bug.php?id=95165

I upgraded from mysql 5.7 to percona 8 and i got this errors and fixed by your helping, thanks so much

changing the ownership of that file, helped me out on an arch system running percona server

$ sudo chown mysql:mysql /var/lib/mysql/mysql_upgrade_info