Normal operation

Nagios should accurately report that the cluster is operating normally. In case you would like additional verification, the following are manual diagnostics to be certain of normal operation.

On denoted systems {master, slaves}:

• master, slaves - # systemctl status mysql (or service mysql status on older servers) shows mysql service is running.
• master, slaves - # ps -lfp `cat /run/mysqld/mysqld.pid` shows mysqld is running.
• master, slaves - # lsof -i | grep LISTEN | grep mysql shows mysqld listening on the correct interface(s).
• master, slaves - # mysql -e 'show variables like "read_only"' shows read_only is OFF for master and ON for slaves.
• master, slaves - # mysqladmin status shows reasonable values. Almost all queries on a slave correspond to write operations on the master.
• master, slaves - # mysqladmin processlist | sed 's/^| //;s/ | /  /g;s/[-+| ]*$//' on the master shows a slave thread for each slave (plus lots of client threads). On each slave it shows an I/O thread and an execution thread.
• master - # mysql -e 'show master status' shows the binlog file and position. Other fields should be empty.
• slaves - # mysql -e 'show slave status\G' shows approximately 40 lines of output. Especially useful are:

               Slave_IO_State: Waiting for master to send event
                  Master_Host: ...
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes
                   Last_Error: 
        Seconds_Behind_Master: 0

• master - # pt-table-checksum --defaults-file=/etc/mysql/pt_checksum.cnf --replicate=percona.checksums --ignore-databases mysql --no-check-binlog-format
• master - # ls -l /var/log/mysql/binlog.* returns a list of files; binlog.index and the most recent file should all have been written very recently today.
• master, slaves - should be clean # ls -l  /var/log/mysql.err