Attended: drallen a2brenna fhgunn ldpaniak

Agenda:

• Progress on Milestones
• Timeline
• Brief summary since the last meeting

Progress on Milestones:

• Backups set up and tested: (Due-date reset to next Wednesday): Remaining: testing restore

Timeline

• Fraser and Daniel are testing and re-moving inventory- likely not finished until mid-next-week (10th or so).
  • Schedule shows wrapup March 31 (!)
    • where the last 4 days is wrapup, and the 5 days before that are finalizing monitoring, tuning, documentation.
    • we're actually doing all three as we go, which will take less time at the end. Good.
  • by next Wednesday: finish inventory move
  • by following Thursday: finish phase two

Brief summary since the last meeting, and upcoming week

• slave grants: Anthony has updated to use hostnames.
• Fraser worked cleaning up permissions and process for re-importing (prod changes)
• Fraser/Daniel worked on failover testing, which had a number of discoveries
  • default binlog_format=STATEMENT may have different failover response than MIXED ("best practice"). Switched in the live setup; will need failover testing (Daniel) - and will set up frequent percona checksum (daily? hourly?) (Fraser)
  • slaves are not read-only; updates are NOT copied to master; so probably want to set read-only ("best practice") - and skip networking (would still allow mysql connections via localhost) (Fraser)
  • master/master/slave has failure-modes. So probably want to go with master/slave/slave. (Anthony)

• Anthony: restore from backups this week - using Fraser's documentation as base, and save edited docs.
• Fraser will be updating configs this week
• Anthony: update salt to reflect recent changes - this week
• nagios checks: Lori to implement this week.
• Anthony: switch the machine to 10Gb (requiring restarting containers) - probably ST#102302
• Backups: Anthony will deb-ify Fraser's mysql-102:/usr/local/lib/mysql scripts and install on mysql{,.student}.cs and mysql-10{2,4,6}

• Anthony: documentation will be sufficient that given a dead master, switch to another master.

Still to do later:

• Fraser: running mysqldumpexcept is quick. run twice separated by 10 sec; if the diff are identical, can use that timestamp.
  • how do we re-run import of permissions without clobbering mysql table?
    • percona command to output grants
      • but mysql.cs has lots of junk- grants for tables/users that don't exist; so fraser is writing a script to trim these.
  • does nscd run? We don't think so; probably we want to run nscd. Anthony will set up nscd- we can benchmark before/after.
• Anthony will review /etc/mysql/conf.d/ and merge into salt what he can.
  • Salt and mysql: best practices for password storage? Anthony suggests making a more secure password to be stored on slave... Anthony will make an ST item to do that before we're done. (/etc/mysql/conf/slave.conf ?)
• Anthony will add marmoset to the slave "do NOT replicate" configs
• Fraser will update the written list of instructions for doing stage two. He will put it in ST for comment.

Next Meeting

• Wed 9 Mar? (day before retreat)

-- DanielAllen - 2016-03-02