Attended: drallen, dlgawley a2brenna fhgunn ldpaniak

Agenda:

• Timeline for project
  • Next round of milestones
• Items to be assigned
• Brief summary since the last meeting

Timeline for project

• Daniel worked up a timing estimate; we've reviewed it as a group with one minor addition. That schedule puts us:
  • moving first database (inventory) Thursday Feb. 11 (soft date)
  • moving the rest of databases Monday Feb. 22 (soft date)
  • wrapping up Thursday March 3rd. (soft date)

New Milestones

• Benchmark current mysql.cs performance - initial low-level tests by Lori: reset the due-date to tomorrow.
  • MUST be certain that low-level testing won't cause undue problems to marmoset or faculty-recruiting (Daniel to look at faculty-recruiting schedule; Lori to ask Nick re: Marmoset).
• Application-level benchmarking of current mysql.cs: to be done by Daniel, Friday 5 Feb.
• Testing HA failover: Anthony and Fraser, Friday 5 Feb.

Items to be assigned:

• Testing failover of saltified database (Anthony and Fraser).
• Establish monitoring of the HA service: resource, latency, health (Anthony will research, can start as soon as saltification is done)

Brief summary since the last meeting:

• Daniel confirmed with Dave that the server room high-speed network interconnects were racked and ready for firmware updating. Lori has done that step; they now have passwords and updated firmware.
• Fraser had a casual conversation with Ken, who agreed that asynch replication was much better than what we have, and a reasonable goal for the project.
  • Management has had a formal discussion with Ken. He is now on the same page as us; and is pleased that we're doing this with project management.
• Lori, Dave, Clayton, and Daniel discussed the "synch vs. asynch" question; Lori did parallel investigation into synchronous clustering.
  • Lori did a full round of testing the weekend of Jan 30th, documented in ST#85594, to indicate synchronous would be possible- but only with a fairly different hardware setup than we've considered, and therefore out of scope for this project this term.
    • "Mysql Cluster" won't work effectively with three data nodes- which would provide no redundancy. It would require four or more; requiring fast network and lots of storage.
    • It would also require a management server (low requirements of CPU/storage/networking)
    • and two or more SQL nodes (high requirements for CPU/networking, low requirements of storage).
• Milestones: by today:
  • mysql configuration by salt - Anthony: documented in ST#85594 Feb 3 2016 14:04
    • access via root from linux.cscf to mysql-10{2,4,6}.cs
  • Benchmark current mysql.cs performance - initial low-level tests by Lori: reset the due-date to tomorrow.
    • MUST be certain that low-level testing won't cause undue problems to marmoset or faculty-recruiting (Daniel to look at faculty-recruiting schedule; Lori to ask Nick re: Marmoset).

• Agreed that for now, we will back up both master and slave. This might change later, but the cost is low for additional assurance.
• Agreed to add an item to saltification: adding a few lines for a shorewall firewall. We should block all but the expected mysql traffic (and ssh from linux.cscf).
• Agreed that monitoring MIGHT extend to using percona; Antony will saltify percona configuration, as well as shorewall.
• Noting that tuning mysql parameters will include choosing mount options - to be done by Dave and Anthony.

• Noting that phase two, the student cluster, could possibly be set to authenticate using active directory rather than a standalone auth setup? However, we might decide to put marmoset on the second cluster, and set up the rest of student databases on the non-HA hardware currently running mysql.cs.

-- DanielAllen - 2016-02-01