DFSc Working Group

Meeting Date

• TEAMS: 2022-06-16

Invitees - Attendees

• Dave, Gouxiang, Lori, Nathan, Nick, Lawrence

Review and accept previous meeting minutes.

• CsDFScWgMeeting20220215

Proposed Agenda Items

Old business

Action items for this meeting 2022-06-16

• ldpaniak/nfish: Ceph upgrade on the 902s
• Omar: Fraser - create a ticket for the plans for local storage option -> https://rt.uwaterloo.ca/Ticket/Display.html?id=1209206
• Anthony: Any plans to deal with cephfs client crashes on teaching systems? - yc2lee/gxshen - RT#1214857 (revert to 5.4 kernel)
• Clayton: NFS servers to ganesha 4.0
• Guoxiang: need to get a full backup before the Ceph upgrade
• Lori: disable/reduce new snapshots in anticipation of Ceph upgrade (?)

New business

• Access to DFSc performance counters, RT#1211673 dlgawley
  • Dave's not here - leave for next meeting

Upgrades

Start with 902 systems. Practice upgrades, work out bugs nfish/a2brenna/ldpaniak

• in progress

Server side

• Want to upgrade to Pacific by end of summer at the latest: strays, upgrade path, less OSD spill from RocksDB(sharding) currently:3, 30, 300GB..., mclock scheduler, graphana daemons. Octopus out of support 2022-06-01: Early May?
• One MDS problem
• Remove all snapshots?
  • in preparation for the upgrade
  • Anthony: we should ensure we have complete backups before that
  • Guoxiang: running low on disk space for index files, backups are slow, new hardware has problems, doesn't know when he can do the full backup (pretty soon?)
• Real downtime with low/no cluster load
• ceph deploy deprecated: Migrate to ceph-adm (docker containerization) then upgrade
  • maybe not as deprecated as originally thought
  • probably can't use ceph-adm
  • will try on the 902s
• splitting cs-teaching into multiple real filesystems: u0-u19(?)

Client side

• • NFS servers to ganesha 4.0 -> ctucker
  • https://launchpad.net/~nfs-ganesha/+archive/ubuntu/libntirpc-4
  • https://launchpad.net/~nfs-ganesha/+archive/ubuntu/nfs-ganesha-4
  • Follow on from previous work https://rt.uwaterloo.ca/Ticket/Display.html?id=1208856
    • should we wait until after the kernel reversion?

Scratch(ish) drives on 211 systems (fhgunn)

• ZFS sends for sync

Action items for next meeting

• Back up email on DFSC/CS systems on August 5 (dlgawley , gxshen)
• Start retiring pre-current term snaps (before May) (ldpaniak, nfish) https://rt.uwaterloo.ca/Ticket/Display.html?id=1228804
• New snapshots continue for now as required by courses (a2brenna, yc2lee)
• Revisit snapshot removal schedule at next DFSC meeting for the purposes of mid-August DFSC upgrade