Meeting Date

• TEAMS: 2024-06-12

Invited

Anthony (group leader), Lori, Clayton, Guoxiang, Nathan, Nick, Todd, Ed, Devon, Gwen

Attendees

Review and accept previous meeting minutes.

CsLWGMeeting20240612

Review last meeting's Action Items

linux.cscf.uwaterloo.ca

• Still blocked on comprehensive review of authentication/authorization configuration, work stalled - need to schedule in person meetings
  • Blocked on staff/manager availability
  • Hoping for an update within the next couple of weeks

New DFSc Hardware

• Hardware is racked, waiting on Phil for inventory work

DFSc Performance

• Snapshots blocked on cleanup of old homedirs
  • Waiting on completion of ongoing tape backup before moving and deleting approximately 9400 retired homedirs, accounting for approximately 20% of space usage in TEACHING * Need to double check that no course accounts are included in the retired homedirs * Should also email users who will have their homedirs deleted - Nick * a2brenna to create list
• Need to send message to users (login message (only applies to login shell), other?) when snapshots are re-enabled, links to appropriate end user documentation - Anthony
  • Will leverage MOTD to point to other more comprehensive documentation
  • Engage management on documentation location (Anthony)
    • Nick Lee maintains a useful page on www.cs.uwaterloo.ca about the general use TEACHING environment, collocating documentation there is an option.
• Quotas blocked on double checking of quota exceptions
  • Nick double checking this
  • Need to double check accounts that don't have a quota set to prevent possible account locking

Endpoint Protection

• IST wants to deploy SentinelOne end point security agents on all devices
• Planning test deployment to linux.student.cs (on a single node) in passive (watch and complain) mode for spring term,
  • Waiting on homedir backups to complete so we can get a clear picture of DFSc traffic caused by S1
    • Still ongoing as of 2024-06-26
  • IST has been informed, is generally supportive

NFS Ganesha server issues (nfs-files.student.cs)

• Soft lockup over weekend(?)
  • futex_abstimed_wait_common
• Analysis of crashes is blocked due to lack of staff time
• Looking into alternatives is also blocked due to lack of staff time
  • File locking is a concern with other options

Future of RT

• CSCF and MFCF met to discuss future of RT
  • Moving forward but exact path and work split between MFCF/CSCF unclear
  • Recent RT troubles have added significant uncertainty and upward pressure on 0.2-0.25 FTE time commitment
  • Management reaching out to IST to resolve some of our outstanding questions, i.e. what will become of the rt.uwaterloo.ca domain
• IST has acknowledged RT is currently breaking frequently, unsure of when these issues might be resolved. Very interested in root cause as it relates to ongoing staffing requirements.
• More news
  • Instead of CSCF running/hosting, will be splitting primary/backup roles for running RT between CSCF and MFCF
  • Modest hardware requirements
  • Domain - hoping to be given the rt.uwaterloo.ca domain, but IST might want to retain it for archival purposes
  • Jira does not support RSG's billing
  • Current plan is to migrate to RT5 when taking over RT

New Business

Cooling woes

• Total cooling failure after hours last Thursday
  • DC 3556 approached a temperature of 40C
  • https://icinga.cscf.uwaterloo.ca/grafana/d/eac3847b-e567-46d3-a305-cb3a25f11cde/chilled-water-outage-monitoring?orgId=1&from=1718596800000&to=1719201599000
• Scheduling post-mortem

Comments