ClusterInformationAggregationTool (CIAT)

This page is for a staff meeting presentation. Usage instructions can be found on the project repository.

Problem

• ~140 nodes of the tembo cluster need to be added to inventory
  • Nodes are not all identical, minor subtleties
• Helpful if barcode were written into BMC

Tech Stack

• Python chosen as scripting language
  • I am familiar with it
  • High level features available ex. JSON Parsing
• Salt Stack
  • Used to run script to collect information from all nodes (Salt selector "*")
  • Can be optionally used to associate barcodes with hostnames
• lshw, lscpu
  • Simple tools to get system specs with JSON output
  • Available though most package managers
• Input and output as CSV files
  • Easy to work with in python
  • Compatible with CSCF inventory
• IPMI tool
  • Only option for writing barcodes into BMC FRU
  • In the form of a binary packaged with CIAT

Usage Overview

Full details in repo Readme

Download into a shared directory accessible by all nodes:

git clone https://git.uwaterloo.ca/qhyun/CIAT.git

Fill out example_data.csv and make accessible to the head node. Give the command:

./ciat.py --output /share/output master /share/example_data.csv

You get out:

tem99.tembo-domain.cs.uwaterloo.ca:
    2019-10-18 13:08:49 INFO     Running as slave
    2019-10-18 13:08:49 INFO     Running lshw
    2019-10-18 13:08:49 INFO     Verifying lshw exists
    2019-10-18 13:09:00 INFO     lshw installed, running now
    2019-10-18 13:09:01 INFO     Determining simple lshw fields
    2019-10-18 13:09:01 INFO     Determining total amount of ram
    2019-10-18 13:09:01 INFO     Determining network info
    2019-10-18 13:09:01 WARNING  Could not load ip of network eno2 / MAC 0c:c4:7a:d9:0c:e9. This network is most likely useless / internal.
    2019-10-18 13:09:01 INFO     Network IDs    : eno1, enp4s0
    2019-10-18 13:09:01 INFO     Network MACs   : 0c:c4:7a:d9:0c:e8, 24:8a:07:79:7f:a0
    2019-10-18 13:09:01 INFO     Network IPs    : 192.168.152.199, 192.168.252.199
    2019-10-18 13:09:01 INFO     Network Speeds : 1Gbit/s, 10Gbit/s
    2019-10-18 13:09:01 INFO     lshw completed and parsed
    2019-10-18 13:09:01 INFO     Running lscpu
    2019-10-18 13:09:01 WARNING  Could not load lscpu as JSON data. Please upgrade lscpu to a version which supports --json. Now trying to fetch data though regex
    2019-10-18 13:09:01 INFO     lscpu completed and parsed
    2019-10-18 13:09:01 INFO     Running salt stack to determine barcode
    2019-10-18 13:09:03 INFO     Running IPMICFG to burn barcode CS010574
    2019-10-18 13:09:04 INFO     Successfully wrote data to file /share/tools/invadd/./output/tem99
    2019-10-18 13:09:04 INFO     Information collection complete
tem54.tembo-domain.cs.uwaterloo.ca:
    Minion did not return. [Not connected]
tem74.tembo-domain.cs.uwaterloo.ca:
    Minion did not return. [No response]
tem36.tembo-domain.cs.uwaterloo.ca:
    Minion did not return. [Not connected]
ERROR: Minions returned with non-zero exit code
2019-10-25 10:58:26 INFO     Opening static data csv
2019-10-25 10:58:26 INFO     Found hostname in column 1 of static data csv
2019-10-25 10:58:26 ERROR    Following hostnames were skipped: tem27, tem36, tem54, tem74, tem75, tem76, tem77, tem78, tem79, tem89
2019-10-25 10:58:26 INFO     Information aggregation complete, see output in ./output/aggregate.csv

hostDomainName","model","cpuModel","memory","numberOfCPUs","numberOfCores","speed","serialNumber","ipAddress","macAddress","barcode","purchaseCost","purchaseOrder","warrantyStart","warrantyStop","accountNumber","active","adminContact","authUser","comments","dns_admin","dns_contact","edocsUrl","equipmentType","operatingSystem","policyEight","purpose","room","vendor","isSaltProvisioned","dhcp","vlanTagging","macName","description" "tem01","SYS-6017TR-TF (To be filled by O.E.M.)","Intel(R) Xeon(R) CPU E5-2620 v2","32GB","2","6","2.10GHz","S11372916903563","192.168.152.101","0c:c4:7a:d8:70:c0","CS010631","7406","221886","2016-09-01","2019-09-01","2950-500-105-2515-120775-6216-0000-000","y","ldpaniak","tozsu","ST#106654; ST#107436; RT#602213","cs-rsg-tembo@cs.uwaterloo.ca","cs-rsg-tembo-logs@cs.uwaterloo.ca","https://cs.uwaterloo.ca/twiki/view/CFPrivate/Tembo","Computer","Linux Ubuntu 16.04","Restricted","Node in Tembo cluster","PHY1002","NetDirect","0","n","n","1GbE Admin","SMC SYS-6017TR-TF; 2x E5-2620 v2 @ 2.10GHz, 32GB RAM, 2x 1TB 7200RPM HDD, 2x GbE + Mellanox 10GbE " "tem02","SYS-6017TR-TF (To be filled by O.E.M.)","Intel(R) Xeon(R) CPU E5-2620 v2","32GB","2","6","2.10GHz","S11372916903563","192.168.152.102","0c:c4:7a:d8:70:c4","","7406","221886","2016-09-01","2019-09-01","2950-500-105-2515-120775-6216-0000-000","y","ldpaniak","tozsu","ST#106654; ST#107436; RT#602213","cs-rsg-tembo@cs.uwaterloo.ca","cs-rsg-tembo-logs@cs.uwaterloo.ca","https://cs.uwaterloo.ca/twiki/view/CFPrivate/Tembo","Computer","Linux Ubuntu 16.04","Restricted","Node in Tembo cluster","PHY1002","NetDirect","0","n","n","1GbE Admin","SMC SYS-6017TR-TF; 2x E5-2620 v2 @ 2.10GHz, 32GB RAM, 2x 1TB 7200RPM HDD, 2x GbE + Mellanox 10GbE "

CSCF Inventory should now happily consume it.

Links

Git Repo

RT