Inventory Service Monitoring Meeting Notes

2021-02-16

Initial meeting with managers; full discussion of the request. Next meeting on 2021-03-10.

2021-02-17

Devs discussion

Problem statement: it's easy for inventory to document an editable list of services that a machine is meant to supply, without tying this to any implementation.

• It is more difficult to flexibly and loosely tie this to an arbitrary set of tools.

Requirements discussion:

• critical: does not hang inventory; (caches icinga data in case it goes down)
• ? can support multiple monitoring agents and multiple salt-masters
• inventory documents the desired status for services (but inventory doesn't apply the changes?)
• ? systems poll inventory and "do it" - inventory triggers?

inventory can query icinga API to get available service-sets (cached); we can offer the user a list to choose from and pair a machine to a service to monitor.

Open question- how to address recording a plaintext name in inventory, then someone adds a service in icinga. Requires followup on the Inventory side.

The paired list of hosts/services can be queried by an external runner which will take actions via Icinga API.

icinga authoritative for some data.

salt authoritative for some data.

• need someone to test the changes on the salt side before it applies.
• possible implementation: we have a table of allowable salt masters and "blessed" services on each of these masters. This table is updated in the Admin area of inventory.
  • Anthony uses inventory access to deliver a salt external pillar which reports any hosts/services for a given master.
  • How salt interprets this is up to the salt master.

Given that Inventory can store multiple hostnames/IPs for a given record, and both icinga and salt control by hostname/IP, any service monitoring or management will require selecting which host/IP is the one being used.

what happens if hostname/ip address changes?

• rename host in icinga via API.
• inventory is authoritative for room, admin contact, possibly parenting

Can we pull icinga service-monitoring parameter names (and values) from API?

• Devon to answer this.

Action items before next meeting with managers (Tues March 2nd):

• Daniel to write up notes/ticket; and generally project manage
• Anthony to talk with Dave about salt ambiguity/challenges and report back by Friday Feb 25th.
• Devon to investigate API abilities and ideally report back by Friday Feb 25th.
• Brian to stay in the loop and be free to make any design suggestions as we go

2021-03-01

Devs discussion

• Anthony conversation with Dave;
  • Inventory to not push changes to salt.
  • Inventory to query lists:
    • icinga offers service sets; salt (and we're not sure what to call the category of things we're naming in salt)
  • inventory consumes these lists; offers lists to users as dropdowns.
  • inventory exposes to icinga/salt the lists that are attached to that host.
  • salt periodically queries lists.
  • cache data so inventory doesn't hang waiting for monitoring.
  • generic framework for "icinga and any future monitoring" is quite difficult.
• phase 2: icinga/salt expose what are currently monitored/deployed and inventory queries to inform user.
  • 3 colour list:
    • green: monitored and good
    • red: monitored and bad
    • yellow: not monitored.
• Devon on APIs: "Director API" vs "Cluster API". He's talking about Director API to control service sets...
  • Director API overloads URL for webpage... which is behind ADFS
    • mod_auth_mellon vs. python direct ADFS module; using apache IP address check for now.
  • endpoints:
    • service-sets endpoint doesn't exist?!
    • can supply a name, it will return IF it exists...
    • could get from icinga deets from mysql query directly (select * from service_sets);
    • anything else he's looked for, exists as end-points. Great!
    • if we want to do phase 2, need to talk to 2nd API to grab current data (or schedule downtime, confirm acknowledgement).
      • issue with inventory: we don't want slowness.

UI design?...

• table view as with existing services? Or a wizard which prompts for what you need to do next.
• Devon's got wizard samples; we all think it's a good approach
• UI be further discussed later this week

Action items:

• Daniel to reschedule larger project team, ideally 1 week
• Daniel to schedule devs to meet this Thursday afternoon.
• Anthony to investigate how to expose this in Salt... not precisely trivial.
• Devon to share images of his UI ideas

• wizard modal example for DNS:
  

• Each step can have field validation
  

• Allows for freedom when choosing UI elements
  

2021-03-04

Devs discussion

Agenda: Requirements and UI

We will present a draft design at the next team meeting next Wednesday, including a mockup UI.

Iteration on Requirements

• Inventory documents the desired list of services for any host.
  • Inventory has a database table which records a list of services, in standardized form, pulled from Icinga's Director API. [ was previously: manually updated via Inventory admin interface; we can pull it from Icinga ]
  • This list can be looked up via SQL by any approved system, such as Icinga and Salt. Inventory does not control/monitor what the systems do with this data.
    • For reliability, inventory does not activate updating the host to install/uninstall these services; that is controlled and monitored within salt.
    • Adding new systems (such as replacing Icinga) will be handled on a one-by-one basis as they are needed.

• Inventory caches icinga's list of monitored service-sets for all hosts; and refreshes the cache regularly (hourly?)
  • Loading the inventory UI will query Icinga for the up-to-date service list for the particular host;
    • if Icinga is down, inventory reports the cached data and disables updates for as long as Icinga is down.
  • Inventory does not cache (or report) service status.

• Phase One does not include monitoring status reports; this requires using a second Icinga API.
  • We can design for how we might present that data in the UI, and implement that in Phase Two.
• Phase One does not include salt status reports; this is also not well-specified at this point and we'd like to implement something we're confident we can finish.

UI Design

We want service monitoring to be easy to understand how to add/remove data, and easy to display existing data.

• Wizard for adding data
  1. select which of the hosts on the record will be monitored.
  2. select the service-set
  3. select additional parameters such as additional contact emails to report to.
  4. report what values from inventory are going to be sent to Icinga (host name, room, admin contact)
  5. button to add
• viewing and deleting: in table format. button to delete.

Do we need UI elements for salt?

• Anthony suggests it's not necessary; he would be happy with querying the list of documented services.

Action items:

• Devon to experiment with balsamiq to show a mockup of what we've discussed.
• Daniel/Brian to work with Devon on iterations before next Wednesday.

2021-03-10

Project team

Agenda:

• Reviewing UI mockup and requirements
• Salt considerations
• Action items

We reviewed the UI mockup and the latest statement of requirements. All agreed with the requirements as described on 2021-03-04

• Q: can we document service descriptions for every service, or only per service-set? A: Devon says the API will only expose the service set descriptions.
• Q: confirmation we can poll and regularly cache the list of available service-sets from icinga? A: yes.

Salt considerations

• Salt will be separate work driven by Inf/Anthony determining what we can consume from Inventory.
• Dave is clear that if we're offering data for salt to consume, we should have a flag for "use this data?" for any host. Don't assume all hosts should be sent to salt just because the data is available. OK!

Action items

• Devon to map out the API access methods.
• Brian to build a sample interface as described.
• Daniel will call another project meeting when we have something to demo.