Icinga

Icinga is an open-source computer and network monitoring system that originally started as a fork of Nagios in 2009.

CSCF is working to migrate from Nagios to Icinga due to the extended feature set including (but not limited to) the RESTful API and "service-level" monitoring that is more applicable to the University of Waterloo's Computer Science network.

While most of the documentation on this page is mostly for internal use, others may benefit from it.

Glossary

Overview

Icinga in our case is a collection of services and applications that are used in conjunction to create the Icinga Stack.

Icinga2

The Icinga2 service is the core of the stack. It runs as a systemd service under the name "icinga2".

It does not have a GUI and can only be directly interacted with via the icingacli command tool.

The Icinga2 service is what reads the written configuration files for hosts, services, commands.etc and schedules/runs checks on hosts/services. Based upon the output of a check, a result script is run that could do anything from sending an email/slack message to automatically trying to fix the issue over SSH.

The Icinga2 service writes its current status (i.e a cache) to the local MySQL instance using IDO2DB. Other services such as NagVis and IcingaWeb2 do not read from the Icinga2 service directly but instead read from the database.

IcingaWeb2

IcingaWeb2 is a web-based frontend.

It consists of multiple components to monitor and generate configuration files for the Icinga2 service.

IcingaWeb2 reads the current status and events from the local MySQL database. The Director module is used to generate and manage configuration files for monitoring hosts and services. Commands (such as scheduling downtime) are sent to the Icinga2 service directly via the REST API.

IDO2DB

IDO2DB (Icinga Data Out to Database) is used by the Icinga2 service to write current status and events to a local database (MySQL in our case) for later reading.

NagVis

NagVis is a graphing and visualization addon that was created for Nagios. As Icinga started as a fork of Nagios and shares the same host/service layout system, it is capable of integrating with Icinga via the MySQL database and IDO2DB feature.

NagVis is used by CSCF to map out the monitored network using automatically generated tree-view graphs. These graphs are generated based upon room hostgroups and host parent/child properties.

Server Installation

Ubuntu 18.04 LTS

IDEA! This server installation guide assumes a fresh install of Ubuntu 18.04 LTS with networking and SSH access already configured. A basic understanding of Apache configuration is also required to tweak and finalize the webserver setup. In order to use Let's Encrypt, the webserver must be externally accessible using either a DNS or HTTP challenge.

  1. Setup Icinga package repository.
    1. Update the current package repository.
      apt-get update
    2. Install curl and GPG packages.
      apt-get install curl gnupg
    3. Install the Icinga package repository key.
      curl https://packages.icinga.com/icinga.key | apt-key add -
    4. Add the Icinga package repository to the apt sources list.
      echo -e "deb https://packages.icinga.com/ubuntu icinga-CodeName main\ndeb-src https://packages.icinga.com/ubuntu icinga-CodeName main" | sed -e "s/CodeName/`lsb_release -c -s`/" > /etc/apt/sources.list.d/icinga.list
    5. Update the current package repository now that the Icinga repository has been added.
      apt-get update
  2. Install Icinga2 and IcingaWeb2 packages.
    apt-get install icinga icingaweb2 icingacli php-gd
    IDEA! Enter a secure password for the icingaadmin user when prompted. When asked whether to enable external commands, select No.
  3. Setup Let's Encrypt/Certbot
    1. Install required packages.
      apt-get install software-properties-common
    2. Add the required Certbot repositories.
      add-apt-repository universe
      add-apt-repository ppa:certbot/certbot
    3. Update the current package repository and install the certbot packages.
      apt-get update
      apt-get install certbot python-certbot-apache
    4. Run certbot to install a certificate on the Apache server setup by Icinga in the previous steps.
      certbot -n --apache -d SERVER_FQDN --agree-tos -m ADMIN_EMAIL --redirect
      IDEA! Replace SERVER_FQDN with the fully qualified domain name for your server. Replace ADMIN_EMAIL with an administrative email for emergency SSL based alerts to be sent to.
  4. Setup the local MySQL server and IDO2DB module for the Icinga service.
    1. Install the required packages.
      apt-get install mysql-server mysql-client icinga2-ido-mysql
      • When prompted, enable the ido-mysql plugin
      • Configure icinga2-ido-mysql with dbconfig-common
      • Enter a secure password for later use
      • IDEA!This will create a mysql database and appropriate user both called "icinga2" and will import the IDO schema. The password for that user is what you entered when prompted. The database information will also be added to /etc/icinga2/features-available/ido-mysql.conf
    2. Run the mysql setup wizard.
      mysql_secure_installation
      • Enable the password validation plugin (and select an appropriate security option)
      • Enter a secure mysql root password to remember (for admin use only)
      • Remove the anonymous users
      • Disallow remote root login for MySQL
      • Drop the test database
      • Reload the privilege tables
    3. Enable the IDO MySQL module and restart the Icinga2 service.
      icinga2 feature enable ido-mysql
      systemctl restart icinga2
      IDEA! The Icinga2 service should now automatically enter information into the local MySQL server's icinga2 database.
  5. Setup the Icinga2 API
    1. Enable the API feature.
      icinga2 api setup
    2. Edit /etc/icinga2/conf.d/api-users.conf and add the following info to create an API user for IcingaWeb2.
      object ApiUser "director" {
        password = "RANDOMPASSWORD"
        permissions = [ "*" ]
      }
      
      object ApiUser "icingaweb2" {
        password = "RANDOMPASSWORD"
        permissions = [ "status/query", "actions/*", "objects/modify/*", "objects/query/*" ]
      }
      IDEA! Replace RANDOMPASSWORD with a secure password. This API user will be used by IcingaWeb2 to control and send commands to the Icinga2 service.
    3. Restart the Icinga2 service.
      systemctl restart icinga2
  6. Setup Mellon/ADFS for authentication. Refer to CSCF's ADFS Documentation for setup information.
  7. Configure IcingaWeb2.
    1. Create the configuration directory.
      icingacli setup config directory --group icingaweb2
    2. Create a setup token.
      icingacli setup token create
      IDEA! Copy the setup key. It will be used in the next step.
    3. Navigate to /icingaweb2 on your webserver via a web browser.
    4. Click on the button to start the setup wizard and enter the previous setup key.
    5. Follow the setup process while using the following information:
      • Ensure that the Doc and Monitoring modules are enabled (no others are needed for now)
      • For Authentication Type, select "External" (since we're using ADFS)
      • Leave Authentication Backend as default
      • Set the Administration Username to your ADFS user
      • Leave Application Configuration as default
      • Configure the Monitoring module
        • Leave the Backend name as "icinga" for now
        • Make sure that the backend type is set to IDO
        • Enter the icinga2 for the database name and user.
        • Enter the user password that was setup in the previous IDO MySQL steps (the same password in /etc/icinga2/features-available/ido-mysql.conf)
      • On Command Transport, enter the FQDN of the server and API username/password for the icingaweb2 API user created earlier (stored under /etc/icinga2/conf.d/api-users.conf)
      • Leave Monitoring Security as default
  8. Setup NagVis.
    1. Install the required packages.
      apt-get install git php-mbstring php-gd php-sqlite3 graphviz sqlite3 rsync icinga-idoutils
    2. Install and enable the NagVis Icinga module.
      git clone https://github.com/Icinga/icingaweb2-module-nagvis.git /usr/share/icingaweb2/modules/nagvis
      icingacli module enable nagvis
      IDEA! This module isn't NagVis itself but is instead a menu option to view the graphs within IcingaWeb2.
    3. Clone the NagVis git repo to your home directory.
      git clone https://github.com/NagVis/nagvis.git nagvis-repo
      cd nagios-repo
    4. Run the install script to install NagVis.
      ./install.sh -a y -q -i ido2db -w /etc/apache2 -s icinga -p /usr/local/nagvis
    5. Enable the NagVis Apache configuration and restart the Apache service.
      a2enconf nagvis
      systemctl restart apache2
    6. Edit the logonmodule, logonenvvar, logonenvcreateuser and logonenvcreaterole options in /usr/local/nagvis/etc/nagvis.ini.php to be set as the following:
      logonmodule="LogonEnv"
      logonenvvar="REMOTE_USER"
      logonenvcreateuser=1
      logonenvcreaterole="Guests"
      IDEA! This sets NagVis to use external authentication.
  9. Setup the Director module
    1. Create a mysql database for the director module (replace some-password with a secure password)
      mysql -e "CREATE DATABASE director CHARACTER SET 'utf8';
         CREATE USER director@localhost IDENTIFIED BY 'some-password';
         GRANT ALL ON director.* TO director@localhost;"
    2. Install the dependencies required for the Director module
      git clone --single-branch --branch stable/0.5.0 https://github.com/Icinga/icingaweb2-module-incubator.git /usr/share/icingaweb2/modules/incubator
      git clone --single-branch --branch stable/0.7.0 https://github.com/Icinga/icingaweb2-module-reactbundle.git /usr/share/icingaweb2/modules/reactbundle
      icingacli module enable incubator
      icingacli module enable reactbundle
    3. Install the Director module
      git clone https://github.com/Icinga/icingaweb2-module-director.git /usr/share/icingaweb2/modules/director
      icingacli module enable director
    4. Navigate to Configuration -> Application on the web interface and open the Resources tab
      1. Create a new resource with the type SQL Database
      2. Set the Resource name to director
      3. Set the Database Name to director
      4. Set the Username to director and password to the password set in the earlier MySQL query
      5. Set Character Set to utf8
      6. Save Changes
    5. Navigate to the Director module on the web interface to finalize the setup process.
      1. Select the "director" resource when asked during the setup. Click Create Schema when asked.
      2. During the kickstart process, enter the FQDN for the server under endpoint name and enter the "director" API user credentials
    6. Setup the Director Background Daemon
      1. Create and setup the icingadirector user
        useradd -r -g icingaweb2 -d /var/lib/icingadirector -s /bin/false icingadirector
        install -d -o icingadirector -g icingaweb2 -m 0750 /var/lib/icingadirector
      2. Create the systemd service file under /lib/systemd/system/icinga-director.service with the following contents:
        [Unit]
        Description=Icinga Director - Monitoring Configuration
        Documentation=https://icinga.com/docs/director/latest/
        Wants=network.target
        
        [Service]
        EnvironmentFile=-/etc/default/icinga-director
        EnvironmentFile=-/etc/sysconfig/icinga-director
        ExecStart=/usr/bin/icingacli director daemon run
        ExecReload=/bin/kill -HUP ${MAINPID}
        User=icingadirector
        SyslogIdentifier=icingadirector
        Type=notify
        
        NotifyAccess=main
        WatchdogSec=10
        RestartSec=30
        Restart=always
        
        [Install]
        WantedBy=multi-user.target
      3. Reload systemd, enable and start the icinga-director service
        systemctl daemon-reload
        systemctl enable icinga-director
        systemctl start icinga-director
  10. Setup the Business Process module
    1. Install and enable the Icinga PHP Library (IPL) module required for the Buisness Process module.
      git clone --single-branch --branch stable/0.4.0 https://github.com/Icinga/icingaweb2-module-ipl.git /usr/share/icingaweb2/modules/ipl
      icingacli module enable ipl
      IDEA! Do not clone the master branch, only clone the latest stable branch on the github repo for IPL.
    2. Install and enable the Business Process module.
      git clone https://github.com/Icinga/icingaweb2-module-businessprocess.git /usr/share/icingaweb2/modules/businessprocess
      icingacli module enable businessprocess
Topic attachments
I Attachment History Action Size Date Who Comment
PNGpng icinga-stack.png r1 manage 443.0 K 2019-12-10 - 13:15 DevonMerner  
PNGpng icinga-stack2.png r1 manage 555.9 K 2019-12-10 - 14:01 DevonMerner  
PNGpng icinga-stack3.png r1 manage 555.3 K 2019-12-10 - 14:17 DevonMerner  
PNGpng logo_icinga1.png r1 manage 58.3 K 2019-12-10 - 13:15 DevonMerner  
Edit | Attach | Watch | Print version | History: r4 < r3 < r2 < r1 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r4 - 2020-01-28 - DevonMerner
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback