Icinga

Download the check_file_size.sh plugin to /usr/lib/nagios/plugins.

Test check_file_size.sh manually by choosing appropriate warning and critical threshold values:

sdi10b#> /usr/lib/nagios/plugins/check_file_size.sh /var/log --maxwarn 1000 --maxcrit 1500 
FILE Critical: Size of 1610 > 1500 for /var/log

sdi10b#> /usr/lib/nagios/plugins/check_file_size.sh /var/log --maxwarn 1000 --maxcrit 2000 
FILE Warning: Size of 1610 > 1000 for /var/log

sdi10b#> /usr/lib/nagios/plugins/check_file_size.sh /var/log --maxwarn 2000 --maxcrit 4000 
FILE OK: All files (1) fall within requested parameters

Prepare for remote execution. Create a directory /etc/nagiosBySsh and the following executable bash script /etc/nagiosBySsh/nagioscheckssh within:

#!/bin/bash

if [ -z "$SSH_ORIGINAL_COMMAND" ]; then
    echo "Environment variable «SSH_ORIGINAL_COMMAND» undefined"
else
    exec $SSH_ORIGINAL_COMMAND
fi

#end

This script will later be executed by a remote ssh call. The actual command to be executed will be provided by the environment variable SSH_ORIGINAL_COMMAND.

Simulate remote execution by using env for setting the desired environment variable:

sdi10b#> env SSH_ORIGINAL_COMMAND='/usr/lib/nagios/plugins/check_file_size.sh /var/log --maxwarn 1000 --maxcrit 1500' /etc/nagiosBySsh/nagioscheckssh

Depending on the chosen parameter values the resulting output should be similar to:

FILE Critical: Size of 1610 > 1500 for /var/log

For the sake of security a non-privileged account will be used for remote ssh execution. Create a system account nagioscheck for this purpose:

adduser --shell /bin/bash --system nagioscheck

As user nagioscheck create an ssh pair of keys using an empty pass phrase:

sdi10b#> su - nagioscheck
sdi10b$> ssh-keygen 
Generating public/private rsa key pair.
Enter file in which to save the key (/home/nagioscheck/.ssh/id_rsa): 
Created directory '/home/nagioscheck/.ssh'.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/nagioscheck/.ssh/id_rsa. ❶
Your public key has been saved in /home/nagioscheck/.ssh/id_rsa.pub. ❷
The key fingerprint is: ...

The private ssh key ❶ will later be used for remote query invocations. After moving it to the querying host it may deleted on the target host for security reasons.

Configure the corresponding public key ❷ to allow for ordinary remote ssh login:

sdi10b#> su - nagioscheck
sdi10b> cd ~/.ssh
sdi10b> cp id_rsa.pub authorized_keys

We now restrict remote logins to execution of /etc/nagiosBySsh/nagioscheckssh from 3. Modify /home/nagioscheck/.ssh/authorized_keys to contain:

from="141.62.75.110",command="/etc/nagiosBySsh/nagioscheckssh" ssh-rsa AAAAB3N...LKPFSJo5 nagioscheck@sdi10b

Explanation: ssh connections origination from 141.62.75.110 will receive the execution result of /etc/nagiosBySsh/nagioscheckssh. In particular no login shell will be provided.

Copy the target host's /home/nagioscheck/.ssh/id_rsa private key to /etc/icinga2/nagioscheck_id_rsa. Since the Icinga process running with the nagios user is requires read access we change the ownnership accordingly:

chown nagios.nagios nagioscheck_id_rsa

The Icinga daemon runs with the effective user's id nagios.We thus have to assure remote ssh login as user nagios:

ssh -i /etc/icinga2/nagioscheck_id_rsa nagioscheck@sdi10b.mi.hdm-stuttgart.de \
  "/usr/lib/nagios/plugins/check_file_size.sh /var/log \
  --maxwarn 1000 --maxcrit 1500"
The authenticity of host 'sdi10b.mi.hdm-stuttgart.de (141.62.75.120)' can't be established.
ECDSA key fingerprint is SHA256:4L5rfTIJPu1lr1gpTyvaywDE01W55roZjNGKkni/060.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'sdi10b.mi.hdm-stuttgart.de,141.62.75.120' (ECDSA) to the list of known hosts.
du: cannot read directory '/var/log/samba': Permission denied
du: cannot read directory '/var/log/unattended-upgrades': Permission denied
FILE Critical: Size of 1742 > 1500 for /var/log

Explanation: Remote command execution of /usr/lib/nagios/plugins/check_file_size.sh as user nagioscheck lacks privileges to completely read the entire /var/log directory tree. This will be fixed in the next step.

Turn back to your target server and allow sudo execution of /usr/lib/nagios/plugins/check_file_size.sh to user nagioscheck without providing a password. Start by:

sdi10b#> apt-get install sudo

Subsequently create a file /etc/sudoers.d/nagios to contain:

nagioscheck     ALL=NOPASSWD : /usr/lib/nagios/plugins/check_file_size.sh

You should now be able to initiate sudo privileged execution from your Icinga host:

ssh -i /etc/icinga2/nagioscheck_id_rsa nagioscheck@sdi10b.mi.hdm-stuttgart.de \
  "sudo /usr/lib/nagios/plugins/check_file_size.sh /var/log \
  --maxwarn 1000 --maxcrit 1500"
FILE Critical: Size of 1726 > 1500 for /var/log

Configure an Icinga check command in conf.d/commands.conf:

object CheckCommand "by_ssh_file_size" {
  import "by_ssh"

  vars.by_ssh_command  = "sudo /usr/lib/nagios/plugins/check_file_size.sh --maxwarn $by_ssh_file_size_warn$ --maxcrit $by_ssh_file_size_crit$ $by_ssh_file_size_path$"
  vars.by_ssh_identity = "/etc/icinga2/nagioscheck_id_rsa"
  vars.by_ssh_logname  = "nagioscheck"

#  Parameters by_ssh_file_size_warn, vars.by_ssh_file_size_crit and
#  by_ssh_file_size_path will be defined in service or host definition
}

Define an Icinga template in conf.d/services.conf:

apply Service for (path => config in host.vars.paths) {
  import "generic-service"

  check_command = "by_ssh_file_size"

  vars += config
}

Finally add two suitable host service checks in conf.d/hosts.conf:

object Host "sdi10b.mi.hdm-stuttgart.de" {
  import "generic-host"

  address = "sdi10b.mi.hdm-stuttgart.de"

  vars.paths["/var/log size"] = {
    by_ssh_file_size_warn = "500"
    by_ssh_file_size_crit = "5500"
    by_ssh_file_size_path = "/var/log"
  }

  vars.paths["/var/lib size"] = {
    by_ssh_file_size_warn = "50000"
    by_ssh_file_size_crit = "900000"
    by_ssh_file_size_path = "/var/lib"
  }

  vars.notification["mail"] = {
    groups = [ "icingaadmins" ]
  }
}