A while back I blogged about how I hooked up Nagios and Git to run the Nagios preflight checks before restarting with a new checkin’s worth of configs. But the more I looked at how it all fit together, the more I knew it could be improved. A sed hack, expecting a certain pattern in the nagios.cfg? Bad bad bad. Most of the improvement revolves around Nagios’s ability to reference relative paths for its config files. Given the path of the ‘main’ nagios.cfg file, you can then reference directories that contain your services, hosts, and other custom commands, in relation to that main file. With this functionality I significantly improved the Git->Nagios pipeline.
First, the pre-receive hook
#!/bin/bash umask 022 while read OLD_SHA1 NEW_SHA1 REFNAME; do export GIT_WORK_TREE=/tmp/nagiostest-$NEW_SHA1 mkdir -p $GIT_WORK_TREE /usr/bin/git checkout -f $NEW_SHA1 sudo /usr/sbin/nagios3 -v $GIT_WORK_TREE/nagios.cfg if [ “$?” -ne “0” ]; then echo “Nagios Preflight Failed” echo “See the above error, fix your config, and re-push to attempt to update Nagios.” exit 1 else echo “Nagios Preflight Passed” echo “Clearing temporary work directory.” rm -rf $GIT_WORK_TREE exit 0 fi done
Using the GIT_WORK_TREE environment variable, which specifies Git’s working directory, I check out the new set of potential configs to a temporary directory. This provides a temporary ‘waiting room’ for the proposed configuration to be tested, before before being put into production. Imagine never (intentionally) breaking Nagios again because of a broken host or service specification. The main thing remember is that all references in the nagios.cfg to other config files (hosts, commands, etc) must be relative paths. I.E., I have lines that look like “cfg_dir=configs” in the nagios.cfg. Note the lack of absolute paths. We now run the Nagios pre-flight check (nagios -v) on the nagios.cfg in the Git work tree. Depending upon the exit value of ’nagios -v’, 0 for success and 1 for failure, we either proceed or die immediately. If success, clean up our temporary run directory.
Now the post-receive hook:
#!/bin/sh echo “Updating repo /etc/nagios3” sudo /usr/bin/update-gitrepo /etc/nagios3
The post-receive hook merely runs a script, noted below, on the Nagios configuration directory.
Update-gitrepo:
#!/bin/sh umask 022 REPO_DIR=$1 cd ${REPO_DIR} /usr/bin/git pull origin master
Given the Git checkout’s directory, we fetch the most recent push to the repository.
For the final step we have to fix some permissions (given that my setup runs the repository through Gitolite as the git user). This hook is located in the actual checkout itself, /etc/nagios3, in the post-merge hook.
#!/bin/sh sudo chown -R nagios:admin /etc/nagios3 sudo /etc/init.d/nagios3 restart
A full commit and restart looks like this:
jforman@merlot:/mnt/raid1/personal/git/monitor/nagios/configs$ git push Counting objects: 7, done. Delta compression using up to 4 threads. Compressing objects: 100% (4/4), done. Writing objects: 100% (4/4), 414 bytes, done. Total 4 (delta 3), reused 0 (delta 0) remote: Previous HEAD position was c80fa03… turn off test notifications with notifications_enabled 0 remote: HEAD is now at f088dbc… Example: Add boilerplate header that file is managed by Git. remote: remote: Nagios Core 3.2.3 remote: Copyright (c) 2009-2010 Nagios Core Development Team and Community Contributors remote: Copyright (c) 1999-2009 Ethan Galstad remote: Last Modified: 10-03-2010 remote: License: GPL remote: remote: Website: http://www.nagios.org remote: Reading configuration data… remote: Read main config file okay… remote: Processing object config file ‘/tmp/nagiostest-f088dbcebf194edbce78068b6004cbbfca703432/commands.cfg’… remote: Processing object config directory ‘/etc/nagios-plugins/config’… remote: Processing object config file ‘/etc/nagios-plugins/config/ftp.cfg’… remote: Processing object config file ‘/etc/nagios-plugins/config/mail.cfg’… remote: Processing object config file ‘/etc/nagios-plugins/config/snmp_int.cfg’… remote: Processing object config file ‘/etc/nagios-plugins/config/nt.cfg’… remote: Processing object config file ‘/etc/nagios-plugins/config/http.cfg’… remote: Processing object config file ‘/etc/nagios-plugins/config/real.cfg’… remote: Processing object config file ‘/etc/nagios-plugins/config/check_nrpe.cfg’… remote: Processing object config file ‘/etc/nagios-plugins/config/snmp_storage.cfg’… remote: Processing object config file ‘/etc/nagios-plugins/config/disk.cfg’… remote: Processing object config file ‘/etc/nagios-plugins/config/mysql.cfg’… remote: Processing object config file ‘/etc/nagios-plugins/config/snmp_load.cfg’… remote: Processing object config file ‘/etc/nagios-plugins/config/fping.cfg’… remote: Processing object config file ‘/etc/nagios-plugins/config/dhcp.cfg’… remote: Processing object config file ‘/etc/nagios-plugins/config/ssh.cfg’… remote: Processing object config file ‘/etc/nagios-plugins/config/rpc-nfs.cfg’… remote: Processing object config file ‘/etc/nagios-plugins/config/mailq.cfg’… remote: Processing object config file ‘/etc/nagios-plugins/config/breeze.cfg’… remote: Processing object config file ‘/etc/nagios-plugins/config/dummy.cfg’… remote: Processing object config file ‘/etc/nagios-plugins/config/netware.cfg’… remote: Processing object config file ‘/etc/nagios-plugins/config/hppjd.cfg’… remote: Processing object config file ‘/etc/nagios-plugins/config/load.cfg’… remote: Processing object config file ‘/etc/nagios-plugins/config/mrtg.cfg’… remote: Processing object config file ‘/etc/nagios-plugins/config/apt.cfg’… remote: Processing object config file ‘/etc/nagios-plugins/config/snmp_cpfw.cfg’… remote: Processing object config file ‘/etc/nagios-plugins/config/snmp.cfg’… remote: Processing object config file ‘/etc/nagios-plugins/config/snmp_process.cfg’… remote: Processing object config file ‘/etc/nagios-plugins/config/snmp_env.cfg’… remote: Processing object config file ‘/etc/nagios-plugins/config/news.cfg’… remote: Processing object config file ‘/etc/nagios-plugins/config/ntp.cfg’… remote: Processing object config file ‘/etc/nagios-plugins/config/telnet.cfg’… remote: Processing object config file ‘/etc/nagios-plugins/config/users.cfg’… remote: Processing object config file ‘/etc/nagios-plugins/config/snmp_mem.cfg’… remote: Processing object config file ‘/etc/nagios-plugins/config/procs.cfg’… remote: Processing object config file ‘/etc/nagios-plugins/config/ifstatus.cfg’… remote: Processing object config file ‘/etc/nagios-plugins/config/games.cfg’… remote: Processing object config file ‘/etc/nagios-plugins/config/disk-smb.cfg’… remote: Processing object config file ‘/etc/nagios-plugins/config/tcp_udp.cfg’… remote: Processing object config file ‘/etc/nagios-plugins/config/snmp_win.cfg’… remote: Processing object config file ‘/etc/nagios-plugins/config/ping.cfg’… remote: Processing object config file ‘/etc/nagios-plugins/config/pgsql.cfg’… remote: Processing object config file ‘/etc/nagios-plugins/config/ldap.cfg’… remote: Processing object config file ‘/etc/nagios-plugins/config/flexlm.cfg’… remote: Processing object config file ‘/etc/nagios-plugins/config/dns.cfg’… remote: Processing object config file ‘/etc/nagios-plugins/config/radius.cfg’… remote: Processing object config file ‘/etc/nagios-plugins/config/snmp_vrrp.cfg’… remote: Processing object config directory ‘/tmp/nagiostest-f088dbcebf194edbce78068b6004cbbfca703432/configs’… remote: Processing object config file ‘/tmp/nagiostest-f088dbcebf194edbce78068b6004cbbfca703432/configs/timeperiods.cfg’… remote: Processing object config file ‘/tmp/nagiostest-f088dbcebf194edbce78068b6004cbbfca703432/configs/services.cfg’… remote: Processing object config file ‘/tmp/nagiostest-f088dbcebf194edbce78068b6004cbbfca703432/configs/commands.cfg’… remote: Processing object config file ‘/tmp/nagiostest-f088dbcebf194edbce78068b6004cbbfca703432/configs/hosts.cfg’… remote: Processing object config file ‘/tmp/nagiostest-f088dbcebf194edbce78068b6004cbbfca703432/configs/abstracts.cfg’… remote: Processing object config file ‘/tmp/nagiostest-f088dbcebf194edbce78068b6004cbbfca703432/configs/contacts.cfg’… remote: Read object config files okay… remote: remote: Running pre-flight check on configuration data… remote: remote: Checking services… remote: Checked 95 services. remote: Checking hosts… remote: Checked 10 hosts. remote: Checking host groups… remote: Checked 7 host groups. remote: Checking service groups… remote: Checked 0 service groups. remote: Checking contacts… remote: Checked 3 contacts. remote: Checking contact groups… remote: Checked 2 contact groups. remote: Checking service escalations… remote: Checked 0 service escalations. remote: Checking service dependencies… remote: Checked 56 service dependencies. remote: Checking host escalations… remote: Checked 0 host escalations. remote: Checking host dependencies… remote: Checked 0 host dependencies. remote: Checking commands… remote: Checked 181 commands. remote: Checking time periods… remote: Checked 4 time periods. remote: Checking for circular paths between hosts… remote: Checking for circular host and service dependencies… remote: Checking global event handlers… remote: Checking obsessive compulsive processor commands… remote: Checking misc settings… remote: remote: Total Warnings: 0 remote: Total Errors: 0 remote: remote: Things look okay - No serious problems were detected during the pre-flight check remote: Nagios Preflight Passed remote: Clearing temporary work directory. remote: Updating repo /etc/nagios3 remote: From monitor:nagios remote: * branch master -> FETCH_HEAD remote: Updating c80fa03..f088dbc remote: Fast-forward remote: configs/commands.cfg | 2 ++ remote: 1 file changed, 2 insertions(+) remote: * Restarting nagios3 monitoring daemon nagios3 remote: Waiting for nagios3 daemon to die.. remote: …done. To git@monitor:nagios.git c80fa03..f088dbc master -> master
Note that I do keep the Nagios package bundled commands in the /etc/nagios-plugins directory and have purposely not put those in the Git tree. This allows for updated Nagios packages from Ubuntu to update those commands accordingly without interfering with the Git repo.
Enjoy.