in    1 minute read

This morning, before getting ready for the day, I took a few moments as usual to check the status of the monitoring, expecting DZ-ARN-01 to have reached “green” Availability and Reliability. This was indeed the case, but I noticed two things when poking about :

  1. MA-01-CNRST had 11 alarms against it - this looks like some kind of drastic failure at the site, perhaps an electrical outage over the weekend.
  2. The nagios messages were not being sent to Slack, even though I had spent some time last week configuring this.

Somewhat miffeed, I went to take a look at the nagios probes running on the AAROC nagios instance , mutting unspeakable epitaphs under my breath. Lo, I was greeted with

Damn right I believe this to be in error !
Horrified, I immediately thought that there had been a break-in. However, after a quick check it was clear that everything was in order, except for the fact that 
                      df -hFilesystem            Size  Used Avail Use% Mounted on/dev/mapper/VolGroup00-LogVol00                      xxx G  xxx G     0 100% //dev/hda1              xxx M   xxx M   xxx M  xxx % /boottmpfs                 xxx G     xxx   xxx G    xxx % /dev/shm
Yep, that'll do it...
Hulk smash full disk ! Actually wait, Hulk diligently clean full disk
A quick check showed that it was the rejected messages director in `/var/spool/nagios2metricstore` that were the offending culprits. I cleaned this out and restarted stuff. We're still not getting messages into Slack, but at least the monitoring is back up.Great way to start a Monday - Bruce

 Tagged with  •   •   •   • 

Next Entry : Reward and Recognition (brief)
Previous Entry : Science: Needs more web