A new solution was called for, and the administrators came across Icinga and its lively community. Thanks to Icinga’s configuration compatibility, migration was relatively easy. Though to make future maintenance even easier, configuration files were reorganised to make full use of groups and inheritance. This way, adding a new machine into an existing category such as database, computing node, storage etc. simply meant changing one configuration file. Today, the LHCb experiment is monitored by a single Icinga instance in a failover setup combined with Mod-Gearman, NRPE and NSClient++. This includes few customised checks such as GPFS and file system speed, beside SNMP checks and other special performance measurements.

The central Icinga server schedules monitoring checks, which 60 distributed Mod-Gearman workers fetch from one of its queues, execute and push back their results to another queue (Figure 2). With this new setup, a single Icinga instance is able to monitor a massive environment of 2000+ hosts and 40,000+ services. To top it of, service check latency improved from an average of 328 seconds to less than 1 second.

CHECKING ON HIGGS BOSON

At a second and third site, detectors at the CMS (Compact Muon Solenoid) and ATLAS (A Toroidal LHC Apparatus) experiments look into the existence of Higgs Boson, extra dimensions and dark matter.

At CMS, Icinga monitors 3000 hosts and 70 switches with one central instance. Combined with a single Mod-Gearman worker, NRPE and check_multi, Icinga handles 90,000 checks every 2 minutes. Ranging from basic network usage, errors and disk utilisation to RAID arrays, equipment temperature and other special services, Icinga keeps an eye on the entire environment.

ATLAS on the other hand, uses a couple Icinga instances run on virtual machines alongside Nagios. Of the total 3000 hosts, these monitor about 90 critical ones on two networks. The monitoring setup helps ATLAS maximise use of LHC beam time, and harvest as much data as possible for physicists to analyse.