Goodbye UKI, hello NGI_UK

At All Hands 2011, in the atrium of the University of York's brand new Ron Cooke Hub conference venue.

On our stand in the middle of the room is a familiar face - helpdesk manager John Kewley - sitting under a slightly less familiar sign.

It doesn't say NGS, or GridPP, although both have posters on display.

The sign says but 'NGI' - aka National Grid Infrastructure - and we have had to to get used to it very quickly.

At  last week's EGI technical forum, what was the UKI ROC - or the UK and Ireland Regional Operation Centre - was offically replaced by two new NGIs called NGI_UK and NGI_IE.

And lots of things broke - including the load monitor and the Nagios testing service.

Names matter. Both the load monitor and Nagios were pulling information about sites and users from the Grid Operations Centre Database. More specifically, they will pulling information about sites and users associated with the UKI ROC.

The UKI ROC is no more: it has no sites or users associated with it.

So... we have spent the last few days tracking down every reference to the 'UKI' in every configuration file for every service and replacing them with NGI_UK.

There were quite a few....

The load monitor is back. We've been working on Nagios today and it should be fully working soon.

On the 97th of April 2011...

Back in February, an over-optimistic fool promised that the NGS would have a working Nagios service in the next few weeks.

The over-optimistic fool was confident because he had a real deadline to meet. Nagios had to be ready by April. April was the month during which the old NGS core sites - which ran the tests for our old INCA-based testing framework - were to be decommissioned.

We are running little late... but I am pleased to say that 2 weeks ago - on Wednesday the 97th of April 2011 - the NGS's Nagios testing service finally went live.

If you have an certificate and it is listed in the Grid Operations Centre database - you can pay it a visit at https://nagios01.ngs.ac.uk/nagios.

If you haven't or aren't - sorry: WLCG Nagios, unlike INCA, denies access to unregistered users by default. We may be able to remove the restriction in future - but, for the moment, we want to focus on fixing the problems it has found.

It is a bit untidy - as we have been without a fully working monitoring service for over 6 months.

While we kept the INCA service running as long as possible, it had become increasingly out of step due to a decision - very early on - to use the 'NeSCForge' software repository as a safe place to keep its configuration.

NeSCForge was not as safe as we had hoped. It vanished in December last year. The list of sites and tests to run remained frozen in their December state... and the Grid moved on.

We have different partner sites offering different services now. INCA wasn't testing them, Nagios is.

More significantly, Nagios takes its list of sites directly from the Grid Operations Centre database. Changes made there should be reflected in Nagios within a day.

My colleagues in the NGS Partnership team are working their way through the Nagios test results. They are identifying problems, finding missing sites and services - and, most importantly, working out how to make things better.

Delayed

In the best traditions of a UK railway station announcement... The NGS would like to apologise for the late arrival of last week's R+D blog posting. This was due to the non-arrival of some good news.

Over the last few weeks: we have made good progress with our Nagios service and with persuading our Workload Management Service to accept certificates generated using institutional credentials and the SARoNGS service.

Unfortunately, the good progress wasn't quite good enough to deliver something that actually worked.

It shouldn't take much longer. Honest.

Good news, bad news

It isn't a case of one step forward... two steps back.

Its more one step forward... with another step forward coming soon.

This week, both the Leeds CREAM-CE and the NGS's Nagios project inched forward.

CREAM

We started with good news thanks to a comment from Ewan at Oxford on Leeds' plans to install a grid front end to our ARC1 High Performance Computing service.

Ewan pointed out that our preferred approach - leaving the grid access on a machine almost-completely-detached from the HPC service - is a) also other people's preferred approach and b) one that actually works.

Which is nice.

And would be nicer if it wasn't for the bad news: Sun Grid Engine support in CREAM never made it as far as the first major release (EMI-1) of the European Middleware Initiatives's grand unified grid software.

Grid Engine support is expected to arrive in a minor release - coming soon.

NAGIOS

We have had a working Nagios development system for some time.

We were trying to build a working 'clean' test system. We were planning to use this to practice the full Nagios install and configuration procedure before being let loose on a proper service.

And when we first practiced - a month or so back - the test server refused to install anything.

Since that time progress has been slow. We blame this on the stubborn refusal of the average day to include more than 24 hours, so cruelly depriving the systems staff of enough time to finish everything else that needs to be done.

Earlier this week, after reading the latest Nagios installation documentation, and comparing notes with the Nagios developers and our colleagues at Oxford who run the GridPP Nagios - we worked out what had gone awry.

There were some unfortunate conflicts between packages in the software repositories defined in /etc/yum.repos.d. We ended up in RPM hell...

Its better now. And as a minor bonus we did developed a utility that can edit YUM repositories in place. It can be found in the UKNGI subversion repository at SourceForge. It isn't pretty or clever but it does work...

See SPOT run

Just when you thought it was safe to go back to the Internet, up pops another Acronym.

Meet SPOT.

You won't - yet - find SPOT in the Grid Acronym Soup because it is not one of ours. It is an escapee from another of the great sources of Acronyms - the world of IT Business Systems.

SPOT stands for Single Point Of Truth.

A SPOT is not a single genuine fact that slipped into a Business Systems sales pitch. Nor does the name imply that Business Systems remind those involved of an inflamed, infected ball of pus. SPOTs are, in general, good.

If you have a SPOT, then you know there is one-and-only-one definitive source for any piece of information - whether it is a price, a name, a salary or an office number.

As anyone who has dealt with a large organisation will appreciate, we do not have as many SPOTs as we should.

On the Grid - which is large, diverse and dispersed by its nature - single points of truth are very hard to find.

Which makes deciding what we should test with Nagios... interesting.

The NCG configuration generator which writes the Nagios configuration needs to know:

• What sites to test.
• What services to test at those sites.
• What tests to run for each service.

Perhaps the closest we have to a SPOT is the Grid Operations Centre Database or GOCDB - which lists every site on the European Grids, their official downtimes and some of the services they provide.

The `some of' is there because the GOCDB defines services in terms of service endpoints - which represent a host within a site acting as, say, a Compute Element or a GSISSH server or a GridFTP server.

There are a comparatively small number of predefined endpoints and these will never cover everything a site can offer - you cannot, for example, advertise an iRODS service.

The GOCDB does not directly provide information about the Virtual Organisations that a service is prepared to support but it should point anyone wanting this information at a site information service willing and able to provide it.

For our first attempt at a WLCG-like Nagios service...

• We collect a list of sites come from the GOCDB - we take any site flagged as belonging to NorthGrid, SouthGrid, Scotgrid or the London Tier 2 subgrids within the UK and Ireland Region.
• We only test services for which GOCDB service endpoints are defined.
• We define the tests for each endpoint within the Perl code of NCG. There is a 'standard' set of tests defined within a perl module called NCG::LocalMetrics::Hash which forms part of the NCG package.
  We modified the module to include local changes from a NCG::LocalMetrics::Hash_local module - a change that has been adopted by the NCG maintainers.

As an approach, it works well enough for Nagios tests but not for the friendly-front-end MyEGI.

MyEGI gets its truth from elsewhere: from the the Aggregated Topology Provider (ATP). The ATP is a sort of single point of single points of truth. It swallows data from the Metric Description Database (MDDB) and from Virtual Organisation feeds. It is scarily complicated in places - as you might be able to gather by looking at the MDDB and ATP database schema

The European Grid Initiative exists to make the grid work better, in part by giving us nicer SPOTs and are encouraging development of the ATP and friends. The curious can find out more on the SAM and Nagios wiki pages at CERN.

The Single Point of Truth is Out There....

Missing the message bus

[With thanks to Konstantin Skaburskas.]

Two weeks ago, we were very nearly at the point where we could deploy WLCG Nagios and phase out our existing testing service.

We had created our own tests and worked out how to add them to the bundle of tests that are sent out onto the Grid.

The tests were actually being run on remote sites.

All that was missing was - well - a big chunk of the test results.

When the tests landed on gLite-based sites - everything worked as expected. In other places - the tests ran... but resolutely refused to let anyone know the results.

We have now found the missing messages - after losing them twice on the way.

It was all due to subtle differences in the environment variables defined at a site. The fix is to set two environment variables by adding something like...

 Environment = {
   "OSG_HOSTNAME=<jdlreqceinfohostname>",
   "LCG_GFAL_INFOSYS=bdii.ngs.ac.uk:2170"
 };

to the template used to generate the JDL file that describes the test.

To understand why, you need to understand how the tests on remote hosts are run. The hard work is done by a script called nagrun.sh - that:

• unpacks the bundle of tests and configures them for the local machine.
• runs them using a bundled copy of nagios
• translates the test results into messages.
• sends the messages to a message broker - which shoves them on the message bus back to the Nagios server.

At the Nagios server end, each message is unpacked and fed to the central Nagios as a passive test result.

If LCG_GFAL_INFOSYS is missing, the tests never make it to the message broker; if OSG_HOSTNAME is missing - they are ignored when they reach the Nagios server.

It is the WLCG Nagios for a reason - it was designed to test machines that sat within the Worldwide LHC Computing Grid. One of its roles was to serve as a replacement for the older 'Service Availability Monitoring' (SAM) tests.

It makes the - perfectly logical - assumption that the environment on the machine running the tests will be like that used for the SAM tests.

WLCG has a dedicated network of message brokers. Any host can find a suitable broker to contact by asking its friendly local information service. The environment variable LCG_GFAL_INFOSYS points to the information service.

Some of the sites we are testing sit outside WLCG. We have our own message broker and pass additional information with the tests to direct messages to it.

A subtle bug, which is been fixed in the current release, meant that even though LCG_GFAL_INFOSYS was not being used, it still had to be set. If it wasn't, nagrun.sh could not find a message broker to contact.

So, the messages were making it back to the Nagios server. The Nagios server was ignoring them.

The reason: the messages sent back are meant to include a reference to the Compute Element (CE) that actually accepted the job. The messages we were sending were all being marked as coming from 'localhost.localdomain' - a dummy name used internally by the nagios tests.

The nagrun.sh script tries to work out the CE name from the local environment and from the output of certain scripts. If all else fails, it assumes nagios knows the answer.

This WLCG Nagios developers had encountered this problem before - when running ATLAS tests against hosts on the US Open Science Grid - and had added code that allows a Open Science Grid hostname to be used as a CE name. It expects the environment variable OSG_HOSTNAME to hold that hostname.

We can also report that the WMS administrators have reconfigured the server so it no longer gets clogging up with CREAM jobs - and the CREAM CE tests are now running via the WMS - as WLCG intended.

We are now ready to deploy WLCG Nagios - unfortunately without the MyEGI friendly front end - and make it available to site administrators.

We will describe how we decide which sites to test and what tests to run in a future posting.

At which point, Nagios related Research and Development will take a break and I will have to find something else to prattle about every couple of weeks.

Adding NGS tests to WLCG Nagios

When we last mentioned the NGS project to deploy WLCG Nagios, we had most of the basic WLCG tests running against 'classic' and 'CREAM' compute elements.

We are now extending the WLCG code with some NGS-specific tests.

In particular, we are adding to the set of tests that are run on individual worker nodes as part of the 'CE', and eventually the 'CREAM-CE' tests.

This is not exactly a common requirement, so documentation is understandably sparse. The best place to start seems to be https://twiki.cern.ch/twiki/bin/view/LCG/PracticalHintsForMigrating2Nagios

The test we will use to test the testing service is deliberately simple. It is a Nagios-style plugin that checks if a site supports the 'Uniform Execution Environment' conventions. It looks for a /usr/ngs directory. If it is missing, this is an error, if it is empty, this warrants a warning, otherwise everything is OK.

We know that WLCG-Nagios uses a mixture of active and passive tests. Active tests deliver results immediately while the results of passive tests filter in slowly via the message broker.

Our initial plan was to extend the CE-probe tests. The CE-probe works by...

• building a compressed tar file containing some nagios tests, a copy of nagios to run them, and bits of python to deliver the results to the message broker.
• generating a JDL that describes how to fetch the tar file and run the tests within it.

The key is a script called nagrun.sh which runs nagios on a remote machine, collects the test results and throws them at a message broker. The broker should deliver them to main nagios server where they reappear as passive test results.

The CE-probe allows additional directory trees to be added to the tar file, as long as they look rather like...

/usr/libexec/grid-monitoring/probes
|
`-- uk.ac.ngs
   `-- wnjob
       |-- uk.ac.ngs
       |   |-- etc
       |   |   `-- wn.d
       |   |       `-- uk.ac.ngs
       |   |           |-- commands.cfg
       |   |           `-- services.cfg
       |   `-- probes
       |       `-- uk.ac.ngs
       |           `-- WN-uee
       `-- uk.ac.ngs.gridJob.jdl.template

This is mostly directories and subdirectories. Real files are marked in bold: WN-uee is the test script, the *.cfg files are nagios configuration files describing how to run it; the *.jdl.template file is used when writing the JDL.

Eagle-eyed readers may have noticed lots of uk.ac.ngs's scattered around.

This serves as a convenient namespace - it exists to stop files in this directory tree inadvertently overwriting those from another tree when the tar file is being created.
The convention used in WLCG Nagios is that the namespace should be your organisation written backwards. Argue not will I.

Incorporating the new directories involves adding extra arguments to the CE-probe

--add-wntar-nag-nosamcfg
--add-wntar-nag /usr/libexec/grid-monitoring/probes/uk.ac.ngs/wnjob/uk.ac.ngs
--jdl-templ /usr/libexec/grid-monitoring/probes/uk.ac.ngs/wnjob/uk.ac.ngs.gridJob.jdl.template

The first of these turns off the standard WLCG 'SAM' tests. GridPP nagios service is already checking those.

At the time this blog post was being written, a grand total of one site has passed - congratulations Glasgow Scotgrid - and it flagged up a few sites that do not provide UEE application.

While we can claim one successful success and several successful failures, there are a lot of sites where the results have yet to arrive.

These laggards include all the old core NGS sites - all of which support UEE, but use Virtual Data Toolkit rather than gLite for grid software. We have tested the test-test on one of these sites and know it works. The next step to to find out why the results are getting lost on the way home.

Three months of basketweaving with Elvis and Maradona.

A little over 3 months ago, we started a project to replace NGS's INCA monitoring service with WLCG Nagios.

This was never going to be easy. There are partner sites in the NGS provide that services that are almost - but not entirely - completely unlike those expected by the Worldwide LHC Computing Grid.

It has been a long, and sometimes tedious, process - documented in long, and no doubt equally tedious, posts on the NGS blog.

We would never have got this far without the WLCG Nagios developers. They have offered advice and produce helpful documentation, fixed quirks and, above all, written code that may be complicated but remains readable and comprehensible.

Over the three months, we have learned how to persuade Nagios to run the tests we need; we have learned how to get a message bus through a firewall, even when said firewall denies that the machine being tested is alive; and we've only broken something important once - when we accidentally clogged up the WMS service with CREAM.

This week, for the very first time, all the bits of the service worked together. We let Nagios run its tests and saw (some of) the results in the MyEGEE and MyEGI 'portals'.

So what took us so long....? Well, we'd accidentally crashed a bus into the database.

The WLCG Nagios software is based around a message bus. Any time anything interesting happens, a message is pushed onto the bus. This relies on a command assigned as a Nagios event handler and the slightly-disturbingly-named 'obsessive compulsive' option that ensures this command is run whenever an interesting test result arrives.

At the same time, a dedicated bus spotter, called msg-to-handler, watches for incoming messages from the bus and stores them in directories on the local disk. Special Nagios plugins checks the directories, react to incoming messages and possibly creating new messages in the process.

The MyEGEE and MyEGI portals are, crudely speaking, pretty views of a complicated MySQL database. They rely on plugins run periodically by Nagios to update the database with details of tests and results.

The messages were arriving. The plugins were running. The database was not being updated.
There were many reasons.

In part, we were simply behind the times....

As WLCG Nagios has developed, the database schema has changed. Earlier versions stuffed different categories of information into separate databases - called things like 'atp', 'metricstore' and 'mddb' - associated with different users and passwords. In newer ones, all the information is kept in one database called 'mrs'.

Some out of date entries in the configuration files for the YAIM configuration tool meant we had components using old style database names instead of the all-conquering mrs.

Fixing the databases names brought us to the point where test results were being processed.
The bad news was that the results they were being rejected in processing.

Yet, it is to the developers credit that WLCG Nagios handles rejection well. Duff data is dumped in special SQL tables - with names ending in 'rejected' - with a reason column explaining what went horribly wrong.

In our case, it was because we had information on test results but no information on tests.
We were missing the data from one vital message - one generated by the NCG configuration generator to announce the safe arrival of a new configuration.

To get the message, we needed to add

<ncg::configpublish>
<configcache>
NAGIOS_ROLE=ROC
VO=ngs.ac.uk
</configcache>
</ncg>

to the ncg.conf configuration file and ensure that the /usr/sbin/mrs-load-services script was run.

And - for some tests, under some circumstances, for certain sites, with a following wind, on a good day - the results appeared.

ICE and too much CREAM

If these is one area where the Grid community excels, it is in the creation of acronyms. The 600-odd entries on GridPP's Grid Acronym Soup page include a FIreMan, two kinds of GENIUS and a PanDA.

This post is brought to you by the acronyms ICE and CREAM - which are not yet in the Soup but are widely deployed by GridPP.

ICE is nothing to do with the white stuff covering most of the UK - it stands for Interface to CREAM Environment; CREAM is Computing Resource Execution And Management.

CREAM provides an alternative, web-service-y, interface for submitting jobs to a compute cluster. ICE allows CREAM services accept jobs from resource brokers such as the NGS's UI/WMS service.

The NGS deployment of WLCG Nagios is having problems swallowing ICE and CREAM.

Our plan is to replace the tests run from the existing INCA service with similar tests from WLCG Nagios. The INCA tests use credentials associated with the ngs.ac.uk Virtual Organisation when submitting jobs, so our Nagios instance is doing the same.

This contrasts with the GridPP Nagios deployment which uses the CERN Ops VO when testing.

Ops membership is acccepted anywhere that processes the CERN data. The ngs.ac.uk VO is accepted, at least in part, by all NGS member and affiliate sites. These include many GridPP sites as well as a number of sites who really aren't bothered by the Higgs Boson.

This is where it gets complicated.

When monitoring a whole region, WLCG Nagios does not submit tests directly to the sites. It passes them to a WMS resource broker where they queue until the site is ready. If the site takes too long to respond, the WMS is told to cancel the test.

Tests aimed at 'Classic' Compute Elements are running. The sites run the tests and, after some tweaks at STFC, we are now able to collect test results from the message bus.

Tests aimed at CREAM services are not running. Worse still, they get stuck in a strange state where cancellations are ignored. Under these circumstances, the CREAM testing bit of Nagios sends another cancellation request... and another... and another...

Eventually the cancel requests clog up the resource broker.

We are not yet sure why CREAM based services and the NGS do not get along.

GridPP people who came to a recent NGS Surgery suggested that it might simply be the presence of an email address in our VO certificate's distinguished name. Comparing distinguished names is far more complicated that it appears and embedded email addresses, in particular, cause no end of hassle.

We've turned off the WMS CREAM tests for now and replaced them with ones sent directly from the Nagios server.

After all no-one wants a broken broker.

[Update: 8-Dec-2010. The Grid Acronym Soup now includes both ICE and CREAM. I suppose this turns it into a Gazpacho.]

Failing more succesfully - getting past Maradona and Condor

It has been nearly a month since the last progress report on Nagios. Which is a shame, because in that time we have made something that looks rather like progress.

The NGS's development Nagios server was at the point where it was throwing tests at NGS partner sites.

The simpler tests - for things such as service certificates reaching their expiry date - are working.

We have have less success with the more sophisticated tests - like those that poke every nook and cranny of a Compute Element.

A few sites - notably those in Scotgrid - are accepting the tests and running them to completion but we only see part of the results. For others sites we get the infamous

Standard output does not contain useful data.Cannot read JobWrapper output, both from Condor and from Maradona.

error message.

In both cases, the same test - the CE-probe - is involved. This is thrown at all sites that advertise Compute Elements in the GOCDB database of all things griddy.

This test makes use of the Nagios concepts of active and passive tests. In an active test, the Nagios service runs some bit of code and expects that bit of code to provide a result. In a passive test, there is no explicit test code and results are fed in by whatever means necessary.

The CE-probe appears within Nagios as one active test and a whole raft of passive ones. The active test delivers a bundle of tests to the site - via a Workload Management service (WMS) - and checks on its progress. At various stages in the life of the bundle, the passive tests results are updated.

Some passive tests results are generated from the Nagios server itself; others are sent directly from the system under test via the next available Message Bus.

When the bundle of tests runs successfully, we see the results generated from within the nagios server but not those coming from the message bus. This is because the development service uses a message broker that sits outside the core set of brokers used by WLCG. A workaround for this is coming any day now.

The Maradona message appears when the bundle of tests doesn't run at all.

It is a by-product of the script generated within the WMS and sent on to the site and, in particular, how this script handles 'Shallow' resubmission.

A shallow failure is one where the job is rejected and can be tried elsewhere. The WMS touts the job around the grid until it finds a system prepared to accept it. Acceptance is signified by the deletion of a marker file using GridFTP.

Which is all very well, as long as the machine on which the script is running has software that is able to delete a file using GridFTP.

gLite-based systems usually have something , those using the NGS VDT based installer do not. If this step fails, the script gives up early and prints the Maradona message.

A VDT based system can be persuaded to run the WMS-generated script by installing the UberFTP tool using

  pacman -get http://vdt.cs.wisc.edu/vdt_181_cache:UberFTP

(Pick a different cache if you are using something other than the elderly version 1.8.1 of VDT.)

UberFTP provides enough GridFTP support to allow the bundle of tests to run - though we have yet to persuade them to run to completion. I would call that a more successful failure.

Anyone attending the HEPSYSMAN meeting in Birmingham on 22 November will have the opportunity to hear, and ask questions, about what we needed to do to persuade WLCG nagios to work on the weirder bits of the NGS.

[Edit 2010-11-24 fixing typos]

I'm a monitoring service, let me out of here

When we last covered the development of the new Nagios monitoring service in the blog - before last week's commercial break - we had just convinced it that all the hosts were alive and ready to be tested.

We can now proudly say that we have coaxed the service towards its first, official complete and utter failure. 

All those highly motivated people who tell you`failure is not an option' - ignore them. If you are running service that tests things, having a test fail means that you actually persuaded that test to run. It isn't failure, it is a different kind of success.

And it is not as easy as it sounds because the Nagios development server is, quite deliberately, kept isolated from the rest of the world. 

This is not a reference to the Harwell Science and Innovation Campus near Didcot: where the people from the STFC e-Science centre who run the service are based, and where the NGS Innovation Forum 2010 will be held.

It is simply that the Nagios development server has limited Internet access - as befits an experimental service. All access to the World Wide Web must be channeled through a web proxy. Privileged access to services is granted only when needed.

Neither the NCG configuration program or the various tests and probes that Nagios uses were written for an environment with a web proxy. Much of the code is written in Perl and support for proxies is already present -it just needed to be turned on. The Nagios developers at CERN have already accepted the changes for the next release.

With web access granted, NCG could build a complete configuration and the tests that suck information from web sites all began to run.

The next problem was getting permission to do things.

This is a grid. To use a grid, you need a certificate. WLCG Nagios has the wherewithall to download a certificate from a MyProxy Credential Management Service - as long as someone has uploaded it in the first place and there is no passphrase required.

The NGS provides a central MyProxy service and MyProxy allows certificates to uploaded so that they can be downloaded using another certificate as authentication. The command to do this isn't exactly short:

env GT_PROXY_MODE=old myproxy-init -s myproxy.ngs.ac.uk -l nagios_dev -x -Z '/C=UK/O=eScience/OU=CLRC/L=RAL/CN=nagios-dev.ngs.ac.uk/emailAddress=sct-certificates@stfc.ac.uk' -k nagios_dev-ngs -c 336

.. but it works.

Or at least it worked after we had added the certificate DN to the authorized_receivers and trusted_receivers entries in the myproxy server configuration file.

... and ensured that the ngs.ac.uk virtual organisation was defined on the Nagios server.

So at long last, the Nagios server could download a certificate, associate it with a virtual organisation and use it to submit jobs via a Workload Management Server.

Which was the point at which we realised that the Workload Management Service endpoint (https://ngswms01.ngs.ac.uk:7443/glite_wms_wmproxy_server) should have been defined in the glite_wms.conf and glite_wmsui.conf files in $GLITE_LOCATION/etc/ngs.ac.uk/.

With that final hurdle overcome, the test jobs started to flow.

The Compute Element tests were sent sites declaring themselves as Compute Elements - including the original NGS core sites at Leeds and RAL.

I admit to rigging it so that Leeds was tested first. The little status box went Green as the job was submitted, then Red as it failed with a friendly:

- Standard output does not contain useful data.Cannot read JobWrapper output, both from Condor and from Maradona.

Sometimes an Argentinian footballer and a large scavenging bird can make your day.

It's alive

If you work at one of those institutions that let users choose names for their own computers: it is inevitable that, sooner or later, someone will claim the name 'Elvis'.

Occasionally, this will be because they like his music. In most cases, it is because they want to be able to run 'ping elvis' and be told that, despite the events of August 16 1977:

 elvis is alive [*]

'Ping', the friendly name of an ICMP echo request packet, was invented as a way of testing network connectivity. The original idea was that if a machine on the Internet was working and it was pinged, it should send back the contents of the 'ping' to the sender as an ICMP echo reply.

Ping dates from when the Internet was a smaller and nicer place. These days, pings are seen as a security issue and are frequently blocked at campus and departmental firewalls.

So, once Elvis has left the building, you may never know if he is still alive.

This is a real issue for the current WLCG Nagios deployment. The Nagios service is hosted at the Rutherford Appleton Laboratory but the sites that make up the NGS and GridPP are spread around the country behind many different firewalls. Some hosts can be pinged from off site, some can't.

Nagios has the concept of hosts and services provided by those hosts. It will only check services if the associated host is working. The usual way of testing a host is by sending a ping. If pings are not permitted, no service on that host is tested.

The NCG utility that generates Nagios configurations can use a dummy test in place of a ping test for all hosts. To enable this, the /etc/ncg/ncg.conf configuration file needs to be changed to include a line setting CHECK_HOSTS to zero:

<NCG::ConfigGen>

  <Nagios>
  ...
  # Disable 'ping' checks of hosts
  CHECK_HOSTS=0

  </Nagios>
</NCG::ConfigGen>

When ncg is run and a new Nagios configuration built, all services on all hosts are tested. On the down side, If a host really has dropped off the network, Nagios will continue to test the services and generate alerts.

Now, if you will excuse me, I better stop writing about Nagios and go back to configuring it. As someone once said: a little less conversation, a little more action please.

[*] For the pedants.. the ping command on Linux run until you stop it and print statistics. You will need to find a Solaris system or a Cisco switch to actually see this message.

Nagios, MyEGEE and MyEGI

[With thanks to Cristina Del Cano Novales at STFC]

The story so far... early last month we started to deploy WLCG Nagios as a replacement for the existing INCA testing service.

STFC set up the servers and deployed the latest published version of WLCG Nagios before handing the baton to Leeds. Leeds are configuring Nagios and the ecosystem of software that supports it.

And it is a there is a lot of supporting software.

In addition to Nagios itself, the NCG configuration tool and the plugins that actually do the work - all of which were described in a previous post - the WLCG package includes..

• pnp4nagios to keep and display historical records of performance metrics.
• an ActiveMQ message bus to pass data to wherever it needs to be. STFC have cunningly configured this on the development box so that it can only talk to itself.
  
• a MySQL database to keep status information .

There are two web tools called MyEGEE and MyEGI that allow people to extract the bits of information in that database that are relevant to them.

Both are built on the kind of general purpose frameworks that have sprung up since people started bandying around the term 'Web 2.0' as if it meant something. Much of what is seen as Web 2.0 is - at its heart - as some way of reading, or updating, a database from a suitably interactive and pretty webpage. Frameworks to do database wrangling are available for all popular languages and a quite a few unpopular ones.

MyEGEE is written in PHP on top of Zend Framework; MyEGI is python and based around Django.

According to a presentation at the EGI Technical Forum - highlighted by one of our colleagues from STFC - MyEGI will replace MyEGEE in the next few months. The developers are expecting to produce the first official MyEGI release in November.

The Featherstone-Kite Openwork Basketweave Mark Two Gentleman’s Flying Machine

In a shopping centre in the middle of Leeds, not far from the Universities, there is - or was - a big glass case.

In the case is a contraption built from wicker-work, string and cogs and lights and bits of old gramophone. Every so often, it springs to life and whirls around and plays a tune.

It is not an Yorkshire-based competitor for the iPod but a sculpture by Rowland Emmet: "The Featherstone-Kite Openwork Basketweave Mark Two Gentleman’s Flying Machine". Leeds shoppers passing by look at it and think...

What on earth is THAT meant to do?

Which is a rather tortuous way of introducing the latest bit of R+D work.

We are investigating how to move the important features of the existing INCA monitoring service to a new monitoring service based on WLCG Nagios.
But - as has been said many times - grids are complicated. Which means that the software needed to monitor grids is complicated. Which means that when you start to look at the software, you spend a lot of time staring at a screen and thinking...

What on earth is THAT meant to do?

So after a week of staring and thinking, here is what we think the bits and pieces of the service are meant to do:

At the core sits Nagios: an open-source monitoring system familiar to many system administrators. It consists of a set of programs called 'plugins' and a scheduler that arranges for these plugins to be run.

A plugin tests if a particular service on a given host is working as expected. Plugins typically return a short message and a status code that means one of: 'OK', 'WARNING', 'CRITICAL' or - if the plugin broke - 'UNKNOWN'. They can also track performance data such as disk usage.

Nagios comes with a set of basic plugins. WLCG Nagios adds a whole raft of Grid specific ones.
In this documentation, plugins within WLCG are referred to as probes.

Next up, a 'configuration generator' called NCG takes data published about a site or set of sites and generates a configuration for Nagios that monitors them.

Statistics and performance metrics generated by the plugins/probes are collected and are delivered via a message bus to a service that stuffs them into a database. A tool called MyEGEE is used to visualise the contents of this database.

If you want to know more...

Staff from STFC and Oxford gave an NGS surgery on WLCG Nagios in late July this year. Their slides describing how WLCG Nagios can be configured and how it has been deployed can be found on the NGS web site.

There is more technical information on twiki.cern.ch in the GridMonitoringNcgOverview and GridMonitoringNcgYaim pages. More information about the plugins/probes can be found on SAMProbesMetrics.

If you want to know more about the NGS R+D activity, we will on on hand at All Hands next week.