Three Little Words

There are these three little words. For some people, these words bring feelings of fulfilment and contentment. For others, they bring nothing but frustration.

Those three little words are:

  Proof of Concept

For that part of the e-Research Community interested in how research will be done in future, A proof of concept is evidence that it is possible to do something new and interesting, using something new and interesting. It might change the way research is done next decade. It is more than enough for a published paper and a presentation at All Hands.

And it is of bog-all use to those for whom e-Research is simply a means to an end. They just want something that works now and works reliably.

There is always a gap between the potentially useful and the actually useful. When you can build something that bridges that gap, you can enable research that would not otherwise be done.

Which brings me to slightly embarrassing news that our project to deploy the ARC middleware in front of the local High Performance Computing service has been a complete success... as a proof of concept.

We have shown that it is possible to deploy ARC services in front of what we should now be calling Oracle Grid Engine.

With some inventive use of ssh copies in prolog and epilog scripts --- that this can be made to work even where there is no file-space shared between the grid 'front end' and the HPC cluster.

We also know that you can support parallel tasks  using ARCs Runtime Environment mechanism --- there are examples at the bottom of the (slightly out of date) Nordugrid documentation --- and make use of to the LCAS/LCMAPS authentication system used by other grid software.

Which is nice....

Whether it is going to be useful is a completely different question.  We do not yet know if the local communities who are best placed to use it --- the rather incongruous pairing of Solar Physics and Social Science --- will want to do so.

Epilogue: Prologs and Epilogs


A quick technical note on faking a shared directory via Grid Engine prolog and epilog scripts.

The scripts run just before the start and just after the end of every job.

ARC-the-middleware obligingly changes directory to the 'shared' scratch directory before submitting the job. This mean that prolog and epilog scripts are presented with the path to this directory in the $SGE_O_WORKDIR environment variable.

The recipe is along the lines of...

• Create a ssh keypair for each user - to be used solely for transfers from HPC backend to grid front end
• Copy the private key to a safe place on the HPC back end, readable only by the user. We will call this $GRID_KEYS.
• Use the public key to create a per-user authorized_key file on the grid front end in somewhere like
     /etc/ssh/authorized_keys.d/$USER
  and change the /etc/ssh/sshd_config (again on the grid-front-end) to set.
      AuthorizedKeysFile  /etc/ssh/authorized_keys.d/%u
• Add code to prolog and epilog to use scp (or rdist) with the -i $GRID_KEYS/$USER to pull files from $SGE_O_WORKDIR at the beginning of the job and push them back at the end.

It is not easy being parallel

As has been said before - there are differences between Grid and traditional High Performance Computing. Some of the differences are due less to the technology and more to the problems being solved.

The more successful grid users are task farmers: they scatter comparatively small compute tasks and data and wait for them to grow into results. The grid - metaphorically speaking - is there to plough the land, spread  the fertilizer and muck out the system administrators.

Traditional HPC concerns itself with big applications and - in particular - applications that are too big to fit on a single computer. HPC systems are built with parallel computing in mind.

The Grid does not do parallel computing well.

Consider the two steps in running any parallel tasks

• Asking for more than one CPU core on the same system.
• Setting those CPU codes to work

For each step, there is definitely more than one way to do it...

Take 4...

So, there you are, sitting by your favourite grid client, a freshly minted X509 proxy ready. All you need to answer one of the great problems of modern science is 4 CPUs.

All you need to do is ask.

How you ask depends on who you are asking and what grid dialect they understand.

Globus GRAM5 and ARC accept tasks defined in Globus  Resource Specification Language (RSL), possibly with some Nordic extensions. In RSL, you can ask for more than one CPU with an additional:

  (count=4)

The web-service-y Globus job submission systems (WS-GRAM) used a similar approach but written as XML.

In Job Description Language, as understood by the gLite CREAM-CE and WMS, you need

  CPUNumber=4;

And in the OpenGridForum-approved XML-based Standard Job Specification Description Language, you have the instantly-memorable and easily-readable:

    <jdsl:TotalCPUCount>

       <jdsl:Exact>4.0</jdsl:Exact>

    </jdsl:TotalCPUCount>

(which you will find buried somewhere under 3 levels of XML tags). 

Yes - I know JSDL isn't really there for humans to read, but it doesn't stop some humans trying.

4 go to work...

That was the easy part.

Now it gets complicated.

And, on this occasion, you can't blame the Grid for the complexity.

Large-scale parallel programs are typically written around libraries implementing the Message Passing Interface (MPI). There is more than one version of the MPI standard and more than one library implementing them.

To add to the confusion, from some MPI variants, you need to build versions for each FORTRAN compiler installed.

Launching a parallel job depends on both the job management software and the underlying mechanisms used for communication. MPI installations typically provide either an mpirun or mpiexec command that ensures that the right processes are started in the right way on the right computers.

It is very likely that each version or each MPI implemention will have its own variant of mpirun or mpiexec. It is equally likely that - at least for mpirun - they will expect different arguments.

In the first and second phases of the NGS, we were funded to provide exemplar Grid clusters at RAL, Oxford, Leeds and Manchester. The grid software we deployed - Pre-WS GRAM from Globus 4 - could launch MPI jobs if

  (jobtype="mpi")

was included in the RSL.

It could only launch one of the many possible mpirun commands. To work around this, devious system administrators cooked up a sort of super-mpirun that would locate the correct version for an applications.

Researcher in Ireland found ways of launching MPI jobs from within JDL jobs - but they could not hide all the complexity.

ARC supports parallel jobs via its Runtime Environments extension - which can tune the environment for an application so that the right number of CPUs are assigned and the right mpirun is run. Again, this needs the  system administrator to do something devious if it is to work.

We haven't even begin to cover parallel programs written outside MPI - such as those using the Java sort-of-MPI library MPJ-Express.

So... what am I trying to say?

It would be nice to have a conclusion, or at least a lame joke, to end this blog post - but I can't think of one.

All I can say is that parallel computing is complicated, distributed computing is complicated and that any attempt to combine the two - either using existing Grid solutions, or something newer, shinier and probably invoking the word Cloud - cannot make either kind of complicated vanish completely.

A good-enough impression

Leeds - as a long standing NGS partner site - want to hook our HPC service into the Grid.

We hope to fill the gap left when the last of our NGS-funded clusters was turned off back in April. Our main requirement was that the grid front end should be completely separate from the HPC service. In addition, we wanted...

• ... to use software provided by the EMI-1 software release from the European Middleware Initiative.
• ... to publish information that can be read into the NGS's central information service.
• .. to accept request from the NGS's workload management system.

We had originally hoped to follow the particle physicists and deploy CREAM.
Unfortunately, EMI-1 was missing the components needed to make CREAM work with the SGE batch system used locally. The only software within EMI-1 that was SGE-friendly was Nordugrid's Advanced Resource Connector - ARC.

After a few months of work and in the great tradition of the grid: it is sort-of-kind-of-working-after-a-fashion. At the moment:

• ARC's compute service - A-REX - is accepting jobs: for a very limited set of users and not from the workload management system.
• ARC's information provider - ARIS - is publishing information about the system and this information is making its way to the NGS's BDII.

I'll will cover A-REX in a future post. This week, you are getting information about the information provider - and in particular, how it links into the NGS.

A bit of background. The NGS information service is a Berkeley Database Information Index service or BDII. BDIIs are built to collate information, some of which comes from other BDIIs. The NGS's central BDII, for example, collates information published by a BDII, or something that looks like a BDII, at each of the partner sites.

ARIS can do a impression of a BDII. Whether it is a convincing impression depends on what it is talking to.

ARIS produces information in its own Nordic-accented schema, designed to feed the ARC tools. This needs to be translated into GLUE format before a BDII will give it a second glance.

Based on documentation on linking ARC and EGI from Nordugrid,, this can all be done via a single ARC configuration file called /etc/arc.conf.  arc.conf consists of blocks, denoted by a [name in square brackets] each containing a set of name=value definitions.

arc.conf needs to be tweaked in three places.

Turn on publishing of Glue 1.2 format information - which  is close enough to the current common Glue version 1.3 - by adding to the '[infosys]' block.

 [infosys]
  ...
 infosys_compat=disable
 infosys_nordugrid=enable
 infosys_glue12=enable

Add in anything that Glue needs and ARIS does not via the '[infosys/glue12]' block:

 [infosys/glue12]
 glue_site_unique_id="NGS-LEEDS"
 ...
 provide_glue_site_info=true

And finally arrange for ARIS to collect its own output and present it as if it were a site BDII by a block called

 [infosys/site/NGS-LEEDS]
 unique_id=NGS-LEEDS
 url=ldap://ngs.arc1.leeds.ac.uk:2135/mds-vo-name=resource,o=grid

Our initial experiments suggest that the information produced by ARIS is good-enough to be accepted the NGS's central BDII but not good enough to fool our Nagios monitoring.

WLCG Nagios includes a number of BDII specific tests including one called org.bdii.Entries. org.bdii.Entries only looks for 'services' - or more accurately objects of the 'GlueService' type. While ARIS generates a lot of information, none of describes a GlueService.

What we don't yet know if it Nagios is being picky, or whether the existence of a GlueService is vital for some bit of grid wizardry.

ARC quirks and keeping track

Another week, another apology for having very little new to say.

Our excuse is that we have all been busy preparing for a meeting of NGS Collaborators (today, 6 July), a workshop on Moonshot, Grid and High Performance Computing (on Thursday) and for a Town Meeting on the future of e-Science and HPC Infrastructures and Applications in the UK (Friday).

On the plus side, when we have all recovered, there should have plenty to write about.

There has been a small amount of time available to work on Leeds' ARC grid software deployment - concentrating on the dull-but-useful task of tracking down and reporting minor bugs.

One such quirk appears when logfiles are rotated - that is renamed and compressed at regular intervals to conserve disk space. ARC continues to write to the original - now renamed - file rather than to a new one. We found the bug, and reported it and discovered that the developers were already aware and it will be fixed in the next release.

Which gives me a chance to opine...

In the 4-years-or-so that I have been involved with The Grid, I have needed to contact software developers all over the world.

With very few exceptions, the developers have been capable, helpful and responsive.
(and I am not going to to identify those very few exceptions)

The open nature of much of the development work, with bug databases and source code repositories readable by anyone, gives bug-hunters from outside the development team as much information as those inside. If used wisely, this information means better bug reports and faster fixes.

Unfortunately, bug-hunting is becoming harder. It is an unfortunate side effect of
the European Grid Infrastructure and European Middleware Initiative - and their remit to co-ordinate development activity from many disparate teams.

The individual development teams have their own development processes and tools....

• The WLCG Nagios developers at CERN use installations of Jira for bug-tracking, Confluence for documentation and and Fisheye for viewing the source code
• The gLite developers use their Savannah service for bugs and feature requests and provide ViewVC for source code browsing. [Addition 8-July: As Brian Bockelman has kindly pointed out - the ViewVC link above no longer takes you to the latest versions as components of gLite have moved to a TRAC instance at CERN.]
  
• ARC has a bugzilla for bugs and uses TRAC for source code browsing.

On top of this is the all-seeing EMI GGUS tracking system that sends bug reports to the individual teams and the EMI wiki.

There are different interfaces for different tools, sometimes you need a certificate, sometimes you need an account. As someone slightly outside - but with an interest in - the development of grid software, I know how hard it can be to check if a bug has been reported, whether it has been fixed and when the fix will be available.

There is clearly work being done to improve the situation and no-one would claim that distributed, international software development is easy to do. What the grid really does not want to do is weaken whatever connections there are with the system administrators who deploy the software and the external developers who use it.

Mind the gap

I can't say that I wasn't warned.

In an attempt to conserve the world's supply of three letter acronyms - and support people studying the Sun and those studying society - we are trying to deploy grid software called ARC on a compute service called ARC.

The Nordugrid Advanced Resource Connector software is the only component within release 1 of the European Middleware Initiative's big bundle of grid stuff that does what we need:

• accepting work requests from a Workload Management Server.
• passing them on to HPC systems which may be running SunOracle GridEngine, Torque/PBS or SLURM batch systems. (Leeds is using Grid Engine, but if we are successful, it could be rolled out to other institutions).

The EMI are in an uncomfortable position. Their job is to take pieces of software from different places - that is similar in intent and very different in design - and persuade them to work together. Sometimes, inevitably, things fall through the gaps.

One of these gaps is between the gLite's BDII information service and ARC's own information services.

This is how it is meant to work...

1. Information is kept in a BDII-friendly database and made available to the world via the LDAP protocol through an OpenLDAP 'slapd' service.
2. On any given system, this information is generated as a set of LDAP 'LDIF' format records by programs called providers and plugins.
3. A program called bdii-update takes the locally generated LDIF, processes it and passes it on to slapd.

What was actually happening...

1. The ARC information system was generating lots of LDIF.
2. The bdii-update process was collating it and passing it onto slapd.
3. slapd was refusing to accept it - complaining of an 'Object class violation'.

After digging into the inner workings of both the BDII and ARC, we've identified the cause. It is all down to a subtle difference between what Nordugrid expect and what gLite expect from their information services.

From this point on, this is going to be technical. Readers of a less geeky disposition can look away now, happy in the knowledge that we know what broke and how to fix it.

Geeks, grab yours Acronyms. Here we go...

Slapd relies on schema files to define what is acceptable: Nordugrid have their own Scandinavian-style nordugrid.schema; gLite use the GLUE schema, including one called Glue-MDS.

Glue-MDS and nordugrid.schema both define an objectClass called 'Mds'. Both agree that it represents a collection of information but in GLUE, an Mds is defined as a STRUCTURAL class whereas Nordugrid defines it as an ABSTRACT class.

So what... as anyone who managed to make it this far down the page might cry.

Well, in the LDAP-world, STRUCTURAL objects can exist whereas ABSTRACT classes can only be used as a basis upon which other objects can be defined. Its all very Object-Oriented-Programming.

ARC's information service generates 'MdsVo' objects, based on Mds objects, but properly STRUCTURAL. This is fine according to the nordugrid schema.

But bdii-update contained code that takes any object that is based on an Mds object and turns it into a plain, simple self-contained Mds object. This is closer to what GLUE expects.

Slapd gets very confused.

A bug report has been raised - and after a bit of bug ping pong between the BDII and ARC developers - it has been decided that bdii-update should, in future, leave Mds objects alone. For the moment, all that is needed is to remove the line in bdii-update that reads

  new_ldif = fix(new_dns, new_ldif)

When acronyms collide...

Now pay attention. This may be a little confusing.

The University of Leeds runs high-end computer clusters as part of its Advanced Research Computing service - known as ARC.

The NorduGrid Collaboration develop, maintain and support grid middleware called the Advanced Resource Connector - also known as ARC.

We had a plan to join ARC-the-computing-service to the grid. The first version of the plan, to use European Middleware Initiative's release of the CREAM software, has been abandoned because support for our batch system - SGE - is not yet ready

So, inspired by work done by Stuart Purdie for Scotgrid, we have a New Plan.

You have probably gathered that The New Plan is (drum roll please) - to deploy ARC on ARC.

Which, when you come to write it down, sounds less like a plan and more like an overexcited performing seal.

It is our only real option: ARC is the only one of the EMI's current middleware menagerie with SGE support.

Round these parts - we like to automate software installations using scripts. Our scripts are based on the instructions from the EMI's Generic installation guide and on Nordugrid's guide to plugging an ARC Compute Element into the European Grid and are now at the point where they can install and configure ARC from the EMI repositories.

The scripts are not yet ready for an official release but - in the unlikely event of anyone wanting to know more - the 'live' versions can be found in the UKNGI code repository at SourceForge. Like the earlier VDT installer scripts - they are intended as runnable documentation. If you don't want to run them, you should be be able to see how they work by reading them.

Incidentally, just to add to the confusion: Leeds also hosts an Antimicrobial Research Centre - known as ARC. Their specialities include some of particular relevance at the moment: antibiotic resistance in bacteria and the transfer of pathogens.

When I last checked, there were people from ARC-the-research-centre using ARC-the-computer-cluster but not via ARC-the-middleware.