How has it affected you?

As you are aware the NGS has gone under some changes since the beginning of the year. The most obvious to our users was the cap on cpu hours that was introduced on the 1st of April with every user being given 2000 free cpu hours per year. If a user requires more than 2000 cpu hours this can be arranged through the NGS on a "pay per use basis".

So 3 months in since the changes were introduced and we want to hear your views. We are asking all registered NGS users to complete a very short survey to feedback their views and experiences on using our services since the 1st of April. It doesn't matter if you haven't used the NGS since before the 1st of April you are still our target for this survey!

The survey will take a few minutes to compete if that and everyone who is a registered NGS user either past or present will be entered into a prize draw.

Delayed

In the best traditions of a UK railway station announcement... The NGS would like to apologise for the late arrival of last week's R+D blog posting. This was due to the non-arrival of some good news.

Over the last few weeks: we have made good progress with our Nagios service and with persuading our Workload Management Service to accept certificates generated using institutional credentials and the SARoNGS service.

Unfortunately, the good progress wasn't quite good enough to deliver something that actually worked.

It shouldn't take much longer. Honest.

That's not my cow

With apologies to Terry Pratchett for the title of this blog post...

You may have spotted an unusual picture on the NGS home page this week. What on earth have cows got to do with the NGS? A very good question...

The answer lies in a recent iSGTW article which once again promotes NGS users research to an international audience.

The article featured the work of the GridQTL team at the University of Edinburgh who provide the GridQTL grid-based platform for running gene analysis. GridQTL is used by researchers all over the world and, unbeknown to many of them, all the computing jobs are submitted to the NGS providing a massive speed up in analysis time.

Team leader Dr Sarah Knott explained “It would be very difficult to run this kind of software without grid computing. We have an uncertain user base and the grid provides the flexibility to sustain an adequate speed of analysis, regardless of online users.”

To see a NGS user case study on GridQTL visit the case study section of the NGS website.

Mind the gap

I can't say that I wasn't warned.

In an attempt to conserve the world's supply of three letter acronyms - and support people studying the Sun and those studying society - we are trying to deploy grid software called ARC on a compute service called ARC.

The Nordugrid Advanced Resource Connector software is the only component within release 1 of the European Middleware Initiative's big bundle of grid stuff that does what we need:

• accepting work requests from a Workload Management Server.
• passing them on to HPC systems which may be running SunOracle GridEngine, Torque/PBS or SLURM batch systems. (Leeds is using Grid Engine, but if we are successful, it could be rolled out to other institutions).

The EMI are in an uncomfortable position. Their job is to take pieces of software from different places - that is similar in intent and very different in design - and persuade them to work together. Sometimes, inevitably, things fall through the gaps.

One of these gaps is between the gLite's BDII information service and ARC's own information services.

This is how it is meant to work...

1. Information is kept in a BDII-friendly database and made available to the world via the LDAP protocol through an OpenLDAP 'slapd' service.
2. On any given system, this information is generated as a set of LDAP 'LDIF' format records by programs called providers and plugins.
3. A program called bdii-update takes the locally generated LDIF, processes it and passes it on to slapd.

What was actually happening...

1. The ARC information system was generating lots of LDIF.
2. The bdii-update process was collating it and passing it onto slapd.
3. slapd was refusing to accept it - complaining of an 'Object class violation'.

After digging into the inner workings of both the BDII and ARC, we've identified the cause. It is all down to a subtle difference between what Nordugrid expect and what gLite expect from their information services.

From this point on, this is going to be technical. Readers of a less geeky disposition can look away now, happy in the knowledge that we know what broke and how to fix it.

Geeks, grab yours Acronyms. Here we go...

Slapd relies on schema files to define what is acceptable: Nordugrid have their own Scandinavian-style nordugrid.schema; gLite use the GLUE schema, including one called Glue-MDS.

Glue-MDS and nordugrid.schema both define an objectClass called 'Mds'. Both agree that it represents a collection of information but in GLUE, an Mds is defined as a STRUCTURAL class whereas Nordugrid defines it as an ABSTRACT class.

So what... as anyone who managed to make it this far down the page might cry.

Well, in the LDAP-world, STRUCTURAL objects can exist whereas ABSTRACT classes can only be used as a basis upon which other objects can be defined. Its all very Object-Oriented-Programming.

ARC's information service generates 'MdsVo' objects, based on Mds objects, but properly STRUCTURAL. This is fine according to the nordugrid schema.

But bdii-update contained code that takes any object that is based on an Mds object and turns it into a plain, simple self-contained Mds object. This is closer to what GLUE expects.

Slapd gets very confused.

A bug report has been raised - and after a bit of bug ping pong between the BDII and ARC developers - it has been decided that bdii-update should, in future, leave Mds objects alone. For the moment, all that is needed is to remove the line in bdii-update that reads

  new_ldif = fix(new_dns, new_ldif)

Hot off the press…

Released the beginning of June, the new edition of the NGS quarterly newsletter is out now. The June edition features a range of articles including –

• Supporting e-Infrastructure Uptake through Community Champions for Research (SeIUCCR) - a new joint project between the NGS and the SSI
• around up of the successful and engaging Collaborations Workshop held by the SSI earlier this year
• an in-depth look at the new NGS load monitor
• Sysadmins corner tips on using Screen
• an introduction to our new member site - the University of Huddersfield

We are always looking for news items for our newsletter and fortnightly email bulletin so if you have anything you would like to contribute please get in touch! We are particularly keen to hear from our users and their application of e-research. Have you recently presented or published a paper featuring research performed on the NGS? If so then we’d love to hear from you!

My data, your data

Edinburgh hosted the Data Intensive Research workshop last week, along with the XLDB Europe. Genially chaired by the formidable Malcolm Atkinson, it was an excellent opportunity to hobnob with other data management folk. While most were database people, many were plain ol' data managers, or people who have data and nowhere to put it.

One of the recurring themes was that there should be more recognition for the work supporting science: if you have a dataset and you spend time preparing, validating, and annotating it, and someone else publishes a result derived from this data, they get the credit for the discovery - naturally - but the work to prepare the data should also be recognised (e.g. in RAEs, maybe as something equivalent to a publication.) Also if it doesn't lead directly to a new discovery. People who share data contribute to science. Moreover, these days, sharing data is often necessary for public funding.

The NGS has a few datasets in bioinformatics already. Having it available already is highly advantageous because you don't have to transfer and update it yourself, and it doesn't take up any space in your storage quota. If there are other datasets you think the NGS could usefully provide, let us know?

Speaking of research support, if you use the NGS you should of course cite NGS in your publications. But maybe you'll also want to buy your friendly local NGS admins a beer or something?

Who do we think you are - revisited.

Once upon a time - before the NGS service at Leeds retired - it was possible to connect to grid-enabled machine using only a web browser, your institutional credentials and an ordinary ssh client.

To achieve this, we combined our Shibboleth 'single-sign on' service with a modified version of our Myproxy enabled GSISSH (MEG) code.

The modifications to MEG allowed us to give access to users without a proper NGS account - but provide them with a very restricted shell. IT security take a dim view of allowing anyone from anywhere to do anything on local computer facilities.

Basically, we needed to replace a gsissh command with a utility that translate a certificate and VOMS information to a local username and password in exactly the same way as a gsissh command.

The technical details were covered back in November. As we are tidying up following the end of NGS R+D activity, we have now packaged up the utility itself - called ngs-x509id - and put it on the UKNGI area on SourceForge.

Ngs-x509id is built around a library of X509 utilities developed by my colleague Robert Frank at Manchester. An early version of the library is bundled with ngs-x509id and Robert has kindly made the latest development version available from the UKNGI subversion repository.

Beep, beep - NGS coming through...

There is a new NGS user case study up on the website and this time it deals with traffic. I'm sure this is a popular topic particularly at 9am on a Monday morning!

NGS user Aengus McCulloch from Newcastle University has been using NGS resources to tackle the explosion in the volume of data relating to real world events such as systems monitoring weather and road traffic etc. Processing the large volumes of data generated by geographical sensors in a short time period is an enormous computational challenge. Guess where the NGS can help out?

Aengus used the real world problem of traffic monitoring and routing as a test case for classifying near real-time processing operations on geographically referenced data using Grid computing architectures. By using the NGS he managed to successfully test his system with up to 250 vehicles.

Aengus said "The NGS has made endless CPU hours available to me through a standards based interface; it has been an invaluable resource for my research".

To read more about his research, see the scalable road traffic case study.

When acronyms collide...

Now pay attention. This may be a little confusing.

The University of Leeds runs high-end computer clusters as part of its Advanced Research Computing service - known as ARC.

The NorduGrid Collaboration develop, maintain and support grid middleware called the Advanced Resource Connector - also known as ARC.

We had a plan to join ARC-the-computing-service to the grid. The first version of the plan, to use European Middleware Initiative's release of the CREAM software, has been abandoned because support for our batch system - SGE - is not yet ready

So, inspired by work done by Stuart Purdie for Scotgrid, we have a New Plan.

You have probably gathered that The New Plan is (drum roll please) - to deploy ARC on ARC.

Which, when you come to write it down, sounds less like a plan and more like an overexcited performing seal.

It is our only real option: ARC is the only one of the EMI's current middleware menagerie with SGE support.

Round these parts - we like to automate software installations using scripts. Our scripts are based on the instructions from the EMI's Generic installation guide and on Nordugrid's guide to plugging an ARC Compute Element into the European Grid and are now at the point where they can install and configure ARC from the EMI repositories.

The scripts are not yet ready for an official release but - in the unlikely event of anyone wanting to know more - the 'live' versions can be found in the UKNGI code repository at SourceForge. Like the earlier VDT installer scripts - they are intended as runnable documentation. If you don't want to run them, you should be be able to see how they work by reading them.

Incidentally, just to add to the confusion: Leeds also hosts an Antimicrobial Research Centre - known as ARC. Their specialities include some of particular relevance at the moment: antibiotic resistance in bacteria and the transfer of pathogens.

When I last checked, there were people from ARC-the-research-centre using ARC-the-computer-cluster but not via ARC-the-middleware.

999 – NGS Surgery

Did you know that the NGS runs Surgery meetings every second Wednesday? The meetings are run over Access Grid / Evo and are open to all.

The surgery meetings have previously been mainly for supporting NGS member sites in installing software etc on their resources. However recent surgery meetings have widened their remit to discuss topics of interest to Campus Champions as well as the Sys Admin community within the NGS.

The NGS is always looking for suggestions for presentations / discussions at forthcoming surgery meetings. If you have any ideas or would like to raise a topic then contact the NGS helpdesk and let us know. Just because you suggest a topic doesn’t mean that you have to present so don’t worry!

To give you an idea of the sort of topics that have been discussed previously, have a look at past NGS surgery meetings on the dedicated section of the NGS website. You can find all the presentations here as well.

If you want to keep up to date with all the forthcoming surgery meetings, make sure you check the NGS website regularly or drop an email to the helpdesk and they will add you to the mailing list.