Sign Here

I would not want to describe the paperwork that goes with University life as something out of
Terry Gilliam's Brazil or Yes Minister.

It would be ill-advised: I haven't completed this month's NGS/B/11347/2(a) (permission to use ironic over-exaggeration within a blog) and submitted it to the appropriate authorities.

Depending on how deeply your institution loves its paperwork, there will be forms to be complete when claiming travel costs, or buying a new HPC system, or obtaining a replacement biro. Inevitably, somebody else needs to sign these forms to show that there has been due diligence and that the trip to Didcot, million pound compute cluster or cheap plastic pen are fully justified.

Somebody else isn't just anybody. When you present your form to the powers-that-be, the powers-that-be will carefully compare the signature with their collection of scribbles from the great-and-the-good.

Only when you have the right name in the right place on the right form will you will receive a new pen and a firm lecture about being more careful in future.

As we have said on a number of occasions, grid security is built on chains of trust. It also relies on the right signature being used in the right place. In our case, these are digital signatures represented by X.509 certificates rather than the spiders-web-on-acid scrawl of a senior University manager.

A certificate in your local list of trusted certificates - typically in /etc/grid-security/certificates - can be accompanied by a file defining its signing-policy. You can see some examples of signing policy files in the UK eScience Certification Authority pages on the website.

The signing policy is particularly influential at the very far end of the chain of trust: the root certificates. The private key associated with root certificates are kept in a Very Safe Place and are taken out only to sign the certificates of Certification Authorities (CAs).

CAs sign the certificates for the rest of us. The UK has two CA's - the main eScience CA and a SARoNGS CA.

Over the last weeks, thanks to the efforts of the dragon-slayers at the Software Sustainability Institute, we finally found out why certificates from our 'SARoNGS' CA were being rejected by the NGS's Workload Management Service.

There was nothing wrong with the certificates themselves.

The SSI developers quickly identify problems with the SARoNGS Certificate Revocation List (CRL) - a list of known-bad certificates that CA's should distribute.

SARoNGS certificates are designed to be short-lived - they expire before anyone gets a chance to do something bad with them - and the revocation list is empty. But all revocation lists - even empty ones - have expiry dates and ours had, unfortunately, gone stale.

Updating the CRL was comparatively easy but it did not solve the problem. The root cause of turned out to be the root certificate's signing policy.

The problem is that there are two signing policies - depending on whether you consider SARoNGS certificates acceptable.

SARoNGS certificates can be obtained using only a UK academic username and password whereas a full eScience certificates requires photo ID and a visit to your local Registration Authority.

The International Grid Trust Federation (IGTF) is responsible for ensuring that certificates are being created and managed in a trust-worthy way. It has strict rules on what constitutes sufficient proof of a users identity and - not to put too fine a point on it - an academic username and password are simply not good enough.

So the signing policy within the IGTF's bundle of UK eScience certificate information does not currently match the version we distribute. The IGFT version will not permit the eScience root to sign for the SARoNGS CA.

The root cause was a misplaced update that installed the IGTF version of the eScience root signing-policy - rather than the NGS's own.

We should have had the 'IGTF+' certificates - a modified version of the IGTF's certificate collection maintained by the NGS blogs' very own Jens Jensen, and incorporating the NGS's signing policy and some additional certificates.

The IGTF+ certificates are available in a number of formats from Jens's avowedly Web-1.0 certificate repository webpage.

[With thanks to James Perry, Steve Crouch and Rob Baxter of the Software Sustainability Institute]

Delayed

In the best traditions of a UK railway station announcement... The NGS would like to apologise for the late arrival of last week's R+D blog posting. This was due to the non-arrival of some good news.

Over the last few weeks: we have made good progress with our Nagios service and with persuading our Workload Management Service to accept certificates generated using institutional credentials and the SARoNGS service.

Unfortunately, the good progress wasn't quite good enough to deliver something that actually worked.

It shouldn't take much longer. Honest.

Looking back - and being differently successful

Failure is not an option... it's a standard feature.

As witicisms go: it is neither particularly funny, or particularly original - and can be seen on T-shirts, fridge magnets and snarky comments across the Internet.

Anyway, not being funny or original has never stopped me before.

We are looking back at the last two years of R+D: we've done the successes, now it is time for the failures.

I'm not talking about those magnificent 'Brown Paper Bag' failures that will haunt us for many years to come. I'll cover those another time.

Here, I want to mention those projects that have been mentioned at conferences or at this blog, which are working or very nearly working, but are not where we hoped they would be.

I am not criticising the highly-capable and skilled developers involved, or the way the work was managed. The NGS is a small organisation: bad luck or the need to solve more pressing problems limit what we can do.

The nearly-successful projects involved are the CA Wizard, SARoNGS and DataMINX.

The CA Wizard (not to be confused with the existing NGS Certificate Wizard) was intended to replace the confusing - and, to some users, downright scary - business of obtaining a certificate with a single desktop tool.

It was delayed by the loss of some key members of staff but is still being actively developed. The developers expect to release a version in May.

DataMINX was a joint UK and Australian service designed to ship data to where it was needed with minimal user involvement. It also fell foul of staff losses and had to finish early.

The developers were concerned about software sustainability and have left the code in a state where someone else could readily take over: it was released with an Open license and is available from Google Code. It is still being recommended in some quarters.

SARoNGS has been mentioned a number of times on the blog. It provides a way of allowing users onto a grid using their institutional credentials.

We know that SARoNGS development has been too slow. We know that the user interface is confusing. We know it doesn't quite work with the UI/WMS. We were reluctant to turn it into a production service until we had overcome some mind-twistingly-obscure bugs.

But SARoNGS has been used - at York, within the NeISS project, and elsewhere - as a simple way to pass authentication information around.

Failure may not be an option but - thanks to the efforts of people within and outside the NGS - it isn't inevitable either.

[With thanks to David Meredith]

Portals, Proxies and making things simpler

The NGS provides the services needed centrally to run a Grid. We provide instructions for institutions that wish to contribute to a Grid and resources for researchers who want access.

We have to leave the interesting bit - connecting the services, institutions and researchers in new ways - to others.

One of the people doing the interesting is Mark Hewitt of the Department of Computer Science at the University of York - who was involved in a project to allow non-technical users to link tasks on the grid together in workflows.

In this guest post, Mark describes how they did it...

The P-Grade portal is a web-based portal system developed by SZTAKI - the Computer and Automation Research Institute of the Hungarian Academy of Sciences - in collaboration with the Centre for Parallel Computing at the University of Westminster.

P-Grade provides a generic job submission system using workflows and can submit to any cluster with the standard NGS Globus stack installation.

The White Rose Grid e-Science Centre deployed P-Grade at York for the use of researchers at York, Leeds and Sheffield. Many users find managing certificates complicated, so we decided to use the NGS SARoNGS system as an alternative.

Integration of SARoNGS with P-Grade was quite straightforward. P-Grade must download a proxy certificate from a proxy server before it can start work. SARoNGS creates proxy certificates, generates username and password and provides a mechanism by which these can be passed to a web page. We were able to modify the P-Grade code, to allow a user to download a SARoNGS certificate, instead of having to use the myproxy tool to upload their e-Science digital certificate.

The addition of SARoNGS support to the portal was vital as it completely removed the need for users to go through the process of signing up for a digital certificate and meant that they could authenticate simply through the portal interface within seconds.

Fixing the mangled XML in the SARoNGS service

A little technical note following from last weeks posting on SARoNGS.

If you are averse to perl, XML and regular expressions, look away now. There will be a proper R+D blog post along shortly.

Someone at a recent NGS Surgery asked for the gory technical details of how we turned the corrupted XML that was breaking the SARoNGS service into something that once-again matched its cryptographic signature.

The SARoNGS web front-end at https://cts.ngs.ac.uk is written in Perl. It relies on Shibboleth to obtain user attributes from identity providers, encode them in Base64 and deliver them via a custom http header called 'Shib-Attributes'.

The Apache web server will eventually present this to our perl code in an environment variable called HTTP_SHIB_ATTRIBUTES.

We realised that, under some circumstances, an additional set of xmlns:xs and xmlns:xsi namespace declarations were being added to the <samlp:response .. > XML tag generated by newer versions of the Shibboleth idP.

These were always inserted at the end of the tag, before the final '%gt;' and just after the responseid attribute. Removing them meant...

• turning the base64 encoded data back into XML,
• using a perl regular expression to remove the cruft and restore the XML to canonical form
• turning the correctly canonicalised XML back into Base64.

or in perl...

use MIME::Base64;
my $encodedData = $ENV{HTTP_SHIB_ATTRIBUTES};

...

my $shibAttrXML=MIME::Base64::decode_base64($encodedData);
for ($shibAttrXML) {
 s{(<saml1p:response.*?responseid="_[0-9a-f]+")(.*?)(>)}{$1$3}m;
};
my $encodedDataCanonical=MIME::Base64::encode_base64($shibAttrXML,'');

It's a workaround, not a fix, but it is a workaround that works.

Single Sign On - The Movie

Ladies and Gentlemen, take a seat, grab some popcorn and don't forget to turn off your mobile phone - because the NGS is going to the movies....

In our feature presentation, we join an intrepid explorer as he connects to the Grid using only his institutional credentials and some slightly-annoying background music.

At 3 minutes and 44 seconds, it is considerably shorter than Avatar, but if you can't wait (spoiler alert!), the plot is:

• Our hero visits the NGS SARoNGS service.
• He authenticates himself using Shibboleth and his institutional username and password.
• He clicks a button or two and is rewarded with credentials that allow him ssh command line access to a grid enabled machine.
• And they all live happily every after.

In the future as seen by Project Moonshot, we will be able to use institutional credentials anywhere. We can already make it most of the way using existing technology - a sort of Project Apollo 13?

The Making Of...

No modern movie is complete without a 'The Making Of...' documentary to fill those extra bytes at the end of the DVD. So we will also let you see behind the movie magic...

When you click on 'Login' on the 'SARoNGS' service provider - https://cts.ngs.ac.uk - your web browser does the Shibboleth Shuffle: passing you via the 'Where Are Your From' (WAYF) service to your home institutions 'Identity Provider' (idP) and then back to http://cts.ngs.ac.uk.

In the last step of the shuffle, a blob of XML is delivered that means `we at the University of Nether Wallop do solemnly swear that this is one of our users'.

Now that it knows that you are a reasonable member of society, the SARoNGS service and your local Identify Provider immediately start talking about you behind your back. In the chatter, your Identity Provider passes on one or more Shibboleth Attributes that describe who you are and what you do.

Shibboleth Attributes can be nearly-anonymous or as personal as names, email addresses or even photos so the UK Access Management Federation has strong recommendations for what can be revealed. Unless legal agreements are in place, an idP only need reveal your unique pseudonymous identifier and your role.

The Shibboleth Assertions are passed from the cts.ngs.ac.uk to a separate authentication service based around a modified MyProxy server.

The authentication service only cares about the unique pseudonymous identifier - or eduPersonTargetedId - and creates and manages short-lived certificates on its behalf. These certificates have a distinguished name that looks like

/DC=uk/DC=ac/DC=ngs/DC=sarongs/CN=(a very long string of hexadecimal digits)

The very long string of hexadecimal numbers is a cryptographic hash of the eduPersonTargetedId.

The authorisation service sends the certificate back to cts.ngs.ac.uk where it is associated with one or more Virtual Organisations (VO).

The default VO, 'ukfederation.org.uk', represents anyone from an institution within the UK Access Management Federation. You can also sign up for an NGS account with a SARoNGS credential, at which point you will be eligible for membership of the 'ngs.ac.uk' VO.

The certificate and VO information is stored on the NGS's official MyProxy server myproxy.ngs.ac.uk under a unique username and a random password.

The SARoNGS service has done its duty. Now the MyProxy enabled Gsisshd (MEG) takes over.

MEG allows an ordinary ssh client to be used to access a grid-enabled service. It accepts a username, a myproxy server and a password - uses these to download a (proxy) certificate and uses that certificate to authenticate you.

ngs.leeds.ac.uk has a version of MEG running on port 2223. We have made some changes - described in technical detail in the bonfire-night R+D posting - to allow certificates with only ukfederation.org.uk membership to log on without being given full command line acccess.

The Out-takes...

The MEG service at Leeds has been running, and accepting SARoNGS and ukfederation.org.uk certificates, since early December 2010.

We have kept quiet about it not because we are naturally modest and unassuming, but because we would have looked like a bunch of bumbling idiots.

There were some places where the SARoNGS service resolutely refused to work. If you were based at one of the unfortunate institutions and tried to reproduce what you saw in the movie, you would have got to the end of the Shibboleth Shuffle and been rudely informed that:

MyProxy didn't like me

We have known why MyProxy is being so unfriendly since November. The XML representing the Shibboleth Attributes is digitally signed and, at some point on its journey, it is corrupted so the signature is no longer invalid.

The fault seemed independent on the version of the idP software deployed but did depend on which attributes were released.

Earlier this week, we worked out why.

It is very subtle, very Shibboleth and another magnificent example of XML biting back.

Before it is signed at the idP the XML the Shibboleth Assertions is first converted to a canonical form, a process that needs to take XML namespaces into account

When the attributes were reconstituted on cts.ngs.ac.uk ready to be passed to the authorization service, additional namespace declarations were inserted, scrambling the signature.

We are still not clear where or why this happens. It might be related to typos in the Shibboleth configuration which left certain Attributes missing a default XML namespace.

The typos are fixed in version 2.2.1 of the idP and, thanks to NeSC Glasgow, we can confirm that this version can send all the attributes it wants with no repercussions.

Working around the problem was trivial. The additional declarations always appeared in the same place - at the very end of a saml1p:Response tag - so we simply removed them again.

The Embarrassing thank-you speech...

Like an Oscar winner, we have a large number of people to thank for their contributions.

These include the people at the NGS partner sites at RAL and Manchester and those people at Glasgow, UCL and Sussex that helped identify and debug the SARoNGS problems.

We would particularly like to thank John Watt from what used to be NeSC Glasgow for taking the time at last weeks NeISS meeting to help generate test cases.

The inevitable sequel...?

SARoNGS is built around an elderly and currently unsupported versions of Shibboleth and Myproxy.

The web user interface is seen as confusing by less experienced users.

If it is to continue running, it will need further development.

SARoNGS is unique in that it make the Grid available to people who cannot or will not use browser-based certificates - and that makes it the real star of the movie.

Too much of an adequate thing

There are two quotes that sum up much of the technology behind grid computing.

The first..

All problems in computer science can be solved by another level of indirection... Except for the problem of too many layers of indirection.

has been variously attributed to the computer scientists Butler Lampson and David Wheeler. In the grid world, indirection is spelled M-I-D-D-L-E-W-A-R-E.

The other?

XML is like violence – if it doesn’t solve your problems, you are not using enough of it.

Variations on this can be found scattered around the internet because XML - the eXtensible Markup Language - is everywhere. Sometimes because it is the right tool for the job, frequently because it is an adequate tool for the job.

It is the right tool for the job when that job is marking up - superimposing a structure on what would otherwise be stream of text.

Scholars in digital humanities use XML schema from the Text Encoding Initiative - and elsewhere - with material as diverse as Chaucer's Canterbury Tales, the Concise Icelandic-English Dictionary and the writings of Mark Twain.

XML parsers such as Apache Xerces can read this data and store it in XML-aware databases such as eXist. These databases can be queried on both their content and their structure using technology such as XQuery and XPath. The tools are complicated to learn but we have the excuse that they are doing a complicated task.

But, as anyone who has experience of modern software will tell you, XML is used for far more than marking up text. It is used as the basis for the SOAP protocol and the Web Services Resource Framework used by the Globus Toolkit and UNICORE.

Pragmatism helped spread XML. It was good enough. It could be understood by looking at it. There are libraries available to generate and process it. Modern development environments such as Eclipse and Netbeans can edit it without the tripping over the angle-brackets.

XML is designed to be flexible enough to cope with ambiguous human text. Computers don't cope well with ambiguity and - as NGS staff discovered during the roll out of the SARoNGS service - XML can be ambiguous in very inventive ways.

SARoNGS, for those who have yet to meet it, is a way of generating restricted certificates for users who can present institutional usernames and passwords. It was intended as a lightweight alternative to applying for a full e-Science certificate.

SARoNGS is uses the Shibboleth service managed by the UK Access Management Federation to authenticate users. Shibboleth is designed to authenticate access to web sites and widely used as a way of accessing academic journals. It is built on SOAP. SOAP is built on XML.

Shibboleth is designed so that the user only ever enters his or her username into a web-page provided by his or her home institution. This is called the Identity Provider or idP. If you try to connect to a Shibboleth-protected service, you will eventually be sent to your local idP.

The idP confirms that you are indeed a fine, upstanding member of the University of Whatever in two distinct messages - known as assertions - one delivered via your web browser, one delivered directly to the Shibboleth-protected service. Think of it as a way of linking your browser, your home institution and the service together.

These messages are heavily disguised XML.

When users from Leeds and some other large Universities tried to use the service - they failed. SARoNGS would see nothing.

Further investigation showed that both messages were being sent but one was being rejected on arrival.

At the time SARoNGS was the only service within the Access Management Federation that insisted on assertions being digitally signed.

Because there are many ways of formatting the same information in XML, signing a lump of XML involves first converting it to a canonical form by - for example - replacing certain character sequences by their Canonical counterparts. This is further complicated by XML Namespaces designed to allow different XML dialects to be used in the same document.

The SARoNGS developers eventually tracked down a small bug - known as JXT-55 - in an XML toolkit used by some versions of Shibboleth. JXT-55 had inadvertently squeezed two different versions of an XML dialect called the Security Assertion Markup Language into the same namespace.

Some time between the XML being canonicalised for signing and being sent, an XML 'cleanup' routine stripped out what looked to it like duplicate data. This invalidated the signature and the message was rejected.

Users in institutions running older Shibboleth services - which included the buggy code - could not use SARoNGS, those running newer ones could.

Discovering this took several weeks of effort and frustration by half-a-dozen people at 4 universities.

XML is like violence in one sense - it can be a source of a great deal of pain.

[Edit: 6-Jun to provide more details of The Little Bug that broke SARoNGS]

NGS at the JISC Conference

If you are attending the JISC conference in Edinburgh next week then be sure to stop by the NGS exhibition stand (no. 28) where Andrew Richards (NGS Executive Director) and Gillian Sinclair (Liaison Officer) will be on hand to answer any questions. So if you want to know how your institution can join the NGS or how you can encourage researchers at your institution to use NGS resources etc then please stop by for a chat!

As well as the exhibition stand there will also be a demonstration of the NGS SARoNGS project. SARoNGS stands for Shibboleth Access to Resources on the NGS and the demo will show how the SARoNGS project is enabling users to gain access to the NGS with their own institutional ID via the Access Management Federation. It will also demonstrate how the NGS is using PERMIS technology from the VPMan project to enable resource providers and projects to control access to individual services within their resource.

The demo will take place on Tuesday on demo stand 4 at 1pm.