SHA2 certificates

We have started to issue certificates with the "new" more secure algorithms, SHA2 (or to be precise SHA256) - basically, it means that the hashing algorithm which is a part of the signature is more secure against attacks than the current SHA1 algorithm (which in turn is more secure than the older MD5).

But only to a lucky few, not to everybody.  And even they get to keep their "traditional" SHA1 certificates alongside the SHA2 one if they wish.

Because the catch is that not everything supports SHA2.  The large middleware providers have started worrying about supporting SHA2, but we only really know by testing it.

So what's the problem?  A digital signature is basically a one-way hash of something, which is encrypted with your private key: S=E(H(message)).  To verify the signature, you would re-hash the message, H(message), and also decrypt the signature with the public key (found in the certificate in the signer): D(S)=D(E(H(message)))=H(message) - and also check the validity of the certificate.

If someone has tampered with the message, the H would fail (with extremely high probability) to yield the same result, hence invalidate the signature, as D(S) would no longer be the same as H(tamper_message).

However, if you could attack the hash function and find a tamper_message which has the property that H(tamper_message)=H(message), then the signature is useless - and this is precisely the kind of problem people are worrying about today, for H being SHA1 signatures (and history repeats itself, since we went through the same stuff for MD5 some years ago.)

So we're now checking if it works. So far, we have started with PKCS#10 requests of a few lucky individuals; I'll do some SPKACs tomorrow.  If you want one to play with, send us a mail via the usual channels (eg email or helpdesk.)

Eventually, we will start issuing renewals with SHA2, but only once we're sure that they work with all the middleware out there... we also take the opportunity to test a few modernisations of extensions in the certificates.

Sign Here

I would not want to describe the paperwork that goes with University life as something out of
Terry Gilliam's Brazil or Yes Minister.

It would be ill-advised: I haven't completed this month's NGS/B/11347/2(a) (permission to use ironic over-exaggeration within a blog) and submitted it to the appropriate authorities.

Depending on how deeply your institution loves its paperwork, there will be forms to be complete when claiming travel costs, or buying a new HPC system, or obtaining a replacement biro. Inevitably, somebody else needs to sign these forms to show that there has been due diligence and that the trip to Didcot, million pound compute cluster or cheap plastic pen are fully justified.

Somebody else isn't just anybody. When you present your form to the powers-that-be, the powers-that-be will carefully compare the signature with their collection of scribbles from the great-and-the-good.

Only when you have the right name in the right place on the right form will you will receive a new pen and a firm lecture about being more careful in future.

As we have said on a number of occasions, grid security is built on chains of trust. It also relies on the right signature being used in the right place. In our case, these are digital signatures represented by X.509 certificates rather than the spiders-web-on-acid scrawl of a senior University manager.

A certificate in your local list of trusted certificates - typically in /etc/grid-security/certificates - can be accompanied by a file defining its signing-policy. You can see some examples of signing policy files in the UK eScience Certification Authority pages on the website.

The signing policy is particularly influential at the very far end of the chain of trust: the root certificates. The private key associated with root certificates are kept in a Very Safe Place and are taken out only to sign the certificates of Certification Authorities (CAs).

CAs sign the certificates for the rest of us. The UK has two CA's - the main eScience CA and a SARoNGS CA.

Over the last weeks, thanks to the efforts of the dragon-slayers at the Software Sustainability Institute, we finally found out why certificates from our 'SARoNGS' CA were being rejected by the NGS's Workload Management Service.

There was nothing wrong with the certificates themselves.

The SSI developers quickly identify problems with the SARoNGS Certificate Revocation List (CRL) - a list of known-bad certificates that CA's should distribute.

SARoNGS certificates are designed to be short-lived - they expire before anyone gets a chance to do something bad with them - and the revocation list is empty. But all revocation lists - even empty ones - have expiry dates and ours had, unfortunately, gone stale.

Updating the CRL was comparatively easy but it did not solve the problem. The root cause of turned out to be the root certificate's signing policy.

The problem is that there are two signing policies - depending on whether you consider SARoNGS certificates acceptable.

SARoNGS certificates can be obtained using only a UK academic username and password whereas a full eScience certificates requires photo ID and a visit to your local Registration Authority.

The International Grid Trust Federation (IGTF) is responsible for ensuring that certificates are being created and managed in a trust-worthy way. It has strict rules on what constitutes sufficient proof of a users identity and - not to put too fine a point on it - an academic username and password are simply not good enough.

So the signing policy within the IGTF's bundle of UK eScience certificate information does not currently match the version we distribute. The IGFT version will not permit the eScience root to sign for the SARoNGS CA.

The root cause was a misplaced update that installed the IGTF version of the eScience root signing-policy - rather than the NGS's own.

We should have had the 'IGTF+' certificates - a modified version of the IGTF's certificate collection maintained by the NGS blogs' very own Jens Jensen, and incorporating the NGS's signing policy and some additional certificates.

The IGTF+ certificates are available in a number of formats from Jens's avowedly Web-1.0 certificate repository webpage.

[With thanks to James Perry, Steve Crouch and Rob Baxter of the Software Sustainability Institute]

E's no good - Distinguishing between distinguished names

This is a sort of companion to Jens's 'On email address in host certificates' post on Monday.

For most people, changes to the policies and standards that describe how the grid should work are met with a resounding 'so what'. For anyone involved in the day-to-day management of grid systems, it is an opportunity to join a collective sign-of-relief.

It is another example of where the 'political' aspects of international research collide with the technical solutions and the needs of researchers who don't give a damn how it works, as long as it lets them do their jobs.

X.509 certificates are complicated because what they represent is complicated - a link in a chain of trust between particular individuals or institutions.

Identities within certificates are tied to Distinguished Names or DNs. A DN is a lists of attributes - such as country, institution and personal name - that uniquely identify a single person, or computer, or service.

The way a DN is stored within a certificate is well-defined but completely incomprehensible to anything that is not a computer program. For many practical purposes, the DN needs to be presented so it can be understood by a person.

A glance at the OpenSSL X509_NAME_print_ex documentation shows how brain-twistingly complicated it can be translating a DN into something that a human being can read.

There is a more detailed explanation on the NGS Wiki. This is the quick tour..

Each individual attribute within a DN has a 'type' and a 'value'.

The type identifies what is being represented - a name, or an email address. It isn't really a name but but a unique sequence of numbers called an Object Identifier. Something like: 1,2,840,113549,1,9,1.

People, inexplicably, find sequences like 1,2,840,113549,1,9,1 hard to remember so for our benefit, 1,2,840,113549,1,9,1 is also known as "Email", "emailAddress" and - occasionally - "E".

The value is depends on the type. For 1,2,840,113549,1,9,1 - it is a string of letters represented in what is known as UTF-8. UTF-8 was developed to represent any letter from any language - but most Grid certification authorities deliberately restrict themselves to the 26 letters of the English alphabet, the numbers 0 to 9 and a few symbols. If they didn't, things would rapidly become even more complicated.

In human-friendly form, the DNs that Jens is working to abolish look very much like

 /C=UK/O=eScience/OU=Manchester/L=MC/CN=voms.ngs.ac.uk/Email=support@grid-support.ac.uk

or maybe

 /C=UK/O=eScience/OU=Manchester/L=MC/CN=voms.ngs.ac.uk/emailAddress=support@grid-support.ac.uk

or even, very rarely

 /C=UK/O=eScience/OU=Manchester/L=MC/CN=voms.ngs.ac.uk/E=support@grid-support.ac.uk

Which variant you get depends on which version of which software is processing the certificate.

The problems appear when DNs are compared as strings of letters rather than in what could be called their 'raw' form.

Most software is smart enough to canonicalise these awkward examples by chosing One True Name for 1,2,840,113549,1,9,1 and substituting this before comparing. Not all software packages agree on which name is the One True Name.

It is now common practice to represent certificate chains in .LSC format - which are simply lists of human-friendly DNs. These may be simple to distribute and do not need to be updated every time the certificate is renewed.

The would be good enough - if it wasn't for that troublesome email address.

On email address in host certificates

Every so often we get questions about email addresses in the names (distinguished names, ie DNs) of host certificates. The problem is that they are deprecated (see the last two paragraphs of section 4.1.2.6 of RFC5280), and they cause all sorts of problems with software which stringifies the DNs because there is no consistent way of doing it (or rather, there are too many consistent ways.) Arguably the software is not coded correctly, but in this case it'd be better to remove the email.

The email is there for historical reasons: when we rekey a certificate we have to give it the same name as before, so that's why it is still there. Dating back ten years or so, the original raison d'être was that before robot certificates, hosts would sometimes run stuff on behalf of users, ie. act as a client, and the email address was meant to give you something to contact when you read the DN in the log file.

The new policy will permit removing the email address from DNs. That's the easy bit.

The trick is to get the software to optionally (at the owner's request) remove the email address from the DN (because some people may genuinely want to keep it, for whatever reason.) Or rather, optionally keep it. The software cannot do this yet.

In fact, it'd be easier to just remove it for all host certificates, or maybe to handle those "manually" who still want to keep it, as with robots for example. If anyone out there has host certificates and depends on email being present in the DN, could you let us know via the usual channels, please? There are no known problems with removing the email address, only with keeping it, but there may of course be unknown problems - there are lots of weird and wonderful things out there.

As for timescale, it'll be ready at the latest when the new (rollover) CA certificates go live at the end of September.

CERN and SSO - your questions answered (maybe)

There have been a number of questions about this recently.

CERN let people sign in to some of some of their stuff if you have a certificate from any approved CA. Which is jolly nice of them - for those of us who don't have CERN accounts, if we need to access something for some reason, we can just log in with our personal certificate.

Last year, around August some time, CERN upgraded their single sign-on (SSO) infrastructure, and one of the security related "improvements" was to check for Certificate Distribution Point (CDP) extensions in intermediate CA certificates. We didn't have that extension (neither, for that matter, had DoESG), as it is not required by the standards, and the requirement was discovered, of course, only after the upgrade. (It doesn't seem to affect any other service that uses certificates.)

As a workaround we created a new certificate just like the existing one but with a CDP added to keep CERN's SSO happy. This was sent to CERN (only), and they deployed it, and it worked, and everybody was happy.

Until fairly recently when it stopped working, for reasons unknown, and of course without any warning. It looks like it was "undeployed" for some reason, and cannot be redeployed?!

So I guess it's up to us to "fix" it - but how? Changing existing CA certificates in the distribution is something that should not be done lightly - we could do this, it should work, technically, but it may at least be an inconvenience for users who have no need for CERN SSO, and one never knows if there's something out there that'll break.

Meanwhile, we have a rollover coming up in September (of which more anon). New CA certificates will have CDP, so should also keep CERN's SSO happy. We can test whether it works at CERN already because they are already deployed. Therefore, the new workaround is to start moving people who need CERN SSO over to the new CA certificates. We'll trial this with a few certificates shortly, and then proceed with the rest.

These are all "just" technical issues, and it is certainly not the first time someone has broken something by upgrading or "improving security" - it happens all the time (and occasionally to us, too). This is a good reason to keep certificates fairly conservative and not try out new exciting features on unsuspecting users. Which is why we have documents like GFD.125. (Which may be due for a revision - something to discuss at OGF32 perhaps.)

Looking back - and being differently successful

Failure is not an option... it's a standard feature.

As witicisms go: it is neither particularly funny, or particularly original - and can be seen on T-shirts, fridge magnets and snarky comments across the Internet.

Anyway, not being funny or original has never stopped me before.

We are looking back at the last two years of R+D: we've done the successes, now it is time for the failures.

I'm not talking about those magnificent 'Brown Paper Bag' failures that will haunt us for many years to come. I'll cover those another time.

Here, I want to mention those projects that have been mentioned at conferences or at this blog, which are working or very nearly working, but are not where we hoped they would be.

I am not criticising the highly-capable and skilled developers involved, or the way the work was managed. The NGS is a small organisation: bad luck or the need to solve more pressing problems limit what we can do.

The nearly-successful projects involved are the CA Wizard, SARoNGS and DataMINX.

The CA Wizard (not to be confused with the existing NGS Certificate Wizard) was intended to replace the confusing - and, to some users, downright scary - business of obtaining a certificate with a single desktop tool.

It was delayed by the loss of some key members of staff but is still being actively developed. The developers expect to release a version in May.

DataMINX was a joint UK and Australian service designed to ship data to where it was needed with minimal user involvement. It also fell foul of staff losses and had to finish early.

The developers were concerned about software sustainability and have left the code in a state where someone else could readily take over: it was released with an Open license and is available from Google Code. It is still being recommended in some quarters.

SARoNGS has been mentioned a number of times on the blog. It provides a way of allowing users onto a grid using their institutional credentials.

We know that SARoNGS development has been too slow. We know that the user interface is confusing. We know it doesn't quite work with the UI/WMS. We were reluctant to turn it into a production service until we had overcome some mind-twistingly-obscure bugs.

But SARoNGS has been used - at York, within the NeISS project, and elsewhere - as a simple way to pass authentication information around.

Failure may not be an option but - thanks to the efforts of people within and outside the NGS - it isn't inevitable either.

[With thanks to David Meredith]

Private keys

The point about public key cryptography is that public keys are public: they are used to prove possession of a secret (namely, the private key) without revealing any information about secret. This is called a Zero Knowledge proof. In other words, much of the level of assurance in the infrastructure rests on management (not just protection) of private keys.

So this is why you have all generated your keys and protected them with strong passphrases.

Recently there has been some discussion among the grid CAs as to whether the rules can be relaxed, without lowering the level of assurance too much. Many people generate their keys on systems which are maintained by someone else: e.g. your desktop at work, or maybe even a UI.

This leads to the proposed loosening of the rules, or perhaps a better description of existing practices.

It is likely that in the future, we will support private keys generated by:

• users themselves, on trusted systems (eg your own machine, or your desktop machine at work)
• institutions, letting them pre-generate keys for their users (apparently some like to do this);
• third parties: e.g. running a credential repository like MyProxy.
  

However, as with much change, it is easy to introduce new rules without fully understanding the problem. There are serious (but fixable) problems with the draft rules. For example, different CAs interpret "third parties" in different ways. Is the CA a third party? I would have thought not. Would the NGS count as a third party, despite the fact that it runs a CA? Probably.

Anyway; the upshot of this is that private key protection rules will be relaxed. What is currently missing is the in-depth understanding of the security aspects of the lifecycle of the private key. I have soapboxed on this topic before. More on this later. Stay tuned.

NGS Innovation Forum incoming!

We really are now in the run up to the third Innovation Forum which will take place on the 23rd - 24th November.

The final speaker for the event was announced a few weeks ago and we are pleased to welcome Andrew Lyall from the European Bioinformatics Institute who will be speaking about the European project ELIXIR.

We will also be providing a "Roaming RA" service at the Innovation Forum so if you would like to use the NGS but do not have a RA at your local institution from whom to obtain a grid certificate, come along to the IF! Not only will you obtain a grid certificate but you will also be able to meet NGS staff and hear about tools which will be of great use to you in running jobs. For more information on how to obtain a certificate at the event, see the instructions on our website.

If you are unable to attend the event you can hopefully follow us on Twitter. We will have a special tag for the event #ngsif10 so look out for this on your Twitter feeds. If you happen to Twitter about the NGS in general please use our tag #ukngs. Help us spread the word!

Remember registration closes on the 12th of November!

Grid certificates drive you mad?

We know that certificates can be a bone of contention with our users but we have a host of ways to help you manage them.

The latest in these is the release of our new series of certificate leaflets. There are 3 in total and are entitled -

• applying, retrieving and renewing certificates
• looking after your certificate
• certificate obligations

They can all be downloaded from our poster and leaflet section of the NGS website.

As well as the leaflets we also have our Certificate Wizard which you can read more about here. We hope you find the leaflets useful and please let us know what you think of them!