Tuesday 20 December 2011

The Training Marketplace

Claire Devereux from the NGS introduces the EGI Training Marketplace.

The Training Marketplace is a service that allows you to search for or advertise training events and resources throughout the EGI community. These can be events or resources that may be open to absolutely anyone, just those within the EGI community, or they may be specific to a small project or to one country only. The Training Marketplace is the one-stop shop for your training needs as a user and is open and free for both academic and commercial providers to advertise in.

Since May the Training Marketplace has evolved from a simple event and material repository into a interactive site where requirements can be captured, events can be rated, and the tool can be customised and embedded into third-party websites using our gadget generator.

The following types of resources are currently supported:
  • traditional training events, usually classroom based or workshops where people attend in person. They can also include virtual events running at specific times. The difference between training events and online training is that training events have a set start and end time whereas online training is accessible either permanently or over a longer time scale.
         You can choose to display events as a list, on a calendar or on an interactive map.
  • online training, available via the web. This category covers a wealth of resources, from self-study courses that require users to log in and complete exercises at their own pace leading to a qualification, through to online tutorials that users can tap into as they wish.
  • training resources, physical resources or services available to the community to assist in training. An example of a training resource is the GILDA Certification Authority. GILDA issues temporary (14 day) personal public key certificates (compliant with the X.509 standard) in order to access the GILDA Testbed for user training.
  • the requirements area is a place for users to describe their training needs and for training providers to see if there is interest in running courses. For example, a provider may post details of potential offerings and ask those interested to visit their website for further details, to check for viability before running a course.
  • University courses is the place for higher education institutions to advertise their Masters and Doctorate training opportunities.

After attending an event or taking part in online training users can rate the event and leave their feedback, letting others know the value of their experience much in the same way are people do with online shopping nowadays.

The latest release is a step towards improving the user's experience and looking towards longer term sustainability. It includes the new online training category, much improved search functionality, notifications to authors once entries are published and an improved calendar view. We are now working improve the functionality and appearance of the Training Marketplace gadget, which allows projects or NGIs to embed the EGI Training Marketplace into their own website, pick and choose which parts to include and even skin some elements with their project colours. If you are an NGI or project we would be interested in hearing about your requirements for the gadget, so please contact us.

Thursday 15 December 2011

CA stuff

Three almost unrelated things, except that they relate to the CA(s), somewhat technical stuff, so bear with me:
  • If you have renewed your certificate recently and found that it didn't work with VOMS, this is due to a misdesigned "feature" of VOMRS that it insists on registering not just the user name but also the issuer name as part of the account. In the Real World(tm), we can keep the issuer (ie CA) name the same all the time but this does not work with grid middleware, so we have to change issuer name whenever we roll over. The "feature" adds no extra security for grid CAs because the user name will always be unique. There is a workaround that tells the server to ignore the feature, which we thought everyone had been using for years - but this, as Steve Traylen points out, will still cause problems if your certificate expires before you can resign the AUP (surely a rare case?!) - but should otherwise work fine. The best option otherwise seems to be to get your VOMS admin (not VO admin!) to duplicate the account entries, one with each issuer name, the old and the new one. CERN (Steve) has done this with theirs, and we are checking the other ones to see if they have failed to enable skipcacheck.  I am both amazed and sorry that we have not discovered (and resolved) this sooner... but Steve is a Wizard(tm) and will fix it...  Incidentally, it is not just us, other CAs roll over too - however, many have chosen to extend the lifetime of the existing certificate instead of renaming it - this will then not cause problems with the VOMRS accounts, but it causes problems with server/client synchronisation for HTTPS instead - if the server and client (browser) are not updated in sync (and they never are), some browsers will print obscure error messages and fail to connect (as we know from past experience).
  • Oh, and another good thing is that the IGTF rules have changed - we can now make the end entity issuing CAs have longer lifetime, so we no longer have to roll over every four years.  Hooray!
  • On the subject of IGTF 1.43 release which came out recently. We're rebuilding the NGS-specific release except we are going back (or forward?) to individual RPMs instead of a single one for the lot - this means we need a couple of dependency RPMs but we should then be able to not mess with the IGTF stuff much and sites can in principle fine tune what they install. We'll have to think about the dependencies carefully.
  • Related to this (so, er, not entirely unrelated), there is this problem with the IGTF root signing policy file. We're trialing the Least Elegant Workaround(tm) this time, having discussed it at some length, by testing a self-signed version of the SLCS toplevel. This makes it independent of the root in the technical sense of building a verification path and checking signing policy files, so would slot in next to an unmodified IGTF release directly - but the downside is that we now have another self signed certificate that we'd need to establish trust in, and the fact that the policy of the SLCS branch (ie the SARoNGS CA, the CEDA CA, etc.) were supposed to be covered by the root CP. Depending on how good this looks (we're testing it from today), this may appear in 1.43.
  • Oh yes, and I know I need to get TACAR corrected and updated. This is not trivial (requires writing forms and pgp signatures) so is awaiting a slot where I have some time...
That was three things, for suitably large values of three. If you have any questions, do get in touch...

Wednesday 14 December 2011

Return of the Champions


I’ve recently taken over the organisation of the NGS Campus Champions programme here at the NGS and last week I chaired the first meeting of the group.

Our Campus Champions come from a number of universities across the UK and are mainly representatives from IT Services.  As for their role and activities on behalf of the NGS?  Well this was one of the main things I wanted to discuss at the meeting!

Thankfully all our Campus Champions agreed with my list of proposed benefits for them (and us) of being a Champion.  In brief the NGS will provide training in using NGS resources, tools and e-infrastructure; provide access to online training materials; produce publicity material to help them publicise the NGS and their role as Champion; hold Campus Champion events at relevant events; hold bi-monthly phone calls for dissemination of news and information.

In return the Campus Champions will actively promote the NGS within their institution offering advice and advising researchers; display NGS Campus Champions publicity material; pass onto the NGS requirements from their institution and researchers; liaise between researchers, the institution and the NGS; attend the bi-monthly Campus Champions meeting.

Currently we have 14 institutions with Campus Champions but we are always looking for more!  If you are interested in becoming a Champion then please contact me.  Your site doesn’t have to be a member of the NGS for it to have a Champion.   The current list of institutions is:
  • Canterbury Christchurch              
  • University of Huddersfield          
  • University of Hull            
  • University of Liverpool
  • University of Manchester           
  • Queen Mary - University of London       
  • University of Oxford        
  • University of Reading    
  • STFC - Daresbury Laboratory      
  • STFC - Rutherford Appleton Laboratory                
  • University of Sheffield  
  • University of Surrey       
  • University of Sussex      
  • University of York
The next meeting of the Campus Champions will take place by phone in February but in the meantime I shall be making some changes to the existing Campus Champion pages on the NGS website.  Watch this space!

Monday 28 November 2011

Interact with the NGS

I recently updated the NGS presence on Facebook to a new interactive page.  Instead of the old-style group page, the NGS now has a new page under the UK NGI banner which reflects our role as the lead in the UK National Grid Initiative along with GridPP.

By “liking” our page you can receive news updates from the NGS all in one place as well as news updates from the Software Sustainability Institute (SSI) and GridPP.  Please feel free to invite your friends and colleagues to "like" our page and to share articles with your friends list - help spread the word about e-infrastructure! 

Wednesday 23 November 2011

NGS at SC'11 - round up

The UK presence was fairly significant this year at SC'11 with attendance by David Wallom, NGS Technical Director and a significant number of the leaders of research computing centres from around UK universities. This included NGS member sites @ Bristol, Leeds, Oxford, Southampton and EPCC.

In this blog post, David gives us a round up of NGS activities at this major computing event.

Having got off of the very long flight from Heathrow
to Seattle we settled onto the metro to get us  downtown. After passable sleep the following morning we headed over to the SC’11 venue – the Washington Convention Centre to collect our badges and then visit the workshop on HPC in Smart Grid, where there was UK interest from the EC FP7 HiPerDNO project being presented by Dr Stef Salvini, OeRC. Following a very productive day where we learnt the state of the art in US Smart grids, how they intend to utilize knowledge developed through the national e-infrastructure for research. We then met up with the EGI team who had an exhibition stand at the conference.

The Monday workshop, Many-Task computing on Grids and Clouds 2011,  started off with an interesting keynote from David Abramson (Monash) a long term friend of the NGS through support for their Nimrod tool which is popular with several of our biosciences users. After this there was a panel session which went slightly off topic to talk about exascale more than Many task but it still attracted several questions around the need for exascale, when we are still struggling to get a significant user base onto smaller HPC systems. Overall a good workshop though having the panel first did mean that a number of people didn’t hang around for the rest of the papers. This workshop was operating in a very competitive market with other sessions on cloud and data management which also attracted significant crowds.

The first full day of the conference allowed for the first good look around the exhibition floor alongside several interesting birds of a feather sessions There was also the  first of a number of conversations with different groups and vendors, including Microsoft, Mathworks and Adaptive Computing.  To give an idea of scale this picture is down one of the main aisles in one of the 5 rooms that were all about this size!!



Pretty impressive stands by a number of people
but the coolest was the multi projection globe on the NOAA stand.


We of course also announced
our activity with Globus Online which created a lot of interest and ended with us having a number of interesting conversations with NSF regarding future collaboration between our national e-infrastructures.





During the meeting the EGI booth was continually visited by a reasonably large number of people, we had the Real Time Monitor showing as normal (having seen a lot of 3D screens this needs to be done in 3d now for next year!). They did though give away a pretty large number of t-shirts as did a lot of stands, so I ended up as the moving poster board around down town  Seattle from 6:30-7am every morning on my morning run!

Friday 18 November 2011

Dinosaurs, DNA and nuclear power


Just a normal month or so in the life of the NGS really but what do all 3 have in common?
They were all research areas investigated using NGS resources.

The research of William Sellers, Phil Manning and Karl Bates on dinosaur locomotion was featured as a “success story” on the EGI website.  They talked about how they used Grid computing to understand better how dinosaurs moved around and what roles they played in their ancient world.  As there are no similar animals around today to compare to dinosaurs such as a T. Rex, the solution is to create a detailed computer simulation of the animal’s skeleton and muscles.

Not only was their research picked up by EGI but it also featured in iSGTW – fantastic publicity for the researchers and for the NGS!

I’ve been busy putting together some user case studies over the last few weeks and I’m pleased to say that there are now a few more up on the website showcasing the large spread of research areas that the NGS facilitates.

First up is Charlie Laughton from the University of Nottingham who has been using the NGS for quite some time now.  He used the NGS to investigate the flexibility and folding properties of DNA as understanding how the tightly packed DNA in human cells can still be read can, in turn, help to understand how cells switch genes on and off.   At present there is no clear understanding of how this works.  Being able to influence this in new ways may ultimately help to find new drugs to treat diseases such as cancer, develop new biofuels, and crops that can resist climate change.

Charlie said “…without the compute power and high-throughput provided by the NGS, we would not have been able to deliver our part of the project in a timely manner. At a more personal level, it led to one of the most highly cited publications I have ever had.”

John Allen from the University of Edinburgh explained how they use NGS resources to power the GridQTL portal which is used worldwide to study gene expression in a wide range of organisms.

The team’s use of the NGS has greatly increased the productivity of their users (currently around 400) in the QTL community. One example of this is a GridQTL user at the University of Missouri Columbia.  They ran a series of studies on carcass, post-natal growth and reproductive traits in commercial Angus cattle and found a speed up of from 20 people-weeks, using their old single server system, to 3 people-weeks to capture and analyse the data with GridQTL. 

Finally we have nuclear power!  Paul Martin from the University of Huddersfield has been using the NGS to investigate the suitability of Thoria asan alternative form of nuclear fuel.  Paul’s research is particularly timely as there is increased interest in the use of thorium dioxide for nuclear power rods not least because of its comparatively high abundance in the earth’s crust and low cost.  It is for this reason that, although the main fuel for nuclear power reactors is currently urania-based, thoria-based fuel is attracting much attention as an alternative high performance nuclear fuel.

All of our case studies can be found on the NGS website and we now have a collection of 26 covering a wide range of research areas.  If you are interested in using the NGS case studies to promote grid resources then please let us know.

Thursday 10 November 2011

A tangled web we weave

In the September 2011 edition of NGS News, we published an article by our Technical Director David Wallom which highlighted the networks of champions that exist within the NGS.  To compliment this I also received and published an article on our website from Simon Hettrick from the Software Sustainability Institute about their network of champions which they have thankfully called Agents.  Too many champions spoil the broth and all that...

It got me thinking about how these networks all fit together like a spiders web as some people are members of more than one champions network and some institutions have more than one person involved.  A tangled web indeed!

This has probably come to the forefront of my mind as, due to some staff changes at the NGS, I'll be a lot more involved in the organisation of the Campus and Community Champions here at the NGS.

So what are these Champions and who are they?

The Campus Champions as suggested by the name, promote the NGS and the services we provide on their university campus or in their institution.  They tend to be people involved in research computing or ITS but we also have researchers involved.  All NGS member sites are expected to nominate a Campus Champion but we welcome Campus Champions from any UK university or institution.  Your site doesn't have to be a member to have a Campus Champion!  If you are interested in promoting the NGS at your institution or university (with help and support from the NGS) then please get in touch!

The Community Champions are funded by the EPSRC funded SeIUCCR project and are researchers who actively use e-infrastructure in their research.  They promote to their research community, peers and colleagues from all institutions and universities.  We are looking for more Community Champions from all and every research area so if you are interested in promoting your research and your use of the NGS then please get in touch!

There will be some slight changes to our Champions networks and hopefully you will see a lot more activity from these already active groups.  I want to highlight their activities more and demonstrate their contribution to the NGS and e-infrastructure as a whole.  I'd also like to link them more closely with the SSI Agents.

Lots of plans are afoot so watch this space!

Thursday 27 October 2011

A smattering of updates

Over conference season there was what can only be described as a smattering of small announcements that you may have missed during your travels.  The blog seems like an ideal place to gather these together!

NGS News
The latest edition of our quarterly newsletter was released in time for conference season.  Available to download from our website, September’s edition contains articles on -
  • A round up of news from Europe including the release of the first Unified Middleware Distribution
  • An introduction to the new Certificate Wizard
  • Championing e-Research and e-infrastructure - the Campus and Community Champions
  • NGS user case study - Using the NGS to model the climate impact of aircraft emissions
  • ...and more!

Tell us what you think!
Also on the website we have a new poll on the home page.  This time we are asking people how easy it is to find the information they are looking for on the NGS website.  So no matter if you are a frequent or occasional visitor to the NGS website, let us know by voting in our poll.  It can be found on the right hand side of the homepage.

 
Busy users
There is a new NGS user case study on the website.  Maria Holstensson from the Institute of Cancer Research explains how she is using the NGS to optimise cancer treatment for children suffering neuroblastoma.

Children with neuroblastoma who are being treated with targeted radionuclide therapy can have their treatment monitored with gamma camera images. These images are used to calculate the amount of drug taken up by the tumour and to estimate the radiation dose. However the image quality can be poor due to scattering and interference. Maria Holstensson from the Institute of Cancer Research is looking at tackling these problems.

Maria said "We have had absolutely fantastic help from the NGS and as a result of using the Grid we have been able to run multiple parallel simulations that we would not have been able to run otherwise".

 
How many?!
We’re pleased to announce that we now have over 1000 subscribers our fortnightly NGS news bulletin.  We have subscribers from all over the UK and much further afield with 21 countries represented amongst our subscribers. 
The news bulletin is delivered to your inbox every second Friday (with some exceptions during conference and holiday season!) and contains news from the NGS, updates from our member sites, details of forthcoming relevant conferences, calls for papers for relevant journals and much more.  You can subscribe to the mailing list from the JISCmail site.

Thursday 20 October 2011

Lounging in Lyon and yawning in York

Oh I wish! 

The September conference season was hectic as always for the NGS team with the EGI Technical Forum (Lyon) and the UK e-Science All Hands Meeting (York) back to back.  There was definitely no time for lounging in the Lyon sunshine although we may have yawned on the train home from York as the conference travel came to an end.

The EGI meeting in Lyon was a great success as always with over 600 attendees.  The meeting is an excellent opportunity for the NGS, in it’s role as the UK National Grid Initiative (NGI), to meet up with other NGI’s from all over Europe.  As well as looking after and organising the UK NGI exhibition stand in conjunction with GridPP, I was also involved in a session in my role as NGS Liaison Officer.

The NGS has held successful roadshow events for several years and these have caught the eye of EGI who are looking at doing something similar through the NGI’s.  I was asked to take part in the EGI / NGI roadshow session and present on my experiences of organising and holding roadshows and measuring the feedback and impact of these events.

There was an interesting discussion after the presentations regarding what the NGI’s would need to host these events and what materials they would find useful.  In some cases more staff and more time would be very helpful but the ability for the EGI to provide these resources are somewhat limited!  I was also asked about practical points such as organising registration and finding the right people in institutions to help host the events.  Hopefully the discussion minutes will be made available at some point.  Some more discussion points were captured on the GridCast blog entry on the session.

The AHM meeting at York attracted 150 people this year and seemed dominated by one word – Cloud!  I lost count of the number of cloud sessions taking place over the 4 days.  The meeting was stimulating and thought provoking judging by the copious amount of notes I took.  However one of the main activities for the NGS was the SeIUCCR organised workshop – Meet the Champions.

The purpose of this workshop was to give attendees an opportunity to meet the researchers that have been promoting and championing research in different e-Science areas and to find out about their work and how they utilise e-infrastructure.  The guest speaker was Scott Lathrop from the XSEDE project where he is the Director for Education, Outreach and Training.  Scott talked about their Campus Champion programme and how they ensure that Campus Champions feel involved and part of the project.  They have many of the same responsibilities as our Campus Champions including raising awareness of XSEDE and even providing training in using the resources. 

Two of our Community Champions also presented at this event outlining the issues in their research areas regarding e-infrastructure, getting started, having the right support at their institution etc.  There was also a very lively discussion panel at the end of the session with many issues raised including how scientific researchers work and the risks involved in devoting time to starting to use new technologies.

It’s great to hear researchers and users speaking about their experiences of actually using this technology, as it sometimes seems, that users get forgotten about in all the discussion about standards and programming.  We need a reminder that at the end of the day this is about building tools and providing a service that people will want to use and find beneficial and that will help further their research.  It’s most definitely not a case of “build it and they shall come”!

I’ll make sure that all the presentations from the AHM workshop are available on the NGS website soon so watch this space!

Thursday 13 October 2011

A long long time ago... or so it seems

Before the mayhem of the September conference season was upon us, the NGS ran a successful summer school at the beginning of the month.  The e-infrastructure summer school was part of the SeIUCCR project.  You can read more about the background to the summer school and project in my blog post from the beginning of August.

We had over 25 students join us down in Coesner's House in Abingdon for 4 days and they came from a wide variety of backgrounds.  We had students who were in their first year or two of their PhD as well as post docs and they came from Edinburgh to Essex and everywhere inbetween.

So how was the summer school?  Was it a success?  Did the students learn and enjoy it?

Well the best people to ask are the students themselves.  All the students were asked to provide us with feedback and some were even willing to write a few more words.

Ed Day from Canterbury Christ Church University attended the summer school and this is what he had to say:


"The recent SeIUCCR Summer School was a very enjoyable and informative experience. As a newcomer to grid research I found the the summer school extemely useful. It contained some important introductory pieces as well as covering many topics in more depth suitable for anyone wishing to use the NGS. Sessions consisted of a good mix of high level overviews and hands-on practicals such as using the P-GRADE portal.

The presenters were very knowledgable and helpful and were eager to inform on all aspects of grid computing. Some sessions involved the speakers talking individually with attendees and I found the staff gave useful advice and were very supportive of my project.

Before the school I thought my particular research area, the forensic investigation of mobile phones, might be a good fit for grid computing, and by the end of the summer school I felt much more able to pursue my research in an informed manner using the NGS. It was useful to know how grid, cloud and high performance computing relate to one another  and over the four days I feel I became much more aware, in a less naive way, of how my research would benefit from the resources the NGS has to offer. In particular I learnt to think about my research differently: how my phone investigation process would be best able to benefit from a grid architecture.

Indeed the school helped support my view that the NGS would be a good resource for ANY project that needs HPC not just vast number crunching modelling applications such as those such by molecular biologists or quantum level physicists (although of course it is good for that too).

I liked the food too!"

And it wasn't just Ed who enjoyed the summer school!

"The summer school has been a fascinating activity. The hands on sessions have familiarized us with HPC/Grid/Cloud, which are useful resourses that I have never had access to or known how to access, whilst the Meeting Champions and Q&A sessions made it possible for us to know how these resourses could facilitate our research. The SeiUCCR summer school provides a great opportunity of learning, communicating and networking. I would like to thank all the people who made this summer school possible"

"The summer school was a relaxed and friendly environment. It provided detailed information about the various resources available to researchers both in terms of computer resources and support. The staff where very approachable and keen to show an interest in the attendees work. Overall it was a great experience I would recommend to anyone who has an interest in grid/cloud/HPC or who's work may benefit from such technologies."


For anyone who was unsuccessful in obtaining a place at the e-infrastructure summer school, the presentations are now available online.  Due to receiving over 120 applications for the summer school and only having a limited number of places, I know many people were keen to see the presentations.

There will be another summer school in 2012 so watch this space for forthcoming announcements next year!




Thursday 29 September 2011

Blogging off

It is the last day of All Hands 2011 and it is my last day working for the NGS.

After 4 years of general griddery, I'm moving on.

Four years is a long time in research, and today's All Hands meeting at York is very different from the first grid event I attended, Open Grid Forum 20 in Manchester.

I remember that the Manchester meeting was huge and full of international delegates.

The UK contingent were based in something called the UK e-Science Village - which conjured up bucollic images of computer scientists dancing around the maypole on the e-Science village green, just next to the local shop for local people.

At the very least, I was hoping to see the UK e-Science Village People giving a rousing chorus of their classic - `(its fun to be at the) STFC.'

The village turned out to be a very large display booth.

All Hands is national, rather than international. The conference and the booths are smaller. As at OGF, people still enthuse about shiny new technology that will solve all our problems in the future.

But in among them are people using the less-shiny, less-all-singing, less-all-dancing software that we have now. And they are using it to do new research that is nothing to do with the technology itself.

And it is those are the people I want to hear - because what I have learned to call e-Infrastructure is very broad - in one session yesterday, the talks covered the behaviour of the heart, and how what the researchers have learned there has been applied to the way muscles move when giving birth; and how to model the way water shapes landscapes over millennia.

I still do not give a damn about how clever, or web-service-y, or standards compliant, a bit of e-Infrstructure is. It is what you do with it that counts.

It is the researchers who have take what we provide and use it to deliver the research that could not otherwise be done. These are the people you can read about in the case studies.

These are the people who have turned e-Research into Research - and will continue to do so for many years to come.

Tuesday 27 September 2011

Goodbye UKI, hello NGI_UK

At All Hands 2011, in the atrium of the University of York's brand new Ron Cooke Hub conference venue.

On our stand in the middle of the room is a familiar face - helpdesk manager John Kewley - sitting under a slightly less familiar sign.

It doesn't say NGS, or GridPP, although both have posters on display.

The sign says but 'NGI' - aka National Grid Infrastructure - and we have had to to get used to it very quickly.

At  last week's EGI technical forum, what was the UKI ROC - or the UK and Ireland Regional Operation Centre - was offically replaced by two new NGIs called NGI_UK and NGI_IE.

And lots of things broke - including the load monitor and the Nagios testing service.

Names matter. Both the load monitor and Nagios were pulling information about sites and users from the Grid Operations Centre Database. More specifically, they will pulling information about sites and users associated with the UKI ROC.

The UKI ROC is no more: it has no sites or users associated with it.

So... we have spent the last few days tracking down every reference to the 'UKI' in every configuration file for every service and replacing them with NGI_UK.

There were quite a few....

The load monitor is back. We've been working on Nagios today and it should be fully working soon.

Monday 19 September 2011

Three Little Words

There are these three little words. For some people, these words bring feelings of fulfilment and contentment. For others, they bring nothing but frustration.

Those three little words are:

  Proof of Concept

For that part of the e-Research Community interested in how research will be done in future, A proof of concept is evidence that it is possible to do something new and interesting, using something new and interesting. It might change the way research is done next decade. It is more than enough for a published paper and a presentation at All Hands.

And it is of bog-all use to those for whom e-Research is simply a means to an end. They just want something that works now and works reliably.

There is always a gap between the potentially useful and the actually useful. When you can build something that bridges that gap, you can enable research that would not otherwise be done.

Which brings me to slightly embarrassing news that our project to deploy the ARC middleware in front of the local High Performance Computing service has been a complete success... as a proof of concept.

We have shown that it is possible to deploy ARC services in front of what we should now be calling Oracle Grid Engine.

With some inventive use of ssh copies in prolog and epilog scripts --- that this can be made to work even where there is no file-space shared between the grid 'front end' and the HPC cluster.

We also know that you can support parallel tasks  using ARCs Runtime Environment mechanism --- there are examples at the bottom of the (slightly out of date) Nordugrid documentation --- and make use of to the LCAS/LCMAPS authentication system used by other grid software.

Which is nice....

Whether it is going to be useful is a completely different question.  We do not yet know if the local communities who are best placed to use it --- the rather incongruous pairing of Solar Physics and Social Science --- will want to do so.

Epilogue: Prologs and Epilogs


A quick technical note on faking a shared directory via Grid Engine prolog and epilog scripts.

The scripts run just before the start and just after the end of every job.

ARC-the-middleware obligingly changes directory to the 'shared' scratch directory before submitting the job. This mean that prolog and epilog scripts are presented with the path to this directory in the $SGE_O_WORKDIR environment variable.

The recipe is along the lines of...

  • Create a ssh keypair for each user - to be used solely for transfers from HPC backend to grid front end
  • Copy the private key to a safe place on the HPC back end, readable only by the user. We will call this $GRID_KEYS.
  • Use the public key to create a per-user authorized_key file on the grid front end in somewhere like
       /etc/ssh/authorized_keys.d/$USER
    and change the /etc/ssh/sshd_config (again on the grid-front-end) to set.
        AuthorizedKeysFile  /etc/ssh/authorized_keys.d/%u
  • Add code to prolog and epilog to use scp (or rdist) with the -i $GRID_KEYS/$USER to pull files from $SGE_O_WORKDIR at the beginning of the job and push them back at the end.




Tuesday 13 September 2011

Conference season approaches...

It's gone slightly quieter for me now that the SeIUCCR e-infrastructure summer school is safely under way.  30 students are now ensconced in Coesner’s House in Abingdon where they are learning about the wonders of e-infrastructure and how it can help their research.  As I type they will have just finished a “hands on” session on the NGS and how to run jobs on our resources.

One event is underway but we still have two to go.  Next week sees many of the NGS staff at the EGI Technical Forum in Lyon.  The NGS in conjunction with GridPP is the UK National Grid Infrastructure (UK NGI) and in turn the UK NGI is part of EGI (European Grid Infrastructure).

It’s a very active meeting for many NGS staff due to the level of involvement we have in this major project.  As well as meetings, there will also be presentations in several sessions from NGS staff.  I’ve been asked to give a presentation on the NGS roadshows as EGI are developing their own roadshows – well they say that imitation is the greatest form of flattery!  As always the UK NGI will have a stand at the event where people can talk to us further about our activities, meet staff and obtain information.  If you are attending the EGI Technical Forum then drop by and see us.

The week following the EGI conference, many of us will be in York for the UK e-Science All Hands Meeting.  Registration for this is open until the 19th of September so if you want to go, make sure you register soon!  Again the NGS will have an exhibition stand along with GridPP at the event.  The exhibition stand will be a hive of activity as there will be several demos taking place here.  The demos are –
  • Applying for UK e-Science Certificates using the new CA Certificate tool
  • Taverna Server: Towards enabling long running workflows on the NGS
Some of our users will be actively taking part in the conference with demos and presentations not to mention NGS staff giving presentations and posters as well.

A major activity at AHM is a workshop organised by SeIUCCR which is a collaboration between the NGS and the Software Sustainability Institute (SSI).    The workshop is entitled  "Meet the Champions" and will take place on the Tuesday 13:30-16:30.

The workshop is an opportunity to meet researchers that have been promoting and leading research over the past decade of e-Science; find out about their work and how they utilise e-Infrastructure, and learn how you can interact with them.  Specifically the "Champions" to meet are members of the Community Champions network from the SeIUCCR (Supporting e-Infrastructure Uptake through Community Champions) project; the NGS Campus Champions and the Software Sustainability Institute Agents Network.

There will also be 2 key presentations -
  • Scott Lathrop is Blue Waters Technical Program Manager for Education and TeraGrid Area Director for Education, Outreach and Training.  Scott's talk is entitled "Engaging Campuses in XSEDE".  XSEDE is the successor to TeraGrid.  Scott will be talking about the XSEDE Campus Champions programme and also the Campus Bridging programme for XSEDE.
  • Steve Brewer is Chief Community Officer of EGI, the European Grid Infrastructure and he will be talking about Community Engagement in Europe.
And if you thought that was enough there will also be a panel session on the question "Why should researchers use e-Infrastructure?".

For the full details of all the NGS activities at the forthcoming AHM meeting please see the news article on the NGS website.

Hopefully we'll see some of you at some point over the next two weeks!

Wednesday 7 September 2011

It is not easy being parallel

As has been said before - there are differences between Grid and traditional High Performance Computing. Some of the differences are due less to the technology and more to the problems being solved.

The more successful grid users are task farmers: they scatter comparatively small compute tasks and data and wait for them to grow into results. The grid - metaphorically speaking - is there to plough the land, spread  the fertilizer and muck out the system administrators.

Traditional HPC concerns itself with big applications and - in particular - applications that are too big to fit on a single computer. HPC systems are built with parallel computing in mind.

The Grid does not do parallel computing well.

Consider the two steps in running any parallel tasks
  • Asking for more than one CPU core on the same system.
  • Setting those CPU codes to work
For each step, there is definitely more than one way to do it...

Take 4...

So, there you are, sitting by your favourite grid client, a freshly minted X509 proxy ready. All you need to answer one of the great problems of modern science is 4 CPUs.

All you need to do is ask.

How you ask depends on who you are asking and what grid dialect they understand.

Globus GRAM5 and ARC accept tasks defined in Globus  Resource Specification Language (RSL), possibly with some Nordic extensions. In RSL, you can ask for more than one CPU with an additional:

  (count=4)

The web-service-y Globus job submission systems (WS-GRAM) used a similar approach but written as XML.

In Job Description Language, as understood by the gLite CREAM-CE and WMS, you need

  CPUNumber=4;

And in the OpenGridForum-approved XML-based Standard Job Specification Description Language, you have the instantly-memorable and easily-readable:

    <jdsl:TotalCPUCount>
       <jdsl:Exact>4.0</jdsl:Exact>
    </jdsl:TotalCPUCount>

(which you will find buried somewhere under 3 levels of XML tags). 

Yes - I know JSDL isn't really there for humans to read, but it doesn't stop some humans trying.

4 go to work...

That was the easy part.

Now it gets complicated.

And, on this occasion, you can't blame the Grid for the complexity.

Large-scale parallel programs are typically written around libraries implementing the Message Passing Interface (MPI). There is more than one version of the MPI standard and more than one library implementing them.

To add to the confusion, from some MPI variants, you need to build versions for each FORTRAN compiler installed.

Launching a parallel job depends on both the job management software and the underlying mechanisms used for communication. MPI installations typically provide either an mpirun or mpiexec command that ensures that the right processes are started in the right way on the right computers.

It is very likely that each version or each MPI implemention will have its own variant of mpirun or mpiexec. It is equally likely that - at least for mpirun - they will expect different arguments.

In the first and second phases of the NGS, we were funded to provide exemplar Grid clusters at RAL, Oxford, Leeds and Manchester. The grid software we deployed - Pre-WS GRAM from Globus 4 - could launch MPI jobs if

  (jobtype="mpi")

was included in the RSL.

It could only launch one of the many possible mpirun commands. To work around this, devious system administrators cooked up a sort of super-mpirun that would locate the correct version for an applications.

Researcher in Ireland found ways of launching MPI jobs from within JDL jobs - but they could not hide all the complexity.

ARC supports parallel jobs via its Runtime Environments extension - which can tune the environment for an application so that the right number of CPUs are assigned and the right mpirun is run. Again, this needs the  system administrator to do something devious if it is to work.

We haven't even begin to cover parallel programs written outside MPI - such as those using the Java sort-of-MPI library MPJ-Express.

So... what am I trying to say?

It would be nice to have a conclusion, or at least a lame joke, to end this blog post - but I can't think of one.

All I can say is that parallel computing is complicated, distributed computing is complicated and that any attempt to combine the two - either using existing Grid solutions, or something newer, shinier and probably invoking the word Cloud - cannot make either kind of complicated vanish completely.

Thursday 1 September 2011

Where did August go?

Usually August is a quiet month outreach wise at the NGS but this year it seems to have been the complete opposite.  The SeIUCCR summer school is fast approaching and I've been busy sorting out registrations, accommodation and queries for that.  The summer school was massively oversubscribed with 4 people applying for each place.  Demonstrates that there is quite a demand out there for training in e-infrastructure across institutions in the UK!  Thankfully all speakers and delegates appear to be sorted so perhaps now I can breathe a sigh of relief.  If you were not able to get a place at the summer school keep an eye on the NGS website and mailing list as the material from the course will be made available online.

In other news there is a new NGS user case study up on our website.  This time Maria Holstensson from the Institute of Cancer explains how she is using NGS resources to improve cancer treatment for children.

Children with neuroblastoma who are being treated with targeted radionuclide therapy can have their treatment monitored with gamma camera images. These images are used to calculate the amount of drug taken up by the tumour and to estimate the radiation dose. However the image quality can be poor due to scattering and interference. Maria Holstensson from the Institute of Cancer is looking at tackling this problem.

It never fails to amaze me the range of research carried out on NGS resources.  We have users from every area from linguistic analysis to high energy physics and we are always looking for more.  There are now 23 user case studies on the NGS website and I hope that they demonstrate that e-infrastructure is for everyone and not just those from physics or computing research areas. 

This may be a good time to mention an interesting blog post from Steve Crouch over on the Ask Steve blog from the Software Sustainability Institute (SSI).  He's been musing on communications between developers and researchers - do they really speak the same language?  Comments will no doubt be welcome!




Friday 26 August 2011

Sign Here

I would not want to describe the paperwork that goes with University life as something out of
Terry Gilliam's Brazil or Yes Minister.

It would be ill-advised: I haven't completed this month's NGS/B/11347/2(a) (permission to use ironic over-exaggeration within a blog) and submitted it to the appropriate authorities.

Depending on how deeply your institution loves its paperwork, there will be forms to be complete when claiming travel costs, or buying a new HPC system, or obtaining a replacement biro. Inevitably, somebody else needs to sign these forms to show that there has been due diligence and that the trip to Didcot, million pound compute cluster or cheap plastic pen are fully justified.

Somebody else isn't just anybody. When you present your form to the powers-that-be, the powers-that-be will carefully compare the signature with their collection of scribbles from the great-and-the-good.

Only when you have the right name in the right place on the right form will you will receive a new pen and a firm lecture about being more careful in future.

As we have said on a number of occasions, grid security is built on chains of trust. It also relies on the right signature being used in the right place. In our case, these are digital signatures represented by X.509 certificates rather than the spiders-web-on-acid scrawl of a senior University manager.

A certificate in your local list of trusted certificates - typically in /etc/grid-security/certificates - can be accompanied by a file defining its signing-policy. You can see some examples of signing policy files in the UK eScience Certification Authority pages on the website.

The signing policy is particularly influential at the very far end of the chain of trust: the root certificates. The private key associated with root certificates are kept in a Very Safe Place and are taken out only to sign the certificates of Certification Authorities (CAs).

CAs sign the certificates for the rest of us. The UK has two CA's - the main eScience CA and a SARoNGS CA.

Over the last weeks, thanks to the efforts of the dragon-slayers at the Software Sustainability Institute, we finally found out why certificates from our 'SARoNGS' CA were being rejected by the NGS's Workload Management Service.

There was nothing wrong with the certificates themselves.

The SSI developers quickly identify problems with the SARoNGS Certificate Revocation List (CRL) - a list of known-bad certificates that CA's should distribute.

SARoNGS certificates are designed to be short-lived - they expire before anyone gets a chance to do something bad with them - and the revocation list is empty. But all revocation lists - even empty ones - have expiry dates and ours had, unfortunately, gone stale.

Updating the CRL was comparatively easy but it did not solve the problem. The root cause of turned out to be the root certificate's signing policy.

The problem is that there are two signing policies - depending on whether you consider SARoNGS certificates acceptable.

SARoNGS certificates can be obtained using only a UK academic username and password whereas a full eScience certificates requires photo ID and a visit to your local Registration Authority.

The International Grid Trust Federation (IGTF) is responsible for ensuring that certificates are being created and managed in a trust-worthy way. It has strict rules on what constitutes sufficient proof of a users identity and - not to put too fine a point on it - an academic username and password are simply not good enough.

So the signing policy within the IGTF's bundle of UK eScience certificate information does not currently match the version we distribute. The IGFT version will not permit the eScience root to sign for the SARoNGS CA.

The root cause was a misplaced update that installed the IGTF version of the eScience root signing-policy - rather than the NGS's own.

We should have had the 'IGTF+' certificates - a modified version of the IGTF's certificate collection maintained by the NGS blogs' very own Jens Jensen, and incorporating the NGS's signing policy and some additional certificates.

The IGTF+ certificates are available in a number of formats from Jens's avowedly Web-1.0 certificate repository webpage.

[With thanks to James Perry, Steve Crouch and Rob Baxter of the Software Sustainability Institute]

Monday 22 August 2011

What do you do on the NGS?

It's been a busy month or so even though it's the holidays as this is the ideal time for me to contact many of the NGS users who promised to write user case studies for me regarding their research.

We have a wide range of reseachers who use the NGS as part of their every day research and it's important for the NGS to highlight that our resources aren't just used by the "typical suspects" such as physicists.

Over the last few weeks I've added another 2 user case studies to the NGS website -
Both these case studies demonstrate how the use of NGS resources is helping to speed up research enabling results to be produced and published faster than previously.

Edwards supervisor, Dr Anna Croft praised the NGS, "The NGS has been an excellent resource for many of our research projects. In particular, I have been able to use it with undergraduate researchers and give them a taste of what it is like to work on large computing infrastructures - an experience that has helped some of them secure PhD funding, both here and overseas, to continue in the computational area. When we had teething problems, the support staff were always friendly, helpful and got things working. Because of this support and the flexibility in requesting computing time, the NGS is one of our first ports of call for projects requiring a larger computing resource."

Thank you Anna!

Thursday 18 August 2011

A good-enough impression

Leeds - as a long standing NGS partner site - want to hook our HPC service into the Grid.

We hope to fill the gap left when the last of our NGS-funded clusters was turned off back in April. Our main requirement was that the grid front end should be completely separate from the HPC service. In addition, we wanted...
We had originally hoped to follow the particle physicists and deploy CREAM.
Unfortunately, EMI-1 was missing the components needed to make CREAM work with the SGE batch system used locally. The only software within EMI-1 that was SGE-friendly was Nordugrid's Advanced Resource Connector - ARC.

After a few months of work and in the great tradition of the grid: it is sort-of-kind-of-working-after-a-fashion. At the moment:
  • ARC's compute service - A-REX - is accepting jobs: for a very limited set of users and not from the workload management system.
  • ARC's information provider - ARIS - is publishing information about the system and this information is making its way to the NGS's BDII.
I'll will cover A-REX in a future post. This week, you are getting information about the information provider - and in particular, how it links into the NGS.

A bit of background. The NGS information service is a Berkeley Database Information Index service or BDII. BDIIs are built to collate information, some of which comes from other BDIIs. The NGS's central BDII, for example, collates information published by a BDII, or something that looks like a BDII, at each of the partner sites.

ARIS can do a impression of a BDII. Whether it is a convincing impression depends on what it is talking to.

ARIS produces information in its own Nordic-accented schema, designed to feed the ARC tools. This needs to be translated into GLUE format before a BDII will give it a second glance.

Based on documentation on linking ARC and EGI from Nordugrid,, this can all be done via a single ARC configuration file called /etc/arc.conf.  arc.conf consists of blocks, denoted by a [name in square brackets] each containing a set of name=value definitions.

arc.conf needs to be tweaked in three places.

Turn on publishing of Glue 1.2 format information - which  is close enough to the current common Glue version 1.3 - by adding to the '[infosys]' block.


 [infosys]
  ...
 infosys_compat=disable
 infosys_nordugrid=enable
 infosys_glue12=enable

Add in anything that Glue needs and ARIS does not via the '[infosys/glue12]' block:

 [infosys/glue12]
 glue_site_unique_id="NGS-LEEDS"
 ...
 provide_glue_site_info=true

And finally arrange for ARIS to collect its own output and present it as if it were a site BDII by a block called


 [infosys/site/NGS-LEEDS]
 unique_id=NGS-LEEDS
 url=ldap://ngs.arc1.leeds.ac.uk:2135/mds-vo-name=resource,o=grid

Our initial experiments suggest that the information produced by ARIS is good-enough to be accepted the NGS's central BDII but not good enough to fool our Nagios monitoring.

WLCG Nagios includes a number of BDII specific tests including one called org.bdii.Entries. org.bdii.Entries only looks for 'services' - or more accurately objects of the 'GlueService' type. While ARIS generates a lot of information, none of describes a GlueService.

What we don't yet know if it Nagios is being picky, or whether the existence of a GlueService is vital for some bit of grid wizardry.

Wednesday 10 August 2011

E's no good - Distinguishing between distinguished names


For most people, changes to the policies and standards that describe how the grid should work are met with a resounding 'so what'. For anyone involved in the day-to-day management of grid systems, it is an opportunity to join a collective sign-of-relief.

It is another example of where the 'political' aspects of international research collide with the technical solutions and the needs of researchers who don't give a damn how it works, as long as it lets them do their jobs.

X.509 certificates are complicated because what they represent is complicated - a link in a chain of trust between particular individuals or institutions.

Identities within certificates are tied to Distinguished Names or DNs. A DN is a lists of attributes - such as country, institution and personal name - that uniquely identify a single person, or computer, or service.

The way a DN is stored within a certificate is well-defined but completely incomprehensible to anything that is not a computer program. For many practical purposes, the DN needs to be presented so it can be understood by a person.

A glance at the OpenSSL X509_NAME_print_ex documentation shows how brain-twistingly complicated it can be translating a DN into something that a human being can read.

There is a more detailed explanation on the NGS Wiki. This is the quick tour..

Each individual attribute within a DN has a 'type' and a 'value'.

The type identifies what is being represented - a name, or an email address. It isn't really a name but but a unique sequence of numbers called an Object Identifier. Something like: 1,2,840,113549,1,9,1.

People, inexplicably, find sequences like 1,2,840,113549,1,9,1 hard to remember so for our benefit, 1,2,840,113549,1,9,1 is also known as "Email", "emailAddress" and - occasionally - "E".

The value is depends on the type. For 1,2,840,113549,1,9,1 - it is a string of letters represented in what is known as UTF-8. UTF-8 was developed to represent any letter from any language - but most Grid certification authorities deliberately restrict themselves to the 26 letters of the English alphabet, the numbers 0 to 9 and a few symbols. If they didn't, things would rapidly become even more complicated.

In human-friendly form, the DNs that Jens is working to abolish look very much like

 /C=UK/O=eScience/OU=Manchester/L=MC/CN=voms.ngs.ac.uk/Email=support@grid-support.ac.uk
or maybe
 /C=UK/O=eScience/OU=Manchester/L=MC/CN=voms.ngs.ac.uk/emailAddress=support@grid-support.ac.uk
or even, very rarely
 /C=UK/O=eScience/OU=Manchester/L=MC/CN=voms.ngs.ac.uk/E=support@grid-support.ac.uk
Which variant you get depends on which version of which software is processing the certificate.

The problems appear when DNs are compared as strings of letters rather than in what could be called their 'raw' form.

Most software is smart enough to canonicalise these awkward examples by chosing One True Name for 1,2,840,113549,1,9,1 and substituting this before comparing. Not all software packages agree on which name is the One True Name.

It is now common practice to represent certificate chains in .LSC format - which are simply lists of human-friendly DNs. These may be simple to distribute and do not need to be updated every time the certificate is renewed.

The would be good enough - if it wasn't for that troublesome email address.

Monday 8 August 2011

On email address in host certificates

Every so often we get questions about email addresses in the names (distinguished names, ie DNs) of host certificates. The problem is that they are deprecated (see the last two paragraphs of section 4.1.2.6 of RFC5280), and they cause all sorts of problems with software which stringifies the DNs because there is no consistent way of doing it (or rather, there are too many consistent ways.) Arguably the software is not coded correctly, but in this case it'd be better to remove the email.

The email is there for historical reasons: when we rekey a certificate we have to give it the same name as before, so that's why it is still there. Dating back ten years or so, the original raison d'être was that before robot certificates, hosts would sometimes run stuff on behalf of users, ie. act as a client, and the email address was meant to give you something to contact when you read the DN in the log file.

The new policy will permit removing the email address from DNs. That's the easy bit.

The trick is to get the software to optionally (at the owner's request) remove the email address from the DN (because some people may genuinely want to keep it, for whatever reason.) Or rather, optionally keep it. The software cannot do this yet.

In fact, it'd be easier to just remove it for all host certificates, or maybe to handle those "manually" who still want to keep it, as with robots for example. If anyone out there has host certificates and depends on email being present in the DN, could you let us know via the usual channels, please? There are no known problems with removing the email address, only with keeping it, but there may of course be unknown problems - there are lots of weird and wonderful things out there.

As for timescale, it'll be ready at the latest when the new (rollover) CA certificates go live at the end of September.

Thursday 4 August 2011

E-infrastructure summer school - registration open now!

A brief hiatus here on the NGS blog as several of us are / have been on holiday. Back to normal service now hopefully!

A lot of my time since I returned from holiday has been devoted not to the NGS but to another project I am currently working on. Catchily named SeIUCCR (pronounced "sucker"), the project was funded by the EPSRC "Crossing the Chasm" call which called for networks and "advocates" to promote the wider uptake of UK e-infrastructures by researchers in engineering and the physical sciences.

Part of the SeIUCCR project is an e-infrastructure summer school which is due to take place in Abdingdon near Oxford in September. The residential summer school will offer an introduction to e-infrastructure including Clouds and Grids to UK PhD students and postdocs over 4 days. The summer school is fully funded including travel expenses and applications are open now.

If you (or anyone you know) would like to apply then be quick as applications will close at 9am on Monday 15th of August. Details of the summer school can be found on the SeIUCCR website and a detailed agenda is available from the registration site.

Wednesday 20 July 2011

On the 97th of April 2011...

Back in February, an over-optimistic fool promised that the NGS would have a working Nagios service in the next few weeks.

The over-optimistic fool was confident because he had a real deadline to meet. Nagios had to be ready by April. April was the month during which the old NGS core sites - which ran the tests for our old INCA-based testing framework - were to be decommissioned.

We are running little late... but I am pleased to say that 2 weeks ago - on Wednesday the 97th of April 2011 - the NGS's Nagios testing service finally went live.

If you have an certificate and it is listed in the Grid Operations Centre database - you can pay it a visit at https://nagios01.ngs.ac.uk/nagios.

If you haven't or aren't - sorry: WLCG Nagios, unlike INCA, denies access to unregistered users by default. We may be able to remove the restriction in future - but, for the moment, we want to focus on fixing the problems it has found.

It is a bit untidy - as we have been without a fully working monitoring service for over 6 months.

While we kept the INCA service running as long as possible, it had become increasingly out of step due to a decision - very early on - to use the 'NeSCForge' software repository as a safe place to keep its configuration.

NeSCForge was not as safe as we had hoped. It vanished in December last year. The list of sites and tests to run remained frozen in their December state... and the Grid moved on.

We have different partner sites offering different services now. INCA wasn't testing them, Nagios is.

More significantly, Nagios takes its list of sites directly from the Grid Operations Centre database. Changes made there should be reflected in Nagios within a day.

My colleagues in the NGS Partnership team are working their way through the Nagios test results. They are identifying problems, finding missing sites and services - and, most importantly, working out how to make things better.

Tuesday 19 July 2011

Avoid meaningless pretty pictures

With the OGF science-in-the-cloud SAUCG workshop closing, it is time to reflect on the many interesting presentations, and try to identify common areas, next steps, etc.

How do we best provision resources for scientists? "Cloud" is a buzzword but there are drivers behind the push for it: increasing resource utilisation (maybe), service provision for small customers (the large, from the service provider perspective, being griddy), dynamically catching up with work and coping with the last-minute work prior to a conference. Lots of projects presented interesting stuff - see the slides - and expect an NGS surgery on the topic. To take this forward we now need to look at roadmaps - eg NIST and SIENA - identify gaps etc.

And the award for the quote-of-the-day goes to Etienne Urbah for the title of this post, and for his recommendation that "Passive sentences should be avoided."

PS. If you pronounce SAUCG "sausage" then it's entirely your own fault.

Thursday 14 July 2011

Going to town

One of the local organisers said that they could not recall a time when the lecture theatre - which officially holds 185 people - had been so full.

It was all the more remarkable given that it was being used for a meeting called - deep breath -  A Town Meeting to discuss UK Strategy for a Research Computing Ecosystem and the Future of e-Science.

As a rule: meetings that mention 'UK Strategy',  'Computing Ecosystem' and - especially - 'e-Science' do not attract huge numbers of people. This one was special because, somehow, the organisers had persuaded every branch of that amorphous thing called e-Science to come along.

There were the people from PRACE - who provide the really big compute for solving really big problems - and  people who run the Institutional High Performance Computing services that drive so much UK research. We had the big data brigade - from Bioinformatics and Earth Systems science - who feed new research. We had representatives from research computing services, institutional IT services and the JANET network. We had the academics who push the limits of what you can do with a computer.

And, of course, there were representatives of the Grid - including the NGS, the Particle Physics community and less-traditional-users such as biology.

And everyone in the room agreed on what we needed to do.

I'll repeat that.

Nearly 200 people involved in academic research gathered in a room and unanimously agreed on what we needed to do next.


That is `what we needed to do' not `how we were going to do it'.

Everyone agreed that 'e-Science' must be driven by what the people who do the research actually need.

Everyone agreed that training for researchers is vital.

Everyone agreed that well-written robust software leads to better research.

Personally, I would like to have heard less agreement and more discussion.  The e-Science community is full of people who have tackled difficult problems - sometimes successfully, sometimes less so - but the town meeting was simply too large scale for discussions.

Technical discussions are best served by gathering small groups of well informed people. They can get quite heated, but this is not necessarily a bad thing. The Moonshot meeting the day before was about the right size.

That is not to say that the discussions - heated or otherwise - did not happen. It just that they happened outside the meeting, by the coffee urn, or in the pub, or on the slow train home - between smaller groups of people who happened to be in London at the same time for a big meeting with a very unwieldy name.

If you weren't among the attendees, you can find some of the presentations on meeting's web page and follow the collective twitterings of some of those who were.

Monday 11 July 2011

On Moonshot and telling the world who you are.

The International Coffee Organisation's Board Room would make a damn good lair for a James Bond villain.

It also served rather well as a venue for a workshop organised by Project Moonshot, held last Thursday and focuses on using moonshot-authentication in Grid and High Performance Computing.

Josh Howlett - JANET's Mr. Moonshot and the workshop organiser - singularly failed to bring a white cat to stroke. And if he had a secret button to drop troublesome guests into his pet shark's tank - he resisted the urge to use it.

He was, however, quite happy to describe his plans to Take Over The World.

We've mentioned Moonshot before: it's goal is to re-use the network of authentication servers that has been created to provide Eduroam to control access to other services.

Moonshot allows people to authenticate themselves securely using their 'home' username and password. It is based around Tunneled Transport Layer Security provided by the Extensible Authentication Protocol and a network of RADIUS servers.

A service can refer authentication decisions onto a remote Authentication (AAA) server. Any chatter between the client and the AAA server that proves the user is who he or she claims to be - such as username, passwords or SPECTRE membership number - is hidden from the service itself. For the simplest uses, all the service needs to see is a simple yes or no.

There are many places where Moonshot could make life easier:
  • Moonshot could make is easier to share High Performance Computers. If it delivers what it promises, you could be granted SSH-access to a service anywhere in the world without needing a separate username and password.
  • In the grid world, adding a sprinkle of Moonshot magic to a Myproxy service or to a Credential Translation Services could make grid certificates available without resorting to a web browser.
This is where things get interesting, or complicated, or political. Depending on your point of view and position in the IT food chain.

For SSH or certificate access, the service needs to obtain some kind of unique, persistent identifier for every user.

But for the current Eduroam service - all you need is confirmation that the user is from a particular institution. It does not need to know if they are the Vice Chancellor, an esteemed professor or a junior researcher.

At the moment all an institution's RADIUS server need do is confirm or deny that the person connecting is a legitimate user of the network. There are no unique identifiers involved.

For Moonshot to be of use in the grid and HPC worlds, institutional RADIUS servers need to release additional information that can be passed back to the service.

The question is what additional information?
  • Should it be an email address? james.bond@mi6.gov.uk
  • Should it be something like a login-identifier? bondjames@mi6.gov.uk.
  • Or should it be pseudo-anonymous? 007@mi6.gov.uk. (If you don't have a license to kill, then this would be something like the Shibboleth eduPersonTargettedId - as used by the UK Access Management Federation - which is unique to a person and a service)
All have advantages and disadvantages....
  • IT Security People really dislike seeing usernames being released. You really don't want to give a potential attacker any help in cracking into a system.
  • There are legal and licensing rules that restricts access to certain classes of data - such as Ordinance Survey maps - to named individuals. Likewise, HPC service managers are far happier granting access based on an email address rather than a random collection of characters.
  • Researchers in some fields, especially Life Sciences, are understandably protective of their personal information and would much prefer pseudo-anonymity.
This is far too complicated a problem to solve at a single meeting, even if one has the the threat of becoming a shark's lunchtime snack to concentrate the mind.

Moonshot is a very impressive project, with international reach and practical contributions from experts in the field. They strike me as the right people to solve it.