NGS

Interested in accessing and managing grid resources?

2012-01-26T16:00:00.000Z

If so then read on!

The NGS is hosting a short but sweet seminar series starting next Wednesday (1st Feb). There will be 3 seminars over the 3 weeks each lasting approximately 30 minutes and the best thing about them is that you can join in no matter where you are - all you need is the internet!

We wanted to make the seminars as open to everyone as we possibly could and, after some deliberation, we decided to use the Evo technology. This is free for everyone to use - all you have to do is to register and I recommend doing this at least the day before. This isn't anything to do with Evo's registration process more that it took several hours for my university email system to allow my confirmation email through...

So what are the topics that we will be discussing?

1st February - Shibboleth Access to Resources on the NGS – Mike Jones, NGS, University of Manchester
This talk will demonstrate how it is possible to access and use NGS resources using institutional login credentials (via the UK Access Management Federation). It will describe how the UK's two main e-Science authentication systems are combined to form an easy to use yet robust identity management environment. It will discuss how this mechanism links together with system, project and Virtual Organisation (VO) registration procedures.

8th February - Certificate Management in the UK - John Kewley, NGS, STFC Daresbury Laboratory
The NGS helpdesk receives many tickets relating to certificates (and certificate renewal in particular): largely due to browser incompatibilities. In order to tackle this problem, the NGS has devised CertWizard which is a browser-independent certificate tool. The presentation will give an introduction to the UK e-Science CA, which has issued over 30,000 certificates, and its associated software and interfaces, including CertWizard.
It will show how modernisations are being made at various stages of the certificate lifecycle, making it easier than ever for users to manage their e-Science Certificate.

15th February - Moonshot - next generation federated identity - Josh Howlett, JANET
Federated identity yields significant benefits for users and services by increasing the usability of services, reducing identity management costs and improving regulatory compliance.
A number of different technical strategies for federating identity have emerged during the past decade, with differing levels of success. These technologies address different types of use case, resulting in significant complexity for both users, services and trust infrastructure providers.
This complexity impedes the adoption of services and increasing operational costs. Moreover, there are many use cases where these technologies do not provide a solution.
Project Moonshot is an ambitious Janet-led initiative, building on existing deployed technologies, that aim to develop a single unified and standardised approach that satisfies all of the authentication and authorisation requirements of the education & research community. Much of the technology has now been implemented, and is now being tested within the Janet Moonshot Technology Pilot.
This presentation will provide an overview of some of the motivating use cases for Moonshot and an overview of the technology and the implementation.

Full details of how to join the seminars are available on the NGS website event page but if you have any queries then please contact the helpdesk and we will do our utmost to help you join in.

It's that time of year again...

2012-01-19T16:11:00.002Z

My inbox seems to be full of emails regarding conference calls for papers, early bird registrations, conference deadlines etc. Yes it's conference preparation season and its in full swing!

I received confirmation today that I'll be giving a paper at the forthcoming EGI Community Forum on our champions networks. I'll be talking about both our Campus and Community champion networks and how we work with each other to promote e-infrastructure in the UK. Several other NGS staff have also had papers accepted on topics including "Linking Authenticating and Authorising Infrastructures in the UK NGI (SARoNGS)" (Mike Jones) and "Tweaking the Certificate Lifecycle for the UK eScience CA" (John Kewley).

Also in my inbox this week was an announcement from the Software Sustainability Institute (SSI) announcing that registration for their Collaboration Workshop 2012 (CW) is now open. This is on of my favourite events as, unlike most conferences, you don't sit passively listening. The CW consists of breakout groups where you discuss topics submitted by the attendees and there's always one of interest to me in every session. After the discussion a member of the break out group volunteers to report back to the CW as a whole. This means that you get to hear what all the other break out groups were talking about and you can still feedback on their outcomes as well.

It's a really lively meeting and you leave after 2 days feeling tired but feeling that you've achieved something worthwhile! It's also a great place for networking with new people as there are researchers from a wide variety of research areas, IT people, community support people and people like myself who represent national initiatives. To see some of the topics already suggested for discussion visit the event website.

Just incase you missed it....

2012-01-12T11:10:00.002Z

A new edition of the quarterly NGS newsletter was released in December so if you missed it in the pre-Christmas rush, now is a chance to catch up!

This edition featured articles on -

the adoption of Globus Online by the NGS
NGS involvement in the EGI Federated Cloud Task Force and the benefits for NGS users
NGS user case study - Scalable Road Traffic Monitoring using Grid Computing
...and more!

I am always looking for new articles or suggestions for articles for the newsletter so if you have anything you would like to see in the next edition (March) then please let me know. The next edition will coincide with conference season so copies of the newsletter will be distributed at the forthcoming SSI Collaboration Workshop and the EGI Community Forum.

Happy new year!

2012-01-05T11:31:00.001Z

I hope you all had a good Christmas and New Year break.

It's back to work and planning for the future here at the NGS with several future events on my to do list.

At the end of last year I finished off the last of our user case studies which highlight how our users have used NGS resources and the advantages it has brought them. The full set of case studies numbers 29 with the latest arrivals listed below -

Using the NGS to run a computer tournament on social learning strategies - Luke Rendell, University of St Andrews

Accelerating the Processing of Large Corpora: Using Grid Computing Technologies for Lemmatizing 176 Million Words Arabic Internet Corpus - Majdi Sawalha, University of Leeds

Computer Simulations of Biological Molecules at the Atomic Level - Sarah Harris, University of Leeds

The next set of case studies are at the planning stage but these will take a slightly different direction. The next set of case studies will look at how the NGS is working with large national and international projects to fulfill their objectives.

Also on the horizon is Easter conference season with several events coming up including the EGI Community Forum which will take place in Munich in March. Several NGS staff have submitted abstracts to this event highlighting work we have carried out in various areas including champion networks, authorisation and authentication.

The week before Munich is the Software Sustainability Institute Collaboration Workshop which you may remember from previous years. This year the event will be held in Oxford and the NGS is involved in several ways including holding a session for our SeIUCCR Community Champions. Watch this space for more information!

Apart from attending other organisation's events I have one of our own to organise. Following the success of last years SeIUCCR e-infrastructure summer school, we will be holding another summer school this year. After the deluge of applications we had last year, I am planning to advertise earlier this year to give a little more time to go through all the applications! Again keep an eye on the NGS website and our mailing list for more information.

The Training Marketplace

2011-12-20T14:25:00.002Z

Claire Devereux from the NGS introduces the EGI Training Marketplace.

The Training Marketplace is a service that allows you to search for or advertise training events and resources throughout the EGI community. These can be events or resources that may be open to absolutely anyone, just those within the EGI community, or they may be specific to a small project or to one country only. The Training Marketplace is the one-stop shop for your training needs as a user and is open and free for both academic and commercial providers to advertise in.

Since May the Training Marketplace has evolved from a simple event and material repository into a interactive site where requirements can be captured, events can be rated, and the tool can be customised and embedded into third-party websites using our gadget generator.

The following types of resources are currently supported:

traditional training events, usually classroom based or workshops where people attend in person. They can also include virtual events running at specific times. The difference between training events and online training is that training events have a set start and end time whereas online training is accessible either permanently or over a longer time scale.

You can choose to display events as a list, on a calendar or on an interactive map.

online training, available via the web. This category covers a wealth of resources, from self-study courses that require users to log in and complete exercises at their own pace leading to a qualification, through to online tutorials that users can tap into as they wish.
training resources, physical resources or services available to the community to assist in training. An example of a training resource is the GILDA Certification Authority. GILDA issues temporary (14 day) personal public key certificates (compliant with the X.509 standard) in order to access the GILDA Testbed for user training.
the requirements area is a place for users to describe their training needs and for training providers to see if there is interest in running courses. For example, a provider may post details of potential offerings and ask those interested to visit their website for further details, to check for viability before running a course.
University courses is the place for higher education institutions to advertise their Masters and Doctorate training opportunities.

After attending an event or taking part in online training users can rate the event and leave their feedback, letting others know the value of their experience much in the same way are people do with online shopping nowadays.

The latest release is a step towards improving the user's experience and looking towards longer term sustainability. It includes the new online training category, much improved search functionality, notifications to authors once entries are published and an improved calendar view. We are now working improve the functionality and appearance of the Training Marketplace gadget, which allows projects or NGIs to embed the EGI Training Marketplace into their own website, pick and choose which parts to include and even skin some elements with their project colours. If you are an NGI or project we would be interested in hearing about your requirements for the gadget, so please contact us.

CA stuff

2011-12-15T12:27:00.002Z

Three almost unrelated things, except that they relate to the CA(s), somewhat technical stuff, so bear with me:

If you have renewed your certificate recently and found that it didn't work with VOMS, this is due to a misdesigned "feature" of VOMRS that it insists on registering not just the user name but also the issuer name as part of the account. In the Real World(tm), we can keep the issuer (ie CA) name the same all the time but this does not work with grid middleware, so we have to change issuer name whenever we roll over. The "feature" adds no extra security for grid CAs because the user name will always be unique. There is a workaround that tells the server to ignore the feature, which we thought everyone had been using for years - but this, as Steve Traylen points out, will still cause problems if your certificate expires before you can resign the AUP (surely a rare case?!) - but should otherwise work fine. The best option otherwise seems to be to get your VOMS admin (not VO admin!) to duplicate the account entries, one with each issuer name, the old and the new one. CERN (Steve) has done this with theirs, and we are checking the other ones to see if they have failed to enable skipcacheck. I am both amazed and sorry that we have not discovered (and resolved) this sooner... but Steve is a Wizard(tm) and will fix it... Incidentally, it is not just us, other CAs roll over too - however, many have chosen to extend the lifetime of the existing certificate instead of renaming it - this will then not cause problems with the VOMRS accounts, but it causes problems with server/client synchronisation for HTTPS instead - if the server and client (browser) are not updated in sync (and they never are), some browsers will print obscure error messages and fail to connect (as we know from past experience).
Oh, and another good thing is that the IGTF rules have changed - we can now make the end entity issuing CAs have longer lifetime, so we no longer have to roll over every four years. Hooray!
On the subject of IGTF 1.43 release which came out recently. We're rebuilding the NGS-specific release except we are going back (or forward?) to individual RPMs instead of a single one for the lot - this means we need a couple of dependency RPMs but we should then be able to not mess with the IGTF stuff much and sites can in principle fine tune what they install. We'll have to think about the dependencies carefully.
Related to this (so, er, not entirely unrelated), there is this problem with the IGTF root signing policy file. We're trialing the Least Elegant Workaround(tm) this time, having discussed it at some length, by testing a self-signed version of the SLCS toplevel. This makes it independent of the root in the technical sense of building a verification path and checking signing policy files, so would slot in next to an unmodified IGTF release directly - but the downside is that we now have another self signed certificate that we'd need to establish trust in, and the fact that the policy of the SLCS branch (ie the SARoNGS CA, the CEDA CA, etc.) were supposed to be covered by the root CP. Depending on how good this looks (we're testing it from today), this may appear in 1.43.
Oh yes, and I know I need to get TACAR corrected and updated. This is not trivial (requires writing forms and pgp signatures) so is awaiting a slot where I have some time...

That was three things, for suitably large values of three. If you have any questions, do get in touch...

Return of the Champions

2011-12-14T12:32:00.001Z

I’ve recently taken over the organisation of the NGS Campus Champions programme here at the NGS and last week I chaired the first meetingof the group.

Our Campus Champions come from a number of universitiesacross the UK and are mainly representatives from IT Services. As for their role and activities on behalf ofthe NGS? Well this was one of the mainthings I wanted to discuss at the meeting!

Thankfully all our Campus Champions agreed with my list ofproposed benefits for them (and us) of being a Champion. In brief the NGS will provide training inusing NGS resources, tools and e-infrastructure; provide access to onlinetraining materials; produce publicity material to help them publicise the NGSand their role as Champion; hold Campus Champion events at relevant events;hold bi-monthly phone calls for dissemination of news and information.

In return the Campus Champions will actively promote the NGSwithin their institution offering advice and advising researchers; display NGSCampus Champions publicity material; pass onto the NGS requirements from theirinstitution and researchers; liaise between researchers, the institution andthe NGS; attend the bi-monthly Campus Champions meeting.

Currently we have 14 institutions with Campus Champions butwe are always looking for more! If youare interested in becoming a Champion then please contact me. Your site doesn’t have to be a member of theNGS for it to have a Champion. Thecurrent list of institutions is:

Canterbury Christchurch
University of Huddersfield
University of Hull
University of Liverpool
University of Manchester
Queen Mary - University of London
University of Oxford
University of Reading
STFC - Daresbury Laboratory
STFC - Rutherford Appleton Laboratory
University of Sheffield
University of Surrey
University of Sussex
University of York

The next meeting of the Campus Champions will take place byphone in February but in the meantime I shall be making some changes to theexisting Campus Champion pages on the NGS website. Watch this space!

Interact with the NGS

2011-11-28T10:53:00.001Z

I recently updated the NGS presence on Facebook to a new interactive page. Instead of the old-style group page, the NGS now has a new page under the UK NGI banner which reflects our role as the lead in the UK National Grid Initiative along with GridPP.

By “liking” our page you can receive news updates from the NGS all in one place as well as news updates from the Software Sustainability Institute (SSI) and GridPP. Please feel free to invite your friends and colleagues to "like" our page and to share articles with your friends list - help spread the word about e-infrastructure!

NGS at SC'11 - round up

2011-11-23T10:10:00.001Z

The UK presence was fairly significant this year at SC'11 with attendance by David Wallom, NGS Technical Director and a significant number of the leaders of research computing centres from around UK universities. This included NGS member sites @ Bristol, Leeds, Oxford, Southampton and EPCC.

In this blog post, David gives us a round up of NGS activities at this major computing event.

Having got off of the very long flight from Heathrow
to Seattle we settled onto the metro to get us downtown. After passable sleep the following morning we headed over to the SC’11 venue – the Washington Convention Centre to collect our badges and then visit the workshop on HPC in Smart Grid, where there was UK interest from the EC FP7 HiPerDNO project being presented by Dr Stef Salvini, OeRC. Following a very productive day where we learnt the state of the art in US Smart grids, how they intend to utilize knowledge developed through the national e-infrastructure for research. We then met up with the EGI team who had an exhibition stand at the conference.

The Monday workshop, Many-Task computing on Grids and Clouds 2011, started off with an interesting keynote from David Abramson (Monash) a long term friend of the NGS through support for their Nimrod tool which is popular with several of our biosciences users. After this there was a panel session which went slightly off topic to talk about exascale more than Many task but it still attracted several questions around the need for exascale, when we are still struggling to get a significant user base onto smaller HPC systems. Overall a good workshop though having the panel first did mean that a number of people didn’t hang around for the rest of the papers. This workshop was operating in a very competitive market with other sessions on cloud and data management which also attracted significant crowds.

The first full day of the conference allowed for the first good look around the exhibition floor alongside several interesting birds of a feather sessions There was also the first of a number of conversations with different groups and vendors, including Microsoft, Mathworks and Adaptive Computing. To give an idea of scale this picture is down one of the main aisles in one of the 5 rooms that were all about this size!!

Pretty impressive stands by a number of people
but the coolest was the multi projection globe on the NOAA stand.

We of course also announced
our activity with Globus Online which created a lot of interest and ended with us having a number of interesting conversations with NSF regarding future collaboration between our national e-infrastructures.

During the meeting the EGI booth was continually visited by a reasonably large number of people, we had the Real Time Monitor showing as normal (having seen a lot of 3D screens this needs to be done in 3d now for next year!). They did though give away a pretty large number of t-shirts as did a lot of stands, so I ended up as the moving poster board around down town Seattle from 6:30-7am every morning on my morning run!

Dinosaurs, DNA and nuclear power

2011-11-18T12:28:00.001Z

Just a normal month or so in the life of the NGS really butwhat do all 3 have in common?

They were all research areas investigated using NGSresources.

The research of William Sellers, Phil Manning and Karl Bateson dinosaur locomotion was featured as a “success story” on the EGIwebsite. They talked about how they usedGrid computing to understand better how dinosaurs moved around and what rolesthey played in their ancient world. Asthere are no similar animals around today to compare to dinosaurs such as a T. Rex,the solution is to create a detailed computer simulation of the animal’sskeleton and muscles.

Not only was their research picked up by EGI but it alsofeatured in iSGTW – fantastic publicity for the researchers and for the NGS!

I’ve been busy putting together some user case studies overthe last few weeks and I’m pleased to say that there are now a few more up onthe website showcasing the large spread of research areas that the NGS facilitates.

First up is Charlie Laughton from the University ofNottingham who has been using the NGS for quite some time now. He used the NGS to investigate the flexibility and folding properties of DNA as understanding how the tightly packed DNA inhuman cells can still be read can, in turn, help to understand how cells switchgenes on and off. At present there is no clear understanding of howthis works. Being able to influence thisin new ways may ultimately help to find new drugs to treat diseases such ascancer, develop new biofuels, and crops that can resist climate change.

Charlie said “…without the compute power and high-throughputprovided by the NGS, we would not have been able to deliver our part of theproject in a timely manner. At a more personal level, it led to one of the mosthighly cited publications I have ever had.”

John Allen from the University of Edinburgh explained howthey use NGS resources to power the GridQTL portal which is used worldwideto study gene expression in a wide range of organisms.

The team’s use of the NGS has greatly increased theproductivity of their users (currently around 400) in the QTL community. Oneexample of this is a GridQTL user at the University of Missouri Columbia. They ran a series of studies on carcass, post-natal growth and reproductivetraits in commercial Angus cattle and found a speed up of from 20 people-weeks,using their old single server system, to 3 people-weeks to capture and analysethe data with GridQTL.

Finally we have nuclear power! Paul Martin from the University ofHuddersfield has been using the NGS to investigate the suitability of Thoria asan alternative form of nuclear fuel. Paul’sresearch is particularly timely as there is increased interest in the use ofthorium dioxide for nuclear power rods not least because of its comparativelyhigh abundance in the earth’s crust and low cost. It is for this reasonthat, although the main fuel for nuclear power reactors is currentlyurania-based, thoria-based fuel is attracting much attention as an alternativehigh performance nuclear fuel.

All of our case studies can be found on the NGS website andwe now have a collection of 26 covering a wide range of research areas. If you are interested in using the NGS casestudies to promote grid resources then please let us know.

A tangled web we weave

2011-11-10T12:00:00.003Z

In the September 2011 edition of NGS News, we published an article by our Technical Director David Wallom which highlighted the networks of champions that exist within the NGS. To compliment this I also received and published an article on our website from Simon Hettrick from the Software Sustainability Institute about their network of champions which they have thankfully called Agents. Too many champions spoil the broth and all that...

It got me thinking about how these networks all fit together like a spiders web as some people are members of more than one champions network and some institutions have more than one person involved. A tangled web indeed!

This has probably come to the forefront of my mind as, due to some staff changes at the NGS, I'll be a lot more involved in the organisation of the Campus and Community Champions here at the NGS.

So what are these Champions and who are they?

The Campus Champions as suggested by the name, promote the NGS and the services we provide on their university campus or in their institution. They tend to be people involved in research computing or ITS but we also have researchers involved. All NGS member sites are expected to nominate a Campus Champion but we welcome Campus Champions from any UK university or institution. Your site doesn't have to be a member to have a Campus Champion! If you are interested in promoting the NGS at your institution or university (with help and support from the NGS) then please get in touch!

The Community Champions are funded by the EPSRC funded SeIUCCR project and are researchers who actively use e-infrastructure in their research. They promote to their research community, peers and colleagues from all institutions and universities. We are looking for more Community Champions from all and every research area so if you are interested in promoting your research and your use of the NGS then please get in touch!

There will be some slight changes to our Champions networks and hopefully you will see a lot more activity from these already active groups. I want to highlight their activities more and demonstrate their contribution to the NGS and e-infrastructure as a whole. I'd also like to link them more closely with the SSI Agents.

Lots of plans are afoot so watch this space!

A smattering of updates

2011-10-27T11:43:00.000+01:00

Over conference season there was what can only be described as a smattering of small announcements that you may have missed during your travels. The blog seems like an ideal place to gather these together!

NGS News
The latest edition of our quarterly newsletter was released in time for conference season. Available to download from our website, September’s edition contains articles on -

A round up of news from Europe including the release of the first Unified Middleware Distribution
An introduction to the new Certificate Wizard
Championing e-Research and e-infrastructure - the Campus and Community Champions
NGS user case study - Using the NGS to model the climate impact of aircraft emissions
...and more!

Tell us what you think!
Also on the website we have a new poll on the home page. This time we are asking people how easy it is to find the information they are looking for on the NGS website. So no matter if you are a frequent or occasional visitor to the NGS website, let us know by voting in our poll. It can be found on the right hand side of the homepage.

Busy users
There is a new NGS user case study on the website. Maria Holstensson from the Institute of Cancer Research explains how she is using the NGS to optimise cancer treatment for children suffering neuroblastoma.

Children with neuroblastoma who are being treated with targeted radionuclide therapy can have their treatment monitored with gamma camera images. These images are used to calculate the amount of drug taken up by the tumour and to estimate the radiation dose. However the image quality can be poor due to scattering and interference. Maria Holstensson from the Institute of Cancer Research is looking at tackling these problems.

Maria said "We have had absolutely fantastic help from the NGS and as a result of using the Grid we have been able to run multiple parallel simulations that we would not have been able to run otherwise".

How many?!
We’re pleased to announce that we now have over 1000 subscribers our fortnightly NGS news bulletin. We have subscribers from all over the UK and much further afield with 21 countries represented amongst our subscribers.
The news bulletin is delivered to your inbox every second Friday (with some exceptions during conference and holiday season!) and contains news from the NGS, updates from our member sites, details of forthcoming relevant conferences, calls for papers for relevant journals and much more. You can subscribe to the mailing list from the JISCmail site.

Lounging in Lyon and yawning in York

2011-10-20T13:50:00.004+01:00

Oh I wish!

The September conference season was hectic as always for the NGS team with the EGI Technical Forum (Lyon) and the UK e-Science All Hands Meeting (York) back to back. There was definitely no time for lounging in the Lyon sunshine although we may have yawned on the train home from York as the conference travel came to an end.

The EGI meeting in Lyon was a great success as always with over 600 attendees. The meeting is an excellent opportunity for the NGS, in it’s role as the UK National Grid Initiative (NGI), to meet up with other NGI’s from all over Europe. As well as looking after and organising the UK NGI exhibition stand in conjunction with GridPP, I was also involved in a session in my role as NGS Liaison Officer.

The NGS has held successful roadshow events for several years and these have caught the eye of EGI who are looking at doing something similar through the NGI’s. I was asked to take part in the EGI / NGI roadshow session and present on my experiences of organising and holding roadshows and measuring the feedback and impact of these events.

There was an interesting discussion after the presentations regarding what the NGI’s would need to host these events and what materials they would find useful. In some cases more staff and more time would be very helpful but the ability for the EGI to provide these resources are somewhat limited! I was also asked about practical points such as organising registration and finding the right people in institutions to help host the events. Hopefully the discussion minutes will be made available at some point. Some more discussion points were captured on the GridCast blog entry on the session.

The AHM meeting at York attracted 150 people this year and seemed dominated by one word – Cloud! I lost count of the number of cloud sessions taking place over the 4 days. The meeting was stimulating and thought provoking judging by the copious amount of notes I took. However one of the main activities for the NGS was the SeIUCCR organised workshop – Meet the Champions.

The purpose of this workshop was to give attendees an opportunity to meet the researchers that have been promoting and championing research in different e-Science areas and to find out about their work and how they utilise e-infrastructure. The guest speaker was Scott Lathrop from the XSEDE project where he is the Director for Education, Outreach and Training. Scott talked about their Campus Champion programme and how they ensure that Campus Champions feel involved and part of the project. They have many of the same responsibilities as our Campus Champions including raising awareness of XSEDE and even providing training in using the resources.

Two of our Community Champions also presented at this event outlining the issues in their research areas regarding e-infrastructure, getting started, having the right support at their institution etc. There was also a very lively discussion panel at the end of the session with many issues raised including how scientific researchers work and the risks involved in devoting time to starting to use new technologies.

It’s great to hear researchers and users speaking about their experiences of actually using this technology, as it sometimes seems, that users get forgotten about in all the discussion about standards and programming. We need a reminder that at the end of the day this is about building tools and providing a service that people will want to use and find beneficial and that will help further their research. It’s most definitely not a case of “build it and they shall come”!

I’ll make sure that all the presentations from the AHM workshop are available on the NGS website soon so watch this space!

A long long time ago... or so it seems

2011-10-13T14:06:00.000+01:00

Before the mayhem of the September conference season was upon us, the NGS ran a successful summer school at the beginning of the month. The e-infrastructure summer school was part of the SeIUCCR project. You can read more about the background to the summer school and project in my blog post from the beginning of August.

We had over 25 students join us down in Coesner's House in Abingdon for 4 days and they came from a wide variety of backgrounds. We had students who were in their first year or two of their PhD as well as post docs and they came from Edinburgh to Essex and everywhere inbetween.

So how was the summer school? Was it a success? Did the students learn and enjoy it?

Well the best people to ask are the students themselves. All the students were asked to provide us with feedback and some were even willing to write a few more words.

Ed Day from Canterbury Christ Church University attended the summer school and this is what he had to say:

"The recent SeIUCCR Summer School was a very enjoyable andinformative experience. As a newcomer to grid research I found the the summerschool extemely useful. It contained some important introductory pieces as wellas covering many topics in more depth suitable for anyone wishing to use theNGS. Sessions consisted of a good mix of high level overviews and hands-onpracticals such as using the P-GRADE portal.

The presenters were very knowledgable and helpful andwere eager to inform on all aspects of grid computing. Some sessions involvedthe speakers talking individually with attendees and I found the staff gaveuseful advice and were very supportive of my project.

Before the school I thought my particular research area,the forensic investigation of mobile phones, might be a good fit for gridcomputing, and by the end of the summer school I felt much more able to pursuemy research in an informed manner using the NGS. It was useful to know how grid,cloud and high performance computing relate to one another and over the four days I feel I became muchmore aware, in a less naive way, of how my research would benefit from theresources the NGS has to offer. In particular I learnt to think about myresearch differently: how my phone investigation process would be best able tobenefit from a grid architecture.

Indeed the school helped support my view that the NGSwould be a good resource for ANY project that needs HPC not just vast numbercrunching modelling applications such as those such by molecular biologists orquantum level physicists (although of course it is good for that too).

I liked the food too!"

And it wasn't just Ed who enjoyed the summer school!

"The summer school has been a fascinating activity.The hands on sessions have familiarized us with HPC/Grid/Cloud, which areuseful resourses that I have never had access to or known how to access, whilstthe Meeting Champions and Q&A sessions made it possible for us to know howthese resourses could facilitate our research. The SeiUCCR summer school providesa great opportunity of learning, communicating and networking. I would like tothank all the people who made this summer school possible"

"The summer school was a relaxed and friendly environment. It provided detailed information about the various resources available to researchers both in terms of computer resources and support. The staff where very approachable and keen to show an interest in the attendees work. Overall it was a great experience I would recommend to anyone who has an interest in grid/cloud/HPC or who's work may benefit from such technologies."

For anyone who was unsuccessful in obtaining a place at the e-infrastructure summer school, the presentations are now available online. Due to receiving over 120 applications for the summer school and only having a limited number of places, I know many people were keen to see the presentations.

There will be another summer school in 2012 so watch this space for forthcoming announcements next year!

Blogging off

2011-09-29T11:06:00.000+01:00

It is the last day of All Hands 2011 and it is my last day working for the NGS.

After 4 years of general griddery, I'm moving on.

Four years is a long time in research, and today's All Hands meeting at York is very different from the first grid event I attended, Open Grid Forum 20 in Manchester.

I remember that the Manchester meeting was huge and full of international delegates.

The UK contingent were based in something called the UK e-Science Village - which conjured up bucollic images of computer scientists dancing around the maypole on the e-Science village green, just next to the local shop for local people.

At the very least, I was hoping to see the UK e-Science Village People giving a rousing chorus of their classic - `(its fun to be at the) STFC.'

The village turned out to be a very large display booth.

All Hands is national, rather than international. The conference and the booths are smaller. As at OGF, people still enthuse about shiny new technology that will solve all our problems in the future.

But in among them are people using the less-shiny, less-all-singing, less-all-dancing software that we have now. And they are using it to do new research that is nothing to do with the technology itself.

And it is those are the people I want to hear - because what I have learned to call e-Infrastructure is very broad - in one session yesterday, the talks covered the behaviour of the heart, and how what the researchers have learned there has been applied to the way muscles move when giving birth; and how to model the way water shapes landscapes over millennia.

I still do not give a damn about how clever, or web-service-y, or standards compliant, a bit of e-Infrstructure is. It is what you do with it that counts.

It is the researchers who have take what we provide and use it to deliver the research that could not otherwise be done. These are the people you can read about in the case studies.

These are the people who have turned e-Research into Research - and will continue to do so for many years to come.

Goodbye UKI, hello NGI_UK

2011-09-27T15:08:00.003+01:00

At All Hands 2011, in the atrium of the University of York's brand new Ron Cooke Hub conference venue.

On our stand in the middle of the room is a familiar face - helpdesk manager John Kewley - sitting under a slightly less familiar sign.

It doesn't say NGS, or GridPP, although both have posters on display.

The sign says but 'NGI' - aka National Grid Infrastructure - and we have had to to get used to it very quickly.

At last week's EGI technical forum, what was the UKI ROC - or the UK and Ireland Regional Operation Centre - was offically replaced by two new NGIs called NGI_UK and NGI_IE.

And lots of things broke - including the load monitor and the Nagios testing service.

Names matter. Both the load monitor and Nagios were pulling information about sites and users from the Grid Operations Centre Database. More specifically, they will pulling information about sites and users associated with the UKI ROC.

The UKI ROC is no more: it has no sites or users associated with it.

So... we have spent the last few days tracking down every reference to the 'UKI' in every configuration file for every service and replacing them with NGI_UK.

There were quite a few....

The load monitor is back. We've been working on Nagios today and it should be fully working soon.

Three Little Words

2011-09-19T16:52:00.001+01:00

There are these three little words. For some people, these words bring feelings of fulfilment and contentment. For others, they bring nothing but frustration.

Those three little words are:

Proof of Concept

For that part of the e-Research Community interested in how research will be done in future, A proof of concept is evidence that it is possible to do something new and interesting, using something new and interesting. It might change the way research is done next decade. It is more than enough for a published paper and a presentation at All Hands.

And it is of bog-all use to those for whom e-Research is simply a means to an end. They just want something that works now and works reliably.

There is always a gap between the potentially useful and the actually useful. When you can build something that bridges that gap, you can enable research that would not otherwise be done.

Which brings me to slightly embarrassing news that our project to deploy the ARC middleware in front of the local High Performance Computing service has been a complete success... as a proof of concept.

We have shown that it is possible to deploy ARC services in front of what we should now be calling Oracle Grid Engine.

With some inventive use of ssh copies in prolog and epilog scripts --- that this can be made to work even where there is no file-space shared between the grid 'front end' and the HPC cluster.

We also know that you can support parallel tasks using ARCs Runtime Environment mechanism --- there are examples at the bottom of the (slightly out of date) Nordugrid documentation --- and make use of to the LCAS/LCMAPS authentication system used by other grid software.

Which is nice....

Whether it is going to be useful is a completely different question. We do not yet know if the local communities who are best placed to use it --- the rather incongruous pairing of Solar Physics and Social Science --- will want to do so.

Epilogue: Prologs and Epilogs

A quick technical note on faking a shared directory via Grid Engine prolog and epilog scripts.

The scripts run just before the start and just after the end of every job.

ARC-the-middleware obligingly changes directory to the 'shared' scratch directory before submitting the job. This mean that prolog and epilog scripts are presented with the path to this directory in the $SGE_O_WORKDIR environment variable.

The recipe is along the lines of...

Create a ssh keypair for each user - to be used solely for transfers from HPC backend to grid front end
Copy the private key to a safe place on the HPC back end, readable only by the user. We will call this $GRID_KEYS.
Use the public key to create a per-user authorized_key file on the grid front end in somewhere like
/etc/ssh/authorized_keys.d/$USER
and change the /etc/ssh/sshd_config (again on the grid-front-end) to set.
AuthorizedKeysFile /etc/ssh/authorized_keys.d/%u
Add code to prolog and epilog to use scp (or rdist) with the -i $GRID_KEYS/$USER to pull files from $SGE_O_WORKDIR at the beginning of the job and push them back at the end.

Conference season approaches...

2011-09-13T12:52:00.002+01:00

It's gone slightly quieter for me now that the SeIUCCR e-infrastructure summer school is safely under way. 30 students are now ensconced in Coesner’s House in Abingdon where they are learning about the wonders of e-infrastructure and how it can help their research. As I type they will have just finished a “hands on” session on the NGS and how to run jobs on our resources.

One event is underway but we still have two to go. Next week sees many of the NGS staff at the EGI Technical Forum in Lyon. The NGS in conjunction with GridPP is the UK National Grid Infrastructure (UK NGI) and in turn the UK NGI is part of EGI (European Grid Infrastructure).

It’s a very active meeting for many NGS staff due to the level of involvement we have in this major project. As well as meetings, there will also be presentations in several sessions from NGS staff. I’ve been asked to give a presentation on the NGS roadshows as EGI are developing their own roadshows – well they say that imitation is the greatest form of flattery! As always the UK NGI will have a stand at the event where people can talk to us further about our activities, meet staff and obtain information. If you are attending the EGI Technical Forum then drop by and see us.

The week following the EGI conference, many of us will be in York for the UK e-Science All Hands Meeting. Registration for this is open until the 19th of September so if you want to go, make sure you register soon! Again the NGS will have an exhibition stand along with GridPP at the event. The exhibition stand will be a hive of activity as there will be several demos taking place here. The demos are –

Applying for UK e-Science Certificates using the new CA Certificate tool
Taverna Server: Towards enabling long running workflows on the NGS

Some of our users will be actively taking part in the conference with demos and presentations not to mention NGS staff giving presentations and posters as well.

A major activity at AHM is a workshop organised by SeIUCCR which is a collaboration between the NGS and the Software Sustainability Institute (SSI). The workshop is entitled "Meet the Champions" and will take place on the Tuesday 13:30-16:30.

The workshop is an opportunity to meet researchers that have been promoting and leading research over the past decade of e-Science; find out about their work and how they utilise e-Infrastructure, and learn how you can interact with them. Specifically the "Champions" to meet are members of the Community Champions network from the SeIUCCR (Supporting e-Infrastructure Uptake through Community Champions) project; the NGS Campus Champions and the Software Sustainability Institute Agents Network.

There will also be 2 key presentations -

Scott Lathrop is Blue Waters Technical Program Manager for Education and TeraGrid Area Director for Education, Outreach and Training. Scott's talk is entitled "Engaging Campuses in XSEDE". XSEDE is the successor to TeraGrid. Scott will be talking about the XSEDE Campus Champions programme and also the Campus Bridging programme for XSEDE.
Steve Brewer is Chief Community Officer of EGI, the European Grid Infrastructure and he will be talking about Community Engagement in Europe.

And if you thought that was enough there will also be a panel session on the question "Why should researchers use e-Infrastructure?".

For the full details of all the NGS activities at the forthcoming AHM meeting please see the news article on the NGS website.

Hopefully we'll see some of you at some point over the next two weeks!

It is not easy being parallel

2011-09-07T14:30:00.001+01:00

As has been said before - there are differences between Grid and traditional High Performance Computing. Some of the differences are due less to the technology and more to the problems being solved.

The more successful grid users are task farmers: they scatter comparatively small compute tasks and data and wait for them to grow into results. The grid - metaphorically speaking - is there to plough the land, spread the fertilizer and muck out the system administrators.

Traditional HPC concerns itself with big applications and - in particular - applications that are too big to fit on a single computer. HPC systems are built with parallel computing in mind.

The Grid does not do parallel computing well.

Consider the two steps in running any parallel tasks

Asking for more than one CPU core on the same system.
Setting those CPU codes to work

For each step, there is definitely more than one way to do it...

Take 4...

So, there you are, sitting by your favourite grid client, a freshly minted X509 proxy ready. All you need to answer one of the great problems of modern science is 4 CPUs.

All you need to do is ask.

How you ask depends on who you are asking and what grid dialect they understand.

Globus GRAM5 and ARC accept tasks defined in Globus Resource Specification Language (RSL), possibly with some Nordic extensions. In RSL, you can ask for more than one CPU with an additional:

(count=4)

The web-service-y Globus job submission systems (WS-GRAM) used a similar approach but written as XML.

In Job Description Language, as understood by the gLite CREAM-CE and WMS, you need

CPUNumber=4;

And in the OpenGridForum-approved XML-based Standard Job Specification Description Language, you have the instantly-memorable and easily-readable:

<jdsl:TotalCPUCount>

<jdsl:Exact>4.0</jdsl:Exact>

</jdsl:TotalCPUCount>

(which you will find buried somewhere under 3 levels of XML tags).

Yes - I know JSDL isn't really there for humans to read, but it doesn't stop some humans trying.

4 go to work...

That was the easy part.

Now it gets complicated.

And, on this occasion, you can't blame the Grid for the complexity.

Large-scale parallel programs are typically written around libraries implementing the Message Passing Interface (MPI). There is more than one version of the MPI standard and more than one library implementing them.

To add to the confusion, from some MPI variants, you need to build versions for each FORTRAN compiler installed.

Launching a parallel job depends on both the job management software and the underlying mechanisms used for communication. MPI installations typically provide either an mpirun or mpiexec command that ensures that the right processes are started in the right way on the right computers.

It is very likely that each version or each MPI implemention will have its own variant of mpirun or mpiexec. It is equally likely that - at least for mpirun - they will expect different arguments.

In the first and second phases of the NGS, we were funded to provide exemplar Grid clusters at RAL, Oxford, Leeds and Manchester. The grid software we deployed - Pre-WS GRAM from Globus 4 - could launch MPI jobs if

(jobtype="mpi")

was included in the RSL.

It could only launch one of the many possible mpirun commands. To work around this, devious system administrators cooked up a sort of super-mpirun that would locate the correct version for an applications.

Researcher in Ireland found ways of launching MPI jobs from within JDL jobs - but they could not hide all the complexity.

ARC supports parallel jobs via its Runtime Environments extension - which can tune the environment for an application so that the right number of CPUs are assigned and the right mpirun is run. Again, this needs the system administrator to do something devious if it is to work.

We haven't even begin to cover parallel programs written outside MPI - such as those using the Java sort-of-MPI library MPJ-Express.

So... what am I trying to say?

It would be nice to have a conclusion, or at least a lame joke, to end this blog post - but I can't think of one.

All I can say is that parallel computing is complicated, distributed computing is complicated and that any attempt to combine the two - either using existing Grid solutions, or something newer, shinier and probably invoking the word Cloud - cannot make either kind of complicated vanish completely.

Where did August go?

2011-09-01T13:43:00.005+01:00

Usually August is a quiet month outreach wise at the NGS but this year it seems to have been the complete opposite. The SeIUCCR summer school is fast approaching and I've been busy sorting out registrations, accommodation and queries for that. The summer school was massively oversubscribed with 4 people applying for each place. Demonstrates that there is quite a demand out there for training in e-infrastructure across institutions in the UK! Thankfully all speakers and delegates appear to be sorted so perhaps now I can breathe a sigh of relief. If you were not able to get a place at the summer school keep an eye on the NGS website and mailing list as the material from the course will be made available online.

In other news there is a new NGS user case study up on our website. This time Maria Holstensson from the Institute of Cancer explains how she is using NGS resources to improve cancer treatment for children.

Children with neuroblastoma who are being treated with targeted radionuclide therapy can have their treatment monitored with gamma camera images. These images are used to calculate the amount of drug taken up by the tumour and to estimate the radiation dose. However the image quality can be poor due to scattering and interference. Maria Holstensson from the Institute of Cancer is looking at tackling this problem.

It never fails to amaze me the range of research carried out on NGS resources. We have users from every area from linguistic analysis to high energy physics and we are always looking for more. There are now 23 user case studies on the NGS website and I hope that they demonstrate that e-infrastructure is for everyone and not just those from physics or computing research areas.

This may be a good time to mention an interesting blog post from Steve Crouch over on the Ask Steve blog from the Software Sustainability Institute (SSI). He's been musing on communications between developers and researchers - do they really speak the same language? Comments will no doubt be welcome!

Sign Here

2011-08-26T21:59:00.011+01:00

I would not want to describe the paperwork that goes with University life as something out of
Terry Gilliam's Brazil or Yes Minister.

It would be ill-advised: I haven't completed this month's NGS/B/11347/2(a) (permission to use ironic over-exaggeration within a blog) and submitted it to the appropriate authorities.

Depending on how deeply your institution loves its paperwork, there will be forms to be complete when claiming travel costs, or buying a new HPC system, or obtaining a replacement biro. Inevitably, somebody else needs to sign these forms to show that there has been due diligence and that the trip to Didcot, million pound compute cluster or cheap plastic pen are fully justified.

Somebody else isn't just anybody. When you present your form to the powers-that-be, the powers-that-be will carefully compare the signature with their collection of scribbles from the great-and-the-good.

Only when you have the right name in the right place on the right form will you will receive a new pen and a firm lecture about being more careful in future.

As we have said on a number of occasions, grid security is built on chains of trust. It also relies on the right signature being used in the right place. In our case, these are digital signatures represented by X.509 certificates rather than the spiders-web-on-acid scrawl of a senior University manager.

A certificate in your local list of trusted certificates - typically in /etc/grid-security/certificates - can be accompanied by a file defining its signing-policy. You can see some examples of signing policy files in the UK eScience Certification Authority pages on the website.

The signing policy is particularly influential at the very far end of the chain of trust: the root certificates. The private key associated with root certificates are kept in a Very Safe Place and are taken out only to sign the certificates of Certification Authorities (CAs).

CAs sign the certificates for the rest of us. The UK has two CA's - the main eScience CA and a SARoNGS CA.

Over the last weeks, thanks to the efforts of the dragon-slayers at the Software Sustainability Institute, we finally found out why certificates from our 'SARoNGS' CA were being rejected by the NGS's Workload Management Service.

There was nothing wrong with the certificates themselves.

The SSI developers quickly identify problems with the SARoNGS Certificate Revocation List (CRL) - a list of known-bad certificates that CA's should distribute.

SARoNGS certificates are designed to be short-lived - they expire before anyone gets a chance to do something bad with them - and the revocation list is empty. But all revocation lists - even empty ones - have expiry dates and ours had, unfortunately, gone stale.

Updating the CRL was comparatively easy but it did not solve the problem. The root cause of turned out to be the root certificate's signing policy.

The problem is that there are two signing policies - depending on whether you consider SARoNGS certificates acceptable.

SARoNGS certificates can be obtained using only a UK academic username and password whereas a full eScience certificates requires photo ID and a visit to your local Registration Authority.

The International Grid Trust Federation (IGTF) is responsible for ensuring that certificates are being created and managed in a trust-worthy way. It has strict rules on what constitutes sufficient proof of a users identity and - not to put too fine a point on it - an academic username and password are simply not good enough.

So the signing policy within the IGTF's bundle of UK eScience certificate information does not currently match the version we distribute. The IGFT version will not permit the eScience root to sign for the SARoNGS CA.

The root cause was a misplaced update that installed the IGTF version of the eScience root signing-policy - rather than the NGS's own.

We should have had the 'IGTF+' certificates - a modified version of the IGTF's certificate collection maintained by the NGS blogs' very own Jens Jensen, and incorporating the NGS's signing policy and some additional certificates.

The IGTF+ certificates are available in a number of formats from Jens's avowedly Web-1.0 certificate repository webpage.

[With thanks to James Perry, Steve Crouch and Rob Baxter of the Software Sustainability Institute]

What do you do on the NGS?

2011-08-22T12:44:00.003+01:00

It's been a busy month or so even though it's the holidays as this is the ideal time for me to contact many of the NGS users who promised to write user case studies for me regarding their research.

We have a wide range of reseachers who use the NGS as part of their every day research and it's important for the NGS to highlight that our resources aren't just used by the "typical suspects" such as physicists.

Over the last few weeks I've added another 2 user case studies to the NGS website -

Modelling the effect of the peptide sequence on the binding affinity to carbon nanotubes from Susana De Tomasio, University of Warwick
Systematic modeling of ionic liquids from Edward Ballard, University of Wales Bangor

Both these case studies demonstrate how the use of NGS resources is helping to speed up research enabling results to be produced and published faster than previously.

Edwards supervisor, Dr Anna Croft praised the NGS, "The NGS has been an excellent resource for many of our research projects. In particular, I have been able to use it with undergraduate researchers and give them a taste of what it is like to work on large computing infrastructures - an experience that has helped some of them secure PhD funding, both here and overseas, to continue in the computational area. When we had teething problems, the support staff were always friendly, helpful and got things working. Because of this support and the flexibility in requesting computing time, the NGS is one of our first ports of call for projects requiring a larger computing resource."

Thank you Anna!

A good-enough impression

2011-08-18T01:42:00.000+01:00

Leeds - as a long standing NGS partner site - want to hook our HPC service into the Grid.

We hope to fill the gap left when the last of our NGS-funded clusters was turned off back in April. Our main requirement was that the grid front end should be completely separate from the HPC service. In addition, we wanted...

... to use software provided by the EMI-1 software release from the European Middleware Initiative.
... to publish information that can be read into the NGS's central information service.
.. to accept request from the NGS's workload management system.

We had originally hoped to follow the particle physicists and deploy CREAM.
Unfortunately, EMI-1 was missing the components needed to make CREAM work with the SGE batch system used locally. The only software within EMI-1 that was SGE-friendly was Nordugrid's Advanced Resource Connector - ARC.

After a few months of work and in the great tradition of the grid: it is sort-of-kind-of-working-after-a-fashion. At the moment:

ARC's compute service - A-REX - is accepting jobs: for a very limited set of users and not from the workload management system.
ARC's information provider - ARIS - is publishing information about the system and this information is making its way to the NGS's BDII.

I'll will cover A-REX in a future post. This week, you are getting information about the information provider - and in particular, how it links into the NGS.

A bit of background. The NGS information service is a Berkeley Database Information Index service or BDII. BDIIs are built to collate information, some of which comes from other BDIIs. The NGS's central BDII, for example, collates information published by a BDII, or something that looks like a BDII, at each of the partner sites.

ARIS can do a impression of a BDII. Whether it is a convincing impression depends on what it is talking to.

ARIS produces information in its own Nordic-accented schema, designed to feed the ARC tools. This needs to be translated into GLUE format before a BDII will give it a second glance.

Based on documentation on linking ARC and EGI from Nordugrid,, this can all be done via a single ARC configuration file called /etc/arc.conf. arc.conf consists of blocks, denoted by a [name in square brackets] each containing a set of name=value definitions.

arc.conf needs to be tweaked in three places.

Turn on publishing of Glue 1.2 format information - which is close enough to the current common Glue version 1.3 - by adding to the '[infosys]' block.

 [infosys]
  ...
 infosys_compat=disable
 infosys_nordugrid=enable
 infosys_glue12=enable

Add in anything that Glue needs and ARIS does not via the '[infosys/glue12]' block:

 [infosys/glue12]
 glue_site_unique_id="NGS-LEEDS"
 ...
 provide_glue_site_info=true

And finally arrange for ARIS to collect its own output and present it as if it were a site BDII by a block called

 [infosys/site/NGS-LEEDS]
 unique_id=NGS-LEEDS
 url=ldap://ngs.arc1.leeds.ac.uk:2135/mds-vo-name=resource,o=grid

Our initial experiments suggest that the information produced by ARIS is good-enough to be accepted the NGS's central BDII but not good enough to fool our Nagios monitoring.

WLCG Nagios includes a number of BDII specific tests including one called org.bdii.Entries. org.bdii.Entries only looks for 'services' - or more accurately objects of the 'GlueService' type. While ARIS generates a lot of information, none of describes a GlueService.

What we don't yet know if it Nagios is being picky, or whether the existence of a GlueService is vital for some bit of grid wizardry.

E's no good - Distinguishing between distinguished names

2011-08-10T22:11:00.005+01:00

This is a sort of companion to Jens's 'On email address in host certificates' post on Monday.

For most people, changes to the policies and standards that describe how the grid should work are met with a resounding 'so what'. For anyone involved in the day-to-day management of grid systems, it is an opportunity to join a collective sign-of-relief.

It is another example of where the 'political' aspects of international research collide with the technical solutions and the needs of researchers who don't give a damn how it works, as long as it lets them do their jobs.

X.509 certificates are complicated because what they represent is complicated - a link in a chain of trust between particular individuals or institutions.

Identities within certificates are tied to Distinguished Names or DNs. A DN is a lists of attributes - such as country, institution and personal name - that uniquely identify a single person, or computer, or service.

The way a DN is stored within a certificate is well-defined but completely incomprehensible to anything that is not a computer program. For many practical purposes, the DN needs to be presented so it can be understood by a person.

A glance at the OpenSSL X509_NAME_print_ex documentation shows how brain-twistingly complicated it can be translating a DN into something that a human being can read.

There is a more detailed explanation on the NGS Wiki. This is the quick tour..

Each individual attribute within a DN has a 'type' and a 'value'.

The type identifies what is being represented - a name, or an email address. It isn't really a name but but a unique sequence of numbers called an Object Identifier. Something like: 1,2,840,113549,1,9,1.

People, inexplicably, find sequences like 1,2,840,113549,1,9,1 hard to remember so for our benefit, 1,2,840,113549,1,9,1 is also known as "Email", "emailAddress" and - occasionally - "E".

The value is depends on the type. For 1,2,840,113549,1,9,1 - it is a string of letters represented in what is known as UTF-8. UTF-8 was developed to represent any letter from any language - but most Grid certification authorities deliberately restrict themselves to the 26 letters of the English alphabet, the numbers 0 to 9 and a few symbols. If they didn't, things would rapidly become even more complicated.

In human-friendly form, the DNs that Jens is working to abolish look very much like

 /C=UK/O=eScience/OU=Manchester/L=MC/CN=voms.ngs.ac.uk/Email=support@grid-support.ac.uk

or maybe

 /C=UK/O=eScience/OU=Manchester/L=MC/CN=voms.ngs.ac.uk/emailAddress=support@grid-support.ac.uk

or even, very rarely

 /C=UK/O=eScience/OU=Manchester/L=MC/CN=voms.ngs.ac.uk/E=support@grid-support.ac.uk

Which variant you get depends on which version of which software is processing the certificate.

The problems appear when DNs are compared as strings of letters rather than in what could be called their 'raw' form.

Most software is smart enough to canonicalise these awkward examples by chosing One True Name for 1,2,840,113549,1,9,1 and substituting this before comparing. Not all software packages agree on which name is the One True Name.

It is now common practice to represent certificate chains in .LSC format - which are simply lists of human-friendly DNs. These may be simple to distribute and do not need to be updated every time the certificate is renewed.

The would be good enough - if it wasn't for that troublesome email address.

On email address in host certificates

2011-08-08T12:05:00.003+01:00

Every so often we get questions about email addresses in the names (distinguished names, ie DNs) of host certificates. The problem is that they are deprecated (see the last two paragraphs of section 4.1.2.6 of RFC5280), and they cause all sorts of problems with software which stringifies the DNs because there is no consistent way of doing it (or rather, there are too many consistent ways.) Arguably the software is not coded correctly, but in this case it'd be better to remove the email.

The email is there for historical reasons: when we rekey a certificate we have to give it the same name as before, so that's why it is still there. Dating back ten years or so, the original raison d'être was that before robot certificates, hosts would sometimes run stuff on behalf of users, ie. act as a client, and the email address was meant to give you something to contact when you read the DN in the log file.

The new policy will permit removing the email address from DNs. That's the easy bit.

The trick is to get the software to optionally (at the owner's request) remove the email address from the DN (because some people may genuinely want to keep it, for whatever reason.) Or rather, optionally keep it. The software cannot do this yet.

In fact, it'd be easier to just remove it for all host certificates, or maybe to handle those "manually" who still want to keep it, as with robots for example. If anyone out there has host certificates and depends on email being present in the DN, could you let us know via the usual channels, please? There are no known problems with removing the email address, only with keeping it, but there may of course be unknown problems - there are lots of weird and wonderful things out there.

As for timescale, it'll be ready at the latest when the new (rollover) CA certificates go live at the end of September.

E-infrastructure summer school - registration open now!

2011-08-04T11:28:00.003+01:00

A brief hiatus here on the NGS blog as several of us are / have been on holiday. Back to normal service now hopefully!

A lot of my time since I returned from holiday has been devoted not to the NGS but to another project I am currently working on. Catchily named SeIUCCR (pronounced "sucker"), the project was funded by the EPSRC "Crossing the Chasm" call which called for networks and "advocates" to promote the wider uptake of UK e-infrastructures by researchers in engineering and the physical sciences.

Part of the SeIUCCR project is an e-infrastructure summer school which is due to take place in Abdingdon near Oxford in September. The residential summer school will offer an introduction to e-infrastructure including Clouds and Grids to UK PhD students and postdocs over 4 days. The summer school is fully funded including travel expenses and applications are open now.

If you (or anyone you know) would like to apply then be quick as applications will close at 9am on Monday 15th of August. Details of the summer school can be found on the SeIUCCR website and a detailed agenda is available from the registration site.

On the 97th of April 2011...

2011-07-20T09:00:00.001+01:00

Back in February, an over-optimistic fool promised that the NGS would have a working Nagios service in the next few weeks.

The over-optimistic fool was confident because he had a real deadline to meet. Nagios had to be ready by April. April was the month during which the old NGS core sites - which ran the tests for our old INCA-based testing framework - were to be decommissioned.

We are running little late... but I am pleased to say that 2 weeks ago - on Wednesday the 97th of April 2011 - the NGS's Nagios testing service finally went live.

If you have an certificate and it is listed in the Grid Operations Centre database - you can pay it a visit at https://nagios01.ngs.ac.uk/nagios.

If you haven't or aren't - sorry: WLCG Nagios, unlike INCA, denies access to unregistered users by default. We may be able to remove the restriction in future - but, for the moment, we want to focus on fixing the problems it has found.

It is a bit untidy - as we have been without a fully working monitoring service for over 6 months.

While we kept the INCA service running as long as possible, it had become increasingly out of step due to a decision - very early on - to use the 'NeSCForge' software repository as a safe place to keep its configuration.

NeSCForge was not as safe as we had hoped. It vanished in December last year. The list of sites and tests to run remained frozen in their December state... and the Grid moved on.

We have different partner sites offering different services now. INCA wasn't testing them, Nagios is.

More significantly, Nagios takes its list of sites directly from the Grid Operations Centre database. Changes made there should be reflected in Nagios within a day.

My colleagues in the NGS Partnership team are working their way through the Nagios test results. They are identifying problems, finding missing sites and services - and, most importantly, working out how to make things better.

Avoid meaningless pretty pictures

2011-07-19T00:19:00.006+01:00

With the OGF science-in-the-cloud SAUCG workshop closing, it is time to reflect on the many interesting presentations, and try to identify common areas, next steps, etc.

How do we best provision resources for scientists? "Cloud" is a buzzword but there are drivers behind the push for it: increasing resource utilisation (maybe), service provision for small customers (the large, from the service provider perspective, being griddy), dynamically catching up with work and coping with the last-minute work prior to a conference. Lots of projects presented interesting stuff - see the slides - and expect an NGS surgery on the topic. To take this forward we now need to look at roadmaps - eg NIST and SIENA - identify gaps etc.

And the award for the quote-of-the-day goes to Etienne Urbah for the title of this post, and for his recommendation that "Passive sentences should be avoided."

PS. If you pronounce SAUCG "sausage" then it's entirely your own fault.

Going to town

2011-07-14T14:53:00.000+01:00

One of the local organisers said that they could not recall a time when the lecture theatre - which officially holds 185 people - had been so full.

It was all the more remarkable given that it was being used for a meeting called - deep breath - A Town Meeting to discuss UK Strategy for a Research Computing Ecosystem and the Future of e-Science.

As a rule: meetings that mention 'UK Strategy', 'Computing Ecosystem' and - especially - 'e-Science' do not attract huge numbers of people. This one was special because, somehow, the organisers had persuaded every branch of that amorphous thing called e-Science to come along.

There were the people from PRACE - who provide the really big compute for solving really big problems - and people who run the Institutional High Performance Computing services that drive so much UK research. We had the big data brigade - from Bioinformatics and Earth Systems science - who feed new research. We had representatives from research computing services, institutional IT services and the JANET network. We had the academics who push the limits of what you can do with a computer.

And, of course, there were representatives of the Grid - including the NGS, the Particle Physics community and less-traditional-users such as biology.

And everyone in the room agreed on what we needed to do.

I'll repeat that.

Nearly 200 people involved in academic research gathered in a room and unanimously agreed on what we needed to do next.

That is `what we needed to do' not `how we were going to do it'.

Everyone agreed that 'e-Science' must be driven by what the people who do the research actually need.

Everyone agreed that training for researchers is vital.

Everyone agreed that well-written robust software leads to better research.

Personally, I would like to have heard less agreement and more discussion. The e-Science community is full of people who have tackled difficult problems - sometimes successfully, sometimes less so - but the town meeting was simply too large scale for discussions.

Technical discussions are best served by gathering small groups of well informed people. They can get quite heated, but this is not necessarily a bad thing. The Moonshot meeting the day before was about the right size.

That is not to say that the discussions - heated or otherwise - did not happen. It just that they happened outside the meeting, by the coffee urn, or in the pub, or on the slow train home - between smaller groups of people who happened to be in London at the same time for a big meeting with a very unwieldy name.

If you weren't among the attendees, you can find some of the presentations on meeting's web page and follow the collective twitterings of some of those who were.

On Moonshot and telling the world who you are.

2011-07-11T12:22:00.015+01:00

The International Coffee Organisation's Board Room would make a damn good lair for a James Bond villain.

It also served rather well as a venue for a workshop organised by Project Moonshot, held last Thursday and focuses on using moonshot-authentication in Grid and High Performance Computing.

Josh Howlett - JANET's Mr. Moonshot and the workshop organiser - singularly failed to bring a white cat to stroke. And if he had a secret button to drop troublesome guests into his pet shark's tank - he resisted the urge to use it.

He was, however, quite happy to describe his plans to Take Over The World.

We've mentioned Moonshot before: it's goal is to re-use the network of authentication servers that has been created to provide Eduroam to control access to other services.

Moonshot allows people to authenticate themselves securely using their 'home' username and password. It is based around Tunneled Transport Layer Security provided by the Extensible Authentication Protocol and a network of RADIUS servers.

A service can refer authentication decisions onto a remote Authentication (AAA) server. Any chatter between the client and the AAA server that proves the user is who he or she claims to be - such as username, passwords or SPECTRE membership number - is hidden from the service itself. For the simplest uses, all the service needs to see is a simple yes or no.

There are many places where Moonshot could make life easier:

Moonshot could make is easier to share High Performance Computers. If it delivers what it promises, you could be granted SSH-access to a service anywhere in the world without needing a separate username and password.
In the grid world, adding a sprinkle of Moonshot magic to a Myproxy service or to a Credential Translation Services could make grid certificates available without resorting to a web browser.

This is where things get interesting, or complicated, or political. Depending on your point of view and position in the IT food chain.

For SSH or certificate access, the service needs to obtain some kind of unique, persistent identifier for every user.

But for the current Eduroam service - all you need is confirmation that the user is from a particular institution. It does not need to know if they are the Vice Chancellor, an esteemed professor or a junior researcher.

At the moment all an institution's RADIUS server need do is confirm or deny that the person connecting is a legitimate user of the network. There are no unique identifiers involved.

For Moonshot to be of use in the grid and HPC worlds, institutional RADIUS servers need to release additional information that can be passed back to the service.

The question is what additional information?

Should it be an email address? james.bond@mi6.gov.uk
Should it be something like a login-identifier? bondjames@mi6.gov.uk.
Or should it be pseudo-anonymous? 007@mi6.gov.uk. (If you don't have a license to kill, then this would be something like the Shibboleth eduPersonTargettedId - as used by the UK Access Management Federation - which is unique to a person and a service)

All have advantages and disadvantages....

IT Security People really dislike seeing usernames being released. You really don't want to give a potential attacker any help in cracking into a system.
There are legal and licensing rules that restricts access to certain classes of data - such as Ordinance Survey maps - to named individuals. Likewise, HPC service managers are far happier granting access based on an email address rather than a random collection of characters.
Researchers in some fields, especially Life Sciences, are understandably protective of their personal information and would much prefer pseudo-anonymity.

This is far too complicated a problem to solve at a single meeting, even if one has the the threat of becoming a shark's lunchtime snack to concentrate the mind.

Moonshot is a very impressive project, with international reach and practical contributions from experts in the field. They strike me as the right people to solve it.

Tweeting - the sound of (e)Science.

2011-07-08T10:05:00.006+01:00

We are in the Sir Ambrose Flemming lecture theatre in University College London at the start of the e-Science town meeting.

The room - which has space for nearly 200 people - is full. The meeting has just started.

On his perch high above me - resplendent in his finest plumage - is Steven Young, our official NGS tweeter.

You can follow Steven as he reduces the sweet sounds of e-science to 160 characters-or-less on the NGS twitter feed.

CERN and SSO - your questions answered (maybe)

2011-07-06T18:40:00.001+01:00

There have been a number of questions about this recently.

CERN let people sign in to some of some of their stuff if you have a certificate from any approved CA. Which is jolly nice of them - for those of us who don't have CERN accounts, if we need to access something for some reason, we can just log in with our personal certificate.

Last year, around August some time, CERN upgraded their single sign-on (SSO) infrastructure, and one of the security related "improvements" was to check for Certificate Distribution Point (CDP) extensions in intermediate CA certificates. We didn't have that extension (neither, for that matter, had DoESG), as it is not required by the standards, and the requirement was discovered, of course, only after the upgrade. (It doesn't seem to affect any other service that uses certificates.)

As a workaround we created a new certificate just like the existing one but with a CDP added to keep CERN's SSO happy. This was sent to CERN (only), and they deployed it, and it worked, and everybody was happy.

Until fairly recently when it stopped working, for reasons unknown, and of course without any warning. It looks like it was "undeployed" for some reason, and cannot be redeployed?!

So I guess it's up to us to "fix" it - but how? Changing existing CA certificates in the distribution is something that should not be done lightly - we could do this, it should work, technically, but it may at least be an inconvenience for users who have no need for CERN SSO, and one never knows if there's something out there that'll break.

Meanwhile, we have a rollover coming up in September (of which more anon). New CA certificates will have CDP, so should also keep CERN's SSO happy. We can test whether it works at CERN already because they are already deployed. Therefore, the new workaround is to start moving people who need CERN SSO over to the new CA certificates. We'll trial this with a few certificates shortly, and then proceed with the rest.

These are all "just" technical issues, and it is certainly not the first time someone has broken something by upgrading or "improving security" - it happens all the time (and occasionally to us, too). This is a good reason to keep certificates fairly conservative and not try out new exciting features on unsuspecting users. Which is why we have documents like GFD.125. (Which may be due for a revision - something to discuss at OGF32 perhaps.)

ARC quirks and keeping track

2011-07-06T10:55:00.010+01:00

Another week, another apology for having very little new to say.

Our excuse is that we have all been busy preparing for a meeting of NGS Collaborators (today, 6 July), a workshop on Moonshot, Grid and High Performance Computing (on Thursday) and for a Town Meeting on the future of e-Science and HPC Infrastructures and Applications in the UK (Friday).

On the plus side, when we have all recovered, there should have plenty to write about.

There has been a small amount of time available to work on Leeds' ARC grid software deployment - concentrating on the dull-but-useful task of tracking down and reporting minor bugs.

One such quirk appears when logfiles are rotated - that is renamed and compressed at regular intervals to conserve disk space. ARC continues to write to the original - now renamed - file rather than to a new one. We found the bug, and reported it and discovered that the developers were already aware and it will be fixed in the next release.

Which gives me a chance to opine...

In the 4-years-or-so that I have been involved with The Grid, I have needed to contact software developers all over the world.

With very few exceptions, the developers have been capable, helpful and responsive.
(and I am not going to to identify those very few exceptions)

The open nature of much of the development work, with bug databases and source code repositories readable by anyone, gives bug-hunters from outside the development team as much information as those inside. If used wisely, this information means better bug reports and faster fixes.

Unfortunately, bug-hunting is becoming harder. It is an unfortunate side effect of
the European Grid Infrastructure and European Middleware Initiative - and their remit to co-ordinate development activity from many disparate teams.

The individual development teams have their own development processes and tools....

The WLCG Nagios developers at CERN use installations of Jira for bug-tracking, Confluence for documentation and and Fisheye for viewing the source code
The gLite developers use their Savannah service for bugs and feature requests and provide ViewVC for source code browsing. [Addition 8-July: As Brian Bockelman has kindly pointed out - the ViewVC link above no longer takes you to the latest versions as components of gLite have moved to a TRAC instance at CERN.]
ARC has a bugzilla for bugs and uses TRAC for source code browsing.

On top of this is the all-seeing EMI GGUS tracking system that sends bug reports to the individual teams and the EMI wiki.

There are different interfaces for different tools, sometimes you need a certificate, sometimes you need an account. As someone slightly outside - but with an interest in - the development of grid software, I know how hard it can be to check if a bug has been reported, whether it has been fixed and when the fix will be available.

There is clearly work being done to improve the situation and no-one would claim that distributed, international software development is easy to do. What the grid really does not want to do is weaken whatever connections there are with the system administrators who deploy the software and the external developers who use it.

How has it affected you?

2011-06-30T12:14:00.002+01:00

As you are aware the NGS has gone under some changes since the beginning of the year. The most obvious to our users was the cap on cpu hours that was introduced on the 1st of April with every user being given 2000 free cpu hours per year. If a user requires more than 2000 cpu hours this can be arranged through the NGS on a "pay per use basis".

So 3 months in since the changes were introduced and we want to hear your views. We are asking all registered NGS users to complete a very short survey to feedback their views and experiences on using our services since the 1st of April. It doesn't matter if you haven't used the NGS since before the 1st of April you are still our target for this survey!

The survey will take a few minutes to compete if that and everyone who is a registered NGS user either past or present will be entered into a prize draw.

Delayed

2011-06-28T10:39:00.004+01:00

In the best traditions of a UK railway station announcement... The NGS would like to apologise for the late arrival of last week's R+D blog posting. This was due to the non-arrival of some good news.

Over the last few weeks: we have made good progress with our Nagios service and with persuading our Workload Management Service to accept certificates generated using institutional credentials and the SARoNGS service.

Unfortunately, the good progress wasn't quite good enough to deliver something that actually worked.

It shouldn't take much longer. Honest.

That's not my cow

2011-06-24T12:22:00.004+01:00

With apologies to Terry Pratchett for the title of this blog post...

You may have spotted an unusual picture on the NGS home page this week. What on earth have cows got to do with the NGS? A very good question...

The answer lies in a recent iSGTW article which once again promotes NGS users research to an international audience.

The article featured the work of the GridQTL team at the University of Edinburgh who provide the GridQTL grid-based platform for running gene analysis. GridQTL is used by researchers all over the world and, unbeknown to many of them, all the computing jobs are submitted to the NGS providing a massive speed up in analysis time.

Team leader Dr Sarah Knott explained “It would be very difficult to run this kind of software without grid computing. We have an uncertain user base and the grid provides the flexibility to sustain an adequate speed of analysis, regardless of online users.”

To see a NGS user case study on GridQTL visit the case study section of the NGS website.

Mind the gap

2011-06-20T10:00:00.009+01:00

I can't say that I wasn't warned.

In an attempt to conserve the world's supply of three letter acronyms - and support people studying the Sun and those studying society - we are trying to deploy grid software called ARC on a compute service called ARC.

The Nordugrid Advanced Resource Connector software is the only component within release 1 of the European Middleware Initiative's big bundle of grid stuff that does what we need:

accepting work requests from a Workload Management Server.
passing them on to HPC systems which may be running ~~Sun~~Oracle GridEngine, Torque/PBS or SLURM batch systems. (Leeds is using Grid Engine, but if we are successful, it could be rolled out to other institutions).

The EMI are in an uncomfortable position. Their job is to take pieces of software from different places - that is similar in intent and very different in design - and persuade them to work together. Sometimes, inevitably, things fall through the gaps.

One of these gaps is between the gLite's BDII information service and ARC's own information services.

This is how it is meant to work...

Information is kept in a BDII-friendly database and made available to the world via the LDAP protocol through an OpenLDAP 'slapd' service.
On any given system, this information is generated as a set of LDAP 'LDIF' format records by programs called providers and plugins.
A program called bdii-update takes the locally generated LDIF, processes it and passes it on to slapd.

What was actually happening...

The ARC information system was generating lots of LDIF.
The bdii-update process was collating it and passing it onto slapd.
slapd was refusing to accept it - complaining of an 'Object class violation'.

After digging into the inner workings of both the BDII and ARC, we've identified the cause. It is all down to a subtle difference between what Nordugrid expect and what gLite expect from their information services.

From this point on, this is going to be technical. Readers of a less geeky disposition can look away now, happy in the knowledge that we know what broke and how to fix it.

Geeks, grab yours Acronyms. Here we go...

Slapd relies on schema files to define what is acceptable: Nordugrid have their own Scandinavian-style nordugrid.schema; gLite use the GLUE schema, including one called Glue-MDS.

Glue-MDS and nordugrid.schema both define an objectClass called 'Mds'. Both agree that it represents a collection of information but in GLUE, an Mds is defined as a STRUCTURAL class whereas Nordugrid defines it as an ABSTRACT class.

So what... as anyone who managed to make it this far down the page might cry.

Well, in the LDAP-world, STRUCTURAL objects can exist whereas ABSTRACT classes can only be used as a basis upon which other objects can be defined. Its all very Object-Oriented-Programming.

ARC's information service generates 'MdsVo' objects, based on Mds objects, but properly STRUCTURAL. This is fine according to the nordugrid schema.

But bdii-update contained code that takes any object that is based on an Mds object and turns it into a plain, simple self-contained Mds object. This is closer to what GLUE expects.

Slapd gets very confused.

A bug report has been raised - and after a bit of bug ping pong between the BDII and ARC developers - it has been decided that bdii-update should, in future, leave Mds objects alone. For the moment, all that is needed is to remove the line in bdii-update that reads

  new_ldif = fix(new_dns, new_ldif)

Hot off the press…

2011-06-16T11:12:00.002+01:00

Released the beginning of June, the new edition of the NGS quarterly newsletter is out now. The June edition features a range of articles including –

Supporting e-Infrastructure Uptake through Community Champions for Research (SeIUCCR) - a new joint project between the NGS and the SSI
around up of the successful and engaging Collaborations Workshop held by the SSI earlier this year
an in-depth look at the new NGS load monitor
Sysadmins corner tips on using Screen
an introduction to our new member site - the University of Huddersfield

We are always looking for news items for our newsletter and fortnightly email bulletin so if you have anything you would like to contribute please get in touch! We are particularly keen to hear from our users and their application of e-research. Have you recently presented or published a paper featuring research performed on the NGS? If so then we’d love to hear from you!

My data, your data

2011-06-13T13:52:00.001+01:00

Edinburgh hosted the Data Intensive Research workshop last week, along with the XLDB Europe. Genially chaired by the formidable Malcolm Atkinson, it was an excellent opportunity to hobnob with other data management folk. While most were database people, many were plain ol' data managers, or people who have data and nowhere to put it.

One of the recurring themes was that there should be more recognition for the work supporting science: if you have a dataset and you spend time preparing, validating, and annotating it, and someone else publishes a result derived from this data, they get the credit for the discovery - naturally - but the work to prepare the data should also be recognised (e.g. in RAEs, maybe as something equivalent to a publication.) Also if it doesn't lead directly to a new discovery. People who share data contribute to science. Moreover, these days, sharing data is often necessary for public funding.

The NGS has a few datasets in bioinformatics already. Having it available already is highly advantageous because you don't have to transfer and update it yourself, and it doesn't take up any space in your storage quota. If there are other datasets you think the NGS could usefully provide, let us know?

Speaking of research support, if you use the NGS you should of course cite NGS in your publications. But maybe you'll also want to buy your friendly local NGS admins a beer or something?

Who do we think you are - revisited.

2011-06-13T10:51:00.004+01:00

Once upon a time - before the NGS service at Leeds retired - it was possible to connect to grid-enabled machine using only a web browser, your institutional credentials and an ordinary ssh client.

To achieve this, we combined our Shibboleth 'single-sign on' service with a modified version of our Myproxy enabled GSISSH (MEG) code.

The modifications to MEG allowed us to give access to users without a proper NGS account - but provide them with a very restricted shell. IT security take a dim view of allowing anyone from anywhere to do anything on local computer facilities.

Basically, we needed to replace a gsissh command with a utility that translate a certificate and VOMS information to a local username and password in exactly the same way as a gsissh command.

The technical details were covered back in November. As we are tidying up following the end of NGS R+D activity, we have now packaged up the utility itself - called ngs-x509id - and put it on the UKNGI area on SourceForge.

Ngs-x509id is built around a library of X509 utilities developed by my colleague Robert Frank at Manchester. An early version of the library is bundled with ngs-x509id and Robert has kindly made the latest development version available from the UKNGI subversion repository.

Beep, beep - NGS coming through...

2011-06-08T11:04:00.003+01:00

There is a new NGS user case study up on the website and this time it deals with traffic. I'm sure this is a popular topic particularly at 9am on a Monday morning!

NGS user Aengus McCulloch from Newcastle University has been using NGS resources to tackle the explosion in the volume of data relating to real world events such as systems monitoring weather and road traffic etc. Processing the large volumes of data generated by geographical sensors in a short time period is an enormous computational challenge. Guess where the NGS can help out?

Aengus used the real world problem of traffic monitoring and routing as a test case for classifying near real-time processing operations on geographically referenced data using Grid computing architectures. By using the NGS he managed to successfully test his system with up to 250 vehicles.

Aengus said "The NGS has made endless CPU hours available to me through a standards based interface; it has been an invaluable resource for my research".

To read more about his research, see the scalable road traffic case study.

When acronyms collide...

2011-06-03T15:18:00.008+01:00

Now pay attention. This may be a little confusing.

The University of Leeds runs high-end computer clusters as part of its Advanced Research Computing service - known as ARC.

The NorduGrid Collaboration develop, maintain and support grid middleware called the Advanced Resource Connector - also known as ARC.

We had a plan to join ARC-the-computing-service to the grid. The first version of the plan, to use European Middleware Initiative's release of the CREAM software, has been abandoned because support for our batch system - SGE - is not yet ready

So, inspired by work done by Stuart Purdie for Scotgrid, we have a New Plan.

You have probably gathered that The New Plan is (drum roll please) - to deploy ARC on ARC.

Which, when you come to write it down, sounds less like a plan and more like an overexcited performing seal.

It is our only real option: ARC is the only one of the EMI's current middleware menagerie with SGE support.

Round these parts - we like to automate software installations using scripts. Our scripts are based on the instructions from the EMI's Generic installation guide and on Nordugrid's guide to plugging an ARC Compute Element into the European Grid and are now at the point where they can install and configure ARC from the EMI repositories.

The scripts are not yet ready for an official release but - in the unlikely event of anyone wanting to know more - the 'live' versions can be found in the UKNGI code repository at SourceForge. Like the earlier VDT installer scripts - they are intended as runnable documentation. If you don't want to run them, you should be be able to see how they work by reading them.

Incidentally, just to add to the confusion: Leeds also hosts an Antimicrobial Research Centre - known as ARC. Their specialities include some of particular relevance at the moment: antibiotic resistance in bacteria and the transfer of pathogens.

When I last checked, there were people from ARC-the-research-centre using ARC-the-computer-cluster but not via ARC-the-middleware.

999 – NGS Surgery

2011-06-01T10:16:00.001+01:00

Did you know that the NGS runs Surgery meetings every second Wednesday? The meetings are run over Access Grid / Evo and are open to all.

The surgery meetings have previously been mainly for supporting NGS member sites in installing software etc on their resources. However recent surgery meetings have widened their remit to discuss topics of interest to Campus Champions as well as the Sys Admin community within the NGS.

The NGS is always looking for suggestions for presentations / discussions at forthcoming surgery meetings. If you have any ideas or would like to raise a topic then contact the NGS helpdesk and let us know. Just because you suggest a topic doesn’t mean that you have to present so don’t worry!

To give you an idea of the sort of topics that have been discussed previously, have a look at past NGS surgery meetings on the dedicated section of the NGS website. You can find all the presentations here as well.

If you want to keep up to date with all the forthcoming surgery meetings, make sure you check the NGS website regularly or drop an email to the helpdesk and they will add you to the mailing list.

Lost in the Workload Management System

2011-05-30T23:13:00.006+01:00

If you work with a complicated technology for any length of time, you build up mental map of how the bits fit together.

In a corner of my mental map of the Grid world is the WMS, aka the Workload Management System. Next to it is a warning: Here Be Dragons.

I know what the WMS does: it's a mixture of matchmaker and postal service. It takes a list of tasks, distributes them around the grid and makes sure they reach their destinations. I know that WMS is part of the NGS's UI/WMS service and provides a comparatively simple way to get work done on the Grid.

What I do not understand is how it works. This is a little embarrassing, as the NGS is currently dealing with two WMS problems: a long term problem that limiting the kinds of users we can support, and a more urgent problem - which surfaced last week - that left the UI/WMS service unusable by all NGS users.

I must emphasise that for the urgent problem: we now know what broke and how to fix it. The fix should be deployed later in the week. Please keep watching the status report on the website or the NGS-STATUS mailing list for updates.

In both cases, we see failures in the authentication between the components of the WMS.

In last week's major incident, the WMS refused to recognise the NGS's virtual organisation service after the associated Virtual Organisation Management Server (VOMS) it was upgraded. If you tried, you were told...

unable to delegate the credential to the endpoint...

Our longer term problem is with the WMS and the `SARoNGS' certificates generated by cts.ngs.ac.uk. CTS - which I think stands for Credential Translation Service - allows you to obtain a grid certificate using just your institutional username and password.

The downside of SARoNGS certificates are signed by a certificate authority that it is not yet recognised by the International Grid Trust Federation. Services must explicitly recognise the certificate authority before anyone can use it.

Somewhere within the WMS, the validation step has gone wrong. If you have a SARoNGS certificate, it tells you...

Connection failed: CA certificate verification failed

The WMS - as you might gather from reading the overview of its architecture - is really a linked collection of services. The official Service Reference Card lists 12 separate components that need to be running for the WMS to function. Some of these depend other pieces of software. In particular, some run within a webserver and use Gridsite to provide access control.

It was a slightly-out-of-date version of Gridsite that caused our major problem last week.

The VOMS server update changed the format of the attribute certificates that link you to a particular Virtual Organisation. Previous releases of the VOMS service used MD5 digital signatures within the attribute certificates. The current one has replaced MD5 with the more secure SHA1.

Our copy of gridsite only knew about MD5 signatures. An updated, SHA1-aware, version was made available late last year. We just hadn't realised that it was needed until last week.

We think that the problems with SARoNGS certificates can be traced to quirks in the way certificate authority information is being passed around the WMS. We are very fortunate that the Software Sustainability Institute has been able to offer the NGS some additional development effort to find these quirks

The Software Sustainability Institute knows its way around the grid. Their developers know how to deal with complicated software. It takes more than a 'Here Be Dragons' warning to stop them...

Getting the users story across

2011-05-24T10:40:00.002+01:00

As you are no doubt aware, I’m always after interesting pieces of user research to feature in our user case study section of the NGS website. User case studies are ideal introductions to people who have never used the grid before as it can demonstrate what can be done with the resources and what the benefits are.

Whilst the production of user case studies benefits the NGS, they also benefit the researcher featured as quite often the user case studies are picked up by other grid organisations.

This was the case with NGS user Cristiano Sabiu who hit the headlines in iSGTW last month with his user case study on his cosmology research.

Cristiano from the University of Portsmouth used NGS resources to study the distribution of galaxies in the universe by creating 2000 mock universes for comparison to actual galaxy distribution in the Sloan Digital Sky Survey (SDSS). By using the NGS he managed to run 20 full scale simulations which required approximately 100,000 cpu hours over the course of a year.

His research was made into a user case study for the NGS website from where it was picked up by International Science Grid This Week (iSGTW) and appeared as their headline article in April. iSGTW is a weekly online publication that provides international coverage of distributed computing and the research it supports. They feature articles on all aspects of distributed computing technology, such as grids and clouds from all research areas and are always on the look out for interesting stories to cover.

The NGS user case studies keep on coming though and the most recent one is on quantum mechanics modelling. Stewart Reed from the University of Leeds used NGS resources to develop new methods of performing accurate computer simulations of quantum mechanical tunnelling.

Stewart explained how "the NGS provides excellent computing resources with which to perform these calculations. The computational capacity available from the NGS allows larger systems to be studied more accurately than are possible with standard workstations”.

To read more about his research, see the quantum mechanics modelling case study.

Good news, bad news

2011-05-20T13:17:00.007+01:00

It isn't a case of one step forward... two steps back.

Its more one step forward... with another step forward coming soon.

This week, both the Leeds CREAM-CE and the NGS's Nagios project inched forward.

CREAM

We started with good news thanks to a comment from Ewan at Oxford on Leeds' plans to install a grid front end to our ARC1 High Performance Computing service.

Ewan pointed out that our preferred approach - leaving the grid access on a machine almost-completely-detached from the HPC service - is a) also other people's preferred approach and b) one that actually works.

Which is nice.

And would be nicer if it wasn't for the bad news: Sun Grid Engine support in CREAM never made it as far as the first major release (EMI-1) of the European Middleware Initiatives's grand unified grid software.

Grid Engine support is expected to arrive in a minor release - coming soon.

NAGIOS

We have had a working Nagios development system for some time.

We were trying to build a working 'clean' test system. We were planning to use this to practice the full Nagios install and configuration procedure before being let loose on a proper service.

And when we first practiced - a month or so back - the test server refused to install anything.

Since that time progress has been slow. We blame this on the stubborn refusal of the average day to include more than 24 hours, so cruelly depriving the systems staff of enough time to finish everything else that needs to be done.

Earlier this week, after reading the latest Nagios installation documentation, and comparing notes with the Nagios developers and our colleagues at Oxford who run the GridPP Nagios - we worked out what had gone awry.

There were some unfortunate conflicts between packages in the software repositories defined in /etc/yum.repos.d. We ended up in RPM hell...

Its better now. And as a minor bonus we did developed a utility that can edit YUM repositories in place. It can be found in the UKNGI subversion repository at SourceForge. It isn't pretty or clever but it does work...

2011-05-18T11:56:00.002+01:00

I’ve recently added a new poll to the NGS website home page which asks if our users collaborate internationally and, if so, if this involves the NGS.

The latest question is the 7th poll to appear on the NGS website so I thought that in this blog post I would recap some of the results of previous posts.

The polls have covered a wide range of topics from “do you like our new website?” way back in November 2009 when we launched the new site, to questions regarding funding and access to resources.

Back in March last year we asked people which operating system they used to access the NGS. This has an important bearing on how our tools and applications are developed in the future as we need to ensure that they are usable by NGS users regardless of their operating system. We had 175 responses and the results were -

Windows – 35%
Linux – 43%
Mac – 19%
Other – 3%

The results were pretty much as expected with Linux users being in the majority. However it was interesting to see that Windows users are catching up and Mac users growing in numbers.

Also many moons ago we asked if our users would like us to offer an academic cloud service. This was a popular poll with 137 replies and the following results –

Yes -66%
No - 20%
Don’t know – 14%

We are pleased to say that the NGS cloud prototype has been very popular and is currently running at capacity as we help and assist NGS users to make the most of this new prototype service.

Of course it’s not just specialised services that we offer, we also offer the day to day stuff that enables people to get started using the NGS. We wanted to see how the users thought we were doing in terms of authorising new applications to use NGS resources. Were we taking too long? Were we meeting expectations? We were pleased to see that nearly half (46%) of applications were approved the same day with 28% approved in the next 1 – 3 days.

We are always looking for new polls for the NGS website so if you have any suggestions then we would love to hear them. You can ask anything about software, hardware, services, usage, user stats etc. All the polls are anonymous.

Stuck in the middle

2011-05-16T11:14:00.014+01:00

To a casual observer, they look very similar: groups of largely male, occasionally slightly dishevelled individuals who spend far too much time starting at screens and who communicate almost exclusively in acronyms.

But those in the know recognise that your High Performance Computing (HPC) geeks and your Grid Computing geeks are very different creatures.

Should you find yourself talking to one - having somehow managed to side-step the awkward initial `Who are you and how did you get into my server room?' conversation - it is very important to know which species it belongs to.

By far the easiest way to find out is to ask the question: how does someone run a program.

If you have found a Grid geek, the answer will feature web services, UIs, WMSs, CEs of various kinds, certificates and, in extreme cases, XML.

The HPC geek will answer Ssh, Ssh, Ssh. Again and again and again.

That is SSH as in Secure Shell. If the person is saying Shh!, you've wandered into the library by mistake.

HPC is all about building the biggest, fastest computer that can fit in the room. HPC systems are designed to be self contained, with fast disks and fast CPUs linked using fast networks. Users may connect to an HPC service from outside - via SSH - but everything they do from that point on stays within an HPC bubble.

Grid is about connecting a disparate set of resources, spread far and wide, so they can do something useful together. There is very definitely more than one way to do it.

At Leeds, we are in the interesting position of trying to connect our HPC service to the grid.

Rather than trying to graft the full Grid software stack onto a very specialised, and customised HPC environment - we are using a separate (virtual) machine to act as a relay, or maybe a translator, between the two worlds. The HPC service is called ARC1. It is only right that the relay will be called NGS.ARC1.

It will talk Grid to the world, but its only channels of communication to the HPC service will be the batch queuing system - SGE - and good old SSH. There is no shared disk space of any kind.

We are now able to submit jobs from the NGS.ARC1 to the HPC service and monitor their progress.

We have the ability to create separate SSH keys for every grid user. Our next step is to configure the HPC service to use these keys in 'scp' commands within 'prolog' and 'epilog' scripts. Data will be pulled onto ARC1 from NGS.ARC1 when a job starts and pushed it back when the job is done.

The latest set of documentation for the CREAM-CE - our choice for the grid side - says that you can set:

  SANDBOX_TRANSFER_METHOD_BETWEEN_CE_WN=LRMS

and let the Local Resource Management Service, the grids general term for batch services like SGE, do the donkey work.

At the moment, we have no idea if weakly-linking the Grid and HPC in this way will work.

Well update you in a week-or-so's time.

And if you are passing through Leeds and want to ask questions, I will - of course - answer `Who are you and what are you doing in my server room?'

Moving pictures

2011-05-11T13:26:00.002+01:00

Back in March I attended the SSI Collaborations Workshop up in Edinburgh. I used to attend these events when OMII-UK organised them and they were always enjoyable and productive meetings. I wrote up my thoughts and activities on the 2 day event in the blog here (day 1) and here (day 2).

If you are interested in finding out more about these meetings such as who attends, how it works and what is discussed then you are in luck. The SSI has just released the videos of the report back sessions onto their YouTube channel. Unlike the usual "death by PowerPoint" these videos show discussion group members reporting back on the issues that their group discussed and the actions their group decided upon. The presentations from the report back sessions are on average about 2-3 mins long thanks to strick timekeeping by the conference chairs. Short and to the point!

SSI have also recently released a list of outcomes from the meeting which they are working on over the forthcoming months.

Looking back - from inside a brown paper bag

2011-05-09T07:22:00.008+01:00

The phrase 'brown paper bag' bug was coined by Linus 'Mr Linux himself' Torvalds to describe an screw-up that is both embarrassing and visible for all to see.

We have already covered the Good and the Not-So-Quite-Good aspects of the last two years R+D. It is now time for the Embarrassingly Bad.

Our brown paper bag moment comes courtesy of the under-used NGS Advanced Reservation service - which allowed any one who needed to to pre-book matching time-slots on multiple computers.

Providing an Advanced Reservation service was a major theme in our original plans for the 'Integrated Infrastructure' part of the R+D work. We wanted to build a service, monitor it, account for use and advertise its existence to the grid.

It started well. We had successfully deployed bits of the HARC co-scheduler as part of phase 2 of the NGS. It was being used to simulate the blood flow through the brain using computers in more than one location.

And then we hit a big problem - described on the blog back in June 2010.

HARC relies on having a network of computers acting as acceptors. Acceptors, unsurprisingly, accept user requests for reservations on a set of compute clusters. They work together: identifying and reserving matching slots of time on each cluster on behalf of the users.

Which is all very well, if the acceptors are working.

We originally piggy-backed on an acceptor network run by the Louisiana Optical Network Initiative (LONI) but, as time went by, this become less-and-less reliable.

So we tried to set up our own. And mostly failed.

You cannot run a production service if the acceptor network is not robust and for robustness, you need to spread the set of acceptors over more than one site. Neither LONI or the NGS could persuade the set of distributed acceptors to stay working for long enough to be useful. When the acceptors worked, they worked well.

When any part of the communication betweeen acceptors went wrong, all the acceptors failed, one after the other, domino style.

The chain of communications could be broken by misplaced firewalls or by small differences in acceptor configuration.

We put a lot of time and effort into sanity checking firewalls and synchronising the configuration between acceptors - ensuring that updates all happened at the same time - but we never managed to build a proper acceptor network.

Eventually we abandoned the project - leaving a small set of working acceptors at Manchester for anyone who wanted to use them

Our advanced reservation service may have failed but the idea of advanced reservation is still alive.

We are now in the era of the data deluge and of cloud computing. In the not to distant future, researchers will have access to massive amounts of data someone on the Internet and will need to get their hands on sufficient computer power and enough network capacity to process it.

There is still a case to be made for a service that can book compute time and network bandwidth, like HARC did. I'm just sorry to say that the NGS was not able to provide it.

Rest assured, the NGS has identified the blithering idiot who decided to spend time and effort on a failed service. The guilty party will be informed - in no uncertain terms - that he is a disgrace to the long and noble history of Grid software development.... the next time I pass a convenient mirror.

NGS TV channel?

2011-05-03T10:48:00.003+01:00

I'm beginning to think that the NGS might need it's own TV channel as there is another video featuring NGS staff available for your viewing pleasure.

This time it is the videos from the UCISA Cloud Computing seminar which was held in February in Loughborough, UK. This was a one day event focusing on a Cloud Computing which is becoming more and more popular in the UK academic sector.

The event was billed as "a perfect opportunity to take a look at the innovation and challenges with Cloud Computing and to reflect on the benefits it can bring to your organisation". The sessions were designed to give a flavour of Cloud Computing possibilities as well as case studies from early adopters.

The presentation videos are all available on the event website and include a presentation from the NGS Technical Director, David Wallom on the FleSSR project - Flexible Services for the Support of Research.

From the weather in 1994 to the climate now

2011-04-28T13:13:00.011+01:00

The most successful users of grid computing are - without doubt - the particle physics community. The Worldwide LHC Computing Grid stores and processes the Terabytes of data generated from the experiments at CERN.

It is all very impressive - even for people like me who don't understand what a Higgs Boson is.

I am not a particle physicist. Back in the mid '90s - when work on the Large Hadron Collider project was beginning - I was at the Department of Meteorology at the University of Reading.

So it was strange to see a familiar name from way-back-then appear in International Science Grid This Week - and asking what the climate science community could learn from CERN.

Back in 1994, the Meteorologists occupied one end of some re-purposed second-world-war huts on the outer edge of campus. On one wall of the huts, opposite the portacabin that served as a coffee room, were the results of one of the field's latest innovations: ensemble forecasts.

Weather forecasts take the best estimate of the state of the atmosphere at one moment in time and calculate how this will change over the next few days.

That best estimate of the state of the atmosphere - called an analysis - is based on observations gathered by the World Meteorological Organisation.

The observations are scattered unevenly over the planet. For entirely practical reasons, there are many more observations over land than over the sea, and many more at the surface than further up in the air. Parts of the world are not covered at all. The best estimate will never be perfect.

Weather forecasting uses huge amounts of computer power and an individual forecast must be completed within a few hours or it is useless. You do not want to be told that a storm is coming after it has arrived.

Until the mid-90's, the most powerful computers available were only able to run one forecast before time ran out.

Weather is chaotic: small features can grow rapidly into huge storms. If such a feature is misrepresented in the analysis - as happened in 1987 - that one forecast can be very, very wrong.

By the mid 90's, enough computer power was available to produce ensembles of forecasts - each using slightly different initial estimates. Sometimes members of the ensemble stayed close together, sometimes they diverged. For the first time, we could estimate how reliable the forecast will be.

The ensembles were produced by the European Centre for Medium Range Weather Forecasting, a short distance up the road from the Meteorology department. This pioneering work is as relevant to the long term forecasting of climate as it was to the short term forecasting of weather.

Tim Palmer was one of those behind the ensembles. He is now a Professor at Oxford and one of the foremost experts on weather and climate prediction forecasting. The talk he gave at the American Geophysical Union fall meeting in 2010 is one of the clearest descriptions of one of successes and failures of the field.

In this week's edition of International Science Grid This Week carries an article by Professor Palmer advocating 'a CERN for Climate Change'.

Human activity is changing the climate - whatever you might read elsewhere on the Internet - but the details of the changes are still not understood.

If those who have studied our climate say we need a CERN-like organisation to understand it, then we should be listening.

[Small edit for clarity, 3 May]

The sun is out

2011-04-26T11:45:00.003+01:00

Given the very sunny Bank holiday weekend we've just had here in the UK, it seems apt to highlight the latest user case study to appear on the NGS website.

The latest case study highlights the research of Marco Califano at the University of Leeds who has been looking into exploiting solar power using the NGS. Marco used NGS resources to try to identify the best materials, sizes and shapes of nanocrystals that could lead to high-efficiency energy conversion in the next-generation solar cells.

Marco explained how "the outcome of my research could help the UK develop alternative, renewable energy technologies that enhance the cleanliness and efficiency of energy production. This, in turn, will contribute to the Government's commitment to meet the greenhouse gas emission targets set in Kyoto".

To read more about his research, see the exploiting solar power case study.

EGI User Forum round up

2011-04-21T13:57:00.003+01:00

Last week saw many NGS staff attend the EGI User Forum 2011 which was held in Vilnius, Lithuania. Many NGS staff attending were actively involved in sessions either chairing or presenting as well as helping out on the UK NGI (UK National Grid Initiative) exhibition stand.
Personally my main involvement was ensuring that the UK NGI exhibition stand got there in one piece and that staff from NGS and GridPP were on hand to answer questions from delegates about our organisation. We had a very busy stand – possibly one of the busiest we’ve had at these events which was good to see.

I was also presenting at the EGI Dissemination session which was chaired by Neasan O’Neill, NGI Dissemination Coordinator. Neasan had asked some of the more established NGI’s to discuss the dissemination methods they had found most useful and effective. I presented on the NGS roadshows of which there have been 12. I discussed the content of a typical roadshow, the feedback we receive directly after the event and the follow up I do approximately 3 months after the event.

I also ran through (very quickly!) the other dissemination methods we have including our user case studies, our quarterly newsletter and the innovation forums. There were quite a few questions and some good discussion after the presentations with quite a few leaflets disappearing from the stand as well which meant less to carry home!

The conference was excellent in general with a brilliant venue and fantastic organisation. The next EGI event is the Technical Forum which will take place in Lyon in France on the 19th – 23rd September 2011.

Looking back - and being differently successful

2011-04-20T12:08:00.014+01:00

Failure is not an option... it's a standard feature.

As witicisms go: it is neither particularly funny, or particularly original - and can be seen on T-shirts, fridge magnets and snarky comments across the Internet.

Anyway, not being funny or original has never stopped me before.

We are looking back at the last two years of R+D: we've done the successes, now it is time for the failures.

I'm not talking about those magnificent 'Brown Paper Bag' failures that will haunt us for many years to come. I'll cover those another time.

Here, I want to mention those projects that have been mentioned at conferences or at this blog, which are working or very nearly working, but are not where we hoped they would be.

I am not criticising the highly-capable and skilled developers involved, or the way the work was managed. The NGS is a small organisation: bad luck or the need to solve more pressing problems limit what we can do.

The nearly-successful projects involved are the CA Wizard, SARoNGS and DataMINX.

The CA Wizard (not to be confused with the existing NGS Certificate Wizard) was intended to replace the confusing - and, to some users, downright scary - business of obtaining a certificate with a single desktop tool.

It was delayed by the loss of some key members of staff but is still being actively developed. The developers expect to release a version in May.

DataMINX was a joint UK and Australian service designed to ship data to where it was needed with minimal user involvement. It also fell foul of staff losses and had to finish early.

The developers were concerned about software sustainability and have left the code in a state where someone else could readily take over: it was released with an Open license and is available from Google Code. It is still being recommended in some quarters.

SARoNGS has been mentioned a number of times on the blog. It provides a way of allowing users onto a grid using their institutional credentials.

We know that SARoNGS development has been too slow. We know that the user interface is confusing. We know it doesn't quite work with the UI/WMS. We were reluctant to turn it into a production service until we had overcome some mind-twistingly-obscure bugs.

But SARoNGS has been used - at York, within the NeISS project, and elsewhere - as a simple way to pass authentication information around.

Failure may not be an option but - thanks to the efforts of people within and outside the NGS - it isn't inevitable either.

[With thanks to David Meredith]

Shooting for the Moon

2011-04-14T09:40:00.005+01:00

The JANET networkshop is now in its third day, abuzz with all things networking and with lots of networkers from industry and academia networking with each other. But also for identity managers was there something of interest.

Project Moonshot is an activity with ambitions to, er, shoot for the moon - to develop an infrastructure for what I call A(A(A))I (usually people "solve" authentication first, then look at authorisation, and then maybe accounting.) The great thing about Moonshot is that it is built entirely on standards.

Moonshot's Sam Hartman gave a very impressive presentation on the next five years in security, covering both grids and clouds. The amount of technical data Sam is able to hold seemingly effortlessly in his head reminds me of the people who play simultaneous chess. Josh Howlett from JANET gave a very interesting presentation on the technical progress and current state of Moonshot: in particular, note how many services work almost unmodified because they use generic security services (JANET will make the presentations available online.)

[Managers' version] Moonshot is a way for users to authenticate to resources. As with IGTF and eduRoam, trust is built from federations - if your user is in the same institute, it is easy enough to trust them; if they are in another country, you need to work your way up the hierarchy. But this is all transparent because trust relationships are established beforehand. So why something new? Shibboleth is tied closely to web resources, IGTF certificates are usually separated from home institution ids, eduRoam is currently mainly for networks. If we can make everything interoperate, you can use your home id to authenticate to everything: the grid, the network, clouds, your toaster, etc. Federated identity brings users single sign-on and single accounts for multiple services.

[Techie version] Moonshot links a RADIUS (2865) service (like eduRoam) into applications using a GSSAPI (2743) module for EAP (3748). So it's all very modular and based on existing standards, but of course much of the module and glue stuff is new. Proposed extensions to GSSAPI will bring support for delegation (5588, 5896). GSSAPI already supports (4121) Kerberos, and this has been tied into MyProxy by Daniel Kouril from CESNET. Sam has built a virtual image which can now be used to demonstrate this. Based on Debian, it should work with several different image hosting environments, such as Xen or VMware. For the NGS, we should now deploy a(nother) MyProxy server, I am thinking of using the training CA, to enable Moonshot access to the NGS.

[User's version] You will be able to access more stuff by just using your home institution password!

Looking back - where did it all go right

2011-04-08T16:12:00.006+01:00

Now all involved have been properly thanked - we can look back at the more-successful and, er, less-successful projects that the NGS has undertaken.

When phase 3 of the NGS started in April 2009, the first thing we did was to make a list.

It wasn't exactly a To-Do list, more of a Could-Do list.

We asked for suggestions for new `services' - 'service' here being a woolly term for something that someone, somewhere might find useful - and collected just short of 40 ideas. These covered the themes of Advanced Reservation, Middleware deployment, User facing services and Data access.

It would have been impossible - given the time and staff available - to implement them all. We had to identify those that were impractical and concentrate on the most promising few. Sometimes we got it more-or-less right; sometimes we didn't. As has been noted before `If we knew what it was we were doing, it wouldn't be called research, would it?'.

In this post, I would like to mention those places where, in my opinion, we got it right - starting with the NGS communities.

Our users have to spend time obtaining a certificate and applying for an NGS account. We wanted to make that account as useful to them as possible.

With communities, individuals can associate themselves with subject area - anything from Aerospace to Zoology - and search for others in the same area. They can look for people based at their home institution, or read the descriptions of their work that others have entered.

Behind the scenes, we also added tags - labels that the NGS support staff can associate with a user and with membership of a group within a Virtual Organisation.

A tag can, for example, identify someone as a legitimate user of a piece of licensed software - and that information can be used to make that piece of software software available on different hosts. This avoids some of the complications of licensing on the Grid.

No list of R+D projects would be complete without a reference to the technology behind the NGS interactive Applications page - where present the list of available software on the NGS website for users to explore.

These are not huge projects or major changes in technology. What we wanted to do - and I hope have done - is to make the grid a little more friendly and a little more useful.

Multi-tasking NGS staff

2011-04-05T13:52:00.002+01:00

Many people involved in the NGS don't just contribute to our organisation, they contribute to many more. For example our Technical Director, David Wallom who is based at the University of Oxford, is also heavily involved with the OGF as the Vice President of Community.

David recently attended the International Symposium on Grids and Clouds 2011 (ISGC 2011) where he was interviewed by the GridCast team regarding the future of the OGF. If you would like to see the interview video, see the GridCast blog post.

Looking back and saying thank you.

2011-04-01T14:58:00.007+01:00

This is the 50-th R+D post on the NGS blog and - despite the date - I will not be trying to be funny. It doesn't seem appropriate given that the NGS's Research and Development activity is winding down.

In the original plan, Phase 3 of the NGS would have finished yesterday. Although we have been granted an extension until the end of September 2011 to prepare for Phase 4, we will be spending this time ensuring that the services we have developed work as well as possible.

It does provide an ideal excuse to look back on what we have achieved - and not achieved - over the last 2 years.

NGS3 arrived at around the time when researchers were becoming interested in cloud computing - and we covered both cloudy and more traditionally griddy services from the earlier phases.

The cloudy bit - run from Oxford and Edinburgh - built an Amazon-ish cloud for academic that eventually formed the the NGS Cloud service.

The griddy bit that - run from Leeds - concentrated on improving the underlying authorization, accounting and monitoring upon which the people who use grid in their research rely.

We'll cover each of these in future postings.

I want to use this one to say that building anything with the help of a group of people spread over 5 institutions and 6 sites, split between many projects and who seldom meet in person - was always going to be interesting.

You can reach new levels of miscommunication when your primary way of talking to one another involves email, or phone, or messages posted to a Basecamp instance, or doing Max Headroom impressions in AccessGrid.

Anyone who can survive the experience and produce something deserves credit.

So if Jonathan Churchill, Simon Collins, Cristina Del Cano Novales, Mark Duller, Matt Ford, Robert Frank, Gokop Goteng, Jens Jensen, Mike Jones, Kevin Haines, Shiv Kaushal, Akay Okcun, David Meredith, Ahmed Sajid, James Scott, David Spence, Steve Thorn, Duncan Tooke, Paul Townend, Matteo Turilli or Steve Young are reading this, I would like to take this opportunity to say:

Thank you.

Getting ready for AHM 2011

2011-03-29T15:59:00.002+01:00

Incase you aren't aware the dates for this year's UK e-Science All Hands Meeting 2011 have been announced as the 26th - 29th of September. It will be held at the University of York and a Call for Papers has now been issued.

The main themes will be shared infrastructures, using the cloud in research, end-user engagement and applications (e-science, e-social science, research in the arts and humanities) ensuring an all encompassing meeting with something for everyone.

There are a number of themes to which papers can be submitted:

* Theme 1: Cloud computing for e-Research
* Theme 2: Shared Infrastructures, Systems and Tools for e-Research
* Theme 3: Applications and end-user engagement in e-Research
* Theme 4: Data-intensive Research
* Theme 5: Organisation, Trust, Security and Validation
* General Paper

As in previous years, authors of selected abstracts will be invited to submit full papers after AHM 2011 to be considered for inclusion in a special edition of a journal. The deadline for papers is the 23rd of May 2011.

There is also a call for workshops and tutorials as there is scheduled time for a limited number of half day workshops. The deadline for workshop proposals is 3rd April 2011 - please see the AHM website for further details.

The Problems of Pilots and Pools

2011-03-29T00:00:00.006+01:00

The Grid relies on users having a unique identity, represented by their certificates.

Which is all very well until you actually have to run something. At this point, the certificate must be mapped to a set of local credentials - on a Unix system this will be a username and a set of groups.

There is no reason why your local credentials will be the same on different hosts or even that they will be the same on different worker nodes within the same compute cluster - especially where pool accounts meet pilot jobs.

A compute service cannot support a large virtual organisation by giving every single member his or her own account - especially if the bulk of the members will never come near the service. On practical grounds, it is more common to set aside pools of accounts and hand them out on a first-come-first-serve basis.

Pilot jobs are widely used in the particle physics world. The sole purpose of a pilot job is to find a big enough chunk of compute power and then and only then find something useful to do with it.

Users submit tasks which are kept on a central queue. When a pilot job runs, it will pick a task from the queue, magically become the task's owner and perform the task.

The magic is provided by a program called glExec which itself depends on the LCAS/LCMAPS framework.

In some of its early incarnations, LCAS/LCMAPS was configured so that every worker node has a separate pools of accounts - and the 'real' user of a pilot job usually ended up with one of these per-worker pools accounts.

These days, it is more common for glExec to likely to pass on requests to a central authorization service.

The ARGUS service, currently being tested by Southgrid at Oxford, is the latest generation of central authorization service. Its behaviour and quirks were covered at a presentation by Kashif Mohammed at a recent NGS Surgery.

Kashif's slides describe how the components of the authorization framework can control access for pilot jobs. The same service can also centrally manage the mapping of certificates to local credentials through an Obligation Handler (OH).

This isn't just relevant to pilot-and-pool-pushing particle physicists. glExec is also provides authorization to the CREAM compute element and we plan to use glExec and ARGUS to centrally manage the mapping of credentials when CREAM is deployed in front of as Leeds' ARC1 cluster.

Read all about it!

2011-03-23T11:39:00.003Z

A new edition of the NGS Quarterly newsletter is out now on our website. Browse whilst drinking your cuppa or download and read on the train!

This edition includes a range of articles on topics such as the imminent changes to the NGS (make sure you read this before the 31st of March if you are a user!), using NX rather than GSISSH to securely access GUI's, tips and hints for using our ever popular UI/WMS and much more.

All editions of the newsletter are available from the newsletter section of the NGS website.

In addition to the newsletter, we also have a new user case study up on the NGS website. This time we have a case study on how NGS resources were used to model the climate impact of aircraft emissions. This research was carried out by Laura Wilcox at the University of Reading who ran FORTRAN code on several NGS sites to complete her models. All the details of Laura's research can be found on the case study page.

Are we there yet?

2011-03-18T23:12:00.010Z

I apologise in advance for this weeks post. I'm afraid that the R+D part of the NGS blog never misses an opportunity for a really lame joke and isn't going to change its ways now.

There are many variants on this one...

An unfortunate traveller took the wrong turn and found himself one of those parts of the world where road-signs - and, indeed, roads - are regarded as a modern fad which will never catch on.

After many hours of driving, he finally saw another person, a genuine Local - possibly leaning over a farm gate and chewing a piece of corn - and asked the way back to the big city. Let's say, for this version, that city was London....

`London?', the Local said, raising an eyebrow.

The Traveller nodded.

`London, you say?'

The Traveller nodded again.

`Well,' continued the Local, scratching his chin, `well.... if I were going to London.... I wouldn't start from here.'

Which takes us nicely to the Unified Middleware Development (UMD) Roadmap discussed in a presentation to an NGS Surgery meeting on 16 March.

It is fair to say that - if they had the choice - the UMD developers at the European Middleware Institute wouldn't start from here either.

They have been given the task of unifying four distinct families of grid software - ARC, gLite, UNICORE and dCache - and making the finished product work with other packages such as Globus.

The roadmap is a high level overview: it doesn't go into detail and describes the components and their dependencies using what seem to be UML Deployment Diagrams.

Yet behind the high level plans - there is concrete and practical work being done. There are regular automated builds being performed using CERN's ETICS system and work is underway on the first proper release - dubbed EMI-1.

The long term aim is for the work of the EMI to be adopted and deployed by the European Grid Initiative.

And - maybe - one of these days we will stop talking about the Middleware and start talking about what we can do with it.

NGS at the JISC conference

2011-03-16T12:13:00.002Z

On Monday several of the NGS staff headed over to Liverpool for the annual JISC conference which was held in the rather impressive BT Convention Centre on the river front.
I was there early to get our stand set up and, once our furniture turned up and sticky stuff was obtained for the posters (even though we had been told the poster boards were velcro compatible!), we were ready to go.

It was good to see so many familiar faces at the event even though for us the JISC conference is a bit of an "oddity" as it doesn't contain a huge number of our target audience. We had a chance to catch up with some collaborators from different parts of the country including Northern Ireland and Scotland and of course our JISC funders.

Tuesday morning saw our parallel session take place entitled "Increasing research efficiency through the NGS" with two gues speakers. Our technical director, David Wallom, opened the session with a brief presentation on what the NGS is / does and also covered our Cloud resources which are definitely a hot topic at the moment!

Our guest speakers were Ian Dunlop, University of Manchester, who presented "e-Infrastructure for Social Science data: Obesity e-Lab & MethodBox" and Susana Sansone, University of Oxford, who presented "ISA Infrastructure - Standards and Software for Annotating, Managing and Sharing Life Science Investigations". Copies of both their presentations are available from the "goodie bag" page for the NGS session.

Full details of the JISC annual conference including all the presentations from the event can be found on the JISC conference event page.

Loaded

2011-03-14T10:15:00.008Z

The most popular page on the NGS web site is the load monitor - which shows the current number of running jobs on selected NGS Partner sites as a set of moving, coloured bars.

We don't think this is because the red, yellow and green graphics look pretty. Many of our users have adopted a simple, effective - and low-tech - approach to scheduling jobs: they have a quick look at the load monitor page and do their work on the least loaded machine.

This is the human-powered counterpart of what the WMS bit of the UI/WMS does.

The load monitor is a nice example of how to present the information that is routinely published by a site on a Grid and defined by a GLUE Schema.

GLUE - in an egregious example of acronym abuse - is meant to stand for Grid Laboratory Uniform Environment. The reality is that it is called GLUE because it is what sticks the Grid together.

As any good Grid standard should, GLUE has its own Working Group, GLUE-WG, and proper published formal specifications. The current version is GLUE 2.0 but its predecessor, GLUE 1.3, is more widely deployed.

The load monitor is a visualisation of two pieces of (GLUE 1.3 style) information, presented to the world as

GlueCEStateFreeCPUs
GlueCEStateTotalCPUs

which, if you ignore the CamelCase-naming scheme and the GlueCEState prefix, are fairly self-explanatory. They are published for every compute element.

The pretty dancing bars are generated using jsProgressBar.

This is not new: the load monitor has been running for as long as the current version of the NGS web site and - before then - researchers used and abused a central Ganglia service for very much the same purpose.

It has back as a Research and Development activity because the NGS is changing.

The hard bit isn't the calculation of the system load or the pretty graphics: it is deciding which sites and compute elements should appear.

The current version is intimately entangled with the INCA monitoring service - the list of hosts is extracted from from a configuration file built for INCA.

INCA - as anyone with a high-enough tolerance of tedium to read the NGS R+D blog regularly will know - is being decommissioned as soon as the Nagios service is ready to replace it and we decided - late last year - to stop updating INCA.

This list has is becoming out of date: it includes a number of machines will disappear from the NGS soon and misses many others which should be there.

We are rewriting the load monitor to:

Select compute elements from information sucked from our Single Point of Truth - the GOCDB.
Filter out only those which support the ngs.ac.uk Virtual Organisation using the snappily-named`GlueCEAccessControlBaseRule' attribute defined by GLUE, and published by the sites.

and use it to generate a list of active Compute Elements in sites.

We can calculate the load on each of these compute elements at regular intervals and present it to the world in as pretty, colourful, wobbling bars. It is the Web 2.0-way.

More castles, more discussion but less lightning

2011-03-10T14:04:00.001Z

Following on from my previous post, day two of the CW11 event kicked off with pastries and caffeine to get us ready for the day ahead and more discussion. Once again the agenda was debated and break out session topics were discussed before a presentation by one of the sponsors of the event, DevCSI which has nothing to do with the infamous TV series set in Las Vegas…

I attended the breakout session on the dissemination of “academic” research software results and ended up scribing for the session which was chaired by the NGS Technical Director, David Wallom, who is a hard person to keep up with!

The basis for this session was discussing where software developers could publish their efforts in a peer reviewed, “reference-able” journal. Nature Methods was held up as a great example but in a different field so could we do something similar for software developers so they could contribute to the RAE and improve their standing in the university etc. It was a very interesting discussion with a lot of bridges to cross before anything concrete can happen but watch this space. Hopefully the slides from this session will be available soon.

After coffee we had a lively reporting back session from all the breakout sessions which also included “Research funding 2011: what's available and how to get it?” and “We have data-management plans, should we have software management plans too?”. The slides from these sessions are available now.

After lunch the audience was divided into 3 groups to discuss the same topic – “Ideas for improving Collaborations” and we were asked to report back with our findings after an hour. The discussion group I was in concentrated more on why’s and why not of collaborating rather than the practical means of doing so that I am usually involved in e.g. Access Grid meetings, collaborative technology such as Skype, blogs etc. However it was still a lively and engaging discussion!

This was the last discussion session of the day and all that remained was for Neil Chue Hong to sum up the past 2 days, for many people to be thanked for their great organisations skills (Simon Hettrick!) and for us all to make our way home by plane, train and automobile.

A great two days yet again and I’m already looking forward to the next one. The SSI is certainly going to be busy following up on all the actions from the meeting and I look forward to seeing their outcomes throughout the next 12 months.

Castles, case studies and lightning

2011-03-08T16:16:00.003Z

Last week I attended the Software Sustainability Institutes Collaboration Workshop in Edinburgh. As with previous meetings this event was very enjoyable with a great deal of discussion, actions and collaboration taking part.

The organisers had revamped the programme for this year’s event and one of the changes was the inclusion of “lightning talks”. The set up was interesting with each presenter having 5 minutes and one slide which shared the screen with a rather large countdown clock. It certainly kept the presenters on their toes and the attention of the audience! Unfortunately given the nature of the presentations you really had to be there to appreciate the talks but some presenters did manage to get a lot of info on their slides. In particular the slides from Taverna, STFC, software preservation, soundsoftware.ac.uk, e-science central, SPRINT, Hartree Institute and Youshare.ac.uk are worth a look. I have linked to the websites for each project where they exist but for the slides, visit the materials section.

After the a lovely lunch it was time to start what I think is the most enjoyable part of the workshops – the breakout sessions. Before the event, delegates have the opportunity to submit topics that they would like to discuss at the breakout sessions and then at the event people vote on what topics they would like to discuss. The delegates break into groups and disappear off to various parts of the building to discuss the topic. Each group has a chair, a scribe and someone who “volunteers” to report back but quite often people take on two of these roles.

I attended the session on preparing case studies to support cases for funding which was rather apt as I prepare the NGS user case studies. There were about 7 of us from a variety of backgrounds including funding council reps and the discussion covered areas such as different types of case studies, ways that case studies could contribute to REF, how to format case studies and much more. It was a very interesting hour and we did come up with several suggestions for further work for the SSI.

The second breakout session of the day took place after some caffeine had been consumed and this time I attended the session on “measuring the research impact of software”. Unfortunately the notes from this aren’t up on the SSI website yet but when they are I will make sure I update the link!

So that was day 1 at the CW11. I’m sure you don’t want to hear about the conference dinner so I’ll leave that out and write up day 2 very soon!

Portals, Proxies and making things simpler

2011-03-07T08:03:00.009Z

The NGS provides the services needed centrally to run a Grid. We provide instructions for institutions that wish to contribute to a Grid and resources for researchers who want access.

We have to leave the interesting bit - connecting the services, institutions and researchers in new ways - to others.

One of the people doing the interesting is Mark Hewitt of the Department of Computer Science at the University of York - who was involved in a project to allow non-technical users to link tasks on the grid together in workflows.

In this guest post, Mark describes how they did it...

The P-Grade portal is a web-based portal system developed by SZTAKI - the Computer and Automation Research Institute of the Hungarian Academy of Sciences - in collaboration with the Centre for Parallel Computing at the University of Westminster.

P-Grade provides a generic job submission system using workflows and can submit to any cluster with the standard NGS Globus stack installation.

The White Rose Grid e-Science Centre deployed P-Grade at York for the use of researchers at York, Leeds and Sheffield. Many users find managing certificates complicated, so we decided to use the NGS SARoNGS system as an alternative.

Integration of SARoNGS with P-Grade was quite straightforward. P-Grade must download a proxy certificate from a proxy server before it can start work. SARoNGS creates proxy certificates, generates username and password and provides a mechanism by which these can be passed to a web page. We were able to modify the P-Grade code, to allow a user to download a SARoNGS certificate, instead of having to use the myproxy tool to upload their e-Science digital certificate.

The addition of SARoNGS support to the portal was vital as it completely removed the need for users to go through the process of signing up for a digital certificate and meant that they could authenticate simply through the portal interface within seconds.

Teacher's pet

2011-03-01T14:01:00.002Z

As well as serving the needs of the UK academic research community, the NGS is also involved in helping to train the upcoming generation of new researchers.

NGS resources are used in a number of university courses including those at Cranfield University and the University of Edinburgh. In a new article on the NGS website, David Fergusson who leads training at the NGS, explains how he has used NGS resources in his teaching of the MSc in Distributed Scientific Computing for the last 6 years.

If you would like further information on how grid and cloud computing can be incorporated into your teaching, then contact the NGS helpdesk.

See SPOT run

2011-02-25T14:14:00.008Z

Just when you thought it was safe to go back to the Internet, up pops another Acronym.

Meet SPOT.

You won't - yet - find SPOT in the Grid Acronym Soup because it is not one of ours. It is an escapee from another of the great sources of Acronyms - the world of IT Business Systems.

SPOT stands for Single Point Of Truth.

A SPOT is not a single genuine fact that slipped into a Business Systems sales pitch. Nor does the name imply that Business Systems remind those involved of an inflamed, infected ball of pus. SPOTs are, in general, good.

If you have a SPOT, then you know there is one-and-only-one definitive source for any piece of information - whether it is a price, a name, a salary or an office number.

As anyone who has dealt with a large organisation will appreciate, we do not have as many SPOTs as we should.

On the Grid - which is large, diverse and dispersed by its nature - single points of truth are very hard to find.

Which makes deciding what we should test with Nagios... interesting.

The NCG configuration generator which writes the Nagios configuration needs to know:

What sites to test.
What services to test at those sites.
What tests to run for each service.

Perhaps the closest we have to a SPOT is the Grid Operations Centre Database or GOCDB - which lists every site on the European Grids, their official downtimes and some of the services they provide.

The `some of' is there because the GOCDB defines services in terms of service endpoints - which represent a host within a site acting as, say, a Compute Element or a GSISSH server or a GridFTP server.

There are a comparatively small number of predefined endpoints and these will never cover everything a site can offer - you cannot, for example, advertise an iRODS service.

The GOCDB does not directly provide information about the Virtual Organisations that a service is prepared to support but it should point anyone wanting this information at a site information service willing and able to provide it.

For our first attempt at a WLCG-like Nagios service...

We collect a list of sites come from the GOCDB - we take any site flagged as belonging to NorthGrid, SouthGrid, Scotgrid or the London Tier 2 subgrids within the UK and Ireland Region.
We only test services for which GOCDB service endpoints are defined.
We define the tests for each endpoint within the Perl code of NCG. There is a 'standard' set of tests defined within a perl module called NCG::LocalMetrics::Hash which forms part of the NCG package.
We modified the module to include local changes from a NCG::LocalMetrics::Hash_local module - a change that has been adopted by the NCG maintainers.

As an approach, it works well enough for Nagios tests but not for the friendly-front-end MyEGI.

MyEGI gets its truth from elsewhere: from the the Aggregated Topology Provider (ATP). The ATP is a sort of single point of single points of truth. It swallows data from the Metric Description Database (MDDB) and from Virtual Organisation feeds. It is scarily complicated in places - as you might be able to gather by looking at the MDDB and ATP database schema

The European Grid Initiative exists to make the grid work better, in part by giving us nicer SPOTs and are encouraging development of the ATP and friends. The curious can find out more on the SAM and Nagios wiki pages at CERN.

The Single Point of Truth is Out There....

Ch-ch-ch-changes as David Bowie would say...

2011-02-23T14:10:00.003Z

Hopefully all NGS users are now aware of the forthcoming changes to the NGS service from the end of March 2011. There will be a slight reduction in the number of free-to-use cores available to NGS users and individual cpu allocations will be reduced to reflect this.

If you haven't already checked please read the full announcement which is available on the NGS website. I would also strongly recommend ensuring that you are subscribed to the NGS news list which is the first port of call (along with the website) for all news about the forthcoming changes.

In the meantime if you have any concerns about the changes please contact the NGS helpdesk. We are there to help and to answer any queries you may have about how the changes will affect you.

Leaving and Joining

2011-02-17T23:23:00.006Z

Most NGS users should now be aware of the major changes due at the end of March 2011 - when some of the machines providing free-to-use CPU time will be retired.

The NGS cluster at Leeds - ngs.leeds.ac.uk - is among those that have reached the end of their useful life. It will be removed from service on 31 March 2011.

This does not mean that Leeds as a site is dropping off the Grid forever. We will be back...

The NGS cluster is one of a number of systems managed by the Research Computing group within the Central IT Service at Leeds. It is also - by modern standards - one of the smallest. In compute terms, it is dwarfed by the one in the room next door - which boasts 4000 CPUs, 7 Tb of memory and 100 Tb of fast disk - and goes by the name of ARC1.

ARC1 is so big because it is two computer clusters rolled into one.

Around half the cluster was funded by the University for use by local researchers - for whom applications such as DL_POLY, AMBER and CASTEP have been are installed.

The rest comes from a UK-wide consortium of Solar Physicists - so there is a need for people from outside Leeds to safely and securely use the service. Cross site access is why the National Grid Service exists. We can do that.

While primarily aimed at the Sun spotters - Leeds has kindly offered some CPU time on ARC1 to the NGS. If, that is, the NGS can get ARC1 onto the Grid.

If we can... applications installed locally will be made available to external users - where licenses permit. The users were are expecting are those who currently access resources via the UI/WMS.

So what do we need...?

A standard way to presenting the applications to the world. We can do that... it is why the Uniform Execution Environment was invented.
A way of limiting access to licensed applications to the right people. We can do that too.
A means of accepting requests from the UI/WMS. Taking the lead from the particle physics community - we are looking at CREAM as deployed by gLite.
An information service that lets people outside see the state of the system and the applications available - the obvious choice here is the BDII, widely deployed and also available from gLite.
A way of accounting for use in APEL. We may send data directly or indirectly via the NGS accounting service and RUS records

Whatever we build, will need to work with what is now called Oracle Grid Engine - the local batch management system - and within a highly specialised and customised Linux environment that is significantly different to that used by the World Wide LHC Grid.

As a first stage, we are going to deploy a separate (virtual) machine - running Scientific Linux and using packages from gLite - to act as the link between ARC1 and the grid.

We want to produce - and make available to others - a kickstart configuration and associated scripts that automate as much of the installation as possible.

It will certain produce a blog post or two.

Leeds has been involved in the NGS for a long time: we have hosted two generations of NGS cluster; been involved in the NGS Outreach, Research and Development and Monitoring activities and created automatic installation systems for deploying and configuring Grid software.

We've had a lot of practice. It certainly hasn't made us perfect but, if you will excuse what sounds like Yorkshire pride coming from someone who is technically an Essex Boy - if anyone can do it- Leeds can.

Are you a researcher looking for a partner?

2011-02-15T13:40:00.004Z

If so then you really should consider coming to the dating event of the year! Well okay it’s not quite a dating event but matchmaking is definitely part of the plan of the Software Sustainability Institutes Collaboration Workshop.

It will be held in Edinburgh at the beginning of March and is described as “the perfect opportunity to meet the researchers, software developers, funders and other software experts that can help you advance your research”.

If you are a researcher who uses software but could do with some help from a computer scientist to make the most of your software, come along and advertise yourself and your situation / problem. You can give a lightning talk or present a poster and then work through the issue with other attendees in small break out groups.

There have already been some break out group discussion topics suggested BUT one of the best things about this meeting is that the agenda constantly changes depending on the interests of the attendees, the emergence of new discussion threads and tangents. There’s never a dull moment!

I’ve attended several of these meetings and I’ve thoroughly enjoyed every single one. No death by PowerPoint and plenty of discussion and positive solutions instead! The constantly changing agenda works really well as we report back to the assembled attendees after every session and then decide on the next set of discussion topics. I’ve always met new and interesting people and came away fully enthused.

If you are a developer there are some free places up for grabs but the deadline is this Friday (18th Feb) so don’t delay!

Missing the message bus

2011-02-11T20:32:00.010Z

[With thanks to Konstantin Skaburskas.]

Two weeks ago, we were very nearly at the point where we could deploy WLCG Nagios and phase out our existing testing service.

We had created our own tests and worked out how to add them to the bundle of tests that are sent out onto the Grid.

The tests were actually being run on remote sites.

All that was missing was - well - a big chunk of the test results.

When the tests landed on gLite-based sites - everything worked as expected. In other places - the tests ran... but resolutely refused to let anyone know the results.

We have now found the missing messages - after losing them twice on the way.

It was all due to subtle differences in the environment variables defined at a site. The fix is to set two environment variables by adding something like...


 Environment = {
   "OSG_HOSTNAME=<jdlreqceinfohostname>",
   "LCG_GFAL_INFOSYS=bdii.ngs.ac.uk:2170"
 };

to the template used to generate the JDL file that describes the test.

To understand why, you need to understand how the tests on remote hosts are run. The hard work is done by a script called nagrun.sh - that:

unpacks the bundle of tests and configures them for the local machine.
runs them using a bundled copy of nagios
translates the test results into messages.
sends the messages to a message broker - which shoves them on the message bus back to the Nagios server.

At the Nagios server end, each message is unpacked and fed to the central Nagios as a passive test result.

If LCG_GFAL_INFOSYS is missing, the tests never make it to the message broker; if OSG_HOSTNAME is missing - they are ignored when they reach the Nagios server.

It is the WLCG Nagios for a reason - it was designed to test machines that sat within the Worldwide LHC Computing Grid. One of its roles was to serve as a replacement for the older 'Service Availability Monitoring' (SAM) tests.

It makes the - perfectly logical - assumption that the environment on the machine running the tests will be like that used for the SAM tests.

WLCG has a dedicated network of message brokers. Any host can find a suitable broker to contact by asking its friendly local information service. The environment variable LCG_GFAL_INFOSYS points to the information service.

Some of the sites we are testing sit outside WLCG. We have our own message broker and pass additional information with the tests to direct messages to it.

A subtle bug, which is been fixed in the current release, meant that even though LCG_GFAL_INFOSYS was not being used, it still had to be set. If it wasn't, nagrun.sh could not find a message broker to contact.

So, the messages were making it back to the Nagios server. The Nagios server was ignoring them.

The reason: the messages sent back are meant to include a reference to the Compute Element (CE) that actually accepted the job. The messages we were sending were all being marked as coming from 'localhost.localdomain' - a dummy name used internally by the nagios tests.

The nagrun.sh script tries to work out the CE name from the local environment and from the output of certain scripts. If all else fails, it assumes nagios knows the answer.

This WLCG Nagios developers had encountered this problem before - when running ATLAS tests against hosts on the US Open Science Grid - and had added code that allows a Open Science Grid hostname to be used as a CE name. It expects the environment variable OSG_HOSTNAME to hold that hostname.

We can also report that the WMS administrators have reconfigured the server so it no longer gets clogging up with CREAM jobs - and the CREAM CE tests are now running via the WMS - as WLCG intended.

We are now ready to deploy WLCG Nagios - unfortunately without the MyEGI friendly front end - and make it available to site administrators.

We will describe how we decide which sites to test and what tests to run in a future posting.

At which point, Nagios related Research and Development will take a break and I will have to find something else to prattle about every couple of weeks.

It's not about us, it's about you

2011-02-08T12:18:00.003Z

As Jason mentioned in his blog post last week, we have a number of NGS user case studies on our website and the number is growing.

The user case studies are a collaboration between the user and myself as the NGS Liaison Officer. I work with the user to produce a short case study outlining their research, how they use the NGS and (rather importantly) the benefits that the NGS has brought them.

I also work with users to make their research more accessible to the wider NGS community which can be pretty difficult given the very specialised nature of some of the research performed on the NGS!

Most of the user case studies come about through users volunteering to write a case study, mainly through our annual user survey. For example this year I had 31 users who volunteered to produce a user case study - a rather large increase in last years number!

I hope the increase has come about through people realising the benefits of advertising their research to the wider community and not just so they can tick a box on their final project report! I have found that the most enthusiastic communicators tend to be PhD students which bodes well for the future of science communication.

The user case studies don't just stay on our website. With over 2000 hits since they were placed on the website, they are picked up by other dissemination teams UK and Europe wide. Quite a few of our case studies have been picked up by iSGTW which has 7300 subscribers and many more unique visitors to their website (over 125,000 at the last count!). EGI have also became interested in our case studies and are looking to produce something similar for their own project.

And it's not just within e-science / grid / e-infrastructure organisations that the case studies are picked up. Hardly an issue of Scientific Computing World goes by without me getting the NGS mentioned in there courtesy of one of our user's research being featured! I am often asked by editors if I know of anyone researching X or Y and, thanks in no small part to the user case studies and the NGS Communities service, I can usually track someone down! SCW is free to read online and you can subscribe to a free printed copy as well.

The latest NGS user case study has recently been put on our website - Simulating carbon nanotubules on the NGS by Rebeca Garcia Fandino at the University of Oxford. Watch this space for more new case studies coming soon!

While interesting things happen... elsewhere.

2011-02-04T22:46:00.015Z

If you look through the case studies on the NGS web site - you can see how the grid has helped researchers study stars, and genes, and criminal activity; to understand how the brain, heart and nervous system work, to search for new drugs to treat disease and study how existing drugs are absorbed by the body. That isn't even mentioning the original goal of the grid - wrangling the enormous amount of data produced when bashing hadrons together in a big pipe under Switzerland.

There is a huge amount of interesting and important research out there.

And a huge amount of very dull - but important - work to be done to enable this research.

Someone needs to make sure the infrastructure is running and keep track of what was used and when. Someone - actually Jens Jensen, one of the contributors to this blog - has to worry about how we look after private keys.

NGS Research and Development's latest excursion to the outer limits of tedium comes courtesy of a project to feed data from our RUS accounting service directly into the EGI's APEL accounting service.

We are already combining data recorded by APEL with that recorded by RUS when updating the CPU usage for account holders within the NGS User Account Service.

In future, we want APEL to be the only place to look for account records and in the great Grid Tradition, this is where things get complicated....

RUS is standards-based. We use the Open Grid Forum Usage Record (UR) format - as defined in great detail in an official specification - to transfer accounting information from individual compute resources to a central store.

UR records can identify the resource which actually did the computing and how much CPU time, or wallclock time or memory was used. It does not carry any information about the relative speed of that resource compared to any other random computer on The Internet.

The NGS accounting clients, and Grid-SAFE all generate UR format data.

APEL is more pragmatic and - as it is part of the gLite software stack - much more widely deployed. It does incorporate a measure (typically derived from the SPECMark) of the speed of the CPU that did the work. It is not explicitly tied to a resource.

So - we are creating a tool that takes records sent to RUS and translates them into APEL updates. It will have to fill in the missing scaling factor - if necessary falling back to a 'custom' scaling to warn potential users of the data that we have no idea how fast the computer is.

These updates will be fed to the central APEL service in exactly the same way as those generated by any other APEL client - and eventually find their way back to the User Account Service.

And the users of the service - well, they don't have to care in the slightest... they have far more interesting things to do.

NGS roadshow event at the University of Huddersfield

2011-02-03T15:50:00.004Z

Last week saw the NGS outreach team at the University of Huddersfield for a NGS roadshow event. The local organiser was Ibad Kureshi who did a fantastic job of organising everything locally and making the day went smoothly. Ibad also did a great job of advertising the event locally meaning that we had over 30 participants at the morning roadshow and over 20 registered for the afternoon training event.

The roadshow kicked off in the usual fashion with an overview of the NGS from our Technical Director, David Wallom from the University of Oxford. Many people in the audience were not familiar with the NGS before the event or had just heard our name mentioned but were unaware of what the NGS actually does so this presentation is designed as a gentle introduction!

Following David we moved onto presentations from NGS users including Paul Martin who is a researcher at the University of Huddersfield. Paul has been using the NGS for computer modelling of thoria in order to determine its suitability for a next generation nuclear fuel. He explained how he has used DL_POLY 2 which scales very well on the NGS. Paul in particular praised the good on-line instructions/training, FAQ’s, blog and helpdesk/support which is great to hear!

Paul was followed by another user, Matt Smith from the University of Liverpool. Ibad had suggested that a presentation by an Abaqus user would go down very well due to the interest in this software package at Huddersfield. We were happy to oblige! Matt spoke about his use of Abaqus on the NGS to model lattice structures. This was a very interesting talk helped by the sample materials Matt brought along to illustrate his presentation. As Ibad had thought, Matt’s presentation prompted plenty of questions about his usage of the NGS and about his research.

David Fergusson from the training team at NeSC then elaborated on how users actually get started on the NGS and how to apply for a certificate, run jobs etc. Again there were plenty of questions showing that Huddersfield researchers are keen to use our resources!

The final presentation was by Ibad who outlined the locally available resources and how these tied in with the resources offered by the NGS. We’ve found in previous roadshows that having presentations about local resources helps the audience to visualise how the NGS fits in locally and that it is there to complement existing resources.

All the presentations from the event are available on the NGS website from the event page.

Private keys

2011-02-02T09:13:00.004Z

The point about public key cryptography is that public keys are public: they are used to prove possession of a secret (namely, the private key) without revealing any information about secret. This is called a Zero Knowledge proof. In other words, much of the level of assurance in the infrastructure rests on management (not just protection) of private keys.

So this is why you have all generated your keys and protected them with strong passphrases.

Recently there has been some discussion among the grid CAs as to whether the rules can be relaxed, without lowering the level of assurance too much. Many people generate their keys on systems which are maintained by someone else: e.g. your desktop at work, or maybe even a UI.

This leads to the proposed loosening of the rules, or perhaps a better description of existing practices.

It is likely that in the future, we will support private keys generated by:

users themselves, on trusted systems (eg your own machine, or your desktop machine at work)
institutions, letting them pre-generate keys for their users (apparently some like to do this);
third parties: e.g. running a credential repository like MyProxy.

However, as with much change, it is easy to introduce new rules without fully understanding the problem. There are serious (but fixable) problems with the draft rules. For example, different CAs interpret "third parties" in different ways. Is the CA a third party? I would have thought not. Would the NGS count as a third party, despite the fact that it runs a CA? Probably.

Anyway; the upshot of this is that private key protection rules will be relaxed. What is currently missing is the in-depth understanding of the security aspects of the lifecycle of the private key. I have soapboxed on this topic before. More on this later. Stay tuned.

Adding NGS tests to WLCG Nagios

2011-01-28T00:35:00.003Z

When we last mentioned the NGS project to deploy WLCG Nagios, we had most of the basic WLCG tests running against 'classic' and 'CREAM' compute elements.

We are now extending the WLCG code with some NGS-specific tests.

In particular, we are adding to the set of tests that are run on individual worker nodes as part of the 'CE', and eventually the 'CREAM-CE' tests.

This is not exactly a common requirement, so documentation is understandably sparse. The best place to start seems to be https://twiki.cern.ch/twiki/bin/view/LCG/PracticalHintsForMigrating2Nagios

The test we will use to test the testing service is deliberately simple. It is a Nagios-style plugin that checks if a site supports the 'Uniform Execution Environment' conventions. It looks for a /usr/ngs directory. If it is missing, this is an error, if it is empty, this warrants a warning, otherwise everything is OK.

We know that WLCG-Nagios uses a mixture of active and passive tests. Active tests deliver results immediately while the results of passive tests filter in slowly via the message broker.

Our initial plan was to extend the CE-probe tests. The CE-probe works by...

building a compressed tar file containing some nagios tests, a copy of nagios to run them, and bits of python to deliver the results to the message broker.
generating a JDL that describes how to fetch the tar file and run the tests within it.

The key is a script called nagrun.sh which runs nagios on a remote machine, collects the test results and throws them at a message broker. The broker should deliver them to main nagios server where they reappear as passive test results.

The CE-probe allows additional directory trees to be added to the tar file, as long as they look rather like...


/usr/libexec/grid-monitoring/probes
|
`-- uk.ac.ngs
   `-- wnjob
       |-- uk.ac.ngs
       |   |-- etc
       |   |   `-- wn.d
       |   |       `-- uk.ac.ngs
       |   |           |-- commands.cfg
       |   |           `-- services.cfg
       |   `-- probes
       |       `-- uk.ac.ngs
       |           `-- WN-uee
       `-- uk.ac.ngs.gridJob.jdl.template

This is mostly directories and subdirectories. Real files are marked in bold: WN-uee is the test script, the *.cfg files are nagios configuration files describing how to run it; the *.jdl.template file is used when writing the JDL.

Eagle-eyed readers may have noticed lots of uk.ac.ngs's scattered around.

This serves as a convenient namespace - it exists to stop files in this directory tree inadvertently overwriting those from another tree when the tar file is being created.
The convention used in WLCG Nagios is that the namespace should be your organisation written backwards. Argue not will I.

Incorporating the new directories involves adding extra arguments to the CE-probe

--add-wntar-nag-nosamcfg
--add-wntar-nag /usr/libexec/grid-monitoring/probes/uk.ac.ngs/wnjob/uk.ac.ngs
--jdl-templ /usr/libexec/grid-monitoring/probes/uk.ac.ngs/wnjob/uk.ac.ngs.gridJob.jdl.template

The first of these turns off the standard WLCG 'SAM' tests. GridPP nagios service is already checking those.

At the time this blog post was being written, a grand total of one site has passed - congratulations Glasgow Scotgrid - and it flagged up a few sites that do not provide UEE application.

While we can claim one successful success and several successful failures, there are a lot of sites where the results have yet to arrive.

These laggards include all the old core NGS sites - all of which support UEE, but use Virtual Data Toolkit rather than gLite for grid software. We have tested the test-test on one of these sites and know it works. The next step to to find out why the results are getting lost on the way home.

Fixing the mangled XML in the SARoNGS service

2011-01-27T23:54:00.006Z

A little technical note following from last weeks posting on SARoNGS.

If you are averse to perl, XML and regular expressions, look away now. There will be a proper R+D blog post along shortly.

Someone at a recent NGS Surgery asked for the gory technical details of how we turned the corrupted XML that was breaking the SARoNGS service into something that once-again matched its cryptographic signature.

The SARoNGS web front-end at https://cts.ngs.ac.uk is written in Perl. It relies on Shibboleth to obtain user attributes from identity providers, encode them in Base64 and deliver them via a custom http header called 'Shib-Attributes'.

The Apache web server will eventually present this to our perl code in an environment variable called HTTP_SHIB_ATTRIBUTES.

We realised that, under some circumstances, an additional set of xmlns:xs and xmlns:xsi namespace declarations were being added to the <samlp:response .. > XML tag generated by newer versions of the Shibboleth idP.

These were always inserted at the end of the tag, before the final '%gt;' and just after the responseid attribute. Removing them meant...

turning the base64 encoded data back into XML,
using a perl regular expression to remove the cruft and restore the XML to canonical form
turning the correctly canonicalised XML back into Base64.

or in perl...

use MIME::Base64;
my $encodedData = $ENV{HTTP_SHIB_ATTRIBUTES};

...

my $shibAttrXML=MIME::Base64::decode_base64($encodedData);
for ($shibAttrXML) {
 s{(<saml1p:response.*?responseid="_[0-9a-f]+")(.*?)(>)}{$1$3}m;
};
my $encodedDataCanonical=MIME::Base64::encode_base64($shibAttrXML,'');

It's a workaround, not a fix, but it is a workaround that works.

Taverna and Cloud on the NGS website

2011-01-25T12:27:00.001Z

Incase you missed it (and we wouldn’t want that now!), a quick heads up of some new articles on the NGS website.

First up is an article on running Taverna workflows on the NGS. Mike Jones (NGS) and Donal Fellow (RCS) from the University of Manchester have been tackling this issue and are making rapid progress. The success of this project could result in an increase in NGS usage by the large Taverna user community.

The second article is an update on the NGS cloud prototype which has been running since last summer when we asked for users willing to try this new service. We were inundated with volunteers and this article by David Fergusson (NGS, NeSC) looks at the user communities taking advantage of this prototype service.

Single Sign On - The Movie

2011-01-20T11:13:00.012Z

Ladies and Gentlemen, take a seat, grab some popcorn and don't forget to turn off your mobile phone - because the NGS is going to the movies....

In our feature presentation, we join an intrepid explorer as he connects to the Grid using only his institutional credentials and some slightly-annoying background music.

At 3 minutes and 44 seconds, it is considerably shorter than Avatar, but if you can't wait (spoiler alert!), the plot is:

Our hero visits the NGS SARoNGS service.
He authenticates himself using Shibboleth and his institutional username and password.
He clicks a button or two and is rewarded with credentials that allow him ssh command line access to a grid enabled machine.
And they all live happily every after.

In the future as seen by Project Moonshot, we will be able to use institutional credentials anywhere. We can already make it most of the way using existing technology - a sort of Project Apollo 13?

The Making Of...

No modern movie is complete without a 'The Making Of...' documentary to fill those extra bytes at the end of the DVD. So we will also let you see behind the movie magic...

When you click on 'Login' on the 'SARoNGS' service provider - https://cts.ngs.ac.uk - your web browser does the Shibboleth Shuffle: passing you via the 'Where Are Your From' (WAYF) service to your home institutions 'Identity Provider' (idP) and then back to http://cts.ngs.ac.uk.

In the last step of the shuffle, a blob of XML is delivered that means `we at the University of Nether Wallop do solemnly swear that this is one of our users'.

Now that it knows that you are a reasonable member of society, the SARoNGS service and your local Identify Provider immediately start talking about you behind your back. In the chatter, your Identity Provider passes on one or more Shibboleth Attributes that describe who you are and what you do.

Shibboleth Attributes can be nearly-anonymous or as personal as names, email addresses or even photos so the UK Access Management Federation has strong recommendations for what can be revealed. Unless legal agreements are in place, an idP only need reveal your unique pseudonymous identifier and your role.

The Shibboleth Assertions are passed from the cts.ngs.ac.uk to a separate authentication service based around a modified MyProxy server.

The authentication service only cares about the unique pseudonymous identifier - or eduPersonTargetedId - and creates and manages short-lived certificates on its behalf. These certificates have a distinguished name that looks like

/DC=uk/DC=ac/DC=ngs/DC=sarongs/CN=(a very long string of hexadecimal digits)

The very long string of hexadecimal numbers is a cryptographic hash of the eduPersonTargetedId.

The authorisation service sends the certificate back to cts.ngs.ac.uk where it is associated with one or more Virtual Organisations (VO).

The default VO, 'ukfederation.org.uk', represents anyone from an institution within the UK Access Management Federation. You can also sign up for an NGS account with a SARoNGS credential, at which point you will be eligible for membership of the 'ngs.ac.uk' VO.

The certificate and VO information is stored on the NGS's official MyProxy server myproxy.ngs.ac.uk under a unique username and a random password.

The SARoNGS service has done its duty. Now the MyProxy enabled Gsisshd (MEG) takes over.

MEG allows an ordinary ssh client to be used to access a grid-enabled service. It accepts a username, a myproxy server and a password - uses these to download a (proxy) certificate and uses that certificate to authenticate you.

ngs.leeds.ac.uk has a version of MEG running on port 2223. We have made some changes - described in technical detail in the bonfire-night R+D posting - to allow certificates with only ukfederation.org.uk membership to log on without being given full command line acccess.

The Out-takes...

The MEG service at Leeds has been running, and accepting SARoNGS and ukfederation.org.uk certificates, since early December 2010.

We have kept quiet about it not because we are naturally modest and unassuming, but because we would have looked like a bunch of bumbling idiots.

There were some places where the SARoNGS service resolutely refused to work. If you were based at one of the unfortunate institutions and tried to reproduce what you saw in the movie, you would have got to the end of the Shibboleth Shuffle and been rudely informed that:

MyProxy didn't like me

We have known why MyProxy is being so unfriendly since November. The XML representing the Shibboleth Attributes is digitally signed and, at some point on its journey, it is corrupted so the signature is no longer invalid.

The fault seemed independent on the version of the idP software deployed but did depend on which attributes were released.

Earlier this week, we worked out why.

It is very subtle, very Shibboleth and another magnificent example of XML biting back.

Before it is signed at the idP the XML the Shibboleth Assertions is first converted to a canonical form, a process that needs to take XML namespaces into account

When the attributes were reconstituted on cts.ngs.ac.uk ready to be passed to the authorization service, additional namespace declarations were inserted, scrambling the signature.

We are still not clear where or why this happens. It might be related to typos in the Shibboleth configuration which left certain Attributes missing a default XML namespace.

The typos are fixed in version 2.2.1 of the idP and, thanks to NeSC Glasgow, we can confirm that this version can send all the attributes it wants with no repercussions.

Working around the problem was trivial. The additional declarations always appeared in the same place - at the very end of a saml1p:Response tag - so we simply removed them again.

The Embarrassing thank-you speech...

Like an Oscar winner, we have a large number of people to thank for their contributions.

These include the people at the NGS partner sites at RAL and Manchester and those people at Glasgow, UCL and Sussex that helped identify and debug the SARoNGS problems.

We would particularly like to thank John Watt from what used to be NeSC Glasgow for taking the time at last weeks NeISS meeting to help generate test cases.

The inevitable sequel...?

SARoNGS is built around an elderly and currently unsupported versions of Shibboleth and Myproxy.

The web user interface is seen as confusing by less experienced users.

If it is to continue running, it will need further development.

SARoNGS is unique in that it make the Grid available to people who cannot or will not use browser-based certificates - and that makes it the real star of the movie.

Last chance for the NGS user survey

2011-01-18T13:36:00.003Z

This is it. The closing date for the NGS user survey is nigh - well this Monday anyway!

We've had a great response to our survey this year and I would like to really thank everyone for completing the form and giving us all your feedback. However I would like a couple more users to complete the survey just to make this one our best ever.

Once the survey closes on Monday I then have the enviable task of taking all your feedback and putting together a report which will be made available on the NGS website summarising all your responses and feedback. The report will also be used to feed into our bid for the continuation of the NGS which we will soon be formulating.

So if you would like your views and opinions to be incorporated into the user survey report and the next phase of the NGS, be sure to complete the user survey before the end of Monday 24th January!

PS and don't forget every completed survey which has an email address included is entered into a prize draw for one of THREE Amazon vouchers!

A different kind of social network

2011-01-14T15:39:00.017Z

Those of us involved in NGS support regularly email our users, sometimes phone them and occasionally see them pixelated in a window of an AccessGrid session - but we seldom get a chance to see what they do for a living.

When we do, it is a welcome reminder that we do the dull-but-useful stuff so others can do new and interesting research.

The latest welcome reminder came courtesy of the National e-Infrastructure for Social Simulation (NeISS) who met in Leeds earlier this week. The local NGS support took the opportunity to come along and listen.

NeISS has been described by as an attempt to build a real life version of`SimCity - with an emphasis changed making life better for real communities rather than tidying-up Tokyo after a visit from Godzilla.

NeISS researchers study the behaviour of people. Their approaches range from statistical analysis of census data to the use of agent-based modelling: following virtual people as they go about their virtual lives, travel to virtual work, have virtual children and eventually virtually die.

The researchers involved have been long-time users of the NGS. Visit the case-studies section of the website and you will see find how NGS resources were used in social simulations to estimate how the population of an area changes over time and study patterns of criminal behaviour

This is proper e-Research - an interdisciplinary collaboration between 8 UK institutions made possible by modern technology and fast computer networks.

They have a website and a software repository; they use applications like Taverna to automate data processing and publish the workflows they create to MyExperiment.

And they want to make their work available to those who plan and run our communities - or simply live in them.

They are developing web-based portals from which simulations can be launched. The idea is that a city planner would be able to log on and see the consequences of, say, building new houses on local traffic and schools.

Simulations are computationally expensive. The computer hardware that hosts the portals lacks the computational oomph to run the simulations so this work need to be offloaded to more powerful computers elsewhere and the data passed around.

It all very grid... or very cloud... or very something-as-a-service.

That is what we are here for and we are doing what we can to help.

NeISS are using resources across the 8 institutions and using or evaluating NGS services including the workload management service to distribute jobs; the SARoNGS service as a means of authentication, and the cloud prototypes for development.

We are working with them to understand how we can improve our services, so that they can use them and they can get on with that whole `making the world a better place' business.

Got a problem with your software?

2011-01-11T13:03:00.002Z

If so then we know who can help! Todays blog post is courtesy of Simon Hettrick from the Software Sustainability Institute (SSI).

In 2010, a crack developer was asked to join the SSI. This man promptly set up as the Institute’s software architect. Today, still wanted by a number of projects, Steve survives as a developer of fortune. If you have a problem, if no one else can help, and if you can email him, maybe you can Ask Steve!

At the SSI, whenever we have a software problem we simply ask Steve. He’s our in-house software architect and all-round guru of code. Then we got to thinking: it’s selfish to keep such a valuable resource to ourselves, we should make Steve’s knowledge available to everyone. And that’s when the idea for the Ask Steve! blog was born.

So what is Ask Steve? The idea is that anyone can email Ask Steve! with a software trouble and query. Each week or so, Steve will work on a problem and post his answer to the blog. People can comment, try out the solution or simply get back to Steve with another question. Steve will sort through the questions he is posed and answer the ones that trouble the most people.
Next time you have a software problem, visit the Ask Steve! blog.