A smattering of updates

Over conference season there was what can only be described as a smattering of small announcements that you may have missed during your travels.  The blog seems like an ideal place to gather these together!

NGS News
The latest edition of our quarterly newsletter was released in time for conference season.  Available to download from our website, September’s edition contains articles on -

• A round up of news from Europe including the release of the first Unified Middleware Distribution
• An introduction to the new Certificate Wizard
• Championing e-Research and e-infrastructure - the Campus and Community Champions
• NGS user case study - Using the NGS to model the climate impact of aircraft emissions
• ...and more!

Tell us what you think!
Also on the website we have a new poll on the home page.  This time we are asking people how easy it is to find the information they are looking for on the NGS website.  So no matter if you are a frequent or occasional visitor to the NGS website, let us know by voting in our poll.  It can be found on the right hand side of the homepage.

 
Busy users
There is a new NGS user case study on the website.  Maria Holstensson from the Institute of Cancer Research explains how she is using the NGS to optimise cancer treatment for children suffering neuroblastoma.

Children with neuroblastoma who are being treated with targeted radionuclide therapy can have their treatment monitored with gamma camera images. These images are used to calculate the amount of drug taken up by the tumour and to estimate the radiation dose. However the image quality can be poor due to scattering and interference. Maria Holstensson from the Institute of Cancer Research is looking at tackling these problems.

Maria said "We have had absolutely fantastic help from the NGS and as a result of using the Grid we have been able to run multiple parallel simulations that we would not have been able to run otherwise".

 
How many?!
We’re pleased to announce that we now have over 1000 subscribers our fortnightly NGS news bulletin.  We have subscribers from all over the UK and much further afield with 21 countries represented amongst our subscribers. 
The news bulletin is delivered to your inbox every second Friday (with some exceptions during conference and holiday season!) and contains news from the NGS, updates from our member sites, details of forthcoming relevant conferences, calls for papers for relevant journals and much more.  You can subscribe to the mailing list from the JISCmail site.

Lounging in Lyon and yawning in York

Oh I wish! 

The September conference season was hectic as always for the NGS team with the EGI Technical Forum (Lyon) and the UK e-Science All Hands Meeting (York) back to back.  There was definitely no time for lounging in the Lyon sunshine although we may have yawned on the train home from York as the conference travel came to an end.

The EGI meeting in Lyon was a great success as always with over 600 attendees.  The meeting is an excellent opportunity for the NGS, in it’s role as the UK National Grid Initiative (NGI), to meet up with other NGI’s from all over Europe.  As well as looking after and organising the UK NGI exhibition stand in conjunction with GridPP, I was also involved in a session in my role as NGS Liaison Officer.

The NGS has held successful roadshow events for several years and these have caught the eye of EGI who are looking at doing something similar through the NGI’s.  I was asked to take part in the EGI / NGI roadshow session and present on my experiences of organising and holding roadshows and measuring the feedback and impact of these events.

There was an interesting discussion after the presentations regarding what the NGI’s would need to host these events and what materials they would find useful.  In some cases more staff and more time would be very helpful but the ability for the EGI to provide these resources are somewhat limited!  I was also asked about practical points such as organising registration and finding the right people in institutions to help host the events.  Hopefully the discussion minutes will be made available at some point.  Some more discussion points were captured on the GridCast blog entry on the session.

The AHM meeting at York attracted 150 people this year and seemed dominated by one word – Cloud!  I lost count of the number of cloud sessions taking place over the 4 days.  The meeting was stimulating and thought provoking judging by the copious amount of notes I took.  However one of the main activities for the NGS was the SeIUCCR organised workshop – Meet the Champions.

The purpose of this workshop was to give attendees an opportunity to meet the researchers that have been promoting and championing research in different e-Science areas and to find out about their work and how they utilise e-infrastructure.  The guest speaker was Scott Lathrop from the XSEDE project where he is the Director for Education, Outreach and Training.  Scott talked about their Campus Champion programme and how they ensure that Campus Champions feel involved and part of the project.  They have many of the same responsibilities as our Campus Champions including raising awareness of XSEDE and even providing training in using the resources. 

Two of our Community Champions also presented at this event outlining the issues in their research areas regarding e-infrastructure, getting started, having the right support at their institution etc.  There was also a very lively discussion panel at the end of the session with many issues raised including how scientific researchers work and the risks involved in devoting time to starting to use new technologies.

It’s great to hear researchers and users speaking about their experiences of actually using this technology, as it sometimes seems, that users get forgotten about in all the discussion about standards and programming.  We need a reminder that at the end of the day this is about building tools and providing a service that people will want to use and find beneficial and that will help further their research.  It’s most definitely not a case of “build it and they shall come”!

I’ll make sure that all the presentations from the AHM workshop are available on the NGS website soon so watch this space!

A long long time ago... or so it seems

Before the mayhem of the September conference season was upon us, the NGS ran a successful summer school at the beginning of the month.  The e-infrastructure summer school was part of the SeIUCCR project.  You can read more about the background to the summer school and project in my blog post from the beginning of August.

We had over 25 students join us down in Coesner's House in Abingdon for 4 days and they came from a wide variety of backgrounds.  We had students who were in their first year or two of their PhD as well as post docs and they came from Edinburgh to Essex and everywhere inbetween.

So how was the summer school?  Was it a success?  Did the students learn and enjoy it?

Well the best people to ask are the students themselves.  All the students were asked to provide us with feedback and some were even willing to write a few more words.

Ed Day from Canterbury Christ Church University attended the summer school and this is what he had to say:

"The recent SeIUCCR Summer School was a very enjoyable and informative experience. As a newcomer to grid research I found the the summer school extemely useful. It contained some important introductory pieces as well as covering many topics in more depth suitable for anyone wishing to use the NGS. Sessions consisted of a good mix of high level overviews and hands-on practicals such as using the P-GRADE portal.

The presenters were very knowledgable and helpful and were eager to inform on all aspects of grid computing. Some sessions involved the speakers talking individually with attendees and I found the staff gave useful advice and were very supportive of my project.

Before the school I thought my particular research area, the forensic investigation of mobile phones, might be a good fit for grid computing, and by the end of the summer school I felt much more able to pursue my research in an informed manner using the NGS. It was useful to know how grid, cloud and high performance computing relate to one another  and over the four days I feel I became much more aware, in a less naive way, of how my research would benefit from the resources the NGS has to offer. In particular I learnt to think about my research differently: how my phone investigation process would be best able to benefit from a grid architecture.

Indeed the school helped support my view that the NGS would be a good resource for ANY project that needs HPC not just vast number crunching modelling applications such as those such by molecular biologists or quantum level physicists (although of course it is good for that too).

I liked the food too!"

And it wasn't just Ed who enjoyed the summer school!

"The summer school has been a fascinating activity. The hands on sessions have familiarized us with HPC/Grid/Cloud, which are useful resourses that I have never had access to or known how to access, whilst the Meeting Champions and Q&A sessions made it possible for us to know how these resourses could facilitate our research. The SeiUCCR summer school provides a great opportunity of learning, communicating and networking. I would like to thank all the people who made this summer school possible"

"The summer school was a relaxed and friendly environment. It provided detailed information about the various resources available to researchers both in terms of computer resources and support. The staff where very approachable and keen to show an interest in the attendees work. Overall it was a great experience I would recommend to anyone who has an interest in grid/cloud/HPC or who's work may benefit from such technologies."


For anyone who was unsuccessful in obtaining a place at the e-infrastructure summer school, the presentations are now available online.  Due to receiving over 120 applications for the summer school and only having a limited number of places, I know many people were keen to see the presentations.

There will be another summer school in 2012 so watch this space for forthcoming announcements next year!

Blogging off

It is the last day of All Hands 2011 and it is my last day working for the NGS.

After 4 years of general griddery, I'm moving on.

Four years is a long time in research, and today's All Hands meeting at York is very different from the first grid event I attended, Open Grid Forum 20 in Manchester.

I remember that the Manchester meeting was huge and full of international delegates.

The UK contingent were based in something called the UK e-Science Village - which conjured up bucollic images of computer scientists dancing around the maypole on the e-Science village green, just next to the local shop for local people.

At the very least, I was hoping to see the UK e-Science Village People giving a rousing chorus of their classic - `(its fun to be at the) STFC.'

The village turned out to be a very large display booth.

All Hands is national, rather than international. The conference and the booths are smaller. As at OGF, people still enthuse about shiny new technology that will solve all our problems in the future.

But in among them are people using the less-shiny, less-all-singing, less-all-dancing software that we have now. And they are using it to do new research that is nothing to do with the technology itself.

And it is those are the people I want to hear - because what I have learned to call e-Infrastructure is very broad - in one session yesterday, the talks covered the behaviour of the heart, and how what the researchers have learned there has been applied to the way muscles move when giving birth; and how to model the way water shapes landscapes over millennia.

I still do not give a damn about how clever, or web-service-y, or standards compliant, a bit of e-Infrstructure is. It is what you do with it that counts.

It is the researchers who have take what we provide and use it to deliver the research that could not otherwise be done. These are the people you can read about in the case studies.

These are the people who have turned e-Research into Research - and will continue to do so for many years to come.

Goodbye UKI, hello NGI_UK

At All Hands 2011, in the atrium of the University of York's brand new Ron Cooke Hub conference venue.

On our stand in the middle of the room is a familiar face - helpdesk manager John Kewley - sitting under a slightly less familiar sign.

It doesn't say NGS, or GridPP, although both have posters on display.

The sign says but 'NGI' - aka National Grid Infrastructure - and we have had to to get used to it very quickly.

At  last week's EGI technical forum, what was the UKI ROC - or the UK and Ireland Regional Operation Centre - was offically replaced by two new NGIs called NGI_UK and NGI_IE.

And lots of things broke - including the load monitor and the Nagios testing service.

Names matter. Both the load monitor and Nagios were pulling information about sites and users from the Grid Operations Centre Database. More specifically, they will pulling information about sites and users associated with the UKI ROC.

The UKI ROC is no more: it has no sites or users associated with it.

So... we have spent the last few days tracking down every reference to the 'UKI' in every configuration file for every service and replacing them with NGI_UK.

There were quite a few....

The load monitor is back. We've been working on Nagios today and it should be fully working soon.

Three Little Words

There are these three little words. For some people, these words bring feelings of fulfilment and contentment. For others, they bring nothing but frustration.

Those three little words are:

  Proof of Concept

For that part of the e-Research Community interested in how research will be done in future, A proof of concept is evidence that it is possible to do something new and interesting, using something new and interesting. It might change the way research is done next decade. It is more than enough for a published paper and a presentation at All Hands.

And it is of bog-all use to those for whom e-Research is simply a means to an end. They just want something that works now and works reliably.

There is always a gap between the potentially useful and the actually useful. When you can build something that bridges that gap, you can enable research that would not otherwise be done.

Which brings me to slightly embarrassing news that our project to deploy the ARC middleware in front of the local High Performance Computing service has been a complete success... as a proof of concept.

We have shown that it is possible to deploy ARC services in front of what we should now be calling Oracle Grid Engine.

With some inventive use of ssh copies in prolog and epilog scripts --- that this can be made to work even where there is no file-space shared between the grid 'front end' and the HPC cluster.

We also know that you can support parallel tasks  using ARCs Runtime Environment mechanism --- there are examples at the bottom of the (slightly out of date) Nordugrid documentation --- and make use of to the LCAS/LCMAPS authentication system used by other grid software.

Which is nice....

Whether it is going to be useful is a completely different question.  We do not yet know if the local communities who are best placed to use it --- the rather incongruous pairing of Solar Physics and Social Science --- will want to do so.

Epilogue: Prologs and Epilogs


A quick technical note on faking a shared directory via Grid Engine prolog and epilog scripts.

The scripts run just before the start and just after the end of every job.

ARC-the-middleware obligingly changes directory to the 'shared' scratch directory before submitting the job. This mean that prolog and epilog scripts are presented with the path to this directory in the $SGE_O_WORKDIR environment variable.

The recipe is along the lines of...

• Create a ssh keypair for each user - to be used solely for transfers from HPC backend to grid front end
• Copy the private key to a safe place on the HPC back end, readable only by the user. We will call this $GRID_KEYS.
• Use the public key to create a per-user authorized_key file on the grid front end in somewhere like
     /etc/ssh/authorized_keys.d/$USER
  and change the /etc/ssh/sshd_config (again on the grid-front-end) to set.
      AuthorizedKeysFile  /etc/ssh/authorized_keys.d/%u
• Add code to prolog and epilog to use scp (or rdist) with the -i $GRID_KEYS/$USER to pull files from $SGE_O_WORKDIR at the beginning of the job and push them back at the end.

Conference season approaches...

It's gone slightly quieter for me now that the SeIUCCR e-infrastructure summer school is safely under way.  30 students are now ensconced in Coesner’s House in Abingdon where they are learning about the wonders of e-infrastructure and how it can help their research.  As I type they will have just finished a “hands on” session on the NGS and how to run jobs on our resources.

One event is underway but we still have two to go.  Next week sees many of the NGS staff at the EGI Technical Forum in Lyon.  The NGS in conjunction with GridPP is the UK National Grid Infrastructure (UK NGI) and in turn the UK NGI is part of EGI (European Grid Infrastructure).

It’s a very active meeting for many NGS staff due to the level of involvement we have in this major project.  As well as meetings, there will also be presentations in several sessions from NGS staff.  I’ve been asked to give a presentation on the NGS roadshows as EGI are developing their own roadshows – well they say that imitation is the greatest form of flattery!  As always the UK NGI will have a stand at the event where people can talk to us further about our activities, meet staff and obtain information.  If you are attending the EGI Technical Forum then drop by and see us.

The week following the EGI conference, many of us will be in York for the UK e-Science All Hands Meeting.  Registration for this is open until the 19th of September so if you want to go, make sure you register soon!  Again the NGS will have an exhibition stand along with GridPP at the event.  The exhibition stand will be a hive of activity as there will be several demos taking place here.  The demos are –

• Applying for UK e-Science Certificates using the new CA Certificate tool
• Taverna Server: Towards enabling long running workflows on the NGS

Some of our users will be actively taking part in the conference with demos and presentations not to mention NGS staff giving presentations and posters as well.

A major activity at AHM is a workshop organised by SeIUCCR which is a collaboration between the NGS and the Software Sustainability Institute (SSI).    The workshop is entitled  "Meet the Champions" and will take place on the Tuesday 13:30-16:30.

The workshop is an opportunity to meet researchers that have been promoting and leading research over the past decade of e-Science; find out about their work and how they utilise e-Infrastructure, and learn how you can interact with them.  Specifically the "Champions" to meet are members of the Community Champions network from the SeIUCCR (Supporting e-Infrastructure Uptake through Community Champions) project; the NGS Campus Champions and the Software Sustainability Institute Agents Network.

There will also be 2 key presentations -

• Scott Lathrop is Blue Waters Technical Program Manager for Education and TeraGrid Area Director for Education, Outreach and Training.  Scott's talk is entitled "Engaging Campuses in XSEDE".  XSEDE is the successor to TeraGrid.  Scott will be talking about the XSEDE Campus Champions programme and also the Campus Bridging programme for XSEDE.
• Steve Brewer is Chief Community Officer of EGI, the European Grid Infrastructure and he will be talking about Community Engagement in Europe.

And if you thought that was enough there will also be a panel session on the question "Why should researchers use e-Infrastructure?".

For the full details of all the NGS activities at the forthcoming AHM meeting please see the news article on the NGS website.

Hopefully we'll see some of you at some point over the next two weeks!