What are they up to in Europe with the grid?

If you are interested in seeing what European researchers get up to with grid resources then check out the new edition of Projects.

A new edition of Projects magazine has recently been released which focuses on Grid projects in Europe. Many of these projects demonstrate how grid technology has been used in a wide variety of research areas such as kidney diseases, HIV, neurosemantics, renewable energy, access to social science data and much more!

As well as research articles the magazine also has articles on how the European Grid Initiative (EGI) was formed and lessons learnt from the previous EGEE project.

The magazine can be found on the interactive website.

Crime and the grid (and a slight return)

If I thought my journey to Sweden for the last ever EGEE user forum was stressful, little did I know about the return one! Thanks to a volcano it took me 3 days, 7 trains, 41 hours, 6 countries and 2 overnight stays to get home. Not doing that again in a hurry...

So updates from the NGS - not a great deal mainly because I've been away! However we do have a new user case study up on the website . Nick Malleson from the University of Leeds has been using NGS resources to model criminal patterns in Leeds. A really interesting and different use of NGS resources!

A quick reminder as well incase you haven't seen the announcements but the registration for our Communicating Science event will close on Wednesday afternoon so register soon if you want to attend!

From desktop to grid

[With thanks to David Spence at Oxford]

The NGS is always looking for ways to make grid software easier to use. We do not want to force people to learn new tools if they can use what is already installed on their desktops and laptops.

One approach we are investigating is "Drop-and-Compute".

The original idea for drop-and-compute came from Ian Cottam, an IT Manager with the Faculty of Engineering & Physical Sciences at Manchester. He wanted to users to be able to submit jobs simply by copying the relevant files to a local directory on their computer.

The approach chosen was to use DropBox - a service available to users of Windows, Macs and Linux can synchronise files between computers. DropBox can ensure that files on your local machine are made available to a central service and that any subsequent changes are copied back.

The original drop-and-computer was designed to run jobs on campus grids using Condor.

As part of the R+D effort, the NGS staff at the Oxford eResearch Centre are investigating if the service can be adapted to use our UI-WMS service - allowing jobs to be sent to any NGS partner site where there are free resources.

Development is far from complete - and there is much work to do on making the service work with the kind of security needed by a National Grid. It has advanced enough for a test service to be made available at Oxford for internal, and selected external, users.

[Original article incorrectly associated Ian Cottom with the MyExperiment project. The MyExperiment project host  the drop-and-computer how-to on their Wiki.] 

Impact!

After a tad stressful journey I am now safely installed in Uppsala with the NGS / UKI Federation stand in full swing at the 5th EGEE User Forum. The conference is being held in Uppsala University which is the oldest university in the Nordic countries. The building we are in is stunning as I’m sure you have seen on some of the photos already posted on the GridCast blog.

Our hosts have looked after us very well so far with excellent organisation and efficiency. There is apparently about 300 people registered for the meeting and it does seem slightly quieter than previous meetings such as those held in Catania last year. However the UKI stand has been busy as always as it becomes a hub for meeting European colleagues and for people wanting to find out what the NGS and GridPP are up to in terms of the EGI.

YEsterday was the first day that I made it to sessions after spending most of Monday setting up the stand and staffing it. I attended the EGI-Inspire session to find out how the EGI is progressing and how the NGS can and will be interacting with this European wide initiative. It did fill in some blanks!

One topic that came up was how to measure success? What is the impact of the EGI on research? Preferably the user community should actively acknowledge the use of the resource but as I know from personal experience at the NGS this is something that we find very difficult to gather information on.

Our funding bodies are constantly asking us to demonstrate our impact and to show the effect that using our resources has on research productivity in the UK. We can only do this by getting information from our users such as the numbers of papers produced, presentations at conferences, posters produced etc. These are all excellent metrics that we are asked for but we find it very difficult to get the information out of our users! Unofrtunately if we can’t prove our impact, we can’t justify more funding. This is a problem that many organisations face and, as yet, no one has come up with a solution yet. Answers on a postcard please!

Déjà vu? No just some duplication…

Next week I’ll be attending the last ever EGEE conference in the slightly chilly surroundings of Uppsala, Sweden. The NGS will be attending the event in good numbers with many staff attending as we are heavily involved in the EGEE successor, the EGI.

As usual we will have an exhibition stand in conjunction with several other UK e-science institutions including GridPP and NeSC. We have lots of examples of how researchers in the UK are using the grid in there every day research as well as details of what we’ve been working on here at the NGS. GridPP will have their Real-time monitor with them so you can see how the data from the LHC is being distributed and sent round the world as it happens.

I’m looking forward to the conference as its an exciting time of transition for the EGEE / EGI project. I feel that countries will start working even more closely together with EGI which can only benefit our researchers who are increasingly involved in international collaborations through European projects etc. The question is how do we get there? How do we work together more closely and more collaboratively? Hopefully some of these questions will be answered at the dedicated EGI sessions at the conference.

So all I have to do now is cross my fingers and hope that all the material for our stand arrives safe and sound in Sweden today, grab some Swedish Krona and work out how I will cope with a beer costing about £5.50 for a 50cl bottle when I usually pay £3!

Oh and I should explain the title of this post – I will be blogging about the conference as part of the GridCast team so my blogs will be reposted on their blog. Apologies if you see them twice!

Who looks after your software when you've gone?

It's a familiar scenario. A PhD student creates a piece of software during their 3 years of research, it's used for their project but then they leave. What happens to the software? Who looks after it? Who maintains it?

EPSRC have also recognised this problem and have recently awarded £4.2 million to establish the UK’s Software Sustainability Institute (SSI). A team of academics and software engineers based at the University of Southampton’s School of Electronics and Computer Science, the School of Computer Science at the University of Manchester and led by EPCC at the University of Edinburgh will work together to tackle these issues.

Together with the research community they will work to manage software beyond the lifetime of its original funding, so that it is strengthened, adapted and customised to maximise its value to future generations of researchers.

The establishment of the SSI obviously has implications for the NGS as there are many pieces of software out there which could be used by the wider community on the NGS - they just need a bit of TLC!

Testing... testing...

In an ideal world, computer systems would tick along nicely, never causing any trouble or going wrong.

And IT staff could sit around doing nothing and drinking coffee all day.

The reality is that - if your computer system seems to be ticking along nicely - then your IT staff are doing their jobs very well indeed... and drinking coffee all day.

Any well-run computer system will have some kind of monitoring in place: to alert the system administrator when a disk starts to fill up or a computer overloads. A decent monitoring system will detect minor problems before they turn into major problems. There are many monitoring systems available, one of the most popular being Nagios.

A well-run Grid needs monitoring too.

A Grid is a collection of services spread over a number of institutions and monitoring brings its own complications . For example: if you want to test if accounts are working as expected on the NGS partner site at Leeds, you will first need to contact a Myproxy server at STFC for a certificate and a VOMS server at Manchester for Virtual Organisation membership.

The NGS monitors all partner sites using a service based on the INCA framework from San Diego Supercomputer Centre. This information is publicly available at http://inca2.ngs.ac.uk.

Our colleagues in GridPP make use of a number of monitoring services.

Yet INCA and many of the the other monitoring services are specialised and not widely used outside the grid community.

Current R+D efforts within the UK Grids focus on using tools such as Nagios which are more familiar to the system administrators.

We already have a Nagios plugin - available from the NGS project pages on NeSCForge - that can be used to integrate INCA results into an existing Nagios service.

There is also a project underway at STFC to determine how far the WLCG Nagios system can replace what we currently do with INCA and whether we can build a monitoring system that covers the NGS and GridPP sites.

[Edit, 11-Apr - changed explaination of the complications of testing a grid]