Thursday 28 April 2011

From the weather in 1994 to the climate now

The most successful users of grid computing are - without doubt - the particle physics community. The Worldwide LHC Computing Grid stores and processes the Terabytes of data generated from the experiments at CERN.

It is all very impressive - even for people like me who don't understand what a Higgs Boson is.

I am not a particle physicist. Back in the mid '90s - when work on the Large Hadron Collider project was beginning - I was at the Department of Meteorology at the University of Reading.

So it was strange to see a familiar name from way-back-then appear in International Science Grid This Week - and asking what the climate science community could learn from CERN.

Back in 1994, the Meteorologists occupied one end of some re-purposed second-world-war huts on the outer edge of campus. On one wall of the huts, opposite the portacabin that served as a coffee room, were the results of one of the field's latest innovations: ensemble forecasts.

Weather forecasts take the best estimate of the state of the atmosphere at one moment in time and calculate how this will change over the next few days.

That best estimate of the state of the atmosphere - called an analysis - is based on observations gathered by the World Meteorological Organisation.

The observations are scattered unevenly over the planet. For entirely practical reasons, there are many more observations over land than over the sea, and many more at the surface than further up in the air. Parts of the world are not covered at all. The best estimate will never be perfect.

Weather forecasting uses huge amounts of computer power and an individual forecast must be completed within a few hours or it is useless. You do not want to be told that a storm is coming after it has arrived.

Until the mid-90's, the most powerful computers available were only able to run one forecast before time ran out.

Weather is chaotic: small features can grow rapidly into huge storms. If such a feature is misrepresented in the analysis - as happened in 1987 - that one forecast can be very, very wrong.

By the mid 90's, enough computer power was available to produce ensembles of forecasts - each using slightly different initial estimates. Sometimes members of the ensemble stayed close together, sometimes they diverged. For the first time, we could estimate how reliable the forecast will be.

The ensembles were produced by the European Centre for Medium Range Weather Forecasting, a short distance up the road from the Meteorology department. This pioneering work is as relevant to the long term forecasting of climate as it was to the short term forecasting of weather.

Tim Palmer was one of those behind the ensembles. He is now a Professor at Oxford and one of the foremost experts on weather and climate prediction forecasting. The talk he gave at the American Geophysical Union fall meeting in 2010 is one of the clearest descriptions of one of successes and failures of the field.

In this week's edition of International Science Grid This Week carries an article by Professor Palmer advocating 'a CERN for Climate Change'.

Human activity is changing the climate - whatever you might read elsewhere on the Internet - but the details of the changes are still not understood.

If those who have studied our climate say we need a CERN-like organisation to understand it, then we should be listening.

[Small edit for clarity, 3 May]

Tuesday 26 April 2011

The sun is out

Given the very sunny Bank holiday weekend we've just had here in the UK, it seems apt to highlight the latest user case study to appear on the NGS website.

The latest case study highlights the research of Marco Califano at the University of Leeds who has been looking into exploiting solar power using the NGS. Marco used NGS resources to try to identify the best materials, sizes and shapes of nanocrystals that could lead to high-efficiency energy conversion in the next-generation solar cells.

Marco explained how "the outcome of my research could help the UK develop alternative, renewable energy technologies that enhance the cleanliness and efficiency of energy production. This, in turn, will contribute to the Government's commitment to meet the greenhouse gas emission targets set in Kyoto".

To read more about his research, see the exploiting solar power case study.

Thursday 21 April 2011

EGI User Forum round up

Last week saw many NGS staff attend the EGI User Forum 2011 which was held in Vilnius, Lithuania. Many NGS staff attending were actively involved in sessions either chairing or presenting as well as helping out on the UK NGI (UK National Grid Initiative) exhibition stand.
Personally my main involvement was ensuring that the UK NGI exhibition stand got there in one piece and that staff from NGS and GridPP were on hand to answer questions from delegates about our organisation. We had a very busy stand – possibly one of the busiest we’ve had at these events which was good to see.

I was also presenting at the EGI Dissemination session which was chaired by Neasan O’Neill, NGI Dissemination Coordinator. Neasan had asked some of the more established NGI’s to discuss the dissemination methods they had found most useful and effective. I presented on the NGS roadshows of which there have been 12. I discussed the content of a typical roadshow, the feedback we receive directly after the event and the follow up I do approximately 3 months after the event.

I also ran through (very quickly!) the other dissemination methods we have including our user case studies, our quarterly newsletter and the innovation forums. There were quite a few questions and some good discussion after the presentations with quite a few leaflets disappearing from the stand as well which meant less to carry home!

The conference was excellent in general with a brilliant venue and fantastic organisation. The next EGI event is the Technical Forum which will take place in Lyon in France on the 19th – 23rd September 2011.

Wednesday 20 April 2011

Looking back - and being differently successful

Failure is not an option... it's a standard feature.
As witicisms go: it is neither particularly funny, or particularly original - and can be seen on T-shirts, fridge magnets and snarky comments across the Internet.

Anyway, not being funny or original has never stopped me before.

We are looking back at the last two years of R+D: we've done the successes, now it is time for the failures.

I'm not talking about those magnificent 'Brown Paper Bag' failures that will haunt us for many years to come. I'll cover those another time.

Here, I want to mention those projects that have been mentioned at conferences or at this blog, which are working or very nearly working, but are not where we hoped they would be.

I am not criticising the highly-capable and skilled developers involved, or the way the work was managed. The NGS is a small organisation: bad luck or the need to solve more pressing problems limit what we can do.

The nearly-successful projects involved are the CA Wizard, SARoNGS and DataMINX.

The CA Wizard (not to be confused with the existing NGS Certificate Wizard) was intended to replace the confusing - and, to some users, downright scary - business of obtaining a certificate with a single desktop tool.

It was delayed by the loss of some key members of staff but is still being actively developed. The developers expect to release a version in May.

DataMINX was a joint UK and Australian service designed to ship data to where it was needed with minimal user involvement. It also fell foul of staff losses and had to finish early.

The developers were concerned about software sustainability and have left the code in a state where someone else could readily take over: it was released with an Open license and is available from Google Code. It is still being recommended in some quarters.

SARoNGS has been mentioned a number of times on the blog. It provides a way of allowing users onto a grid using their institutional credentials.

We know that SARoNGS development has been too slow. We know that the user interface is confusing. We know it doesn't quite work with the UI/WMS. We were reluctant to turn it into a production service until we had overcome some mind-twistingly-obscure bugs.

But SARoNGS has been used - at York, within the NeISS project, and elsewhere - as a simple way to pass authentication information around.

Failure may not be an option but - thanks to the efforts of people within and outside the NGS - it isn't inevitable either.

[With thanks to David Meredith]

Thursday 14 April 2011

Shooting for the Moon

The JANET networkshop is now in its third day, abuzz with all things networking and with lots of networkers from industry and academia networking with each other. But also for identity managers was there something of interest.

Project Moonshot is an activity with ambitions to, er, shoot for the moon - to develop an infrastructure for what I call A(A(A))I (usually people "solve" authentication first, then look at authorisation, and then maybe accounting.) The great thing about Moonshot is that it is built entirely on standards.

Moonshot's Sam Hartman gave a very impressive presentation on the next five years in security, covering both grids and clouds. The amount of technical data Sam is able to hold seemingly effortlessly in his head reminds me of the people who play simultaneous chess. Josh Howlett from JANET gave a very interesting presentation on the technical progress and current state of Moonshot: in particular, note how many services work almost unmodified because they use generic security services (JANET will make the presentations available online.)

[Managers' version] Moonshot is a way for users to authenticate to resources. As with IGTF and eduRoam, trust is built from federations - if your user is in the same institute, it is easy enough to trust them; if they are in another country, you need to work your way up the hierarchy. But this is all transparent because trust relationships are established beforehand. So why something new? Shibboleth is tied closely to web resources, IGTF certificates are usually separated from home institution ids, eduRoam is currently mainly for networks. If we can make everything interoperate, you can use your home id to authenticate to everything: the grid, the network, clouds, your toaster, etc. Federated identity brings users single sign-on and single accounts for multiple services.

[Techie version] Moonshot links a RADIUS (2865) service (like eduRoam) into applications using a GSSAPI (2743) module for EAP (3748). So it's all very modular and based on existing standards, but of course much of the module and glue stuff is new. Proposed extensions to GSSAPI will bring support for delegation (5588, 5896). GSSAPI already supports (4121) Kerberos, and this has been tied into MyProxy by Daniel Kouril from CESNET. Sam has built a virtual image which can now be used to demonstrate this. Based on Debian, it should work with several different image hosting environments, such as Xen or VMware. For the NGS, we should now deploy a(nother) MyProxy server, I am thinking of using the training CA, to enable Moonshot access to the NGS.

[User's version] You will be able to access more stuff by just using your home institution password!

Friday 8 April 2011

Looking back - where did it all go right

Now all involved have been properly thanked - we can look back at the more-successful and, er, less-successful projects that the NGS has undertaken.

When phase 3 of the NGS started in April 2009, the first thing we did was to make a list.

It wasn't exactly a To-Do list, more of a Could-Do list.

We asked for suggestions for new `services' - 'service' here being a woolly term for something that someone, somewhere might find useful - and collected just short of 40 ideas. These covered the themes of Advanced Reservation, Middleware deployment, User facing services and Data access.

It would have been impossible - given the time and staff available - to implement them all. We had to identify those that were impractical and concentrate on the most promising few. Sometimes we got it more-or-less right; sometimes we didn't. As has been noted before `If we knew what it was we were doing, it wouldn't be called research, would it?'.

In this post, I would like to mention those places where, in my opinion, we got it right - starting with the NGS communities.

Our users have to spend time obtaining a certificate and applying for an NGS account. We wanted to make that account as useful to them as possible.

With communities, individuals can associate themselves with subject area - anything from Aerospace to Zoology - and search for others in the same area. They can look for people based at their home institution, or read the descriptions of their work that others have entered.

Behind the scenes, we also added tags - labels that the NGS support staff can associate with a user and with membership of a group within a Virtual Organisation.

A tag can, for example, identify someone as a legitimate user of a piece of licensed software - and that information can be used to make that piece of software software available on different hosts. This avoids some of the complications of licensing on the Grid.

No list of R+D projects would be complete without a reference to the technology behind the NGS interactive Applications page - where present the list of available software on the NGS website for users to explore.

These are not huge projects or major changes in technology. What we wanted to do - and I hope have done - is to make the grid a little more friendly and a little more useful.

Tuesday 5 April 2011

Multi-tasking NGS staff

Many people involved in the NGS don't just contribute to our organisation, they contribute to many more. For example our Technical Director, David Wallom who is based at the University of Oxford, is also heavily involved with the OGF as the Vice President of Community.

David recently attended the International Symposium on Grids and Clouds 2011 (ISGC 2011) where he was interviewed by the GridCast team regarding the future of the OGF. If you would like to see the interview video, see the GridCast blog post.

Friday 1 April 2011

Looking back and saying thank you.

This is the 50-th R+D post on the NGS blog and - despite the date - I will not be trying to be funny. It doesn't seem appropriate given that the NGS's Research and Development activity is winding down.

In the original plan, Phase 3 of the NGS would have finished yesterday. Although we have been granted an extension until the end of September 2011 to prepare for Phase 4, we will be spending this time ensuring that the services we have developed work as well as possible.

It does provide an ideal excuse to look back on what we have achieved - and not achieved - over the last 2 years.

NGS3 arrived at around the time when researchers were becoming interested in cloud computing - and we covered both cloudy and more traditionally griddy services from the earlier phases.

The cloudy bit - run from Oxford and Edinburgh - built an Amazon-ish cloud for academic that eventually formed the the NGS Cloud service.

The griddy bit that - run from Leeds - concentrated on improving the underlying authorization, accounting and monitoring upon which the people who use grid in their research rely.

We'll cover each of these in future postings.

I want to use this one to say that building anything with the help of a group of people spread over 5 institutions and 6 sites, split between many projects and who seldom meet in person - was always going to be interesting.

You can reach new levels of miscommunication when your primary way of talking to one another involves email, or phone, or messages posted to a Basecamp instance, or doing Max Headroom impressions in AccessGrid.

Anyone who can survive the experience and produce something deserves credit.

So if Jonathan Churchill, Simon Collins, Cristina Del Cano Novales, Mark Duller, Matt Ford, Robert Frank, Gokop Goteng, Jens Jensen, Mike Jones, Kevin Haines, Shiv Kaushal, Akay Okcun, David Meredith, Ahmed Sajid, James Scott, David Spence, Steve Thorn, Duncan Tooke, Paul Townend, Matteo Turilli or Steve Young are reading this, I would like to take this opportunity to say:
Thank you.