This quarters edition contains features on -
- Using the NGS Cloud Protoype in teaching
- Running Taverna Workflows on the NGS
- Research Communities on the NGS Cloud Prototype
- Quantitative genetic analysis on the NGS
- ... and much more!
<ncg::configpublish>to the ncg.conf configuration file and ensure that the /usr/sbin/mrs-load-services script was run.
<configcache>
NAGIOS_ROLE=ROC
VO=ngs.ac.uk
</configcache>
</ncg>
Apparently 'Grid Computing is Dead'.
Again.
It wasn't Colonel Mustard, with the lead piping, in the library. It was David De Roure, with a posting, on the Nature eResearch blog.
To be fair on David: he is an eyewitness - not the perpetuator to the dastardly deed. He was highlighting a panel discussion at the IEEE eScience conference in Brisbane entitled "Grid Computing is dead: Let's all move to the Cloud".
That title looks like another round of that popular panel game: my vague terminology is better that your vague terminology. As Simon Hettrick has pointed out - Clouds computing has one big advantage over Grid computing - the Name. Clouds sound nice and fluffy; grids sound hard and rigid.
You cannot really discuss Cloud computing in general. You need to talking about the the various Somethings-as-a-Service.
Most cloudy discussions concentrate on IaaS - Infrastructure-as-a-Service. Through the wonders of virtualisation, imaginary computers are conjured up on a magic box somewhere on The Internet. You can ask for an imaginary computer and use and abuse it just like a real computer under your desk. This has changed the way computing is delivered.
It is not the only option. Some Research Institutions, and commercial companies, are offering access to High Performance Computing systems and calling it 'Cloud Computing'. More accurately, this is SaaS - Software-as-a-service - and PaaS - Platform-as-a-Service. This is good when you need access to a particular application or the messy bits needed to build an application.
A few years ago, they might have been called such an offering 'Grid Computing'.
The Grid is a Platform. We are offering it as a Service.
It might be a slightly-rickety platform but we have used it to support many applications and enable new research.
The Grid isn't dead. It has PaaS-ed over to the other side.
After all no-one wants a broken broker.
[Update: 8-Dec-2010. The Grid Acronym Soup now includes both ICE and CREAM. I suppose this turns it into a Gazpacho.]
So the NGS Innovation Forum is over for another year and, although it seemed to consume most of my time over the last month, I’ll miss it!
This years event was well received by all those who attended according to the feedback I was given both at the event and afterwards by email. Always nice to know we’re doing the right thing!
The event kicked off on Tuesday with a day focused primarily on our users and I’m glad to say there were some present in the audience. Steven Young gave a brief summary of the Campus Champions who are our eyes and ears in institutions – ready to help users and to feedback comments and suggestions to us. We then moved onto a series of talks about user tools. The aim of this session was to talk through some of the tools that we offer in order that users could head home from the event and actually apply them in their research. The tools covered were –
The day also featured three presentations from users who make a great deal of use of the NGS resources. We had presentations from a variety of research areas to demonstrate just how widely used our resource are. Luke Rendell from St. Andrews University talked about simulating learning strategies, Zhongwei Guan talked about modelling composite structures and Narcis Fernandes-Fuentes talked about using the NGS for early stage drug discovery. A bit of a range of uses!
Day 1 was really good with lots of questions and discussion which continued right the way from the last session through the drinks reception and poster viewing until the end of the event dinner!
Wednesday was aimed primarily at IT staff, sys admins etc so there were a few new faces on this day. In order to bring everyone up to speed, David Wallom re-capped the discussion from findings the day before. We then kicked off with a presentation from the University of Westminster who have been a NGS member for some time before moving onto a discussion session about how the NGS can help to facilitate collaboration between researchers and institutions.
Presentations on two NGS projects followed – accessing the NGS with Shibboleth and updates to the NGS accounting provision. The last session was dedicated to the EU with an update from the EGI Director, Steven Newhouse followed by presentations from two ESFRI projects – CLARIN and ELIXIR.
An exhausting couple of days but well worth it!
From an outreach point of view I’m now busy organising a couple of new roadshow events that people requested during the IF, I’m gathering the presentations from the event to go on the NGS website (watch this space!) and announcing the winner of the best poster at the event.
Congratulations to Jarmila Husby from the School of Pharmacy, University of London whose poster “Molecular Modelling Studies of the STAT3β homodimer:DNA complex” was voted the winner by the delegates. Jarmila won an Amazon voucher which is very handy with Christmas coming up! All the posters from the event will also be on the website soon.
If you missed the event there are a number of ways to catch up – the Twitter posts are available, a blog post from Catherine Gater of EGI, an article on Cloud computing from Simon Hettrick at SSI and photos from the event are available on the NGS Flickr account.
Thank you once again to all those who attended and hopefully we’ll see you all next year!
Standard output does not contain useful data.Cannot read JobWrapper output, both from Condor and from Maradona.error message.
pacman -get http://vdt.cs.wisc.edu/vdt_181_cache:UberFTP
xmlsec1 verify --id-attr:AssertionID Assertion shibdata.xmlWhere shibdata.xml here is a file containing the assertion.
<Reference URI="#_39e459384b39f1ddce64e11c58155abc">The URI is meant to point you at the bit of XML that has actually been signed. The code expects to find an attribute
ID="_39e459384b39f1ddce64e11c58155abc"attached to that element.
This posting is going deep into the innards of Grid software.
Think of it as a computer programmer's version of Inside Nature's Giants - a wonderful example of TV science but not necessary suitable for watching over dinner. So before we are get out the (metaphorical) scalpels, I want to explain why we need to do this.
The NGS provides the SARoNGS service - that provides certificates to people using their institutional credentials and store these in a MyProxy server.
We have developed the Myproxy enabled GSISSH to give users command line access to a grid compute service from any SSH client - this reads credentials from a MyProxy server.
By linking SARoNGS and Myproxy-enabled GSISSH, using the ability to create accounts on demand and opening the service to anyone in the UK Access Management Federation, it would be possible to provide such a service anyone in the UK academic community who needed it.
The big practical problem with this plan - and the one most likely to give your IT security people nightmares - is stopping this service being abused.
The missing link is the ability to provide very restricted access to users who are being nosy - enough to prove that it can be done, not enough to do anything - and full access to ones who have signed up to a suitable acceptable use policy.
If you offer a service that runs actual real programs on behalf of actual real grid users, then at some point you are going to be handed a blob of data that contains:
For services such as Globus GSI-OpenSSH and GRAM you need to associate the proxy certificate with an account on a compute service. The account will be used when running anything on behalf of the user.
This sounds simple. Lots of things about Grid computing sound simple.
This particular problem fails to be simple because there are many, many different ways by which the users proxy certificate can be delivered.
For GSI-OpenSSH, delivery is left to the Generic Security Service (GSS). Technical details can be found on Globus development webpages.
The code that provides GSS authentication plays a complicated game of network ping-pong as client and server bounce messages at one another until they come to a mutual agreement or give up trying. The people behind the Heimdal project have bravely attempted to explain how it works on their blog.
At the end of the game, the credentials are delivered to Globus in the form of a 'context' stored in a variable of type gss_ctx_id_t.
There is a function within the Globus libraries called globus_gss_assist_map_and_authorize that uses this context, feeds it to whatever authorization mechanism is used locally and returns a local user account.
globus_gss_assist_map_and_authorize is used in both the Globus GRAM gatekeeper and GSI-OpenSSH but does not seem to be part of the official application programming interface.
It will either look up the user in the Globus gridmap file or call out to an external authorization service such as LCAS/LCMAPS. The exact behaviour depends on environment variables and configuration files.The idea that code and code history is valuable in itself has been mentioned before in this blog and in much more prestigious publications before - and this applies even if the code was never finished.
We have one more problem to overcome. NeSCForge will be closing down on 20 December and we are not going to lose our source code when it does. The details of exactly how we will save our code will have to wait for another day and another posting.
[With thanks to Robert Frank at Manchester]
When we last covered the development of the new Nagios monitoring service in the blog - before last week's commercial break - we had just convinced it that all the hosts were alive and ready to be tested.
We can now proudly say that we have coaxed the service towards its first, official complete and utter failure.
All those highly motivated people who tell you`failure is not an option' - ignore them. If you are running service that tests things, having a test fail means that you actually persuaded that test to run. It isn't failure, it is a different kind of success.
And it is not as easy as it sounds because the Nagios development server is, quite deliberately, kept isolated from the rest of the world.
This is not a reference to the Harwell Science and Innovation Campus near Didcot: where the people from the STFC e-Science centre who run the service are based, and where the NGS Innovation Forum 2010 will be held.
It is simply that the Nagios development server has limited Internet access - as befits an experimental service. All access to the World Wide Web must be channeled through a web proxy. Privileged access to services is granted only when needed.
Neither the NCG configuration program or the various tests and probes that Nagios uses were written for an environment with a web proxy. Much of the code is written in Perl and support for proxies is already present -it just needed to be turned on. The Nagios developers at CERN have already accepted the changes for the next release.
With web access granted, NCG could build a complete configuration and the tests that suck information from web sites all began to run.
The next problem was getting permission to do things.
This is a grid. To use a grid, you need a certificate. WLCG Nagios has the wherewithall to download a certificate from a MyProxy Credential Management Service - as long as someone has uploaded it in the first place and there is no passphrase required.
The NGS provides a central MyProxy service and MyProxy allows certificates to uploaded so that they can be downloaded using another certificate as authentication. The command to do this isn't exactly short:
env GT_PROXY_MODE=old myproxy-init -s myproxy.ngs.ac.uk -l nagios_dev -x -Z '/C=UK/O=eScience/OU=CLRC/L=RAL/CN=nagios-dev.ngs.ac.uk/emailAddress=sct-certificates@stfc.ac.uk' -k nagios_dev-ngs -c 336
.. but it works.
Or at least it worked after we had added the certificate DN to the authorized_receivers and trusted_receivers entries in the myproxy server configuration file.
... and ensured that the ngs.ac.uk virtual organisation was defined on the Nagios server.
So at long last, the Nagios server could download a certificate, associate it with a virtual organisation and use it to submit jobs via a Workload Management Server.
Which was the point at which we realised that the Workload Management Service endpoint (https://ngswms01.ngs.ac.uk:7443/glite_wms_wmproxy_server) should have been defined in the glite_wms.conf and glite_wmsui.conf files in $GLITE_LOCATION/etc/ngs.ac.uk/.
With that final hurdle overcome, the test jobs started to flow.
The Compute Element tests were sent sites declaring themselves as Compute Elements - including the original NGS core sites at Leeds and RAL.
I admit to rigging it so that Leeds was tested first. The little status box went Green as the job was submitted, then Red as it failed with a friendly:
- Standard output does not contain useful data.Cannot read JobWrapper output, both from Condor and from Maradona.
Sometimes an Argentinian footballer and a large scavenging bird can make your day.
Following on from my previous blog post about the JISC- Future of Research? conference, the next parallel session I attended was “Evolution and Revolution in ICT and Arts and Humanities research”.
The first presenter was Simon Tanner from King’s College London who spoke about some JISC digitised collections. No slides from this one I’m afraid as he showed us pretty pictures instead!
The next speaker was from Mimas who spoke about “long tails and efficiencies of scale”. The slides for this one are available and of particular relevance to myself and the NGS, was the observation that services have to show –
This is something that the NGS has been undertaking recently and will continue to do so as we come up for refunding next year. You may have seen a small flurry of stats etc on the website and we have been undertaking more stats gathering behind the scenes from the proliferation of data available from usage stats and user applications. From this we are building up a picture of user demand, impact and value etc.
We hope that our users will help us by contributing to our forthcoming annual user survey and our follow up roadshow survey. If you have attended any NGS roadshows we would be grateful if you could complete the short survey which will take about a minute to complete! This will help us shape future roadshow events and also analyse the benefits that users get from attending these events.
The final speaker was John Coleman from the University of Oxford who presented on “Large Scale Computational Research in Arts and Humanities”. He started with an interesting fact that in 2008 YouTube was the second most popular search engine with people looking for speech / audio instead of text. He is currently working on a JISC funded project to “mine a year of speech” which aims to annotate a years worth of data in the form of a corpus. To do this he is currently using about 20 computers set up as a cluster in a local lab. However he is now looking at placing the data in several other universities which he says is “like grid computing”. John also highlighted a report available on "ICT Tools for searching, annotation and analysis of audiovisual material" which may be of interest to people.
The JISC Future of Research? conference had some very interesting parallel sessions and I hope it continues next year!
Last Tuesday the NGS was out on the road with an exhibition stand at the JISC Future of Research conference which was held in London and online.
As well as speaking to delegates browsing the exhibition stands, I also went to some very interesting parallel sessions. The first session I attended was “Centralising your IT Support for Research” and consisted of 3 presentations including Mary Visser, Director of IT from the University of Leicester. Mary talked about how researchers want “free at point of use” as funders are unwilling to pay FEC for these, seeing IT as basic facilities which should be provided by the institution. Leicester currently have an IT research liaison manager who speaks to researchers about their IT needs and provides guidance and advice about local and national resources. Sounds like something that many of our users would like!
Mary’s presentation for a centralised IT support service was countered by Rob Procter from Manchester e-Research Centre who argued the case for more distributed IT provision within schools and departments. Rob pointed out that many researchers do not trust IT services to provide what they need as they are teaching focused with most of their effort going in this direction rather than towards research. Robs argument for embedding IT staff in departments is certainly one I’ve heard many times before and I’ve also seen some very good results and collaborations come out of these situations.
Here at the NGS we realise that not all universities have IT services that can help with grid computing or even the use of computing in research. The NGS can’t have technical staff based in every institution in the country but we do have a variety of means to try and help from afar.
Check this list to see if you have a local Campus Champion who can provide some advice or support to you in your institution. If you don’t have a Campus Champion, we have our helpdesk where knowledgeable staff can answer your queries by email. We also have a variety of tutorials to talk you through getting started and running jobs. If there is anything else you would find useful regards training material then please let us know!In that strange parallel universe that exists only in TV adverts: two friends sit in a remarkably spacious and clean kitchen, sipping low-calorie-but-surprisingly-tasty beverages, and talking.
And what are they talking about? Which wonderful washing powder washes whites whitest.
If we lived in their world, I would be able to introduce:
The all-new NGS schema washing service - removing unpleasant stains from your grid information and leaving it huggably soft and smelling of Summer Meadows.
elvis is alive [*]'Ping', the friendly name of an ICMP echo request packet, was invented as a way of testing network connectivity. The original idea was that if a machine on the Internet was working and it was pinged, it should send back the contents of the 'ping' to the sender as an ICMP echo reply.
<NCG::ConfigGen>
<Nagios>
...
# Disable 'ping' checks of hosts
CHECK_HOSTS=0
</Nagios>
</NCG::ConfigGen>
export TOP_SECRET="if we told you, then it wouldn't be a secret any more."
openssl enc -e -blowfish -in secret-data.plain -out secret-data.enc -pass env:TOP_SECRET
This is symmetric encryption so now that we need to play pass the password.
export TOP_SECRET="that password in the line above that we are still not telling you"
openssl enc -d -blowfish -e -in secret-data.enc -out secret-data.plain -pass env:TOP_SECRET
The password.enc file can be sent to the certificate holder who can unscramble it using something like
echo $TOP_SECRET | \
openssl rsautl -encrypt -certin -inkey someones-public-cert.pem \
-out password.enc
This can be used to update the password securely and ensure and all the secrets stay secrets.
openssl rsautl -inkey /path/to/my/userkey.pem -decrypt -in password.enc
A bit like sending a letter to Santa (you have no idea where it is going and you can be fairly sure you won't hear anything back).I suppose that makes us Santa's little helpers: less of a National Grid Service, more of a National Elf Service.
Standard output does not contain useful data. Cannot read JobWrapper output, both from Condor and from Maradona.and people running grid software quickly get used to seeing the classic
GSS failed Major:01090000 Minor:00000000 Token:00000003These are nothing to do with large birds, Argentinian footballers or unsuccessful members of the military. Roughly translated these mean 'Sorry... your job went missing' and 'Oops... invalid certificate' respectively.
An extra special reminder that the Call for Poster Abstracts for the NGS Innovation Forum closes this Friday. If you or anyone you know would like to submit a poster abstract to this event please make sure that you submit your 200 word contribution to the NGS website by 5pm on Friday (24th).
Remember that there will be a prize for the best poster as voted for by the delegates and all abstracts will be peer reviewed by the Innovation Forum Programme Committee. We would like to encourage all users to submit a poster and to attend the event in order to hear about the latest developments and tools from the NGS and also to leave with the knowledge of how to apply these tools in their research. Delegates are welcome to attend for one or both days of the event.
Registration for the event is also open now and further details are available on the event page on the NGS website.