NGS at SC'11 - round up

The UK presence was fairly significant this year at SC'11 with attendance by David Wallom, NGS Technical Director and a significant number of the leaders of research computing centres from around UK universities. This included NGS member sites @ Bristol, Leeds, Oxford, Southampton and EPCC.

In this blog post, David gives us a round up of NGS activities at this major computing event.

Having got off of the very long flight from Heathrow
to Seattle we settled onto the metro to get us  downtown. After passable sleep the following morning we headed over to the SC’11 venue – the Washington Convention Centre to collect our badges and then visit the workshop on HPC in Smart Grid, where there was UK interest from the EC FP7 HiPerDNO project being presented by Dr Stef Salvini, OeRC. Following a very productive day where we learnt the state of the art in US Smart grids, how they intend to utilize knowledge developed through the national e-infrastructure for research. We then met up with the EGI team who had an exhibition stand at the conference.

The Monday workshop, Many-Task computing on Grids and Clouds 2011,  started off with an interesting keynote from David Abramson (Monash) a long term friend of the NGS through support for their Nimrod tool which is popular with several of our biosciences users. After this there was a panel session which went slightly off topic to talk about exascale more than Many task but it still attracted several questions around the need for exascale, when we are still struggling to get a significant user base onto smaller HPC systems. Overall a good workshop though having the panel first did mean that a number of people didn’t hang around for the rest of the papers. This workshop was operating in a very competitive market with other sessions on cloud and data management which also attracted significant crowds.

The first full day of the conference allowed for the first good look around the exhibition floor alongside several interesting birds of a feather sessions There was also the  first of a number of conversations with different groups and vendors, including Microsoft, Mathworks and Adaptive Computing.  To give an idea of scale this picture is down one of the main aisles in one of the 5 rooms that were all about this size!!



Pretty impressive stands by a number of people
but the coolest was the multi projection globe on the NOAA stand.

We of course also announced
our activity with Globus Online which created a lot of interest and ended with us having a number of interesting conversations with NSF regarding future collaboration between our national e-infrastructures.





During the meeting the EGI booth was continually visited by a reasonably large number of people, we had the Real Time Monitor showing as normal (having seen a lot of 3D screens this needs to be done in 3d now for next year!). They did though give away a pretty large number of t-shirts as did a lot of stands, so I ended up as the moving poster board around down town  Seattle from 6:30-7am every morning on my morning run!

My data, your data

Edinburgh hosted the Data Intensive Research workshop last week, along with the XLDB Europe. Genially chaired by the formidable Malcolm Atkinson, it was an excellent opportunity to hobnob with other data management folk. While most were database people, many were plain ol' data managers, or people who have data and nowhere to put it.

One of the recurring themes was that there should be more recognition for the work supporting science: if you have a dataset and you spend time preparing, validating, and annotating it, and someone else publishes a result derived from this data, they get the credit for the discovery - naturally - but the work to prepare the data should also be recognised (e.g. in RAEs, maybe as something equivalent to a publication.) Also if it doesn't lead directly to a new discovery. People who share data contribute to science. Moreover, these days, sharing data is often necessary for public funding.

The NGS has a few datasets in bioinformatics already. Having it available already is highly advantageous because you don't have to transfer and update it yourself, and it doesn't take up any space in your storage quota. If there are other datasets you think the NGS could usefully provide, let us know?

Speaking of research support, if you use the NGS you should of course cite NGS in your publications. But maybe you'll also want to buy your friendly local NGS admins a beer or something?

Delivering data

There now follows a Public Service announcement from The National Grid Service Department of stating the bleeding obvious.

There is very little point in using Grid software on a machine in Daresbury to run an application on a computer near Didcot if the data you need is stuck on a server in Darwin.

That statement is not going to be a surprise to anyone. After all, the Worldwide LHC Computing Grid was built to ship the flood of data from CERN to somewhere where it could be stored and then on to somewhere where it can can be analysed.

When delivering data, there is definitely more than one way to do it: you could use GridFTP or SRB or iRODS or SRM, or SFTP or FTP or WEBDAV or HTTP or even, if you are feeling old fashioned, read and write to files on a local disk.

Things get more complicated when you need to collect data through one mechanism and deliver it through another. In practice, this almost inevitably means that the data is copied onto local storage before being sent to its final destination.

This is not practical if there is a lot of data and you are on a comparatively slow network connection.

This is one of the problems that the DataMINX Data Transfer Service (DTS) aims to solve.

The DTS is an international collaboration jointly funded by the Australian Research Collaboration Service and OMII-UK. It isn't really NGS R+D but it is built on earlier work from the NGS and staff from the NGS have provided much of the development effort.

The idea behind DTS is that you give the job of delivering your data to the DTS in very much the same way as you would give the job of delivering a favourite Aunt's birthday present to a parcel courier service.

A courier will have a network of planes, trains, vans and delivery drivers to collect the parcel and carry it to its destination. You just have to book your collection. Auntie just needs to sign for the parcel.

Delivery in the DTS is done by pools of worker nodes with fast network connections and the wherewithall to send and receive data using the many network protocols. An internal messaging system that allows requests for data transfers to be made and for the status of the transfers to be reported.

In software terms, the developers of DTS have deliberately avoided reinventing the wheel - something for which the Grid has a not-entirely-undeserved reputation. Where possible, they have adopted and adapted existing widely-used libraries. For example:

• File access is provided by commons-vfs-grid - extensions to the Apache Commons Virtual File System to support griddy protocols.
• The messaging system is Apache ActiveMQ
• The Spring Java framework provides features such as batch processing.

There is much more to DTS than can be covered in a blog post. If you want to know more: a powerpoint presentation describing of how DTS works can be found, with the source code, on the projects web site (http://dtsproject.googlecode.com) and a formal paper describing the work due to be published in Philosophical Transactions of the Royal Society A in late July or early August.

[With thanks to David Meredith of the DTS project.]

Wish you were here

For those of you not at the Innovation Forum, you are missing lots of interesting presentations and discussions. Applications, clouds, outreach, "campus champions", EGI, and more. Oh, and real research. I encourage you to see the slides if and when they're available.

In my areas of expertise, I note in particular the requirements for data management and identity management. On the latter, we are working to improve the way certificates are managed in various stages of their lifetime; recent developments in the IGTF will, for once, help make some things easier. The CA will be modernised. And SARoNGS will of course also be available.

For data management, it is interesting to note the very different requirements of communities: security, performance, volume, metadata, filesize, etc. The good thing about the NGS is we're stronger together. For a new community, to solve their problems themselves would take a long time, but using infrastructure set up for other communities makes things much easier.