Send in the clouds

Today's vague term from the NGS Research and Development effort is 'Cloud'. Everyone wants a cloud.

Large companies have private clouds. The heavyweights of the IT world - Microsoft, Google and Amazon - all offer some form of cloudiness on a commercial basis. The UK Government has announced its own G Cloud project.

One of the major themes within the Research and Development work in the NGS is the provision of an Agile Deployment Environment - which can provide compute services quickly when needed.

This is yet another cloud, the kind usually known as Infrastructure-as-a-service. The idea is to provide virtual machines on request that can be set up in any way that the user wants.

The work is being managed from within the National eScience Centre at Edinburgh and done jointly by staff at NeSC and the Oxford eResearch Centre.

Development has focused on Eucalyptus, an open source implementation of Amazon's EC2 service.

Those involved presented their work at one of the regular NGS surgery meetings held on 11 March. You can find their slides in PDF and Powerpoint form on the NGS web site.

The broad conclusion is that - with current technology - it is possible to build a cloud that works like Amazon's but not (yet) one that scales like Amazon's.

We know that some researchers already use commercial cloud services. Which begs the question, why is the NGS adding another cloud to the crowd?

It partly down to data. Where there are enormous quantities of data to be crunched, or where the data is highly sensitive, a cloud hosted at a University has advantages.

Commercial cloud service charges reflect their costs and even Google, Amazon and Microsoft have to pay for an Internet connection. This is why commercial clouds typically charge for every megabyte of data transfered in-and-out of their service.

If you are dealing in terabytes of data, this can be very expensive. If you host a cloud for academic use in a UK academic institution, you can take advantage of the JANET network linking UK academia and JANET do not charge by the byte.

Some data is made available to academics under very strict rules - which mandate where it can be stored. It is quite common for the owners to forbid the export of data outside the UK or the EU. Again, if you process your data in a UK academic cloud, you can be sure it hasn't left the country.

By building an example cloud service and documenting the cloudy way, the NGS is providing a basis for future institutional and cross-institutional clouds.