Tuesday, 31 August 2010

Registration for NGS Innovation Forum open now!

The registration for the NGS Innovation Forum is now open - details of how to register can be found on the event page on the NGS website.

We are pleased to announce another speaker for the event. Neil Geddes, director of the NGS, will be speaking about the future of the NGS so if you are a long term NGS user who wants to know the future direction of the NGS or a new user who is planning to use the NGS for the long term, then make sure you attend!

A reminder that we are still looking for NGS users to submit poster abstracts demonstrating how they use our resources in their research. The deadline for abstracts is the 10th of September so it's approaching soon! There are many benefits of submitting an abstract and attending the event -

  • Walk through demos of new NGS tools
  • NGS staff on-hand to answer your questions
  • The opportunity to contribute and feedback to the future of the NGS
  • The poster abstracts will be peer reviewed by the NGS IF'10 programme committee
  • Publicity for your research both at the event and through accepted posters being placed on the NGS website
  • The chance to win a prize as "best poster" as voted for by IF'10 delegates
All that is required is a short 200 word abstract! Of course you are more than welcome to attend the event without submitting an abstract and you can attend for one or both days. We hope you can come along!

Friday, 27 August 2010

Ravioli code

The ngs-vo-tool - a utility for organising your virtual organisations - was released to the NGS's area on NeSCForge earlier this week.

Previous postings have described the reasons why the tool was written and how we worked out what it needed to do. This one is about developing and testing the program code itself.

So if you thought the 'Rough guide to the User Account Service' was dull, look away now.

The ngs-vo-tool was meant to be an example of 'software as documentation'.

If you are running complex software, which needs configuration information to be scattered around multiple files in different formats, and want to describe how you set it up then you have two options...
  • Write a long and detailed description of every stage in the process and expect people to read and follow the documentation.
  • You write a program to do the dirty work and expect anyone who wants to know how it works to read the program.
We are doing the latter. This is why ngs-vo-tool is written in Python - a language designed to be readable by people who don't actually know the language. Take this example from the code:

class LcmapsMapper(BaseMapper):
"LCMAPS gridmapfile and groupmapfile entries"

... snip ...

def add_mapping(self, vo, acinfo):

gridmap_e,groupmap_e = mapfile_entries(vo,acinfo)

if gridmap_e not in self.gridmap_entries:
logger.debug("Adding <%s> to gridmapfile" % gridmap_e)

if groupmap_e not in self.groupmap_entries:
logger.debug("Adding <%s> to groupmapfile" % groupmap_e)

... snip ...

As long as you can wrap your brain around Python's way of using indentation to group related code together, that is, describes the process of adding VO information to the contents of a LCMAPS gridmapfile and groupmapfile.

The ngs-vo-tool was also an attempt to apply the kind of software engineering techniques that our colleagues at the Software Sustainability Institute want to encourage in academia. In particular, we tried to use something approximating to Test Driven Development (TDD) to keep the bugs under control.

in TDD, for each bit of program code, there is a corresponding bit of test code to put it through its paces. A test harness is used to run all the tests whenever the code is changed.

If you have the source code, run:

python setup.py test

to watch all 57 tests fly past.

In proper TDD, the test is written first and run before any attempt is made to write the thing being tested. I'm afraid that I was more pragmatic and wrote the tests and code at the same time.

This approach encourages the developer to split the code into largely-self-contained modules with well defined interfaces simply because these are much easier to write tests against. Some people refer to this as ravioli code.

I would be a fool to claim that the program is bug-free - although the testing did shake out bugs early in the development process and I do think that the code is cleaner and easier to read because of this approach.

Someone at eScience centre at Southampton once came up with the phrase 'making the useful usable' as a way of describing their work. I hope that someone will find the ngs-vo-tool useful in making their services to the grid usable.

Tuesday, 24 August 2010

Application hosting environment and UI/WMS - preparations continue!

I've just added some details of more presentations at the forthcoming Innovation Forum to the website. We have two demos - one of the UI/WMS which has proved to be a big hit with our users and the Application Hosting Environment (AHE). Full details of the presentations are available on our event page on the website. Delegates will see walk-throughs of how to use these tools on the NGS and the benefits they can bring to your research.

We have also updated the UI/WMS tutorials if you want to give the UI/WMS a go beforehand!

Friday, 20 August 2010

User success stories at the NGS

The programme for the 3rd NGS Innovation Forum is really beginning to come together now. I've managed to secure presentations from 3 NGS users from completely different research areas to talk about how they have used the NGS in their work.

First up we have Zhongwei Guan from the University of Liverpool who leads a research group where many researchers use the NGS. They research into the impact of explosions on aircraft fuselages as well blasts and impacts on concrete amongst many other things. I have seen Zhongwei speak at previous events and his presentations are colourful and interesting!

Next up in a complete change of direction, we have Luke Rendell from the Centre for Social Learning and Cognitive Evolution at the University of St Andrews. Luke came to the NGS to ask for resources to run an international computer tournament on the evolution of learning. The tournament and results were so productive that the resulting paper was published in Science and was featured in New Scientist.

Finally, and by no means least, we have Narcis Fernandes-Fuentes who has used the NGS for several years to discover novel therapeutic agents. Narcis has had his research turned into a NGS user case study as an example of the type of research that can be performed on the NGS.

The deadline for poster abstracts for the event is approaching (10th Sept) so if you would like to submit an abstract and be in with a chance of winning the best poster prize, please submit soon!

Wednesday, 18 August 2010

Source code archeology

I'm afraid this is going to be technical.

For the last month or so - in the gaps between holidays, meetings and dealing with a power-glitch that has knocked-out some rather important bits of ngs.leeds.ac.uk - I've been working on the ngs-vo-tool.

The ngs-vo-tool is a utility program that does the tweaking and fiddling needed when adding or removing support for all, or part, of a particular Virtual Organisation (VO).

One of the things the ngs-vo-tool needs to tweak and fiddle is the LCMAPS version of the gridmapfile. This controls how which bits of of which VO get assigned to which local accounts and consists of entries like...

"/training.ngs.ac.uk/*" .ngstrain
"/monitoring.ngs.ac.uk/lcas_lcmaps/*" .ngsmon

This particular example assigns anyone in the NGS's Training VO to an account in the 'ngstrain' pool but only members of the 'lcas_lcmaps' group within the NGS Monitoring VO to the ngsmon pool.

Groups can contain subgroups. You can also cherry-pick VO members with a particular role or a particular capability.

Among the last features due to be added to ngs-vo-tool is one to allow any combination of group/role/capability to specified and have this turned into something fit for an LCMAPS gridmapfile.

As always, it is not a simple as it first appears.

The bit in quotes is a pattern that matches a Fully Qualified Attribute Name (FQAN).

The FQAN is a representation of VO, group, subgroup, role and capability defined in http://edg-wp2.web.cern.ch/edg-wp2/security/voms/edg-voms-credential.pdf as

with the additional complication is that the FQAN for a VO member with no role can either omit the Role=role bit or explicitly include 'Role=NULL'.

So in order for the code to do the right thing, I need to work out..
  • What the LCMAPS uses to match a pattern to a string. In particular, is it fussy about where the '*' can be placed and how many '*'s can be used.
  • How the FQAN is constructed.
and do this for the slightly elderly version of LCMAPS that some NGS sites have deployed.

So a spot of source code archeology is required and luckily, I don't need to dig too deep as CERN kindly provide access to their source code repository on the web.

The LCMAPS code, and the rest of the gLite code, can be found at http://jra1mw.cvs.cern.ch. Released versions are even conveniently 'tagged' with the version number at that release - allowing the incurably geeky to jump directly to the relevant files.

This is work in progress. So far, I've worked out that the venerable Unix fnmatch function is used to match the pattern to the string and fnmatch allows '*'s to be used anywhere.

The exact details of FQAN construction are still buried somewhere but suitable fnmatch patterns should cope with the many variants of Role.

The code is in our local version control system. It will be copied to the source code repository at NeSCForge as soon as the important bits of ngs.leeds.ac.uk are back in service.

Thursday, 12 August 2010

Innovative NGS

The more observant amongst you may have noticed a recent addition to the top level tabs on the NGS website.

As well as making sure that the NGS runs all the services required by our users and sites on a day to day basis, we are also working on services for the future. To keep you up to date with our work "behind the scenes", we have added a new Innovation section to the website.

This section currently includes information on our prototype cloud service and user interfaces such as as GSI-PuTTY and R through the NGS (Windows). Some of these services are looking for people to help develop the applications or to be the first users and to help develop them for the wider community.

We hope that you find this section interesting and useful!

Thursday, 5 August 2010

Want to know how our databases work?

Then come along to the NGS Innovation Forum in November!

We have recently announced the details of our first few presentations with Simon Collins from the NGS team at the University of Manchester bravely giving two presentations.

His first presentation on day 1 will cover using databases on the NGS. It will give an overview of the databases supported on the NGS, what offerings and advantages these give to users and real-life case studies of how NGS users are taking advantage of this service.

Simon will also be giving a second presentation on day 2 aimed more at our member sites which will look at accounting services and solutions for NGS member sites. Current and potential sites will find out how accounting is supported on the NGS, the different choices available for sites to report accounting information and the information and tools that are available to sites that choose to do so.

Keep your eye on the NGS website over the next few weeks to see more details of presentations at the IF.

Monday, 2 August 2010

Splendid Isolation

I wandered lonely as a.... typical - I come back from a week's leave with a post on cloud computing to discover that every cloud related title has already been used.

The last few weeks have seen what can only be called a storm of cloud activity - the release of the Openstack cloud software; Amazon adding a cloud tuned for High Peformance Computing applications to their collection; the publishing of two reports on academic use of clouds and, or course, the public launch of the NGS's own prototype cloud service.

Some commentators have claimed the launch of the Amazon service means that The Grid is Dead. Obviously, those of us in the grid world disagree - as Craig Lee of the Open Science Grid explains.

Alternatively, we are now all officially zombies - mmm.. brains.

People who see the cloud as a computional grid with all the nasty authentication stuff ripped out are misunderstanding the point of nasty authentication stuff in a world where not all data is open and access control and audit trains are vital.

The people who say that the cloud is the greatest thing since sliced bread are also missing the point. Most clouds are quick-and-easy ways to get your hands on a virtual machine. It is virtual machines that are the greatest thing since sliced bread.

Why? It is all about keeping things separate.

Many modern applications are bad neighbours. They do not sit happily on a server but need plumbing into web service containers such as Tomcat or Glassfish and databases such as MySQL or PostgreSQL. It is not uncommon for the applications to require specific versions of their supporting software.

Trying to persuade two such applications to co-exist on the same server is the stuff of system administrators nightmares - sitting on the sysadmin scale of horror somewhere near an operating system upgrade or running out of coffee.

Virtual Machines let you dump these applications and their hangers on in an isolated environment comparatively cheaply and easily. Isolating applications isn't a new idea - it was implemented in the mainframe systems of the 1960s - and system admins have been using techniques such as chroot jails for decades to provide some level of separation.

With apologies for borrowing great poetry to make a cheap point: Wordsworth's poem Daffodils - the one that starts 'I wandered lonely as a cloud' - mentions the 'the bliss of solitude'. It is the solutude of computing clouds that makes them useful.