It's good to be back at All Hands and catch up with all the Old Hands, and shake New Hands as well.
As usual it is necessary to clone oneself for the parallel sessions, keep track of all the discussions, keep the todo list up to date, sniff around for new things, while simultaneously keeping up with email back home, other projects ticking along at home, documents, proposals, reports, arrangements. The mental equivalent of an octopus. All in a day's work. Hey ho.
Tuesday, 14 September 2010
Seen at All Hands
The All Hands meeting exists so that those involved in one branch of e-Intrastructure / e-Science / e-Research / e-Social-Science / e-Whatever can see what those in the other e-*s are up to.
One project that particularly caught my eye was BlogMyData.
Much academic research takes place in corridors, pubs and even - occasionally - in the loo. Researchers will discuss their latest discoveries with colleagues when they bump into one another on the way to somewhere else - and get a new perspective or a new idea as they do so. Call it serendipity at work - or possibly serendipity in the bar of the Dog and Duck.
This is the kind of material that occasionally appears as `Bloggs, Fred (Personal Communcation)' in papers.
BlogMyData extends this chatter about the work to researchers who are in different institutions and so - unless they happen to be at All Hands - are very unlikely to be in the same coffee room, or pub, at the same time.
It allows researchers to post visualisations of the data they are working on blogs which can be read - and commented on - by collaborators. It combines two projects: the Godiva 2 visualisation package from Reading and the LabBlog blogging tool from platform.
This has the big advantage that the data and the conversation will be recorded for future reference unlike, say, a chat in the bar or an unexpected encounter in the gents...
One project that particularly caught my eye was BlogMyData.
Much academic research takes place in corridors, pubs and even - occasionally - in the loo. Researchers will discuss their latest discoveries with colleagues when they bump into one another on the way to somewhere else - and get a new perspective or a new idea as they do so. Call it serendipity at work - or possibly serendipity in the bar of the Dog and Duck.
This is the kind of material that occasionally appears as `Bloggs, Fred (Personal Communcation)' in papers.
BlogMyData extends this chatter about the work to researchers who are in different institutions and so - unless they happen to be at All Hands - are very unlikely to be in the same coffee room, or pub, at the same time.
It allows researchers to post visualisations of the data they are working on blogs which can be read - and commented on - by collaborators. It combines two projects: the Godiva 2 visualisation package from Reading and the LabBlog blogging tool from platform.
This has the big advantage that the data and the conversation will be recorded for future reference unlike, say, a chat in the bar or an unexpected encounter in the gents...
Hello from AHM
A selection of NGS staff are all now settled into conference mode at the UK e-Science All Hands Meeting in a wet Cardiff.
The stand was put up last night and our first demo was held this morning. A big thanks to Jonathan Churchill whose demo of the UI/WMS managed to pull in a substantial crowd despite a quiet start to the event as people continue arrive during the morning.
Jonathan will be doing a follow up demo this lunchtime on the "gLite WMS Enabled NGS Applications Portal" so pop by our stand with your lunch if you are in Cardiff!
The conference proper kicks off this afternoon with the first themes and workshops. I'll be going to the "Sharing, collaboration and interfaces for e-Research" which looks as though it will have some interesting presentations about user tools.
The stand was put up last night and our first demo was held this morning. A big thanks to Jonathan Churchill whose demo of the UI/WMS managed to pull in a substantial crowd despite a quiet start to the event as people continue arrive during the morning.
Jonathan will be doing a follow up demo this lunchtime on the "gLite WMS Enabled NGS Applications Portal" so pop by our stand with your lunch if you are in Cardiff!
The conference proper kicks off this afternoon with the first themes and workshops. I'll be going to the "Sharing, collaboration and interfaces for e-Research" which looks as though it will have some interesting presentations about user tools.
Saturday, 11 September 2010
The Featherstone-Kite Openwork Basketweave Mark Two Gentleman’s Flying Machine
In a shopping centre in the middle of Leeds, not far from the Universities, there is - or was - a big glass case.
In the case is a contraption built from wicker-work, string and cogs and lights and bits of old gramophone. Every so often, it springs to life and whirls around and plays a tune.
It is not an Yorkshire-based competitor for the iPod but a sculpture by Rowland Emmet: "The Featherstone-Kite Openwork Basketweave Mark Two Gentleman’s Flying Machine". Leeds shoppers passing by look at it and think...
We are investigating how to move the important features of the existing INCA monitoring service to a new monitoring service based on WLCG Nagios.
But - as has been said many times - grids are complicated. Which means that the software needed to monitor grids is complicated. Which means that when you start to look at the software, you spend a lot of time staring at a screen and thinking...
So after a week of staring and thinking, here is what we think the bits and pieces of the service are meant to do:
At the core sits Nagios: an open-source monitoring system familiar to many system administrators. It consists of a set of programs called 'plugins' and a scheduler that arranges for these plugins to be run.
A plugin tests if a particular service on a given host is working as expected. Plugins typically return a short message and a status code that means one of: 'OK', 'WARNING', 'CRITICAL' or - if the plugin broke - 'UNKNOWN'. They can also track performance data such as disk usage.
Nagios comes with a set of basic plugins. WLCG Nagios adds a whole raft of Grid specific ones.
In this documentation, plugins within WLCG are referred to as probes.
Next up, a 'configuration generator' called NCG takes data published about a site or set of sites and generates a configuration for Nagios that monitors them.
Statistics and performance metrics generated by the plugins/probes are collected and are delivered via a message bus to a service that stuffs them into a database. A tool called MyEGEE is used to visualise the contents of this database.
If you want to know more...
Staff from STFC and Oxford gave an NGS surgery on WLCG Nagios in late July this year. Their slides describing how WLCG Nagios can be configured and how it has been deployed can be found on the NGS web site.
There is more technical information on twiki.cern.ch in the GridMonitoringNcgOverview and GridMonitoringNcgYaim pages. More information about the plugins/probes can be found on SAMProbesMetrics.
If you want to know more about the NGS R+D activity, we will on on hand at All Hands next week.
In the case is a contraption built from wicker-work, string and cogs and lights and bits of old gramophone. Every so often, it springs to life and whirls around and plays a tune.
It is not an Yorkshire-based competitor for the iPod but a sculpture by Rowland Emmet: "The Featherstone-Kite Openwork Basketweave Mark Two Gentleman’s Flying Machine". Leeds shoppers passing by look at it and think...
What on earth is THAT meant to do?Which is a rather tortuous way of introducing the latest bit of R+D work.
We are investigating how to move the important features of the existing INCA monitoring service to a new monitoring service based on WLCG Nagios.
But - as has been said many times - grids are complicated. Which means that the software needed to monitor grids is complicated. Which means that when you start to look at the software, you spend a lot of time staring at a screen and thinking...
What on earth is THAT meant to do?
So after a week of staring and thinking, here is what we think the bits and pieces of the service are meant to do:
At the core sits Nagios: an open-source monitoring system familiar to many system administrators. It consists of a set of programs called 'plugins' and a scheduler that arranges for these plugins to be run.
A plugin tests if a particular service on a given host is working as expected. Plugins typically return a short message and a status code that means one of: 'OK', 'WARNING', 'CRITICAL' or - if the plugin broke - 'UNKNOWN'. They can also track performance data such as disk usage.
Nagios comes with a set of basic plugins. WLCG Nagios adds a whole raft of Grid specific ones.
In this documentation, plugins within WLCG are referred to as probes.
Next up, a 'configuration generator' called NCG takes data published about a site or set of sites and generates a configuration for Nagios that monitors them.
Statistics and performance metrics generated by the plugins/probes are collected and are delivered via a message bus to a service that stuffs them into a database. A tool called MyEGEE is used to visualise the contents of this database.
If you want to know more...
Staff from STFC and Oxford gave an NGS surgery on WLCG Nagios in late July this year. Their slides describing how WLCG Nagios can be configured and how it has been deployed can be found on the NGS web site.
There is more technical information on twiki.cern.ch in the GridMonitoringNcgOverview and GridMonitoringNcgYaim pages. More information about the plugins/probes can be found on SAMProbesMetrics.
If you want to know more about the NGS R+D activity, we will on on hand at All Hands next week.
Tuesday, 7 September 2010
Social Sciences and Humanities and the NGS Innovation Forum
We are pleased to say that Martin Wynne from the CLARIN project will be presenting at the forthcoming NGS Innovation Forum. We have a number of social scientists who use the NGS and this is an excellent opportunity to demonstrate to the community how the NGS can be of relevance in this research area.
Martins presentation will look at what is necessary at the national level to support CLARIN, a European infrastructure providing services relating to language resources and tools to researchers in the Humanities and Social Sciences.
Continuing with the European theme we are also pleased to announce that Steven Newhouse, Director of EGI, will also be presenting at the event. Steven will be reflecting on the experiences of the first 6 months and provide an overivew for the plans that have now been established for future years.
Registration for the Innovation Forum is now open with full details available on the event page on the NGS website.
Martins presentation will look at what is necessary at the national level to support CLARIN, a European infrastructure providing services relating to language resources and tools to researchers in the Humanities and Social Sciences.
Continuing with the European theme we are also pleased to announce that Steven Newhouse, Director of EGI, will also be presenting at the event. Steven will be reflecting on the experiences of the first 6 months and provide an overivew for the plans that have now been established for future years.
Registration for the Innovation Forum is now open with full details available on the event page on the NGS website.
Friday, 3 September 2010
The messages on the bus go back and forth
In a rare display of synchronised software development - a raft of NGS research and development projects have safely reached their final destination.
We are proudly showing off our cloud; our HARC acceptors are now accepting to an acceptable level; the new User Interfaces are on display in the Innovation section of the NGS web site; we have come up with a saner way to manage access to licensed software and the ngs-vo-tool to manage virtual organisation configuration has been released.
There will be more on HARC in a future posting. The rest have already been well and truly blogged.
This posting is about the what we are going to do with all our newly available free time in the final 7 months of the third phase of the NGS.
We are here to do the dull-but-useful bits and the dull-but-useful things we are concentrating on are those needed as the NGS and GridPP join forces to form the UK National Grid Initiative (UK NGI) within the European Grid Initiative...
Yes... you wait ages for a bus and then three turn up at once.
A message bus is simply a way getting a lump of data from A to B. The important bit is that your application does not need to know how to get from A to B - it just needs to hop on the bus at A and hop off again at B.
Finding a reliable route from A to B becomes somebody else's problem.
WLCG Nagios uses its message bus to pass the results of tests to a central monitoring service. Technical details can be found on https://twiki.cern.ch/twiki/bin/view/EGEE/UseLocalActiveMQForMessaging.
The latest APEL client sends records wrapped in messages rather than attempting immediate database updates. Again there are more technical details available from http://goc.grid.sinica.edu.tw/gocwiki/ApelHome.
In both cases, they are using the message bus to avoid information being lost when there is a backlog of data to be processed. As users of the UK public transport service are all too aware, buses are very good at waiting in traffic.
We are proudly showing off our cloud; our HARC acceptors are now accepting to an acceptable level; the new User Interfaces are on display in the Innovation section of the NGS web site; we have come up with a saner way to manage access to licensed software and the ngs-vo-tool to manage virtual organisation configuration has been released.
There will be more on HARC in a future posting. The rest have already been well and truly blogged.
This posting is about the what we are going to do with all our newly available free time in the final 7 months of the third phase of the NGS.
We are here to do the dull-but-useful bits and the dull-but-useful things we are concentrating on are those needed as the NGS and GridPP join forces to form the UK National Grid Initiative (UK NGI) within the European Grid Initiative...
- Incorporating the important features of the NGS's current 'INCA' monitoring service into a NGI one based on WLCG Nagios.
- Allowing the data collected by the NGS accounting service to be interchanged with that collected by the APEL (Accounting Processor for Event Logs) accounting service.
Yes... you wait ages for a bus and then three turn up at once.
A message bus is simply a way getting a lump of data from A to B. The important bit is that your application does not need to know how to get from A to B - it just needs to hop on the bus at A and hop off again at B.
Finding a reliable route from A to B becomes somebody else's problem.
WLCG Nagios uses its message bus to pass the results of tests to a central monitoring service. Technical details can be found on https://twiki.cern.ch/twiki/bin/view/EGEE/UseLocalActiveMQForMessaging.
The latest APEL client sends records wrapped in messages rather than attempting immediate database updates. Again there are more technical details available from http://goc.grid.sinica.edu.tw/gocwiki/ApelHome.
In both cases, they are using the message bus to avoid information being lost when there is a backlog of data to be processed. As users of the UK public transport service are all too aware, buses are very good at waiting in traffic.
Thursday, 2 September 2010
A flurry of activity
There has been a batch of new speakers recently announced for the NGS Innovation Forum '10 and further information about their presentations are now on the website. The latest presentations are NGS tool demos which will consist of walk-throughs of NGS tools using real research examples so delegates can leave the event with the knowledge of new tools to use in their research.
NGS tool demos
1. Transcriptome Analysis using the NGS User Interface /Workload Management System (UI/WMS) – Jonathan Churchill, NGS, STFC RAL
The UI/WMS is a tool which allows users to easily submit jobs to the whole of the NGS relying on the WMS to chose which NGS resources to use for their jobs. Use of the UI/WMS will be demonstrated with a user case study in which analysis time of mRNA was decreased from a month to less than 12 hours.
2. Accessing the NGS using the Application Hosting Environment (AHE) – Stefan Zasada, UCL
An overview of how access to the NGS can be simplified using the Application Hosting Environment, a lightweight application portal system.
3. Using the HERMES data management tool – David Wallom, NGS, University of Oxford
Here we will show how easy it is to install and connect into various NGS resources to move data between them, your home institution and your desktop.
4. The NGS from the CCP4 desktop – Matteo Turilli, NGS, University of Oxford
The NGS R&D theme have been working to build access to the NGS into the desktop tools that researchers use on a day-to-day basis, in this presentation we look at the example of CCP4: Software for Macromolecular X-Ray Crystallography.
We also have a presentation from the Director of the NGS -
The future of the NGS – Neil Geddes, NGS Director, STFC RAL
This presentation will look at the focus of activities for the NGS for the coming 2-3 years and possible longer term opportunities.
Remember that registration for the event is now open and that the call for poster abstracts closes on the 10th of September!
NGS tool demos
1. Transcriptome Analysis using the NGS User Interface /Workload Management System (UI/WMS) – Jonathan Churchill, NGS, STFC RAL
The UI/WMS is a tool which allows users to easily submit jobs to the whole of the NGS relying on the WMS to chose which NGS resources to use for their jobs. Use of the UI/WMS will be demonstrated with a user case study in which analysis time of mRNA was decreased from a month to less than 12 hours.
2. Accessing the NGS using the Application Hosting Environment (AHE) – Stefan Zasada, UCL
An overview of how access to the NGS can be simplified using the Application Hosting Environment, a lightweight application portal system.
3. Using the HERMES data management tool – David Wallom, NGS, University of Oxford
Here we will show how easy it is to install and connect into various NGS resources to move data between them, your home institution and your desktop.
4. The NGS from the CCP4 desktop – Matteo Turilli, NGS, University of Oxford
The NGS R&D theme have been working to build access to the NGS into the desktop tools that researchers use on a day-to-day basis, in this presentation we look at the example of CCP4: Software for Macromolecular X-Ray Crystallography.
We also have a presentation from the Director of the NGS -
The future of the NGS – Neil Geddes, NGS Director, STFC RAL
This presentation will look at the focus of activities for the NGS for the coming 2-3 years and possible longer term opportunities.
Remember that registration for the event is now open and that the call for poster abstracts closes on the 10th of September!
Subscribe to:
Posts (Atom)
Posts
