DGET · Data-Grid Environment and Tools for Distributed Management and Analysis of Large Sets of Scientific Data

Final Activity Report Summary - DGET (Data-Grid Environment and Tools for Distributed Management and Analysis of Large Sets of Scientific Data)

Grid based computational infrastructure has emerged as a natural response to the demands of computing applications worldwide which are becoming more complex and demanding in terms of resources as well as increasingly collaborative and interdisciplinary. We have concentrated on two aspects of Grid computing: firstly, devising a framework for efficient, scalable and robust computing for a system which by its nature is dynamic, heterogeneous and decentralised; secondly, providing a user-friendly environment which hides the complexity of the Grid, and allows the application scientist to interact with their data. The first challenge has been addressed using the concepts of peer-to-peer (P2P) computing. Usually Grid systems were designed to run sophisticated applications with intensive computing and storage needs across the traditional organisational boundaries and are characterised by sophisticated resource management and data transfer components. P2P systems on the other hand were mainly designed for resource sharing, mostly files, and therefore focuses on sophisticated resource discovery capabilities. Each approach has its own advantages and disadvantages and our DGET project combines ideas from both approaches. We developed an efficient and robust virtual network topology that can manage resources in a dynamic computing environment where nodes can join or leave the Grid without warning. Nodes that join the system are organized in a virtual tree-hierarchy topology following a model known as B+tree. We have shown that by choosing a tree topology and having the lower branches report solely to their parents, the diameter of the network only increases logarithmically with the number of nodes. Updates due to nodes migration only introduce small overheads. We have performed several experiments on systems of heterogenous computers that confirm the results of our theoretical work. We have produced a prototype that allows one implement DGET on a given system. Most Grid based workflow management systems rely on a centralised workflow enactment engine to take care of the execution of the entire workflow application. Our approach to the problem though is fully distributed. The graphical interface to DGET is divided into four modules: a workbench for viewing and monitoring user application workflow; a desktop for building user application and defining its corresponding workflow; the DGET environment for defining and setting a specific user environment; and finally, a DGET monitor for discovering, pooling, and managing grid resources. The second challenge has been to hide the complexity of this approach from the non-computer scientist. The test-bed application chosen was in the area of particle physics where the volumes of data are so large (petabytes) as is the CPU required to analyse it, that distributed computing using the Grid is the only viable solution and is a major research topic at CERN, the European Centre for Particle Physics, and our partner organisation on this project. We have designed a middleware layer called 'Feicim' (meaning 'I see' in Gaelic) which hides the code complexity and provides a unified representation of data and algorithms through an intuitive interactive GUI. Specifically Feicim allows: viewing of all distributed data files in a tree structure; selection and viewing (e.g. histogramming) of data on these files; discovery and selection of available algorithms (filters) that can be applied to data; implementation of algorithms either in real time on local resources or on the Grid; return of output results as data-files or histograms. The GUI and datafile-browsing of Feicim have been integrated into the core code of the LHCb experiment, a collaboration of over 500 physicists from 48 institutes in 13 countries and it allows them locate their data of interest in a simple intuitive fashion. Work is on-going to incorporate the other aspects of Feicim and DGET to allow simple data analysis in a robust environment.

Текст от CORDIS, на английски · Данни: CORDIS, © Европейски съюз