NESTOR · Next gEneration Sequence sTORage
Horizon 2020 — Marie Skłodowska-Curie Actions
- Duration
- 2017-09-01 → 2020-08-31
- EU contribution
- €246,668
- Participants
- 2
- Scheme
- MSCA-IF-GF
Lines connect the coordinator with its partners.
Results in brief
Next gEneration Sequence sTORage
A data series, is a sequence of data points ordered over a dimension (time in the case of time-series). Data series are present in virtually every scientific and social domain: from health care, astronomy and biology, to finance and the internet-of-things. It is not unusual for applications in these domains to involve numbers of sequences (or subsequences) in the order of billions. As a result, analysts are unable to handle the vast amounts of data series that they have to filter and process. It is noteworthy that common practice in various domains is to use custom data analysis solutions, usually built using higher level programming languages, such as R/Python. Such techniques, while commonly acceptable in small data processing scenarios, are unfit for larger scale data management and exploration. This is because they come in contrast to all previous database research, not taking advantage of indexes, physical data independence, query optimization, and data processing methods designed for scalability. The reason why existing systems are unsuitable for data series, and as a result, not used, is that they have been designed and optimized for fundamentally different data and query models. Current relational storage layers cannot handle the access patterns that analysts are interested in, without scanning large amounts of unnecessary data or without large processing overhead. Thus, making complex analytics inefficient. Moreover, relational query languages are not an intuitive interface for expressing data series specific queries. In order to alleviate these problems, and exploit the potential of sequential data analytics, specialized data series storage and retrieval systems have to be developed,. Such systems, will allow analysts – across different fields – to efficiently retrieve and manipulate their sequences of interest. Along these lines, our goal is to develop a specialized data series storage layer that goes beyond relational and array databases, by being able to organize data series in a way that fits specialized data series workloads. Such a system has the potential to optimize data processing pipelines in many different domains as diverse as biology, IoT data analytics, and finance to personalized medicine.
Data: CORDIS, © European Union
Project objective
Sequential data are everywhere, from DNA sequences to astronomical light curves, and from aircraft engine monitoring data to the prices of stock options. Recent advances in various fields such as those of data storage, networking and sensing technologies, have allowed organizations to gather overwhelming amounts of sequential data at unprecedented speeds.This wealth of information enables analysts to identify patterns, find abnormalities, and extract knowledge. It is noteworthy that common practice in various domains is to use custom data analysis solutions, usually built using higher level programming languages, such as R/Python. Such techniques, however, while commonly acceptable in small data processing scenarios, are unfit for larger scale data management and exploration. This is because they come in contrast to all previous database research, not taking advantage of indexes, physical data independence, query optimization, and data processing methods, designed for scalability. In these domains, database systems are used merely for storing and retrieving data and not as the sophisticated query processing systems they are.Current relational storage layers cannot handle the access patterns that analysts of sequential data are interested in, without scanning large amounts of unnecessary data or without large processing overhead. Thus, making complex analytics inefficient.In order to exploit this new opportunity, we plan to develop specialized data series storage and retrieval systems, which will allow analysts – across different fields – to efficiently manipulate the sequences of interest.The proposed research project, named NESTOR (Next gEneration Sequence sTORage), has the potential of great economic and social impact in Europe as multiple scientific and industrial fields are currently in need of the right tools, in order to handle their massive collections of data series.
Original text from CORDIS.
Participants
- UNIVERSITE PARIS CITE · ParisCoordinatorFrance
- PRESIDENT AND FELLOWS OF HARVARD COLLEGE · CambridgeUnited States
Links
Data: CORDIS, © European Union