NEO-QE · NEtwork-aware Optimization for Query Executions in Large Systems

NEtwork-aware Optimization for Query Executions in Large Systems

Efficient execution of query operations is crucial for the overall performance of a data system, and one of the main performance challenges in large distributed environments is the network communications. In the past years, significant performance improvements have been achieved by using state-of-the-art methods, designed in the data management and data communication domain. However, all the proposed techniques in both fields just view each other as a black box, and the additional gains in performance from a co-optimization perspective have not yet been explored. This project aims to bridge the gap and further improve the performance of query execution in large distributed systems. Data warehouses and data centres are the foundation of computing infrastructure for data analytics. The project aims to improve the performance of one of the core tasks in the scenarios, i.e., query execution. The developed system can either make applications faster or make time-restricted applications complete within deadlines, and consequently big data users will benefit from the designs on making decisions in a more efficient way. In the meantime, using energy-efficient data movement strategies, money cost on power consumption (e.g., by switches and cooling systems) and carbon emissions of the underlying hardware in data centers will be also efficiently reduced. Moreover, since the proposed system is designed on the basis of the coflow model, the proposed techniques on data flow-scheduling will be able to be applied to current big data computing platforms such as Hadoop and Spark, and consequently contribute to the big data community. The project focuses on mathematically modeling the co-optimization problem between application-level data locality assignments and network-level data communications for query executions, and on that basis to design, development and analysis of a novel query execution system to make sure that a best runtime performance can be achieved. Moreover, to meet query latency/deadline/energy requirements in different application scenarios, bandwidth reserving and routing strategies in complex networks will be also explored. The final developed system will either make big data analytics faster or complete within deadlines, with energy consumption being efficiently reduced.

Data: CORDIS, © European Union