DIME · Distributed Inference for Energy-efficient Monitoring at the Network Edge

Distributed Inference for Energy-efficient Monitoring at the Network Edge

Deep Learning (DL) inference on data from end devices such as IoT sensors, smartphones, and drones enhances operational efficiency and functionality in sectors like industrial automation, smart cities, remote healthcare, and smart agriculture. Performing the DL inference locally on the devices, aka edge ML inference, promises to boost energy efficiency, responsiveness, and alleviate privacy concerns. Nevertheless, the research emphasis had been in design of embedded DL models with higher accuracy without systematically studying their latency and energy consumption on the devices. Consequently, existing distributed DL inference techniques does not achieve important trade-offs between accuracy, latency, and energy consumption. In this context, the DIME project has made significant contributions toward building efficient edge ML systems. In order to understand the trade-offs between accuracy, latency, and on-device energy consumption per inference, in this project we conducted a comprehensive measurement study on multiple devices that span different processor types, including CPU, GPU and TPU, using datasets of varying complexity. Further, to assess the performance of distributed DL inference between a device and a server, we also conducted measurements for offloading the data samples using different communication protocols WiFi and Bluetooth, and the measurements for runtimes of state-of-the-art DL models on two distinct servers, one with NVDIA Tesla GPU and the other with A100 GPU. We also studied the algorithmic problem of distributing the inference tasks between devices and the edge servers. We proposed a new inference load balancing algorithm that maximizes the inference accuracy while satisfying a delay constraint for the application. Further, we proposed a new distributed DL inference framework called Hierarchical Inference (HI). Using this framework, we did significant work on improving the efficiency of the DL inference systems. We also evaluated the performance of the HI systems using the measurements. The algorithmic strategies proposed in DIME enable reliable and energy-efficient inference on devices by augmenting their capabilities with large DL models in the cloud leading to the large-scale adoption of edge AI systems with significant societal and economic benefits

Data: CORDIS, © European Union