SSDM · Deployable Decision-Making: Embracing Semantics for Robotic Safety in Everyday Scenarios

Deployable Decision-Making: Embracing Semantics for Robotic Safety in Everyday Scenarios

As robots move from traditional industrial environments into everyday human-centric settings, their ability to act safely increasingly depends on more than simply avoiding collisions or respecting predefined analytical constraints. Robots must develop a semantic understanding of their operating environment and take actions that are contextually appropriate. For instance, if a robot is assigned the task of heating food in a microwave, it must not only identify the microwave and the food but also understand that metallic materials should be removed beforehand to avoid potential fire hazards. Recent progress in computer vision and machine learning has enabled robots to extract rich semantic information from sensory inputs such as RGB-D images and language. However, embedding this semantic knowledge into safe and contextually appropriate behaviour remains an open challenge. A significant bottleneck arises at the interface between the perception and action modules of robotic systems. While various safe control frameworks enable robots to adhere to safety constraints, these constraints are often assumed to be provided in particular analytical forms beforehand (e.g., as control barrier functions). Bridging the gap between semantic understanding and safe action execution necessitates translating semantic constraints from perception into explicit functions defined in the robot's state and action space. This translation process must account for perceptive uncertainties (e.g., sensor noise and incomplete maps), dynamic environmental conditions, and possible interactions with humans in shared spaces. Addressing these complexities is pivotal to achieving the safe and reliable deployment of robotic systems in real-world, everyday scenarios. This project aims to address the challenge of semantically safe robot decision-making with the following actionable steps: (1) providing a comprehensive review of semantics-driven robot decision-making frameworks as well as the necessary perception and safe control building blocks, (2) exploring efficient environment representations that facilitate downstream language-conditioned contextual reasoning and semantically-informed decision-making, (3) developing the theoretical foundation for seamlessly integrating perception and action modules to enable semantically safe actions, (4) deriving uncertainty-aware approaches to account for perceptive uncertainties and environment variations, and (5) demonstrating the effectiveness of the overall approach in real-world scenarios. Through these steps, we aim to establish the theoretical foundations and algorithmic tools necessary for semantically safe robot decision-making. These advancements will represent a step towards designing safe and efficient decision-making algorithms for robots operating in unstructured, human-centric environments, where intelligent and reliable robotic systems can not only improve daily life but also support human counterparts in specialized applications (e.g., collaborative construction, manufacturing, and healthcare).

Data: CORDIS, © European Union