AI-Based Techniques for Enhancing Human-Robot Collaboration

This PhD thesis advances human-robot collaboration by developing an adaptive robotic assistant that ensures safe, seamless, and productive collaboration between humans and machines. In modern industry is not possible to apply full automation in some contests, humans contribute with essential knowledge and skills to specialized processes, while robots can assist by reducing physical strain, improving workflow, and enhancing operator well-being. This research leverages artificial intelligence techniques such as machine learning, deep learning, and large multimodal models to create a collaborative workstations where humans and robots work together intuitively, sharing tasks and workspace as one team. The proposed robotics systems are equipped with advanced perception capabilities, allowing them to detect, interpret, and respond to human actions and inputs in real time. Using sensors and cameras data, the robot adapts dynamically to the operator’s movements and intentions, prioritizing safety and fluency while optimizing workflow and minimizing the human effort. This capability was tested in various real-world robotic workstations, such as assembly and manufacturing tasks, where the robot provided safety and efficiency enhancements. A starting point of this work includes a human body pose perception system using RGB-D cameras, tracking joint positions in real time at 18 Hz with accuracy validated against an IMU wearable based system. By identifying the operator’s position, the robot maintains a safe distance and dynamically avoids collisions in various scenarios. Building on this skeleton tracking and obstacle avoidance framework, the thesis further integrates gesture-based tool selection and automatic tool retrieval, allowing the robot to respond fluidly to operator commands and to avoid it, effectively reversing the traditional human-machine interaction model. The research further developed, in collaboration with the University of Coimbra, D-RMGPT, a system employing large multimodal models for assembly tasks. This system reduces assembly time by 33% for novices and offers high flexibility through viiimage-based prompts without the need for extensive data training. Additionally, a case study involving a process execution, in this circumstance as an example a pizza preparation, demonstrates the use of a GPT model to compose the process sequence and manage the robot actions. In fact, receiving the operator’s inputs such as ingredient selection via voice/gestures and recognizing the human actions, the robot can adjust its responses accordingly to provide assistance to the operator during the process. Together, all these contributions establish a robust framework for adaptable, intuitive real-time human-robot collaboration that improves the human wellbeing and productivity