Manufacturing is shifting from autonomous robots to collaborative robots (cobots) that work alongside humans. To foster effective human-robot collaboration, the robot must be aware of the operator's needs and be able to adapt accordingly. A fundamental step in this collaboration is the robot's ability to recognize and understand the operator's actions online. This study aims to develop a computer vision system to recognize and classify human actions in a collaborative assembly scenario. The goal has been to create a lightweight action recognition model, by investigating various models, including Convolutional Neural Networks (CNNs) and Long Short-Term Memory (LSTM) networks, combined with YOLO-based object recognition. This work presents several model architectures designed to classify human actions based on different types of input data. Specifically, the models process hand landmark data from an RGB-D camera. Four actions commonly found in manufacturing environments are classified. A custom dataset was created from scratch to support this research. From the analysis of the different approaches, the best-performing model achieved an offline action recognition accuracy of 95%, and an online accuracy exceeding 90%. Comparing different methodologies and input data for human action recognition helps to highlight their strengths and limitations, enabling the identification of the most effective strategy for recognizing actions in human-robot collaboration. This approach contributes to significant advancements in developing more intuitive and efficient collaborative environments within manufacturing settings. Data, code, and results are available at: https://github.com/grgzpp/human-action-recognition-and-anticipation.

Recognizing Human Actions in Collaborative Human-Robot Assembly Manufacturing / Zoppi, G., Forlini, M., Palmieri, G., Neto, P.. - (2025). (30th IEEE International Conference on Emerging Technologies and Factory Automation, ETFA 2025 Porto, Portugal 9 - 12 September 2025) [10.1109/ETFA65518.2025.11205748].

Recognizing Human Actions in Collaborative Human-Robot Assembly Manufacturing

Zoppi G.
Primo
;
Forlini M.;Palmieri G.;
2025-01-01

Abstract

Manufacturing is shifting from autonomous robots to collaborative robots (cobots) that work alongside humans. To foster effective human-robot collaboration, the robot must be aware of the operator's needs and be able to adapt accordingly. A fundamental step in this collaboration is the robot's ability to recognize and understand the operator's actions online. This study aims to develop a computer vision system to recognize and classify human actions in a collaborative assembly scenario. The goal has been to create a lightweight action recognition model, by investigating various models, including Convolutional Neural Networks (CNNs) and Long Short-Term Memory (LSTM) networks, combined with YOLO-based object recognition. This work presents several model architectures designed to classify human actions based on different types of input data. Specifically, the models process hand landmark data from an RGB-D camera. Four actions commonly found in manufacturing environments are classified. A custom dataset was created from scratch to support this research. From the analysis of the different approaches, the best-performing model achieved an offline action recognition accuracy of 95%, and an online accuracy exceeding 90%. Comparing different methodologies and input data for human action recognition helps to highlight their strengths and limitations, enabling the identification of the most effective strategy for recognizing actions in human-robot collaboration. This approach contributes to significant advancements in developing more intuitive and efficient collaborative environments within manufacturing settings. Data, code, and results are available at: https://github.com/grgzpp/human-action-recognition-and-anticipation.
2025
9798331553838
File in questo prodotto:
File Dimensione Formato  
Zoppi_Recognizing-Human-Actions-Collaborative_2025.pdf

Solo gestori archivio

Tipologia: Versione editoriale (versione pubblicata con il layout dell'editore)
Licenza d'uso: Tutti i diritti riservati
Dimensione 1.42 MB
Formato Adobe PDF
1.42 MB Adobe PDF   Visualizza/Apri   Richiedi una copia

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11566/367240
Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus ND
  • ???jsp.display-item.citation.isi??? ND
social impact