This paper presents an innovative robotic framework enabling real-time mimicry of human arm gestures with a single-arm TIAGo robot, aimed at assisting neurorehabilitation therapies for post-stroke and neurodegenerative patients. Initially, we adopted a single-camera approach utilizing MediaPipe for human pose detection; however, accuracy evaluations against ground-truth data from the OptiTrack motion capture system demonstrated substantial limitations. Specifically, root mean square error (RMSE) metrics highlighted significant inaccuracies in single-camera solutions due to depth ambiguity, especially for compact gestures with limited spatial extent and minimal lateral displacement. Consequently, we transitioned to a dual-camera RGB setup by integrating TIAGo’s onboard Xtion camera with an external RealSense D455 camera. This dual-camera configuration, combined with triangulation techniques, significantly improved spatial accuracy and reliability in 3D joint position estimation. To meet the demanding real-time requirements of gesture recognition and mimicry, we replaced MediaPipe with NVIDIA DeepStream, leveraging GPU acceleration through CUDA. Performance evaluations demonstrated that DeepStream achieves substantially higher frame rates, enabling smooth and stable pose detection. Our proposed robotic mimicry algorithm operates in two main stages: geometric approximation of joint angles based on spherical coordinates and numerical refinement using a damped pseudoinverse inverse kinematics method. Scaling is also incorporated to accommodate biomechanical differences between human and robotic limbs, ensuring gestures coherence. Simulation results with Gazebo and MATLAB validate the framework’s effectiveness, confirming accurate real-time gesture mimicry and its potential for practical applications.

REAL-TIME MIMICRY OF HUMAN ARM GESTURES ON A SINGLE-ARM TIAGO ROBOT USING RGB-BASED TRIANGULATION, AI POSE DETECTION, AND NVIDIA DEEPSTREAM / Bajrami, A., Beraldo, G., Costa, D., Palpacelli, M.C., Cortellessa, G.. - 5:(2025). (ASME 2025 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, IDETC-CIE 2025 Hilton Anaheim, CA, USA 2025) [10.1115/DETC2025-168757].

REAL-TIME MIMICRY OF HUMAN ARM GESTURES ON A SINGLE-ARM TIAGO ROBOT USING RGB-BASED TRIANGULATION, AI POSE DETECTION, AND NVIDIA DEEPSTREAM

Bajrami A.;Costa D.;Palpacelli M. C.;
2025-01-01

Abstract

This paper presents an innovative robotic framework enabling real-time mimicry of human arm gestures with a single-arm TIAGo robot, aimed at assisting neurorehabilitation therapies for post-stroke and neurodegenerative patients. Initially, we adopted a single-camera approach utilizing MediaPipe for human pose detection; however, accuracy evaluations against ground-truth data from the OptiTrack motion capture system demonstrated substantial limitations. Specifically, root mean square error (RMSE) metrics highlighted significant inaccuracies in single-camera solutions due to depth ambiguity, especially for compact gestures with limited spatial extent and minimal lateral displacement. Consequently, we transitioned to a dual-camera RGB setup by integrating TIAGo’s onboard Xtion camera with an external RealSense D455 camera. This dual-camera configuration, combined with triangulation techniques, significantly improved spatial accuracy and reliability in 3D joint position estimation. To meet the demanding real-time requirements of gesture recognition and mimicry, we replaced MediaPipe with NVIDIA DeepStream, leveraging GPU acceleration through CUDA. Performance evaluations demonstrated that DeepStream achieves substantially higher frame rates, enabling smooth and stable pose detection. Our proposed robotic mimicry algorithm operates in two main stages: geometric approximation of joint angles based on spherical coordinates and numerical refinement using a damped pseudoinverse inverse kinematics method. Scaling is also incorporated to accommodate biomechanical differences between human and robotic limbs, ensuring gestures coherence. Simulation results with Gazebo and MATLAB validate the framework’s effectiveness, confirming accurate real-time gesture mimicry and its potential for practical applications.
2025
  
File in questo prodotto:
Non ci sono file associati a questo prodotto.

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11566/367384
 Attenzione

Attenzione! I dati visualizzati non sono stati sottoposti a validazione da parte dell'ateneo

Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus 1
  • ???jsp.display-item.citation.isi??? ND
social impact