ICRA@40 — Visuo-Tactile Sensing with Haptic Feedback for Teleoperated Manipulation

This work integrates visuo-tactile sensing with haptic feedback for teleoperated robot manipulation. Both arms of a TIAGo humanoid robot are controlled through an HTC Vive VR setup combined with MANUS haptic feedback gloves; GelSight visuo-tactile sensors on the gripper measure contact, and the measured contact state is rendered back to the operator’s fingertips in real time as vibration signals.

Contact forces are estimated from the sensor images with two complementary methods: tracking the movement of markers embedded in the sensor surface, and a deep-learning estimator. On the feedback side, a ROS node streams per-finger force values to a modified MANUS SDK client driving the gloves' vibrotactile actuators. The work also introduces a novel setup for evaluating normal force, shear force, and slip (arXiv:2404.19585).

I presented this work as the primary author at the IEEE ICRA@40 conference in Rotterdam (2024).

Technologies: computer vision, haptic motor control, virtual reality, teleoperation, ROS, Python