My master’s thesis at TU Darmstadt investigates how robots can measure how fast an object slips through a gripper—a step toward deliberately controlling sliding motion during manipulation. I combine GelSight images and tactile marker motion with vibration signals: the camera captures contact detail, while vibrations reveal rapid stick-slip events that are difficult to resolve at camera frame rates.
I co-developed a test bench with the mechanical engineering faculty to collect labelled slip data under controlled grip forces and pulling speeds. A separate camera tracks a marker on the object to provide ground-truth velocity. Using these recordings, I compared late-fusion, Transformer, and hybrid neural networks, and tested how sensor inputs, model size, and prediction horizon affect accuracy and latency.
The best hybrid model achieved a mean absolute error of 2.13 mm/s during slip. Removing vibration increased slip-velocity error by 20–21%, demonstrating the benefit of combining the sensors. Training the model to predict 50 ms ahead yielded an error of 3.35 mm/s, suggesting a way to anticipate slip and compensate for sensing delays. I also explored a fast vibration-driven architecture conditioned on slower visuotactile context to reduce inference latency.
The evaluation was limited to the test bench: generalisation across materials and rotations, robustness to robot motion, and closed-loop gripper control remain future work.
Technologies: Python, ROS2, PyTorch, GelSight, vibration sensing, CAD
View the thesis defense presentation.