Slip is a very valuable manipulation tool for humans, and my thesis aims to bring these capabilities to robots. Using visuotactile sensing and vibration sensing, I infer the slip direction and speed, which can in turn be used to control a gripper to let an object slip at a predetermined velocity — an until-now unsolved problem.
Robotic grasping fails silently: by the time a slipping object has visibly moved, it is often already too late to correct the grip. My thesis tackles this problem at its source, asking whether slip velocity can be estimated continuously and fast enough to close a real-time control loop. The approach fuses a GelSight visuotactile sensor, which captures rich contact geometry through an embedded camera, with a high-frequency vibration sensor that picks up the mechanical micro-events humans rely on to catch slip within milliseconds — cues that conventional camera frame rates simply cannot resolve.
Because no existing dataset captures continuous slip velocity under controlled conditions, I co-designed and built a low-cost, reproducible test bench that induces slip while recording an independent, high-precision ground truth, documenting every hardware and software component so the setup can be rebuilt and extended by others. On top of this dataset, I study which sensor modalities and data representations (raw signal, spectrogram, learned tactile embeddings, and others) are necessary and sufficient for accurate velocity estimation, using a branching multimodal network with modality dropout to quantify each input’s contribution.
The core modeling contribution is a multimodal fusion transformer that tokenizes self-supervised tactile features and vibration spectrograms with modality and relative-time embeddings, letting the model reason over asynchronous, multi-rate sensor streams. Compared to a conventional late-fusion baseline, this approach improves both in-distribution accuracy and out-of-distribution robustness; properties that matter more in deployment than benchmark performance alone. I evaluate the full pipeline against the latency budget a reactive grasp controller would require, and close the loop conceptually with an open-loop replay study of a proportional controller driven by the predicted slip velocity.
Technologies: Python, ROS2, PyTorch, GelSight, force/torque sensing, CAD, Docker
You can view the slides from my thesis defense here.