Prediction of Slip Velocity for Control in Robotic Grippers

Table of Contend

ToDo

Human manipulation needs slip control

Research Question

Can slip velocity be estimated from visuotactile and vibration data?

Related Work

Chen, Prepscius, Lee & Lee, IEEE RA-L 6(2) (2021)

Waltersson & Karayiannidis, arXiv:2606.11952 (2026)

Related Work - Waltersson

Optical and Hall effect sensors
Error ranges from less than 1% of slip velocity on wood
to 100% on translucent materials
Impressive manipulation for selected materials

Related Work - Chen

BioTac sensors
Slip estimation with an average error of 5 mm/s
1.5 ms Inference time
These will be our goal metrics

Visuotactile sensing

Limits of visuotactile sensing

Smooth surfaces
High latency
Low framerate can’t detect
- Fast movement
- Slip vibrations

Stick-slip

Every material is rough on a microscopic scale
Relative movement causes stick-slip vibrations
Faster sliding ∝ higher frequency
Faster sliding => different frequency components

Hanaor, Gan & Einav, Tribology International 93 (2016)

Giovani Luís Rech, youtu.be/1fHGTir78Hk

Test bench

Built with the mechanical engineering faculty
Gripper at the top, object pulled down by a spindle
Slip speed set by the spring and the motor speed
Grip force and pull velocity are varied
Built to precisely lable data with slip velocity lables.
Webcam captures marker to determine ground truth

Inputs and output

Visuotactile

precise, directional
low framerate, data heavy

raw image

marker flow

depth map

Vibration

raw acceleration,
128 sample windows

Slip velocity

movement of the object relative to the gripper

Research questions

Does slip velocity estimation work?
Is fusion needed?
Does modelling the slip spike help?
Can latency be reduced?
Can future slip be predicted?
Does pre-training help?
Does a minimal input reduce latency?
Do more representations help?
Does architecture size matter?

Research questions

Does slip velocity estimation work?
Is fusion needed?
Does modelling the slip spike help?
Can latency be reduced?
Can future slip be predicted?

Appendix

Does pre-training help?
Does a minimal input reduce latency?
Do more representations help?
Does architecture size matter?

Fusion

MLP-based late fusion model
Concatenates encodings
Double-head architecture

Transformer

Hybrid

Does slip velocity estimation work? Yes

Best is Hybrid at 2.13 mm/s slip-MAE
Below goal error of 5mm/s used by Chen et al. for control.
Exact cause for the performance difference hard to pin down
Possibly: Tokens give improvement, but only when time is not encoded

Is fusion needed? Yes

Drop the vibration sensor: slip-MAE +20–21 %
Drop the visuotactile input: +5–16 %
Drop marker flow on top: +23–30 %
=> Vibration alone is already a strong modality
GelSight is likely still needed for directionality

Does modelling the slip spike reduce the error? Event

parametric term in the output head to model slip spike

Does modelling the slip spike reduce the error? No

The peak arrives on average 36 ms after onset
Every network underpredicts the spike
Event and twin Event worse than Hybrid
Might work with more complex models,
But results encourage other avenues for improving performance

How can latency be reduced? Rate-Core

fast causal vibration core, conditioned by slow visuotactile context

Can latency be reduced? Yes

Depth is expensive
Inference is dominated by the shared CNN image stem
Latency correlates with parameter size
Rate-core is the only one that meets the 1.5ms budget from Chen et al.

Can future slip be predicted? Surprisingly, yes

Hybrid trained on time-shifted labels,
with time shift of: 50 / 100 / 200 ms
timeshift of 50ms:
slip-MAE 3.35 mm/s, +57 %
100ms usable for anticipating slip

Limitations

No material invariance
No rotation invariance
Only on the test bench
- Robustness to robot arm noise?
- Effect of the hand moving while gripping?

Böhm, A., Schneider, T., Belousov, B., Kshirsagar, A., Lin, L., Doerschner, K., … & Peters, J., What matters for active texture recognition with vision-based tactile sensors, in 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 15099–15105, IEEE (2024).

youtu.be/KWCL-jCnivI, arx-x.com

Outlook

More variance

Combinations of architecture

Slip velocity control

Thank you for your attention, self-, cross-, and otherwise.I hope you’ll let that pun slide and stick around for the questions.

Apendix

Does pre-training improve performance? Twin

Cross-modal self-supervised pre-training of encoders

Does pre-training improve performance? Not measurably

Labels are expensive, raw recordings are not: the point of pre-training
Pre-trained on 10 / 30 / 203 files, each against scratch on the same budget
Best case, the full corpus: 4 %
And the pre-trained net sees the data twice, once to pre-train and once to train

How can latency be reduced?

Strip no image stream at all, physically motivated scalars instead

Do more modalities help? No

depth added to Fusion (L), Transformer, Hybrid

Accuracy gets worse: +4.4 %, +1.9 %, +0.7 % slip-MAE
End-to-end latency +49–57 %, about 32–38 ms
Detection unchanged
The engineered vibration representations show no consistent effect either

Does architecture size matter? No

Width-scaled ladders, ~1 % to ~20 % of the base parameter count
Hybrid S beats Fusion L by 10 % with 13.5× fewer parameters
20 % of the parameters buys 90 % of the performance
Hybrid and Transformer lose accuracy at the top rung: overfitting
Architecture beats parameter count

`Evidence`

Detection is solved within the first few epochs

The error is undershoot, not overshoot

Returns diminish up the size ladder

The onset spike, event by event

What a prediction actually looks like

The spread between recordings

Where does the error come from? The fast tail

Slip frames are 30 % of the test set but carry 70–80 % of the error
MAE grows with velocity: 0.9 mm/s below 1.5 mm/s, 9 mm/s above 14.8 mm/s
Cause is the target, not the network: log(1+v) under a masked Huber loss compresses the fast end
The curve has the same shape for all nine networks