Prediction of Slip Velocity for Control in Robotic Grippers

via Transformer-Based Fusion of Visuotactile Data, Image Data, and Vibration Signals

Noah Becker

Intelligent Autonomous Systems

18.09.2026

Table of contents

  1. Motivation and research question
  2. Slip sensing: related work and visuotactile limits
  3. Stick-slip vibrations
  4. Test bench, inputs, and ground truth
  5. Networks and evaluation questions
  6. Results
  7. Limitations and outlook

Human manipulation needs slip control

Human manipulation needs slip control

Human manipulation needs slip control

Phone video: self-recorded. Video sources: 1, 2

Research Question

Can slip velocity be estimated from visuotactile and vibration data?

Related Work - Waltersson

Related Work - Waltersson

  • Optical and Hall effect sensors
  • Error ranges from less than 1% of slip velocity on wood
  • to 100% on translucent materials
  • Impressive manipulation for selected materials

Related Work - Chen

  • BioTac sensors
  • Slip estimation with an average error of 5 mm/s
  • 1.5 ms Inference time
  • These will be our goal metrics

Visuotactile sensing

Visuotactile sensing

Limits of visuotactile sensing

  • Smooth surfaces
  • High latency
  • Low framerate can’t detect
    • Fast movement
    • Slip vibrations

Vibration is everywhere

Vibration is everywhere

Video sources: 3, 4, 5

Stick-slip

  • Every material is rough on a microscopic scale
  • Relative movement causes stick-slip vibrations
  • Faster sliding ∝ higher frequency
  • Faster sliding => different frequency components

Hanaor, Gan & Einav, Tribology International 93 (2016)

Giovani Luís Rech, youtu.be/1fHGTir78Hk

Test bench

  • Built with the mechanical engineering faculty
  • Gripper at the top, object pulled down by a spindle
  • Slip speed set by the spring and the motor speed
  • Grip force and pull velocity are varied
  • Built to precisely lable data with slip velocity lables.
  • Webcam captures marker to determine ground truth

Inputs and output

Visuotactile

  • precise, directional
  • low framerate, data heavy

raw image

marker flow

depth map

Vibration

  • raw acceleration,
    512 sample windows

Slip velocity

  • 1D movement of the object relative to the gripper

A dataset for slip velocity

  • Continuous slip velocity, rather than only slip/no-slip labels
  • Synchronized modalities with independent optical ground truth
  • 290 controlled runs—about three hours across varied surfaces, grip forces, and pull speeds
  • No directly comparable benchmark combines visuotactile sensing, high-frequency vibration, and continuous velocity labels
Dataset

Research questions

  1. Does slip velocity estimation work?
  2. Is fusion needed?
  3. Does it transfer to an unseen surface?
  4. Can latency be reduced?
  5. Can future slip be predicted?
  6. Does modelling the slip spike reduce the error?
  7. Does pre-training improve performance?
  8. Do more modalities help?
  9. Does architecture size matter?

Research questions

  1. Does slip velocity estimation work?
  2. Is fusion needed?
  3. Does it transfer to an unseen surface?
  4. Can latency be reduced?

Appendix

  1. Can future slip be predicted?
  2. Does modelling the slip spike reduce the error?
  3. Does pre-training improve performance?
  4. Do more modalities help?
  5. Does architecture size matter?

Branch encoders

Input preparation

Six consecutive GelSight frames are stacked.

Marker flow is rasterized onto a 7 × 9 grid.

Vibration is Fourier-transformed with a short-time Fourier transform (STFT).

The vibration summary contains root mean square (RMS) amplitude, zero-crossing rate, spectral centroid, and dominant peak frequency.

Fusion

Transformer

Hybrid

Does slip velocity estimation work? Yes

  • Best is Hybrid at 2.13 mm/s slip-MAE
  • Below goal error of 5mm/s used by Chen et al. for control.
  • Exact cause for the performance difference hard to pin down
  • Possibly: Tokens give improvement, but only when time is not encoded

Is fusion needed? Yes

  • Drop the vibration sensor: MAE +61–91 %
  • Drop the visuotactile input: MAE +2–26 %
  • Slip-MAE rises +72 % without vibration, versus +8 % without GelSight
  • ⇒ Vibration alone is already a strong modality
  • GelSight is likely still needed for directionality and material detection

Does it transfer to an unseen surface? Not yet

  • Train on the dotted surface; test on the diamond surface
  • Slip-MAE rises by 64–102 % across the four networks
  • Overall MAE rises by 40–72 %
  • Slip detection largely survives: at most a 0.9 % drop

How can latency be reduced? Rate-Core

fast causal vibration core, conditioned by slow visuotactile context

Can latency be reduced? Yes

  • Inference latency is mostly the shared CNN image stem

  • Rate-core is the only one that meets the 1.5ms budget from Chen et al.

  • Slip-MAE is 17% higher for Rate-Core than Hybrid (2.49 vs. 2.13 mm/s)

Limitations

  • No material invariance
  • No rotation invariance
  • Only on the test bench
    • Robustness to robot arm noise?
    • Effect of the hand moving while gripping?

Böhm, A., Schneider, T., Belousov, B., Kshirsagar, A., Lin, L., Doerschner, K., … & Peters, J., What matters for active texture recognition with vision-based tactile sensors, in 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 15099–15105, IEEE (2024).

youtu.be/KWCL-jCnivI, arx-x.com

Outlook

More variance

Combinations of architecture

Slip velocity control

Thank you for your attention, self-, cross-, and otherwise.I hope you’ll let that pun slide and stick around for the questions.

Appendix

Fusion architecture

Transformer architecture

Hybrid architecture

Rate-Core architecture

Event architecture

Absolute error by architecture

Overall accuracy table

Network Params [M] Inputs Slip-MAE [mm/s] MAE [mm/s] No-slip-MAE [mm/s] RMSE [mm/s] Gate AUROC Fwd [ms] E2E [ms] Rate [Hz]
Hybrid 0.94 T+F+A 2.13 0.76 0.21 2.68 0.999 8.7 62.2 16
Event 3.38 T+A+S+U+F 2.23 0.80 0.24 2.74 0.999 12.9 66.3 15
Evidence 2.58 T+A+S+U+F 2.29 0.93 0.39 3.26 0.999 13.7 67.2 15
Twin 0.66 T+F+A 2.32 0.84 0.24 3.39 0.999 9.2 62.6 16
Fusion (L) 3.38 T+A+S+U+F 2.35 0.88 0.29 3.50 0.999 12.7 66.2 15
Transformer 1.09 T+A+S+U+F 2.37 0.85 0.25 3.43 0.999 10.0 63.5 16
Fusion (S) 0.18 T+A+S+U+F 2.40 0.87 0.27 3.71 0.999 7.1 60.6 17
Strip 0.24 A+F 2.49 0.91 0.27 3.70 0.999 2.5 55.9 18
Rate-Core 0.70 T+F+A 2.49 0.89 0.25 4.00 1.000 9.6 63.0 16

Inputs: T tactile frames, D depth, F marker flow, A vibration sensor, S vibration spectrogram, U vibration summary.

Relative slip-frame error

Surface-transfer table

Network In-domain slip-MAE [mm/s] In-domain MAE [mm/s] Unseen-surface slip-MAE [mm/s] Unseen-surface MAE [mm/s] Gate AUROC
Transformer 1.25 0.46 2.04 0.79 0.988
Hybrid 1.22 0.45 2.47 0.67 0.989
Fusion (S) 1.19 0.45 2.40 0.62 0.988
Fusion (L) 1.20 0.45 2.19 0.69 0.981

Slip-gate learning curves

Error decomposition

Tail-error behaviour

Under- and over-prediction

Sensor ablation

Sensor-ablation table

Network Input set Slip-MAE [mm/s] Gate AUROC
Transformer all sensors 1.25 1.000
accelerometer only 1.45 0.990
GelSight only 1.50 0.999
GelSight, no marker flow 1.54 0.999
Hybrid all sensors 1.22 1.000
accelerometer only 1.28 0.995
GelSight only 1.48 0.999
GelSight, no marker flow 1.59 0.999
Fusion (S) all sensors 1.19 0.998
accelerometer only 1.30 0.995
GelSight only 1.43 0.999
GelSight, no marker flow 1.49 0.999

Depth-input ablation

Depth-ablation table

Network No-depth slip-MAE +Depth slip-MAE Change [%] No-depth E2E [ms] +Depth E2E [ms] No-depth AUROC +Depth AUROC
Hybrid 2.13 2.14 +0.7 62.2 97.6 0.999 0.999
Fusion (L) 2.35 2.46 +4.4 66.2 103.8 0.999 0.999
Transformer 2.37 2.42 +1.9 63.5 95.2 0.999 1.000

All errors are in mm/s; latency is end-to-end cost per frame.

End-to-end latency

Slip-spike metrics

Slip-spike metrics table

Network Peak MAE [mm/s] Peak ratio Recall@70 In-event false spike [%] No-slip false spike [%]
Hybrid 7.54 0.830 0.717 5.9 1.2
Event 8.01 0.759 0.641 7.3 1.4
Evidence 9.55 0.731 0.582 7.0 4.3
Twin 8.35 0.769 0.693 6.6 1.5
Fusion (L) 10.17 0.703 0.502 7.4 1.7
Transformer 7.24 0.816 0.709 7.3 1.5
Fusion (S) 9.47 0.732 0.590 8.1 1.7
Strip 9.01 0.804 0.665 8.6 2.1
Rate-Core 9.07 0.779 0.681 7.5 1.5

Future-slip prediction horizon

Future-slip horizon table

Horizon [ms] All MAE Slip MAE No-slip MAE Persistence Skill Peak ratio Recall@70 No-slip false spike [%] Gate AUROC Gating change [%]
0 0.77 2.14 0.23 — — 0.824 0.721 1.4 0.999 +11.7
50 1.27 3.35 0.46 1.71 0.253 0.733 0.562 3.4 0.999 -2.3
100 1.56 3.84 0.67 2.19 0.288 0.682 0.478 5.3 0.998 -9.8
200 2.09 4.25 1.25 2.77 0.245 0.562 0.274 15.3 0.987 -17.4

MAE values are in mm/s. Onset metrics are dimensionless unless marked as percentages.

Can future slip be predicted? Surprisingly, yes

  • Hybrid trained on time-shifted labels,
    with time shift of: 50 / 100 / 200 ms
  • timeshift of 50ms:
    slip-MAE 3.35 mm/s, +57 %
  • 100ms usable for anticipating slip

Metric-rank disagreement

Does modelling the slip spike reduce the error? Event

parametric term in the output head to model slip spike

Does modelling the slip spike reduce the error? No

  • The peak arrives on average 36 ms after onset
  • Every network underpredicts the spike
  • Event and twin Event worse than Hybrid
  • Might work with more complex models,
  • But results encourage other avenues for improving performance

Does pre-training improve performance? Twin

Cross-modal self-supervised pre-training of encoders

Does pre-training improve performance? Not measurably

  • Labels are expensive, raw recordings are not: the point of pre-training
  • Pre-trained on 10 / 30 / 203 files, each against scratch on the same budget
  • Best case, the full corpus: 4 %
  • And the pre-trained net sees the data twice, once to pre-train and once to train

Pre-training efficiency table

Training Files Slip-MAE [mm/s] MAE [mm/s] Gate AUROC
Fine-tuned 10 3.66 1.27 0.995
30 3.25 1.14 0.997
All (203) 2.32 0.84 0.999
From scratch 10 3.87 1.66 0.994
30 3.17 1.15 0.998
All (203) 2.43 0.87 0.999

How can latency be reduced?

Strip no image stream at all, physically motivated scalars instead

Do more modalities help? No

depth added to Fusion (L), Transformer, Hybrid

  • Accuracy gets worse: +4.4 %, +1.9 %, +0.7 % slip-MAE
  • End-to-end latency +49–57 %, about 32–38 ms
  • Detection unchanged
  • The engineered vibration representations show no consistent effect either

Does architecture size matter? No

  • Width-scaled ladders, ~1 % to ~20 % of the base parameter count
  • Hybrid S beats Fusion L by 10 % with 13.5× fewer parameters
  • 20 % of the parameters buys 90 % of the performance
  • Hybrid and Transformer lose accuracy at the top rung: overfitting
  • Architecture beats parameter count

Architecture-size table

Ladder Rung Params Slip-MAE [mm/s] Latency [ms]
Transformer nano 11k 2.79 6.1
micro 21k 2.71 6.0
tiny 57k 2.51 6.6
small 214k 2.46 7.2
base 1,090k 2.37 10.0
large 2,107k 2.43 8.3
Hybrid nano 11k 2.59 7.7
micro 20k 2.47 7.8
tiny 65k 2.39 7.9
small 250k 2.12 9.0
base 1,244k 2.14 15.6
Fusion nano 8k 2.79 6.2
micro 14k 2.60 6.0
tiny 47k 2.46 6.3
small 183k 2.40 7.1
large 3,377k 2.35 12.7

Latency is median forward-pass time. Superscript daggers in the thesis mark separately timed runs.

Evidence

Detection is solved within the first few epochs

The error is undershoot, not overshoot

Returns diminish up the size ladder

The onset spike, event by event

Onset–peak scatter

What a prediction actually looks like

The spread between recordings

Where does the error come from? The fast tail

  • Slip frames are 30 % of the test set but carry 70–80 % of the error
  • MAE grows with velocity: 0.9 mm/s below 1.5 mm/s, 9 mm/s above 14.8 mm/s
  • Cause is the target, not the network: log(1+v) under a masked Huber loss compresses the fast end
  • The curve has the same shape for all nine networks

Bibliography

[1] Pexels, “A person giving a voting ballot,” [Online]. Available: https://www.pexels.com/video/a-person-giving-a-voting-ballot-7103504/. [Accessed: Sep. 18, 2026].

[2] Pexels, “Man doing rock climbing,” [Online]. Available: https://www.pexels.com/video/man-doing-rock-climbing-4463069/. [Accessed: Sep. 18, 2026].

[3] Pexels, “An orchestra playing music,” [Online]. Available: https://www.pexels.com/video/an-orchestra-playing-music-7095757/. [Accessed: Sep. 18, 2026].

[4] Pexels, “Man touching a sculptured wall,” [Online]. Available: https://www.pexels.com/video/man-touching-a-sculptured-wall-6577556/. [Accessed: Sep. 18, 2026].

[5] Pexels, “Making ceramic bowl on potter’s wheel,” [Online]. Available: https://www.pexels.com/video/making-ceramic-bowl-on-potters-wheel-11761089/. [Accessed: Sep. 18, 2026].