Project : LensGesture

LensGesture: Augmenting Mobile Interactions with Back-of-Device Finger Gestures


We present LensGesture, a pure software approach for augmenting mobile interactions with back-of-device finger gestures. LensGesture detects full and partial occlusion as well as the dynamic swiping of fingers on the camera lens by analyzing image sequences captured by the built-in camera in real time. We report the feasibility and implementation of LensGesture as well as newly supported interactions. Through offline benchmarking and a 16-subject user study, we found that 1) LensGesture is easy to learn, intuitive to use, and can serve as an effective supplemental input channel for today’s smartphones; 2) LensGesture can be detected reliably in real time; 3) LensGesture based target acquisition conforms to Fitts’ Law and the information transmission rate is 0.53 bits/sec; and 4) LensGesture applications can improve the usability and the performance of existing mobile interfaces.


  • Xiao, X., Han, T., and Wang, J. LensGesture: augmenting mobile interactions with back-of-device finger gestures. In Proceedings of ACM International Conference on Multimodal Interaction (ICMI 2013), Sydney, Australia, Dec 9 – 13, 2013. ( pdf )


Source Code (BSD License) Video