Publication Details
Download |
Frederick Choi, Sven Mayer, Chris Harrison
3D Hand Pose Estimation on Conventional Capacitive Touchscreens Submission Proceedings of the 23rd International Conference on Mobile Human-Computer Interaction (MobileHCI'21), 2021-09-27 (bib) |
Contemporary mobile devices with touchscreens capture the X/Y position of finger tips on the screen and pass these coordinates to applications as though the input were points in space. Of course, human hands are much more sophisticated, able to form rich 3D poses capable of far more complex interactions than poking at a screen. In this paper, we describe how conventional capacitive touchscreens can be used to estimate 3D hand pose, enabling richer interaction opportunities. Importantly, our software-only approach requires no special or new sensors, either internal or external. As a proof of concept, we use an off-the-shelf Samsung Tablet flashed with a custom kernel. After describing our software pipeline, we report findings from our user study, which shows our 3D joint tracking accuracy is around. We conclude with several example applications we built to illustrate the potential of our approach. |