Markerless Vision-Based Hand-Tracking for Interaction




Ever since the creation of the first Virtual/Augmented-Reality (VR/AR) applications Human-Computer-Interaction research has taken a great interest in using the human hand as an input-device. This is explained by one of the main requirements of VR/AR: actions/responses familiar from our everyday life have to be at the user’s disposal. Furthermore, the overall user experience should be enhanced by allowing the user to carry out tasks, which are impossible in real life (e.g. flying). Everyday interaction with our environment ubiquitously involves our hands as an "interaction-device", therefore, it should also be available in its virtual counterpart. To enable VR/AR systems to react to hand movements, the user’s hand has to be tracked. In this context, a markerless approach is particularly desirable, to ensure the interaction to be instant and immersive (one does not need to wear an extra device e.g. a glove to start grasping or pointing at objects in the real world).

The ultimative goal would be a hand tracking method that allows tracking the full 27 degrees of freedom (DOF) of a hand (global pose and joint angles) in order to enable all kinds of interaction purposes. Unfortunately, these methods suffer from several drawbacks so far, such as no real-time support or special initialization procedures. Hence, these methods are not applicable in practice yet. Therefore, several other approaches restrict themselves to a subset of the full DOFs (only global hand pose and stiff gestures) to solve these drawbacks. Unfortunately, they were still not capable of tracking the position of the hand as well as all the three rotational DOFs without limitations. Because of that, we recently developed a novel computer vision based hand-tracking technique, which is capable of robustly tracking the 6 DOFs of the human hand pose while distinguishing between 4 different gestures without such limitations. Moreover, it works in real-time (at least 25 frames per second) with the help of 3 (or more) off-the-shelf consumer cameras. A key feature of our system is its fully automatic real-time initialization procedure, which, along with a sound tracking-lost detector, makes the system fit for real-world applications.

Interfaces & Applications

We built several interaction interfaces for our hand-tracking system. Moreover, we connected our system to different commercial applications. A selection is described in the following:

  • Self-Implemented Interfaces/Applications
  • Commercial Applications
    • Zugspitze 3D, 3D Reality Maps undefined[Video]
    • RTT DeltaGen, Real-Time Technology AG undefined[Video]
    • 3D Games

Synopsis (German)


626 K

Press Coverage


  • Winner of the RTT Emerging Technology Contest, 8th RTT Conference, May 10-11 in Berlin, Germany, 2007.
  • Laval Virtual Award for "Bare-Handed 3D Interaction", best project in the category "Interfaces and Materials", 11th Laval Virtual Conference, April 22-26 in Laval, France, 2009.
    undefined[Description]  undefined[Video] extern[Link]
  • Outstanding Paper Award for "Real-Time Bare-Hands-Tracking for 3D Games" at the IADIS International Conference Game and Entertainment Technologies, June 17-19 in Algarve, Portugal, 2009.


Katharina Stollenwerk, Anna Vögele, Ralf Sarlette, Björn Krüger, André Hinkenjann, and Reinhard Klein
In proceedings of 13. Workshop Virtuelle Realität und Augmented Reality der GI-Fachgruppe VR/AR, Bielefeld, Germany, Sept. 2016
Markus Schlattmann, Tan Zheng, Johannes Broekelschen, and Reinhard Klein
In: IADIS International Journal on WWW/Internet (Dec. 2010), 8:2(1-16)
In: Journal of Virtual Reality and Broadcasting (Oct. 2010), 7:8
In proceedings of SIGGRAPH ASIA '09: ACM SIGGRAPH ASIA 2009 Sketches, Yokohama, Japan, pages 1, ACM, Dec. 2009
Markus Schlattmann, Johannes Broekelschen, and Reinhard Klein
In proceedings of IADIS International Conference Game and Entertainment Technologies (GET '09), pages 59-66, IADIS Press, June 2009
In proceedings of Virtual Reality International Conference (VRIC), Apr. 2009
Markus Schlattmann, Tanin Na Nakorn, and Reinhard Klein
In proceedings of International Conference on Computer Graphics, Visualization and Computer Vision (WSCG '09), Feb. 2009
In proceedings of ACM Symposium on Virtual Reality Software and Technology (VRST '07), Nov. 2007
D. Cohen-Or and P. Slavík (Editors)
In: Computer Graphics Forum (Sept. 2007), 26:3(467-476)