Gesture recognition

During the blue-c, a marker-less method was developed that detects a stretched arm as an indication of a pointing gesture from multiple camera streams. This method allowed basic interaction with the virtual environment. The aim for blue-c-II is to go towards a more general gesture recognition framework. Based on the full body pose of the user, the extended framework will provide more complex interaction schemes for the collaboration facilities developed during the second project phase.

The reconstruction of the full body pose from multiple camera streams without using any markers is still a challenging research task. Our approach will be based on an articulated body model which will be fitted to the image data of multiple cameras. The body model is built out of superellipsoids and is driven by an articulated skeletal structure. As a first step, by using a background-subtraction algorithm, the user will be separated from the background in all camera images. Then, from these foreground masks, the 3D shape of the person will be computed using a voxel-based procedure. In order to recover the personís joint positions and limb dimensions, a stochastic optimization process will fit the body model into the 3D observation on each time instant.

Once the movements of the user over time are known, gestures can be detected and used to trigger actions in the system. For this purpose, a probabilistic framework will be trained to recognize a set of well defined gestures, such as grasping, pushing, etc.
Using this prior information, the framework will then classify the recovered temporal joint data of the performing user into gestures and will trigger the corresponding actions in the system.

3D reconstruction result of the user from 6 views using our voxel-based procedure. Alignment of the model with respect to the 3D reconstruction after the stochastic optimization process. Result of the foreground / background segmentation of four simultaneously captured input images. Model built out of superellipsoids. The left figure shows an explosion view with all articulated chains separated. On the right, the skeletal structure is overlaid.


Last update:
March 31. 2012 04:21:40
Powered by CMSimple