Extending the iGesture Framework with Multimodal Gesture Interaction Functionality
Nowadays, more and more commercial products support gesture based interaction. Some of the best known examples are Nintendo's Wii gaming console and Apple's iPhone. The commercial success of these products validates the usefulness of gesture interaction. Firstly, it makes devices and software easier to use by providing more natural interfaces. Secondly, they support and attract a broader range of users. Audiences that normally do not frequently play games, such as women or adults, represent a steadily increasing audience since the introduction of gesture-controlled devices. However, gesture interaction can not ony be used for gaming but also gains popularity in desktop computing. Unfortunately, it is still diffcult to develop applications that support gesture interaction. Existing frameworks to build these types of applications either offer a limited and fixed number of gestures or provide limited support for algorithm and gesture designers. In many cases, the gesture interaction functionality is hard-coded in specific applications, resulting in a more cumbersome and complex development process.
iGesture is an open source, Java-based framework for gesture recognition that provides support for the application developer as well as algorithm and gesture designers. New devices and recognition algorithms can easily be added. Gestures can be grouped and managed in gesture sets and new gestures can be defined by sample or textual descriptors.
In this thesis, we have extended the iGesture framework with support for composite gestures and multi-modal interaction. Composite gestures allow the gesture designer to define complex gestures based on a set of simpler gestures in a declarative manner. A small set of basic gestures may lead to better recognition rates since the gestures forming part of this set are more distinctive. Multi-modal interaction makes it possible to, for example, combine gesture input with voice input and thereby supports the invocation of actions in a more natural way.
To achieve the goal of declarative gesture composition, a multi-modal recogniser has been developed to recognise composite and multi-modal gestures. Furthermore, we also de ned an extensible set of composite gesture constraints. Last but not least, tool support for designing and testing composite gestures is provided as well.