Multimodal interfaces are user-machine interfaces making use of numerous different input modalities including speech, gestures or emotions. Multimodal interfaces offer a number of specific advantages compared to standard WIMP interfaces due to the concurrent management of the different modalities. They further offer the possibility to combine different modalities to increase the expressive power and provide a more natural interaction with applications.

 

 

 

The extraction of meaningful information from different modalities occurring in parallel is a complex problem. This problem becomes even more complex if one takes the inherently unpredictable user behaviour and the imperfect performance of probabilistic recognisers into account. Therefore, conflicts resulting between input data from different modalities have to be taken into account and managed in order to unveil the full capabilities of multimodal interfaces. In this thesis we will investigate and come up with effective algorithms for conflict resolution in multimodal interaction. The results of this thesis will be integrated with an existing framework for the creation of multimodal interaction that is currently developed by WISE lab members [1,2].

 

[1] Lode Hoste, Bruno Dumas and Beat Signer, Mudra: A Unified Multimodal Interaction Framework, Proceedings of ICMI 2011, 13th International Conference on Multimodal Interaction, Alicante, Spain, November 2011.

[2] Bruno Dumas, Frameworks, Description Languages and Fusion Engines for Multimodal Interactive Systems, PhD Thesis no. 1695, Faculty of Sciences, University of Fribourg, Switzerland, December 2010.