Mudra: A Unified Multimodal Interaction Framework
In recent years, multimodal interfaces have gained momentum as an alternative to traditional WIMP interaction styles. Existing multimodal fusion engines and frameworks range from low-level data stream-oriented approaches to high-level semantic inference-based solutions. However, there is a lack of multimodal interaction engines offering native fusion support across different levels of abstractions to fully exploit the power of multimodal interactions. We present Mudra, a unified multimodal interaction framework supporting the integrated processing of low-level data streams as well as high-level semantic inferences. Our solution is based on a central fact base in combination with a declarative rule-based language to derive new facts at different abstraction levels. Our innovative architecture for multimodal interaction encourages the use of software engineering principles such as modularisation and composition to support a growing set of input modalities as well as to enable the integration of existing or novel multimodal fusion engines.
Publication Reference
Hoste, L., Dumas, B. and Signer, B.: "Mudra: A Unified Multimodal Interaction Framework", Proceedings of the 13th International Conference on Multimodal Interaction (ICMI 2011), Alicante, Spain, November 2011
Available:
WISE Authors
Related Research Topic(s)
Related Research Project(s)
- Midas - A Declarative Multi-Touch Interaction Framework
- MobiCraNT - Second Generation Mobile Cross-media Applications - Scalability, Heterogeneity and Legacy
- Mudra - A Unified Multimodal Interaction Framework