Multimodal Interaction and Motivational Intention Characterization

MIMIC

Motivational peer interaction, such as encouragement and value-related talk, plays a critical role in sustaining engagement and persistence in collaborative learning by shaping a shared motivational climate. These interactions are not conveyed through spoken language alone but are embedded in multimodal cues, including vocal prosody, pauses, gestures, and gaze, which carry important social and emotional meaning. However, existing research and AI-based approaches have largely focused on verbal content, overlooking how motivational messages are expressed and interpreted through implicit, contextual, and embodied signals. As a result, current AI models remain limited to surface-level analyses and are unable to capture the subtle dynamics that sustain motivation in collaborative learning settings. This project addresses this gap by developing and piloting an integrative multimodal approach for recognizing motivational peer interaction using multimodal large language models (MLLMs). By combining speech content with prosodic, embodied, and contextual information, the project aims to enable real-time interpretation of motivational interaction as it unfolds in classroom settings. This work lays the foundation for hybrid intelligence systems that complement human expertise by supporting teachers and learners in fostering motivational classroom climates and promoting socially shared regulation of motivation in collaborative learning.

Funders

HI OULU
human au

Project information

Project duration

-

Funded by

Other Finnish

Project funder

University of Oulu
Hybrid Intelligence: Human-AI Co-evolution and Learning in Multi-realities (HI)

Funding amount

20 000 EUR

Project coordinator

University of Oulu

Unit and faculty

Learning and Learning Processes
Faculty of Education and Psychology

Contact information

Project leader

Project description

This NRI positions motivational classroom interaction recognition as a hybrid-intelligence problem: neither AI nor human analysis alone can deliver scalable, valid identification of motivational interaction in learning contexts. The initiative, therefore, combines multimodal AI development with motivation and learning sciences expertise. We aim to collaboratively develop and pilot an MLLM for recognising motivational peer interaction from existing multimodal interaction data, alongside shared construct definitions, annotation practices, and human-in-the-loop validation routines that enable later upscaling. This establishes a hybrid pathway in which AI is iteratively trained through expert guidance, while researchers gain understanding of how motivational interaction is expressed across modalities.

​​The project responds to the research question: ”Can a multimodal large language model recognize motivational interactions among students in collaborative learning settings by integrating speech content, vocal prosody, and embodied cues?” We expect the multimodal model to outperform speech-content-only models, as prosodic and embodied signals provide critical context for interpreting motivational meaning

Project actions

MIMIC will be implemented through three interrelated work packages (WPs): two focusing on methodological and technical aspects of LLM development, and the third one focusing on collaborative activities between project partners and administrative issues. The work will be carried out in collaboration with the University of Potsdam and the Technical University Berlin (Germany).

Project results

The project produces one of the first empirically validated, multimodal approaches for analysing motivational interaction in peer collaboration by integrating semantic information from fine-tuned large language models (LLMs) with paralinguistic cues derived from high-quality speech and video data. The resulting methodological pipeline will contribute directly to current research on collaborative learning, learning analytics, and hybrid intelligence by offering a new way to detect motivational processes as they unfold in authentic educational settings. The multimodal dataset and analysis framework will provide researchers with tools to examine the interplay between verbal content, prosody, and interaction dynamics in unprecedented detail, supporting new directions in motivational research, social interaction modelling, and the development of AI systems capable of interpreting nuanced human behaviour.
The project’s primary contribution lies in developing robust research methods, such as multimodal large language models, that enable precise analysis of motivational interactions in classroom interactions. These methodological innovations will produce evidence-based insights that guide educators in designing practices that sustain engagement and equity. Furthermore, the methods will inform the creation of intelligent tools capable of detecting and responding to motivational dynamics in real time. By advancing both scientific understanding and practical applications, the project strengthens the foundation for scalable, research-driven educational technologies, ultimately promoting motivated lifelong learning.