Succeeding together: Haoyu Chen and Tiina Törmänen train AI to interpret emotions

A shared desire to understand emotions unites learning researcher Tiina Törmänen and Haoyu Chen, Assistant professor of machine learning. Both study emotions and AI from their own perspectives, and they are now working together to fine-tune AI to recognise emotions in teamwork situations.

In his AI research, Chen has focused on understanding emotions and human behaviour. Along with his team, he has developed the AffectGPT artificial intelligence model, which recognises human emotions from video footage.

"The model we have developed is currently one of the best emotion-recognising AI models in the world. We wondered whether it could be further developed to work in a specific context. That was the starting point for our collaboration with Tiina's team," Chen says.

Together, Chen and Törmänen have collected video material that is reviewed by the AI model, the researchers, and the students who were filmed in the situation.

"Humans evaluate the analysis made by the AI, and in this way, we train the model to become a more skilled interpreter of emotions in learning situations," Törmänen explains.

Törmänen approaches the study of emotions from an educational science perspective. She investigates how learners' emotions affect collaboration and how emotions are regulated in teamwork situations. She is interested in exploring how AI could help us if it could detect cues of emotions such as anger or sadness and respond to them constructively.

"Could AI, in the future, help us resolve interpersonal conflicts by supporting emotion regulation? At present, this possibility is being studied in controlled research settings, with careful attention to ethical considerations.”

Research partner found through a multidisciplinary research programme

Finding a well-functioning partnership with a researcher from another field is not always easy. Chen and Törmänen found the collaborator they were looking for through the multidisciplinary Hybrid Intelligence research programme, where both are currently working.

"A barrier to multidisciplinary collaboration is often simply not knowing who you could work with. Research programmes that bring together different disciplines are important because they allow you to discuss and develop ideas with researchers from other fields," Chen says.

Developing artificial intelligence that understands human emotions is impossible without interdisciplinary collaboration. The challenges currently being solved require expertise in the human sciences and an understanding of how humans behave and interpret one another.

"Algorithms allow us to map signals and structure them into models. We still have a lot to learn about defining these models, and for that, we need experts in human behaviour," Chen notes.

The duo’s research is a prime example of hybrid intelligence, where humans and AI work together effectively, learning from one another.

Advancing technology allows Törmänen to conduct research with much larger datasets than before.

"There is plenty of research on emotion regulation in teamwork, but it is based on small datasets in limited situations. Analysing data collected from group work is incredibly laborious, and this is where AI could bring efficiency," Törmänen says.

Students also benefit from this bringing together of disciplines. The duo's research setup is generating master’s theses on the topic, one from the perspective of human sciences and the other from computer science.

United by a genuine desire to understand the phenomenon

Collaboration with a representative from another discipline introduces new approaches to the work and helps researchers understand the phenomenon under study on a completely new level.

"We understand emotions as clear categories, such as joy, sadness, or surprise. Through Tiina, I gain depth in interpreting the data," Chen says.

Multidisciplinary research takes time and requires immersing oneself in the terminology and practices of an unfamiliar field. Törmänen believes that her collaboration with Chen has been exceptionally functional because both are committed and motivated.

"In this kind of collaboration, you have to challenge yourself and your ways of thinking, and it is not for everyone. Both of us genuinely want to better understand the phenomenon we are researching, human emotions."

The Achievers

Haoyu Chen

Work: Assistant Professor of Machine learning, Hybrid Intelligence research programme
Education: Doctor of Science, Computer Science of Engineering
Motto: Understanding humans is the key to building better AI.

Tiina Törmänen

Work: Postdoctoral Researcher, Hybrid Intelligence research programme
Education: Doctor of Philosophy, Education
Motto: What matters is where you focus: solutions or problems.