6+E: Understanding non-typical emotions from facial videos with multimodalities.
Emotion is an important factor in many aspects of our lives. Computer scientists train computers to understand humans’ emotions and have made great progresses, but one major limitation is that most of previous studies simplify the task of emotion understanding as classification of the six basic emotions (6BE: happiness, sadness, anger, fear, surprise and disgust), while other non-typical emotions are seldom concerned due to many difficulties. This plan propose to take the challenge and explore non-typical emotions (6+E) starting from: boredom, confusion, interest, shame, nervousness and confidence. We will build computational models by integrating info from multiple modalities of facial videos, including expressional movements, contextual facial behaviors, facial color changes, and physiological signals. We hope the research could be a foundation and inspire more studies to focus beyond the 6BE for emotion understanding.