Center for Machine Vision and Signal Analysis (CMVS)

Core part of research in CMVS is fundamental research on generic algorithms for computer vision. The Center has achieved ground-breaking research results in many areas of its activity, including texture and dynamic texture analysis.

Image and video descriptors play a key role in most computer vision systems and applications. The function of descriptors is to convert pixel-level information into a useful form, which captures the most important factors into a useful form but is insensitive to irrelevant aspects caused by the varying environment. While the definition of irrelevant depends on the application, the most common cases are related to imaging conditions like illumination, viewing angle, scale, noise, and blur. The main focus of our research has been in texture descriptors, in which area we have long traditions and rank among the world leaders.

Texture is an important characteristic of many types of images and videos. The Local Binary Pattern (LBP) texture operator has been highly successful in numerous applications around the world, and has inspired plenty of new research on related methods, including the blur-insensitive Local Phase Quantization (LPQ) method, Weber Law Descriptor (WLD), Binarized Statistical Image Features (BSIF), and Local Orientation Adaptive Descriptor (LOAD) also developed by researchers of CMVS. Spatio-temporal dynamic texture descriptors for video analysis have also been proposed. The most popular one among these is LBP-TOP. One focus of current research is in combining these kind of “traditional” descriptors with deep learning based approaches, with applications in face analysis and biomedical image analysis, for example.

Local Binary Patterns

Local Binary Pattern (LBP) is a simple yet very efficient texture operator which labels the pixels of an image by thresholding the neighborhood of each pixel and considers the result as a binary number. Due to its discriminative power and computational simplicity, LBP texture operator has become a popular approach in various applications. It can be seen as a unifying approach to the traditionally divergent statistical and structural models of texture analysis. Perhaps the most important property of the LBP operator in real-world applications is its robustness to monotonic gray-scale changes caused, for example, by illumination variations. Another important property is its computational simplicity, which makes it possible to analyze images in challenging real-time settings. LBP has been most widely used as a dense texture descriptor, but can also be used as a sparse descriptor like SIFT or computed on a grid like HOG.

For a more detailed description of LBP in spatial and spatiotemporal domains, see LBP in Scholarpedia.

Dynamic texture descriptors

Dynamic textures (DT) or temporal textures are textures with motion, extending spatial domain texture to the temporal domain. Dynamic texture analysis methods provide effective tools for motion analysis. Potential applications include remote monitoring and various type of surveillance in challenging environments, such as monitoring forest fires to prevent natural disasters, traffic monitoring, homeland security applications, and animal behavior for scientific studies. DT methods are also used for analysis of facial dynamics, recognition of actions, video synthesis, motion segmentation, and video classification. CMVS has done extensive research on spatiotemporal dynamic texture descriptors, DT recognition, segmentation, and synthesis. Among the application areas are face analysis and recognition of actions.

Automatic face analysis is a very active topic in computer vision research as it is useful in several applications, like biometric identification, visual surveillance, human-machine interaction, video conferencing and content-based image retrieval.

Facial analysis is one of the most important abilities used in our everyday lives. Humans use face analysis for many different tasks. They recognize other people mainly from their faces, identify them as young or old, men or women, Caucasian or Asian. We may also see from the face whether a person is happy or sad, tired or nervous, feels pain, is telling the truth or lying. Faces are very important in social communications between humans. Emotional faces communicate both the emotional state and behavioral intentions of a person. The mouth movements are an important cue for speech recognition in noisy conditions, and hearing-impaired people can even read from lips. The head pose and gaze direction provide information about the direction of attention. Faces can also provide hints about health. For example, a person with autism has difficulties to understand and express emotions.

Due to its great importance, facial image analysis has been a topic of intensive research in computer vision, image processing and related fields. Among its potential areas of application are natural human computer/robot interfaces, biometric identification, image and video retrieval, multimedia management, video surveillance, forensics, audio-visual speech recognition, and affect-sensitive systems for various applications.

A common goal for our research on different areas of face analysis is to develop methods for unconstrained conditions (“in the wild”), allowing changes in illumination, head pose, image quality, and background.

Face Recognition

In 2004, we proposed an approach to facial representation based on LBP features. The face image is divided into regions from which the LBP features are extracted and concatenated into an enhanced feature vector to be used as a face descriptor. This approach is one of the milestones in face recognition research, receiving the prestigious Koenderink Prize at ECCV 2014 for fundamental contributions in computer vision that stood the test of time. It has been adopted and further developed by many groups and companies around the world. In 2007, the face descriptor was extended to videos using spatiotemporal LBP-TOP operators to describe facial dynamics, Also this approach has been adopted by many others.

In face biometrics, our current key focus of research is in developing effective methods for anti-spoofing. Another area of the recent research is soft biometrics, with an aim to develop robust methods, e.g. for gender recognition, age estimation and kinship verification from facial images.

Face anti-spoofing (presentation attack detection)

Without dedicated countermeasures, most of the existing facial biometric systems are vulnerable to spoofing (presentation attacks) because they try to maximize the discriminability between identities without regards to whether the presented trait originates from a living legitimate client or not. CMVS has been actively developing software-based methods for presentation attack detection (PAD). The proposed countermeasures to spoofing analyze the facial texture, motion and contextual information for describing the inherent disparities between genuine faces and fake ones. In 2013, we received the IET Biometrics Premium Award 2013 for our paper published in IET Biometrics journal in 2012.

See video illustrating face anti-spoofing

Facial expression recognition

Automatic emotion analysis and understanding has received much attention over the years in affective computing. One of the key modalities is facial expressions. CMVS has done extensive research on facial expression recognition using dynamic information analysis. Our experimental results also demonstrate promising performance for group happiness intensity analysis.

A goal of our current research is in developing methodology for recognizing spontaneous expressions from continuous video data in unconstrained conditions (“in the wild”).

Micro-expression recognition and spotting

Micro-expression is very rapid involuntary facial expressions which reveal suppressed affect. Facial micro-expressions reveal contradictions between facial expressions and the emotional state, enabling recognition of suppressed emotions. The method (and corpus) developed in CMVS is the first to recognize spontaneous facial micro-expressions and achieves very promising results.

We have also investigated methods for spotting micro-expressions from video data to be used, e.g. as a preprocessing step prior to recognition. An aim l of our current research is to make automated micro-expression analysis a practical tool for real-world applications.

The work has been higlighted in international media such as MIT Technology Review and Daily Mail.

Heart rate monitoring from face videos

Heart rate is an important indicator of people's physiological state. Recent research shows that physiological signals like the heart rate can be measured just by a normal camera. The human visual system has limited spatio-temporal sensitivity, but many signals that fall below this capacity can be informative. For example, human skin color varies slightly with blood circulation. This variation, while invisible to the naked eye, can be exploited to extract pulse rate of a person. The methodology developed in CMVS outperforms other solutions that have performance degradation when environmental illumination variations and subjects’ motions are involved. A goal of the current research is to adapt the methodology to different applications, including health monitoring, face anti-spoofing, and affective computing.

Visual speech recognition and animation

Visual speech information plays an important role in both speech recognition and human computer interaction. For visual speech recognition local spatiotemporal descriptors and graph embedding were proposed to capture video dynamics, represent and recognize spoken isolated phrases based solely on visual input. CMVS has also developed a visually realistic animation system for synthesizing a talking mouth. Video synthesis is achieved by first learning generative models from the recorded speech videos and then using the learned models to generate videos for novel utterances.

3D computer vision has been one of the core research areas in CMVS since early 1990’s. This research has resulted in many novel methods and software tools that have been widely used in the research community and companies.

During the last few years our focus has been in 3D computer vision techniques that enable more advanced features in augmented reality applications, where real and virtual objects co-exist in the same environment. Wearable computers such as Google Glass have created a strong demand for such technology. One of the fundamental problems investigated in our work is accurate localization of the user with respect to the environment. Other important research problems include estimation of the 3D scene structure and building a dense 3D model from multiple images.

Camera calibration

Geometry is an important aspect of computer vision. The laws of geometry and optics describe how the three-dimensional world is imaged on the camera sensor and, hence, an understanding of imaging geometry is important for the development of automatic image analysis methods. Geometric camera calibration is an important topic because it is a prerequisite for imagebased metric 3D measurements, and we have a strong expertise in that field.

Camera calibration toolboxes:

• Camera calibration toolbox for Matlab
• Camera calibration toolbox for generic central cameras
• Kinect Calibration Toolbox

3D Reconstruction

Automatically acquiring a three-dimensional model of a scene from multiple photographic images is an important research area in computer vision. During the last few years, we have also focused on 3D modeling from point clouds produced with image-based 3D reconstruction techniques.

Recognizing humans, their actions and emotions with computer vision, and providing an intelligent machine's response, will address some of the fundamental problems of affective human-computer interaction.

Development of affective human-computer interfaces (HCI) is of great interest in building future ubiquitous computing systems. We should be able to interact with computers in a natural way, similar to the human-human interaction. Computer vision and speech recognition will play a key role. The computer should be able to detect and identify the user, recognize user’s emotions, communicate easily by understanding speech and gestures, detect and track humans and recognize their actions, and provide a natural response based on its observation. Our research on image and video descriptors, facial image analysis, emotion recognition, gaze analysis, gesture analysis, visual speech analysis and synthesis, and 3D vision provide a unique basis for building affective HCI systems for various applications.

Experimental systems for face analysis and human-robot interaction

An experimental real-time face analysis system has been developed in our unit. The system includes modules for face detection and tracking, facial landmark detection, geometric and illumination normalization , face recognition, gender recognition, facial expression recognition, and heart-rate measurement from face videos. Face recognition component is the centerpiece of this analysis as it can dynamically learn new people that appear in front of the camera and remember previous analysis results and combine them with new ones for each different person. The system also displays how the recognition is done by visualizing each individual step and their intermediate results. The system has been used to demonstrate our research for visitors and in different exhibitions.

Key parts of the system were implemented on Vuzix M100 Smart Glasses. Software uses only local processing and does not rely on network connectivity to offload computation to servers. Face analysis technology on wearable devices enables many different kind of services. Familiar persons could be associated with extra information that is displayed when they are detected. In the future, software could also automatically analyze surroundings, e.g. for the visually impaired and use other modalities to convey this information for the user.

Together with the Robotics team, we developed a distributed system for affective human–robot interaction which combines all the basic sensory elements and can collaborate with a smart environment and obtain further knowledge from the Internet. The realized HRI robot, Minotaurus, runs in real time and is capable of interacting with human beings. Minotaurus forms a rather generic platform for experimenting with human–robot collaboration in different applications and environments.

See video of interaction with social robot "Minotaurus"

Human gaze analysis

One important aspect of perceptual interfaces is the analysis of human attention. Since eye movement reveals the regions of interest (ROI) of the human visual system, it is widely utilized in researches regarding visual attention understanding. Observing the focus of human interest allows for improved human computer interaction. To facilitate the research in visual attention analysis, CMVS has designed and establish a new task-driven eye tracking dataset that provides fixation density maps of 47 subjects.

CMVS has also explored gaze direction analysis from video data and compiled the Oulu Multi-pose Eye Gaze Dataset that contains eye gaze images captured under multiple head poses.

The capability to recognize human emotions plays a significang role in applications ranging from human computer interaction and entertainment to psychology and education. In CMVS we believe that combining complementary information from different modalities increases the accuracy of emotion recognition.

Affective computing (AC) is a modern research direction to solve the great challenge of creating emotional intelligence for a machine. AC is a cross-disciplinary research area on the design of systems that can recognize, interpret, and simulate human emotions and related affective phenomena. Our main goal is create breakthrough technology in affective computing based on computer vision, audio-visual speech, physiological signals and other data. The world-class expertise of CMVS from these areas provides a unique basis for this research.

The most informative way for machine perception of affect and emotions is through facial expressions in video. Also body movements, audio-visual speech, head, eye, and different physiological signals that are closely related to emotional changes (e.g., heart rate, respiration, galvanic skin response) should be considered.

Emotion analysis from facial expressions and electroencephalogram

Recently, we proposed an approach for multi-modal video-induced emotion recognition, based on facial expression and electroencephalogram (EEG) technologies. Spontaneous facial expression is utilized as an external channel. A new feature, formed by percentage of nine facial expressions, is proposed for analyzing the valence and arousal classes. Furthermore, EEG is used as an internal channel supplementing facial expressions for more reliable emotion recognition. Discriminative spectral power and spectral power difference features are exploited for EEG analysis. Finally, these two channels are fused on feature-level and decision-level for multi-modal emotion recognition. Experiments are conducted on MAHNOB-HCI database, including 522 spontaneous facial expression videos and EEG signals from 27 participants. Moreover, human perception in emotion recognition compared to the proposed approach is tested with 10 volunteers. The experimental results and comparisons with the average human performance show the effectiveness of the proposed multi-modal approach.

Emotion recognition from voice

The human voice is one of the most important channels for affective signaling, and is commonly used in any multimodal affective analysis. Voice characteristics are heavily influenced by emotions both intentionally and implicitly. When the voice is used in natural language speech these characteristics are embedded directly in the acoustic speech signals paralinguistic properties (i.e. suprasegmental properties or prosody) as well as in any intended lexical messages. These relatively long term emotional changes of paralinguistics are also largely involuntary and automatic, and, as such, convey information about the emotional state of the speaker (both portrayed and actual). Emotion recognition is performed by analysing the various acoustic features of voice/speech for emotional patterns. In the case of speech, specifically, the prosody of speech (i.e. intonation, intensity, rhythm, and voice quality) is parameterized and utilized directly, or with other optional multimodal features, in a machine learning approach to achieve emotion recognition.

Multimodal fusion

The capability to recognize human emotions plays a significant role in applications ranging from human computer interaction and entertainment to psychology and education. In CMVS we believe that combining complementary information from different modalities increases the accuracy of emotion recognition. The key modalities include machine vision, speech and various physiological signals.

Emotion recognition is a key first step in enabling affective functionality. The affective state of a human user is often critical in evaluation of affective context and affectually correct functionality as human behavior is strongly modified and motivated by psychological aspects such as stress, mood, and feelings. State-of-the-art emotion recognition solutions rely heavily on multimodality by utilizing data fusion and classification of physiological signal analysis, natural language processing, and facial expression and gesture analysis.

Human communicate via many non-verbal channels such as facial expressions, voice characteristics and bodily gestures. We hypothesize that fusing information from various modalities in affective computing increases performance over single-modal approaches in many applications. In addition to externally observable affective behavior patterns, physiological signal can also be utilized. Physiological signals that we have used include central nervous system signals (EEG) and autonomic nervous system (sympathetic and parasympathetic; heart rate variability, EDA). We have combined modalities such as facial videos, voice recordings and physiological signals by using machine learning techniques.

Low-energy computing requires that the power demands are on par with energy that can be harvested from the environment. In practice, only a few mW may be available for sensing, communications, and computing. In this domain we investigate energy efficient fully programmable architectures and high-level design tool chains.

Selected research: Transport Triggered Architecture (TTA) applications

Industrial condition monitoring

We have studied energy efficient signal processing methods for condition monitoring of various industrial applications. We have implemented methods on the transport triggered architecture (TTA) signal processor in order to achieve performance and energy efficiency needed by energy autonomous wireless sensor nodes. The sensor node implemented together with VTT is illustrated in figure below. Our TTA signal processor was implemented on the Flash FPGA on the node.

Application specific accelerator for computer vision

The research of energy efficient signal processing methods has also expanded to cover future IoT solutions, where the smart sensors have signal processing capabilities in order to reduce the transmitted data amounts. As a first step, an application specific accelerator for computer vision was implemented. The accelerator was implemented using multicore TTA and programmed with Portable computing language (PoCL) implementation of OpenCL. The accelerator was benchmarked against commercial CPUs and GPUs and was shown to provide higher performance together with significantly lower energy consumption. The simplified TTA architecture of our computer vision accelerator is presented in figure below.

Application specific accelerator for HEVC video coding

In this work, we have accelerated HEVC in-loop filters using energy efficient programmable processor architecture. By assigning in-loop filtering to an appropriate coprocessor instead of a general-purpose processor it is possible to achieve substantial improvements in terms of computation performance and energy consumption. The architecture can be fully programmed using a high level programming language, such as C, which means that it can be used to accelerate also other applications. The system architecture, which is shown below, is based on transport triggered architecture (TTA) cores which are shown to be energy efficient and have high performance for a wide range of applications.

In recent years, increasing resolving power and automation of biomedical imaging systems have resulted in an exponential growth of the image data. Manual analysis of these data sets is extremely labor intensive and hampers the objectivity and reproducibility of results. Hence, there is a growing need for automatic image processing and analysis methods. In CMVS, our aim has been to apply modern computer vision techniques to biomedical image analysis which is one of our emerging research areas.

Since 2008, we have carried out some research on texture-based methods for analyzing microscopic and x-ray images, and for classifying HEp-2 cell images.

In CMVS, we have been also investigating computer vision based methods for analyzing living cells from microscopic images and videos since 2011. During this time we have had several joint projects, and one of those, Algorithm-based Combination and Analysis of Multidimensional Video and Open Data (ABCdata, 2012-2015), was a strategic opening project funded by the Finnish Funding Agency for Innovation (Tekes). We develop computer vision based methods for analyzing living cells from microscopic images and videos, and offer solutions that can,

• detect and segment individual objects or particles,
• model their shapes and morphology,
• track their movement across the image sequences, and
• detect events or anomalies based on the features computed from the data.

Cell Tracking

In cooperation with Biocenter Oulu we have developed a solution for analyzing dynamic behavior of cells in phase-contrast images. Our current cell tracking system can track individual cells and provide detailed spatio-temporal measurements of cell behaviors including cell count and migration speed together with direction. Paper link

Detection of tumor cell spheroids from co-cultures

We have developed a method for detecting tumor cell spheroids in phase contrast images of co-cultures of fibroblasts and tumor cells in an experimental 3D model. It depends on local features defined over superpixels. The method learns appearance based features, statistical features, textural features and their combinations. Paper pdf

Cell Segmentation

We propose a method for cell segmentation which can handle arbitrary cell shapes. It uses an edge detector's probability map to create a wedge between touching cells and uses graph cuts to segment individual cells. The main advantage of our method is that it is capable of handling complex cell shapes because it does not make any assumptions about the cell shape. Paper link

Fig.: Maximum intensity projection (view from top) for a sample sequence. From left to right: Original stack, our method, 3D Graph Cuts, Farsight, MINS and 3D MLC.

Joint Cell Segmentation and Tracking

Analysis of microscopic images can be very challenging due to frequent interaction of cells with each other, low contrast, variations in cell shapes and similar cell appearances. Often a single frame does not contain enough information to make the correct decision about segmentation and tracking. In these situations it helps to consider content of adjacent frames, which can be computationally very expensive for long dense sequences. We have developed a greedy joint cell segmentation and tracking method which overcomes this challenge.

Cell Proposal Generation:

Our method uses multiple filter banks to detect cells, and uses watershed to split cell clusters and obtain cell proposals (Fig. 1 shows the main steps). It then create a hierarchical forest from these cell proposals (Fig. 2 shows a tree in the forest).

Tracking:

Our method connects proposals in adjacent frames and constructs a directed graphical model. This model contains nodes for different events that a cell can go through (mitosis, apoptosis, move, leave, enter). Our method then iteratively finds the shortest path in this graph, providing cell segmentations and tracks. Paper link

Fig. 1: A segmentation proposal tree. Left: Segmentations for 3 different filter banks. Right: Hierarchical tree constructed from the proposals.

Texture-based methods in biomedical image analysis

Texture is a key feature in the visual diagnosis of medical settings. Manual inspection of specimens with a light microscope is still to date the gold standard. In collaboration with the Institute of Molecular Medicine in Finland (Adjunct Professor Johan Lundin), we explored high throughput computer assisted methods for automated analysis of digitized breast and colorectal cancer samples, and microbiological samples, e.g. malaria parasites.

More recently, in collaboration with Prof. Osmo Tervonen from Oulu University Hospital we investigated the problem of thorax disease diagnosis from x-ray images using an approach based on deep convolutional neural network.

HEp-2 Cell Classification

Automated cell classification in Indirect Immunofluorescence (IIF) images has potential to be an important tool in clinical practice and research. Recently, classification of Human Epithelial Type 2 (HEp-2) cell images has attracted great attention. However, the HEp-2 cell classification task is quite challenging due to large intra-class and small inter-class variations. We have proposed several effective approaches to this problem, using texture and shape descriptors as well as convolutional neural networks.

Dataset description:

The Spontaneous Micro-Gesture (SMG) dataset comprises of 40 video sequences (821,056 frames in total, more than 8 hours) from 40 participants (each sequence lasts for around 15 minutes) with RGB, depth, silhouette video streams, and skeleton data. 40 participants are recruited from a European university with multicultural backgrounds (16 countries), 27 males and 13 females with an average age of 25.

The dataset contains 71 relaxed emotional state (RES) instances and 71 stressed emotional state (SES instances). Lengths of those instances range from two to five minutes. Each participant contributes two instances or four instances (half are RESs and half are SESs). 3,692 gesture clips are labeled manually, and the average length of those MGs is 51.3 frames.

Some micro-gesture examples:

Relevant publications:

The detailed description of the database, and related baseline results can be found in:

Chen, H., Liu, X., Li, X., Shi, H., & Zhao, G (2019) Analyze spontaneous gestures for emotional stress state recognition: A micro-gesture dataset and analysis with deep learning. 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019): 1-8.

Contacts:

The database can be used, for example, in studying body gesture based emotion analysis. For more details or inquiring obtaining the database, please contact Dr. Guoying Zhao.

Dataset description:

The iMiGUE database collects 359 links of public available videos from YouTube channel Australian Open TV. These videos are interviews or conferences carried out right after a tennis match, and each video has a property of ‘Win’ or ‘Lose’ which is result of the match. Totally, these videos involve 72 subjects where 36 of them are males, 36 of them are females. Besides, from these videos, a number of micro-gestures are manually identified and annotated into 32 classes.

Some micro-gesture examples:

Relevant publications:

The detailed description of the database, and related baseline results can be found in:

Liu X, Shi H, Chen H, Yu Z, Li X & Zhao G (2021) iMiGUE: An Identity-Free Video Dataset for Micro-Gesture Understanding and Emotion Analysis. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition: 10631-10642.

Contacts:

The database can be used, for example, in studying body gesture based emotion analysis. For more details or inquiring obtaining the database, please contact Dr. Guoying Zhao.

OuluVS database includes the video and audio data for 20 subjects uttering ten phrases: Hello, Excuse me, I am sorry, Thank you, Good bye, See you, Nice to meet you, You are welcome, How are you, Have a good time. Each person spoke each phrase five times. There are also videos with head motion from front to left, from front to right, without utterance, five times for each person.

The details and the baseline results for visual speech recognition can be found in:

Zhao G, Barnard M & Pietikäinen M (2009) Lipreading with local spatiotemporal descriptors. IEEE Transactions on Multimedia 11(7):1254-1265.

The database can be used, for example, in studying the visual speech recognition (lipreading). If you want to get this database, please contact Guoying Zhao.

Oulu-CASIA NIR&VIS facial expression database contains videos with the six typical expressions (happiness, sadness, surprise, anger, fear, disgust) from 80 subjects captured with two imaging systems, NIR (Near Infrared) and VIS (Visible light), under three different illumination conditions: normal indoor illumination, weak illumination (only computer display is on) and dark illumination (all lights are off).

The database can be used, for example, in studying the effects of illumination variations to facial expressions, cross-imaging-system facial expression recognition or face recognition. If you want to get access to this database, please contact Dr. Guoying Zhao.

SPOS database includes spontaneous and posed facial expressions of 7 subjects. Emotional movie clips were shown to subjects to induce spontaneous facial expressions, which include six categories of basic emotions (happy, sad, anger, surprise, fear disgust). Subjects were also asked to pose the six kinds of facial expressions after watching movie clips. Data are recorded by both visual and near infer-red camera. All together 84 posed and 147 spontaneous facial expression clips were labeled out from the starting point to the apex.

So far, spontaneous and posed facial expressions are usually found in different databases. The difference between databases (different experimental setting and different participants) hindered researches which considered both spontaneous and posed facial expressions. This database offers data collected on the same participants and under the same recording condition, which can be used for comparing or distinguishing spontaneous and posed facial expressions.

Related Publication

Pfister, T.; Xiaobai Li; Guoying Zhao; Pietikainen, M.; "Differentiating spontaneous from posed facial expressions within a generic facial expression recognition framework". Computer Vision Workshops (ICCV Workshops), 2011 IEEE International Conference, vol., no., pp.868-875, 6-13 Nov. 2011.

Contact information

To request a copy of SPOS database, please send email to Dr. Xiaobai Li.

Micro-expressions are important clues for analyzing people's deceitful behaviors. So far the lack of training source has been hindering research about automatic micro-expression recognition, and SMIC was developed to fill this gap. SMIC database includes spontaneous micro-expressions elicited by emotional movie clips. Emotional movie clips that can induce strong emotion reactions were shown to subjects, and subjects were asked to HIDE their true feelings while watching movie clips. If they failed to do so they will have to fill in a long boring questionnaire as punishment. This kind of setting is to create a high-stake lie situation so that micro-expressions can be induced.

In our ICCV2011 paper we released a first version of SMIC (referred as SMIC-sub), which only includes 6 subjects data recorded by a high speed camera of 100 fps.

Later in our FG2013 paper we released the full version of SMIC (referred as SMIC). For the full version of SMIC, 20 subjects were recorded and all together 164 spontaneous micro-expressions were found from 16 subjects. All data are recorded by a high speed (HS) camera of 100 fps. Also normal speed cameras with 25 fps of both visual (VIS) and near inferred (NIR) light range were added in the recording of the latter 10 subjects. So three datasets are included in SMIC database, which are SMIC-HS, SMIC-VIS and SMIC-NIR.

*UPDATE!

We now release an extended version of SMIC (referred as SMIC-E), which includes long video clips that contains some extra non-micro frames before and after the labelled out micro frames. SMIC-E also has three datasets according to their counterparts in SMIC, which are SMIC-E-HS, SMIC-E-VIS and SMIC-E-NIR. The SMIC-E-VIS and SMIC-E-NIR dataset both include 71 long clips of average duration of 5.9 seconds. The SMIC-E-HS dataset contains 157 long clips of an average length of 5.9 seconds. Three samples from the original SMIC-HS dataset are not included in the SMIC-E-HS due to data loss of the original videos. SMIC-E database could be used for purpose of micro-expression spotting.

Related Publication

Xiaobai Li; Pfister, T.; Xiaohua Huang; Guoying Zhao; Pietikainen, M., "A Spontaneous Micro-expression Database: Inducement, collection and baseline," in Automatic Face and Gesture Recognition (FG), 2013 10th IEEE International Conference and Workshops on , vol., no., pp.1-6, 22-26 April 2013

Pfister, T.; Xiaobai Li; Guoying Zhao; Pietikainen, M., "Recognising spontaneous facial micro-expressions," in Computer Vision (ICCV), 2011 IEEE International Conference on , vol., no., pp.1449-1456, 6-13 Nov. 2011

Contact information

To request a copy of SMIC database (please indicate which version/subset you need), please send email to Dr. Xiaobai Li.

Data description

Micro-expressions (MEs) are important clues to reveal people’s true feelings, but are difficult or impossible to be captured by ordinary persons with naked-eyes as they are very short and subtle. It is expected that robust computer vision methods can be developed to automatically analyze MEs which requires lots of ME data.

The current ME datasets mostly contain only one single form of 2D frontal color videos which is limited as some MEs might be more visible from side views. Research on 4D data of ordinary facial expressions has prospered, and we believe 4D data will be helpful for ME study. We collected the 4DME dataset, which is a new spontaneous ME dataset including 4D data along with three other video modalities (Grayscale, RGB, and Depth).

We are now releasing the first version of 4DME data (as 4DME). It includes two subsets, i.e., 1) data recorded by the Dimension Imaging 4D (DI4D) capturing system and reconstructed as sequences of 3D meshes (as 4DME-4D), and 2) data recorded by the frontal grayscale camera (60FPS, 640x480, as 4DME-gray). For each subset, we provide short clips of cut-out micro-expressions and macro-expressions (from onset to offset) for the recognition test, and also long clips (with context frames) for the spotting test. All data are annotated with emotion categories and action unit labels. Other versions of 4DME may be released later.

Relevant publications

Xiaobai Li, Shiyang Cheng, Yante Li, Muzammil Behzad, Jie Shen, Stefanos Zafeiriou, Maja Pantic, Guoying Zhao. "4DME: A spontaneous 4D micro-expression dataset with multimodalities," in IEEE Transactions on Affective Computing 2022.

Contact information

To request a copy of 4DME database, please send email to Dr. Xiaobai Li.

Contact via Firstname.Lastname@oulu.fi

Professors

• Dr. Janne Heikkilä, Director, room TS304 [Personal page] [G Scholar] [LinkedIn]
• Dr. Tapio Seppänen, Associate Director, TS303 [G Scholar]
• Dr. Guoying Zhao, Academy Professor, TS302 [Personal page] [G Scholar] [ResearchGate]
• Dr. Juho Kannala, TS326
• Dr. Mourad Oussalah, TS379
• Dr. Matti Pietikäinen (emeritus) [Personal page] [G Scholar] [dblp]
• Dr. Olli Silvén (emeritus) [G Scholar]

Associate Professors

• Dr. Miguel Bordallo López, TS349 [Personal page] [G Scholar] [LinkedIn]
• Dr. Mourad Oussalah, TS379

Assistant Professors

• Dr. Haoyu Chen, TS321

Adjunct Professors (Docents)

• Dr. Jukka Kortelainen, TS313 [Personal page] [G Scholar]
• Dr. Xiaobai Li [LinkedIn] [G Scholar]
• Dr. Li Liu [G Scholar]
• Dr. Qing Liu, TS325 [Personal page] [G Scholar]

Senior researchers and post-docs

• Dr. Snehal Bhayani
• Dr. Constantino Álvarez Casado, TS301 [Personal page] [ResearchGate]
• Dr. Md. Ziaul Hoque, TS324
• Dr. Fang Kang, TS321
• Dr. Anja Keskinarkaus [ResearchGate] [G Scholar]
• Dr. Puneet Kumar, TS330
• Dr. Yante Li, TS311
• Dr. Yang Liu, TS316
• Dr. Matti Matilainen, TS324
• Dr. Janne Mustaniemi, TS301 [Personal page] [G Scholar]
• Dr. Le Nguyen, TS322
• Dr. Matteo Pedone, TS329 [LinkedIn] [G Scholar]
• Dr. Mehrdad Rostami, TS327
• Dr. Mehdi Safarpour, TS301
• Dr. Pekka Sangi, TS328 [Personal page] [G Scholar]
• Dr. Zhuo Su
• Dr. Praneeth Susarla, TS315
• Dr. Xiaoyun Zhang, TS324
• Dr. Xiaoting Wu, TS317
• Dr. Qianru Xu

Researchers

• Amir Adilkhan, TS327
• Nastaran Taefi Aghdam, TS381
• Taufiq Ahmed, TS309
• Ahmed Hammad Alhawwary, TS317
• Samuel Boccara, TS308
• Hassan Bozorgmanesh, TS315
• Getnet Demil, TS377
• Gueddou Djaafer, TS317
• Mohamed Elhabebe, TS334
• Fatima Ez-Zahraa, TS336
• Muhammad Fahad Khalil, TS376
• Rabiul Hasan, TS317
• Timo Hyart
• Moinul Islam, TS377
• Xingxun Jiang, TS314
• Yan Jiang, TS327
• Miro Kakkonen, TS317
• Huai-Qian Khor, TS323
• Hui Kuurila-Zhang, TS323
• Manuel Lage, TS313
• Yante Li, TS311
• Yanying Liang, TS314
• Guiyu Liu, TS327
• Haotian Liu, TS314
• Abdelrahman Mostafa, TS316
• Maryam Munawar, TS336
• Nhi Nguyen, TS313
• Kai Noponen, TS309 [ResearchGate] [G Scholar]
• Enea Pedretti, TS309
• Niklas Saari, TS333
• Ali Salar, TS308
• Umer Saleem, TS376
• Amir Mollazadeh Sardroudi, TS310
• Marco Savic, TS316
• Seyedata Jodeiri Seyedian, TS309
• Atif Shah, TS316
• Ahmed Shaheen, TS381
• Fuzel Shaik, TS308
• Sasan Sharifipour, TS313
• Niranjan Sreegith, TS334
• Licai Sun, TS316
• Shuzhou Sun, TS310
• Youcai Sun, TS336
• Zhaodong Sun, TS334
• Joonas Tapaninaho, TS336
• Tuomas Varanka
• Alexander Vedernikov, TS334
• Huan Vo
• Tharindu Muthukuda Walawwe, TS327
• Mengting Wei, TS333
• Haidong Wu, TS317
• Yushun Xiao, TS311
• Wuti Xiong
• Huali Xu, TS322
• Huiyu Yang
• Yueyi Yang, TS311
• Hao Yu, TS336
• Kejuan Yue, TS333
• Yuanjun Zhang, TS376
• Jiehua Zhang, TS323
• Boyi Zheng, TS314
• Xiaoyan Zhou, TS317

Assistant researchers

• Tuomas Määttä, TS381
• Anastasia Voitenko, TS317

Visitors

Support staff

• Hannu Rautio, TS359

Alumni and past visitors

• Carlos De Oliveira Affonso [G Scholar]
• Gun Ahn
• Jeena Ainikkal Velayudhan
• Dr. Saad Ullah Akram [Personal page] [G Scholar]
• Shariful Md. Alam
• Iman Alikhani [G Scholar] [LinkedIn]
• Muhammad Awais Aslam [ResearchGate][G Scholar]
• Nooshin Bahader
• Hao Ban
• Dr. Neslihan Bayramoglu [Personal page]
• Djamila Romaissa Beddiar
• Dr. Muzammil Behzad [Personal page] [G Scholar] [LinkedIn] [ResearchGate]
• Mohamed Bengana
• Bilel Bennadji
• Dr. Salah Eddine Bekhouche [Personal page] [G Scholar]
• Hugo Bernicard
• Tuukka Bogdanoff
• Anastasiia Borodulina
• Dr. Zinelabidine Boulkenafet [ResearchGate] [G Scholar]
• Dr. Wheidima Carneiro de Melo
• Dr. Jie Chen [Personal page] [G Scholar]
• Dr. Weitong Chen
• Dr. Xu Cheng
• Dr. Yawen Cui
• Fan Dai
• Yulan Dai
• Wanxia Deng
• Berkay Elagöz
• Dr. Yuhua Fan
• Linpu Fang
• Dr. Angelos Fylakis [Personal page] [G Scholar]
• Mina Ghobrial
• Eric Granger
• Yundi Guo
• Dr. Abdenour Hadid [G Scholar] [LinkedIn]
• Ilyas Hamadouche
• Silviya Hasana
• Dr. Arto Hautala [LinkedIn] [G Scholar]
• Dr. Ilkka Hautala [ResearchGate] [G Scholar]
• Riku Hietaniemi [LinkedIn] [G Scholar]
• Dr. Jukka Hirvasniemi [G Scholar]
• Tuomas Holmberg
• Dr. Xiaopeng Hong [Personal page]
• Youssef Hosni
• Dr. Xiaohua Huang [Personal page] [G Scholar] [ResearchGate]
• Khor Huai-Qian
• Dr. Lam Huynh [Personal page] [G Scholar]
• Dr. Sami Huttunen [LinkedIn] [G Scholar] [GitHub]
• Brian Irvine
• Antti Isosalo [G Scholar] [solecris]
• Dr. Ilkka Juuso [LinkedIn][solecris]
• Anri Kivimäki
• Dr. Jukka Komulainen [PersonalPage] [G Scholar]
• Merja Kreivi-Kauppinen
• Dr. Jari Korhonen [G Scholar]
• William Kretzschmar
• Dr. Zuzana Kukelova
• Theo Lassalle
• Dr. Bo Li
• Guifeng Li
• Zhaofeng Li
• Bofan Lin
• Na Liu
• Dr. Xin Liu [Personal page] [G Scholar]
• Atiq Mahmud
• Arman Mansouri
• Dr. Jiri Matas [Personal page]
• Seyed Mirmojarabian
• Ahmed Mohammed
• Surat-E Md Mostafa
• Dr. Usman Muhammad
• Sofia Mäkikyrö
• Dr. Sofeem Nasim
• Dr. Phong Nguyen [Personal page] [G Scholar]
• Xuesong Niu
• Shuichi Otani
• Yuma Ouchi
• Niina Palmu
• Dr. Juha Partala [Personal page] [G Scholar]
• Dr. Wei Peng
• Charmin Pingamage
• Zalan Rajna [ResearchGate] [G Scholar]
• Yohani Ranasinghe
• Dr. Mohammad Sabokrou
• Thibault Sap
• Mahalakshmy Seetharaman
• Jaakko Seppänen
• Tiina Seppänen, TS377 [ResearchGate] [G Scholar]
• Changchong Sheng
• Dr. Henglin Shi
• Dr. Jingang Shi [G Scholar]
• Xin Shu
• Jiaojiao Su
• Kaushik Sundarajayaraman
• Rokayia Sultana
• Mohammad Tavakolian [G Scholar] [LinkedIn] [Researchgate] [Personal page]
• Aleksei Tiulpin
• Khanh Tran-Thuong [G Scholar]
• Tuomas Tuokkola
• Sercan Türkmen
• Dr. Nhat Vo
• Xiaowen Wang
• Takahiro Watanabe
• Muhammad Waqas
• Dr. Liang Wu
• Dr. Baiqiang Xai
• Yuxi Xia
• Zhaoqiang Xia
• Dr. Yingyue Xu [G Scholar]
• Kohei Yamamoto
• Ruijing Yang
• Dr. Markus Ylimäki [Personal page] [G Scholar]
• Dr. Zitong Yu
• Dr. Stefanos Zafeiriou [Personal page]
• Dr. Delu Zeng
• Jingliang Zhang
• Jiawei Zhang
• Jinghua Zhang
• Xiaochao Zhao [G Scholar]
• Yunrui Zhao
• Dhiaa Eddine Zidane

Center for Machine Vision and Signal Analysis
P.O Box 4500
FI-90014 University of Oulu
Finland

Internal mail: 9CMVS

Our street address is

Center for Machine Vision and Signal Analysis
Erkki Koiso-Kanttilan katu 3, door E
90570 Oulu

The research groups of CMVS are located in the third floor of the Tietotalo, near Linnanmaa main bus stop (Yliopisto E).