Infotech Oulu Annual Report 2012 - Intelligent Systems Group (ISG)

Professor Juha Röning and Professor Jukka Riekki,

Department of Computer Science and Engineering , University of Oulu

juha.roning(at), jukka.riekki(at)


Background and Mission

The main scientific objective of the basic research conducted in the Intelligent Systems Group is to generate new applicable knowledge on intelligent systems, and to generate positive societal impacts by applying this knowledge using scientifically plausible methods and state-of-the-art technology. We conduct research on spatial computing, collective intelligence, sensor networks, optimization of industrial processes, mobile robotics and cybernetics, human-computer interaction (HCI), human-robot interaction (HRI), computer networks, and security in complex information processing systems.

Our strategic research themes, whereby we aim to conduct world class basic research, are: 1) Safety and security on all levels of intelligent systems, 2) Data mining, with a special focus on optimization of industrial processes and well-being, 3) Human-environment interaction with a special focus on novel physical user interfaces, robot-environment and human-robot interaction, 4) Mobile robotics and cybernetics with a special focus on spatial computing, adaptation, and field robotics, and 5) Sensor networks with a special focus on understanding behaviour and patterns in our everyday environment.

We have conducted basic research related to these research themes for over ten years. Our team consists of two professors, five postdoctoral researchers and 21 doctoral students. The annual external funding of the group is more than two million Euros, in addition to our basic university funding. There have been 17 completed doctoral degrees from the group. From the research of the group, eight spin-out companies have been established so far: Codenomicon, Clarified Networks, Hearth Signal, Nose Laboratory, Nelilab, Atomia, Probot and IndoorAtlas.

We co-operate with many international and domestic partners. In applied research, we are active in European projects. In addition, several joint projects are funded by the Finnish Funding Agency for Technology and Innovation (Tekes) and industry. We were a research partner in the Data to Intelligence (D2I), Internet of Things (IoT) and Cloud Software ICT SHOKs. In the Cloud Software Program ICT SHOK, we are responsible for the cloud security CoP.

In the II City project, we collaborated with the University of Lapland and Sonic Studio from Piteå, Sweden, and in the Pervasive Service Computing project with Shanghai Jiao Tong University from China. These projects were funded by the European Union (Interreg IVA North) and the Academy of Finland (the MOTIVE program). Also we are collaborating with the Tokyo University of Agriculture and Technology in the Interactive Context-aware System for Energy Efficient Living project, funded by the Academy of Finland.

We are active in the scientific community. For example, Prof. Juha Röning was appointed as visiting professor of Tianjin University of Technology. He was also invited as the Robot Science Adviser of Tianjin Science and Technology Center for Juveniles. He served as a member of the Board of Directors in euRobotics and as a member of the SAFECode International Board of Advisors, and as a chief judge in the European Land Robot Trial (Elrob 2012) in Thun. Elrob is the biggest outdoor robot event in Europe, and participants are research facilities and companies that represent the state-of-the-art in Europe in this research area. Prof. Jukka Riekki gave an invited talk at the 4th International Research Workshop with focus on Near Field Communication (NFC2012) on March 13th, 2012 in Helsinki, Finland. Several doctoral students made research visits during 2012 as well. Teemu Leppänen started a one-year research visit to the University of Tokyo and Mikko Polojärvi continued his visit to Hokkaido University. Several members of the group co-chaired workshops and were also on the committees of international conferences.


Scientific Progress

Research on mobile robotics

This research includes several topics to support technologies of mobile robotic systems. Our main research topics include motion planning, energy management, magnetic localization, information representation and development of a mobile robot system. A practical robotic system is being build that can operate in indoor among people. It is called Minotaurus.

Dynamic motion planning: Research on making better motion planning in a real world environment is being studied. The planner combines local obstacle avoidance with dynamic model and performance capabilities of the robot, as well as the capabilities of perception system. Local obstacle avoidance gets its input from several sensor systems, combines the information to knowledge of the current state of the environment and makes a route plan among obstacles. Perception modules include laser scanners, data processing, and several vision algorithms using a Kinect range camera. These can detect obstacles and humans in the surrounding environment. Several avoidance algorithms can be run in parallel, and the same algorithms can be run with varying parameters, with best result for the current situation being selected. This improves the planner to be able to find a proper route, even when one of the algorithms might fail, or certain parameters are not suitable for the current situation.

The objective of our indoor localization research is to develop methods for exploiting magnetic field anomalies in positioning. The idea is based on the analysis with various indoor magnetic field datasets showing that indoor magnetic fields provide sufficient spatial variation and temporal stability to permit inference about sensing locations, given noisy measurements. Recent years, we have published various papers presenting magnetic field localization in robotic and human contexts.

Based on the magnetic field localization studies, a new start-up company, called Indoor Atlas Ltd., was founded in 2012. This company offers indoor positioning technologies for various application areas. The company has generated high interest in international technological magazines.

In autonomous water quality monitoring research, a new robotic platform was developed in 2012 (illustrated in Figure 1). This platform is designed to ease and improve the collection of water quality parameters currently performed manually or through fixed wireless sensors placed on the lakes and on-shore. The first pilot tests will be performed during the summer of 2013.

Figure 1. Robotic vessel for autonomous water quality measurement studies. The vessel has an on-board computer, sensor winch, camera surveillance and radio connection to communicate with the human operator on-shore.

In our research on magnetic field localization and SLAM, we have put a strong emphasis on light-weight methods running entirely on mobile platforms, such as Android smartphones and tablets. Compact map representation and effective algorithms are essential when using devices with very limited resources, and we have developed methods to tackle the problems arising from very sparse data and high uncertainty levels produced by low-cost and noisy smartphone sensors. Our work is continuing toward an autonomous mobile robot system based solely on smartphone sensors that is able to intelligently build a map of the magnetic environment.

Social robot scenarios are particularly difficult because of the dynamic (often crowded) environment. Magnetic field localization is not affected by surrounding people like laser scanners and cameras for example, and is therefore very promising in these kinds of scenarios. While our method is used to localize the robot, the other sensors can be assigned to handle the social tasks. We have also developed localization methods that are usable by both robots and humans equipped with similar mobile devices (see Figure 2 and 3).

Figure 2. An iRobot Create equipped with a Samsung Galaxy Nexus smartphone utilizing the same magnetic map for localization as the person holding an Asus Nexus 7 tablet. The same localization unit can be used by both humans and robots in social robot scenarios.

Intelligent battery systems are being developed partly in the WintEVE project. The focus is on developing an energy management system with improved energy efficiency, and operation in varying conditions (like cold), as well as on gaining a better understanding of power usage and how long a robot is still operable. Work has included study of upcoming battery technologies that might improve drastically the capacity weight ratio in the near future, as well as making a system that can profile and control power usage in the robot. For example, if a certain task can be performed with fewer sensors, the other sensors of the robot are turned off. Neural networks are also used for creating a better model of the true capacity state of the battery. Further, the capacity and consumption will be used for predicting upcoming operating time as part of planning and task scheduling.

Figure 3. CSE lobby (above) and Discus Entrance hall (below) at the University of Oulu, with corresponding magnetic field maps. The figure illustrates the strong variation in the magnetic fields in these environments that makes the localization possible.

The Evolutionary Active Materials (EAM) project, which is funded by the Academy of Finland, is a joint effort of the Computer Science and Engineering laboratory (CSE) and the Microelectronics and Materials Physics laboratories. The aim of the EAM project is to develop novel evolutionary computation (EC) based design methods for active and versatile materials and structures. The first components are being developed through a novel holistic design process utilizing constantly increasing computation power, the development of multi-physics simulators, and EC techniques such as genetic algorithms.

During 2012, the structure of a cymbal type piezoelectric actuator was developed further by using a combination of a genetic algorithm, and a multiphysics simulator. The shape of the steel cap was optimized by directing control points of a spline curve with a genetic algorithm. The cymbal (Figure 4) was successfully optimized to produce maximum displacement which was 28.9% higher compared to the linear endcap profile. The cymbal with an optimized shape of the steel cap was also build and measured. It was discovered that reality and simulation coincide very well.

Figure 4. The side profile of the optimized cymbal.

Also the behaviour of the cymbal structure was studied with genetic algorithms by optimizing the maximum force with different values of displacement and thickness of the steel cap (cf. Figure 5).

Figure 5. Force as a function of displacement for different thicknesses of the steel cap. Every dot has been optimized with GA.

The aim of the EAM project for 2013 is to develop further the implementation of the optimization module to allow more efficient distribution of computational tasks, and to provide an opportunity to optimize more complex hybrid structures.

Research on human-environment interaction

Our current work on human-environment interaction focuses on physical user interfaces and human-robot interaction. In physical user interfaces, mobile terminals are used as physical objects rather than as traditional I/O devices. Our current studies concentrate on touch-based interaction: users interact with the local environment by touching objects with their mobile terminals. The touch-based user interfaces use NFC technology (i.e. RFID technology for mobile phones): an act of touching brings an NFC reader near an NFC tag, and hence the data in the tag is read and delivered to the system. The objects that can be touched are advertised in the environment by NFC icons, graphical icons resembling the icons of the graphical user interfaces of computers and other user terminals.

In 2012, we started developing a novel NFC device, NFC Pad, together with Offcode Ltd, an Oulu based embedded systems company. This work started in the Active Learning Spaces project funded by Tekes and coordinated by the University of Tampere. Moreover, we started developing applications for the NFC Pad in the Digital Services ICT SHOK, also in this project together with Offcode, the University of Tampere, and several other project partners. Furthermore, we continued our participation in the Future School Research Second Wave project that is funded by the European Social Fund and the city of Oulu and is coordinated by the Faculty of Education at the University of Oulu. We develop also applications for NFC phones in these three projects.

The NFC Pad allows teachers and students to create study material using traditional methods; drawing on paper and moulding objects from clay, for example. The pad is a flat surface divided into 6 x 4 cells, each equipped with an NFC reader. A student places first a sheet of A4 paper on the pad; this paper presents a task, for example a sentence with some missing words. Then the student performs the task by placing pieces of paper or other objects augmented with NFC tags on the task paper, for example objects representing the missing words. The pad reads the information from the tags (the task id from the task paper’s tag) and provides visual (LEDs) and audio feedback. Teachers can program their own pad applications easily by drawing a task paper, placing the paper and correct objects on the pad, and recording the correct answer (Figure 6).

Figure 6. An example of tasks that can be realized with the NFC Pad. In this task, a child has to select correct pairs of fruit (blue cards) and sums (red cards) based on the values of fruit that is presented on the task paper - and place the pairs to the top left corner of the task paper.

For NFC phones, we are developing situated mobile learning applications. This work is performed in the Active Spaces project together with Context Learning Finland. The applications enable learning experiences in which the virtual and real (i.e. physical) environments are interconnected. The first application teaches children at schools about security measures in dangerous situations. NFC tags are placed on school premises, and scenarios are prepared for each tag location as multimedia presentations. When a child touches a tag with a phone, the application presents the corresponding multimedia presentation and asks questions about the scenario; for example, how to put out fire in the presented situation.

In the Digital Services ICT SHOK, we have created an authoring tool targeted at primary school teachers named NFC-ACT. This PC application can be used to create NFC-enabled learning games for mobile phones. The games follow the same structure: a game presents a challenge on the phone screen and the  child solves the challenge by touching the correct NFC tag among the tags placed on objects in the environment, as is illustrated in Figure 7. Teachers create games by defining the application flow and the multimedia content to be shown on the screen at each stage. This tool can also be used to write the NFC tags using an external NFC reader and to deploy the games to children’s mobile phone using Bluetooth.

In the Future School 2nd Wave, we have collaborated with researchers from the Department of Education and with teachers to create NFC based learning games for pupils with heterogeneous backgrounds (ages from 7 to 12, different nationalities). Our aim is to enhance learning Finnish, which the target group acquires as a second language. The teachers used the games as tools to introduce vocabulary (colours, body parts, Christmas words, etc.) to the pupils.

Figure 7. Children are practising the Finnish names of the parts of the body using our NFC-based mobile application.

We continued our work on a general purpose data visualization framework in the Internet of Things ICT SHOK. The purpose of this framework is to simplify the creation of systems that connect user-specific data from data sources to visualization applications or other applications that modify their behaviour based on data input. We developed a prototype visualization application on top of the framework. The application visualizes each user’s data as a fish; the values given by a user determine the behaviour of the corresponding fish. The application is shown in Figure 8. This visualization prototype was deployed for two classes at the Ritaharju School. 

Figure 8. Visualizing users’ data in a virtual aquarium; the values given by a user determine the behaviour of the corresponding fish.

We also started to search for answers to the more interesting question: what is there beyond simple visualization that can be done to drive motivation? Sensor data alone is not sufficient; it is the representation that makes the data accessible to normal users and is able to persuade as well.

We also started working on a general framework for visualizing the status of IoT systems on a 3D virtual environment. We started exploring the visualization possibilities by using data from a home environment provided by a ThereGate device, the Home Energy Management System (HEMS) gateway, and a Finwe Key2phone, a lock system operated by mobile phones. We are developing a general architecture for collecting and visualizing data acquired from the home environment. The data is stored in a Cloud database and visualized using the RealXtend platform for 3D virtual worlds. Visualized data, for example power consumption and temperature values, can motivate people to use the devices in their homes in a smarter and more energy-efficient way, as shown in Figure 9.

Figure 9. Home environment data visualization system architecture.

We continued exploring smart space interaction, specifically how knowledge-based technologies enable smart spaces not only to adapt their behaviour according to the actions of users, but also to initiate interaction when necessary. Moreover, we studied the management of smart spaces (see Figure 10). We suggest distinguishing the context-based adaptation activities of smart spaces from monitoring and controlling of smart spaces. With this vision, we have created a framework for meta-level control of smart spaces. Simulation study was conducted as proof-of-concept. Currently we are working on a real prototype to validate this approach for a single ubiquitous system.

Figure 10. Meta-level control of smart spaces.

Research on sensor networks

The research on sensor networks is targeted at understanding behaviour and patterns in our everyday environment. We are studying sensor data processing platforms, sensor data processing algorithms, lightweight knowledge representations suitable for sensor networks, and reasoning. In 2012, we continued work in the Internet of Things ICT SHOK and Data to Intelligence ICT SHOK.

In the Data to Intelligence SHOK, we worked on collecting, processing and storing traffic related data from the City of Oulu region. We built a cloud service for storing real time information collected by taxi cabs operating in the city. We studied different data cleaning and synchronization procedures for dual laser measurements, and car engine and location data. We also fused environmental data with these measurements to enable better situation awareness and driver recommendation systems for different actors in traffic, as shown in Figure 11. Situation awareness enables the users to understand what is happening around them, what is going to happen next and what kind of possibilities are there to reason and act. 

Figure 11. Information from digital maps, weather sensors and traffic infrastructure can be fused to data collected by vehicles to enable situation awareness.

We continued our research on adding semantics to Internet of Things systems in the Internet of Things ICT SHOK. We studied how the benefits of semantics can be enjoyed without sacrificing energy efficiency and short latencies. We utilize the existing Semantic Web technologies as their interoperability allows application developers to use nodes implemented and deployed by others. In 2012, we built a system connecting sensor nodes to knowledge-based components, and evaluated in this system different data formats in terms of their semantic expressivity and resource consumption.

We extended our previous research on the UBI-AMI home energy consumption monitoring system by developing a wireless sensor network based application for monitoring the energy consumption of buildings in real-time. This architecture is based on ideas of dynamically configuring the sensor network for applications, dynamically composing sensor-based services from the resources in the network and distributing computational load in the network. An example of this is shown in Figure 12.

Figure 12. An example of a dynamically configurable sensor network system.

In the MAMMOTH project, we continued research on distributed system architecture for M2M systems and the Internet of Things. This architecture is based on the above ideas, with the addition of mobile sensing platforms and generic IoT devices. Extending both the UBI-AMI system and activities in the MAMMOTH project were conducted in co-operation with MediaTeam. In the fall 2012, we started co-operation with the University of Tokyo, where one of our doctoral students, Teemu Leppänen, is studying participatory sensing applications utilizing smartphones as mobile sensing platforms.

Research on human-robot interaction

Two on-going projects, the Academy funded AFHRI (Affective-Human-Robot interaction) and the European Regional Development Fund (ERDF) funded Minotaurus com-bine the theoretical and practical aspects of creating service robotics of the future. Both projects are in co-operation with the Center for Machine Vision Research (CMV). Human-Machine communication and robot operation in real world conditions are major topics. Key aspects are putting research results into practice, and integrating them as a part of the operation of the robot. The main challenges are representation of information, and adapting a set of algorithms to the current state in the surrounding environment. Human friendliness of the outward appearance of the robot is driving the design of the robot.

Studies on interaction with a machine have been continued from various perspectives. The capability of the Minotaurus robot has been improved with new sensors, perception processing and physical capabilities. The robot has improved with an attached commercial arm, developed by Jaco Kinova. Jaco is originally for wheelchair operation, and it provides a great reachability-weight ratio. Development of arm software and coordination on a mobile platform will continue on 2013, combining perception and motion planning to real-time operation of the arm, and providing the possibility to pick up small objects from a table and the floor.

At the end of the year, the Minotaurus robot took a part on demonstration for a University funding club meeting with great success, performing its capabilities of dialogue with humans, navigating among obstacles, and basic picking up of objects.

Interaction with a machine has been studied from various perspectives. The combination of machine vision, speech recognition and synthesizing, touch and touchless interactions, along with the robot’s operation in the environment requires a software platform that processes, distributes and stores information efficiently. Real-time Linux based operating system services, along with general purpose representation for information (called Markers) have been developed to support integration of the algorithms. Marker representation is used for representing the output of sensor processing algorithms, for representing a model of environment and obstacles around the robot, and information related to the robot’s tasks.

Speech and language processing in Human-Robot Interaction: in the Minotaurus project, speech and natural language processing have been studied to be able build a natural dialogue interface between a robot and a human. The research has been concentrating on automatic speech recognition (ASR) based on a CMU Sphinx engine. Furthermore, a simple language understanding model, based on keyword spotting and name entity recognition, is implemented to be able to recognize the meaning of a speech utterance. We have used a pre-trained English acoustics model based on hidden Markov models (HMMs), as well as trained new HMMs for the Finnish language in conjunction with appropriate dictionaries and language models for robot communication application. Finally, the speech understanding module is connected to a talking mouth, which is able to synthesis speech audio and lip movements. Responses are based on the robot’s internal knowledge and local database, as well as on external knowledge from the web. In the future, work will be concentrating on statistical language processing and understanding, as well as modelling of uncertainty in communication, to be able to implement natural dialogue and interaction between a human and a robot.

The Human-Robot Interaction system contains several “contact points”, where interaction with the system occurs. These contact points are large displays, equipped with multiple cameras, a microphone system and software for producing dialogue and virtual face animation that matches the synthesized speech. The mobile robot is one contact point in the system.

The robot is based on a Segway RMP200 platform, and equipped with several sensors to receive information from the surrounding environment and humans. Two Kinect sensors are used as a 3D vision camera, one looking towards the floor and one looking forwards. Multiple 2D scanning lasers are additionally used for close distance obstacle avoidance, and for refining the estimation of the robot’s movement. Further, the robot is equipped with several microphones to reason the source direction for audio and for speech recognition.

The robot uses a graph based navigation map for finding a raw route for driving around the laboratory, and an RRT (Rapidly-exploring Random Tree) for finding a free path from the current location to the next route position. For raw location, the robot uses a Kinect sensor and detection of visual clues like room signs and room numbers. Also a laser scanner profile is used for matching with prior knowledge of the corridor shape profile. Later, technology using magnetic field quality for localization will be merged to the robot.

Research on data mining

The future of living and housing: a project entitled “An Interactive Context-aware System for Energy Efficient Living” (INCA) started in 2011, and lasted until the end of 2012. INCA was funded by the Academy of Finland, and the project goal was to develop an interactive and context-aware feedback and control system for rationalizing energy efficient living. The system itself collects personalized data from the environment and offers information about consumption habits for the individuals. With the help of the information, it is possible to advise people on more energy efficient living, and to give recommendations for avoiding unnecessary energy consumption. The research in the INCA project was performed jointly with Associate Professor Kaori Fujinami from the Tokyo University of Agriculture and Technology (TUAT). During 2012, our research topics have been concentrating on feedback systems, while continuing to develop the low-level energy consumption measurement systems that are needed to make the feedback systems work. We are developing easy-to-install, low-cost electricity and water meters, which could be installed externally without any permanent changes in the house infrastructure. Fujinami’s group at TUAT are developing techniques for detecting activities of wasteful energy usage. In addition, they are studying persuasive control and feedback technologies.

Non-Intrusive Appliance Load Monitoring: it is possible to estimate per appliance energy consumption using a single sensor installed at the main electrical panel or using the power company’s smart meter if available. This can be done by noticing the changes in power consumption when appliances turn on or off. These changes are unique to each appliance so it is possible to determine which appliance caused the change. This way we can estimate the operating time of each appliance and therefore approximate the amount of energy used by the appliance. A prototype feedback system that works in real-time was developed for the project. A screenshot of the system in use can be seen in Figure 13.

Figure 13. A real-time feedback system showing energy consumption divided between individual appliances.

A Water Consumption Meter based on Mechanical Vibration Sound and Machine Learning techniques: in this research topic, we developed an easy-to-install, low-cost water consumption meter, which could be installed on an existing house without any permanent changes in infrastructure. The water flow is estimated from the mechanical vibration of the water pipe. A contact microphone was used to be able to eliminate background noise and environmental sounds. The water flow estimation was performed using machine learning algorithms based on statistical regression. A supervised training and calibration set is collected using a commercial flow meter. After a short calibration period, our meter can estimate water flow and consumption only using the mechanical vibration of the water pipe, where different frequency characteristics of water flow sound, together with the weighted K-nearest neighbour and Gaussian process regression models are applied to perform predictions of time-series data. In addition, an embedded system is developed to process incoming data in real-time.

Our project partners from TUAT focused more on wasteful activity recognition and feedback systems. They used energy consumption information from our work to aid their systems. In addition to energy consumption information, they used several sensors to detect wasteful activities related to refrigerator use. A long-term study was performed to test the effects of feedback on improving energy efficiency of fridge use. A picture of what the feedback system looks like in practice can be seen in Figure 14.

Figure 14. The PerFridge graphical user interface uses a nature inspired metaphor for feedback in the main screen. Bar graphs and numbers are also available for those who are interested.

New methods for activation of young men: the multidisciplinary MOPO project combines traditional health promotion, modern technology and the measurement of physical activity. The aim of the study is to provide knowledge on physical activity, the relationship towards physical activity, information behaviour, fitness, health, nutrition, life habits and cultures of young men. A novel wellness coaching service for preventing marginalization and promoting physical activity and health in young men is being developed in the project. The information obtained in the study can be used to promote the wellness of young adults in education, and in the study and decision-making of the professionals in social and health services.

Altogether about 5000 conscription-aged men (four call-up age classes) have been/will be invited to the MOPO study. The condition, wellbeing, health, relationship towards physical activity, information behaviour and the use of media and technology will be clarified during the years 2009-2013 with questionnaires, measurements and interviews. The contents of the service network will be developed together with the city of Oulu, conscription-aged men and several companies.

The data mining group has been responsible for the development of the gamified webportal, Figure 15. Young men collect data about their daily activity using a wrist-worn Polar Active or a mobile application developed by ISG. The mobile application recognizes the user’s daily activities (walking, running, biking, riding a car and idling) using mobile phone accelerometers. The algorithms used in the recognition process are independent of the user and the mobile phone’s operating system. Information about daily activity is uploaded to the portal, and based on the data, tailored feedback is given to the user. In addition, through the portal, tailored information, for instance about health, and nutrition is given to the user. Users of the portal also have access to virtual Oulu, see Figure 16. As mentioned, the portal is gamified: based on physical activity and activity in the portal, points are given to the user. These points can be used in a mobile game called “Clans of Oulu” where different clans are trying to conquer the city of Oulu, Figure 17. The game has been developed by LudoCraft.

Access to the portal and/or the activity monitor was given to approximately 300 conscription-aged men. The test started in autumn 2012 and lasted 3 months. The purpose of this test was to find answers to the following questions:

  • Did the portal and the game, along with success in it, motivate the user to exercise?
  • What was the overall opinion about the ICT platform?
  • Did the health and exercise messages given through the platform motivate the user to exercise?
  • Did the sensors alone motivate the user to exercise?

During the test, the behaviour of the portal users was tracked and this information has been exploited to develop a new version of the portal for 2013 call-ups.

Figure 15. Webportal: offers tailored information, for instance, about physical activity, fitness, health, and nutrition.

Figure 16: Virtual Oulu.

For several years now, one of the interests of ISG in data mining research has been the study of ethical issues associated with data mining. These include the well-established issue of protecting the privacy of data subjects, and also a number of previously overlooked issues such as fairness and trust. The ethics research of ISG also emphasizes the potential of data mining as an instrument for pursuing things of value rather than just a threat to values that needs to be controlled. All these perspectives are combined in the MOPO projects: the outcome of the project, a persuasive video game, may help the study subjects adopt healthier lifestyles, but its development involves potential ethical problems that must be actively identified and worked against. An ethical analysis of the technical work in the project is carried out and will be published alongside the work itself.

The final service will be developed based on the experiences of the pilot. The intervention started in the autumn of 2012. Operators of the study are the Oulu Deaconess Institute’s Department of Sports and Exercise Medicine, the University of Oulu, the City of Oulu, the Virpiniemi Sports Institute, the Finnish Defence Forces and wellness technology companies from Northern Finland. The project website can be found from

Figure 17: Clans of Oulu is a game where you cannot succeed unless you are physically active and use the portal actively. Based on activity the user gets points which can be used to conquer the districts of Oulu by the user’s clan.

Data mining methods for steel industry applications: the Intelligent Systems Group has long experience in advancing data mining methodologies in the steel industry, and is a member of the Centre for Advanced Steels Research - CASR, which is one of the interdisciplinary umbrella organizations of the University of Oulu. In 2012, advanced data analysis solutions for the needs of the steel industry were developed in one active project and in two PhD studies.

Production efficiency with probability predictions: in the prob2E-project (Probability Predictions to Production Efficiency) the aim was to develop further a probability prediction approach to develop and utilize statistical models and validate the benefits that industry can achieve by employing distributional predictions instead of point predictions. The project was funded by Tekes, Ruukki, Ovako, and Valio. In June, the final results of the project, as well as, the viewpoints of the participating companies were presented in a seminar at the University of Oulu.

The applied algorithms were based on probabilistic predictors, and one of the challenges was to develop effective ranking methods for these kinds of models. The proposed Exceedance Probability Score (EPS) was published in the Journal of Statistical Theory and Practise. New methods for predicted distribution visualization were developed as well, and the results will be published in Communications in Statistics – Theory and Methods. In Figure 18, an example of the predicted distribution visualization with beanplots is presented. The left hand side of each bean corresponds to the real distribution with visualized observations, and the right hand side corresponds to the predicted distribution.

Figure 18. The predicted (on the right hand of each bean) and observed distributions of Charpy-V data when the data is clustered to nine clusters based on predicted mean and deviation.

In the first of the two PhD studies, the goal lies in the development of new data analysis methods for modelling of hot steel plate rolling schedules. Previous attempts to model statistical temperatures and forces in each pass during rolling have not succeeded due to the challenge of the problem. The advanced functional data analysis methods will enable achievement of better accuracy of time-dependent phenomena like pass schedules modelling. Commonly, physical models have been applied to the task. Here, statistical models are suggested instead. The advantages of a statistical model are generality, maintenance and accuracy. A statistical model is not limited just to one production plant but the same models can be applied also to other plants. As the plate temperature and force cannot be measured directly during rolling, there is a strong need for well-adapted models. The industrial partner for the research is Ruukki Metals, Raahe Works.

In the research, the problem of missing data has been considered. The measurements of rolling temperature contain much missing data due to the hot and moist circumstances that occur near the pyrometers. By employing semi-supervised learning, the information contained in the observations with missing response measurements was utilized to improve the learning process by employing the COREG-algorithm. As a result, a slight increase in the prediction accuracy was observed.

In 2012, significant efforts have been made to find features that describe the rolling process, so that the feature matrix can be used to fit regressors that predict accurately the temperature at different stages of the rolling. The compression of the history of the pass schedule has been one of the most challenging feature extraction tasks. The photograph in figure 19 has been taken during the steel plate rolling process.

Figure 19. The steel plate rolling mill at Ruukki Metals, Raahe Works.

The other PhD study concerns methods for exceedance probability estimation in the case of highly scattered measurement sets. The situation may occur when product quality is verified with several test samples, and thus, traditional point prediction based modelling methods are not sufficient. Density forecasting methods are needed when not only the mean but also the deviance and the distribution shape of the response depend on the explanatory variables.

In the research, the impact toughness of the steel products has been modelled. The rejection probability in the  Charpy-V quality test has been predicted with mean and deviation models, a distribution shape model and a quantile regression model. The proposed methods have been employed in two steel manufacturing applications with different distributional properties, and it has been shown that especially the case with diverse product assortment benefits from the distribution modelling. The industrial partners of the research are Ruukki Metals, Raahe Works and Ovako, Imatra.

Cognitive load study: Eija Haapalainen Ferreira is collaborating with Prof. Anind K. Dey from the Human-Computer Interaction Institute of Carnegie Mellon University (USA) to study how to assess the cognitive load of a person in real-time, based on psycho-physiological measurements. Task interruption, divided attention and multitasking cause split attention that increases our cognitive demands. This is a problem, especially while performing tasks during which momentary meandering can have serious consequences (e.g. driving, manoeuvring heavy machinery, controlling air traffic, walking in traffic, performing surgery, etc.). The ability to measure real-time changes in the cognitive load of individuals would provide a key tool to help mediate our attention resources when performing cognitive intensive tasks (Figure 20). In addition, our cognitive abilities change as we age.  Therefore, understanding issues of attention demand is particularly essential for the elderly. One of the primary research questions of the study is to explore whether the same tools can be used to assess cognitive load for both younger and older adults.

Figure 20. Real-time cognitive load assessment has wide applications in predicting drivers’ ability to maintain vehicle control.

Research on software security

Within the Intelligent Systems Group, the Oulu University Secure Programming Group (OUSPG) has continued research on security and safety in intelligent systems. Security and safety challenges in intelligent systems are threefold: increasing complexity leads to unforeseeable failure modes, quality is not the priority and awareness is lacking. We have approached the challenges from these three directions in our research.

Complexity - Model Inference and Pattern Recognition: we work under the premises of unmanageable growth in software and system complexity and emergent behaviour (unanticipated, not designed) having a major role in any modern non-trivial system. We have worked on natural science approaches to understanding artificial information processing systems. We have developed and applied model inference and pattern recognition to both content and causality of signalling between different parts of systems.

Quality - Building Security in: software quality problems, wide impact vulnerabilities, phishing, botnets, criminal enterprise have proven that software and system security is not just an add-on despite the past focus of the security industry. Instead, security, trust, dependability and privacy have to be considered over the whole life-cycle of the system and software development, from requirements all the way to operations and maintenance. This is furthermore emphasized by the fact that large intelligent systems are emergent and do not follow a traditional development life-cycle. Building security in not only makes us safer and secure, but also improves overall system quality and development efficiency. Security and safety are transformed from inhibitors to enablers. We have developed and applied black-box testing methods to set quantitative robustness criteria. International recognition of the Secure Development Life Cycle has provided us with a way to map our research on different security issues.

Awareness - Vulnerability Life Cycle: Intelligent systems are born with security flaws and vulnerabilities, new ones are introduced, old ones are eliminated. Any deployment of system components comes in generations that have different sets of vulnerabilities. Technical, social, political and economic factors all affect this process. We have developed and applied processes for handling the vulnerability life-cycle. This work has been adopted in critical infrastructure protection. Awareness of vulnerabilities and the processes to handle them all increase the survivability of emergent intelligent systems for developers, users and society.

These research goals are reached through a number of research activities.

Secure Software Development Lifecycle in Cloud Computing as part of the Cloud Software project - we approach all three goals by researching practical ways of building security into complex Cloud Computing services, from the design phase to actual operational use. The dependencies of a single cloud based web service visualized by technology and location are shown in Figure 21.

Figure 21. Dependencies of a single cloud based web service visualized by technology and location.

Development and Industrial Application of Multi-Domain Security Testing Technologies (DIAMONDS) enables efficient and automated security testing methods of industrial relevance for highly secure systems in multiple domains (including for example banking, transport or telecommunication). In particular, over 90% of software security incidents are caused by attackers exploiting known software defects. DIAMONDS addresses this increasing need for systematic security testing methods by developing techniques and tools that can efficiently be used to secure networked applications in different domains. DIAMONDS will leverage systematic, model-based testing and monitoring approaches for security testing to enable highly secure systems by early testing and test automation. Advanced model-based security testing and visualization methods will allow the early identification of design vulnerabilities and efficient system/test design targeting security issues. In Figure 22 is an example of port scanning visualization. The DIAMONDS project convinced visitors at the ITEA & ARTEMIS Co-summit 2012 in Paris, and received the best-booth award.

Figure 22. Port scanning visualized in an industrial automation network.

Identification of a protocol gene: this research, PROTOS-GENOME, approaches the problems of complexity and quality by developing tools and techniques for reverse-engineering, and identification of protocols based on using protocol genes - the basic building blocks of protocols. The approach is to use techniques developed for bioinformatics and artificial intelligence. Samples of protocols and file formats are used to infer structure from the data. This structural information can then be used to effectively create large numbers of test cases for this protocol. In 2012, the project further developed the existing methodology, resulting in improvements in efficacy and discovering a number of vulnerabilities in web browsers.

Economic Models for Collaborative Access Network Provisioning (EMCANP) was launched in 2012 in cooperation with the Centre for Wireless Communications and the Virginia Polytechnic Institute and State University. The project explores the potential for capacity increase in heterogeneous wireless networks through increased collaboration, both among network operators and among end users. The task of OUSPG is to create a socio-technical security model for the purpose of identifying the information security threats involved in such collaboration and the appropriate responses to those threats. The chosen modelling approach is to adapt and apply the PROTOS-MATINE methodology, characterized by its ability to synthesize data collected from diverse sources and has previously proven effective when used to evaluate targets such as antivirus software and voice-over-IP systems. We are thus pursuing two objectives in parallel: to develop a new version of the methodology, and to use it to analyse the EMCANP case. We made progress toward both objectives in 2012 and expect publication-ready results in 2013.

The Electric Vehicle ecosystem studies security and privacy issues involved in future infrastructures for charging and testing electric vehicles in northern environments, as well as associated end-user services. The work aims to bring our research themes of Quality, Complexity and Awareness into new application areas, where connectivity to a  global infrastructure is applied to traditionally isolated systems.

Hash function security research is a part of cryptography, where hash functions and their properties are studied. Hash functions play a major role in many modern communication protocols, and are at the moment a very hot topic since NIST (National Institute for Standards in Technology) proposed a competition for a new and more secure hash function standard, SHA-3. This competition vets the one best algorithm from 64 different propositions. At OUSPG, the study of hash functions has led to some practical results, but mostly concerning theoretical advances in the field of hash function security. The results gained form this research and the cryptographic expertise are then applied to the practical information security work in OUSPG.


Exploitation of Results

The results of our research were applied to real-world problems in many projects, often in collaboration with industrial and other partners. Efficient exploitation of results is one of the core objectives of the national Tivit ICT SHOK projects like Cloud Software, D2I and IoT; in these projects we work in close collaboration with companies throughout the projects.

The Intelligent Systems Group utilizes a robotics laboratory and a pressure-sensitive floor (EMFi material) installed in our laboratory as part of a smart living room. Other equipment includes a home theatre, two degree-of-freedom active cameras, four mobile robots and one manipulator, a WLAN network, and various mobile devices. Our aim is to gradually build a versatile infrastructure that offers various generic services for pervasive applications. Naturally, this kind of environment enables realistic experiments that lead to a better understanding of such applications.

We are also building an immersive space into our laboratory. This affordable cave is built from six white screens covering 360 degrees and six projectors, one for each white screen. We are currently adapting the realXtend 3D virtual environment platform for this setting.

The generic system for the semi-automatic maintenance of statistical prediction models is now in use for product planning at the Ruukki steel plate mill. The system enables us to keep up-to-date those models which are important for ensuring that the products fulfil the customer needs. As new products and production methods are continuously developed, and novel process settings are taken into use, data from previously unseen conditions occur, and thus there is a constant need to update the prediction models. Another major reason that causes a need for model updates is process drift. The system developed generates semi-automatically new model versions when the newest data makes it possible to improve the prediction accuracy. Each model version is a dynamic link library (DLL) that can be directly plugged in to the on-line applications that actually utilize the models. The system will give economic benefits because improved prediction accuracy in the product planning decreases production costs, and the costs needed to maintain good prediction accuracy will now be small.

During the reporting year, the group continued utilizing outdoor robotic systems. Development and utilization of Mörri, a multipurpose, high performance robot platform continued. More focus was put on perception in natural conditions, representation of detections, knowledge, and an environment model of the operating environment. The software architecture further developed the earlier work on Property Service Architecture, and the Marker concept as general purpose representation was further developed.


Future Goals

We will continue to strengthen our long term research and researcher training. We will also continuously seek opportunities for the exploitation of our research results by collaborating with partners from industry and other research institutions on national and international research programs and projects. The University of Oulu is a founding member of euRobotics. Juha Röning is a member of the Board of Directors of euRobotics.

We will strengthen our international research co-operation. Over recent years, the group has created collaboration projects with Japanese researchers. Two doctoral students have funding for one-year visits to Japanese universities.  We will continue the collaboration with Sonic Studio from Sweden and the Shanghai Jiao-Tong University from China. With the University of Tianjin in China, we have a joint project in which methods and a system will be developed for vision-based navigation of Autonomous Ground Vehicles, which utilize an omni-directional camera system as the vision sensor. The aim is to provide a robust platform that can be utilized in both indoor and outdoor AGV (Autonomous Ground Vehicles) applications. This co-operation will continue.

In the USA, we will continue to co-operate with the Human-Computer Interaction Institute in Carnegie Mellon University with Assistant Professor Anind K. Dey. The research is on human modelling in the area of human-machine interaction. We continue and strengthen US-Finland co-operation through an NSF grants. With Virginia Polytechnic Institute and State University we have our collaborative research project on Techno Economic Models for Collaborative Access Network Provisioning. In 2013, we start another US-Finland project: NSF EAGER: Global RF Spectrum Opportunity Assessment.

Shorter research visits to European partners in EU-funded projects are also planned.

In 2013, the aim is to utilize more widely the know-how from sensor technology and data mining. New application areas will be studied, including rehabilitation, exercise motivation and energy efficiency in households, and the benefits of our expertise will be highlighted to actors in the areas.

In human-environment interaction and sensor networks, our research will continue mainly in Tekes and ICT SHOK projects. Our main goals are to develop analysis methods for sensor network data and to develop applications utilizing physical user interfaces. Research on novel software architectures, reasoning and knowledge representations will continue as well. Field trials in realistic settings, and close collaboration with research groups (national and international) and companies will be emphasized.



professors, doctors


doctoral students






person years



External Funding



Academy of Finland

277 000

Ministry of Education and Culture

110 000


996 000

other public

50 000

domestic private

111 000


145 000


1 689 000


Doctoral Theses

Davidyuk O (2012) Automated and interactive composition of ubiquitous applications. Acta Universitatis Ouluensis. Technica C 420.

Halunen K (2012) Hash function security: Cryptanalysis of the Very Smooth Hash and multicollisions in generalised iterated hash functions. Acta Universitatis Ouluensis. Technica C 433.

Suutala J (2012) Learning discriminative models from structured multi-sensor data for human context recognition. Acta Universitatis Ouluensis. Technica C 421.


Selected Publications

Alasalmi T, Suutala J, Röning J (2012) Real-time non-intrusive appliance load monitor feedback system for single-point per appliance electricity usage. Proc. International Conference on Smart Grids and Green IT Systems (SMARTGREENS 2012), SciTePress, 203-208.

Belo F, Birk A, Brunskill C, Kirchner F, Lappas V, Remy C, Roccella S, Rossi C, Tikanmäki, A & Visentin G. (2012), The ESA lunar robotics challenge: Simulating operations at the lunar south pole. Journal of Field Robotics. 29(4): 601–626.

Feng W, Röning J, Kannala J, Zong X & Zhang B (2012) A general model and calibration method for spherical stereoscopic vision. SPIE Proc. Intelligent Robots and Computer Vision XXIX: Algorithms and Techniques, 8301: 830107.

Gilman E & Riekki J (2012) Smart spaces: a metacognitive approach. Grid and Pervasive Computing Workshops, LNCS 7096: 148-155, Springer.

Hosio S, Kostakos V, Kukka H, Jurmu M, Riekki J & Ojala T (2012) From school food to skate parks in a few clicks: using public displays to bootstrap civic engagement of the young. Pervasive Computing: 10th International Conference, Pervasive 2012, LNCS 7319: 425-442, Springer.

Juutilainen I, Tamminen S & Röning J (2012) A tutorial to developing statistical models for predicting disqualification probability. Computational Methods for Optimizing Manufacturing Technology, IGI Global, 368-399.

Juutilainen I, Tamminen S, Röning J (2012) Exceedance probability score: a novel measure for comparing probabilistic predictions. Journal of Statistical Theory and Practice, 6(3): 452-467.

Kauppinen M & Röning J (2012) Software-based neural network assisted movement compensation for nanoresolution piezo actuators. Proc. SPIE Intelligent Robots and Computer Vision XXIX: Algorithms and Techniques, 8301: 830102.

Koskimäki H, Huikari V, Siirtola P & Röning J (2012) Behavior modeling in industrial assembly lines using a wrist-worn inertial measurement unit. Journal of Ambient Intelligence and Humanized Computing, Online first.

Leppänen T, Närhi P, Ylioja J, Riekki J, Tobe Y, Ojala T (2012) On using continuations in wireless sensor networks. 9th International Conference on Networked Sensing Systems, 11-14.

Leppänen T, Riekki J (2012) Dynamic data processing middleware for sensor networks. Grid and Pervasive Computing Workshops, LNCS 7096: 141-147, Springer.

Leppänen T, Ylioja J, Närhi P, Räty T, Ojala T & Riekki J (2012) Holistic energy consumption monitoring in buildings with IP-based wireless sensor networks. Proc. Fourth ACM Workshop on Embedded Sensing Systems for Energy-Efficiency in Buildings (BuildSys ‘12), 195-196.

Liljedahl M, Lindberg S, Delsing K, Polojärvi M, Saloranta T & Alakärppä I (2012) Testing two tools for multimodal navigation. Advances in Human-Computer Interaction, ID: 25138410, 10 p.

Oja M & Riekki J (2012) Ubiquitous framework for creating and evaluating persuasive applications and games. Grid and Pervasive Computing Workshops, LNCS 7096: 133-140, Springer.

Parkkila T, Kääriäinen J, Tanner H & Riekki J (2012) Middle-of-life PLM solutions for reconfigurable networked mechatronic products. ARPN Journal of Systems and Software 2(5): 177-186.

Polojärvi M & Riekki J (2012) Lightweight Service-Based Software Architecture. Grid and Pervasive Computing Workshops, LNCS 7096: 172-179, Springer.

Pyykkönen M P, Riekki J, Alakärppä I, Sanchez I, Cortes M & Saukkonen S (2012) Designing tangible user interfaces for NFC phones. Advances in Human Computer Interaction, Hindawi Publishing Corporation, 575463, 12 p.

Rautiainen M, Korhonen T, Mutafungwa E, Ovaska E, Katasonov A, Evesti A, Ailisto H, Quigley A, Häkkilä J, Milic-Frayling N & Riekki J (eds.) (2012) Grid and pervasive computing workshops. LNCS 7096, Springer.

Riekki J, Cortés M, Hytönen M, Sánchez I & Korkeamäki R-L (2012) Touching nametags with NFC phones: a playful approach to learning to read. LNCS Transactions on Edutainment. Special Issue on Interactive Digital Storytelling.

Röning J & Casasent DP (eds.) (2012) Proc. SPIE Intelligent Robots and Computer Vision XXIX: Algorithms and Techniques 8301.

Siirtola P, Röning J (2012) User-independent human activity recognition using a mobile phone: offline recognition vs. real-time on device recognition. Proc. Distributed Computing and Artificial Intelligence, Springer, 151: 617-627.

Siirtola P & Röning J (2012) Recognizing human activities user-independently on smartphones based on accelerometer data. International Journal of Interactive Multimedia and Artificial Intelligence 1(5): 38-45.

Su X, Riekki J & Haverinen J (2012) Entity notation: enabling knowledge representations for resource-constrained sensors. Personal and Ubiquitous Computing 16(7): 819-834.

Zhou J, Athukorala K, Gilman E, Riekki J & Ylianttila M (2012) Cloud architecture for dynamic service composition. International Journal of Grid and High Performance Computing 4(2): 17-31.

Last updated: 15.4.2014