Infotech Oulu Annual Report 2011 - Intelligent Systems Group (ISG)

Professor Juha Röning and Professor Jukka Riekki,

Department of Computer Science and Engineering , University of Oulu

juha.roning(at), jukka.riekki(at)

Background and Mission

The main scientific objective of the basic research conducted in the Intelligent Systems Group is to generate new applicable knowledge on intelligent systems, and to generate positive societal impacts by applying this knowledge using scientifically plausible methods and state-of-the-art technology. We conduct research on spatial computing, collective intelligence, sensor networks, optimization of industrial processes, mobile robotics and cybernetics, human-computer interaction (HCI), human-robot interaction (HRI), computer networks, and security in complex information processing systems.

Our strategic research themes, whereby we aim to conduct world class basic research, are: 1) Safety and security on all levels of intelligent systems, 2) Data mining with a special focus on optimization of industrial processes and well-being, 3) Human-environment interaction with a special focus on novel physical user interfaces, robot-environment and human-robot interaction, 4) Mobile robotics and cybernetics with a special focus on spatial computing, adaptation, and field robotics, and 5) Sensor networks with a special focus on understanding behaviour and patterns in our everyday environment.

We have conducted basic research related to these research themes for over ten years. Our team consists of two professors, five postdoctoral researchers and 21 doctoral students. The annual external funding of the group is more than two million Euros in addition to our basic university funding. There have been 15 completed doctoral degrees from the group. From the research of the group, seven spin-off companies have been established so far: Codenomicon, Clarified Networks, Hearth Signal, Nose Laboratory, Nelilab, Atomia and Probot.

We co-operate with many international and domestic partners. In applied research, we are active in European projects. In addition, several joint projects are funded by the Finnish Funding Agency for Technology and Innovation (Tekes) and industry. We were a research partner in the Cooperative Traffic (CoopTraffic), and Devices and Interoperability Ecosystem (DIEM) ICT SHOKs. In the Cloud Software Program ICT SHOK we are responsible for the cloud security theme. The CoopTraffic and DIEM projects were completed in 2011, and we are participating to the preparations of a new ICT SHOKs that will start during 2012.

In the II City project, we collaborated with the University of Lapland and Sonic Studio from Piteå, Sweden, and in the Pervasive Service Computing project with Shanghai Jiao Tong University from China. These projects were funded by the European Union (Interreg IVA North) and the Academy of Finland (the MOTIVE program). Also we are collaborating with the Tokyo University of Agriculture and Technology in the Interactive Context-aware System for Energy Efficient Living project, funded by the Academy of Finland.

We are active in the scientific community. For example, Prof. Juha Röning acted as a member of the SAFECode International Board of Advisors, 2011 and as a chief judge in the European Land Robot Trial (Elrob 2011) in Leuven in May. Elrob is the biggest outdoor robot event in Europe, and participants are research facilities and companies that represent the state-of-the-art in Europe in this research area. Prof. Jukka Riekki acted as Conference Chair of the 6th Grid and Pervasive Computing Conference that was organized on May 11-13 2011 at the University of Oulu. The team organizing the conference consisted mainly of researchers from ISG and MediaTeam Oulu. Prof. Jukka Riekki visited several universities in Wuhan, Shanghai, Hefei, and Peking during his one-month visit to China in December 2010 – January 2011. Dr. Susanna Pirttikangas visited Tsinghua University for two months to start collaboration with the Cisco Joint Laboratory for Green Technology and studied the large data sets the Beijing taxi transport system provides. Several doctoral students made research visits during 2011 as well. Teemu Leppänen made a research visit to the Tokyo Denki University in the beginning of 2011. Several members of the group co-chaired workshops and were also on the committees of international conferences.


Scientific Progress

Research on Prototyping: from a Smart Environment towards Remote Distributed Intelligence

Verification of the developed methods and models in prototypes is an important part of our research. To support this activity, we develop software and hardware architectures for smart environments. In addition to verification, prototypes speed up the commercialization of the research results. In prototyping, we have set and tackled the following objectives:

Developing Hardware and Software for Prototypes and Commercial Devices: The goal is to make designing of embedded systems faster and easier, while preserving the commercial applicability of the resulting devices. This has been approached via the Embedded Object Concept (EOC). EOC is a concept that utilizes common object-oriented methods used in software by applying them to combined Lego-like software-hardware entities. This approach enables people without a comprehensive knowledge in electronics design to create new embedded systems. For experts, it shortens the design time of new embedded systems. The conceptual idea of embedded objects has been successfully implemented with the Atomi II framework, which provides the so called Atomi modules.

The EOC research has progressed by making the concept easier to grasp for new users. This has happened by developing the documentation, tutorials and example cases that are to be presented in workshops and lectures for local companies and research facilities.

Research on Nanoresolution Tools for Interdisciplinary Applications: The project “Nanoresolution tools for interdisciplinary applications” (NRT) develops novel research, manipulation, and manufacturing methods for micro- and nanotechnology components and instruments. This means ever smaller objects which are placed on surfaces not measurable with existing tools. A network of contacts and collaborators is used to select the objectives where commercial instruments are not yet available. One basis of the selection is a study of the commercial potential and value. Commercial potential affects the cost and selection of the components to be used thereby enabling the commercialization process. In order to improve the regional effect of the project, market studies are carried out.

The tools developed in NRT are based on a Scalable Modular Control Architecture (SMCA) that is being developed in the project, and the previously mentioned embedded objects. SMCA is a generalized modular architecture for both the device hardware and the control software on a PC. The architecture is extensible, scalable and portable, and it enables reuse of modules. It is a hierarchically layered hybrid architecture for implementing research equipment. SMCA enables swift changing of actuators, sensors and tools with minimal effort, thus being an ideal frame for various applications.

Implementation of SMCA is included in the framework within the project. The framework aims to improve the quality of the developed tools and enable fast prototyping. To achieve these goals, the framework includes reusable components that perform the tasks needed in all developed measurement tools, such as data management, data visualization and configuration of devices. It also supports run-time specialization of measurement tasks to enable the use of the same hardware to perform different measurement tasks easily. Future work with SMCA will concentrate on applying the architecture in new areas, and the dissemination of the technologies for the local companies and research facilities.

Research on Mobile Robotics

The objective of our research is to develop methods for exploiting magnetic fields in localization. Magnetic field localization is based on the observation that indoor magnetic fields provide sufficient spatial variation and temporal stability to permit inference about sensing sites, given noisy measurements. One essential question is how to collect magnetic field information in an efficient way using mobile robots with localization uncertainties.

In traditional SLAM approaches that are based on laser, sonar, or machine vision techniques, sensor coverage plays an important role, enabling previously observed structures to be utilized to decrease motion model uncertainties. In magnetic field SLAM, however, sensor coverage restricted by point-wise magnetometers requires frequent visits to already measured areas. While we exploit already gathered information in localization, we lose some degree of optimality in sensing quality (or modelling quality). How can we measure this quality, and is there some way to overcome the so-called exploration/exploitation dilemma?

In our previous studies, we examined near-optimal exploration in magnetic field SLAM with a constant optimality factor. In each iteration, we used the same factor to select an action that guaranteed near-optimal sensing while minimizing localization uncertainty. A problem arose from a tendency to select actions that provided extrapolating data with large prediction variance. After a short period of time, the sensing site distribution, presented by particle discretization, was too dispersed for the algorithm to find any near-optimal action.

In 2011, we continued our studies by proposing a scalable optimality factor which still guarantees a constant level of optimality for exploration. This provides a mechanism for achieving a balance between exploration and exploitation, enabling exploitation actions in sensing regions that have smaller prediction variance. We also proposed a method for comparing different models in terms of how well they are suited for localization. This is expressed as the expected sensing space coverage required producing noisy measurements drawn from the model.

In 2011 we developed an unmanned surface vehicle (USV) as a part of a TULI project which aimed at extending CSE’s autonomous environmental modelling studies to modelling water quality parameters. (Tekes’ TULI program supports research innovations that have potential commercial values.) In this project, we developed a prototype vessel (see the figure) that was able to autonomously collect information on water quality parameters given a sensing route. The experiments showed 1) that robot vessels with minor localization instruments can perform successful modelling tasks, and 2) that spatial and spatio-temporal heterogeneity of parameters advocates adaptive methods for modelling and route planning when the objective is to build a cost-efficient (sensing time with regard to modelling accuracy) sensing system.


The USV developed in CSE. The USV can be used as an autonomous measurement instrument in aquatic environments. Top left: the USV is performing a measurement mission in the Baltic Sea, Pietarsaari, Finland.


The Evolutionary Active Materials (EAM) project, which is funded by the Academy of Finland, is a joint effort of the Computer Science and Engineering laboratory (CSE) and the Microelectronics and Materials Physics laboratories. The aim of the EAM project is to develop novel evolutionary computation (EC) based design methods for active and versatile materials and structures. The first components are being developed through a novel holistic design process utilising constantly increasing computation power, the development of multi-physics simulators, and EC techniques such as genetic algorithms. During 2011, the first optimization experiments were conducted in the EAM project. Two different piezo structures were optimized by using a combination of a genetic algorithm, and a multiphysics simulator. The optimized piezo actuator structures were a cantilever beam, and a cymbal. The cantilever beam structure (in the next figure) was successfully optimized to produce maximal deflection on a predetermined resonance frequency (4000 Hz), while simultaneously addressing the stress affecting the piezo structure. The cymbal type piezoelectric actuator structure was successfully optimized to produce maximum force and displacement. The aim of the EAM project for the year 2012 is to develop further the implementation of the optimization module to allow more efficient distribution of computational tasks, and to provide an opportunity to optimize more complex hybrid structures.


The cantilever beam structure was optimized to produce a maximal displacement at a 4000 Hz resonance frequency. The amount of displacement is indicated with the colour ranging from blue to red.


The magnetic field based localization technique were further developed and tested for different operational environments. Underground mines and other underground environments pose several challenges to positioning technologies. These challenges are related to harsh operating conditions, emergency situations, application requirements such as portability, and the overall feasibility of the technology. The key motivations for applying a positioning system in underground environments are the safety of people, and logistics. Emergency situations pose the most serious challenge to positioning techniques as the whole technological infrastructure, such as the radio communication network and the electric power system, might be harmfully affected by an emergency situation such as fire. A geomagnetic field based positioning technique was proposed for underground mining environments. The proposed technique utilizes the anomalies of the geomagnetic field present in underground environments. The main source of the magnetic anomalies is the complex distribution of metallic minerals such as iron ore. The distribution of metallic minerals produces unique spatial magnetic patterns in underground mines which can be utilized for positioning. Preliminary results were published using the data collected from Pyhäsalmi (in the next figure), which is an underground copper and zinc mine located in central Finland. The data used in the experiments were collected from tunnels located approximately 1400 meters below the surface. The obtained results suggest that the proposed positioning technique can provide pose estimates with an accuracy of 1.5 meters inside underground tunnels. The proposed technique can potentially provide a robust, and a cost efficient positioning solution for underground environments with minor infrastructure requirements. The instrument used to collect the magnetic field data for the positioning experiments is shown in second figure.


The aeromagnetic map [µT] of the Pyhäsalmi mining area superimposed with an aerial photo. The aeromagnetic data is collected from an altitude of 150 m. The magnetic field of the mining area has a unique spatial pattern which stands out from the background magnetic field. This is a typical phenomenon in areas where magnetic minerals (e.g. iron ore) can be found.


The magnetometer array used to collect the magnetic field data. The magnetic field maps used in the experiments were generated using the measurements performed by one horizontal row of the sensor array. The photo has been taken on level 2 of the Pyhäsalmi mine, 1440 m below the surface.


Research on Human-Environment Interaction

Our current work on human-environment interaction focuses on physical user interfaces and human-robot interaction. In physical user interfaces, mobile terminals are used as physical objects rather than as traditional I/O devices. Our current studies concentrate on touch-based interaction: users interact with the local environment by touching objects with their mobile terminals. The touch-based user interfaces use NFC technology (i.e. RFID technology for mobile phones): an act of touching brings an NFC reader near an NFC tag, and hence the data in the tag is read and delivered to the system. The objects that can be touched are advertised in the environment by NFC icons, graphical icons resembling the icons of the graphical user interfaces of computers and other user terminals.

In 2011, we completed our fifth version of a visual language for advertising NFC tags. The next figure presents an example how a multimedia service at a tourist attraction might be advertised. When a user touches the “Play” symbol with her/his phone, a video is shown on a nearby display. When the stop icon is touched, the video is stopped. We participated also in defining a Finnish standard for advertising NFC tags; that standard will be published in 2012.


A multimedia player service for a tourist attraction.


We continued research on multimodal interaction for interactive spaces in the DIEM ICT SHOK together with researchers from VTT, the University of Tampere, and the Tampere University of Technology. Our work focused on interaction using NFC technology. During 2011, we built multimodal user interfaces to control and acquire power consumption data from different home appliances. In the prototype, a ThereGate platform from ThereGate Corporation is used as a gateway that relays commands given with an NFC phone to other system components. Commands are stored on NFC tags and graphical icons advertising the commands are placed on top of the tags. When a user touches a tag, the phone reads a command from the touched tag and sends the command to the gateway. A user can touch a tag to turn on and off home appliances (e.g. a lamp, a TV) and configure them; to get information about appliances’ power consumption; and to subscribe alarms to the phone. In the last case, an alarm is sent when an event occurs at home (e.g. a window is opened). All this software is freely available at the project webpage.

In the DIEM project we also created a working prototype of a general purpose data visualization framework. The purpose of this framework is to simplify the creation of systems that connect user-specific data from data sources to visualization applications or other applications that modify their behaviour based on data input. We developed a prototype visualization application on top of the framework. The application visualizes each users’ data as a fish; the values given by a user determine the appearance and behaviour of the corresponding fish. The application is shown in the figure below. We worked with Polar Electro to show data from wrist-worn activity meters in this aquarium application. Furthermore, our DIEM project partners from the Department of Information Processing Science visualized data from two of their applications with the aquarium: a micropayment system visualized the number of transactions for each different item as a fish while a feedback system connected the feedback score of each question to a fish.


A prototype visualization application.


We continued creating NFC based applications for school environments, in collaboration with the Department of Education at the University of Oulu. We developed an application to teach English vocabulary to 10-to-12-year-old pupils. With this application, pupils can write down their own stories and associate words to them. Each word is written on a separate card that is equipped with an NFC tag. Other pupils read the story and select those words that belong to the story by touching the corresponding cards with their phones; the phone then informs the pupils whether the selection was correct. This application was tested in a local school during a one week period. Furthermore, we continued developing small NFC based applications for a local kindergarten. NFC tags were attached to different cards presenting numbers, letters, colours, and family members. When a child touches a card the phone says out loud the name of the colour, the number, the letter or the family member and shows that word on the phone screen as well.

In the II City project, we developed two novel applications in collaboration with Sonic Studio from Piteå, Sweden, and the University of Lapland. PING! is a new kind of mobile navigation aid utilizing fuzzy routing and audio instead of turn-by-turn routing and visual maps. Fuzzy routing guides a user by telling in what direction and how far the target is - instead of providing a detailed route and turning instructions. Giving the user the freedom and possibility of choosing one’s own route to the target has proven to work well within a city environment. Users can find targets by sweeping the mobile device around them (left-to-right and back). Auditory and visual feedback tell the user the targets in these directions; how many and how far they are. After the user has selected a target, graphical direction and distance indicators and sound guide the user towards this goal.

Echo Range is an audio-based mobile game designed to give its players a new kind of experience of a city environment. It is a game based on the old battleship idea: players hunt and shoot other players’ vessels. Each player is equipped with a phone and a stereo headset. The players are located with GPS and the orientation of the phone (detected with the onboard compass) determines the direction of the vessel. Submarines hunt corvettes and corvettes hunt submarines. Submarines can shoot torpedoes and corvettes can drop depth charges. The relation of the players in the real environment determines the effect of these weapons when used. Players try to locate each other by listening to the soundscape the game creates; no other information (e.g. graphical) is given about the other players’ locations. The vessel engines produce sounds and so do the weapons, the torpedoes and the depth charges. Submarine players listen passively to sounds with a hydrophone; they can hear corvettes inside the hydrophone’s detection cone. To observe a larger area the players need to sweep the phone, that is, to turn the hydrophone in different directions. Corvettes on the other hand locate submarines with a sonar. Corvette players can send ping sounds in every direction around a corvette and listen to echoes. The figure below shows the user interfaces for submarines and corvettes. Both PING! and Echo Range applications were tested at downtown Oulu in autumn 2011.


Echo Range user interfaces for submarines (left) and corvettes (right).


In the Pervasive Service Computing project, we continued the development of service-oriented infrastructure unobtrusively supporting people in their daily activities. This work involves collaboration with MediaTeam Oulu and Shanghai Jiaotong University. Several prototypes were developed to demonstrate the capabilities of dynamic service composition and coordination. We developed a RESTful middleware, a perception framework that simplifies and accelerates the development of pervasive systems. This framework defines the overall infrastructure for pervasive services that utilize rules and context. Moreover, we explored cloud computing to enable dynamic pervasive service composition. A cloud based on-demand service composition prototype was implemented with the Windows Azure Cloud platform. In addition, we studied distributed reasoning with two prototypes; with an application composition and control prototype, and with a prototype allowing mobile users to share information about common interests.

Research on Sensor Networks

The research on sensor networks is targeted at understanding behaviour and patterns in our everyday environment. We are studying sensor data processing platforms, sensor data processing algorithms, lightweight knowledge representations suitable for sensor networks, and reasoning. In 2011, we continued work in the Cooperative Traffic ICT SHOK. This research was conducted in the Sensor Data Fusion and Applications (SDFA) project, together with the University of Jyväskylä, the Tampere University of Technology, the Information Processing Department of the University of Oulu, and VTT. We developed a pattern recognition system for detecting road conditions from accelerometer and GPS readings.

In mid 2011, we joined the Urban Flows and Networks project, lead by MediaTeam. Our focus is on modelling transport and pedestrian flows using measurements from Oulu’s Urban Pervasive Infrastructure, which consists of a Bluetooth network, the PanOulu WLAN network and loop detectors associated with traffic lights at intersections. We recently completed our first study relating traffic volume measurements from the loop counters and the Bluetooth network. Ongoing work studies movement patterns in WLAN and Bluetooth networks.

In the UBI program, we continued the development of a platform on collecting, processing and storing sensor data from a heterogeneous wireless sensor network. The first prototype for measuring the energy consumption in households, using wireless sensor nodes over IEEE 802.15.4 networks with 6LoWPAN protocol stack, was completed in 2010. In total, 11 households in Oulu participated in the first pilot testing. The test users found the system useful and feasible for home use, although interest towards energy saving seems to wear out quickly without real economic incentive. We extended this prototype to also offer to users sensor-based data about local weather, indoor air quality, and tracing of people’s movement. This data can be used in providing context-aware services and supporting people in their everyday activities. This work was conducted in co-operation with MediaTeam and the Tokyo Denki University, Japan. We studied the database developed at Tokyo Denki University, TomuDB, that features simple interfaces for location-based data queries in WSN. We also studied composing dynamically sensor-based services and exposing them via Internet. This work targets dynamic configuration of a sensor network, easy context-switching in the sensor network, dynamically composing sensor-based services, and distributing computational load in the network.

In the RealUbi project, a collaboration project with Tsinghua University in China was started with a view to studying the trajectories of 28000 taxis moving around Beijing. From the data, encounter times between different vehicles, peak times at given places and other statistics could be found. This kind of information helps in city and traffic planning, finding the attractive areas within cities, and in taxi dispatching, for example.


Studying a prototype at Tsinghua University. This device is assembled on the roof of a vehicle, where it collects environmental data. (Photograph by Judy Zhou.)


In the Pervasive Service Computing project, we developed a novel lightweight data and knowledge representation, Entity Notation (EN), to build knowledge-based systems for pervasive environments. Such a representation facilitates connecting sensors, actuators and other mobile devices to a Semantic Web and inference engines, and supports intelligence in pervasive environments. Several prototypes have been developed to demonstrate this representation and evaluate it. The next figure M shows the role of EN in connecting a sensor to an inference engine.


Resource-constrained sensors and an inference engine communicate by exchanging EN packets.


Research on Human-Robot Interaction

Two on-going projects, the Academy funded AFHRI (Affective-Human-Robot interaction) and the European Regional Development Fund (ERDF) funded Minotaurus combine the theoretical and practical aspects of creating service robotics of the future. Both projects are in co-operation with the Center for Machine Vision Research (CMV). Human-Machine communication and the robot’s operation in real world conditions are major topics. Key aspects are putting research results into practice, and integrating them as a part of the operation of the robot. The main challenges are representation of information and adapting a set of algorithms to the current state in the surrounding environment. Human friendliness of the outward appearance of the robot is driving the design of the robot.

Interaction with a machine has been studied from various perspectives. The combination of machine vision, speech recognition and synthesizing, touch and touchless interactions, along with the robot’s operation in the environment requires a software platform that processes, distributes and stores information efficiently. Real-time Linux based operating system services, along with general purpose representation for information (called Markers) have been developed to support integration of the algorithms. Marker- representation is used for representing the output of sensor processing algorithms, for representing a model of environment and obstacles around the robot, and information related to the robot’s tasks.

Speech and language processing in Human-Robot Interaction: In the Minotaurus project, speech and natural language processing have been studied to be able build a natural interface between a robot and human. At the first stage, the research has been concentrating on automatic speech recognition (ASR) based on a CMU Sphinx engine, which is used to recognize greetings, commands, and questions. Furthermore, a simple language understanding model, based on keyword spotting, is implemented to be able to recognize the meaning of a speech utterance. We have used a pre-trained English acoustics model based on hidden Markov models (HMMs), as well as trained new HMMs for the Finnish language in conjunction with appropriate dictionaries and language models for robot communication application. Finally, the speech understanding module is connected to a talking mouth, which is able to synthesis speech audio and lip movements. Responses are based on the robot’s internal knowledge as well as external knowledge from the web.

Different pre-processing methods are applied to realize continuous ASR. To detect speech and non-speech frames, the signal power in each frame is computed and accumulated in a histogram as a feature. A two-state state machine (i.e., silence or speech) in conjunction with adaptive thresholds is used to detect the transitions over multiple frames. Using these techniques, we are able to detect and recognize speech from a distance. Moreover, speaker orientation is estimated using multiple microphones and phase correlation analysis to be able to detect a current active speaker and her orientation. This could be used to eliminate background noise, as well. In the future, work will be concentrating on medium and large size vocabulary speech recognition and language processing based on statistical machine learning to be able to implement more natural communication techniques between a human and a robot.

The Human-Robot Interaction system contains several “contact points”, where interaction with the system occurs. These contact points are large displays, equipped with multiple cameras, a microphone system and software for producing dialog and virtual face animation that matches the synthesized speech. The mobile robot is one contact point in the system.

The robot is based on a Segway RMP200 platform, and equipped with several sensors to receive information from the surrounding environment and humans. Two Kinect sensors are used as a 3D vision camera, one looking towards the floor and one for forward. Multiple 2D scanning lasers are additionally used for close distance obstacle avoidance, and for refining the estimation of the robot’s movement. Further, the robot is equipped with several microphones to reason the source direction for audio and for speech recognition.


Test robot for Human-Robot Interaction.


The robot uses a graph based navigation map for finding a raw route for driving around the laboratory, and an RRT (Rapidly-exploring Random Tree) for finding a free path from the current location to the next route position. For raw location, the robot uses a Kinect sensor and detection of visual clues like room signs and room numbers. Also a laser scanner profile is used for matching with prior knowledge of the corridor shape profile. Later, technology using magnetic field quality for localization will be merged to the robot.

Research on Data Mining

The future of living and housing: A new project titled “An Interactive Context-aware System for Energy Efficient Living” (INCA) started in 2011. INCA is funded by the Academy of Finland and the project goal is to develop an interactive and context-aware feedback and control system for rationalizing energy efficient living. The system itself collects personalized data from the environment and offers information about consumption habits for the individuals. With the help of the information, it is possible to advise people on more energy efficient living and to give recommendations for avoiding unnecessary energy consumption. The research in the INCA project is performed jointly with Associate Professor Kaori Fujinami from the Tokyo University of Agriculture and Technology (TUAT). During 2011, our research topics have been concentrating on low-level energy consumption measurement systems. We are developing easy-to-install, low-cost electricity and water meters, which could be installed externally without any permanent changes in the house infrastructure. Fujinami’s group at TUAT are developing techniques for detecting activities of wasteful energy usage. In addition, they are studying persuasive control and feedback technologies.

Non-Intrusive Appliance Load Monitoring: It is possible to estimate per appliance energy consumption using a single sensor installed at the main electrical panel or using the power companies’ smart meter if available. This can be done by noticing the changes in power consumption when appliances turn on or off. These changes are unique to each appliance so it is possible to determine which appliance caused the change. This way we can estimate the operating time of each appliance and therefore approximate the amount of energy used by the appliance. In the INCA project, we have used sensors that are plugged into electrical sockets to develop the load disaggregation algorithm. This kind of sensor can be used to demonstrate the possibilities of this approach, but it is not possible to measure a whole household with this sensor. The same algorithm could later be used with data from an apartments´ smart meter so a whole household could be measured and total energy consumption could be divided between different appliances or appliance groups. This would allow giving users more accurate feedback on electricity consumption habits to motivate them in saving energy. Threshold-based edge detection is applied to recognition of on-off events from power and reactive power measurements. K-nearest neighbours and support vector machine classifiers are applied to discriminate between different devices. The following figure illustrates the device on-off event detection, as well as visualizes typical power characteristics of different devices in the feature space.


A water flow estimation system based on the mechanical vibration of the water pipe and statistical learning methods.


A Water Consumption Meter based on Mechanical Vibration Sound and Machine Learning techniques: In this research topic, we are developing an easy-to-install, low-cost water consumption meter, which could be installed on an existing house without any permanent changes in infrastructure. The water flow is estimated from the mechanical vibration of the water pipe. Contact microphone is used to be able to eliminate background noise and environmental sounds. The water flow estimation is performed using machine learning algorithms based on statistical regression. A supervised training and calibration set is collected using a commercial flow meter. After a short calibration period, our meter can estimate water flow and consumption only using the mechanical vibration of the water pipe, where different frequency characteristics of water flow sound, together with the K-nearest neighbour and Bayesian kernel regression models are applied. In addition, an embedded system is developed to process incoming data. In the future, the consumption meter can be applied to monitor the water usage of the household. The figure below presents sound-based water measurement systems and the water flow rate estimation.

Detection of appliances on-off events from single point measurements. Furthermore, typical power “finger prints” in the feature space to be used to discriminate between different devices in the load monitoring system are visualized.


New methods for activation of young men: The multidisciplinary MOPO project combines traditional health promotion, modern technology and the measurement of physical activity. The aim of the study is to provide knowledge on physical activity, the relationship towards physical activity, information behaviour, fitness, health, nutrition, life habits and cultures of young men. A novel wellness coaching service for preventing marginalization and promoting physical activity and health in young men is developed in the project. The information obtained in the study can be used to promote the wellness of young adults in education, study and decision-making of the professionals of social and health services.

Altogether about 5000 conscription-aged men (four call-up age classes) have been/will be invited to the MOPO study. The condition, wellbeing, health, relationship towards physical activity, information behaviour and the use of media and technology will be clarified during the years 2009-2013 with questionnaires, measurements and interviews. The contents of the service network will be developed together with the city of Oulu and conscription-aged men. 

The data mining group has been responsible for the development of the gamified ICT platform which forms the basis for the different applications. It integrates diverse types of data from various sources. Activity data collected using a number of different devices has to be combined with static and dynamic information on the in-game universe and events of the persuasive video game. Additionally, the platform needs to support message passing from administrators to end users (for health information and various announcements), and also between end users (for in-game conversations). To satisfy these requirements, a client-server architecture was designed and implemented, consisting of a central database and a set of tools and interfaces for producing, importing, exporting and visualizing data in accordance with the information needs of each of the user groups of the platform.

Contrary to previous game solutions, subjects do not exercise while playing the game but their daily activity and exercise is monitored and rewarded in the game. Data collected through the ICT application and participant observation will be analyzed when evaluating the effectiveness of gamification in encouraging participants towards an active lifestyle.


The game: the player earns points based on his physical activity. After collecting enough points he proceeds to the next level.


The first version of the developed service combined with different activity monitors was tested during the 3 months pilot intervention in autumn 2011. In the tests, a pilot version of a computer game, called Space Pioneer, was used. Instead of motivating user directly towards more active lifestyle the user is motivated to play the computer game. However, daily activity measures are used as game input thus the playing of the game and success in it correspond to the daily activity levels. The daily activity measures were collected using pedometer, Polar Active monitor or mobile phone application. The mobile phone application was developed by ISG in the project for Symbian^3 operating system. The application used the accelerometers of the phone to recognize daily activities (walking, running, biking, riding a car and idling) as well as the daily step counts.


Space Pioneer game by ISG.


In the pilot answers were sought for the following questions:

  • Which sensors are preferred by young people?
  • Did the game, and success in it, motivate to exercise?
  • What was the overall opinion of the ICT platform?
  • Did the health and exercise messages given through the platform motivate to exercise?
  • Did the sensors alone motivate to exercise?

For several years now, one of the interests of ISG in data mining research has been the study of ethical issues associated with data mining. These include the well-established issue of protecting the privacy of data subjects, and also a number of previously overlooked issues such as fairness and trust. The ethics research of ISG also emphasizes the potential of data mining as an instrument for pursuing things of value rather than just a threat to values that needs to be controlled. All these perspectives are combined in the MOPO projects: the outcome of the project, a persuasive video game, may help the study subjects adopt healthier lifestyles, but its development involves potential ethical problems that must be actively identified and worked against. An ethical analysis of the technical work in the project is carried out and will be published alongside the work itself.

The final service will be developed based on the experiences of the pilot. The intervention will start in the autumn 2012. Operators of the study are the Oulu Deaconess Institute Department of Sports and Exercise Medicine, the University of Oulu, the City of Oulu, the Virpiniemi Sports Institute, the Finnish Defence Forces and wellness technology companies from Northern Finland. The project website can be found from

Data mining methods for steel industry applications: The Intelligent Systems Group has long experience in advancing data mining methodologies in the steel industry, and is a member of the Centre for Advanced Steels Research - CASR, which is one of the interdisciplinary umbrella organizations of the University of Oulu. In 2011, advanced data analysis solutions for the needs of the steel industry were developed in two active projects.

In the PISKET project (Improving Pass Scheduling Calculation Taking into Account Flatness), prediction models were developed to predict the rolling temperatures and loads during the plate rolling process. The statistical prediction of rolling loads and temperatures is demanding and the application of advanced data analysis methods is needed. In 2011, semi-supervised learning methods were developed for predicting the rolling temperatures, and non-linear regression models were developed for predicting the rolling force.

The measurements of rolling temperature contain much missing data due to the hot and moist circumstances that occur near the pyrometers. By employing semi-supervised learning, the information contained in the observations with missing response measurements were utilized to improve the learning process by employing the COREG-algorithm. As a result, a slight increase in the prediction accuracy was observed. Another important achievement in temperature modelling has been the success in the selection of a feature set that describes the deformation history of the earlier pass schedule, but is invariant to the pass number.

The developed prediction method for the rolling force outperformed completely the reference model used by the plate mill: The average prediction error was about 260 tons with the new model, and about 520 tons with the reference model. The model developed has been tested on-line in the Ruukki steel plate mill and it will soon replace the earlier models. The aim is that the quality of steel plates can be improved by optimizing the rolling pass schedule using the models developed.


A variable importance plot on the variables most valuable in the prediction of rolling temperatures.


Production efficiency with probability predictions: In the prob2E-project (Probability Predictions to Production Efficiency) the aim is to develop further a probability prediction approach to develop and utilize statistical models and validate the benefits that industry can achieve by employing distributional predictions instead of point predictions. The project is funded by Tekes, Ruukki, Ovako, and Valio.

In 2011, ISG developed two generally applicable algorithms for applying disqualification probability prediction models in the optimization, planning and control of production processes. Also, an approach to formulate, fit and validate prediction models for probability prediction was published. The proposed probability prediction procedure focuses on accurate prediction of the dependence of deviation and tail distribution shape from input variables.

A 3D-scatterplot on the predicted disqualification probability as a function of two control variables.



The benefits are measured by applying the developed probability prediction procedure into three different manufacturing industry processes (cheese manufacturing, steel strip rolling and the tempering of steel bars) and comparing the savings with the case in which the prevailing modelling approach is employed in the same process optimization task. The main hypothesis is that decision making that is based on predicted disqualification risk decreases disqualifications, and decreases variation in the product quality characteristics. The results obtained confirm that decision making that is based on predicted disqualification risk gives very significant benefits in one of the applications while in the two another applications the benefits are of less importance, or unclear. Currently, the on-line testing phase of the developed algorithms is on-going both in Ovako and Ruukki.

The predicted and observed densities for homogenous groups of the cheese manufacturing test data set.


The predicted and observed deviations in the steel bar quenching test data set.


The density functions of the predicted disqualification probability calculated from the composition and planned production method for products that are finally qualified or disqualified in the qualification test.


Cognitive load study: Eija Haapalainen returned from a two-year research visit (Fulbright - Technology Industries of Finland Grant) at Carnegie Mellon University where she worked on a study that aims to assess the cognitive load of a person based on psycho-physiological measurements. Task interruption, divided attention and multitasking cause split attention that increases our cognitive demands. This is a problem especially while performing tasks during which momentary meandering can have serious consequences (e.g. driving, manoeuvring heavy machinery, controlling air traffic, walking in traffic, performing surgery, etc.). The ability to measure the real-time changes in the cognitive load of individuals would provide a key tool to help mediate our attention resources when performing cognitive intensive tasks. In addition to the above critical activities, the ability to measure cognitive load in real-time would also benefit our daily life, where it could be used for example as an aid to increase productivity and efficiency in a professional environment (office work, learning) and to monitor leisure activities (i.e. sports, meditation and gaming).

In the study, a large number of different biosignals are examined to find the best measure or set of measures for assessing cognitive load. The signals being measured include pupil size, heart rate, skin temperature and skin conductance as well as electrical brain activity. In the first phase of the study these signals were recorded while study participants solved elementary cognitive tasks to learn how changes in cognitive load are reflected on the psycho-physiological signals. Future goals of the study include implementing the developed methods in real world applications.


Sensor devices on a study participant solving computer-based tasks.


Research on Software Security

Within the Intelligent Systems Group, the Oulu University Secure Programming Group (OUSPG) has continued research on security and safety in intelligent systems. Security and safety challenges in intelligent systems are threefold: increasing complexity leads to unforeseeable failure modes, quality is not the priority and awareness is lacking. We have approached the challenges from these three directions in our research.

Complexity - Model Inference and Pattern Recognition: We work under the premises of unmanageable growth in software and system complexity and emergent behaviour (unanticipated, not designed) having a major role in any modern non-trivial system. We have worked on natural science approaches to understanding artificial information processing systems. We have developed and applied model inference and pattern recognition to both content and causality of signalling between different parts of systems.

Quality - Building Security in: Software quality problems, wide impact vulnerabilities, phishing, botnets, criminal enterprise have proven that software and system security is not just an add-on despite the past focus of the security industry. Instead, security, trust, dependability and privacy have to be considered over the whole life-cycle of the system and software development, from requirements all the way to operations and maintenance. This is furthermore emphasized by the fact that large intelligent systems are emergent and do not follow a traditional development life-cycle. Building security in not only makes us safer and secure, but also improves overall system quality and development efficiency. Security and safety are transformed from inhibitors to enablers. We have developed and applied black-box testing methods to set quantitative robustness criteria. International recognition of the Secure Development Life Cycle has provided us with a way to map our research on different security aspects.

Awareness - Vulnerability Life Cycle: Intelligent systems are born with security flaws and vulnerabilities, new ones are introduced, old ones are eliminated. Any deployment of system components comes in generations that have different sets of vulnerabilities. Technical, social, political and economic factors all affect this process. We have developed and applied processes for handling the vulnerability life-cycle. This work has been adopted in critical infrastructure protection. Awareness of vulnerabilities, and the processes to handle them all increase the survivability of emergent intelligent systems for developers, users and society.

These research goals are reached through a number of research activities.

Secure Software Development Lifecycle in Cloud Computing as part of the Cloud Software project - we approach all three goals by researching practical ways of building security into complex Cloud Computing services, from the design phase to actual operational use.

Situational awareness of modern information networks and applications in the MOVERTI project where we approach information systems from all three viewpoints, and have developed methods for combining measurements from network traffic, vulnerability feeds and scans, and socio-economic networks to gain situational awareness of modern information systems.

Development and Industrial Application of Multi-Domain Security Testing Technologies (DIAMONDS) enables efficient and automated security testing methods of industrial relevance for highly secure systems in multiple domains (incl. e.g. banking, transport or telecommunication). In particular, over 90% of software security incidents are caused by attackers exploiting known software defects. DIAMONDS addresses this increasing need for systematic security testing methods by developing techniques and tools that can efficiently be used to secure networked applications in different domains. DIAMONDS will leverage systematic, model-based testing and monitoring approaches for security testing to enable highly secure systems by early testing and test automation. Advanced model-based security testing methods will allow the early identification of design vulnerabilities and efficient system/test design targeting security aspects. The DIAMONDS project convinced visitors at the ITEA & ARTEMIS Co-summit 2011 in Helsinki, and received the best-booth award.

Identification of a protocol gene: This research, PROTOS-GENOME, approaches the problems of complexity and quality by developing tools and techniques for reverse-engineering, and identification of protocols based on using protocol genes - the basic building blocks of protocols. The approach is to use techniques developed for bioinformatics and artificial intelligence. Samples of protocols and file formats are used to infer structure from the data. This structural information can then be used to effectively create large numbers of test cases for this protocol. In 2011, the project further developed the existing methodology, resulting in improvements in efficacy and discovering a number of vulnerabilities in web browsers.

Hash function security research is a part of cryptography, where hash functions and their properties are studied. Hash functions play a major role in many modern communication protocols, and are at the moment a very hot topic since NIST (National Institute for Standards in Technology) proposed a competition for a new and more secure hash function standard, SHA-3. This competition vets the one best algorithm from 64 different propositions. At OUSPG, the study of hash functions has led to some practical results, but mostly concerning theoretical advances in the field of hash function security. The results gained form this research and the cryptographic expertise are then applied to the practical information security work in OUSPG.


Web browser fuzzing in progress.


Exploitation of Results

The results of our research were applied to real-world problems in many projects, often in collaboration with industrial and other partners. Efficient exploitation of results is one of the core objectives of the national Tivit ICT SHOK projects like Cloud Software, DIEM and Cooperative Traffic – in these projects we work in close collaboration with companies throughout the projects.

In the II City project, we organized two interregional seminars in October 2011 to disseminate the results of the project, offered presentations on timely topics related to the project, and brought together organizations from Finland and Sweden. Altogether, 60 participants participated to the seminars that were organized in Oulu and in Piteå. The participants were from the Interactive Institute, the University of Oulu, other universities, the City of Oulu, and from Swedish and Finnish companies. Several fields of business were represented: content production, software development, mobile device manufacturing, wellness device manufacturing, game development, embedded systems development, ICT consulting and marketing.

The Intelligent Systems Group utilizes a robotics laboratory and a pressure-sensitive floor (EMFi material) installed in our laboratory as part of a smart living room. Other equipment includes a home theatre, two degree-of-freedom active cameras, four mobile robots and one manipulator, a WLAN network, and various mobile devices. Our aim is to gradually build a versatile infrastructure that offers various generic services for pervasive applications. Naturally, this kind of environment enables realistic experiments that lead to a better understanding of such applications.

The generic system for the semi-automatic maintenance of statistical prediction models is now in use for product planning at the Ruukki steel plate mill. The system enables us to keep up-to-date these models which are important for ensuring that the products fulfil the customer needs. As new products and production methods are continuously developed, and novel process settings are taken into use, data from previously unseen conditions occur, and thus there is a constant need to update the prediction models. Another major reason that causes a need for model updates is process drift. The system developed generates semi-automatically new model versions when the newest data makes it possible to improve the prediction accuracy. Each model version is a dynamic link library (DLL) that can be directly plugged in to the on-line applications that actually utilize the models. The system will give economic benefits because improved prediction accuracy in the product planning decreases production costs, and the costs needed to maintain good prediction accuracy will now be small.

During the reporting year, the group continued utilizing outdoor robotic systems. Development and utilization of Mörri, a multipurpose, high performance robot platform continued. More focus was put on perception in natural conditions, representation of detections, knowledge, and an environment model of operating environment. The software architecture further developed the earlier work on Property Service Architecture, and the Marker concept as general purpose representation was further developed.

Future Goals

We will continue to strengthen our long term research and researcher training. We will also continuously seek opportunities for the exploitation of our research results by collaborating with partners from industry and other research institutions on national and international research programs and projects. The group is a founding member of the European Robotic Network of Excellence (EURON). The group is a contract member of EURON II which was approved for the EU’s FP6 as a Network of Excellence.

We will strengthen our international research co-operation. Within the last years, the group has created collaboration projects with Japanese researchers. Two doctoral students have funding for one-year visits to Japanese universities. . We will continue the collaboration with Sonic Studio from Sweden and the Shanghai Jiao-Tong University from China. With the University of Tianjin in China we have a joint project in which methods and a system will be developed for vision-based navigation of Autonomous Ground Vehicles, which utilize an omni-directional camera system as the vision sensor. The aim is to provide a robust platform that can be utilized in both indoor and outdoor AGV (Autonomous Ground Vehicles) applications. This co-operation will continue.

In the USA, we will continue to co-operate with the Human-Computer Interaction Institute in Carnegie Mellon University with Assistant Professor Anind K. Dey. The research is on human modelling in the area of human-machine interaction. We have started new US-Finland co-operation through an NSF grant with Virginia Polytechnic Institute and State University. The research is on Techno Economic Models for Collaborative Access Network Provisioning.

Shorter research visits to European partners in EU-funded projects are also planned.

In 2012, the aim is to utilize more widely the know-how from sensor technology and data mining. New application areas will be studied including rehabilitation, exercise motivation and energy efficiency in households, and the benefits of our expertise will be highlighted to the actors in the areas.In human-environment interaction and sensor networks our research will continue mainly in Tekes and ICT SHOK projects. Our main goals are to develop analysis methods for sensor network data and to develop applications utilizing a physical user interface. Research on novel software architectures, reasoning and knowledge representations will continue as well. Field trials in realistic settings, and close collaboration with research groups (national and international) and companies will be emphasized.



professors & doctors


graduate students






person years



External Funding



Academy of Finland

262 000

Ministry of Education and Culture

237 000


1 123 000

domestic private

76 000


363 000


2 061 000

Doctoral Theses

Zhou J (2011) Pervasive service computing: community coordinated multimedia, context awareness, and service composition. Acta Universitatis Ouluensis. Technica 399.

Selected Publications

Davidyuk O, Georgantas N, Issarny V & Riekki J (2011) MEDUSA: middleware for end-user composition of ubiquitous applications. Mastrogiovanni F & Chong NY (Eds.) Handbook of Research on Ambient Intelligence and Smart Environments: Trends and Perspectives. IGI Global, 197-219.

Davidyuk O, Gilman E, Sánchez I, Mäkipelto J, Pyykkönen M & Riekki J (2011) iCompose: Context-aware physical user interface for application composition. Central European Journal of Computer Science 1(4): 442-465.

Davidyuk O, Sanchez I & Riekki J (2011) CADEAU: Supporting autonomic and user-controlled application composition in ubiquitous environments. Malatras A (Ed.) Pervasive Computing and Communications Design and Deployment: Technologies, Trends, and Applications, IGI Global, 74-102.

Gilman E, Su X, Davidyuk O, Zhou J & Riekki J (2011) Perception framework for supporting development of context-aware web services. International Journal of Pervasive Computing and Communications 7(4): 339-364.

Haverinen J & Kemppainen A (2011) A geomagnetic field based positioning technique for underground mines. International Symposium on Robotic and Sensors Environments (ROSE 2011), September 17-18, Montreal, QC, Canada, 7-12.

Juutilainen I, Koskimäki H, Laurinen P & Röning J (2011) BUSDM - An algorithm for the bottom-up search of departures from a model. Journal of Statistical Computation and Simulation 81 (5): 561-578.

Juutilainen I, Tuovinen L, Laurinen P, Koskimäki H & Röning J (2011) Semi-automatic maintenance of regression models: an application in the steel industry. Journal of Computing and Information Technology 19(3): 71-82.

Kemppainen A, Haverinen J, Vallivaara I & Röning J (2011) Near-optimal exploration in Gaussian process SLAM: scalable optimality factor and model quality rating. 5th European Conference on Mobile Robots (ECMR 2011), September 7-9, Orebro, Sweden, 283-290.

Leppänen T, Perttunen M, Kaipio P & Riekki J (2011) Sensor data fusion middleware for cooperative traffic applications. International Journal On Advances in Networks and Services 4(1‑2): 149-158.

Liu M, Koskela T, Ou Z, Zhou J, Riekki & Ylianttila M (2011) Super-peer-based coordinated service provision. Journal of Network and Computer Applications 34(4): 1210-1224.

Mikkola H, Rahikkala A, Kumpulainen K & Riekki J (2011) ActiveAquarium - virtual fish reflecting your physical activity. ETEN Conference 2010. Kumpulainen K & Toom A (Eds.) Journal of the European Teacher Education Network JETEN 6, 7-21, University of Helsinki.

Perttunen M, Mazhelis O, Cong F, Kauppila M, Leppänen T, Kantola J, Collin J, Pirttikangas S, Haverinen J, Ristaniemi T & Riekki J (2011) Distributed road surface condition monitoring using mobile phones. Ubiquitous Intelligence and Computing. Hsu C-H, Yang L, Ma J, Zhu C (Eds.) Lecture Notes in Computer Science 6905, Springer Berlin/Heidelberg, 64-78.

Pietikäinen P, Helin A, Puuperä R, Kettunen A, Luomala J & Röning J (2011) Security testing of web browsers. Communications of the Cloud Software 1(1).

Röning J, Casasent DP, & Hall EL (Eds.) (2011) Intelligent Robots and Computer Vision XXVIII: Algorithms and Techniques. Proc. SPIE-IS&T Electronic Imaging, SPIE 7878.

Röning J, Tuhkanen V, Sipola R & Vallius T (2011) Facilitating software construction for nano- and microscale measurement, SPIE Newsroom, Published Online, April 14.

Röning J, Tuhkanen V, Sipola R & Vallius T (2011) Software framework for nano- and microscale measurement applications, Intelligent Robots and Computer Vision XXVIII: Algorithms, Techniques, and Active Vision, 24-25 January 2011, San Francisco, California, USA, 787802:1-11.

Sanchez I, Cortes M, Riekki J & Pyykkönen M (2011) NFC-based physical user interfaces for interactive spaces. Ahson SA & Ilyas M (Eds.) Near Field Communications Handbook (Internet and Communications), Auerbach Publications, 175-230.

Schaberreiter T, Kittilä K, Halunen K, Röning J & Khadraoui D (2011) Risk assessment in critical infrastructure security modelling based on dependency analysis, 6th International Workshop on Critical Information Infrastructures Security (CRITIS11), September 8-9, 2011, Lucerne, Switzerland.

Siirtola P, Koskimäki H, Huikari V, Laurinen P & Röning J (2011) Improving the classification accuracy of streaming data using SAX similarity features. Pattern Recognition Letters 32: 1659-1668.

Su X, Riekki J & Haverinen J (2011) Entity notation - enabling knowledge representations for resource-constrained sensors. Personal and Ubiquitous Computing, doi: 10.1007/s00779-011-0453-6.

Tiensuu H, Juutilainen I & Röning J (2011) Modeling the temperature of hot rolled steel plate with semisupervised learning methods. Proceedings of 14th International Conference on Discovery Science (DS 2011). Elomaa T, Hollmen J & Mannila H (Eds.) Lecture Notes in Artificial Intelligence 6926, Berlin, Springer, 351-364.

Tikanmäki A & Röning J (2011) Markers - toward general purpose information representation. IROS 2011 Workshop: Knowledge Representation for Autonomous Robots, 6 p.

Vallivaara I, Haverinen J, Kemppainen A & Röning J (2011) Magnetic field based SLAM method for solving the localization problem in mobile robot floor cleaning task. The 15th International Conference on Advanced Robotics, June 20-23, Tallinn, Estonia, 198-203.

Vu T, Sokan A, Nakajo H, Fujinami K, Suutala J, Siirtola P, Alasalmi T, Pitkänen A & Röning J (2011) Detecting water waste activities for water-efficient living, 13th International Conference on Ubiquitous Computing (Ubicomp 2011), Beijing, China, September 17-21, 579-580.

Vu T, Sokan A, Nakajo H, Fujinami K, Suutala J, Siirtola P, Alasalmi T, Pitkänen A & Röning J (2011) Feature selection and activity recognition to detect water waste from water tap usage. The 1st International Workshop on Cyber-Physical Systems, Networks, and Applications (CPSNA 2011), August 28, Toyama, Japan, 138-141.


Last updated: 23.6.2016