Infotech Oulu Annual Report 2013 - Intelligent Systems Group (ISG)

Professor Juha Röning and Professor Jukka Riekki,
Department of Computer Science and Engineering , University of Oulu

juha.roning(at)ee.oulu.fi, jukka.riekki(at)ee.oulu.fi

http://www.oulu.fi/cse/isg


Background and Mission

The main scientific objective of the basic research conducted in the Intelligent Systems Group is to generate new, applicable knowledge on intelligent systems, and to generate positive societal impacts by applying this knowledge using scientifically plausible methods and state-of-the-art technology. We conduct research on spatial computing, collective intelligence, sensor networks, optimization of industrial processes, mobile robotics and cybernetics, human-computer interaction (HCI), human-robot interaction (HRI), computer networks, and security in complex information processing systems.

Our strategic research themes, whereby we aim to conduct world class basic research, are: 1) Safety and security on all levels of intelligent systems, 2) Data mining, with a special focus on optimization of industrial processes and well-being, 3) Human-environment interaction with a special focus on novel physical user interfaces, robot-environment and human-robot interaction, 4) Mobile robotics and cybernetics, with a special focus on spatial computing, adaptation, and field robotics, and 5) Sensor networks, with a special focus on understanding behaviour and patterns in our everyday environment.

We have conducted basic research related to these research themes for over ten years. Our team consists of two professors, seven post-doctoral researchers and 24 doctoral students. The annual external funding of the group is more than two million Euros, in addition to our basic university funding. There have been 17 completed doctoral degrees from the group. From the research of the group, eight spin-out companies have been established so far: Codenomicon, Clarified Networks, Hearth Signal, Nose Laboratory, Nelilab, Atomia, Probot and IndoorAtlas.

We co-operate with many international and domestic partners. In applied research, we are active in European projects. In addition, several joint projects are funded by the Finnish Funding Agency for Technology and Innovation (Tekes) and industry. We were a research partner in the Data to Intelligence (D2I), Internet of Things (IoT) and Cloud Software ICT SHOKs. In the Cloud Software Program ICT SHOK, we are responsible for cloud security CoP.

In the II City project, we collaborated with the University of Lapland and Sonic Studio from Piteå, Sweden, and in the Pervasive Service Computing project with Shanghai Jiao Tong University from China. These projects were funded by the European Union (Interreg IV A North) and the Academy of Finland (the MOTIVE program). Also we are collaborating with the Tokyo University of Agriculture and Technology in the Interactive Context-aware System for Energy Efficient Living project, funded by the Academy of Finland.

We are active in the scientific community. For example, Prof. Juha Röning is acting as visiting professor of Tianjin University of Technology and as the Robot Science Adviser of Tianjin Science and Technology Center for Juveniles. He served as a member of the Board of Directors in euRobotics and as a member of the SAFECode International Board of Advisors, and as a chief judge in the euRathlon 2013, which took place between 23rd and 27th September in Berchtesgaden, Germany. euRathlon is a new outdoor robotics competition which invites teams to test the intelligence and autonomy of their robots in realistic mock emergency-response scenarios. During the reporting year, the group organized two workshops on security, the 11th Finnish Vulnerability Researcher Meeting on November 18th, and the 7th International Crisis Management Workshop and Winter School (CrIM’13), which brought together both Finnish and international information security experts. A workshop on Human Activity Sensing Corpus and Its Application (HASCA2013) was also organized by Susanna Pirttikangas in conjunction with Ubicomp’13.

Prof. Jukka Riekki gave an invited talk at the National E-learning Conference in Helsinki. Several doctoral students continued their research visits during 2013 as well: Teemu Leppänen completed a one-year research visit to the Institute of Industrial Science, at the University of Tokyo, Japan, and Mikko Polojärvi visited Hokkaido University in Japan. Several members of the group co-chaired workshops and were also on the committees of international conferences.


Scientific Progress

Research on mobile robotics

This research includes several topics to support technologies of mobile robotic systems: motion planning, localization, energy management, information representation and development of a mobile robot system. A practical robotic system is being build that can operate indoors among people.

The Finnish Academy and European Regional Development Fund (ERDF) funded Minotaurus robot combines the theoretical and practical aspects of creating service robotics of the future. Both projects are in co-operation with the Center for Machine Vision Research (CMV). Another platform is being developed to realize a mobile robot that operates in a harsh outdoor environment.

Dynamic motion planning

Making improved motion planning in a real world environment is being studied. The planner combines perceptions from several sources, local obstacle avoidance with dynamic model and the performance capabilities of the robot, as well as the capabilities and limitations of the perception system. The developed system has been tested both indoors (using the Minotaurus robot) and outdoors (using the Modular Outdoor platform, Figure 1).

Figure 1. A real robot and its model using simulation software. For raw location, the robot uses a Kinect sensor and detection of visual clues like room signs and room numbers. Also a laser scanner profile is used for matching with prior knowledge of the corridor shape profile. Later, technology using magnetic field quality for localization will be merged to the robot.

 

Local obstacle avoidance gets its input from several sensor systems, combines the information to knowledge of the current state of the environment, and plans a route among obstacles. The perception modules include laser scanners, data processing and several vision algorithms using a Kinect range camera. These can detect obstacles and humans in the environment. Several obstacle avoidance algorithms can be run in parallel, and the same algorithms can be run with varying parameters, from which the best result for the current situation is selected (Figure 2). This improves the planner so it is able to find a proper route, even when one of the algorithms might fail, or some parameters are not suitable for the current situation. Further the parameters of each algorithm are tuned online, using genetic algorithms to be able to adapt to new conditions.

 

 

Figure 2. Calculating additional paths to the target (green) using several algorithms (like RRT,VFF) and varying parameters.

 

The robot uses a graph based navigation map for finding raw route points, and an RRT (Rapidly-exploring Random Tree) for finding a free path from the current location to the next route position. Indoors, the graph is predefined from the floor map. Outdoors, the graph is generated from the Open Street Map data.

A dynamic model of a robot is used to set restrictions for the path planner and predict the forces that would affect the robot as it moves along the planned route. The restrictions come from environmental parameters and the robot’s performance capabilities. The dynamic model can be continually adjusted, based on the difference between the robot’s predicted and actual behaviour. The algorithms for the dynamic model are developed using a highly accurate physics simulator.

Magnetic field localization and SLAM

The objective of our indoor localization research is to develop methods for exploiting magnetic field anomalies in positioning. The idea is based on analysis with various indoor magnetic field datasets showing that indoor magnetic fields provide sufficient spatial variation and temporal stability to permit inference about sensing locations, given noisy measurements. In recent years, we have published various papers presenting magnetic field localization in robotic and human contexts (Figure 3).

Based on the magnetic field localization studies, a new start-up company, called Indoor Atlas Ltd., was founded in 2012. This company offers indoor positioning technologies for various application areas. The company has generated high interest in international technology magazines.

Figure 3. Magnetic field SLAM in an apartment building (top left). The strong variations in the magnetic field (top right) allow the robot to construct a geometrically correct map (bottom left) despite the errors in its odometry (bottom right).

 

In our research on magnetic field localization and SLAM, we have put a strong emphasis on light-weight methods running entirely on mobile platforms, such as Android smartphones and tablets. Compact map representation and effective algorithms are essential when using devices with very limited resources, and we have developed methods to tackle the problems arising from very sparse data and high uncertainty levels produced by low-cost and noisy smartphone sensors. Our work is continuing toward an autonomous mobile robot system based solely on smartphone sensors that is able to intelligently build a map of the magnetic environment (Figure 4).

Figure 4. An iRobot Create equipped with a Samsung Galaxy Nexus smartphone utilizing the same magnetic map for localization as the person holding an Asus Nexus 7 tablet. The same localization unit can be used by both humans and robots in social robot scenarios.

 

Social robot scenarios are particularly difficult because of the dynamic (often crowded) environment. Magnetic field localization is not affected by surrounding people like laser scanners and cameras for example, and it is therefore very promising in these kinds of scenarios. While our method is used to localize the robot, the other sensors can be assigned to handle the social tasks. We have also developed localization methods that are usable by both robots and humans equipped with similar mobile devices.

Winter operability for electric vehicles

Intelligent battery systems for robots and light-weight electric vehicles are being developed partly in the WintEVE project. The focus is on developing scalable battery powered systems with integrated energy management, improved energy efficiency and the ability to operate in varying conditions, such as those experienced during winter. In addition, interest is in gaining a better understanding of power usage and operational times of the related equipment, such as mobile robots. Work has included a study of upcoming battery technologies that might drastically improve the capacity and power to weight ratios in the near future, as well as making of systems that can profile and control power usage in robots. For example, robots may need to turn off unnecessary sensors, when they are not needed, to conserve energy. Artificial neural networks are being utilized for creating better models for representing the true state of the charge and condition of the battery system. Further, the models can be used for predicting the capacity and energy available for the upcoming operating periods as a part of forward planning and task scheduling.

Energy efficient model predictive control relates to the development of intelligent battery systems, where the goal is to optimize energy usage in varying equipment and operating conditions. In mobile robotics, surveying the environment utilising flying drones in conjunction with a ground unit, utilizing Simultaneous Localisation And Mapping (SLAM), is considered in tasks related to movement surface classification and route planning processes. Machine learning is employed in adaptation of models related to predictive control for automatizing optimal route learning and planning. This work may be further expanded to full electric cars, operating in large road networks, by generating automated mappings of road conditions, for example road surface classification based on recorded energy consumption, traction and slope (Figure 5). The classification data may then be used by model predictive route planning algorithms for internal and external car control, aimed at driving energy prediction, congestion avoidance and possibly for larger scale traffic flow optimization. Weather and its effects on energy consumption is one of the focus points for this research, especially for full electric vehicles (FEVs).

 

Figure 5. Mapping of consumed (red) and braking regenerated (green) energy of an electric car during a single driving session.

 

To further improve energy efficient control, our outdoor robot platforms are being upgraded with energy harvesting capabilities by hardware development, primarily utilising braking energy collection via advancing technologies like super-capacitors (Figure 6). Much of the required hardware has been manufactured during 2013; initial testing and control system development should, therefore, be performed during 2014. In order to achieve the highest efficiency possible for mobile platforms, the research needs to combine the results of dynamic motion planning, intelligent battery management, SLAM and model predictive control.

Figure 6. Outdoor robot combined with a possible super-capacitor braking system (red).

 

Autonomous water quality sensing

Currently, collecting a large amount of water quality data using either field sensors or sampling bottles requires enormous manual efforts. Not only the time spent on changing environmental conditions, but also positioning and route planning bring challenges, especially when these are required to be performed in-situ. A typical task for such a case would be searching for pollution leaks, or modelling how some water quality parameter is distributed over a lake or shore.

Our research on autonomous water quality sensing aims at cost-efficient sensing strategies in terms of sensing time and modelling accuracy controlling robotic vessels in the target water environment. The key approach is to apply our previous research on indoor magnetic field exploration to the water quality sensing, where an additional challenge follows from the multiple variables (called water quality parameters), such as dissolved oxygen, turbidity, electrical conductivity, pH and fluorescent dissolved organic matter. Also, in many cases, this information is required to be measured from different depths so that the spatial models are three dimensional.

In order to operate in water environments, following from the Tekes TULI project (projects aiming to commercialize research innovations), two unmanned surface vehicles were constructed during 2013 (Figure 7). These robotic platforms enable users to monitor through video-surveillance sensing tasks; meanwhile, all the information is transferred to the database and can be visualized through a web service. A new spin-off Aquamarine Robots Ltd. was founded in December 2013. This company offers robotic vessels and an aquatic information system to collect, store and visualize relevant information from freshwater and mining environments.

Figure 7. A robotic vessel performing a navigation task in July 2013.

 

Research on human-robot interaction

Key aspects are putting research results into practice, and integrating them as a part of the operation of the robot. The main challenges are representation of information, and adapting a set of algorithms to the current state in the surrounding environment. The human friendliness of the outward appearance of the robot is driving the design of the robot. Studies on interaction with a machine have continued from various perspectives. The capability of the Minotaurus robot has been improved with new sensors, perception processing and physical capabilities. The robot is also capable of engaging in simple dialogs using English or Finnish. Now we are adding the ability of recognizing and picking up objects from a surface upon request (Figure 8).

Figure 8. The Minotaurus robot grasping a mug.

 

The robot is based on a Segway RMP200 platform, and equipped with several sensors to receive information from the environment and humans. Two Kinect sensors are used as a 3D vision camera, one looking towards the floor and one looking forwards. Multiple 2D scanning lasers are additionally used for close distance obstacle avoidance, and for refining the estimation of the robot’s movement. Further, the robot is equipped with several microphones to reason the source direction for audio and for speech recognition.

This year a commercial arm has been implemented on the robot (Jaco Kinova). The arm was originally developed for wheelchair operation, and it provides a good reachability-weight ratio. The arm is flexible and the robot can reach the ground, tables and even higher located objects using it. Further research has also been done in the areas of perception and dynamic motion planning. The robot can now find objects on a table and calculate the best possible grasping pose.

During the year, the Minotaurus robot was demonstrated in several events, including Robottipäivät in Tietomaa and the University Science day at the University of Oulu.

Interaction with a machine has been studied from various perspectives. The combination of machine vision, speech recognition and synthesizing, touch and touchless interactions, along with the robot’s operation in the environment requires a software platform that processes, distributes and stores information efficiently. Real-time Linux based operating system services, along with general purpose representation for information (called Markers) have been developed to support integration of the algorithms. Marker representation is used for representing the output of sensor processing algorithms, for representing a model of environment and obstacles around the robot, and for representing information related to the robot’s tasks.

The Evolutionary Active Materials

The Evolutionary Active Materials (EAM) project, which is funded by the Academy of Finland, is a joint effort between the Computer Science and Engineering laboratory (CSE) and the Microelectronics and Materials Physics laboratories. The aim of the EAM project is to develop novel, evolutionary computation (EC) based design methods for active and versatile materials and structures. The first components are being developed through a novel holistic design process utilizing constantly increasing computation power, the development of multi-physics simulators, and EC techniques, such as genetic algorithms.

During 2013, a Cymbal type piezoelectric actuator was optimized by using a combination of a genetic algorithm and FEM modelling software Comsol Multiphysics. From the optimized results, maps of electromechanical capabilities of different structures were generated. The blocking force of the actuator was maximised for different values of displacement by optimizing the height of the cap and the length flat region of the end cap profile. By using values obtained from a genetic algorithm optimization process, a function was formulated for design parameters. Using the function, a map of displacement, the steel thickness and the height of the end cap the optimized length of flat region was constructed (Figure 9). A similar map with the length of the flat region for the optimized height of end cap was created (Figure 10).

Figure 9. Height of the steel cap as a function of steel thickness and displacement for Cymbal.

 

Figure 10. Length of the flat region of the cap as a function of steel thickness and displacement for Cymbal.

 

Also parameters of a new structure called “mikbal” were optimized with a genetic algorithm. The advantage of mikbal compared to Cymbal is its ability to generate large displacements using less piezo material than Cymbal. With a 25 mm piezo diameter and 40 mm steel diameter mikbal produces 114 µm displacement. Cymbal with 25 mm diameter generates only a 67 µm displacement and Cymbal with 40 mm produces 189 µm.

Research on human-environment interaction

Our current work on human-environment interaction focuses on physical user interfaces and human-robot interaction. In physical user interfaces, mobile terminals are used as physical objects rather than as traditional I/O devices. Our current studies concentrate on touch-based interaction: users interact with the local environment by touching objects with their mobile terminals. The touch-based user interfaces employ NFC technology (i.e. RFID technology for mobile phones): an act of touching brings an NFC reader near an NFC tag, and hence the data in the tag is read and delivered to the system. The objects that can be touched are advertised in the environment by NFC icons, graphical icons resembling the icons of the graphical user interfaces of computers and other user terminals.

In 2013, we have focused our research on building innovative user interfaces for learning environments in which learner’s engagement, social interaction and collaboration among learners are emphasized.

 We have also continued the development of two novel NFC devices, an Activity Pad and an NFC Mouse, together with Offcode Ltd, an Oulu based embedded systems company. This work was carried out in the Active Learning Spaces project funded by Tekes and coordinated by the University of Tampere. Moreover, we developed applications for the Activity Pad and the NFC Mouse in the Digital Services ICT SHOK, this project is also together with Offcode, the University of Tampere, and several other project partners. Furthermore, we continued our participation in the Future School Research Second Wave project that is funded by the European Social Fund and the City of Oulu, and coordinated by the Faculty of Education at the University of Oulu. We are also developing applications for NFC phones in these three projects.

The Activity Pad combines a 4x6 grid of programmable NFC readers together with printed sheets of A4-sized paper to allow teacher-driven creation of interactive learning applications featuring application-specific tangibles, and promoting co-located interaction. Teachers and students create study material using traditional methods; drawing on paper and molding objects from clay, for example. Once the material has been generated, a student first places a sheet of A4 paper on the pad; this paper presents a task, for example, a sentence with some missing words. The pad identifies the task in question from an NFC tag attached to the paper sheet. Then the student performs the task by placing pieces of paper or other objects augmented with NFC tags on the task paper. The pad reads the information from the tags and provides visual (LEDs) and audio feedback. Teachers can program their own pad applications easily by drawing a task paper, placing the paper and correct objects on the pad, and recording the correct answer (Figure 11).

Figure 11. An example of tasks that can be realized with the Activity Pad. This task teaches children how to recycle. The child has to place different waste material in the appropriate bin.

 

The NFC Mouse is a portable device that enables reading NFC tags from the environment. Applications are stored in the device’s memory. A child starts an application by touching the corresponding NFC tag. Then s/he solves the given task by finding objects from the local environment and touching the NFC tags attached to these objects with the mouse. The device provides auditory, haptic and visual feedback (LEDs). We have also developed an intuitive GUI to program both devices by connecting jigsaw pieces. No knowledge of any programming language is needed to program the devices.

For NFC phones, we are developing situated mobile learning applications for adult education and training. This work is performed in the Active Spaces project together with Context Learning Finland. The applications enable learning experiences in which the virtual and real (i.e. physical) environments are interconnected. We have continued the work started last year, implementing and testing another application in order to instruct social workers about fire safety. NFC tags are placed behind posters on a training centre, and scenarios are prepared for each tag location as multimedia presentations. Each poster presents a situation that is linked to the physical place where it is located. When a trainee touches a tag with a phone, the application presents the corresponding multimedia presentation. Trainers can monitor the progress of the learners at any time.

In the Digital Services ICT SHOK, we have developed a second version of NFC-ACT, an authoring tool targeted at primary school teachers. This PC application can be used to create NFC-enabled learning games for mobile phones. The games follow the same structure: a game presents a challenge on the phone screen and the child solves the challenge by touching the correct NFC tag among the tags placed on objects in the environment. Teachers create games by defining the application flow and the multimedia content to be shown on the screen at each stage. This tool can also be used to write the NFC tags using an external NFC reader, and to deploy the games to children’s mobile phone using Bluetooth. Furthermore, in this project we have tested a peer-to-peer WiFi connectivity solution in order to distribute learning content among a group of children and their teachers using low-end phones. Children solve questions proposed in exercises and send the results to teachers.

In the Future School 2nd Wave, we have continued our collaboration with researchers from the Department of Education and with teachers to create NFC based learning games for pupils with heterogeneous backgrounds (ages from 7 to 12, different nationalities). Our aim is to enhance learning Finnish, which the target group is acquiring as a second language. The teachers have used the games as tools to introduce vocabulary (colours, body parts, Christmas words, etc.) to the pupils.

We have also created a similar NFC based application for cognitive stimulation of elderly. We collected audio and pictures from well-known people, TV programs, and events from the 50 s and 60 s to stimulate short-term and long-term memory recall. We printed pictures and placed NFC tags behind them. When the users touched the tag with the phone a corresponding audio was triggered. Our research results show that this type of application also promotes social interaction among the elderly.

We continued our work on a general purpose data visualization framework in the Internet of Things ICT SHOK. The purpose of this framework is to simplify the creation of systems that connect user-specific data from data sources to visualization applications, or other applications that modify their behaviour based on data input. In 2013, we used this framework to build a new prototype. Our agenda was to work with games and game-like concepts where users can play with their data. Instead of merely seeing visual improvement, users can now turn their behavioural progress into virtual furniture and game pieces. In our prototype, users furnish and energize a virtual office (Figure 12).

Figure 12. This chaotic office is from our month long testing period with Laturi Corporation. Coloured portions of the office are energized.

 

In general, motivational games can be more involving than simple visualizations. In addition to creating the game prototype, we were able to discover design principles that should be taken into account when creating games and other interactive applications that use sensor data in what can be described as an after-the-fact fashion.

We also started working on a general framework for visualizing the status of IoT systems in two different methods: a 3D virtual environment implemented on the RealXtend platform, and a 360-degree panorama picture in mobile phones. We started exploring the visualization possibilities by using data from a home environment provided by a ThereGate device (Home Energy Management System (HEMS) gateway), Finwe Key2phone (a lock system operated by mobile phones) and taxi traffic information collected from sensors in the city of Oulu. We are developing a general architecture for collecting, storing and visualizing data acquired from the home environment. Figure 13 shows an example of the power consumption visualized in the RealXtend environment and panorama pictures visualizing power consumption and taxi activity. Visualized data can help the user form a general picture of the underlying system. For example, visualizing real-time power consumption on home appliances can motivate people to use the devices in a smart and more energy-efficient way.

Figure 13. Panorama power consumption of a microwave (left), panorama taxi activity in downtown Oulu (upper right) and the power consumption of a television in virtual 3D environment (bottom right).

We continued exploring smart space interaction, specifically how knowledge-based technologies enable smart spaces not only to adapt their behaviour according to the actions of users, but also to initiate interaction when necessary. Moreover, we continued exploring the usage of metareasoning concepts in ubiquitous systems. A real prototype of a traffic-related system, which implements our ideas, is under development. Also, we started to explore how metareasoning concepts can support ubiquitous learning systems, making them more useful for both students and teachers.

Research on situation awareness

Understanding behaviour and patterns in our everyday environment is essential in human-computer interaction, and our approach is to utilize different kinds of sensor networks to enable advanced services that utilize this knowledge. Naturally, the possibilities that emerge from collecting data from the environment bring also challenges as big data processing requires special knowledge about big data storage and distributed data processing. One of our most interesting application areas is traffic, as several different actors produce large amounts of data continuously. In 2013, we continued work in the Internet of Things ICT SHOK and the Data to Intelligence ICT SHOK.

In the Data to Intelligence SHOK, we are working on collecting, processing and storing traffic related data from the City of Oulu region. We have built a Cloud service for storing real time information collected by taxi cabs operating in the city. We have developed, together with our project partners (several ITS companies and universities), the Oulu Traffic Pilot that realizes data collection from moving objects, digital maps, weather and environmental sensors, and it visualizes the cleaned and analysed information on a web client (Figure 14). Together with our partners, we are designing more flexible ways of storing, processing and possibly selling the data and analysis results.

Figure 14. Visualizing taxi cab emission statistics at interesting road segments in the city of Oulu (Oulu Traffic Pilot). The user interface is by Infotripla, the data is collected by EC-tools, and it is cleaned and analysed by the University of Oulu. The data provider is Oulun aluetaksi.

 

We fuse moving object data, road weather data produced by the Finnish meteorological institute, the Digiroad digital map data on the Finnish road and street network maintained by the National Land Survey of Finland, the Finnish Transport Agency and individual municipalities (Figure 15). We utilize the information in creating different profiles for road segments that can be exploited in advanced pollution-aware routing or making assumptions on the current situation, for example (Figure 16). We are also developing our own algorithms for extending the existing GIS analysis tools.

Figure 15. Fetching data from interesting road segments from the Digiroad digitial map database to be matched with data on taxi cab trips.

 

Different kinds of statistics can be derived from the moving object data. The dangerous/safe spots, congestion areas, areas of high deceleration, etc. can be found when vehicle data is collected. It is also interesting to try to find how this information can be used in taxation, as well. When social network data is fused to physical data, the possibilities are extended.

Figure 16. Average pollution level (green low, blue moderate, red high) from a vehicle’s OBDII data during half a year in the City of Oulu.

 

We are studying adding semantics to Internet of Things systems, specifically how the benefits of semantics can be enjoyed without sacrificing energy efficiency and short latencies. We utilize the existing Semantic Web technologies, as their interoperability allows application developers to use nodes implemented and deployed by others. In 2013, we built a system connecting sensor nodes to knowledge-based components and in this system evaluated different data formats in terms of their semantic expressivity and resource consumption. This work suggests to IoT application developers data formats offering the best compromise between the usually conflicting characteristics of good expressivity and modest resource requirements. Moreover, we are studying an approach to transforming one well-known sensor data format SenML (Media Types for Sensor Markup Language) to Semantic Web representations (Figure 17). Hence, many intelligent IoT applications can be built on SenML-enabled devices.

Figure 17. Energy consumption of both encoding and decoding different data formats based on this sensor system.

 

We extended our previous research on the UBI-AMI v2 wireless sensor network-based home energy consumption monitoring application, where the benefit is offering a holistic real-time view of the energy consumption of a building and, on the other hand, of the individual household appliances. The holistic view presents real-time and historical energy consumption apartment-wide, as often the infrastructure elements, such as heating or saunas, are the most energy consuming elements in comparison to the individual everyday appliances. This architecture is based on ideas of dynamically configuring the wireless sensor network for particular applications, and dynamically composing sensor-based services from the sensor data. Users can monitor their individual energy usage and inject user-defined computational tasks into the system in runtime from a Web service. The tasks are represented by mobile agents, migrating in the system devices based on their resource utilization, and are capable of both sensing and actuation. The agents inherently distribute the computational load in the participating devices in the system, which additionally contributes to the energy saving in the wireless sensor network infrastructure.

In the MAMMOTH project, we continued research on distributed system architectures for the M2M systems and the Internet of Things. This architecture is based on the above idea of a generic software framework, enabling the use of mobile agents as the user or application defined computational task representation. We extended the software framework with mobile sensing platforms, such as smartphones, towards generic IoT devices. The framework enables utilization of heterogeneous data sources, including low-power resource constrained wireless sensor platforms, as data producers for the IoT ecosystem. This sensor-based or refined data can be utilized by data producers in their applications, or by the dynamically injected mobile agents in their computations locally in the IoT devices (Figure 18). Additionally, we studied mobile agents-based smart objects in the IoT environment and demonstrated an evaluation method for the assessment of energy efficient resource utilization in the smart spaces on top of the IoT ecosystem. The work with the UBI-AMI v2 system, and the activities in the MAMMOTH project were conducted in co-operation with MediaTeam.

Figure 18. The developed software framework for IoT applications.

 

During the year 2013, we started co-operation with the University of Tokyo and Aoyama Gakuin University, and continued our previous co-operation with the Tokyo Denki University in Japan, as one of our doctoral students, Teemu Leppänen, was a visiting research associate with the Institute of Industrial Science at the University of Tokyo. This work included studying energy efficient participatory sensing applications with the mobile sensing platforms and mobile agents, furthermore extending the above mentioned software framework.

Research on data mining

GlobalRF: In 2013, the Global Spectrum Opportunity Assessment (GlobalRF) project has started. GlobalRF is a collaborative research project with a joint effort undertaken by WiFiUS (Wireless Innovation between Finland and U.S.), leveraging research and education sources in Finland and the U.S. in the area of wireless communications. The collaborating institutions are the Illinois Institute of Technology (IIT) and the Virginia Polytechnic Institute and State University (Virginia Tech) in the U.S., and the VTT Technical Research Centre, Turku University of Applied Sciences, and the University of Oulu in Finland. All these institutions have ongoing research and education programs in wireless communications, and bring significant expertise and resources to the proposed project.

Currently the data analysis and inference group is working with large-scale statistical analysis in the GlobalRF project where a fundamental radio frequency (RF) spectrum shortage problem is tackled by developing methods for understanding the current and evolving use of the spectrum in various environments. The study consists of a combination of the RF spectrum data with open data, such as human behaviour (e.g., mass events) and weather (e.g., extreme weather conditions), to discover different correlations, possible anomalies and variables explaining frequency band usage in both the frequency and temporal domains (e.g, multi-dimensional time-series). Moreover, the research in the area includes the development of big data management, processing, and visualization tools, as well as building predictive models to realize novel ways and guidance for dynamic sharing of spectrum usage. Conversely, RF spectrum measurement (and open datasets), intelligent data analysis and machine learning algorithms could provide novel ways to model environmental and human related contextual variables in urban city areas. At the moment, RF measurement units are installed in downtown Chicago and Turku, producing ongoing data for analysis.

New methods for activation of young men: the multidisciplinary MOPO project combines traditional health promotion, modern technology and the measurement of physical activity. The aim of the study is to provide knowledge about physical activity, the attitude towards physical activity, information behaviour, fitness, health, nutrition, life habits and cultures of young men. A novel wellness coaching service for preventing marginalization and promoting physical activity and health in young men is being developed in the project. The information obtained in the study can be used to promote the wellness of young adults in education, and in the study and decision-making of professionals in social and health services.

Altogether about 6000 conscription aged men (five call-up age classes) have been/will be invited to participate in the MOPO study. The condition, wellbeing, health, relationship towards physical activity, information behaviour and the use of media and technology will be clarified during the years 2009–2014 with questionnaires, measurements and interviews. The contents of the service network were developed together with the city of Oulu, conscription aged men and several companies.

The data mining group has been responsible for the development of a gamified webportal for mobile phones (Figure 19). Young men collect data about their daily activity using a wrist worn Polar Active, or a mobile application developed by ISG. The mobile application recognizes the user’s daily activities (walking, running, biking, riding a car and idling) using mobile phone accelerometers. The algorithms used in the recognition process are independent of the user, phone location and the mobile phone’s operating system. Information about daily activity is uploaded to the portal, and based on the data, tailored feedback is given to the user. In addition, through the mobile webportal, tailored information, for instance about health, and nutrition, is given to the user. As mentioned, the portal is gamified: based on physical activity and activity in the portal, points are given to the user. These points can be used in a mobile game called “Clans of Oulu” where different clans are trying to conquer the city of Oulu, Figure 20. The game has been developed by LudoCraft.

Figure 19. The webportal offers tailored information, for instance, about physical activity, fitness, health, and nutrition.

 

In autumn 2013, access to the portal and the activity monitor was given to approximately 250 conscription aged men. The intervention started in autumn 2013 and lasts 6 months. The purpose of this intervention is to find answers to the following questions:

  • Did the portal and the game motivate the user to exercise?
  • Which features of the portal did users use most?
  • What was the overall opinion about the ICT platform?
  • Did the health and exercise messages given through the portal motivate the user to exercise?
  • Did the sensors alone motivate the user to exercise?

During the test, the behaviour of the portal users is tracked, and this information can be exploited to develop a new version of the portal.

Figure 20. Clans of Oulu is a game where you cannot succeed unless you are physically active and use the portal actively. Based on activity, the user gets points which can be used to conquer the districts of Oulu by the user’s clan.

 

For several years now, one of the interests of ISG in data mining research has been the study of ethical issues associated with data mining. These include the well-established issue of protecting the privacy of data subjects, and also a number of previously overlooked issues such as fairness and trust. The ethics research of ISG also emphasizes the potential of data mining as an instrument for pursuing things of value rather than just a threat to values that needs to be controlled. All these perspectives are combined in the MOPO projects: the outcome of the project, a persuasive video game, may help the study subjects adopt healthier lifestyles, but its development involves potential ethical problems that must be actively identified and worked against. An ethical analysis of the technical work in the project is being carried out and will be published alongside the work itself.

Operators of the study are the Oulu Deaconess Institute’s Department of Sports and Exercise Medicine, the University of Oulu, the City of Oulu, the Virpiniemi Sports Institute, the Finnish Defence Forces and wellness technology companies from Northern Finland. The project website can be found from www.tuunaamopo.fi.

Data mining methods for steel industry applications: the Intelligent Systems Group is a member of the Centre for Advanced Steels Research - CASR, which is one of the interdisciplinary umbrella organizations of the University of Oulu. In 2013, together with other groups in CASR, an application for a large national research programme System Integrated Metal Production – SIMP was prepared. SIMP focuses on metal processes, and related model and algorithms creation, which requires advanced mathematics and chemistry research. The actual quantum jump in the industry’s competitiveness requires resolution of ICT technology related challenges i.e. real time adaptation of process simulation and multi-physics models, multidimensional sensor data analysis and handling over the production systems, big data reduction methods, multivariate analysis tools, prediction of product quality solutions, etc. The programme budget is 43.8 M€ and its duration is 5 years from the beginning of 2014. There are 19 companies and SMEs participating, 7 Finnish academic and research institutions and 31 international participants that will improve the profitability of the industry (Figure 21).

Figure 21. A steel plate during manufacturing.

 

There were two PhD studies that developed data analysis solutions for the needs of the steel industry. In the first of the two PhD studies, the topic is the modelling of the quality of steel products in extreme conditions of a rolling mill. The goals of the study lie in in the development of new data analysis methods in the steel manufacturing processes, such as modelling of hot steel plate rolling schedules and the flatness of the steel plate or strip. So far, the temperature model for the steel plate during the rolling process has been made, and it has achieved a prediction accuracy of about 20 degrees for the whole pass schedule, and 16 degrees for the last rolling passes. This is a major improvement in comparison to earlier attempts. Additionally, significant efforts have been made to find features that describe the rolling process and enable the prediction of the temperature at different stages of the rolling. The compression of the history of the pass schedule has been one of the most challenging feature extraction tasks. The results show that the features that sum over the whole pass schedule are most important.

During the study, the problem of missing data has been considered. The measurements of rolling temperature contain a large amount of missing values due to the hot and moist conditions of the rolling process. By employing semi-supervised learning, the information contained in the observations with missing response measurements was utilized to improve the learning process by employing the COREG-algorithm. Such learning methods utilize both the labelled and the unlabelled data to produce better learners than those using only the labelled data. As a result, a slight increase in the prediction accuracy was observed. The lack of reliable measurements is a relevant issue in many rolling mills.

The future research will concentrate on two-dimensional quality properties like shape and flatness as a part of the SIMP-programme. The industrial partners for the research are Ruukki Metals, Raahe Works and Outokumpu Stainless Oy, Tornio.

The other PhD study concerns methods for exceedance probability estimation in the case of multiple measurement test sets. The situation may occur when the variability of the quality within the product is high, and thus, traditional point prediction based modelling methods are not sufficient. Density forecasting methods are needed when not only the mean, but also the deviance and the distribution shape of the response depend on the explanatory variables.

In the research, the impact toughness of the steel products has been modelled. The rejection probability in the Charpy-V quality test has been predicted with mean and deviation models, a distribution shape model and a quantile regression model. The proposed methods have been employed in two steel manufacturing applications with different distributional properties, and it has been shown that especially the case with diverse product assortment benefits from the distribution modelling. The industrial partners of the research are Ruukki Metals, Raahe Works and Ovako, Imatra. The thesis was sent to pre-examination in September.

Cognitive load study: Eija Ferreira is collaborating with Prof. Anind K. Dey from the Human-Computer Interaction Institute of Carnegie Mellon University (USA) to study how to assess the cognitive load of a person in real-time, based on psycho-physiological measurements. Task interruption, divided attention and multitasking cause split attention that increases our cognitive demands. This is a problem, especially while performing tasks during which momentary meandering can have serious consequences (e.g. driving, manoeuvring heavy machinery, controlling air traffic, walking in traffic, performing surgery, etc.). The ability to measure real-time changes in the cognitive load of individuals provides a key tool to help mediate our attention resources while performing cognitive intensive tasks (Figure 22). In addition, our cognitive abilities change as we age. Therefore, understanding issues of attention demand is particularly essential for the elderly. Our research has shown that it is possible to build systems that respond to changes in a person’s cognitive load at a time granularity of 10 seconds, and that the same tools and sensors can be used to assess cognitive load for both younger and older adults.

Figure 22. Real-time cognitive load assessment has wide applications in predicting drivers’ ability to maintain vehicle control.

 

Management of semantic data in virtual environments: In the Reaxity (Future City as Open Mixed Reality Space) project, ISG is working as part of a multi-disciplinary consortium to create concepts and enabling technologies for mixed reality-based services in future urban spaces. The project is funded by Tekes as a small strategic opening, and it focuses initially on developing demonstrations of innovative service concepts and roadmaps for future work. In the next phase, the goal is to secure funding for a large strategic opening, and then proceed to pursue the long-term vision of the project by following the roadmaps.

The role of ISG’s data mining researchers in Reaxity is to study questions related to the management of semantic metadata in mixed reality (MR) spaces. Semantic metadata is used to encode the meaning of MR objects in a machine-readable format, making it easier, for instance, for a user to discover relevant services and content. The principal questions to be answered are: how the metadata is authored, how it is represented, and how it is incorporated in MR objects. By designing and prototyping a metadata management solution that addresses these questions, ISG is participating in the development of an open technology platform for MR applications in future cities.

Research on software security

Within the Intelligent Systems Group, the Oulu University Secure Programming Group (OUSPG) has continued research on security and safety in intelligent systems. Security and safety challenges in intelligent systems are threefold: increasing complexity leads to unforeseeable failure modes, quality is not the priority and awareness is lacking. We have approached the challenges from these three directions in our research.

Complexity - Model Inference and Pattern Recognition: we work under the premises of unmanageable growth in software and system complexity and emergent behaviour (unanticipated, not designed) having a major role in any modern non-trivial system. We have worked on natural science approaches to understanding artificial information processing systems. We have developed and applied model inference and pattern recognition to both content and causality of signalling between different parts of systems.

Quality - Building Security In: software quality problems, wide impact vulnerabilities, phishing, botnets, and criminal enterprise have proven that software and system security is not just an add-on, despite the past focus of the security industry. Instead, security, trust, dependability and privacy have to be considered over the whole life-cycle of the system and software development, from requirements all the way to operations and maintenance. This is furthermore emphasized by the fact that large intelligent systems are emergent and do not follow a traditional development life-cycle. Building security in not only makes us safer and secure, but also improves overall system quality and development efficiency. Security and safety are transformed from inhibitors to enablers. We have developed and applied black-box testing methods to set quantitative robustness criteria. International recognition of the Secure Development Life Cycle has provided us with a way to map our research on different security issues.

Awareness - Vulnerability Life Cycle: Intelligent systems are born with security flaws and vulnerabilities, new ones are introduced, old ones are eliminated. Any deployment of system components comes in generations that have different sets of vulnerabilities. Technical, social, political and economic factors all affect this process. We have developed and applied processes for handling the vulnerability life-cycle. This work has been adopted in critical infrastructure protection. Awareness of vulnerabilities and the processes to handle them all increase the survivability of emergent intelligent systems for developers, users and society.

These research goals are reached through a number of research activities.

Secure Software Development Lifecycle in Cloud Computing as part of the Cloud Software project - we approach all three goals by researching practical ways of building security into complex Cloud Computing services, from the design phase to actual operational use (Figure 23). This work culminated in a practical handbook of the Secure Agile Software Development Cycle.

Figure 23. Dependencies of a single cloud based web service visualized by technology and location.

Development and Industrial Application of Multi-Domain Security Testing Technologies (DIAMONDS) enabled efficient and automated security testing methods of industrial relevance for highly secure systems in multiple domains (including for example banking, transport or telecommunication). In particular, over 90% of software security incidents are caused by attackers exploiting known software defects. DIAMONDS addressed this increasing need for systematic security testing methods by developing techniques and tools that can efficiently be used to secure networked applications in different domains. DIAMONDS leveraged systematic, model-based testing and monitoring approaches for security testing to enable highly secure systems by early testing and test automation. Advanced model-based security testing and visualization methods allowed for the early identification of design vulnerabilities and efficient system/test design targeting security aspects (Figure 24).

Figure 24. Port scanning visualized in an industrial automation network.

 

Identification of a protocol gene: this research, PROTOS-GENOME, approaches the problems of complexity and quality by developing tools and techniques for reverse-engineering, and identification of protocols based on using protocol genes - the basic building blocks of protocols. The approach is to use techniques developed for bioinformatics and artificial intelligence. Samples of protocols and file formats are used to infer structure from the data. This structural information can then be used to effectively create large numbers of test cases for this protocol. In 2013, the project further developed the existing methodology, resulting in improvements in efficacy and discovering a number of vulnerabilities in web browsers.

Economic Models for Collaborative Access Network Provisioning (EMCANP) was launched in 2012 in cooperation with the Centre for Wireless Communications and the Virginia Polytechnic Institute and State University. The project explores the potential for capacity increase in heterogeneous wireless networks through increased collaboration, both among network operators and among end users. The task of OUSPG is to create a socio-technical security model for the purpose of identifying the information security threats involved in such collaboration and the appropriate responses to those threats. The chosen modelling approach is to adapt and apply the PROTOS-MATINE methodology, characterized by its ability to synthesize data collected from diverse sources, and it has previously proven effective when used to evaluate targets such as antivirus software and voice-over-IP systems. We are thus pursuing two objectives in parallel: to develop a new version of the methodology, and to use it to analyse the EMCANP case. We made progress toward both objectives in 2013.

Internet of Things studies security and privacy issues in large-scale sensor networks. Topics of interest are alternative ways of authentication, such as proof of work and cryptocurrencies, secure update mechanisms, software defined networking and related service-level agreements for data centres. The work brings together our research themes of Quality, Complexity and Awareness to an application area where resource limits are combined with global connectivity.

The Electric Vehicle ecosystem studies security and privacy issues involved in future infrastructures for charging and testing electric vehicles in northern environments, as well as associated end-user services. The work aims to bring our research themes into new application areas, where connectivity to a global infrastructure is applied to traditionally isolated systems.


Exploitation of Results

The results of our research were applied to real-world problems in many projects, often in collaboration with industrial and other partners. Efficient exploitation of results is one of the core objectives of the national Tivit ICT SHOK projects like Cloud Software, D2I and IoT; in these projects we work in close collaboration with companies throughout the projects.

The Intelligent Systems Group utilizes a robotics laboratory and a pressure-sensitive floor (EMFi material) installed in our laboratory as part of a smart living room. Other equipment includes a home theatre, two degree-of-freedom active cameras, four mobile robots and one manipulator, a WLAN network, and various mobile devices. Our aim is to gradually build a versatile infrastructure that offers various generic services for pervasive applications. Naturally, this kind of environment enables realistic experiments that lead to a better understanding of such applications.

We are also building an immersive space into our laboratory. This affordable cave is built from six white screens covering 360 degrees and six projectors, one for each white screen. We are currently adapting the realXtend 3D virtual environment platform for this setting.

During the reporting year, the group continued utilizing outdoor robotic systems. Development and utilization of Mörri, a multipurpose, high performance robot platform continued. More focus was put on perception in natural conditions, representation of detections, knowledge, and an environment model of the operating environment. The software architecture further developed the earlier work on Property Service Architecture, and the Marker concept as general purpose representation was further developed.


Future Goals

The partnership in the SIMP programme that belongs to the SHOK concept of Tekes enables us to continue our steel research into new areas. The new goals are in quality prediction at different process stages and for more challenging properties. As a result more advanced expert systems can be developed to aid the operators with different roles in steel making.

We will continue to strengthen our long term research and researcher training. We will also continuously seek opportunities for the exploitation of our research results by collaborating with partners from industry and other research institutions on national and international research programs and projects. The University of Oulu is a founding member of euRobotics. Juha Röning is a member of the Board of Directors of euRobotics.

We will strengthen our international research co-operation. Over recent years, the group has created collaborative projects with Japanese researchers. Two doctoral students have funding for one-year visits to Japanese universities. We will continue the collaboration with Sonic Studio from Sweden and the Shanghai Jiao-Tong University from China. With the University of Tianjin in China, we have a joint project in which methods and a system will be developed for vision-based navigation of Autonomous Ground Vehicles, which utilize an omni-directional camera system as the vision sensor. The aim is to provide a robust platform that can be utilized in both indoor and outdoor AGV (Autonomous Ground Vehicles) applications. This co-operation will continue.

In the USA, we will continue to co-operate with the Human-Computer Interaction Institute in Carnegie Mellon University with Assistant Professor Anind K. Dey. The research is on human modelling in the area of human-machine interaction. We continue and strengthen US-Finland co-operation through an NSF grants. With Virginia Polytechnic Institute and State University we have our collaborative research project on Techno Economic Models for Collaborative Access Network Provisioning. In 2013, we started another US-Finland project: NSF EAGER: Global RF Spectrum Opportunity Assessment.

Shorter research visits to European partners in EU-funded projects are also planned.

In 2014, the aim is to utilize more widely the know-how from sensor technology and data mining. New application areas will be studied, including rehabilitation, exercise motivation and energy efficiency in households, and the benefits of our expertise will be highlighted to actors in the areas.

In human-environment interaction and sensor networks, our research will continue mainly in Tekes and ICT SHOK projects. Our main goals are to develop analysis methods for sensor network data and to develop applications utilizing physical user interfaces. Research on novel software architectures, reasoning and knowledge representations will continue as well. Field trials in realistic settings, and close collaboration with research groups (national and international) and companies will be emphasized.


 

Personnel

professors, doctors

9

doctoral students

24

others

17

total

50

person years

33


 

External Funding

Source

EUR

Academy of Finland

151 000

Ministry of Education and Culture

73 000

Tekes

2 060 000

other public

50 000

domestic private

13 000

international

160 000

total

2 507 000


 

Doctoral Theses

Schaberreiter T (2013) A Bayesian network based online risk prediction framework for interdependent critical infrastructures. Acta Universitatis Ouluensis. Technica C 466.


Selected Publications

Ahola, R, Pyky R, Jämsä, T., Mäntysaari, M, Koskimäki, H,, Ikäheimo T, Huotari, M-L, Röning J, Heikkinen H & Korpelainen R (2013) Gamified physical activation of young men – a multidisciplinary population-based randomized controlled trial (MOPO study), BMC Public Health 2013, 13:32.

Gilman E, Davidyuk O, Su X & Riekki J (2013) Towards interactive smart spaces. Journal of Ambient Intelligence and Smart Envi-ronments. IOS Press. Vol. 5, Issue 1, 5-22.

Riekki J, Cortés M, Hytönen M, Sánchez I & Korkeamäki RL (2013) Touching nametags with NFC phones: A Playful Approach to Learning to Read. In: Pan Z et al. (Eds.): Transactions on Edutainment X, LNCS 7775. Springer-Verlag, 228-242.

Pyykkönen M, Riekki J, Jurmu M, Sanchez I (2013) Activity Pad: Teaching tool combining tangible interaction and affordance of paper. ACM International Conference on Interactive Tabletops and Surfaces (ITS’13), St Andrews, United Kingdom. 135-144.

Polojärvi M, Saloranta T, Riekki J (2013) Navigating by audio-based probing and fuzzy routing. International Conference on Making Sense of Converging Media, Tampere, Finland. 87-93.

Jurmu M, Ogawa M, Boring S, Riekki J, Tokuda H (2013) Waving to a touch interface: descriptive field study of a multipurpose multimodal public display. The International Symposium on Pervasive Displays (PerDis’13), Mountain View, California.

Kauppinen M & Röning J (2013) Control issues and recent solutions for voltage controlled piezoelectric elements utilizing artificial neural networks. Proc. SPIE Intelligent Robots and Computer Vision XXX: Algorithms and Techniques, 8662: 866202:1-13.

Kagatsume S, Murata, S, Fujinami K, Alasalmi T, Suutala J, & Röning J (2013) PerFridge: An augmented refrigerator with the awareness of wasteful usage of electricity, UbiComp’13 Adjunct, Zurich, Switzerland.

Koskimäki H, Huikari V, Siirtola P & Röning J (2013): Behaviour modelling in industrial assembly lines using a wrist-worn inertial measurement unit, Journal of Ambient Intelligence and Humanized Computing: Volume 4, Issue 2 (2013), 187-194.

Röning J & Casasent DP (eds.) (2013) Proc. SPIE Intelligent Robots and Computer Vision XXX: Algorithms and Techniques 8662.
Schaberreiter T, Varrette S, Bouvry P, Röning J & Khadraoui D (2013) Dependency analysis for critical infrastructure security modelling: A case study within the Grid’5000 Project, The 8th ARES Conference (ARES 2013).

Siirtola P & Röning J (2013): Ready-to-use activity recognition for smartphones. IEEE Symposium on Computational Intelligence and Data Mining (CIDM 2013), 16-19 April 2013, 59-64.
Tamminen S, Juutilainen I & Röning J (2013) Exceedance probability estimation for a quality test consisting of multiple measurements. Expert Systems with Applications, September, 40(11): 4577-4584.

Tamminen S, Tiensuu H, Juutilainen I & Röning J (2013) Steel property and process models for quality control and optimization. Materials Science Forum, Trans Tech Publications, June, 762:301-306.

Vallivaara I, Kemppainen A, Poikselkä K, Röning J (2013) Monty Hall Particle Filter – A new method to tackle predictive model uncertainties. IEEE International Conference on Advanced Robotics, Montevideo, Uruguay.
 

Last updated: 26.2.2015