Infotech Oulu Annual Report 2016 - Biomimetics and Intelligent Systems Group (BISG)

Professor Juha Röning and Dr. Heli Koskimäki, Biomimetics and Intelligent Systems Research Unit, Faculty of Information Technology and Electrical Engineering , and Professor Seppo Vainio, Oulu Center for Cell-Matrix Research, Faculty of Biochemistry and Molecular Medicine, University of Oulu
juha.roning(at)oulu.fi, heli.koskimaki(at)oulu.fi, seppo.vainio(at)oulu.fi
http://www.oulu.fi/bisg

Background and Mission

Biomimetics and Intelligent Systems Group (BISG) is a fusion of expertise from the fields of computer science and biology. In BISG, our basis are intelligent systems and our research areas include data mining, machine learning, robotics, and information security. More precise research topics vary from data mining algorithm development and optimization of industrial manufacturing processes all the way to environmental monitoring with mobile robots.

Bringing expertise from ICT and Biotech together, we will reach the skills to make use of the mechanisms common in information processing and the biological data processing system and extrapolate this to intelligent solution making in ICT. One important goal of this program is to be able to physically link living cells via identified signaling systems to establish learning complex that involves Bio and ICT in a unified bifunctional interactive machine.

The group consists of four sub-groups: Data Analysis and Inference Group, Organ BioEngineering Biology, Robotics and Secure Programming

We have conducted basic research in intelligent systems and tissue engineering for over ten years as individual groups. Now we have joint our efforts. Our team consists of 2 professors, 10 post-doctoral researchers and 15 doctoral students. The annual external funding of the group is more than two million Euros, in addition to our basic university funding. In the reported year, there have been 23 completed doctoral degree from the group. From the research of the group, 11 spin-off companies have been established so far: Codenomicon, Clarified Networks, Hearth Signal, Nose Laboratory, Nelilab, Atomia, Indalgo Probot, Aquamarine Robots, Radai and IndoorAtlas.

We co-operate with many international and domestic partners. In applied research, we are active in European projects. In addition, several joint projects are funded by the Finnish Funding Agency for Technology and Innovation (Tekes) and industry. We were a research partner in the SIMP and CyberTrust SHOKs. Prof. Juha Röning was selected as ACO (Academic coordinator) of the Cyber Trust program.

We are active in the scientific community. For example, Prof. Juha Röning is acting as visiting professor of Tianjin University of Technology and as the Robot Science Adviser of Tianjin Science and Technology Center for Juveniles. He served as a member of the Board of Directors in euRobotics and as a member of the SAFECode International Board of Advisors. He chaired the euRathlon / TRADR Summer School 2016 in Oulu, Finland, 22nd to 26th of August. It was a five-day course to provide participants with a full overview and hands-on experience with multi-domain real robotic systems. He also chaired with prof. Othmane the First International Workshop on Agile Development of Secure Software (ASSD’16) in Salzburg 1st of September. With robotics group, he participated NORDRUM project where aim was to collect radiation data from the environment using an unmanned aerial vehicle (UAV). The testing area was located in Norway, Hauerseter Leir military campsite (5th to 7th of September).

During the reporting year, the group organized the 9th International Crisis Management Workshop and Winter School (CrIM’16) and NordSec conference which brought together both Finnish and international information security experts. The group also organized Summer School 2016 of Data mining, big data and open data together with Exactus DP and Aurora DP 15th to 19th of August.

Representing Finland as a Partner for Peace (PfP) na-tion, BISG / Prof. Röning participated the Specialists’ Meeting on Intelligence and Autonomy in Robotics, held in Wachtberg, Bonn, Germany on 25 – 27 October 2016.

Celentano and Röning are co-editors, together with collaborating partners, of an IEEE Access special section on Recent Advances in Socially-aware Mobile Networking.

Prof. Seppo Vainio has been the chair in the Minisymposium ”Omics in Biomedicine” (2016). He is also part of a European nanotechnology ”HyNanoDend” network.

Scientific Progress

Intelligent Systems Incorporating Security

Within the Biomimetics and Intelligent Systems Group, the Oulu University Secure Programming Group (OUSPG) has continued research on security and safety in intelligent systems. Security and safety challenges in intelligent systems are threefold: increasing complexity leads to unforeseeable failure modes, quality is not the priority and awareness is lacking. We have approached the challenges from these three directions in our research.

Complexity - Model Inference and Pattern Recognition: we work under the premises of unmanageable growth in software and system complexity and emergent behaviour (unanticipated, not designed) having a major role in any modern non-trivial system. We have worked on natural science approaches to understanding artificial information processing systems. We have developed and applied model inference and pattern recognition to both content and causality of signalling between different parts of systems.

Quality - Building Security In: software quality problems, wide impact vulnerabilities, phishing, botnets, and criminal enterprise have proven that software and system security is not just an add-on, despite the past focus of the security industry. Instead, security, trust, dependability and privacy have to be considered over the whole life-cycle of the system and software development, from requirements all the way to operations and maintenance. This is furthermore emphasized by the fact that large intelligent systems are emergent and do not follow a traditional development life-cycle. Building security in not only makes us safer and secure, but also improves overall system quality and development efficiency. Security and safety are transformed from inhibitors to enablers. We have developed and applied black-box testing methods to set quantitative robustness criteria. International recognition of the Secure Development Life Cycle has provided us with a way to map our research on different security issues.

Awareness - Vulnerability Life Cycle: Intelligent systems are born with security flaws and vulnerabilities, new ones are introduced, old ones are eliminated. Any deployment of system components comes in generations that have different sets of vulnerabilities. Technical, social, political and economic factors all affect this process. We have developed and applied processes for handling the vulnerability life-cycle. This work has been adopted in critical infrastructure protection. Awareness of vulnerabilities and the processes to handle them all increase the survivability of emergent intelligent systems for developers, users and society.

These research goals are reached through a number of research activities.

Secure Software Development Lifecycle as a part of the Cyber Trust project - we approach all three goals by researching practical ways of building security into Secure Platforms, Cloud Computing services and Critical Infrastructure, from the design phase to actual operational use (Figure 1).

Figure 1. Dependencies of a single cloud based web service visualized by technology and location.

Situational Awareness in Information and Cyber Security aims to understand critical environments and accurately predict and respond to potential problems that might occur. Networked systems and networks have vulnerabilities that present significant risks to both individual organizations and critical infrastructure. By anticipating what might happen to these systems, leaders can develop effective countermeasures to protect their assets (Figure 2).

Figure 2. Port scanning visualized in an industrial automation network.

Coverage based robustness testing: Modern web browsers are feature rich software applications available for different platforms ranging from home computers to mobile phones and modern TVs. Because of this variety, the security testing of web browsers is a diverse field of research. Previously, we have found a number of bugs in browsers, but previous methods were seeing diminishing returns. By utilizing code coverage we were able to improve on existing state of the art.  This work introduces a cross-platform testing harness for robustness testing, called CovFuzz. In the design of CovFuzz, test case generators and instrumentation are separated from the core into separate modules. This allows the user to implement feature specific test case generators and platform specific instrumentations, and to execute those in different combinations.

Identification of a protocol gene: This research, PROTOS-GENOME, approaches the problems of complexity and quality by developing tools and techniques for reverse-engineering, and identification of protocols based on using protocol genes - the basic building blocks of protocols. The approach is to use techniques developed for bioinformatics and artificial intelligence. Samples of protocols and file formats are used to infer structure from the data. This structural information can then be used to effectively create large numbers of test cases for this protocol.

OUSPG Open: This activity brought together over 60 people from 23 academic and commercial organizations by organizing regular events throughout the summer. Five novel security tools were produced as collaborative projects. For example, TryTLS, a tool for the software and library developers, vulnerability researchers, and end-users, who want to use TLS safely, resulted in a number of issues being found in commonly used programming libraries.

Privacy and Security and Online Social Networks: Exploiting Social Structure for Cooperative Mobile Networking (SOCRATE), a two -year (2015-2016) Tekes funded project under the Wireless Innovation between Finland and U.S. programme WiFiUS [http://wifius.org/], is a collaboration between the University of Oulu (co-PI Dr Ulrico Celentano), VTT, Aalto University, Arizona State University, and University of Nevada, supported by NSF funding on the US side. BISG contribution focused on privacy and security issues in online social networks data mining and the related architecture aspects and on privacy and security issues.

The final goal of SOCRATE project is to exploit for optimised radio network operation the knowledge about the social structure of network users. Clearly, this potentially exposes users to disclosure of sensitive information. The disclosure of personal information, if not anonymised, exposes also to additional threats such as identity theft or even physical security or denial of service or sabotage. Similar privacy and security questions are found in other applications, such as those enabled by the Internet of Things (IoT) paradigm. In this extension (Figure 3, top), we may use the term user to mean a person or an entity possibly related to a person, and the word social is used to refer to a person or equivalently in more abstract terms to the contextual relationship of devices.

IoT devices are increasingly permeating the human environment. Connected sensors and/or actuators are found for example in smart environments, cars and wearables, in both industrial and nonindustrial scenarios. Whereas they serve a range of applications and services, virtually any IoT device has access to sensing data about humans or it acts on the environment humans live within and in particular on appliances and services humans rely upon. Clearly, information security, hence privacy, and physical security, hence safety, are therefore unavoidable themes in such people-centric IoT. More generally, security enforcement is a fundamental enabler of the success of IoT and people-centric IoT in particular.

Figure 3. A people-centric ICT scenario, above, and the conceptual framework supporting it, below. From Celentano et al. (manuscript).

BISG research in this area follows data minimisation principles, where 1) occasions for collection of sensitive data, 2) the extent of data collection, and 3) the time duration of data storage are minimised. The above, and the second point in particular, have direct impact on architectural choices, see Figure 3.

At the top of the figure are depicted on the left the people-centric IoT service and on the right the wireless network supporting it. The objective is to infer observations of features at one domain and exploit them at the other domain, through the enhancement protocols shown in yellow at the bottom of Figure 3. By data minimisation principles, attributes are stored at various parts of the system to mitigate the threats of possible attackers. Separated interfaces towards individual sources guarantee flexibility of the design. Attackers may target either side of the system (Figure 3, top).

Different scenarios enabled by the knowledge of social structure and users’ features are compared in Höyhtyä et al. (2016). Relationship among users and users’ preferences can be used to identify alternative data distribution strategies, for example relying on clusters. These strategies are then associated with corresponding transmit power requirements. Options include changes in the topology (direct data transfer from base station BS to user equipment UE; or from BS to cluster-head CH and then from CH to UEs; or having chunks of data sent from BS to UEs, and then sharing them among UEs). Considering the availability of various radio access techniques (RAT), the above options can be combined with changes in the RAT for communication between CH and UEs and among UEs (LTE; WiFi). The most power-efficient scheme depends also on the required data rate, and in turn the data rate impacts on transmission time and hence on energy efficiency.

Related to the above research and in the framework of SOCRATE project co-operation, Celentano and Röning are co-editors, together with collaborating partners, of an IEEE Access special section on Recent Advances in Socially-aware Mobile Networking.

Intelligent Systems Incorporating Machine Learning and Data Mining

Data mining methods for steel industry applications: BISG is a member of the Centre for Advanced Steels Research - CASR, which is one of the interdisciplinary umbrella organizations of the University of Oulu. Year 2016 was the third year in participation to a large national research programme System Integrated Metals Processing – SIMP.

One of the main goals in SIMP programme has been the development of an innovative supervisor system to assist the process development personnel and the operators of a steel production line over the whole production chain, and to help discover new alternative solutions for improving both the products and the manufacturing process. The quality monitoring tool (QMT) is based on statistical models that predict different quality properties and rejection risks in several process steps, and it provides also model visualization (Figure 4). The tool has been developed in co-operation with VTT, In year 2016, QMT was delivered for online use at Outokumpu, Tornio and for offline use at SSAB, Raahe. The development work continues, and the functionality of the tool can be improved with the feedback of the test period.

Figure 4. The tool for quality monitoring and visualization during the steel making process.

One of the models in QMT predicts the rougness of the stainless steel surface with hot rolling parameters. In previous case, the roughness was visually evaluated to 9 classes by experts in Germany. The goal was to predict the quality after hot rolling, before the product was shipped to Germany for further processing. In 2016, a new data was gathered with roughness measured with a device. Although, the roughness type was in this case different, the research revealed that also the finishing step has a significant effect on the surface quality. The new models will be implemented into QMT next year.

In the year 2016 one goal of the SIMP project was reached, as the selection of combination parameters for slab design were updated and improved with statistical models. The aim of the combination is to ensure the sufficiency of the material to produce the ordered product with desired dimensions. The variance modelling increased the knowledge of process deviation and the factors behind it. As a result, the new selection procedure is expected to increase yield, reduce the risk of rejection, energy consumption and emissions, which in its turn improves the cost-effectiveness of the steel mill. This study showed that using specific statistical modelling methods and classification, the knowledge behind the steel process can be pointed out and utilized in manufacturing. Powerful data mining methods enable the effective use of process data in slab design. This research is brought out vital problems in production line and there has been a lot of development work done also at SSAB, Raahe due to this study.  The results of this research were published in journal of Ironmaking & Steelmaking on August 2016.

In the steel plate production process it is important to minimize the wastage piece produced when cutting a mother steel plate to the size ordered by a customer. The uneven shapes at the plate end sides and lateral sides cause yield loss, amounting to about 5% to 6% of a total tonnage of slab used. To minimize the loss, aim is to produce plates with concave side edges because wastage from concave side edges is smaller than from convex. We developed a method for automatic recognition of steel plate edge shape with classification and regression models. First, we defined the curvature of a time series describing the steel plate side edge, and used this information to build statistical distribution model to visualize what kind of curve shapes the studied data set includes and how the amount of curvature is distributed in the manufacturing process. This information can then be used to optimize manufacturing parameters to manufacture more plates with desired shape. Data for the study was collected from the steel plate mill at SSAB, Raahe.

A tool for finding clusters of inclusions in SEM (Scanning Electron Microscope) specimens of steel samples was developed. The inclusion clusters may have a significant effect on mechanical properties of the steel; especially novel ultra-high strength steels require very high steel purity. This tool enables quick and efficient inspection of the specimens. The summaries of the clusters are produced, as well as, visualizations of the whole test area or interesting parts of it, The visual presentation of the chemical composition of the clusters helps to understand the birth mechanism of the formation, and thus, to find a way to prevent it in steel products. An example of a test sample from finished product can be seen in Figure 5. One cluster has been selected for a closer inspection, and it can be seen that in this case there are only MnS particles in quite large inclusion formation. The shape of the cluster is unusual, but the direction of the test sample may explain it. The research was carried out in the co-operation with SSAB, Raahe.

Figure 5. An example of a clustered SEM specimen with zoomed in visualizations and the chemical composition of the selected cluster,

SIMP programme will continue for another 6 months, and we will present our research results annually on SIMP and DIMECC seminars around Finland as well as at international publishing venues.  

Uncertainty of classification models. Many real-world data sets contain missing data values. These might be the result of e.g. malfunctioning sensors or some measurements being too expensive to measure from every sample etc. Having missing values when classifying a sample means that there is an increase in uncertainty in the final classification result. Knowing how uncertain the result can sometimes be as important information as the classification result itself. In an Infotech doctoral program project, we are quantifying that uncertainty so that interpreting the classification results becomes easier.

Classification algorithms have traditionally been developed using complete data sets and most require values for all variables to be present to work. Many real world data sets are, however, cursed with missing data. To tackle this problem, we developed an algorithm that uses multiple imputation to handle the missing values. The algorithm can be used with any classifier that supports estimation of class posterior probabilities. The developed algorithm performs as well or even better as a benchmark algorithm (see Figure 6) and it does not require the classifier to support handling of missing values.

The uncertainty does not, however, behave consistently across different data sets. In a follow-up work we addressed this issue and the results of this work were presented in a conference, see Figure 7.

Figure 6. Modelling the uncertainty as a function of classification rate.

Figure 7. Prediction of test sample uncertainty based on the uncertainty model from Figure 4.

To generalize these results, the classification confidence algorithm was modified to a calibration algorithm. In practice this means that the prediction scores of a classification algorithm are attempted to get more closely to resemble true posterior probabilities. Still unpublished example of the performance this calibration algorithm compared to two state-of-the-art can be seen in Figure 8.

Figure 8. Calibration plots from raw prediction scores, two commonly used calibration algorithms and our novel algorithm. Unpublished results.

Data mining methods for data of wearable sensors: BISG has a long experience is studying data from wearable sensors. Earlier the study has concentrated on human activity recognition based on accelerometer data from a wrist-worn device or mobile phone. This year BISG extended the application area from human activity recognition to early diagnosis of diseases. In addition, we have used variety of sensors including electromyogram, thermometer, electrodermal activity sensor and photoplethysmography sensor.

Human activity recognition: The activity recognition approaches can be used for entertainment, to give people information about their own behavior, and to monitor and supervise people through their actions. Thus, it is a natural consequence of that fact that the amount of wearable sensors based studies has increased as well, and new applications of activity recognition are being invented in the process. In 2016 BISG concentrated on studying human activity recognition in two scenarios: adaptive models and comparing electromyogram data to accelerometer data.

Usually human activity recognition is based on user-independent models. However, as people are different these models do not work equally well with every subjects. Therefore, in order to obtain high recognition rates with all users, models need to adapt to each user’s personal movements. In our study, it was shown that personal models can be trained without a separate data gathering session if wearable device has several types of sensors. On the other hand, it was shown that personal models always do not provide as high recognition accuracies as reported in the literature. The reason for this is that models are not general enough and therefore they cannot react to changing conditions.  In fact, in order to build reliable user-dependent recognition model, a lot of personal data needs to be collected. This requires an extensive, separate data collection session for each user. If the aim is to build a commercial application for the masses, this is far from ideal situation. However, BISG introduced an noise injection based method to expand the area covered by training data, and in this way, make the models trained using it more general and less vulnerable to changing conditions. This is shown to improve the recognition rates, especially if the amount of the training data is small. An another approach to tackle the same problem a solution combining the human independent and personal models more effectively using self-organizing maps based distance as a selection criteria was introduced. By using the approach, the selection can be done in real time and within wearable device itself. The results show that the approach clearly outperforms posterior probability based approach in preserving the high recognition accuracy regardless of which model is used.

BISG also studied the possibility to use electromyogram data to recognize human activities. The actual research problem tackled is one of the major drawbacks in activity recognition, namely to add completely new activities in real life to the recognition models. In this study, it was shown that in gym settings electromyogram signals clearly outperforms the accelerometer data in recognition of completely new sets of gym movements from streaming data even though the sensors would not be positioned directly to the muscles trained. However, it was noted than when the task is to recognize previously known activities, accelerometer data outperforms electromyogram signals.

Early diagnosis: The progress in the sensor development including improved memory and battery properties has made possible to measure human physiology 24/7, and more importantly with such accurate readings that have previously been possible only in laboratory settings. BISG is aiming to use a single, easy and comfortable device to measure 24/7 data from persons bio-signals and based on these recognize upcoming seizures. For this purpose BISG got a six month funding from TEKES from Challenge Finland program. Though the funding period was short, promising results were obtained. The first use case was based on migraine. The migraine is a chronic, incapacitating neurovascular disorder, characterized by attacks of severe headache and autonomic nervous system dysfunction. Moreover, migraine headache is usually associated with nausea, vomiting, or sensitivity to light, sound, or movement and when untreated, typically lasts 4 to 72 hours. Typical migraine attack consist of five phases: prodrome (e.g., food craving), aura (e.g., visual, sensory, or motorsymptoms preceding the headache), headache (usually unilateral, pulsating), resolution (pain wanes), and recovery. The illustration of these phases are shown as Figure 9. The migraine condition starts in over 40% of the cases before 18 years of age, thus making it also a childhood disease. The medication of migraine can be divided into two categories: preventive (daily dose) and acute medication (when symptoms start). The preventive medication is an expensive solution and especially with children, it is avoided as long as possible. On the other hand, the problem with acute medication is that some people do not have early symptoms, some tend to easily dismiss the early symptoms, sometimes even on purpose. Thus the early diagnosis of migraine attack would be a valuable addition to treatment of the disease. The research concept was aimed to early detection of migraine attach based on human bio-signals collected with wearable sensors is presented. In our approach, the idea is to use a single, easy and comfortable device to measure 24/7 data. Results from this project will be published in 2017. Moreover, BISG is actively seeking new funding possibilities and instruments to continue in studying the early diagnosis also in 2017.

Figure 9. Phases of typical migraine attack.

Developing software for data mining in public health: The multidisciplinary MOPO project combined traditional health promotion, modern technology and measurement of physical activity. Altogether about 6000 conscription aged men (five call-up age classes) were invited to participate in the study, where the participants’ physical condition, well-being, health, relationship towards physical activity, information behaviour and use of media and technology were investigated during the years 2009–2014 using questionnaires, measurements and interviews. In addition to this, a novel wellness coaching service for preventing marginalization and promoting physical activity and health in young men was developed in the project. The service, which took the form of a gamified Web portal optimized for mobile devices (Figure 10), was developed by BISG personnel and incorporated functionality for collecting, storing and analysing physical activity data and presenting the results of the analysis to the user as personalized feedback.

Figure 10. The Web portal offers tailored information about topics such as physical activity, fitness, health, and nutrition.

The Web portal went through a number of iterations that were evaluated in a series of intervention studies, culminating in an intervention where access to the portal and a wrist-worn activity monitor was given to approximately 250 conscription-aged men (Figure 11). The intervention started in autumn 2013 and lasted 6 months. Following this final intervention, the lessons learned over the course of the MOPO study were generalised into a set of design principles intended to be applied by medical researchers and software developers implementing digital interventions for health behaviour change. A paper on the proposed design principles was published in 2016. Also in 2016, BISG researchers contributed to an analysis of MOPO data comparing self-reported versus measured physical activity and sedentary behaviour.

The operators of the MOPO study were the Oulu Deaconess Institute’s Department of Sports and Exercise Medicine, the University of Oulu, the City of Oulu, the Virpiniemi Sports Institute, the Finnish Defence Forces and several wellness technology companies in Northern Finland. The project website can be found at www.tuunaamopo.fi.

Figure 11. In autumn 2013, 250 conscription aged men were recruited from call-ups to test the wellness coaching service developed in the MOPO project.

Foundations of knowledge discovery and data mining:

Knowledge discovery in data (KDD) was defined in 1996 by Fayyad et al. as “the nontrivial process of identifying valid, novel, potentially useful, and ultimately understandable patterns in data”. Although this definition still has its merits, it represents a rather narrow interpretation of the concept of knowledge that may prove a hindrance to the development of more advanced KDD tools. Meanwhile, the seminal process model proposed by Fayyad et al., which depicts the KDD process as a sequence of five major steps, is still embedded in most KDD process models, including the standard model CRISP-DM. This established model, while essentially correct, represents a limited perspective on the KDD process that is likely to prove inadequate in the long run.

Figure 12. The actors of the KDD process can be illustrated as the vertices of a triangle, with technology in the center, being both an actor in its own right and a mediator of interactions among human actors (a). The process can take on a number of different forms, characterized by which of the actors are present and how they interact: the standard KDD process (b), KDD using personal data (c), KDD using volunteer computing (d), and KDD driven by a non-expert actor (e). A good example of the latter is the so-called Quantified Self movement.

In its research on the foundations of KDD and data mining, BISG has sought to expand this traditional view of the nature of KDD. The resulting model, like the established one, accounts for the data transformations required in order to get from raw data to knowledge, but also for the actors of the process and the interactions among them that need to take place for the process to move forward. Furthermore, the model explicitly considers the contributions of non-expert actors, as well as the possibility of technology taking on a more autonomous role in the process, which is likely to be realized in the near future as KDD software grows more intelligent and becomes capable of handling tasks that currently require a human actor. Having a model that provides a more complete account of the KDD process is essential in unlocking the full potential of KDD technology, which in turn is crucial in making sense of the deluge of digital data that seems to have become a permanent feature of high-technology societies. Figure 12 illustrates the process actors and how different interactions among them lead to different types of KDD processes.

Intelligent Systems Incorporating Robotics and Cybernetics

euRathlon Summer School

The ERL Emergency/TRADR summer school 2016 was organized from the 22nd to 26th of August by the robotics group members and the staff from TRADR (Long-Term Human-Robot Teaming for Disaster Response). The summer school was attended by 55 students, mostly doctoral, originating from 17 different countries (Figure 13). Also, 6 invited lecturers from TRADR held lectures during the summer school. According to the satisfaction survey, the participants were very pleased with the summer school as all of those who answered the survey would recommend it to others.

Figure 13. Attendees and organizers in the ERL Emergency/TRADR summer school 2016 held at the University of Oulu.

This year, the ERL Emergency summer school focused on developing algorithm for controlling land robots with a strong focus on SLAM and multi-source persistent data integration.

 In total, the summer school lasted for four and a half days consisting roughly 35% of lectures and 65% of practical exercises in which the students developed control and SLAM algorithms. These practical sessions were held indoors in the University of Oulu’s facilities and outdoors in a nearby (less than 1km of walking) botanical garden where electricity, shelter and internet access were also provided. The hands-on practices culminated to a challenge scenario that each team performed on the last day at the botanical garden.

The students were provided with two of TRADR’s UGVs and one UAV (Figure 14) provided by Ascending Technologies. The students could modify and develop software for the two UGVs but the UAV was flown only by the trained representative of Ascending Technologies. Before the summer school, these UGVs and UAV were also used to gather preliminary data for software development during the exercises. The raw data was used to form initial maps that were given to the students to work on for testing and performing simulations. This was done to reserve the students’ time for more meaningful tasks as the generation of maps from raw data takes many hours of processing on a desktop computer.

During the registration process the students were asked to provide a brief description of their programming experience. This info was used on the first day to form eight balanced teams of six or seven persons. The balancing was done mainly in regard of C, Python and ROS experience as at least one person in each group had to have at least basic understanding of these to ensure that the practices would proceed in a timely fashion.

Figure 14. TOP; One of the two identical unmanned ground robots (UGV) provided by TRADR. BOTTOM; The UAV, AscTec Falcon 8, provided by TRADR partner Ascending Technologies.

On the first day the student teams were presented with a challenge scenario they would perform and compete on the fifth, and last, day of the summer school. The scenario consisted of a simulated toxin leak at the botanical garden and the teams’ task would be to find and localize the toxic materials using the UGV, UAV and the pre-recorded maps of the area.

To fulfil this task, the teams needed to fulfil the sub-tasks:

  • Map the area using one UGV and one UAV.
  • Update and refine the map based on new data.
  • Develop strategies to safely navigate the UGVs in the danger zone.
  • Navigate the UGV to designated points of interest with the highest degree of autonomy possible.
  • Detect objects automatically if possible.
  • Have the UGV perform automatic collision avoidance if possible.

The practical sessions focused on developing ways to fulfil these tasks and mapping the scenario area (see Figures 15-18).

Figure 15. The used waypoint planner and simulator.

Figure 16. Outdoor testing at the botanical garden.

The challenge scenario was run on the final day of the summer school. The toxic leaks were simulated with bright balloons and rough estimates of their locations were given to the teams. Compared to previous days, the scenario environment was somewhat changed by added obstacles (chairs, tables, etc.). Each team had 30 minutes of time to complete the mission, during which they had full access to the UGV. The teams also had a limited 5 minutes access to the UAV, flown by the trained operator, to get a rough overview of the environment.

Figure 17. Scenario briefing and composed map of the area.

The students were also given a demonstration of the Aquamarine Robots Dolphin marine robot.

Figure 18. Aquamarine Robots Dolphin robot shown during the summer school.

 

Robotics Research

In 2016 BISG had a wide range of research in the area robotics, including industrial safety, aerial data gathering, battery life management and control of complex wheeled land robots.

ReBorn

The EU funded ReBorn project has ended. The project had participants from 17 industrial and academic institutions from 10 different countries. During the project, the sufficiency of current standards related to robot development and reusability in industrial environments was investigated by a paper review and by surveys sent to the project partners. The current standards (Figure 19) for designing user safe robots were deemed sufficient for fulfilling the requirements to implement safe robots for traditional industrial applications. However, in some applications shortcomings in the currently available standards were found.

Figure 19. Main standards applicable in implementing user safety in industrial robotics.

One of the commonly mentioned issues was that there is a lack of standardized commonly applicable performance descriptions for the existing line of robots. This especially hinders the flexible use of heterogeneous and modular robotics. Also, reuse and repurposing old robots for new applications is more difficult without a common form of performance descriptions and capabilities of the robots. It was also found out there currently are no dedicated ISO/EN/DIN standards specifically related to safety and design assisting in implementation of reconfigurable manufacturing cells. Other area that was mentioned in the survey responses from the project partners was the lack of LCC (Life Cycle Cost) standards, similar to what are already in use in the building construction industry.

One area of interest in the project was the requirements for implementing CWS (Collaborative Work Spaces). In this field, some standards already exists (Figure 20) that can be utilized to implement the minimum safety features required to avoid serious injuries. However, from the applications side, the standards are currently vague on how the software should be implemented and how the user should be taken in to account as an agent acting in the control loop when performing tasks co-operatively with a robot. The actions of the human in the loop needs quite a lot of prediction and behavior observation to implement co-operation efficiently and safely in flexible manufacturing units. This is an area that is currently under a lot of research and appropriate standards should also be developed on how the user monitoring and behavior prediction should be performed on hardware and algorithm level.

Cloud computing is an area that could be better utilized in industrial environments as a channel for data processing, learning and teaching of industrial robots in the future. Cloud computing could also be used for openly collecting and sharing data about robot reliability for evaluation of the reusability, safety and costs of running specific types of robots in certain tasks.

Figure 20. Standards applicable for collaborative work spaces.

 

NORDUM Exercise

NORDUM (Intercomparison of Nordic unmanned aerial monitoring platforms) exercise was organized in the Hauerseter Leir military campsite, Gardermoen, Norway. The NKS-B activity NORDUM is the first joint Nordic exercise for unmanned systems. All in all, five teams participated in this event coming from different universities and radiation safety related institutions located in Norway, Sweden and Finland.

In the NORDUM exercise, the objective was to locate and identify potential radioactive materials from the arranged scenario areas. The scenario areas varied from cluttered areas containing large shipping containers and various metal structures to open field and forest scenarios. For the measurements, a stand-alone sensor package was constructed containing a RTK (Real Time Kinematic) capable GPS (u-blox C94-M8P-3), measurement computer (Raspberry Pi 3 Model B), a 433 MHz (3DR) radio, a 3.7V Li-ion battery and a gamma radiation spectrometer (Kromek GR1-A).

In the arranged scenarios, the teams needed to localize hidden radiation sources and visualize their location utilizing GPS. In the scenarios, the constructed stand-alone sensor package performed well in most scenarios, although some radio link related issues were encountered especially near large metal structures. The utilized Kromek GR1-A was sensitive enough that radiation sources could also be identified from local spectrum histograms when enough flybys near the radiation source was made.

The stand-alone sensor package was carried with a DJI Inspire 1 T600 quadcopter, hanging 1.5 meters below the vehicle. This made possible manoeuvring the sensor very close to the objects being measured. The quadcopter had a flight time of 10 minutes with a 5.7 Ah 22.2V Li-ion flight battery and the sensor package. The 4k resolution camera was utilized during flight to observe the sensor position and to manoeuver it to wanted positions. Although the conditions were rather windy, flying with the sensor package was manageable. The quadcopter with the sensor package is shown in Figure 21. The measurement results from one of the three scenarios is shown in Figure 22 and Figure 23.

Figure 21. The DJI Inspire 1 carrying the constructed stand-alone sensor package containing a GPS, a radio transceiver and a gamma radiation detector. On the right is a still image captured by the onboard 4k resolution camera.

Figure 22. Gamma radiation measurements made with the stand-alone sensor package carried by the quadcopter in one of the testing scenarios. The brightness of the green color indicates the intensity of the detected gamma radiation activity.

Figure 23. Local histogram collected from area enclosed by the red circle 3 in the scenario image. The detected spike corresponds with Cs-137 (Cesium with a theoretical gamma radiation energy emissions of 661.64 keV).

 

Robots

The Mörri robot has been equipped with more easily maintainable and more powerful electronics in its reincarnation. The main drive electronics are now mostly off-the-shelf components controlled with an Ardupilot APM2 based controller that is connected to an onboard computer handling the overall robot control and communications with a remote control station. The Mörri platform is also used in testing the test batch of intelligent battery modules that have been constructed. The functional diagram of the new drive system is shown in Figure 24 and the Mörri mobile platform in Figure 25.

Figure 24. The overview of the renewed fundamental electrical system required to drive the Mörri robot.

Figure 25. Mörri with Microsoft’s Kinect 2 sensor driving on a field and on snow with tracks put on.

In anticipation of performing joint missions simultaneously with multiple UAVs and UGVs, also custom quadcopter platforms are being constructed. The basic platform, shown below in Figure 26, is low-cost and is constructed from off-the-shelf components for better maintainability. The quadcopter platform is built around the open-source ArduPilot PX4 flight controller, allowing more freedom for customization and testing our own implementations required for autonomous operation, which is not as easy to do with most prebuilt and significantly more expensive quadrotors. Combined with LIDAR (Light Detection And Ranging), the copters will be used for SLAM and environment classification efforts in joint missions with UGVs, such as Mörri. Because both Mörri and the quadcopter utilize ArduPilot based controllers, the development of both platforms is simpler due to having very similar protocols for using the controller responsible for inertial measurements and platform control.

Figure 26. A semi-ready customizable low-cost quadcopter platform.

 

Intelligent battery modules

Intelligent battery modules (Figure 27) have been developed in collaboration with Probot Ltd. and the test batch is being tested with our robot platforms. The initial tests of the test batch have showed that the designed battery electronics are functioning as was intended. The battery module has an integrated heater for winter operation and a charger module allowing energy transfer from one module to another in any parallel connected energy bus. With a developed charge control module, the bus can also be potentially used to recover energy from multiple power sources, such as solar panels.

Figure 27. Assembled intelligent battery module for general use in modular robotics.

 

Control of Complex Wheeled Robots

Pseudo-omnidirectional robots with individually steer-able wheels offer a good balance between payload, robustness and mobility.  However, the non-holonomic nature of the regular wheels and the often redundantly actuated structure of these robots make their control a complex issue. This complexity of control is further exacerbated when the wheels are not rigidly connected to the robot body but are instead connected via actuated chains which allow the wheels move relative to the body. BISG has developed control algorithms for such Articulated Wheeled Vehicles (AMW). The control algorithms are mathematically simple closed-form analytical functions and are thus computationally light but are currently limited to planar cases. The computational load is only linearly dependant on the number of wheels making the developed control algorithm suitable for multi-wheel configurations and/or low-powered embedded MCUs. The control algorithms synchronize the rolling and steering velocities of complex planar robots (plausible simulated example in Figure 28) with freely located wheels forming fixed or variable footprints. The rolling and steering velocities remain synchronized even with very complex motions of the robot (Figure 29). With the developed control algorithms, the traversable path, robot’s heading on different points of the path and the path velocity can be controlled separately, thus offering great freedom on how to control the robot on a given practical task. The control algorithms do not in practice suffer from representation singularities which are a common problem in wheeled control. The control algorithms also compensate for the proximity of mechanical singularities by adjusting the robot’s path velocity according to the maximum capabilities of its wheels’ steering and rolling actuators. In fact the developed control algorithms are time optimal in a sense that at any given moment the robot is either traversing with maximum allowed path velocity or at least one of its steering or rolling actuators is turning at its maximum velocity (Figure 30), i.e. the robot traverses the given path in the given way with the given velocity restrictions as fast as it possibly can.

Figure 28. Example of complex wheeled planar robot.

Figure 29. Simulation run of Figure 26’s robot traversing a given yellow path while keeping its front directed at all times to a point of interest (green larger dot). Note the smooth convergence of the robot (black line) and the target path.

Figure 30. (Top) wheel rolling speeds, (Middle) wheel steering speeds and (Bottom) robot path velocity for the first 30 seconds of a simulation run.

 

In summary, the developed control algorithms can be used in a wide range of robot configurations and scenarios with low computational cost. The control algorithms are currently limited to planar surfaces and can cause sudden and large changes in velocity and the control algorithms are being extended to work also with uneven surfaces and limited motor torques.

In year 2016 the control algorithms have been enhanced to allow the wheels to have non-zero lateral and longitudinal offsets, making the algorithm suitable for practically any configuration of a wheeled planar robot. In addition, a path tracking algorithm was developed.  The algorithm is very simple yet provides smooth and robust path convergence in simulated environments (Figure 31).

Figure 31. Smooth path convergence in cluttered environment.

Two ERDF project started; Labrobot and OuluZone+ projects. Labrobot focuses on Food industry, and OuluZone+ for autonomous vehicles in harsh conditions.

Labrobot-project focuses on boosting regional Food industry by technology transfer demonstrations, building up test facility and network of stakeholders. By surveying challenges in factories, combined with knowledge of robotics, big data, machine vision and biotechnologies; new kind of solutions are searched for base of new business possibilities. This project is done in cooperation with Center of Machine Vision and signal processing, Biocenter Oulu and Luke. Project is partly funded by City of Oulu, Yaskawa, Probot, Maustaja, Antel, Kinnusen mylly, mekitech, and SR-Intruments.

In the OuluZone+ project the focus is on automatic road building machines and smaller mobile robots (UGV and UAV) for supporting operation on the field.  In the project are studied how the capabilities of autonomous cars could be formally verfied, and tested from perspectives of operting in all weather conditions and all situations. Project is partly funded by City of Oulu, OSEKK, Ouluzone Operointi Oy and industrial partners.

The Evolutionary Active Materials

The Evolutionary Active Materials (EAM) project, which is funded by the Academy of Finland, is a joint effort between the Computer Science and Engineering laboratory (CSE) and the Microelectronics and Materials Physics laboratories. The aim of the EAM project is to develop novel, evolutionary computation (EC) based design methods for active and versatile materials and structures. The first components are being developed through a novel holistic design process utilizing constantly increasing computation power, the development of multi-physics simulators, and EC techniques, such as genetic algorithms (GA).

During 2016, the height and the top diameter of Cymbal type piezoelectric actuator were optimized by genetic algorithm and FEM modelling. From the optimized results, maps of electromechanical capabilities of different structures were generated. The blocking force of the actuator was maximized for different values of displacement by optimizing the height of the cap and the length flat region of the end cap profile. By using values obtained from a genetic algorithm optimization process, a function was formulated for design parameters. Using the function, a map of displacement, the steel thickness and the height of the end cap the optimized length of flat region was constructed (Figure 32). A similar map with the length of the flat region for the optimized height of end cap was created. The results will be published at 2017.

Figure 32. The top diameter of the steel cap as a function of steel thickness and displacement for Cymbal.

New type of actuator called Mikbal (Figure 33) was invented, optimized with genetic algorithm and realized. Mikbal was developed from Cymbal by adding additional steel structures around the steel cap to increase displacement and save the amount of used piezoelectric material. The best displacement to amount of used piezo material ratio was achieved with 25 mm piezo material diameter in the case of 40 mm steel structures, and lower height and top diameter of the cap increased the displacement. The results will be published during 2017.

Figure 33. The von Mises stresses in Mikbal actuator under 500 V voltage.

Also optimization of the end cap structure of the Cymbal type energy harvester was done with genetic algorithm and FEM modeling software Comsol Multiphysics. The aim was to improve harvested power levels from human walking (Figure 34).  The power produced by the energy harvester was increased by allowing the algorithm to modify thickness in certain regions as grooves in the end cap. By evolution of the structure, power produced by the harvester increased by 38 % compared to traditional linear type Cymbal harvester which was also optimized by the algorithm. Increase in power was obtained by change of mode in mechanics of the harvester by grooves.

Figure 34. Cymbal type energy harvester in a shoe and an optimised profile for the harvester. In the profile piezoceramic disc is depicted in yellow and steel cap in grey. The grooves shown in the left side of the profile have been found by the genetic algorithm.

New grooved Cymbal energy harvester (Figure 35) gave promising results in physical measurements producing same power with less force than uniform shape. The model was invented based on results given by genetic algorithm optimization process with spline shapes. Grooved Cymbal is easy to produce compared to spline shape. Depth and place of grooves were optimized by genetic algorithm. The parameters of the algorithm itself were optimized also with GA, called metaGA. Results of the metaGA will be published during 2017.

Figure 35. Grooved cymbals.

 

Intelligent Systems Incorporating Bio-IT solutions

We have taken part in the Ruby/Diamond HILLA project. This was based on collaboration between the polytechniques, VTT and BISG. This and a previous Tekes project lead to establishment of four strategies that should offer openings in the aims to establish minimally invasive of non-invasive wellness and health parameter monitoring technologies. Via a collaborative network, we acquired novel nanomaterials offering ways to couple electronics to biomonitoring behaviour of live cells.

Developing novel real-time biosensors for glucose monitoring. For developing “second generation biosensors”, we have taken use of our skills to purify and culture the skin derived progenitor cells that are responsible in skin renewal and regeneration. We obtained for the project a Tekes strategic opening funding. With this support, we have advanced the work to develop of a novel biosensor strategy (Figure 36).

Figure 36.  Novel biosensor strategy. Donor skin renewing cells are set to culture and a specific responsive component is engineered to target a tag to the 3´end of the coding sequence in the genome. Such a cell is then implanted to the donor to serve as a measure for a given physiological parameter. These serve to offer novel ways to biomonitor in real time physiologically relevant factors with and external electronic reader that is coupled wirelessly to the cloud to data analysis of multiple sensors at the end.   

By now, we have been able to conduct the proof of principle set up in the sensor construction.  These indicate that the skin is indeed responsive to the changes in certain serum constituents. The data also indicated that the cells with in the skin can also be engineered and be converted genetically to serve as biosensors, thus to report changes in the physiological parameters such as glucose. We have screened in selected biological phenomena with the proteomics and transcriptomics the respective mediators in the glucose response in the skin.  We also generated experimental diabetic models to identity diabetes associated and insulin independent responders. The approach has turned a successful one. First of all the skin appears responsive for physiological levels of glucose. Due to this reason we also were able to identify candidate factors whose genes and encoded are currently being engineered to convert the respective protein into an isoform whose activity can read with an external electronic device.

We have also tested the capacity to culture of FACS purified cells of the skin and if such cells can be transplanted with a fluorescent tagged vital sensor cells to the donor so that the cells indeed become incorporated. We assayed the stability of the sensor cells as transplants. The data suggest that a syngeneic host suggesting that the aimed biosensor strategy is feasible accepts the skin progenitor graft.

In collaboration with VTT we have also developed the electronic unit, a tunable spectral camera. This has the capacity to measure the changes in the skin basal progenitor cell integrated sensor. We have filed a patent of these biotechnological avenues with VTT.      

Developing an ex vivo supernatural personal mobile biosensor device. To advance the goal to develop novel wearable sensory devises we started to assemble first via a HILLA funded project a micro fluidistic set up that will be converted to a bio recognition tool. During the research period, several micro fluidistic prints were planned, made and tested. Out of these a configuration was obtained that collected successfully, the skin associated fluids as depicted by the presence of color dye in the fluidistic chamber (Figure 37). A patent search of the strategy has been conducted.

Figure 37. A micro filudistic print design is able to collect the skin-associated fluids as depicted by the accumulation of a blue indicator dye in the chamber.

During 2016, we developed capacity to the micro fluidistic set up to monitor specific biomolecules present in the skin fluids. This work lead to an opening via identification of novel types of biological nanomaterial’s from the skin. These components are generated normally by the cells, they cargo wealth of physiologically relevant biomolecules and they can cross the biological barriers. Given the numerous amounts, small size of the nano scale components, the opening has stimulated a need to establish both bio and databanks. This is currently being conducted with via deep sequencing and proteomics to diagnose the samples that are derived from cohorts.

During 2016, wealth of medical technical developmental lines with VTT and companies have been initiated and also a new Tekes project grant filed. We obtained a new Academy of Finland funded grant from the Bio Future 2025 program to advance the nanobioelectronic analysis strategies, one of the Infotech Oulu research program targets.  

To advance the biosensor openings we have started to develop at the same time more complex diagnostic platforms as the fluidic champers. To be able to read the fluorescence that is revealed by specific antibodies bound to the diagnostic components reagents against these factors are being developed during 2017 with our collaborator. Our partners in the HILLA project were able to develop a mobile phone based micro fluidistic reader capacity. Together with the developed biochips, such printable materials are likely to set the stage for the point of care diagnostics in the field of personalized medicine during 2017.     

Screening of electromagnetic and opto/chemo/electro genetic responses in organs generated from stem cells. The genetic engineering offers opportunities to developed technologies where the cellular in or output signals can also be regulated by certain wavelengths in the electromagnetic spectrum. Alternatively the cellular actions can be genetically constructed so that a signal will be transmitted to a biosensor that will convert it to a form readable by an electric device. To advance these tasks we have initiated with private funding screens that aim to identify cellular channels that are regulated by specific spectral frequencies such as the RF ones. Such diagnostics use a paradigm shift where the cellular responses to given stimuli will be screened primarily via vital “biosensors” with live cellular tags. Thus the approach in the bioelectronics analytics have become possible via the crisp Cas9 genome-editing technologies where libraries of gene edited diagnostic cells can be generated.   

During 2016, we developed novel tissue engineering technologies that do enable introduction of specific gene expression constructs to individual cells of the model organ such as the mammalian kidney. Here the organ primordia is dissociated to single cells, the genetic construct encoding the protein of interest such as the opto, chemo or radiogenetic responsive component is transduced to such a cell with a reporter for the read out screens. There after the organ is let to self-assemble and placed for a long-term culture (Figure 38).

Figure. 38. An organ primordia can be dissociated to single cells, the constitute cells transduced with a genetic construct to acquire opto-, chemo- and radio genetic guidance capacity to the morphogenetic cells ex vivo.

With the developed model systems we have taken use of the image analysis technologies to visualize how the morphogenetically active cells behave in three dimension in the 4D conditions that offer a whole organ primordia to be cultured ex vivo.  To achieve this we applied defined pressure to the assembled organ primordia in ex vivo setting depicted in Figure 39.

Figure 39. The 3D kidney organ primordia that is relatively thick being composed of multiple cell layers develops also under a mild pressurize in ex vivo. Here the mechanical pressure converts the 3D development more towards a 2D configuration. The developed setup will offer ways to identify pressure sensors in the cells and also to develop novel organ pressure monitoring tools. The power of this novel “organoid” culture set up is that it enables for the first time is complex organs image analysis and follow up of the behavior of the individual constituent cell while the complex 3D anatomical structure of the organ become laid down. It is important that the quality of the data good enough to offer segmentation and “computer vision” analysis. With such “Fixed Z-Dimension” (FZD) culture we are in a process of illustrating the fine details how biological shapes, namely the organ structure in 3D becomes constructed from the cellular building blocks. These data serves also as the digital 3D landscape for developing 3D bio printing when advancing a European Union FET FLAGSHIP representing a regenerative medicine and nanotechnology initiative. 

We found that under a defined pressure the organ flattens towards two dimension (2D) but yet morphogenesis progressed (Figure 40). This novel set up has made it possible follow the fate of individual cells is the cells are constricting a detailed manner while the natural form.

Figure 40. Operetta confocal workstation coupled to a robotic set up and an incubator was assembled. A) A holder for plates and transported by the robotic arm (B) and the cells with in will be transported to an incubator (C). The whole set up is inside a hood (D) and the robotic arm transports the plates to the Operetta confocal semi-high throughout microscope fluorescent reader. The data is analyzed by wealth of machine vision/image analysis programs present with in the assembled bio robotic set up. The bio robotic core facility will be used to screen with a library of live indicators cellular response to specific frequencies in the electromagnetic spectra. 

To target the detailed dynamics by which the form is assembled in a model organ we took use of the genetically engineered Wnt4CreGFP knock in mouse model. This was crossed to the floxed Rosa26 Yellow Fluorescent Protein (YFP) transgenic mice. In this genetic crossing the stem cells that generate whole of the nephron will become labeled with the YFP.

With the fixed Z-dimension culture we have captured 3D movies from the developing kidney with the confocal microscope in a time-lapse setting.  We are in a process of analyzing the detailed cell behavior via the machine learning/computer based image analysis with Prof. Janne Heikkilä. With Dr. Jari Juuti we aim to construct a specific device that allows detailed measure of the pressure forced encountered by the tissue undergoing morphogenesis. These novel capabilities now allow analysis in great detail the mode by which the spatial and temporal organization of the cells go on to construct natural form that is open at present in any developing organ system. We will use models to identify the pressure sensors from the cells with the OMICS technologies. 

Developing high throughput robotic aided platforms to screen complex cellular responses to magnetic/electric fields via signaling pathway reporters. To advance the strategies to measure in a high through put manner the cellular responses to stimuli we have assembled a bio robotic workstation. Here an Operetta confocal microscope was obtained and this was coupled to a hood that contains an automated plate-cargo arm, a rack for the plates with a bar code reader and incubator for long term exposure of the cells to compounds such as drugs or specific electromagnetic spectral radiation (Figure 40). The Operetta confocal microscope has machine learning/image analysis capacity for wealth of measurements to be conducted from the cells.

To take use of the set up a yeast cell library was obtained and three replica clones from it was generated and stored for later use. The library is composed of cell where each of the 3´end of each of the yeast gene was targeted by a green fluorescent protein (GFP) tag. The next goal is to obtain capacity to start to use the set up to define the oscillating properties of the cellular genes and to use it as live measures for screening responses to stimuli such as those mediated by the opsins for the visible light frequencies. Such genome wide screens vital bio indicator based scan be expected to lead to identification of novel biosensor pathways for certain spectral frequencies. When the strategy will be subjected to patient derived gene edited human induced pluripotent (iPS) cells and those whose fate has been engineered to defined directions this technology should  offer avenues for the era personalized medicine diagnostic developmental aims.   

Intelligent Systems with cohort data sets: Cohort data set is a special data set from the medical domain, which has not been studied with a machine learning approach before. The data set, Northern Finland Birth Cohort 1966 (NFBC 1966), is a unique data set with over 14 000 original variables in various yet heterogeneous formats (numerical, ordinal, categorical, images, text etc.) from a population of over 12 000 mothers and their children without any complete data points. The amount of variables rises to millions if genetics and epigenetics are considered (p >> n).

There are two extremely important aspects of modeling this type of data: confidence of the predictions made with the model and model interpretability. Steps towards instance level confidence estimates have been made in our previous work (see above) and we will continue to pursue this goal, along with keeping model interpretability in focus also, when we start digging into this fascinating data set. Our goal is to use a machine learning approach to make novel discoveries from the data that traditional data analysis approach has not yet uncovered.

Elders are an increasingly large fraction of the population in developed countries. From one hand people expect an independent life also in presence of more or less important diseases. On the other hand the treatments to care those diseases, often together with co-morbidities, imply larger costs. To respond to both these goals, the disease progress should be kept as low as possible (see Figure 41), which means early disease detection, deinstitutionalisation and personalised medicine, striving to allow a better quality of life, a more cost-efficient healthcare system and a more inclusive access to healthcare both in developing countries and in remote areas in developed countries.

Novel Bio-ICT technologies are needed to achieve these targets and BISG is active in this area in many fronts summarised below.

By tracking health status of large groups and including in the analysis a wealth of metrics and parameters, large amounts of data are generated. On the other hand, by downscaling biology-based technologies down to the nanoscale including sensing biological parameters directly from living cells, potential security threats are correspondingly moving into human bodies, but promising tools are offered for personalised medicine and treatments, including tight biological interaction, prostheses and their control (Celentano and Röning 2015). BISG is strong in all these areas (data analysis, security and robotics) and it is therefore pushing itself among the world leaders in this growingly important area.

Figure 41. Progress of a disease (left), outcome (right) and access to healthcare (Celentano and Röning 2015).

 

Towards a Holistic Self-awareness in Humans and AI

Artificial entities like robots and unmanned or autonomous vehicles are more and more present in the human environment. Social interaction among all the players in such a heterogeneous scenario (Figure 42) calls for a number of research issues to be addressed and its study offers interesting potentialities.

Figure 42. Interwork among heterogeneous agents and within them. From Celentano & Röning (2016b).

Self agency. Self-awareness in humans plays a role in a number of brain functions and disturbances. On the other hand, self-awareness improves the efficiency in robotic systems (Celentano and Röning 2016a). Awareness of the self is achieved through analysis of observations, or measurements, of various entities involved. This interwork in a heterogeneous multi-agent system (Figure 42) may occur with different topologies: sensing the actuation of other entities, as in Figure 43a; acquiring information shared by others, as in Figure 43b; exploiting different functions for self/nonself discrimination. In short, through perrception, action, and sharing information (Celentano & Röning 2016a).

Figure 43. a) Left: An entity (bottom) sensing the actuation of two entities (top). b) Right: Entities (bottom) acquiring instructions shared by another entity (top). From Celentano & Röning (2016a).

Embodied agent. As psychologist James Gibson observed, there is an interdependency of perception and action (“perceive to act, act to perceive”). We study the social intelligent entity as embodied (Celentano & Röning 2016b), where are brought to evidence not only the interaction among entities but also the interwork within them (cf. Figure 42 and Figure 44).

Considering that an instantiation of the agent may possess only part of its functions, the same generic model can be used at different scales, applied to entities, at brain, body and world domains (the latter possibly including other entities or agents), benefitting modularity and scalability (Celentano & Röning 2016b).

Figure 44. The embodied agent and its environment. From Celentano & Röning (2016b).

Structured information representation and instruction logic. Interwork among modular agents include as seen perception and action but also the exchange of information or commands, both referred to as instructions as in Celentano & Röning (2016a). These communications may be subject to noise, as it is the case in an operating room or in air traffic control (Figure 45, top). Whereas machines are subject to environmental noise only, humans suffer both environmental noise and cognitive noise (Celentano & Röning 2016c), affecting different steps of the information communication process (Figure 45, bottom).

Figure 45. Top: Interaction among heterogeneous agents in a noisy environment. Bottom: Information communication between remote source and destination entities in noisy conditions. From Celentano & Röning (2016c).

For reliable information exchange among heterogeneous agents is needed a formal representation of the exchanged instructions, usable by both humans and machines (Figure 46).

Figure 46. Interaction through specified processes (languages and representation). From Celentano & Röning (2016c).

Using the instruction logic in Celentano & Röning (2016c), the example situation in which mobile m0 at x0 orders mobile m3 to be in x1 at t1 to search a book b and bring it immediately to m0 can be represented by

«report_to,m3,m0,x0,t,1»

^ «move_to,m3,-,x1,t<t1,1»

^ «search,m3,b,-,t,1»

^ «bring,m3,b,x0,0,1».

 

Exploitation of Results

BISG continued co-operation with the SpAtial, Motor & Bodily Awareness (SAMBA) research group [http://dippsicologia.campusnet.unito.it/do/gruppi.pl/Show?_id=hhuv] at the Department of Psychology of the University of Turin, Italy. Several initiatives for EU projects including Horizon 2020 are ongoing and these efforts will be continued.

Outside Europe, BISG is currently co-operating with the University of Nevada, Arizona State University and Carnegie Mellon University.

The results of our research were applied to real-world problems in many projects, often in collaboration with industrial and other partners. Efficient exploitation of results is one of the core objectives of the national Digile and FIMEC ICT SHOK projects like SIMP, IoT and Cyber Trust; in these projects we work in close collaboration with companies throughout the projects.

During the reporting year, the group continued utilizing outdoor robotic systems. Development and utilization of Mörri, a multipurpose, high performance robot platform continued. More focus was put on perception in natural conditions, representation of detections, knowledge, and an environment model of the operating environment. The software architecture further developed the earlier work on Property Service Architecture, and the Marker concept as general purpose representation was further developed.

Future Goals

The partnership in the SIMP programme that belongs to the SHOK concept of Tekes enables us to continue our steel research into new areas. The new goals are in quality prediction at different process stages and for more challenging properties. As a result more advanced expert systems can be developed to aid the operators with different roles in steel making.

We will continue to strengthen our long term research and researcher training. We will also continuously seek opportunities for the exploitation of our research results by collaborating with partners from industry and other research institutions on national and international research programs and projects. The University of Oulu is a founding member of euRobotics. Juha Röning is a member of the Board of Directors of euRobotics.

We will strengthen our international research co-operation. With the University of Tianjin in China, we have a joint project in which methods and a system will be developed for vision-based navigation of Autonomous Ground Vehicles, which utilize an omni-directional camera system as the vision sensor. The aim is to provide a robust platform that can be utilized in both indoor and outdoor AGV (Autonomous Ground Vehicles) applications. This co-operation will continue.

In the USA, we will continue to co-operate with the Human-Computer Interaction Institute in Carnegie Mellon University with Assistant Professor Anind K. Dey. The research is on human modelling in the area of human-machine interaction. We continue and strengthen US-Finland co-operation through an NSF grants. The co-operation within SOCRATE co-operation with the University of Nevada can be exploited complementary expertise in the area of multi-layer security. Two new BISG project proposals for co-operative projects under the WiFiUS programme are currently under review at the Academy of Finland and NSF. 

Shorter research visits to European partners in EU-funded projects are also planned. The cooperation with Prof. Raffaella Ricci and her colleagues, focuses on bridging neuroscience and artificial intelligence. This research aims at cross-fertilising the two scientific domains, continuing and strengthening the research paths currently active at respective sides.

In 2017, the aim is to utilize more widely the know-how from sensor technology and data mining. New application areas will be studied, including rehabilitation, exercise motivation and energy efficiency in households, and the benefits of our expertise will be highlighted to actors in the areas.

In human-environment interaction and sensor networks, our research will continue. Our main goals are to develop analysis methods for sensor network data and to develop applications utilizing physical user interfaces. Research on novel software architectures, reasoning and knowledge representations will continue as well. Field trials in realistic settings, and close collaboration with research groups (national and international) and companies will be emphasized.

Personnel

professors

2

postdoctoral researchers

10

doctoral students

15

other research staff

5

total

32

person years for research

25

 

<--break->

External Funding

Source

EUR

Academy of Finland

156 000

Tekes

735 000

domestic private

176 000

international

200 000

total

 1 267 000

 

Doctoral Theses

Latvakoski, Juhani (2016) Small world for dynamic wireless cyber-physical systems. VTT Science 142.

Selected Publications

Alasalmi T., Koskimäki H., Suutala J. and Röning J. (2016). Instance level classification confidence estimation. Advances in Intelligent Systems and Computing. The 13th International Conference on Distributed Computing and Artificial Intelligence 2016, Springer.

Celentano U (2016) Panel: European Project Space on Intelligent Technologies for Innovation and Sustainability. Invited. 8th International Conference on Agents and Artificial Intelligence (ICAART). 24–26 Feb 2016, Rome, Italy.

Celentano U, Röning J (2016a) Multi-robot systems, machine-machine and human-machine interaction, and their modelling. 8th International Conference on Agents and Artificial Intelligence (ICAART), vol. 1, pp. 118–125. 24–26 Feb, Rome, Italy.

Celentano U, Röning J (2016b) Modular agents for heterogeneous human-robot systems. ERL Emergency & TRADR Workshop on Heterogeneity in Robotic Systems, Oulu, Fin-land, 22–26 Aug.

Celentano U, Röning J (2016c) Structured information representation and exchange in heterogeneous multi-agent systems in mission-critical scenarios. Information Systems Technology Panel: Specialists Meeting on Intelligence & Autonomy (Ro-botics). Bonn, Germany, 25–27 Oct.

Celentano U, Röning J, Yang L, Zhang J, Ermolova N, Tirk-konen O, Chen T, Höyhtyä M (manuscript) Information and physical security in people-centric IoT.

Höyhtyä M, Mämmelä A, Celentano U, Röning J (2016) Power-efficiency in social-aware D2D communications. Proc. European Wireless Conference (EW 2016). Oulu, Finland, 18–20 May 2016.

Koskimäki H and Siirtola P (2016) Recognizing Unseen Gym Activities from Streaming Data - Accelerometer vs. Electromyogram Advances in Intelligent Systems and Computing, International Conference on Distributed Computing and Artificial Intelligence

Koskimäki H and Siirtola P (2016) Adaptive Model Fusion for Wearable Sensors Based Human Activity Recognition International Conference on Information Fusion, ISIF, 07, 1709-1713.

Koskimäki H and Siirtola P (2016) Model Update in Wearable Sensors Based Human Activity Recognition IEEE Symposium on Computational Intelligence and Data Mining, accepted.

Tuovinen L (2016) A conceptual model of actors and interactions for the knowledge discovery process. In Proc. 8th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management – Volume 1: KDIR, 240–248.

Tuovinen L, Ahola R, Kangas M, Korpelainen R, Siirtola P, Luoto T, Pyky R, Röning J & Jämsä T (2016) Software design principles for digital behavior change interventions: Lessons learned from the MOPO study. In Proc. 9th International Conference on Biomedical Engineering Systems and Technologies – Volume 5: HEALTHINF, 175­–182.

Niemelä Maisa, Ahola Riikka, Pyky Riitta, Jauho Anna-Maiju, Tuovinen Lauri, Siirtola Pekka, Tornberg Jaakko, Mäntysaari Matti, Keinänen-Kiukaanniemi Sirkka, Röning Juha, Jämsä Timo, Korpelainen Raija (2016) Nuorten miesten fyysinen aktiivisuus ja istuminen itsearvioituna ja mitattuna Liikunta & Tiede, 53(2-3):73-79.

Pietikäinen P, Kettunen A & Röning, J (2016) Steps Towards Fuzz Testing in Agile Test Automation. International Journal of Secure Software Engineering, Volume 7 Issue 1, January 2016, pp. 38-52.

Siirtola P & Röning J (2016) Reducing Uncertainty in User-independent Activity Recognition - a Sensor Fusion-based Approach International Conference on Pattern Recognition Applications and Methods, Rome, Italy 24-26 February 2016, 611--619.

Siirtola P, Koskimäki H & Röning J (2016) From User-independent to Personal Human Activity Recognition Models Exploiting the Sensors of a Smartphone 24th European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning, ESANN 2016., Bruges, Belgium 27-29 April 2016, 471--476.

Siirtola P, Koskimäki H & Röning J (2016) Personal models for eHealth - improving user-dependent human activity recognition models using noise injection IEEE Symposium on Computational Intelligence and Data Mining, December, accepted.

Siirtola P.; Tamminen S.; Ferreira E.; Tiensuu H.; Prokkola E.; Röning J. (2016) Automatic Recognition of Steel Plate Side Edge Shape Using Classification and Regression Models The 9th Eurosim Congress on Modelling and Simulation, September.

Tiensuu H, Tamminen S, Pikkuaho A & Röning J (2016) Improving the yield of steel plates by updating the slab design with statistical models.  Ironmaking and Steelmaking: Processes, Products and Applications, accepted, August 2016.

Tuovinen L, Ahola R, Kangas M, Korpelainen R, Siirtola P, Luoto T, Pyky R, Röning J, Jämsä T (2016)  Software Design Principles for Digital Behavior Change Interventions: Lessons Learned from the MOPO Study Proceedings of the 9th International Joint Conference on Biomedical Engineering Systems and Technologies - Volume 5: HEALTHINF, 175-182.

Drelon C, Berthon A, Sahut-Barnola I, Mathieu M, Dumontet T, Rodriguez S, Batisse-Lignier M, Tabbal H, Tauveron I, Lefrançois-Martinez AM, Pointud JC, Gomez-Sanchez CE, Vainio S, Shan J, Sacco S, Schedl A, Stratakis CA, Martinez A, Val P. PKA inhibits WNT signalling in adrenal cortex zonation and prevents malignant tumour development. Nat Commun. 2016 Sep 14;7:12751. doi:10.1038/ncomms12751. PubMed PMID: 27624192; PubMed Central PMCID: PMC5027289.

Nagy II, Xu Q, Naillat F, Ali N, Miinalainen I, Samoylenko A, Vainio SJ. Impairment of Wnt11 function leads to kidney tubular abnormalities and secondary  glomerular cystogenesis. BMC Dev Biol. 2016 Aug 31;16(1):30. doi: 10.1186/s12861-016-0131-z. PubMed PMID: 27582005; PubMed Central PMCID:PMC5007805.

Rak-Raszewska A, Vainio S. Nephrogenesis in organoids to develop novel drugs and progenitor cell based therapies. Eur J Pharmacol. 2016 Nov 5;790:3-11. doi:10.1016/j.ejphar.2016.07.011. PubMed PMID: 27395798.

Xu Q, Krause M, Samoylenko A, Vainio S. Wnt Signaling in Renal Cell Carcinoma. Cancers (Basel). 2016 Jun 17;8(6). pii: E57. doi: 10.3390/cancers8060057. Review. PubMed PMID: 27322325; PubMed Central PMCID: PMC4931622.

Vidal V, Sacco S, Rocha AS, da Silva F, Panzolini C, Dumontet T, Doan TM, Shan J, Rak-Raszewska A, Bird T, Vainio S, Martinez A, Schedl A. The adrenal capsule is a signaling center controlling cell renewal and zonation through Rspo3. Genes Dev. 2016 Jun 15;30(12):1389-94. doi: 10.1101/gad.277756.116. PubMed PMID: 27313319; PubMed Central PMCID: PMC4926862.

Halt KJ, Pärssinen HE, Junttila SM, Saarela U, Sims-Lucas S, Koivunen P, Myllyharju J, Quaggin S, Skovorodkin IN, Vainio SJ. CD146(+) cells are essential for kidney vasculature development. Kidney Int. 2016 Aug;90(2):311-24. doi: 10.1016/j.kint.2016.02.021. PubMed PMID: 27165833. 

Pietilä I, Prunskaite-Hyyryläinen R, Kaisto S, Tika E, van Eerde AM, Salo AM, Garma L, Miinalainen I, Feitz WF, Bongers EM, Juffer A, Knoers NV, Renkema KY, Myllyharju J, Vainio SJ. Wnt5a Deficiency Leads to Anomalies in Ureteric Tree Development, Tubular Epithelial Cell Organization and Basement Membrane Integrity Pointing to a Role in Kidney Collecting Duct Patterning. PLoS One. 2016 Jan 21;11(1):e0147171. doi: 10.1371/journal.pone.0147171. PubMed PMID: 26794322; PubMed Central PMCID: PMC4721645.

Prunskaite-Hyyryläinen R, Skovorodkin I, Xu Q, Miinalainen I, Shan J, Vainio SJ. Wnt4 coordinates directional cell migration and extension of the Müllerian duct essential for ontogenesis of the female reproductive tract. Hum Mol Genet. 2016 Mar 15;25(6):1059-73. doi: 10.1093/hmg/ddv621. PubMed PMID: 26721931; PubMed Central PMCID: PMC4764189. 

Krause M, Samoylenko A, Vainio SJ. Exosomes as renal inductive signals in health and disease, and their application as diagnostic markers and therapeutic agents. Front Cell Dev Biol. 2015 Oct 20;3:65. doi: 10.3389/fcell.2015.00065. Review. PubMed PMID: 26539435; PubMed Central PMCID: PMC4611857.

Daniel E, Onwukwe GU, Wierenga RK, Quaggin SE, Vainio SJ, Krause M. ATGme: Open-source web application for rare codon identification and custom DNA sequence optimization. BMC Bioinformatics. 2015 Sep 21;16:303. doi: 10.1186/s12859-015-0743-5. PubMed PMID: 26391121; PubMed Central PMCID: PMC4578782.

Bibikova O, Popov A, Bykov A, Prilepskii A, Kinnunen M, Kordas K, Bogatyrev V, Khlebtsov N, Vainio S, Tuchin V. Optical properties of plasmon-resonant bare and silica-coated nanostars used for cell imaging. J Biomed Opt. 2015 Jul;20(7):76017. doi: 10.1117/1.JBO.20.7.076017. PubMed PMID: 26230637. 

Rak-Raszewska A, Hauser PV, Vainio S. Organ In Vitro Culture: What Have We Learned about Early Kidney Development? Stem Cells Int. 2015;2015:959807. doi: 10.1155/2015/959807. Review. PubMed PMID: 26078765; PubMed Central PMCID: PMC4452498.

Berry RL, Ozdemir DD, Aronow B, Lindström NO, Dudnakova T, Thornburn A, Perry P, Baldock R, Armit C, Joshi A, Jeanpierre C, Shan J, Vainio S, Baily J, Brownstein D, Davies J, Hastie ND, Hohenstein P. Deducing the stage of origin of Wilms' tumours from a developmental series of Wt1-mutant mice. Dis Model Mech. 2015 Aug 1;8(8):903-17. doi: 10.1242/dmm.018523. PubMed PMID: 26035382; PubMed Central PMCID: PMC4527280.

Ali N, Hosseini M, Vainio S, Taïeb A, Cario-André M, Rezvani HR. Skin equivalents: skin from reconstructions as models to study skin development and diseases. Br J Dermatol. 2015 Aug;173(2):391-403. doi: 10.1111/bjd.13886. Review. PubMed PMID: 25939812.

Krause M, Rak-Raszewska A, Pietilä I, Quaggin SE, Vainio S. Signaling during Kidney Development. Cells. 2015 Apr 10;4(2):112-32. doi: 10.3390/cells4020112. Review. PubMed PMID: 25867084; PubMed Central PMCID: PMC4493451.

Naillat F, Yan W, Karjalainen R, Liakhovitskaia A, Samoylenko A, Xu Q, Sun Z, Shen B, Medvinsky A, Quaggin S, Vainio SJ. Identification of the genes regulated by Wnt-4, a critical signal for commitment of the ovary. Exp Cell Res. 2015 Mar 15;332(2):163-78. doi: 10.1016/j.yexcr.2015.01.010. PubMed PMID: 25645944.

Rajaram RD, Buric D, Caikovski M, Ayyanan A, Rougemont J, Shan J, Vainio SJ, Yalcin-Ozuysal O, Brisken C. Progesterone and Wnt4 control mammary stem cells via myoepithelial crosstalk. EMBO J. 2015 Mar 4;34(5):641-52. doi: 10.15252/embj.201490434. PubMed PMID: 25603931; PubMed Central PMCID: PMC4365033.

Junttila S, Saarela U, Halt K, Manninen A, Pärssinen H, Lecca MR, Brändli AW, Sims-Lucas S, Skovorodkin I, Vainio SJ. Functional genetic targeting of embryonic kidney progenitor cells ex vivo. J Am Soc Nephrol. 2015 May;26(5):1126-37. doi: 10.1681/ASN.2013060584. PubMed PMID: 25201883; PubMed Central PMCID: PMC4413750.

Naillat F, Veikkolainen V, Miinalainen I, Sipilä P, Poutanen M, Elenius K, Vainio SJ. ErbB4, a receptor tyrosine kinase, coordinates organization of the seminiferous tubules in the developing testis. Mol Endocrinol. 2014 Sep;28(9):1534-46. doi: 10.1210/me.2013-1244. PubMed PMID: 25058600.

Rymer C, Paredes J, Halt K, Schaefer C, Wiersch J, Zhang G, Potoka D, Vainio S, Gittes GK, Bates CM, Sims-Lucas S. Renal blood flow and oxygenation drive nephron progenitor differentiation. Am J Physiol Renal Physiol. 2014 Aug 1;307(3):F337-45. doi: 10.1152/ajprenal.00208.2014. PubMed PMID: 24920757; PubMed Central PMCID: PMC4121567.

Maezawa Y, Onay T, Scott RP, Keir LS, Dimke H, Li C, Eremina V, Maezawa Y, Jeansson M, Shan J, Binnie M, Lewin M, Ghosh A, Miner JH, Vainio SJ, Quaggin SE. Loss of the podocyte-expressed transcription factor Tcf21/Pod1 results in podocyte differentiation defects and FSGS. J Am Soc Nephrol. 2014 Nov;25(11):2459-70. doi: 10.1681/ASN.2013121307. PubMed PMID: 24904088; PubMed Central PMCID: PMC4214535.

Pietilä I, Vainio SJ. Kidney development: an overview. Nephron Exp Nephrol. 2014;126(2):40. doi: 10.1159/000360659. Review. PubMed PMID: 24854638.  

Halt K, Vainio S. Coordination of kidney organogenesis by Wnt signaling. Pediatr Nephrol. 2014 Apr;29(4):737-44. doi: 10.1007/s00467-013-2733-z. Review. PubMed PMID: 24445433; PubMed Central PMCID: PMC3928513.

Prunskaite-Hyyryläinen R, Shan J, Railo A, Heinonen KM, Miinalainen I, Yan W, Shen B, Perreault C, Vainio SJ. Wnt4, a pleiotropic signal for controlling cell polarity, basement membrane integrity, and antimüllerian hormone expression during oocyte maturation in the female follicle. FASEB J. 2014 Apr;28(4):1568-81. doi: 10.1096/fj.13-233247. PubMed PMID: 24371124.

Cheddad A, Nord C, Hörnblad A, Prunskaite-Hyyryläinen R, Eriksson M, Georgsson F, Vainio SJ, Ahlgren U. Improving signal detection in emission optical projection tomography via single source multi-exposure image fusion. Opt Express. 2013 Jul 15;21(14):16584-604. doi: 10.1364/OE.21.016584. PubMed PMID: 23938510.

 

 

Last updated: 12.5.2017