Machine vision replaces expert presence - Augmented reality technology developed by the University of Oulu can guide step-by-step even in computer repair
The University of Oulu has developed an application focused on AR-assisted task guidance, which allows the performance of tasks requiring special expertise independently without the presence of an expert. The application works by having an expert initially create instructions for performing the task using a digital twin produced by 3D scanning technology of the target device and adding graphical annotations such as arrows and text boxes. There can be several instructions for different tasks, and the device manufacturer can create them in advance. The actual mobile application works by the user downloading the necessary instructions for the task from the instruction library, which they can then follow step-by-step on the mobile device's screen. The instructions are projected onto the device using AR technology.
"AR-assisted guidance has indeed been studied worldwide, and there are also commercial applications, but they typically require the presence of an expert and interaction with the user, whereas our proposed solution allows the task to be performed independently without personal guidance. The application could be compared to a much more precise version of popular tutorial videos that many use when they need to do something beyond their expertise," says Professor of Machine vision and signal analysis Janne Heikkilä from the University of Oulu.
According to Heikkilä, the development work is still ongoing, and currently, the application can mainly be used to demonstrate the technology with simple tasks. However, the intention is to release an open-source implementation in the future, which will include all the essential tools.
This way, in principle, anyone can create a digital twin of the target, create instructions for necessary maintenance or repair tasks, and use the mobile application to repeat them in a real environment. For example, changing a bicycle tire could be easily accomplished with the application when the instructions are created by the bicycle manufacturer specifically for that model.
"In the future, the focus will be on utilizing artificial intelligence in creating instructions, making it more effortless and reducing the amount of expert work. The mobile application also requires a lot of development to make it user-friendly. Additionally, the aim is to enable the use of AR or XR glasses, eliminating the need to hold the mobile device in hand," says Heikkilä.
The first demo version has been built quickly, as the project is based on the existing strengths of both the University of Oulu and the University of Central Sweden's research.
'The project combines our university's years of experience in image-based 3D reconstruction and the recent developments in the field, which have made digital twins even more accurate,' says IMMERSE's project manager in Oulu, researcher Janne Mustaniemi.
The purpose of the IMMERSE project is to promote remote work and operation using immersive augmented reality technologies to address the unique challenges and skill shortages of northern regions. The project's beneficiaries are primarily organizations in the mining, forestry, and manufacturing industries, whose business efficiency and work safety the new technology improves. IMMERSE is a joint effort of Finnish and Swedish expertise, an Interreg Aurora-funded joint project with Mid Sweden University, which began in August 2024.
In the picture, the IMMERSE team is seen through the application's demo view from left to right: Business development manager Janne Haverinen from the University of Oulu, researcher Mohammadreza Shamshirgarha from Mid Sweden University, Associate Professor Emin Zerman from Mid Sweden University, Professor Mårten Sjöström from Mid Sweden University (centre), researcher Janne Mustaniemi from the University of Oulu, Professor Janne Heikkilä from the University of Oulu, and researcher Ahmed Alhawwary from the University of Oulu.