A comfortable and immersive user experience in this type of a scenario is already supported by the latest intelligent high-end head-mounted devices and software where spatial audio and high-rate streaming (4K-and-beyond) create a sensation of nearly physical presence in a virtual world.
In real life, the immersive experience places extremely high performance requirements for wireless networks. Associate Professor Antti Tölli is now developing an innovative physical layer connectivity framework which combines directional mmWave connectivity, use of affordable in-device memory and multipoint coded caching (CC) techniques.
“In our emerging vision, cache-aided mmWave-based connectivity solutions will be able to manage extreme peak rates while ensuring the reliability of increasingly demanding XR applications,” Tölli says. “The storage capacity of mobile devices is constantly increasing and therefore becoming one of the cheapest network resources, readily available closer than we think. With our innovative solution, the realization of immersive digital environments in demanding dynamic wireless settings may become affordable in the near future."
Tackling wireless bottlenecks
One of the key paths towards the intelligent use of network resources and device memory departs from so-called bottleneck areas which suffer from limited wireless connectivity. “Advanced content distribution, wireless delivery and proactive caching algorithms can dramatically increase the system performance in such areas,” Tölli says. “In an immersive scenario, coded and collaborative caching is especially beneficial because the location-dependent multimedia content becomes a dominant part of the wireless data delivery.”
However, current techniques of data placement in cache memories cannot yet handle a dynamic scenario where the users move freely and where the immersive virtual environment is captured based on the exact location and tilt angle of each user’s headset. “We need to acquire large-scale detailed information of the environment in the proximity of each headset,” Tölli notes. “Each user’s streamed data is unique and also highly location and time dependent, and the delivery delays must be lower than the human perception thresholds."
The magic of in-device memory
In actual fact, a large part of the rich multimedia content for rendering a certain viewpoint is common among several users. “If we can identify and select the common location-specific content, it offers opportunities for efficient use of pooled in-device memory resources,” Tölli says. “We can then employ intelligent cache placement and network coded multicast content delivery mechanisms which make the pooled memory an additional degree of freedom to be exploited, in addition to wireless resources.”
In practice, the cache memory of the users could be opportunistically filled, for instance, with multimedia content which represents the fixed structures of the virtual world. In the memory placement, content which is relevant to the identified system-wide bottleneck areas with poor connectivity, is preferred and incentivized,” Tölli says. “We foresee that, ultimately, a significant portion of the multimedia content rendered on the eyewear would originate from the local cache available conveniently at the user device.”
Less cache involvement, on the contrary, is needed closer to the access points where the wireless connectivity is fluent. “We are exploring various cache placement opportunities to establish appropriate schemes for distributing files across the selected cache memories,” Tölli says. “When we combine dynamic and proactive cache replacement strategies with intelligent multiantenna content delivery, common coded content can be delivered to maximize the quality of experience of all the users, given the limited total memory across the devices.”
The remaining requested content across the network can be delivered e.g. via multipoint multicast beamforming of the network coded content. On the other hand, part of content can also be exchanged directly employing nearby device-to-device connections (D2D). After the cache (re-)placement has been carried out, the network can potentially offload the requested content onto the D2D connections. For this purpose, the caches of the nearby users are utilized. “These hybrid scenarios are highly useful,” Tölli says. “The network can then decide whether it shares the content directly, possibly to multiple users simultaneously, using unicast or multicast beamforming, or whether it allocates the request to a short-range D2D connection.”
Challenges of mmWave links
As software companies are ramping up, larger audiences consume extended reality applications. With the constantly growing usage intensity and unconstrained user mobility, a dynamic full immersion scenario causes major capacity and quality challenges for existing networks. Field tests in the current 5G networks have proven 5G capability for a Gbps access link on mmWaves. “Immersive digital experience in multi-user environments requires extremely high area capacity and ultra-low over-the-air latencies, which are not yet fully achieved with 5G,” Tölli notes. “Communication over mmWave links is by default unreliable, subject to random blockage. Solving these challenges will become critical as we strive to enable future dense and dynamic immersive networks.”
The considered immersive viewing application, in all of its complexity, offers valuable results for other fields of technol-ogy too including autonomous driving, rural network connectivity and remote surgery, to mention a few. The algorithms, system design solutions, and performance evaluation frameworks that Tölli and his team innovate, will be able to direct cellular operators, telecommunication research organizations, equipment vendors, and mobile software companies, among others, towards affordable future solutions beyond 5G.
Industry relevance of the solutions is highlighted in an on-going project with Nokia Bell Labs with funding from Finnish Research Impact Foundation. “The goal is to validate selected technical solutions in the early phase of 6G system development, through proof-of-concept systems and demonstrations, jointly with our industry partner,” Tölli says.
Although he immerses himself into the challenges of virtual content delivery during working hours, Tölli has managed to keep his home and three sons detached from virtual worlds. As a fervent outdoor enthusiast, he is not likely to replace the real world with a digital twin soon, either. “Fresh air and Finnish forests are much too compelling and hard to replicate,” Tölli notes. “My ultimate immersion happens in the middle of nowhere - with my feet on pedals or skis."
Antti Tölli graduated with a doctoral degree in 2008. In his research, he focuses especially on multiant-enna communications and related signal processing challenges. He has also accumulated strong experience in industry.
Read more: cc.oulu.fi/~atolli
Last updated: 26.4.2021