New breakthrough study into understanding depth perception – results could be applied to improve machine vision

Similar neural (so-called phase-coding) principles likely underpin stereo vision in other animals, including humans. These principles – based on mirror-symmetrically moving sensors to sample 3-dimensional information from the world – could be applied to improve robotics and machine vision, according to a new breakthrough study into understanding depth perception.

Fruit flies are able to see the world more clearly and accurately than first thought. It had been previously thought that the inner workings of fruit flies’ eyes were immobile, since the insect’s compound eyes appear stationary from the outside. Therefore it has been long thought that they could only see a low-resolution, “pixelated” image of the world with little or no depth perception.

The researchers from University of Oulu participated in international team led by scientists at the University of Sheffield, which now discovered that photoreceptors, cells in the eye which react to changes in light, twitch in an organised way, which create a fine detailed image. Research benefitted novel in-vivo synchrotron imaging in unique context.

The team discovered that flies are able to see in high-resolution stereo vision because their photoreceptors gather more information about the surroundings than previously thought possible. The photoreceptors do this by responding to light changes with ultrafast mirror symmetrical movements, called photoreceptor microsaccades, so when the fly moves forward they are gathering images that both move with the world and against it simultaneously.

The image information the photoreceptors gather is therefore not coarsely “pixelated” but continuous and much more detailed, which is then sent to the brain and processed into what the flies see.

Professor Mikko Juusola, from the University of Sheffield said: “Many animals have two eyes to see the world in depth and because the eyes are apart, they capture two separate images of the same scene. It is well-known that vision must use the slight positional differences between these images, called the disparity, to determine where and how far visual objects are from us and their 3D structure. However, it is not well understood how the eyes and brain take this 2D information and turn it into a 3D understanding of the surroundings.

The eyes of fruit flies examined in a particle accelerator

“The new research results on depth perception and stereo vision are so groundbreaking that textbooks will have to be rewritten. In the study, the researchers from the University of Oulu innovated and participated to in-vivo imaging at synchrotron. During this phase, a particle accelerator was used to, for example, examine the dynamics between the photoreceptors in the two eyes of a fruit fly,” says Professor of Experimential Electron Spectroscopy Marko Huttula from the Nano and Molecular Systems Research Unit NANOMO, read more about the unit.

To make these discoveries required the team of scientists to devise many new research methods, never before used in vision research. For example, to measure photoreceptor microsaccade dynamics locally and globally across the two eyes, they pioneered high-speed microscale infrared and nanoscale X-ray imaging in living flies. The findings of these experiments were then implemented into biophysically-accurate multi-scale morphodynamic sampling models to simulate the compound eyes' newly-found high-resolution adaptive optics. The models gave theoretical predictions of how well the flies should see the world, and these predictions were used to design novel visual stimuli. Finally, these stimuli were used in testing the super-resolution signalling in the fruit fly brain networks and learning behaviours in the flight simulator system to distinguish super-resolution 3-dimensional objects, revealing the true limits of the flies' amazingly stereo vision.

This could have implications for human eyesight, as the nervous system of a fruit fly has evolved to perceive the three dimensional world efficiently and it’s very likely that us and fruit flies use similar principles in order to see in stereo. Furthermore, the researchers propose how the principle and mathematical theory about mirror-symmetric visual information sampling based on the new results could be used to improve man-made sensors, and is directly applicable for machine vision and robotics.

Highly respected Proceedings of the National Academy of Sciences (PNAS) published the results online March 17, 2022:

• Binocular Mirror-Symmetric Microsaccadic Sampling Enables Drosophila Hyperacute 3D-Vision. Joni Kemppainen, Ben Scales, Keivan Razban Haghighi et al. PNAS 2022: https://doi.org/10.1073/pnas.2109717119