by Busola Olukoya
figures by Rebecca Senft

In the past year, I was excited to see Ready Player One, Black Panther and Avengers: Infinity Wars in theaters. What these three movies have in common – besides the action-packed storylines that kept me on the edge of my seat, gripping my boyfriend’s bicep until he was in a lot of pain – is their depiction of the possibilities of virtual reality (VR) and augmented reality (AR) technology. VR is an immersive computer-generated simulation of an environment. In Ready Player One, players are completely immersed in a VR world called “the Oasis” and wear special clothing to complete challenges and interact with other players. AR, on the other hand, superimposes virtual objects on the user’s view of the real world. We see AR used for communication in Black Panther.

Outside of movies, VR and AR are the new wave of entertainment and communication technology; Microsoft and Facebook are among the top companies competing in this market. The future of VR and AR is touted to be the development of mixed/merged reality (MR) technology, in which the virtual and the real worlds are more seamlessly interwoven. The technologies that integrate the real and virtual worlds are rapidly evolving, opening the door for many exciting applications of MR technology in years to come.

Him/Her: Human beings as the operating system

MR is a form of AR in which the virtual objects appear more realistic. That is, an MR user would be able to experience and interact with a virtual object as if it were real (Figure 1). For example, players of the well-known Pokémon Go game would be able to perceive relative distance from a character as they would from a sign-post prohibiting trespassing. MR technology gives the user the ability to interact with both the virtual and real environment at the same time as opposed to being completely immersed in the virtual environment – as is the case with VR – or having the virtual environment pop up on the real environment (AR).

Figure 1. MR at work  Holoportation enables MR users to interact with people who are far away as if they were physically present. The MR user in this image is interacting with the holograph of another MR user in a distant location.

MR is thought to be superior to both VR and AR. The key advantage of MR over VR is the use of transparent lenses in MR headsets. Opaque lenses are used to create the immersive environment of VR, but MR users can transition between the virtual and real worlds more easily. However, in contrast to AR, where virtual objects are held in a fixed position, MR computers take into account the position and orientation of the user in placing the virtual objects so that they seem more real and are easier to interact with.

Tricking the brain: Reality perception in MR

What we perceive as the real world is constructed by our brains. Information from the sensory organs is taken in by the brain and matched to previously stored information in memory to create the reality that we perceive and interact with. Developers of MR technology rely on data obtained from studying how the mind constructs reality to create technology that can trick the brain into perceiving virtual objects as real.

Understanding how the eye works is essential for MR developers, as the eye relays visual stimuli to the brain. The eye perceives the color and shape of an object, as well as its size, thickness, and location in space. When we look at a landscape, the picture we see is a seamless integration of all this information and is dependent on factors such as the orientation of the head relative to the rest of the body and the height of the body relative to other objects. A child whose head cannot reach the kitchen counter has a completely different perception of the world than an adult of average height.

In tricking the eye to pick up virtual objects as real, MR headsets are equipped with scanners which take in 3D information about the environment and sensors which relay information about the position and orientation of the user’s head relative to the body and the environment. These pieces of information are used to orient the placement of the virtual world within the real environment. This is also dynamic. In a Microsoft demo of HoloLens, a holographic computer developed for MR use, the aliens the user is fighting respond to the change in orientation of the user by changing the location of their attack in the living room setup.

Additionally, spatial sound technology enhances the user’s perception of the “realness” of virtual objects (Figure 2). In the real world, you can hear an ambulance getting closer because the sound it makes gets louder. You can also tell what direction it is coming from or going to. With spatial sound, the aliens in the HoloLens demo are made to sound louder as they come closer. This makes the aliens feel more realistic to the user.

Figure 2. Auditory contributions in MR Spatial sound technology is used in VR, AR and MR to enhance the user’s perception of the nearness and directionality of virtual objects. In this virtual environment, the VR user is able to tell that the bird is closer to him than the whale.

Ready player one?

MR is poised to be the next big technology in enhancing workplace productivity, global communication and education. For example, Microsoft HoloLens developers have coined the term holoportation by which people can be seen as holographs and can be interacted with as if they were physically present. Through holoportation, MR can be used in distant learning and to communicate with loved ones who are far away.

The rise in media portrayal of VR and AR technology — as evidenced by my movie experiences this year — disguises the long history of their development. High costs of VR and AR equipment hindered their widespread use until the advent of Google Cardboard; a virtual reality platform developed for use on a smartphone. Currently, there is a push towards subsidizing equipment pricing for VR and AR and increasing access to MR equipment, with a subsequent increase in awareness and demand for these technologies. A recent estimate for global HoloLens use was stated to be around fifty-thousand people. The use of MR is projected to rise in the next few years as the technology becomes more developed and more developers join in the effort to make MR a part of our daily lives.

Busola Olukoya is a PhD student in the Program in Neuroscience at Harvard Medical School.

For more information:

  • To learn about the use of MR for education, check out this Cyber Acoustics article
  • For more information about how MR is being used today, see this article on MR applications

This article is part of the 2018 Special Edition — Tomorrow’s Technology: Silicon Valley and Beyond 

5 thoughts on “It’s All in the Mind: Insights to the development of mixed reality technology

  1. My own parents dont believe me and. Had me committed for observation against my will. My ex tried to falsely send me to prison for horrible offenses that never happened. I did 1 year in jail and was aquitted on 15 felony counts. She lied entirely and i was either going to prison for 35 years or going home. It was a nightmare. Now they threaten to put illegal porn on my phone or set me up again for things i didnt do. True story. They tell people lies about me and isolate me from everyone. I cant prove the electronic harrassment. Help please. Cops wont believe me because i used drugs and they think im seeing things.

  2. I have an ex whose friends are harrassing me with vr or ar or mr inside my home and everywhere i go. And yes i am serious. I believe they are in Tulsa. Theyve done it for years and even come in my bathroom and harrass and virtually rape me. What can i do to stop them. Please help.. I am 100% serious.

    1. Hey did you find out how they are doing it and what the technology is called it’s happening to my family and police don’t believe me any help

  3. By way of MR using VR and AR together,
    I thanks for your painstaking and wonderful illusionary approach!

Leave a Reply

Your email address will not be published. Required fields are marked *