by Nivanthika K. Wimalasena
figures by Rebecca Clements

Imagine going in for a surgery where the surgeon, instead of looking down and seeing only your swollen leg, can see the exact location of your fracture before making a single incision. Now imagine that this doesn’t require x-ray vision or the stuff of science fiction, but is possible through augmented reality (AR), used to overlay an image of a standard x-ray onto your broken leg. In fact, this technology is already being used in the medical field to provide better care for patients.

What is augmented reality?

Defined most simply, AR is a computerized projection of images or other information like scans, blueprints, or maps that is overlaid with, and anchored to, the physical world. AR need not require anything more than the phone you already have—at its simplest it can mean the ability to add an animated figure to a picture, or to find virtual Pokémon in the physical world around you, as in the popular game Pokémon Go.

More complex applications are also underway, such as in GPS navigation, in which wearable glasses could take the map from your phone screen to the three-dimensional physical streets in front of you, telling you which way to go and simultaneously eliminating the need to look back and forth between a map and the road.

However, perhaps one of the most exciting applications lies in the medical field, where surgeons can use these types of wearable AR glasses to visualize crucial information from scans overlaid with the body of the patient as they operate. There are several features of AR glasses that are necessary to make this kind of application possible.

How do wearable AR glasses work?

A main requirement of AR glasses is that they have clear lenses, allowing us to see the world in front of us, while mapping virtual information on top of it. To do this, they contain sensors that are used to create an internal, three-dimensional representation of the physical world in front of the user (currently with up to a 120° field of view). This is achieved using a combination of standard cameras and depth-sensing infrared cameras. This representation is used as the anchor to which to tether the projected holograms (Figure 1).

Figure 1: Schematic of AR glasses. AR glasses include standard and infrared cameras, sensors (including a gyroscope and accelerometer), and clear lenses. This image was based on the Microsoft HoloLens2.

An important quality of this projection then, is that it moves as we move. For this, the glasses contain various sensors that allow it to estimate the position of the head in space—such as an accelerometer, which measures the rate of change of our motions, and a gyroscope, which measures the angle of these motions. Using all of this information, the glasses “know” where we are, how we’re moving, and what the world in front of us looks like.

Therefore, the sensors in AR glasses allow projected holographic images to be tethered not only to physical space, but also the user’s body. The crucial function of these measurements is to adjust for head and eye movements in order to create a stable and believable augmentation of reality, which is layered over the real world. In more complicated applications, this same data allows the glasses to “see” the location of the user’s hands, which allows the user to not only view these projections, but also to use their hands to manipulate them, zooming in and out, and switching between images.

Creating a pair of glasses with this set of qualities is not an easy task, and the technology is continually improving. Creating the required, faithful internal representation of the physical world seen by the user requires the storage and processing of huge amounts of data. The ability to deal with this data in a self-contained manner, using only the glasses themselves and not requiring a phone or computer, is a significant and recent advancement. Before the glasses were equipped with their own processing power, there were significant delays as data was transmitted from the glasses to a computer and back again. Therefore, previous iterations of AR glasses have been plagued with issues of a lagging projected images as the user moves. Perceivable lag between a head movement and the movement of the image disallows interaction with the image in real time, and would result in an out-of-sync overlay with the real world. Progress on these fronts is ongoing, as both hardware and software improvements are made.

How can AR be applied in medicine?

Currently, surgeons use a variety of scanning methods to visualize tumors to be removed, aneurisms to be clipped, or fractured bones to be repaired. Information from x-ray, magnetic resonance imaging (MRI), ultrasound, computed tomography (CT), and/or others is gathered to create a plan before a surgery. However, these scans and their critical information have the limitation of being separate from the physical patient.

At Imperial College London, surgeons have partnered with Microsoft and are using an AR headset (the HoloLensÔ) while performing reconstructive surgery on victims with severe leg injuries. In these surgeries, the location of the blood vessels surrounding the break is critical. Tissue grafts from other parts of the body are often required, and this involves connecting grafts to the local blood supply at the site of injury. In order to use AR for these surgeries, doctors at ICL first used computed tomography angiography (CTA), a technique allowing visualization of blood vessels, to create a 3D image of the vessels, bones, and soft tissue in the leg (Figure 2). Then, using custom software, these scans were rendered in a form compatible with the HoloLensÔ glasses.

Figure 2: The use of AR for surgery. This image depicts a rendering of a patient’s broken leg, generated by HoloLens2 software using computed tomography angiography scans. Blood vessels are depicted in pink and can be seen running through the leg around the fractured bone. Yellow arrows denote the location of perforating arteries, which surgeons use to connect tissue grafts during the reconstructive leg surgery. This Figure was adapted from Pratt et al. 2018, European Radiology Experimental.

During the surgery, surgeons were able place these holographic projections over the leg, enabling them to see the locations of bone fractures and the surrounding blood vessels before making an incision3. In this way, the ability to visualize the vessels during the surgery has the potential to improve patient outcomes and reduce the length of surgeries, which is better for both patients and their doctors. In fact, at ICL, surgeons report that this AR-based method is already more reliable and therefore less time-consuming than audible Doppler ultrasound, the current standard.

Why isn’t AR being used in every operating room?

Current medical applications of AR are not without their limitations. First, digitizing and integrating various types of scans into three-dimensional models takes time, meaning that AR assistance is currently not an option in emergency situations. Secondly, the implementation of a pipeline in hospitals to make these models, and the custom software needed to render these 3D images so that they can be projected via hologram, are limiting factors in their widespread adoption. Finally, doctors will undoubtedly require training to use the glasses and take full advantage of their functionality.

That said, this technology has a huge potential to make specific routine procedures, such as the reconstructive leg surgery discussed above, safer and faster. In these cases, once the software exists, scans from new patients could easily be put into an existing pipeline. For complex or intricate procedures, increased precision in visualization of injury sites, tumors, blood clots, etc. through AR could be a tremendous boon for both doctors and patients.

Ultimately, there is a lot of potential and room for innovation in AR, particularly in medicine. Improvements in software to render and digitize the two-dimensional information contained in CT, MRI, and other medical scans into three dimensions has the potential to make this technology truly paradigm shifting. And, as engineers make technical advancements on the hardware and software of AR glasses themselves, they will become more seamlessly integrated with the physical world and more intuitive to use. In time, these improvements will hopefully lead to widespread adoption of the technology, adding a new depth to the phrase “the doctor will see you now.”


Nivanthika K. Wimalasena is a Ph.D. student in the Program in Neuroscience at Harvard Medical School.

Rebecca Clements is a third-year Ph.D. candidate in the Biological and Biomedical Sciences program at Harvard. You can find her on Twitter as @clements_becca.

For more information:

This article is part of our SITN20 series, written to celebrate the 20th anniversary of SITN by commemorating the most notable scientific advances of the last two decades. Check out our other SITN20 pieces!

2 thoughts on “Augmented Medicine: the power of augmented reality in the operating room

  1. Indeed this is amazing discovery in the face of prevailing accidents/ emergency situations faced by both patients and doctors. If operationally harnessed and adopted, will go along way in the rate at which lives are saved. kudos to the amiable PhD students, more grease to your elbows. God bless you. pst. Samuel

Leave a Reply

Your email address will not be published. Required fields are marked *