Augmented Reality in Health Care

Augmented Reality (AR) is defined as “a live direct or indirect view of a physical, real-world environment whose elements are "augmented" by computer-generated or extracted real-world sensory input such as sound, video, graphics, haptics or GPS data.” For most people, AR is a new and entertaining technology that creates a “3D” virtual image of a structure that appears to be floating on their mobile device screen and can be modified by motion and various interactive controls. The best-known example is Pokemon Go.
It is estimated that over a billion smartphone users will create AR content at least once in 2018, with at least 300 million doing so monthly, and tens of millions weekly. (Deloitte's 2018 Technology, Media, and Telecommunications Predictions Report). In anticipation, AR development tools like Apple’s ARKit and the new iPhone X and iPad Pro will further the expansion and integration of augmented reality at an accelerated pace. It has been predicted that the next version of ARKit will enable new AR game development to include multiple players.
In terms of ease of use and development platforms for new AR content, Apple’s new computer OS (High Sierra) introduced support for VR in Metal 2 and has partnered with Valve, Unity, and Unreal to bring VR creation tools to the Mac. A new feature is persistent tracking, which will “remember” where an object is placed in a virtual space. AR content creation will become easier and available for everyone and applications will be integrated into almost every facet of learning and engagement. 

Health Care 

In health care, AR has been integrated into a number of applications for several years. Surgery, anatomy, education and simulation AR apps have been developed with variable success to date, but most everyone agrees that augmented reality will only get better and become a part of clinical care and education in the coming years.
Jump Simulation, a part of OSF Innovation, is one of the top innovation centers in the world in medical innovation, simulation, education and research. The Advanced Imaging and Modeling (AIM) program staff at Jump Sim explores and develops unique and engaging solutions that address the most critical challenges facing health care in clinical settings and in medical education.
“At Jump and through the Advanced Imaging and Modeling (AIM) program we have demonstrated the value of virtual and augmented reality for clinical decision making, patient education, and clinical education,” said John Vozenilek, MD, Vice President and Chief Medical Officer for Simulation at Jump Sim. “The basic premise is quite simple, we are activating and exciting more of the senses. When people are engaged in VR and AR, they are using binocular and 3-dimensional vision. They are in motion and manipulating their environments, interacting in ways that cannot be accessed during traditional and didactic settings." 

"There is no doubt that VR is effective. AR will also be effective, and perhaps in different ways. The interaction with the real-world permits additional connectedness to objects and people in the IRL environment. This will be a pathway to astonishing discovery in the field of team and distance-team work. The smallest of the advantages may just be that shared visualization around augmented reality objects will permit multiple viewpoints to be shared around a common understanding. Many of the future impacts are as yet untold, but we anticipate our role in discovering and leveraging the advantages of AR for the benefit of our patients,” said Vozenilek.

But how does Jump Sim continue to discover, develop and leverage the impact of technologies such as AR, VR and mixed reality for the benefit of our patients? One way is through collaborative relationships with colleagues at both the Urbana-Champaign (UIUC) and Chicago (UIC) campuses of the University of Illinois. The Biomedical Visualization graduate program at UIC is a global leader and partners with Jump Sim through internships and research projects. Samantha Bond, a faculty member in Biomedical Visualization at UIC, is one of only a few people in the US who teaches a course in AR specifically for biomedical and health care applications.
“Both augmented and virtual reality technologies are in a rapid stage of growth. As developers continue to use mixed reality technology for health care, we can see clearer definitions emerging for new types of projects and their applications in medicine,” said Bond. “Where virtual reality holds the power to transport a user to a new world, providing access to new environments and empathy, augmented reality can bring elements of any new world into the real context of the user's life." 

"Augmented reality, which can be often overlooked as a powerful education tool, is remarkably efficient for health care communication. As quickly as opening an app on your phone, you can bring complex medical concepts to your own fingertips, bridging a gap between knowledge in your own environment and the content on the screen. Patients with varying degrees of health literacy can begin connecting their own realities with their medical content, and physicians can train themselves to better perform procedures by bringing virtual elements into their real worlds. As students in visualization programs continue to develop augmented reality software, I believe we'll start to see long-lasting impacts of augmented reality on medical student education, patient education, and clinical training,” said Bond. 


Some of the best AR apps in health care have been created for teaching anatomy. An interactive anatomical software leader has been 3D4Medical, LLC and the Complete Anatomy product line which inludes the Complete Heart and Anatomy modules. 
In the most recent versions of the Complete Anatomy apps, a sophisticated AR feature enables 3D virtual projection of the dissections that the user can create in the app. This functionality adds new dimension to the exploration of the human body from a tablet (my favorite method to use AR is on the iPad Pro), and can be integrated into most any curricula (from STEM courses for K-12 learners to advanced medical anatomy).
The resources in human anatomy have never been better. Combined cadaver dissection in the lab, life-size touch devices such as the Anatomage Table, mobile apps (Complete Anatomy), DICOM data (CT, MR and ultrasound), and VR apps (such as those in the AIM Lab at Jump Simulation) will integrate new and valuable tools in a hybrid curriculum. As more of these resources continue to be integrated into clinical settings, AR integration in surgery, simulation and procedures skills will have a direct impact on patient care.     


Augmented reality and virtual reality are promising platforms for surgery and surgical planning. In surgical education, AR provides a three-dimensional space for demonstration, interaction, simulation of a specific procedure and an opportunity to collaborate from distant locations using common mobile devices.
In a recent TEDWomen 2017 presentation, Dr. Nadine Hachach-Haram provided a live demonstration of the use of AR in remote surgery. From the podium of her live TED presentation in New Orleans, she connected online with an orthopaedic surgeon at the University of Minnesota utilizing the augmented reality collaboration software that her company (Proximie) developed. Dr. Hachach-Haram stresses the future advantages through virtual AR interaction in surgery will provide expert access for remote populations around the world, especially in underserved areas.
In a recent American College of Surgeons ACS-AEIs Virtual Grand Rounds entitled “Augmented Reality: Debating its Value in Simulation and Surgery,” two surgeons presented the pros and cons of using AR in surgery.
Steven Schwaitzberg, MD, FACS from the University of Buffalo Jacobs School of Medicine and Biomedical Sciences, described the benefits of AR as providing a totally safe environment for residents to practice surgery. In simulated surgery using AR, residents can take exams that enable automated score-keeping that provides instant automatic feedback. Dr. Schwaitzberg went on to cite a published paper “Mixed Reality for Robotic Treatment of a Splenic Aneurysm” (Pietrabassa et al, 2010) where surgeons applied AR overlay on the patient to guide the dissection of the surgical field.
In describing the current limitations of AR in surgery, Lee Swanstrom, MD, FACS, from the Oregon Clinic GMIS, Institute for Image Guided Surgery (and an affiliation with IHU-Strasbourg), a basic issue is the required manual effort to create segmented anatomy for AR (a challenge we also have at Jump Simulation and at other institutions). The hope is that one day, these “3D models” will be created in real time. Also, most operating rooms are not yet set-up to incorporate AR in surgical procedures.
The use of AR in telesurgery is in its infancy, but holds promise. With AR, VR and mixed reality, we now have much better tools to use and with mobile devices, surgical training can be shared throughout the world. 

Future Applications 

AR has many current applications in gaming, education and health care. The value of AR will increase exponentially with the integration of IoT data (Internet of Things), artificial intelligence (AI, such as IBM Watson integrated in decision making and data analytics), tactile and haptic feedback capabilities and the printing of 3D models in surgical planning.
Product development (sharing development of medical devices collaboratively in a 3D space over long distances), a democratization of innovation (using easy to use mobile devices) and interactive learning (such as product manuals and immersive simulation) will advance AR across most disciplines. 
At Jump Simulation and elsewhere, the value of AR is in the early stages of development and application. As we continue to address the challenges in health care, AR, VR and mixed reality will continue to play an increased role in the goal of improved patient outcomes, clinical education, expense efficiency and learner access. 

--Grace Hsu, Experiential Instructional Designer-Develop at Jump contributed to this story. 
Categories: Advanced Imaging and Modeling (AIM), Augmented Reality (AR), Education, Innovation, Medical Visualization, Surgery, Virtual Reality (VR)