– Written by Bruno Borralho Gobbato, Brazil

THE DAY THE MONITOR DISAPPEARED

It was an ordinary Thursday list. My scrub nurse was humming the same Tom Jobim tune she always does when the arthroscopy cart refused to boot. While the rep fiddled with cables, I donned a Apple Vision Pro. The rotator‑cuff tear suddenly floated in front of me—4‑K, three‑metres wide—untethered to any cart. I could pin MRI slices to my left, vital signs to my right and, for the first time in two decades, keep my neck in neutral. So this is what the metaverse feels like, I thought. This is not exactly how it happened, but it is a taste of my feelings.

INTRODUCTION

Surgical innovation continues to evolve with the goal of improving outcomes, ensuring patient safety, enhancing surgical precision, and optimizing procedural efficiency. Among emerging technologies, mixed reality (MR)—a continuum between the real and virtual worlds—has gained particular relevance in surgical disciplines. Enabled by head-mounted displays like the Microsoft HoloLens, MR allows surgeons to interact with 3D holographic projections using voice commands, gaze, and hand gestures. Unlike traditional data interfaces, which rely on 2D screens and images, MR integrates interactive 3D reconstructions into the surgeon’s field of view, bridging the gap between digital planning and physical execution. The global rise in AR/VR investment and academic research reflects the growing recognition of MR’s transformative potential¹.

In orthopedic surgery, MR is increasingly being applied to enhance preoperative planning, intraoperative navigation, and postoperative rehabilitation. By superimposing patient-specific 3D data directly onto the operative field, MR systems can improve accuracy, reduce radiation exposure, shorten operative times, and maintain sterility while displaying relevant clinical information. Procedures such as vertebroplasty, K-wire placement, tumor resection, and total hip arthroplasty have already demonstrated tangible benefits. Within shoulder surgery, particularly in arthroplasty, MR enables detailed visualization, real-time guidance, and remote collaboration, suggesting a broader role for this technology in upper extremity procedures. As MR systems become more refined and accessible, they may serve as a cost-effective alternative to robotics, revolutionizing orthopedic surgical training and practice.

MIXED REALITY IN THE ARTHROSCOPY

Surgical advancements continuously seek to enhance patient outcomes and operational safety. In the realm of orthopaedic surgery, the integration of technology has markedly improved procedural accuracy and recovery times. Notably, the development of three-dimensional (3D) anatomical reconstructions and computer-assisted surgical planning has revolutionized pre-operative strategies and patient care. Building upon these innovations, the transition from pre-operative planning to intra-operative application represents a significant step forward, facilitated by augmented reality (AR) technologies.

Operating rooms (ORs) around the world can vary significantly in terms of size, video availability, size of monitors, and their positions. These variations often depend on the healthcare facility’s resources, the type of surgeries performed, and regional practices and standards. The size of operating rooms can range from small, compact spaces in older or resource-limited facilities to large, spacious environments in modern hospitals. In advanced facilities, ORs are often equipped with multiple high-definition (HD) or even 4K cameras to provide detailed views of the surgical field. These cameras can be mounted on the ceiling, walls, or on surgical lights. The availability of video technology can vary greatly; in some developing regions, the use of video may be limited or absent due to resource constraints. Conversely, in state-of-the-art hospitals, comprehensive video integration systems are standard, allowing for real-time video streaming, recording, and remote consultations.

The size of monitors in operating rooms can also vary widely. In high-resource settings, large monitors, ranging from 27 to 55 inches or larger, are common. These monitors provide clear, detailed views of surgical procedures and are often HD or 4K to enhance visual clarity. Smaller monitors, around 20 to 24 inches, may still be used in less advanced ORs or for secondary displays. 

Virtual reality (VR) headsets can revolutionize operating rooms by democratizing access to advanced surgical visualization, reducing the need for physical monitors, and enabling more efficient use of space. Unlike fixed physical monitors, VR headsets offer flexible and customizable display options. Surgeons can position virtual screens in their field of view as needed, eliminating the need for multiple large monitors in the operating room. This flexibility allows for a more ergonomic and personalized setup, where information can be displayed directly within the surgeon’s line of sight, reducing the need to turn away from the patient.

By using VR headsets, the physical space requirements of operating rooms can be significantly reduced. Traditional setups often require ample space to accommodate large monitors, multiple workstations, and other equipment. VR headsets can streamline these requirements, making it possible to design smaller, more efficient operating rooms without compromising on functionality. This is particularly advantageous in resource-constrained settings where space is limited.

VR headsets enable remote collaboration, allowing surgeons to consult with experts and colleagues from anywhere in the world in real-time. Through VR, remote surgeons can view the same surgical field and provide guidance, making it easier to democratize access to specialized expertise.

Spatial Flexibility of Virtual Displays:

During the shoulder arthroscopy surgeries, surgeons experienced a significant enhancement in their visual field using the Headsets. The headset allowed for the positioning of virtual screens within the surgeon’s field of view at any desired location and size. This adaptability proved crucial, enabling surgeons to customize their visual workspace according to the specific demands of each procedure. The ability to adjust the virtual display dynamically during the surgery allowed for optimal viewing angles and ergonomics, which minimized physical strain and improved procedural focus.

Integration of Notes and Diagnostic Imaging:

A pivotal feature of the MR technology in use was its capacity to integrate essential patient data directly into the surgical view. Surgeons were able to overlay real-time MRI scans and patient notes onto their field of vision. This integration provided immediate reference without the need to divert attention away from the surgical site. The seamless access to a comprehensive set of data—including pre-operative planning notes and dynamic MRI images—enhanced decision-making accuracy and confidence during complex surgical maneuvers.

Responsiveness and Real-Time Performance:

A critical aspect of integrating augmented reality in surgical settings is the system’s responsiveness. The Apple Vision Pro Headset displayed an impressive lack of latency, with real-time updates of visual data as the surgery progressed. The absence of perceptible lag in displaying MRI data and virtual annotations ensured that the surgeons could rely on the augmented visuals to accurately reflect the current state of the surgical field. This reliability was crucial for maintaining the flow of the surgery and for the effective use of augmented reality tools in a high-stakes environment.

DYNAMIC ANTERIOR STABILISATION – FROM VIRTUAL BLUEPRINT TO HOLOGRAPHIC DRILL PATH

If Bankart repair is skateboarding, Dynamic Anterior Stabilisation (DAS) is wingsuit flying: the margin for error is millimetres, the penalty for drift is the suprascapular nerve. In 2024 our team asked a simple question: Could we design the perfect tunnel in virtual reality the night before and then project it, life‑size, onto the bone when it mattered?

SCULPTING THE BLUEPRINT IN VR

The workflow starts in a workstation that automatically segments the patient’s CT scan into scapula, humerus, and nerve channels. I export those meshes as STL files and pull them into a VR sandbox (ShapesXR). Inside the headset the scapula floats. With haptic controllers I trace the ideal posterior drill trajectory, hugging subchondral bone but steering 15 mm clear of the suprascapular notch. The software calculates cortical thickness, warns of potential blow‑outs, and colour‑codes safety margins—green for ≥20 mm nerve distance, amber for 15 – 20 mm, red for anything tighter. With medical images of the supraescapular nerve, we 3D-draw the nerve on the patient (cadaver model) 3D model scapula (from the CT). Now we have a 3D scapula model, with planning execution and nerve information.

FROM FILE TO FIELD: MIXED‑REALITY EXECUTION

Day of the surgery we fire up a HoloLens 2, and import yesterday’s .glb. After a point surface‑mapping routine, the holographic scapula locks onto reality with sub‑centimeter fidelity. The planned tunnel appears suspended in space. The instrument to drill is also planned and the scapula with nerve is aligned with the cadavers torso.

This is one of the first shoulder arthroscopy procedures used with Mixed Reality guidance.

Mixed reality did not make DAS easy; it made it repeatable. By hard‑coding yesterday’s perfect plan into today’s imperfect reality, we traded anxiety for alignment—and, I hope, signed the suprascapular nerve’s parole.

MIXED REALITY FOR ARTHROPLASTY

Nothing exposes a surgeon’s insecurities like a CT scan of a Walch‑B2 glenoid.

Shoulder arthroplasty poses unique challenges, particularly in the presence of glenohumeral deformities and glenoid bone loss. These anatomical alterations are common in degenerative shoulder arthritis and cuff tear arthropathy. Traditional two-dimensional CT scans often fail to adequately reveal the full extent of these deformities, prompting the adoption of three-dimensional (3D) reconstructions for a more accurate assessment of glenoid vault morphology, including depth, version, and the containment of bone loss. Understanding these parameters preoperatively is crucial for planning successful reconstruction.

Glenoid preparation is a critical step in shoulder arthroplasty. Malpositioning of the glenoid component, particularly in cases of severe retroversion or bone loss, has been associated with high rates of implant failure, including component migration and loosening. Excessive corrective reaming to address retroversion can further compromise glenoid bone stock, while insufficient correction may lead to poor implant biomechanics. Therefore, surgical goals include maximizing implant backside contact, restoring version and inclination, minimizing unnecessary reaming, preventing peg or cage perforation, optimizing fixation, and avoiding mechanical impingement.

To achieve these objectives, preoperative planning software using 3D CT data has been developed to improve implant positioning. These platforms allow surgeons to manipulate virtual reconstructions to determine optimal component orientation and fit. More recently, the use of patient-specific instrumentation and 3D-printed models has facilitated improved transfer of the preoperative plan to the operative field. However, this approach can be time-consuming and cost-intensive.

Mixed reality (MR) technology represents a significant advancement in this domain. MR platforms enable immersive visualization of preoperative plans and patient-specific anatomy during surgery through head-mounted displays. These devices project interactive 3D holograms into the surgeon’s field of view, which can include segmented CT or MRI data, the glenoid or humerus in virtual form, and even dynamic intraoperative navigation overlays. Unlike conventional navigation systems that require the surgeon to look away at an external monitor or sterile tablet, MR allows continuous focus on the surgical site with enhanced ergonomics.

Studies have demonstrated that MR-guided systems improve the precision of glenoid pin placement and component orientation, outperforming manual instrumentation in cadaveric models. Furthermore, MR reduces the need for physical 3D-printed models by providing holographic representations directly within the sterile field. The ability to customize the display layout—whether highlighting implant trajectories, viewing step-by-step guides, or accessing virtual controls—further enhances intraoperative flexibility and efficiency.

The integration of MR into shoulder arthroplasty workflows offers numerous advantages: real-time navigation without visual distraction, improved tactile feedback, and better surgeon posture. These benefits are especially relevant in complex glenoid reconstructions where accuracy is paramount and bone preservation is limited. As this technology continues to evolve, MR may offer a cost-effective, scalable alternative to robotic-assisted surgery, helping surgeons achieve more predictable outcomes in both primary and revision shoulder arthroplasty^3-10.

ORTHOPEDIC INTERNATIONAL ROUND TABLE IN MIXED REALITY

Mixed Reality (MR) has rapidly evolved from an intraoperative tool to a versatile platform capable of transforming how surgeons collaborate across the globe. By merging digital holography with real-world visualization, MR allows users to interact with patient-specific anatomical data, imaging, and surgical plans in an immersive, three-dimensional space. This capability opens new avenues for real-time, interactive, and remote case discussions among orthopedic specialists. In a pioneering international initiative, four orthopedic surgeons based in Brazil, Switzerland, the United States, and Scotland convened in a shared MR environment using the Microsoft HoloLens 2 headset in conjunction with the Mesh Meeting platform. Each participant contributed complex shoulder surgery cases, uploading preoperative imaging, 3D reconstructions, and intraoperative visuals into the virtual space. Within this digital environment, surgeons could use hand gestures and voice commands to manipulate and discuss each case collaboratively, examining glenoid morphology, implant planning, and potential complications. The system maintained stable connectivity and minimal latency across all locations, allowing seamless interaction. One of the most striking features was the ability to point, rotate, and enlarge 3D holograms using virtual hands—an interaction that closely replicated the dynamics of a face-to-face surgical planning session. This format offered clear advantages over conventional video conferencing by enabling detailed spatial understanding and surgeon-to-surgeon engagement centered on shared virtual content. Additionally, the MR system eliminated the need for physical 3D-printed models, reduced the reliance on screens, and preserved sterile-field communication when applied in operative settings. While the cost of MR headsets remains a consideration—approximately $3,500 USD per device—the rapid expansion of the metaverse market and the emergence of new hardware options suggest that accessibility will improve in the near future. Ultimately, this experience highlights the potential of MR technology to establish a new paradigm in surgical education, international collaboration, and case-based learning—especially valuable in complex fields such as shoulder reconstruction, where subtle anatomical nuances are critical. As MR continues to integrate into orthopedic practice, it is poised to become a cornerstone of global surgical connectivity.

THE ROAD AHEAD

What keeps me bullish?

• Precision — If mixed reality can consistently shave ±2 mm off implant positioning, patient‑specific guides may become museum pieces.
• Democratisation — A junior surgeon in Manaus (BRA) —or Turkey —can scrub virtual shoulders with Mayo faculty before touching a live patient.
• Well‑being — My neck could no longer aches after a marathon cuff‑repair list, and that alone is worth the visor fog.
• Collaboration — I’ve coached surgeons from my office in Jaraguá do Sul (BRA) via a holographic feed, turning 3,000 km into the distance of an arm gesture.
• Radiation‑free OR — Our mixed‑reality guide can reduce fluoroscopy
• Greener Surgery — Less physical. More digital.

What keeps me cautious?

• Overload — Too many floating panels and you feel like flying economy inside Times Square.
• Dependence — If the headset glitches mid‑case, have you rehearsed the low‑tech bailout?
• Access — A USD 4‑k visor is lunch money in Silicon Valley, but a year’s salary in many parts of the world.
• Data Security — Shared holograms travel through the cloud; a single misconfigured server could broadcast PHI across continents.
• Regulatory Drift — ANVISA, FDA and EU MDR rules evolve yearly, turning yesterday’s compliant headset into tomorrow’s contraband.
• Bandwidth — A glitchy 3‑G connection can morph a holographic drill guide into a pixelated guess, especially in remote Amazon or rural clinics.
• Learning Curve — Senior consultants may dismiss “video‑game medicine”, while residents can’t imagine life without it—creating a generational rift in OR workflow.

FINAL THOUGHTS

The metaverse will never suture a cuff or reduce a fracture, but it will change how we perceive and plan those tasks. Tool adoption in surgery follows a familiar arc: ridicule, resistance, acceptance, then dependence. With mixed reality we are somewhere between acceptance and dependence. My advice? Step in early while curiosity outweighs cynicism. Train with the visor, question its metrics, log your complications. Only then can we steer the tech toward patient benefit rather than novelty.

Bruno Borralho Gobbato MD

IDOMED Jaraguá

Jaraguá do Sul, Brazil

Contact: bgobbato@gmail.com

References

1. Gobbato B, Garcia JC. Mixed Reality for Shoulder Arthroscopy. J Clin Med. 2024;13:x.
2. Calem DB, Lubiatowski P, Trenhaile S, Gobbato B, Alkhateeb J, Erickson J. Mixed reality applications in upper extremity surgery: the future is now. EFORT Open Rev. 2024;9:1034–46. https://doi.org/10.1530/EOR-24-0080
3. Lubiatowski P, Stupnicki S, Nizinski J, Tuczynski P, Roszak K, Lubiatowski B, et al. Mixed reality guidance in humeral osteotomy provides superior precision and accuracy: validation and comparative study. Abstract presented at: ICSES 2023; 2023 Sep 5; Rome, Italy.
4. Scalise JJ, Codsi MJ, Bryan J, Brems JJ, Iannotti JP. The influence of three-dimensional computed tomography images of the shoulder in preoperative planning for total shoulder arthroplasty. J Bone Joint Surg Am. 2008;90:2438–45. https://doi.org/10.2106/JBJS.G.01341
5. Shapiro TA, McGarry MH, Gupta R, Lee YS, Lee TQ. Biomechanical effects of glenoid retroversion in total shoulder arthroplasty. J Shoulder Elbow Surg. 2007;16(3 Suppl):S90–5. https://doi.org/10.1016/j.jse.2006.07.010
6. Nyffeler RW, Sheikh R, Atkinson TS, Jacob HAC, Favre P, Gerber C. Effects of glenoid component version on humeral head displacement and joint reaction forces: an experimental study. J Shoulder Elbow Surg. 2006;15:625–9. https://doi.org/10.1016/j.jse.2005.09.016
7. Iannotti J, Baker J, Rodriguez E, Brems J, Ricchetti E, Mesiha M, et al. Three-dimensional preoperative planning software and a novel information transfer technology improve glenoid component positioning. J Bone Joint Surg Am. 2014;96:e71. https://doi.org/10.2106/JBJS.L.01346
8. Kriechling P, Loucas R, Loucas M, Casari F, Fürnstahl P, Wieser K. Augmented reality through head-mounted display for navigation of baseplate component placement in reverse total shoulder arthroplasty: a cadaveric study. Arch Orthop Trauma Surg. 2023;143:169–75. https://doi.org/10.1007/s00402-021-04025-5
9. Sanchez-Sotelo J, Berhouet J, Chaoui J, Freehill MT, Collin P, Warner J, et al. Validation of mixed-reality surgical navigation for glenoid axis pin placement in shoulder arthroplasty using a cadaveric model. J Shoulder Elbow Surg. 2024;33:1177–84. https://doi.org/10.1016/j.jse.2023.09.027
10. Rojas JT, Jost B, Zipeto C, Budassi P, Zumstein MA. Glenoid component placement in reverse shoulder arthroplasty assisted with augmented reality through a head-mounted display leads to low deviation between planned and postoperative parameters. J Shoulder Elbow Surg. 2023;32:e587–96. https://doi.org/10.1016/j.jse.2023.05.002

Header Image by filip bossuyt (Cropped)