Intra-operative visual guidance through AI

Understanding Computer Vision in Surgery

Computer vision, often considered a subfield of AI, focuses on enabling machines to automatically interpret and understand visual information from the world, much like humans. In the context of intra-operative surgical applications, computer vision mostly involves the analysis of video streams, which are mainly derived from laparoscopic or robotic endoscopic systems. Traditional computer vision methods that did not use AI laid the groundwork, but the integration of AI has elevated mankind’s capacity to understand complex situations present in surgery to unprecedented levels.

Enhancing Precision through AI Algorithms
Machine learning models, and particularly deep neural networks, are trained on vast datasets to recognise patterns, anomalies, and structures within images. This enables the algorithms to process real-time visual information, offering surgeons valuable insights and guidance. In real-time surgical guidance, AI algorithms hold the potential to accurately identify and highlight critical structures such as blood vessels, nerves, and organs. This can enhance a surgeon's ability to navigate complex anatomical structures with precision, reducing the risk of inadvertent damage and improving overall surgical outcomes. Even more important is the potential of this technology for surgical training and teaching, where AI, having seen thousands of cases,is looking over junior surgeons’ shoulders.

At present, such computer vision based systems have already proven capable in other domains, such as identifying planes of dissection for cholecystectomy [1] and the improved detection of colonic polyps [2], while similar approaches are being investigated for urothelial cancer during transurethral bladder tumor resection [3].

Objective Surgical Feedback and improved training
AI is expected to take on a leading role for surgical teaching and training [4]. While a future is to be expected where surgeons get real-time feedback on errors, or even the prevention of errors, today objective feedback consists of intelligent post operative case reviews. AI has shown the potential to differentiate between surgical expertises using
GEARS, but the explainability of such approaches generally stays low. Furthermore, such methods have recently been shown to be lacking when compared to other proficiency metrics and teaching methodologies, such proficiency based progression (PBP) [5]. As PBP focuses on the errors made by trainees, new AI systems for surgical teaching should prioritise error detection as an explainable step forwards toward automated skill assessment. Already today, AI systems are capable of screening every frame in a surgery. As such, they can help classify phases for instant review and assessment [6] and can help compare current performance to previous cases, or to peers, to facilitate inter-surgeon learning and knowledge sharing.

Furthermore, AI can also help in safe and private streaming of endoscopic video streams. Indeed, novel real-time AI algorithms are being developed which automatically blur the endoscopic view during phases such as port placement or camera cleaning.

Augmented Reality and Surgical Navigation
AI's role in computer vision goes beyond image analysis; it extends into augmented reality (AR) for surgical navigation. By overlaying pre-operative imaging information, such as 3D models derived from CT scans, onto the surgeon's field of view, AR can provide an immersive and intuitive way for surgeons to visualise critical structures during surgery. During these EAU live cases, multiple surgeries will be displayed using AR, but the attentive viewer will notice that surgical and 3D model alignment is, at present, far from perfect. This is where AI can benefit AR. Already AI has been shown to solve problems in AR such as instrument occlusion [9], as depicted in Figure 1.b.

Improved Efficiency and Reduced Workload
The automation capabilities provided by AI-powered computer vision contribute to improved efficiency and reduced workload in the operating room. Surgical phase analysis in particular is expected to improve planning and operating room logistics, while the novel robotic driven systems now hitting the market allow roboticized and intelligent instrument tracking, obliviating the need for surgical personnel holding endoscopes whilst providing a constant and stable image [10].

Future Directions and Innovations
The field of AI in computer vision for real-time surgical guidance is still in its infancy. Nonetheless, at EAU24 we will see first-hand useful, clinical, real-time AI applications already impacting patient care.

Future developments will include the integration of AI with robotics to leverage multiple sensor inputs to the maximal capacity [11]. Advancements in AI explainability and interpretability are needed to enhance the trustworthiness of AI systems, further encouraging their adoption in surgical practice.

Furthermore, collaborative efforts between urologists, engineers, data scientists and other healthcare professionals are crucial for developing AI solutions that seamlessly integrate into existing surgical workflows. Interdisciplinary collaboration ensures that AI technologies are not just technologically advanced, but first and foremost align with clinical needs and surgical practices.

Conclusion
The role of AI in computer vision for real-time surgical guidance is leading a paradigm shift in the field of surgery. From enhancing precision and navigating anatomical variability to providing real-time feedback and enabling augmented reality, AI is reshaping the way urologists approach and perform procedures. As technology continues to advance and ethical considerations are addressed, the synergy between AI and surgery holds the promise of safer, more efficient, and more personalised patient care. Embracing these innovations with a cautious and collaborative approach will undoubtedly lead to a future where AI becomes an indispensable ally in the operating room, revolutionising surgical practice and improving patient outcomes.