Collaborative Virtual Surgery : Techniques , Applications and Challenges

While considerable effort has been dedicated to improve medical education with virtual reality based surgical simulators, relatively little attention is given to the simulation of the collaborative procedures in distributed environments. In this paper, we first present a literature review of techniques involved in the development of collaborative simulators, including network architecture, transmission protocol, collaboration mechanism, schedule algorithm, collaborative user-interaction feature and haptic communication. We introduce the details of each technique and discuss the advantages and drawbacks. Then, we review some of the existing applications to illustrate how to apply these techniques to implement an efficient and robust collaborative simulator. Finally, we discuss the challenges that need to be addressed in the future.


INTRODUCTION
In conventional surgical training practice, surgeons learn the techniques using animals, cadavers, volunteers and patients.Although recent years have witnessed rapid technological advance in these procedures, the effectiveness and safety of these traditional methods have been questioned.For example, the anatomy of animals and human is different and cadavers cannot provide appropriate physiological responses.The surgical training performances on patients or volunteers are hazardous and any errors may do great harm to their health [1].
In recent decades, virtual reality (VR) based medical simulations have become more and more prevalent considering the system reusability, cost effectiveness and the better learning efficiency they offered to medical education.Considerable research efforts have been dedicated to the development of computer-assisted surgical simulators to enable doctors and students to learn human anatomy and physiology in a virtual environment.These simulators could assist in surgical planning, training and rehearsal, as well as the investigation of new medical procedures.
While many VR-based surgical simulation systems have been developed, most of them focus on the simulation of surgical interventions involving a single user in the virtual Manuscript Received on 11 May， 2010 Email: jqin@cse.cuhk.edu.hkenvironment.In reality, surgical operation is usually a teamwork requiring a group of medical practitioners to cooperate with each other.In this regard, collaborative architecture is a necessary component of future surgical simulators.
On the other hand, with the development of computer network and distributed system, multiple users can carry out a common task collaboratively in a distributed environment [2].Incorporating information sharing, 3D visualization and user-interaction, the distributed environments evolve to collaborative virtual environments (CVE).CVEs have found many applications in real life such as collaborative design, multi-user computer games and tele-conferences.In particular, collaborative surgical simulation is becoming a promising application of CVEs in computer-assisted medicine.
Nevertheless, it is a challenging task to design and implement a high performance distributed surgical simulation system because of the sharp conflict between the requirement of maintaining high levels of state consistency and the limitation of network transmission capacity, in particular when every practitioner involved in the collaborative virtual surgery should be informed of any events and the updated information of virtual objects in a real-time level [3]- [5].This is not a small consumption of network resources.Furthermore, tough network conditions such as restricted network bandwidth, stochastic and unbounded network delay and uncertain network reliability exacerbate the situations [6].The situation is further worsened by the requirement of real-time interactions where time-consuming modeling and simulation are prohibitive.To meet these challenges, a collaborative surgical simulation system should be able to reduce the effect of network latencies on its performance, manage events exchange, maintain system consistency, support reliable collaboration mechanism and ensure the availability, accuracy and extensibility of the collaborative simulation.
It is therefore critical to contrive robust network architecture to enable efficient collaboration and consistency control, where adaptive protocols are developed for mediating flow of events among the practitioners, synchronizing geographically distributed databases and reducing the insurmountable communication latencies.Work has been done recently to enhance the collaboration in surgical simulation.However, comprehensive survey of related techniques is not readily available.This paper is a brief survey of collaborative virtual surgery; our aim is to present the documented approaches, mechanisms and architecture and to share our own experiences with researchers and designers to help them to develop collaborative surgical simulation systems and extend existing single-alone systems to collaborative ones.
The next section presents a literature review of essential issues involved in the development of collaborative simulators, including network architecture, transmission protocol, collaboration mechanism, schedule algorithm, collaborative user-interaction feature and haptic communication.We provide the details of each technique and discuss their advantages and drawbacks.Section 3 reviews some existing applications, including the authors' work, to illustrate how to apply these techniques to implement an efficient and robust collaborative simulator.Section 4 discusses some of the challenges that need to be addressed in the future.

II. TECHNIQUES
A. Network Architecture Network architecture supporting collaborative virtual environments has conventionally been dominated by client-server (CS) and peer-to-peer (P2P) strategies.To decrease network latencies, some CVEs systems adopt P2P network architecture such as DIVE [7] and SIMSET [8].Although the P2P architecture can minimize the latency, consistency is not guaranteed because of the absence of a server as an arbiter.Meanwhile, Client-Server (CS) architecture is adopted in some CVEs applications such as MASSIVE-3 [9] and NAVL [10] to maintain system consistency, but the network latency is relatively high because of the transmission bottleneck at the server.Many techniques have been developed to decrease the latencies in CS-based CVEs.For example, in Distributed Open Inventor (DOI) [11], each client has a copy of the global scene graph and the synchronization is done by using replicated scene graph protocol in which some local variations on the graph, e.g.different levels of detail, are allowed.
Nevertheless, neither of them can sufficiently support collaborative surgical simulation where high level of consistency and minimal network latency are both needed.In [12], the authors proposed a multi-servers architecture to avoid potential inconsistency in simulating cooperative tasks while supporting low-latency interactions.Other similar hybrid architectures include PaRADE [13] and SPLINE [14], where multiple servers with coordinated data flow between them can provide scalability and redundancy [15].However, in these architectures, with the increase in the number of clients attached to one of the servers, network latency is still a problem.In addition, when the number of servers is large, synchronization between these servers is a time-consuming task.
In one of our previous projects, we present an adaptive framework to support real-time collaboration in surgical simulation [16], where cluster-based hybrid network architecture is proposed to decrease network latency using a circular topology, where the consistency of the system is maintained through an administrative server.Cluster architecture is commonly employed in the areas of parallel and distributed systems.They are usually implemented for load balancing and high performance computing [17] [18] [19].In the computer graphics and virtual reality research, cluster architecture is mostly used to support parallel visualization and the synchronization of avatar migration in virtual environment [20] [21] [22].The objective of this framework, on the other hand, is to support real-time 3D user interactions in collaborative surgical simulation.The reliability of this framework is achieved by the failure detection chain and master-slaver model.In the failure detection mechanism, a specific monitor is assigned to each participant based on the circular topology, which is different from similar approaches where a small number of randomly chosen nodes are delegated as monitors [23].

B. Protocols and Schedule Algorithm
Another important component of collaborative virtual surgery is the communication protocol developed for efficient network management and the handling of the message flow that is specific to the interactions during collaborative surgical simulation.The most common transport protocols are transport control protocol (TCP) and user datagram protocol (UDP).Although TCP can guarantee reliable transmission because of its "flow control" mechanisms, its latency is usually higher than UDP for the same reason [24].There is thus a trade-off between the reliability and the speed in the choice of a suitable transmission protocol.
In our previous work [16], we use two communication mechanisms in the above-mentioned cluster-based architecture.When participants transmit the updated information end-to-end according to the topology view, TCP protocol is used to ensure communication reliability.For some surgical simulators where computation-intensive deformable models are involved, a multicast mechanism using UDP protocol could be adopted to save network resources and reduce network latency.Reliable message exchange on top of the multicast transmission is maintained by using two techniques: distributed message acknowledgment (DMA) and sliding window (SW).
In a later work [25], we further propose an adaptive transmission protocol to timely and accurately transmit force information in a distributed virtual environment, as it is of great importance in sharing dynamic states among users.A hybrid derivative polynomials prediction algorithm is proposed in force prediction to reduce network latencies.When the function of the force is known, the system can choose a proper order of derivative polynomial based on the network bandwidth.If the function of the force is unknown, we use zero-order polynomial to predict the magnitude and direction of the force in the current time step until the updated information arrives.Meanwhile, state regeneration is used to correct the estimated prediction.
Besides transport protocols, it is also necessary to design novel and specific application-level protocols to fulfill the special requirements of collaborative virtual surgery.For example, in [26] token control messages among all participants.The procedures involved in distributed deformable simulation can thus be implemented by using a combination of these five types of primitives.Other famous application-level protocols for distributed virtual environment include the virtual reality transfer protocol (VRTP) [27] and the distributed worlds transfer and communication protocol (DWTP) [28] and the interactive sharing transfer protocol proposed by Waters et al [29].

C. Collaboration and Consistence Mechanism
Besides network architecture and communication protocols, data management mechanisms for collaboration and consistency are necessary for collaborative virtual surgery.In one of our early work [4], we propose a new attitude dead reckoning mechanism in a collaborative medical visualization system (CMVS).Using quaternion as the description of attitude, we derive a general trajectory construction scheme of attitude that extrapolates a number of previous packets in order to form the future trajectory of objects.An adaptive prediction and convergence approach is developed based on this cumulative trajectory.The method allows smooth transition between consecutive attitudes obtained from the network.
In our recent work [16], we implement several mechanisms to support efficient and flexible collaboration based on our cluster-based architecture.First, our framework is optimized by switching between two computational policies depending on network traffic and resources.Second, to maintain the consistency of the common scenes among all participants, collaborative mechanisms including coupling control and token control are developed.The coupling control mechanism decides which operations should be executed collaboratively among all users and which operations should be handled individually by the participants.Thus, users can filter some messages by defining them as uncoupling messages to reduce the traffic in the network environment.The well-known token-control mechanism is integrated to regulate the operations of the multiple participants performing on the common virtual organs.Third, a flexible transmission policy is provided for view synchronization.Other techniques should be investigated for collaboration and consistency, including load balancing [30], state synchronization [31], prediction and convergence [32] and relevance filtering [33].

D. Haptic Communication
While visual feedback is the most important perception channel of VR-based surgical simulation systems, haptic force feedback is proven to be essential for enhancing the sense of presence and immersion in virtual environment [34] [35].In this case, collaborative surgical simulators should address the issues on the synchronization of visual-haptic feedback in a distributed environment.
A number of strategies have been proposed in attempt to achieve the level of consistency and synchrony required for networked haptic virtual environment by reducing the effect of network delay, packet loss and jitter.Global time coordinate frame [36] has been employed to maintain state consistency whereas sequence number is introduced as timestamp to haptic data so that users can determine the amount of network delay by subtracting the sequence number associated with the received data packets.Decorators, i.e. visual cues such as changing the color of haptic cursor, can be provided to inform users of the network delay [37].Virtual time and rollback mechanism have been implemented by using a system of logical clocks to adjust the simulation time adaptively according to network conditions [36].
To counteract network delay jitters, virtual-time rendering (VTR) media synchronization algorithm is adopted to dynamically adjust the packet transmission rate with respect to the network load [38], where virtual-time contraction and expansion, skipping and pausing [39], are applied appropriately.Buffering schemes are also devised to tackle the issue by rearranging the data packets stored in a buffer in the order of sequence number [40][41].Analogous to the transmission of audio and videos signals, media synchronization mechanisms [42] are proposed to counteract the effect on haptic media transmission when network delay is bounded [43] or unknown [44].
Owing to the high temporal correlation of position data in haptic rendering, dead reckoning has been used estimate lost data from previously received data [37] [42], unless the delay jitters become so significant that destroy the correlation.Linear prediction schemes are also developed to estimate missing in-between data by interpolation [45].Furthermore, force averaging algorithms have also been developed to smooth the feedback forces generated from noisy haptic information received from the network [46].

III. APPLICATIONS
In this section, we attempt to recount some important or impressive projects in the history of using CVEs and related techniques in medical applications, from which we can see the development of related techniques and the challenges that we are still facing now.

A. Distributed Medical Consultation and Diagnosis Systems
In the last decade, with the development of computer network, some distributed medical consultation and diagnosis systems have been developed, targeting to facilitate the exchange of medical information and diagnosis report, or to allow for the co-investigation of medical data among multiple users.Bai et al. developed a Java-based teleconsultation system to demonstrate the use of the World Wide Web (WWW) for telemedicine and interactive medical information exchange [47].Makris et al. described methods for the design of both synchronous and asynchronous computer-supported cooperative work (CSCW) procedures suitable for the medical application area and specifically for the purpose of medical teleconsultation and remote diagnosis support [48].Subsequently, Steele et al. described a simulation-based medical planning environment developed for the WWW using the Virtual Reality Modeling Language (VRML) and the Java programming language [49].However, these systems do not support collaborative real-time 3D user interactions efficiently and cannot be used to carry out collaborative surgical simulation.

B. Co-Surgeon System
Yeongho Kim et al. proposed a CS-based architecture called Co-Surgeon which combines 3D surgery simulation with CSCW technology to enable surgeons to simulate treatment plans collaboratively [50].Co-Surgeon enables multiple users in remote locations to manipulate 3D anatomical models and to simulate surgical operations while sharing a view of the simulation.In addition, the system can store the simulated procedure so that offline users can later replay it and participate asynchronously.One of the main disadvantages of Co-Surgeon is that the transmission bottleneck at the server results in additional latency, especially when complicated surgical operations are involved.

C. Collaborative Simulation of Temporal Bone Surgery
Dan Morris et al. implemented a collaborative virtual environment for the simulation of temporal bone surgery [51].The system allows two users to independently observe and manipulate a common model, and allows one user to experience the forces generated by the other upon contact with the bone surface.This permits an instructor to remotely observe a trainee and provide real-time feedback and demonstration.But this system can only support two users to interact with a generic bone model.In reality, a lot of interventions need more than two surgeons to cooperate with each other.

D. Surgical Simulator Development Toolkits
Some surgical simulation projects have provided modules to support networked virtual environments, such as Gipsi [52], Spring [53], and SOFA [54].However, their goal is primarily to integrate heterogeneous models or processes seamlessly and give little attention to collaboration.Their network modules do not have much concern on the development of CVE techniques.

E. A Serious Game for Blood Management
Recently, Qin et al. have presented a serious game for learning blood management in orthopedic surgery [55].This game includes networking capability to support players' collaboration on operative tasks based on a specifically designed CS architecture.In this architecture, the clients interact with shared virtual models, whereas the server manages the network connections among clients, schedules the events, and transmits updated information to participants.On the server side, we built a service manager module that handles system initialization and manages network services.The server maintains a topology view of the clients, and players can join the game until maximum capacity is reached.To buffer messages and transmit them to the corresponding destinations, we used first-come-first-served queuing.We also developed a communication protocol on the basis of primitive commands-for connection operations, transformation operations, deformation operations, and so on-which we predefined using the game system's current requirements.As mentioned, the transmission bottleneck at the server side prohibits its application in scenarios with large number of participants.

IV. CHALLENGES AND FUTURE DIRECTIONS
In this section, we explain current challenges we face and present some emerging approaches and future directions to address these challenges.

A. Robustness of Network Architecture
We shall further investigate techniques to ensure the robustness of our cluster architecture, without compromising its efficiency.For example, we have implemented a failure detection chain based on the cluster topology, where each participant sends heartbeat signals to its monitor at a regular interval.However, more simulation and experiments should be conducted to determine the optimal heartbeat signal rate in various network conditions.Mechanisms to further ensure the reliability of the system when a participant and its monitor are powered off at the same time will also be investigated.Robustness is essential for all architectures, as during collaborative simulation, it is likely that some failed instances, caused by the malfunction of software or network connection, may block the message transfer in the cluster and affect the progress of simulation.This may have great impact on the effect of collaborative surgical simulation, for surgical training or planning.

B. Sophisticated Techniques for Reliable Multicast Transmission
In collaborative surgical simulators and planners, reliable multicast transmission is necessary to save network resources and reduce network latency.In our previous work, we have implemented reliable multicast transmission.In the future, we shall study the use of artificial intelligence techniques based on the number of participants and network conditions to identify the optimal parameters in our implementation.

C. Support of dynamic shared state management for visual-haptic communication
Robust network architecture and reliable multicast transmission are insufficient to support collaborative surgical simulation and planning, although they are already hard research tasks.It is also necessary to provide modules to support efficient and flexible collaboration in various environments.We aim at developing and elaborating two key mechanisms.First, two computational policies will be implemented and refined, namely local-computation policy where computation-intensive tasks will be off-load to individual participant, and global computation policy where a participant with powerful computational capability handles the computation and distribute the results to other participant.We will develop approaches to support dynamic switching between these two policies based on network traffic and resources.Second, relevance filtering will be investigated to analyze the semantics of packets on transmission or reception, thereby selecting only those that match a pre-defined criterion (our system is an example), deciding which operations should be executed collaboratively by all users and which operations should be handled individually by the participants themselves.This relevance filtering can be used to control data flow and reduce network traffic.

D. Multi-component objects and multi-resolution model
To enhance the flexibility of the collaborative system and further reduce network traffic, study will be conducted to split a physical object into a collection of independent components, each of them describing one feature of the model, so that only relevant information of a state update is transmitted to other participants.Besides, a multi-resolution model shall be investigated to represent virtual tissues with different levels of details using variable mesh structures.Thus, flexible transmission schemes can be implemented based on it.

E. Comprehensive Software Platform
Finally, we will modularize and package above-mentioned techniques as a software platform to allow rapid prototyping of a collaborative surgical simulation and planning system by making use of the application programming interface provided.We will further demonstrate the viability of our software platform by extending our existing stand-alone single-user simulators into collaborative systems.

V. CONCLUSION
Recent years have witnessed significant progress in virtual reality based surgical simulation and planning.However, development of an interactive and robust simulator is still a difficult task due to the involvement of many labor and computation intensive modules.The work is even more demanding when virtual surgery system is extended to simulate co-operative work among multiple medical professionals over the network.
In this paper, we briefly reviewed the essential techniques involved in the development of collaborative surgical simulators and presented the challenges.We share a lot of our experiences and views in this area.We believe the software platform proposed in last subsection, once established, will free engineers and surgeons from tedious implementation of network infrastructure and collaborative mechanisms so that they can focus on the design and simulation of surgical tasks.Collaborative systems constructed based on the software platform can facilitate real-time collaboration of surgeons all over the world.In particular, it will potentially benefit the patients in rural areas of developing countries, where the level of medical care are usually not as high as those in cities, by delivering professional clinical care through the network in an interactive manner.
, a communication protocol for collaborative deformable simulation is developed with the definition of a number of primitive operations.The primitives of the protocol are categorized into five distinct types, namely, connection The functions of these five operations are defined in the following: • Connection operations: monitor and manage connections (e.g.accept, establish or release a connection), and synchronize the clocks of clients and server.• Configuration operations: initialize and distribute the common virtual models as well as the configuration parameters, keeping them consistent among the clients in the distributed collaborative virtual environment.• Transformation operation: handle transformation manipulations such as translation, rotation and scaling.• Deformation operations: handle the flow of events and synchronize the nodal position of the shared model after various deformation manipulations among clients.• Collaboration operations: transmit coupling control and