Anatomical site relocalisation using dual data synchronisation

1. A medical system enabling a manual or automatic repositioning of a flexible endoscope at a specific location during at least one new exploration subsequent to a first or reference exploration during which said specific location in a tubular, at least partially deformable organ has already been explored by said endoscope, said medical system comprising: a flexible endoscope at least equipped with an image taking device; a display and movement controller associated with said endoscope; an end tip tracking device or system providing a 3D position and orientation of an end tip of the endoscope, which is affixed to or mounted into the end tip of the flexible endoscope; and a computer and associated storage, the storage being configured to store a reference exploration video acquired through the flexible endoscope during the first or reference exploration during which the specific location in the tubular, at least partially deformable organ has already been explored by the endoscope, wherein each of a plurality of captured video images captured during the new exploration and the reference exploration is associated with a respective synchronously-recorded endoscope tip 3D position and orientation, wherein the medical system is configured to: compute a spatial correspondence between at least one of the endoscope 3D tip positions of the new or subsequent exploration and at least one of the endoscope 3D tip positions of the reference exploration, search for a spatially closest neighbor image of the reference exploration video that is spatially closest to the specific location during the new exploration as a best matching image of the first or reference exploration, and display, in parallel on or in two different windows on one or more screens, a live image of a new or subsequent exploration video acquired through the flexible endoscope during the new exploration and the best matching image of the reference exploration video.

2. The medical system according to claim 1 , further comprising at least two markers, whose positions are recordable with the end tip tracking device or system or a marker tracking system separate from the end tip tracking device or system, the at least one markers being configured to be placed on given anatomical locations on the subject having the tubular organ, said anatomical locations depending on the explored tubular organ and not undergoing noticeable deformation or displacement when the subject changes position, said markers providing referential points in the previous and subsequent explorations.

3. The medical system according to claim 1 , further comprising one or more processors configured to provide real-time endoscopic video analysis, perform an extraction and a matching of similar features visible during both the first or reference exploration and the subsequent or new exploration.

4. The medical system according to claim 1 , further comprising a human interface device configured to receive an input to precisely select a pixel or a group of pixels in a video, in order to perform an interactive virtual tagging of one or more images of the first or reference exploration flexible endoscopic video indicating specific points of interest in the body of the explored subject, the virtual tagging being realized from a specific reference exploration video reader that receives input to add tags in any selected image of the video.

5. The medical system according to claim 1 , further comprising one or more processors configured to execute a video analysis algorithm to extract visible points of interest, the visible points of interest being anatomical, pathological or surgical specific features in the body of the explored subject and to realize an automatic virtual tagging on one or more images of the reference exploration flexible endoscopic video, indicating said specific points of interest.

6. The medical system according to claim 4 , further comprising one or more processors configured to analyze the subsequent new exploration video, in order to localize in the subsequent new exploration video the same specific point of interest in the body of the explored subject tagged in an image of the reference exploration video, and to add the same specific point of interest in the same location on the synchronized subsequent new exploration video, with a superimposition of the virtual tags.

7. The medical system according to claim 1 , further comprising one or more processors configured to fuse two synchronized flexible endoscopic videos, enhanced or not by virtual tags indicating specific points of interest, using augmented reality techniques with a virtual rendering of the body 3D models of the anatomical and/or pathological structure, the 3D models being extracted from a preoperative medical image realized before the new exploration of the body of the subject.

8. The medical system according to claim 1 , wherein the flexible endoscope is further equipped with at least one tool or instrument, and the tracking device or system is an electromagnetic tracking device or a fiberoptic tracking device.

9. A method for repositioning, realized by the medical system according to claim 1 , one or more times, at the specific location, which has previously been explored during the first or reference exploration, the flexible endoscope during one or more successive endoluminal or extraluminal subsequent explorations of the at least one new exploration the method comprising: synchronizing, by a synchronization process between a subsequent exploration flexible endoscope video and a reference exploration flexible endoscope video displayed in parallel on or in two different windows, on one or more screens.

10. The method according to claim 9 , wherein the synchronization process is based only on the position and orientation registration of the end tip of the flexible endoscope recorded during the first or reference exploration and the subsequent or new exploration from the tracking device, providing the location and orientation of said end tip of the endoscope according to a fixed external reference frame.

11. The method according to claim 9 , wherein the synchronization process is based on the position and orientation registration of the end tip of the flexible endoscope recorded during the reference exploration and the subsequent or new exploration from the tracking device or system, improved by a real-time endoscopic video analysis, performing an extraction and a matching of similar features visible during both the first or reference exploration and the subsequent or new exploration.

12. The method according to claim 9 , wherein the two synchronized flexible endoscopic videos are enhanced by virtual tags indicating specific points of interest, the virtual tags being defined on the first or reference exploration flexible endoscopic video interactively by a user and in the subsequent or new exploration flexible endoscopic video automatically due to image analysis based on the comparison of the two synchronized videos.

13. The method according to claim 9 , wherein the two synchronized flexible endoscopic videos are enhanced by virtual tags indicating specific points of interest, the virtual tags being defined on the first or reference exploration flexible endoscopic video automatically by an automatic video image analysis extracting visible points of interest including anatomical, pathological or surgical specific features, and in the subsequent or new exploration flexible endoscopic video automatically due to image analysis combining a comparison algorithm of the two synchronized videos with an automatic video image analysis extracting the same visible points of interest in the subsequent or new exploration flexible endoscopic video.

14. The method according to claim 9 , wherein the two synchronized flexible endoscopic videos, enhanced or not by virtual tags indicating specific points of interest, are fused using augmented reality techniques with a virtual rendering of body 3D models of an anatomical and/or pathological structure, the 3D models being extracted from a preoperative medical image realized before the subsequent or new exploration of the body of the subject.

15. The method according to claim 10 , wherein the synchronization process comprises: recording the location of the flexible endoscope end tip using a tracking system providing the location and orientation of the end tip of the endoscope according to a fixed external reference frame such as an electro-magnetic, fiberoptic or similar tracking device, and the associated flexible endoscope video during the endoluminal or extraluminal first or reference exploration, registering the location of the flexible endoscope end tip using the end tip tracking device or system providing the location and orientation of the end tip of the endoscope according to a fixed external reference frame, and the associated flexible endoscope video during an endoluminal or extraluminal subsequent or new exploration, and synchronizing the subsequent or new exploration video and the first or reference exploration video from or using the registration of the flexible endoscope end tip location between the real-time subsequent exploration and the first exploration.

16. The method according to claim 15 , wherein the synchronizing comprises synchronizing the subsequent or new exploration video and the first or reference exploration video from or by a registration of the flexible endoscope end tip location between the real-time exploration and the reference exploration, improved by a real-time flexible endoscopic video analysis extracting and matching one or more similar features visible in both first reference exploration and real-time subsequent exploration.

17. The method according to claim 15 , wherein the synchronization comprises: recording the location of the flexible endoscope end tip using the end tip tracking device or system providing the location and orientation of the end tip of the endoscope according to a fixed external reference frame, and the associated flexible endoscope video during the endoluminal or extraluminal first or reference exploration, adding an interactive virtual tagging of one of more images of the first or reference exploration flexible endoscopic video indicating specific points of interest in the body of the explored subject, the virtual tagging being realized from a specific first or reference exploration video reader that allows a user to add tags in any selected image of the video through any interaction or human interface device that allows the user to precisely select a pixel or a group of pixels in a video, registering the location of the flexible endoscope end tip using the end tip tracking device or system providing the location and orientation of the end tip of the endoscope according to a fixed external reference frame, and the associated flexible endoscope video during an endoluminal or extraluminal subsequent or new exploration, and synchronizing the subsequent or new exploration video and the first or reference exploration video from or using the registration of the flexible endoscope end tip location between the real-time subsequent exploration and the first exploration, the registration process being improved by adding an analysis of the subsequent or new exploration video localizing in the subsequent or new exploration video the same specific points of interest in the body of the tagged explored subject, and adding the analysis in the same location on the synchronized subsequent or new exploration video, with a superimposition of the virtual tags.

18. The method according to claim 15 , wherein the synchronization process comprises: recording the location of the flexible endoscope end tip using the end tip tracking device or system providing the location and orientation of the end tip of the endoscope according to a fixed external reference frame, and the associated flexible endoscope video during the endoluminal or extraluminal first or reference exploration, adding an automatic virtual tagging on one or more images of the first or reference exploration flexible endoscopic video, indicating specific points of interest in the body of the explored subject, the virtual tagging being preferably realized automatically through a specific video analysis algorithm, based on one or more of color, brightness, contrast, and textures analysis and based on features tracking, extracting visible points of interest, registering the location of the flexible endoscope end tip using the end tip tracking device or system providing the location and orientation of the end tip of the endoscope according to a fixed external reference frame, and the associated flexible endoscope video during an endoluminal or extraluminal subsequent or new exploration, and synchronizing the subsequent or new exploration video and the first or reference exploration video from or using the registration of the flexible endoscope end tip location between the real-time subsequent exploration and the first exploration, the registration process being improved by adding an analysis of the subsequent new exploration video, localizing in the subsequent or new exploration video the same specific point of interest in the body of the explored tagged explored subject, and adding the analysis in the same location on the synchronized subsequent new exploration video, a superimposition of the virtual tags, the same point of interest detection algorithm.

19. The method according to claim 9 , wherein the exploration with the flexible endoscope is performed in a tubular organ of a human subject.

20. The method according to claim 9 , wherein at least two markers, whose positions can be recorded with the end tip tracking device or system or a marker tracking system separate from the end tip tracking device or system, are previously placed on given anatomical locations on the subject, said locations depending on the explored tubular organ and not undergoing noticeable deformation or displacement when the subject changes position, are used to provide referential points in the previous and subsequent explorations, together with at least one other internal easily identifiable anatomical point.

21. The method according to claim 20 , wherein the referential points are used to define and attach a frame or referential to the subject in the first and subsequent explorations.

22. The method according to claim 9 , wherein, during the first or previous exploration, a limited number of images are recorded or selected, which contain relevant information, marked or tagged, and wherein, during the subsequent exploration, the video image processing and synchronizing is performed, and guidance information provided, only when the current or live endoscope position is close to the position associated with images containing the relevant information.

23. The method according to claim 9 , wherein, when a tubular organ is explored, an image processing is performed which analyzes a lumen position in the video image at a time t during the subsequent exploration and selects the image from the previous or reference exploration associated to a position close to the subsequent live position with a similar lumen position.

24. The method according to claim 9 , wherein, when the explored organ is a colon, three points or parts of the colon which are attached to an abdominal wall are used as fixed reference points or locations, the reaching of a target during a subsequent exploration being performed through backward motion.

25. The method according to claim 9 , wherein the orientation of the tip of the flexible endoscope is recorded, and exploited, in addition to its 3D position, by the end tip tracking device or system, to evaluate the rotation difference of the endoscope orientation between the reference and the subsequent exploration.

26. The method according to claim 9 , wherein precise relocalization of the tip of the flexible endoscope is performed through two consecutive operational steps: a gross localization, by performing an approximate positioning of the tip of the flexible endoscope close to a reference point or position determined in a previously-conducted procedure; and a fine positioning, by referring to a mapping of a target site or points, taken during a previous procedure onto the video images in the subsequent or current exploration.