Cone Beam and 3d Fluoroscope Lung Navigation

The disclosure relates to methods and systems for reducing divergence between computed tomography images and a patient through the use of cone beam computed tomography imaging.

Pulmonary disease may cause one or more portions of a patient’s lungs may lose its ability to function normally and thus may need to be treated. Lung treatment procedures may be very complex and would be greatly aided if the surgeon performing the procedure can visualize the way airways and other structures in the patient’s lungs are shaped and where tools are located. Traditional pre-operative images are helpful, to an extent, with the former, but provide no guidance with regard to the latter.

Systems for displaying images and tracking tools in the patient’s lungs generally rely on pre-operative data, such as from computed tomography (CT) scans performed before, sometimes days or weeks in advance, the treatment procedure begins. However, such systems do not account for changes that may have occurred after the CT scan was performed, or for movement occurring during the treatment procedure. Systems, devices, and methods for improving on the process of identifying and visualizing a patient’s lungs, as well as structures and tools located therein, are described below.

3 3 3 3 3 The disclosure is directed to a systems and method of a method of registering an image to a luminal network including detecting a position of a sensor in a luminal network. The method of registering also includes receiving images forD reconstruction of the luminal network with the sensor within the luminal network; presenting theD reconstruction image on a user interface; receiving indication of location of target in theD reconstruction image; generating pathway through the luminal network to a target; and determining if the sensor moved from detected position following receipt of the images forD reconstruction, where when it is determined that the position of the sensor is the same as detected position, the luminal network and theD reconstruction are registered. Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the methods.

3 3 3 2 3 3 3 Implementations may include one or more of receiving survey data when it is determined that the position of the sensor has changed, registering the luminal network to theD reconstruction based on the survey data or generating aD model of the luminal network. The method may further include displaying the pathway on in theD reconstruction,D slices images derived from theD reconstruction, aD model derived from theD reconstruction, or a virtual bronchoscopy. Additionally, or alternatively, the method may further include displaying the position of the sensor along the pathway in a user interface.

3 3 3 3 3 Another aspect of the disclosure is a method of registering an image to a luminal network including receiving a pre-operative computed tomography (CT) image of the luminal network, receiving an indication of a target within the luminal network, generating a pathway through the luminal network to the target. The method of registering also includes receiving images forD reconstruction of the luminal network, transforming coordinates of the pre-operative CT image to coordinates of theD reconstruction to register the pre-operative CT image to theD reconstruction, and updating a position of a catheter in theD reconstruction image or aD model upon detection of movement of the catheter.

3 2 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 The method may further include displaying theD reconstruction,D slices images derived from theD reconstruction, aD model derived from theD reconstruction, or a virtual bronchoscopy on a user interface. In another aspect the method includes generating aD model from theD reconstruction image before transforming the pre-operative CT coordinates and theD reconstruction coordinates and may also include matching features from the CT images to theD reconstruction andD model derived from theD reconstruction. Alternatively, the method includes generating aD model from theD reconstruction after transferring the target and pathway from the pre-operative CT image to theD reconstruction. The method may include receiving survey data, where the survey data is received prior to receipt of theD reconstruction or the survey data is received after transfer of the target and pathway to theD reconstruction from the pre-operative CT image to register theD reconstruction to the luminal network.

3 3 3 A further aspect of the disclosure is a method for registering an image to a luminal network including receiving a pre-operative computed tomography (CT) image of the luminal network, receiving an indication of a target within the luminal network, generating a pathway through the luminal network to the target, generating a CTD model, detecting a position of a catheter within the luminal network, registering the pre-operative CT image to the detected position of the catheter, receiving an indication of a location of a sensor in the pre-operative CT or CTD model and update the location in a user interface until proximate the target, receiving images forD reconstruction of the luminal network, and detecting a position of the catheter and updating the position on a use interface.

3 3 3 3 3 3 3 3 3 3 3 3 3 The method may further include generating a 3D model from theD reconstruction. Still further the method may include recalling survey data from memory, presenting theD reconstruction on a user interface, receive an indication of a location of a target in theD reconstruction, and generating a pathway in theD reconstruction orD model. Still further the method may further include determining a relative position of the target and the catheter in theD reconstruction and updating the relative position of the target and the catheter in the pre-operative CT image and CTD model based on the determined relative position in theD reconstruction orD model. Additionally, the method may include registering the pre-operative CT image with theD reconstruction and transferring the target and pathway from the pre-operative CT image andD model to theD reconstruction andD model.

The disclosure is directed to a system and method for using a cone beam computed tomography (CBCT) image or a 3D fluoroscopy image in connection with intraluminal navigation techniques and systems.

3 3 3 There exist a number of systems that utilize the output from a pre-procedural computed tomography (CT) scan (e.g., CT image data) for purposes of identifying areas of interest or targets to which navigation of an endoscope or catheter is desired. Typically, this navigation will be of luminal networks such as the airways of the lungs or the biliary tract, but they could also be of spaces such as the thoracic cavity generally or other locations within a patient. These systems generally have two phases. A first phase is a planning phase where the targets are identified, and a three-dimensional (D) model is generated. A second phase is a navigation phase where the location of the catheter within the patient is detected and depicted on theD model or other images to allow the clinician to navigate to the identified targets. By updating the position of a catheter within theD model, the clinician is able to perform procedures such as biopsy or treatment at the target location. One such systems is the ILLUMISITE system sold by Medtronic PLC, which is an electromagnetic navigation (EMN) system.

1 FIG. 1 FIG. 100 100 40 100 50 30 70 80 depicts a systemsuitable for implementing methods described herein. As shown in, systemis used to perform one or more procedures on a patient supported on an operating table. In this regard, systemgenerally includes a bronchoscope, monitoring equipment, a tracking system, and a computing device.

50 50 30 50 50 90 90 92 96 96 50 90 50 90 91 96 92 96 96 50 90 92 94 96 94 93 96 94 93 96 76 72 80 Bronchoscopeis configured for insertion through the patient’s mouth and/or nose into the patient’s airways. Bronchoscopeincludes a source of illumination and a video imaging system (not explicitly shown) and is coupled to monitoring equipment, for example, a video display, for displaying the video images received from the video imaging system of bronchoscope. In an embodiment, bronchoscopemay operate in conjunction with a catheter guide assembly. Catheter guide assemblyincludes a locatable guide (LG)and catheter. Cathetermay act as an extended working channel (EWC) and be configured for insertion through a working channel of bronchoscopeinto the patient’s airways (although the catheter guide assemblymay alternatively be used without bronchoscope). Catheter guide assemblyincludes a handleconnected to catheter, and which can be manipulated by rotation and compression to steer LGand catheter. catheteris sized for placement into the working channel of bronchoscope. In the operation of catheter guide assembly, LG, including an EM sensor, is inserted into catheterand locked into position such that EM sensorextends a desired distance beyond a distal tipof catheter. The location of EM sensor, and thus distal tipof catheter, within an EM field generated by EM field generator, can be derived by tracking moduleand computing device.

92 96 99 70 70 90 94 96 70 72 74 76 76 76 74 72 74 74 74 80 81 74 1 FIG. LGand catheterare selectively lockable relative to one another via a locking mechanism. A six degrees-of-freedom tracking systemis utilized for performing navigation, although other configurations are also contemplated. Tracking systemmay be configured for use with catheter guide assemblyto track a position of EM sensoras it moves in conjunction with catheterthrough the airways of the patient, as detailed below. In an embodiment, tracking systemincludes a tracking module, a plurality of reference sensors, and an EM field generator. As shown in, EM field generatoris positioned beneath the patient. EM field generatorand the plurality of reference sensorsare interconnected with tracking module, which derives the location of each reference sensorin the six degrees of freedom. One or more of reference sensorsare attached to the chest of the patient. The six degrees of freedom coordinates of reference sensorsare sent as data to computing device, which includes an application, where the data from reference sensorsare used to calculate a patient coordinate frame of reference.

94 92 94 62 64 92 92 94 96 96 96 92 Although EM sensoris described above as being included in LG, it is also envisioned that EM sensormay be embedded or incorporated within a treatment tool, such as a biopsy toolor an treatment tool(e.g. an ablation catheter), where the treatment tool may alternatively be utilized for navigation without need of LGor the necessary tool exchanges that use of LGrequires. EM sensormay also be embedded or incorporated within catheter, such as at a distal portion of catheter, thereby enabling tracking of the distal portion of catheterwithout the need for LG.

62 64 90 92 62 70 62 62 64 66 92 92 96 1 FIG. According to an embodiment, biopsy and treatment tools,are configured to be insertable into catheter guide assemblyfollowing navigation to a target location and removal of LG. Biopsy toolmay be used to collect one or more tissue samples from the target location, and in an embodiment, is further configured for use in conjunction with tracking systemto facilitate navigation of biopsy toolto the target location, and tracking of a location of biopsy toolas it is manipulated relative to the target location to obtain the tissue sample. Treatment toolis configured to be operated with a generator, such as a radio frequency generator or a microwave generator and may include any of a variety of ablation tools and/or catheters. Though shown as a biopsy tool and microwave ablation tool in, those of skill in the art will recognize that other tools including for example RF ablation tools, brachytherapy tools, and others may be similarly deployed and tracked without departing from the scope of the present disclosure. Additionally, a piercing tool and/or puncture tool may be used with and/or incorporated in LGto create an exit point where LG, and thereby catheter, is navigated outside of the patient’s airways and toward the target location, as further described below.

20 100 20 3 3 3 20 30 3 A radiographic imaging device, such as a C-arm imaging device capable of capturing images of at least a portion of the patient’s lungs is used in conjunction with system. Radiographic imaging devicecaptures images from which aD reconstruction can be generated such as a CBCT device or aD fluoroscopy device. Generally, both CBCT images andD fluoroscopy images are captured by sweeping the radiographic imaging devicethrough a defined sweep angle (e.g.,-180 degrees and any integer value within that range). By processing the individual images or video captured during the sweep, aD reconstruction can be generated which is similar to a traditional CT image. As will be understood CBCT images have similar resolution to CT images whereas fluoroscopy images have a lower resolution.

1 FIG. 1 FIG.A 20 80 81 20 20 20 80 20 80 81 20 40 As shown in, radiographic imaging deviceis connected to computing devicesuch that applicationmay receive and process image data obtained by radiographic imaging device. However, radiographic imaging devicemay also have a separate computing device located within itself, within the treatment room or in a separate control room to first receive the image data obtained by radiographic imaging deviceand relay such image data to computing device. In one example, the radiographic imaging deviceis connected to a picture archiving and communications system (PACS) server which in turn is connected to the computing deviceand application. To avoid exposing the clinician to unnecessary radiation from repeated radiographic scans, the clinician may exit the treatment room and wait in an adjacent room, such as the control room, while radiographic imaging deviceperforms the CBCT and/or fluoroscopic scans.depicts an end view of the radiographic imaging device as it might be used to image a patient while they are laying on tablein accordance with the disclosure.

80 81 3 3 80 3 3 3 3 3 80 Computing deviceincludes software and/or hardware, such as application, used to facilitate the various phases of an EMN procedure, including generating theD model, identifying a target location, planning a pathway to the target location, registering theD model with the patient’s actual airways, navigating to the target location, and performing treatment at the target location. For example, computing deviceutilizes data acquired from a CT scan, CBCT scan, magnetic resonance imaging (MRI) scan, positron emission tomography (PET) scan, and/or any other suitable imaging modality to generate and display theD model of the patient’s airways, to enable identification of a target location on theD model (automatically, semi-automatically or manually) by analyzing the image data and/orD model, and allow for the determination and selection of a pathway through the patient’s airways to the target location. While the image data may have gaps, omissions, and/or other imperfections included in the image data, theD model is a smooth representation of the patient’s airways, with any such gaps, omissions, and/or imperfections in the image data filled in or corrected. TheD model may be presented on a display monitor associated with computing device, or in any other suitable fashion.

3 3 81 3 3 20 81 3 3 20 Though described herein generally as generating aD model from either pre-operative CT images, CBCT images, orD fluoroscopy images, applicationmay not need to generate theD model or even aD reconstruction. Instead, that functionality may reside in the computing device associated with the radio graphic imaging deviceor the PACS server. In such scenarios, the applicationneed merely import theD reconstruction orD model generated from a CT image, CBCT image, fluoroscopy images by the radiographic imaging deviceor the PACS server.

80 3 3 3 3 3 Using computing device, various views of the image data and/orD model may be displayed to and manipulated by a clinician to facilitate identification of the target location. As noted above, the target location may be a site within the patient’s lungs where treatment is to be performed. For example, the treatment target may be located in lung tissue adjacent to an airway. TheD model may include, among other things, a model airway tree corresponding to the actual airways of the patient’s lungs, and show the various passages, branches, and bifurcations of the patient’s actual airway tree. Additionally, theD model may include lesions, markers, blood vessels and vascular structures, lymphatic vessels and structures, organs, other physiological structures, and/or aD rendering of the pleural surfaces and fissures of the patient’s lungs. Some or all of the aforementioned elements may be selectively displayed, such that the clinician may choose which elements should be displayed when viewing theD model.

81 92 96 94 96 96 96 96 92 62 64 96 96 After identifying the target location, applicationmay determine a pathway between the patient’s trachea and the target location via the patient’s airways. In instances where the target location is located in lung tissue that is not directly adjacent an airway, at least a portion of the pathway will be located outside of the patient’s airways to connect an exit point on an airway wall to the target location. In such instances, LGand catheterwill first be navigated along a first portion of the pathway through the patient’s airways to the exit point on the airway wall. LGmay then be removed from catheterand an access tool, such as a piercing or puncture tool, inserted into catheterto create an opening in the airway wall at the exit point. cathetermay then be advanced through the airway wall into the parenchyma surrounding the airways. The access tool may then be removed from catheterand LGand/or tools,reinserted into catheterto navigate catheteralong a second portion of the pathway outside of the airways to the target location.

94 70 94 93 96 62 64 94 94 50 96 62 64 96 3 50 96 62 64 3 During a procedure, EM sensor, in conjunction with tracking system, enables tracking of EM sensor(and thus distal tipof catheteror tools,) as EM sensoris advanced through the patient’s airways following the pathway planned during the planning phase. Though generally described herein in connection with EM sensors, the disclosure is not so limited. Rather, the position of the bronchoscope, catheteror tools,can be determined through the use of flex sensors (E.g., Fiber-Bragg sensors) which are used to match the shape of the catheterwith the shape of the airways in theD model. By sensing the shape of the sensors, and matching the sensor’s shape the airways, an accurate determination of the position of the sensor or a distal portion of the bronchoscope, catheteror tools,can be determined and displayed on theD model.

3 3 81 94 3 94 As an initial step of the procedure, when using aD model generated from CT scan, theD model must be registered with the patient’s actual airways to enable applicationto display an indication of the location of EM sensoron theD model corresponding to the location of EM sensorwithin the patient’s airways. The registration is necessary because the CT scan may have been taken days, and even weeks or months prior to the actual procedure. Even if the CT scan were taken the same day, such CT scans are not undertaken within a surgical suite thus registration is still necessary.

92 92 3 94 3 One potential method of registration involves performing a survey of the patient’s lungs by navigating LGinto each lobe of the patient’s lungs to at least the second bifurcation of the airways of that lobe. The position of LGis tracked during this registration phase, and theD model is iteratively updated based on the tracked position of the sensorwithin the actual airways of the patient’s lungs. While the registration process focuses on aligning the patient’s actual airways with the airways of theD model, registration also ensures that the position of vascular structures, pleural surfaces, and fissures of the lungs are accurately determined.

3 3 3 Registration, however, does not achieve a perfect match of the position of the patient’s lungs and theD model. There are a number of reasons for this mismatch, typically called CT-to-body divergence. As an initial matter, traditional CT images are taken at full breath hold. That is, the patient is asked to expand their lungs to a maximum and hold that position while undergoing the imaging. This has the benefit of inflating the airways and increasing their visibility in the CT images and make it easier to generate a highly detailedD model. However, when performing the procedure, the patient is not at a full breath hold, rather they are typically sedated and experiencing tidal volume breathing. This results in a difference in shape and position of the airways in the lungs of the patient during the procedure as compared to during the CT imaging. As a result, even when the airways have been registered to theD model (e.g., using the airway sweep or another method) there will be differences between the relative positions of the airways or targets identified in the lungs in the model and the actual relative positions of the patient’s airways and the target.

20 3 3 3 3 3 One method of addressing the CT-to-body divergence is to utilize a CBCT image data set from radiographic imaging deviceand not a traditional CT scans as the starting point for the procedure. In this process, the CBCT image data is used to generate and display theD model of the patient’s airways, to enable identification of a target location on theD model (automatically, semi-automatically or manually) by analyzing the image data and/orD model, and allow for the determination and selection of a pathway through the patient’s airways to the target location. Though the following techniques are described in connection with CBCT images those of skill in the art will appreciate that they are equally applicable to any imaging technique capable of generating aD reconstruction such asD fluoroscopy, as noted above.

2 FIG. 1 FIG. 200 100 40 74 70 50 96 30 94 50 96 70 80 202 presents a methodfor employing CBCT in conjunction with systemofsuch that the planning phase occurs in conjunction with the navigation and treatment of the patient. As will be appreciated, the patient is situated on the table, reference sensorsare on the patient’s chest, and connected to EM tracking system. A bronchoscopeand/or catheteris inserted into the patient’s airways and images may be displayed on the monitoring equipment. A position of the sensor(e.g., one associated with the bronchoscopeor catheteror another tool) can be detected and indication that a sensor position has been received by tracking systemcan be presented on a user interface on computing deviceat step.

20 80 204 80 206 3 3 3 204 3 3 120 Radiographic imaging devicemay then be engaged and the computing devicereceives the CBCT at step. The computing deviceincludes one or more applications for processing the CBCT data and presenting it on one or more user interfaces for manipulation and assessment. At step, the application analyzes the CBCT data and generates aD model of the airways. ThisD model can be manipulated by the user via a user-interface to ensure that it has sufficient resolution and sufficiently captures the airways of the patient (e.g., to a particular bifurcation point). A rejection of theD model may be received by the application at which point a further CBCT image may be acquired and the process restarted at step. The rejection may be based for example, on the clinician not being satisfied with theD model (e.g., insufficient bifurcation generation, missing a lobe or a significant portion thereof), alternatively, theD model may simply appear incorrect based on the clinician’s experience with the physiology of patients. These types of deficiencies may be the result of improper or insufficient CBCT imaging or an improper setting on the radio graphic imaging device.

3 208 210 212 214 80 3 216 Acceptance of theD model is received by the application at stepand the user-interface presents CBCT images or virtual CBCT images of the patient’s lungs from the CBCT at step. These CBCT images are slice images either taken at or generated for different points of the patient’s lungs in cross section. The user interface allows the user to scroll through these images which show one or more of the axial, coronal or sagittal planes (though others are also possible) of the lungs and allows the user to identify a target within the images. The application receives the indication of a target at stepand generates a pathway to reach the target through airways at step. The target indication may be a manual marking by a clinician providing the indication through a user interface on computing device. Alternatively, the application 81 may perform an image analysis and automatically detect the target and provide the indication of its location. The user interface then displays the pathway though the airways in one or more CBCT images, virtual CBCT images, or a virtual bronchoscopy view of theD model at step. The user interface may additionally or alternatively display the pathway on one or more of CBCT images or virtual CBCT images.

50 96 217 94 94 204 202 50 96 3 3 The CBCT images will also show the presence of the bronchoscopeor catheterthat had been previously inserted into the patient. At stepthe application can determine whether the sensorhas moved since the acquisition of the CBCT images. If the sensorhas not moved since the taking of the CBCT images in step, then the position of the EM sensor detected in stepcorresponds to the position of the distal end of the bronchoscopeor catheterin the images. As such the EM coordinate system and the CBCT image system are registered to one another and no further steps need be taken to register the patient’s lungs to theD model generated from the CBCT images and further navigation can be undertaken following the planned pathway through theD model with confidence.

217 94 81 218 94 70 3 3 3 3 220 222 224 If the determination at stepis that the sensorhas moved, or moved greater than some threshold, then an indicator can be presented on the user interface suggesting that the user perform a survey, as described above, and the applicationreceives the survey data at step. The survey involves the insertion of the EM sensorinto the lobes of the lungs receipt by the tracking systemof the position of the EM sensor as it moves through the airways. As many hundreds or thousands of these positions (EMN coordinates) are collected a point cloud of positions is created. The point cloud, of which all points are assumed to be taken from within the luminal network has aD dimensional shape that can then be matched to theD shape of the airways to register to theD model and the airways of the patient. Once registered the detected position of the EM sensor can be used to follow a pathway in theD model to the identified target. The detected position of the EM sensor relative to the pathway and the target is continually updated on the user interface at stepuntil determining that the target has been is arrived at stepand a procedure is undertaken at step. The procedure may be a biopsy or a treatment of the target such as ablation (e.g., RF, microwave, cryo, thermal, chemical, immunotherapy, or combinations of these).

94 216 218 3 Whether the patient and the CBCT images are registered because the sensordid not move following the imaging (step), or by use of the survey (step), this registration using a CBCT image should essentially eliminate any CT-to-body divergence issue as the CBCT images were acquired with the patient in exactly the same position as when the navigation procedure commences. Moreover, the CBCT images are taken while the patient is undergoing tidal breathing as opposed to full breath hold, thus the differences between the patient’s lungs and theD modeling when tradition CT images are used while the patient is at full breath hold.

94 50 96 90 Though not described in detail here, the positioning and navigation of the EM sensor(e.g., on bronchoscope, catheter, or other tools) may be done manually as described above in connection with catheter guide assemblyor may be achieved using a robotically driven catheter guide assembly.

300 100 3 80 302 81 304 302 304 3 FIG. A further methodthat may be used with systemis described in connection with. In the method 300 a CT image and/or a CTD model is received and stored in a memory associated with computing deviceat step. This is a standard pre-operative CT image taken with traditional CT imaging systems while the patient is at full breath hold, as described above. This pre-operative CT image is processed by the applicationand a pathway is generated to targets within the luminal networks which have been imaged (e.g., the airways of the lungs) at step. Stepsandachieve the planning phase.

306 40 94 70 81 80 308 81 20 50 94 310 3 308 3 20 80 81 At optional step, which may be at any time following completion of the planning phase, the patient is situated on the tableand the data from an survey (e.g., insertion of an EM sensorinto the airways) is received by the tracking systemand processed by applicationin computing device. At stepCBCT image is acquired by applicationof the desired portion of the patient using radiographic imaging device. This CBCT image may include the bronchoscopeor another device including EM sensor. Optionally, at stepa CBCTD model may be generated from the CBCT image. Alternatively, the acquired CBCT image received at stepmay include aD model that was generated by software resident on the radio graphic imaging device, or on the PACS server, and supplied to the computing deviceand applicationwith the CBCT image.

308 81 312 20 312 Both the pre-operative CT image that was used for the planning phase and the CBCT image acquired in stepare in Digital Imaging and Communications in Medicine (DICOMM) format. The DICOMM format includes reference to the coordinate system with which the image was acquired. As a result, the application, at steptransforms the coordinate system of the pre-operative CT image with the coordinate system of the CBCT image taken by the radiographic imaging device. Stepeffectively registers the pre-operative CT image with the CBCT image.

311 81 3 310 3 302 3 3 3 3 3 81 Alternatively, at stepthe applicationaligns the CBCTD model generated at stepaD model generated from the pre-operative CT image and received at step. The alignment of the twoD models registers the pre-operative CT image with the CBCT image. The application may present either or both of the pre-operative CTD model and the CBCTD model on a user interface and request confirmation of alignment by a user or allow for interaction by the user to finalize the orientation of the twoD models relative to each other to finalize the registration of the twoD models. Alternatively, this may be automatically performed by application.

20 40 40 40 81 200 81 40 A further alternative with respect to registration is to make an assumption as to alignment of the patient in the pre-operative CT image and the CBCT image. This process relies on the fact that during imaging with the radiographic imaging devicethe patient is always lying flat on the tablewith their chest away from the tablealong the length of the tableand that they will be in essentially this position during the acquisition of the pre-operative CT. In this registration process, the applicationmay request via the user interface that the clinician identify a common point in both the pre-operative CT and the CBCT image. This point could be the target, as described above with respect to method, or it could be a point such as a main carina of the lungs or a rib or some other feature which appears in both image data sets. Alternatively, the applicationmay utilize various image processing techniques to identify these common features in the two image data sets and to register them to one another. Once identified, either manually or automatically, because of the assumption that the patient is aligned on the tableessentially in the same position in both images, the two image data sets (e.g., pre-operative CT and CBCT images) are registered to one another. As will be appreciated, the identification of 2, 3, 4, 5, 10 points, either automatically or by a clinician using the user interface will refine the registration even more, where desired. In some aspects this may be achieved using mutual information techniques of image brightness matching. This may be assisted by various deep learning methodologies where empirical algorithms are developed by the processing of hundreds or thousands or more images and performing the registration.

314 3 308 3 310 3 310 316 312 3 310 3 3 311 3 318 81 3 320 At step, once the two CT images orD models are registered to one another, all the planning data that was generated using the pre-operative CT image can be transferred to the CBCT image acquired at stepor to theD model acquired at step. With features such as the target and a pathway to the target, among others, transferred from the pre-operative CT image to the CBCT image, if aD model of the CBCT image was not generated at step, it can now be generated at stepand will include the target and pathway that has been transferred from the pre-operative CT image to the CBCT image at step. Alternatively, where the CBCTD model was generated at step, but the pre-operative CTD model and the CBCTD model were not registered to one another at step, the features transferred can be matched to the CBCTD model at optional step. Regardless of when the transfer to the features occurs, the applicationcan cause a user interface to display the CBCTD model and CBCT images and the features from the planning phase identified in the pre-operative CT image can be displayed therein on a user interface at step.

306 322 3 94 50 96 306 308 94 216 3 3 94 324 81 326 328 94 20 96 20 80 81 96 3 96 In instances where a survey was not undertaken at step, a survey can be undertaken at step. This survey registers the CBCT image and the CBCTD model to the patient’s lungs by navigating the sensor, which is embodied on the bronchoscope, catheteror another tool, into the airways of the patient, generating the point cloud discussed above. As will be appreciated, other methods of registration may also be employed without departing from the scope of the present disclosure. If the survey were conducted in step, above, the application may proceed during the acquisition of the CBCT image at stepto conduct the EM sensormovement analysis, described above in stepto register the patient’s airways to the CBCT image andD model generated therefrom. Once registered the detected position of the EM sensor can be used to follow a pathway in the CBCTD model to the target originally identified in the pre-operative CT image. The detected position of the EM sensorrelative to the pathway and the target is continually updated on the user interface at stepuntil the applicationdetermines that the target has been arrived at stepand a procedure may be undertaken upon arrival at step. As an alternative, to use of an EM sensorand detection of its position, the radiographic imaging devicemay be capable of generating fluoroscopic images. The position of the cathetermay be detected in one or more fluoroscopic images that are acquired by the radio graphic imaging device. This detection may be manual by the clinician using a user interface on computing deviceor may be performed by the applicationvia image processing techniques. Because the coordinate system is the same between the CBCT images and the fluoroscopic images acquired by the same device, the detected position of the catheterin the fluoroscopic images can be transferred to the CBCT images or CBCTD model. The fluoroscopic images may be acquired periodically as the catheteris navigated towards the target. The procedure may be a biopsy or a treatment of the target such as ablation (e.g., RF, microwave, cryo, thermal, chemical, immunotherapy, or combinations of these).

2 FIG. 3 FIG. 3 94 3 As with the method of, the method of, eliminates the CT-to-body divergence because the CBCT image and model are generated with the patient in the same position they are in for the navigation procedure. Further the target and pathways are shown in the CBCTD model and CBCT images. Further, any discrepancies in registration are minimized either by the DICOMM registration process, the acquisition of the CBCT image with the sensorin the image, and/or receiving survey data and matching it to the airways of the CBCT images and CBCTD model.

400 400 402 80 404 81 3 4 4 FIGS.A andB A methodis described with reference to. In accordance with method, a pre-operative CT image is acquired at stepand saved in a memory of computing device. At stepthe applicationprocesses the CT image, generates aD model, presents on a user interface CT images on which to receive an indication of a target, and generates a pathway through the airways of a patient (or another luminal network) to reach the target. These steps complete the planning phase using a pre-operative CT image.

40 50 96 94 70 81 406 94 70 408 3 410 94 96 3 After the planning phase is complete, the patient may be placed on the tableand a bronchoscopeor catheterinserted such that a sensorcan be detected by the tracking systemand that data provided to the applicationat step. Next a survey can be conducted and a point cloud of positions of the sensorreceived by the tracking systemas the survey is conducted at step. With the point cloud, the patient and the pre-operative CT image as well as theD model generated therefrom are registered to one another at step. As noted above, the sensormay be an EM sensor, a flex sensor, or other sensor useable to determine a position of the catheteror bronchoscope in a patient and depict that position in the pre-operative, thus registering the patient and the pre-operative CT image andD model.

70 94 412 With the patient and the pre-operative CT image registered navigation can commence with the tracking systemreceiving indications of new locations of the sensoras it is moved through the airways of the patient and the detected positions being updated on a user interface at stepas the pathway is followed to an identified target.

94 50 96 94 20 414 416 3 418 408 80 3 420 3 408 404 3 3 422 3 424 94 40 94 426 94 3 428 4 FIG.B 4 FIG.B Once the sensor, and more particularly the bronchoscope, catheter, or other tool including the sensor, is proximate the target a CBCT image can be generated with radiographic imaging deviceat step. At this point at least two different options are available. In accordance with one option, at step, a CBCTD model is generated from the CBCT image. Next at stepthe point cloud that was generated by the survey at stepmay be recalled from a memory in the computerized devicein which it is stored, and fit to the CBCT image, and the CBCTD model. Alternatively, the method may skip forward to step, where the CBCT model and the CBCT images are registered by any of the methods described herein and can be presented on a user interface. Because the CBCT image andD model are registered with the patient based on the survey from step, the pathway and targets identified at stepcan be transferred from the pre-operative CT imageD model to the CBCT image and CBCTD model at stepin. Alternatively, inthe CBCT image and CBCTD model may be presented on the user interface at stepsuch that the target can be identified in the CBCT images. Once identification of the target is received by the application, the application generates a pathway from the location of the sensor, as depicted in the CBCT images and CBCT model to the target. Again, because the CBCT image is generated about the patient while they are in position on the tableon which the navigation is being undertaken, there is no CT-to-body divergence. The result is that the “last mile” of navigation to the target (e.g., the final 3 mm to a target) can be undertaken with heightened confidence that the target will be properly reached for biopsy or treatment. Subsequent movement of the sensoris detected at stepand the position of the sensorin the CTD model can be updated and a procedure can be undertaken at step.

414 430 50 96 94 3 50 96 81 50 96 50 96 2 3 81 432 94 3 3 94 426 428 As noted above, after stepan alternative method can be followed. At a stepthe CBCT image, which includes the bronchoscopeor catheter(or other tool) with sensortherein is within the CBCT image, the CBCT image and/or CBCTD model can be analyzed to determine the relative position of the target and a distal end of the bronchoscopeor catheter. This relative position determination can be automatically derived by the application. Alternatively, the relative position can be determined by receipt of an indication of the location of the bronchoscopeor cathetervia the user interface, where one or more of the target and the distal end of the bronchoscopeor catheterare shown inD images or theD model. The position of the target can be assumed to be the same in both the pre-operative CT image and the CBCT image. The relative position data can then be used by the applicationat stepto update the detected position of the sensorin the pre-operative CT image and theD model derived from the pre-operative CT. This update of position will account for the CT-to-body divergence that results from the use of the pre-operative CT image and theD model for navigation. Again, the last mile movement of the sensorto the target can be detected at stepand a procedure can be performed at step.

100 81 80 20 81 81 200 400 500 50 96 94 502 81 20 504 20 5 FIG. As will be appreciated, the system, and particularly applicationbeing run on computing device, can be configured to control operation of the radiographic imaging device. This control may be via user input to a user interface. As such according to this Alternatively, the application, can be configured, following registration of the pre-operative CT or an initial CBCT image to a patient (if required) and the identification of target, to adjust the imaging field of the CBCT to focus on the target. The applicationmay, using the location of the target in the patient, focus all future CBCT imaging on the target. This may be done without any intervention by the user. Similarly, the applicationmay initiate CBCT imaging at points during any of the methods described with respect to methods-, without interaction from a user. For example in connection with a methoddepicted in, as a bronchoscope, catheter, or any other tool including sensoris navigated to and detected within a pre-determined distance from a target at step, the applicationsignals the radiographic imaging deviceto initiate a CBCT image acquisition process at step. Alerts may be provided to the clinicians and surgical staff allowing them to move away from the patient and limit their exposure to the radiation emitted by the radiographic imaging device. These alerts may be audible or visual via the user interface.

20 81 506 400 200 400 The CBCT image is acquired via radiographic imaging deviceand received by applicationat step. This CBCT image may be used as described particularly with respect to the CBCT imaging described in the method. Alternatively, the use of CBCT imaging may be reviewed and considered completely separate from methods-and simply as another visual tool employed by the clinician to confirm placement, locations, and other clinical observations.

600 74 70 81 602 604 81 606 200 400 100 6 FIG. can A further aspect of the disclosure is directed to breathing detection to assist in CBCT imaging. The methodis described with respect to. Employing the output from reference sensors, tracking systemand therewith applicationdetect the phase of the phase of the patient’s breathing at step. This phase can be monitored throughout a procedure at step. Whenever a request for a CBCT image is received (either directly or via an automatic process) the applicationcan determine the phase of the patient’s breathing at step. There are a variety of options for the application at this point depending on which method-in which the systemis engaged.

81 608 3 If the CBCT image is the first CT image acquired for a procedure, the applicationcan at stepdirect the radiographic imaging device to only acquire images when the reference sensors are within a tolerance of a desired portion of the breathing phase (e.g., nearing end of exhale phase, or nearing end of inhale phase). For example, nearing the end of the inhale phase may allow for the airways to be in an expanded state resulting in potentially cleaner images that can generate a more accurateD model due to the contrast of the airways that results from airways being expanded. Alternatively, when the breathing phase is approaching the end of the exhale phase, there may be a longer duration of the breathing cycle where there is substantially no movement of the lungs, thus allowing for more images to be captured and enhancing the stability of the images as they are acquired in the CBCT image.

610 81 20 81 612 20 614 616 618 81 74 610 20 616 81 20 200 400 At stepthe applicationsignals the radiographic imaging deviceto being imaging. When applicationdetermines at stepthat the desired portion of breathing phase is about to end the application signals the radiographic imaging deviceto stop imaging the patient at step. At stepthe application can determine whether the CBCT imaging is complete. If not, the method continues to stepwhere the applicationdetermines that the desired breathing phase is about to be entered, by monitoring the position of the reference sensors, and the method returns to stepwhere the radiographic imaging deviceagain acquires images during the desired portion of the breathing phase. If the CBCT imaging is complete at step, then the applicationstops the radiographic imaging deviceat step 6192 and proceeds to the next steps in methods-.

81 608 20 20 81 81 20 608 Where a registration is desired between a either a pre-operative CT image or a previously acquired CBCT image, the applicationcan at stepsignal the radiographic imaging deviceto acquire images only during those portions of the breathing cycle that most closely match the breathing cycle of the previously acquired CT or CBCT images. By matching the breathing cycles as closely as possible, the two image data sets will more closely resemble one another making registration between the two easier and to transfer features such as a target or a pathway from the first CT image to a second. For example, in instances where registration to a pre-operative CT image is desired, the radiographic imaging device, can be directed by the applicationto acquire CBCT images only during portions of the breathing cycle approaching the maximum inhale of normal tidal breathing position. When it is two CBCT images that are to be acquired the applicationcan store in memory the breathing phase of the first CBCT image and direct the radiographic imaging deviceacquire images at the same phase of breathing at step.

74 40 81 74 74 20 622 81 624 81 20 626 610 As will be appreciated, by limiting imaging to a specified portion of the breathing phase the time required to acquire a CBCT image may be increased and may take several breathing cycles to complete. This may minimally extend the time required to acquire the CBCT image but results in greater fidelity of the captured image as the lungs are always imaged in about the same position of the breathing cycle. In addition, by monitoring the reference sensors, if a patient were to cough or move on the tableduring the imaging process, the applicationwhich is monitoring the positions of the reference sensorsthorough the breathing cycle can detect the rapid movement of the sensor. If such a movement is detected during imaging by the radiographic imaging deviceat step, the applicationcan reject the most recently acquired portion of the CBCT image at step. The applicationcan then direct the radiographic imaging deviceto reposition itself at stepto reacquire a portion of the CBCT image that corresponds to that which was rejected in the next breathing phase and the method proceeds back to step.

7 FIG. 80 80 702 704 706 708 710 712 702 81 514 81 704 706 716 81 94 Turning now to, there is shown a simplified block diagram of computing device. Computing devicemay include a memory, a processor, a display, a network interface, an input device, and/or an output module. Memorymay store applicationand/or image data. Applicationmay, when executed by processor, cause displayto present user interface. Applicationmay also provide the interface between the sensed position of EM sensorand the image and planning data developed in the pathway planning phase, described above.

702 704 80 507 702 704 704 80 Memorymay include any non-transitory computer-readable storage media for storing data and/or software that is executable by processorand which controls the operation of computing device. In an embodiment, memorymay include one or more solid-state storage devices such as flash memory chips. Alternatively, or in addition to the one or more solid-state storage devices, memorymay include one or more mass storage devices connected to the processorthrough a mass storage controller (not shown) and a communications bus (not shown). Although the description of computer-readable media contained herein refers to a solid-state storage, it should be appreciated by those skilled in the art that computer-readable storage media can be any available media that can be accessed by the processor. That is, computer readable storage media includes non-transitory, volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data. For example, computer-readable storage media includes RAM, ROM, EPROM, EEPROM, flash memory or other solid state memory technology, CD-ROM, DVD, Blu-Ray or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by computing device.

708 710 80 712 Network interfacemay be configured to connect to a network such as a local area network (LAN) consisting of a wired network and/or a wireless network, a wide area network (WAN), a wireless mobile network, a Bluetooth network, and/or the internet. Input devicemay be any device by means of which a user may interact with computing device, such as, for example, a mouse, keyboard, foot pedal, touch screen, and/or voice interface. Output modulemay include any connectivity port or bus, such as, for example, parallel ports, serial ports, universal serial busses (USB), or any other similar connectivity port known to those skilled in the art.

8 FIG. 8 FIG. 800 800 100 40 90 50 800 802 20 80 804 81 20 80 806 96 96 80 81 96 802 20 96 describes a methodrequiring no pre-procedure CT scan data. Inmethodfollows steps of set up of the system, placement of the patient on the operating table, and initial navigation of catheter guide assemblyeither alone or in combination with bronchoscopeto a location within the lungs. The location within the lungs could be a target lobe or other anatomical point. As an example, the location may be the third bifurcation in a desire lobe of the lung. Once at this location, methodstarts with capturing a CBCT scan at stepusing radiographic imaging device. The computing devicereceives the CBCT scan at step. For example, the applicationmay be retrieve the CBCT scan from a database in which the scan was stored following capture, or a user may direct the radiographic imaging deviceto output the CBCT scan directly to the computing device. At step, the distal end of the catheterand a target (e.g., a lesion or other location for treatment) are identified in one or more images of the CBCT scan. This identification can be manual where the user marks the distal end of the catheterand the target in one or more of the images of the CBCT scan. These images may be displayed in a user interface on computing device. Alternatively, the applicationmay be configured to conduct image analysis and to automatically identify the distal end of the catheterand the target. If either or both of the distal portion of the catheter or the target cannot be identified in the CBCT images from the scan, the process can return to stepto conduct another CBCT scan. This may require repositioning of the patient, radiographic imaging device, or the catheter.

808 80 70 20 40 76 Following identification, at stepthe computing devicecan register the CBCT scan data with the electromagnetic field generated by the tracking system. Registration may be undertaken in a variety of ways. If, for example, the coordinate system of the radiological image deviceis perpendicular to the operating table, all that is required is translation of the CBCT coordinates to match the tracking system (e.g., EM coordinates of the field produced by EM field generator). Alternatively, registration may be achieved utilizing a pose estimation technique.

76 20 76 96 70 70 To determine the pose for each slice making up the CBCT scan, fiducial markers which are formed in or on the EM field generatorplaced under the patient are analyzed. The markers may be evenly spaced or may be varyingly spaced from one another in a known pattern. Regardless of how spaced, the orientation and placement of the markers is know and the spacing and positioning of the markers in any slice of the CBCT can be analyzed to determine the angle of the device relative to the radiographic imaging devicerelative to the EM field generator. With the known position of the markers, and both a marked position of the distal portion of the catheterand a detected position of the catheter as identified by the tracking system, a mathematical transform from the coordinate system of the CBCT scan data to the coordinate system of the tracking system(e.g., EM coordinates).

810 3 312 3 96 3 814 302 92 816 Once registration is complete, at step, aD model of the patient’s lungs can be generated from the CBCT scan, similar to the process described above with the pre-procedure CT image. At step, a pathway is generated through theD model from the marked position of the distal portion of the catheterto the marked position of the target. This pathway may be manually created by a user, semi-automatically, or automatically derived, much as it might be in aD model from a pre-procedure CT scan. Navigation to the target may now be undertaken. If at any time during the navigation the user wishes to perform another CBCT scan, the decision can be made at stepand the process can revert back to step. The use of multiple CBCT scans may be desirable, for example, when performing microwave or RF ablation procedures within the lungs to ensure accurate placement of an ablation catheter in a desired location in the target. Once navigated to an appropriate location, a user or a robot may remove the LGto allow for placement of an ablation catheter or other tool (e.g., a biopsy tool) to perform a procedure at step.

9 FIG. 900 800 900 100 40 800 902 80 904 81 20 80 906 3 908 3 910 94 96 912 3 20 76 96 914 916 918 96 920 96 94 96 922 96 3 3 81 920 922 924 926 916 depicts a methodrequiring no pre-procedure CT scan data. As with method, methodfollows steps of set up of the systemand placement of the patient on the operating table. Rather than initiate navigation as described in method, at stepa CBCT scan of the patient is undertaken. The computing devicereceives the CBCT scan at step. For example, the applicationmay be retrieve the CBCT scan from a database in which the scan was stored following capture, or a user may direct the radiographic imaging deviceto output the CBCT scan directly to the computing device. At stepaD model of the airways is generated from the CBCT scan. At stepeither theD model, or slice images from the CBCT image are analyzed to identify a target (e.g., a lesion) and to generate a pathway to the target through the airway in the same manner as can be done with a pre-procedure CT image. Following target identification and pathway planning, a sweep of the airways may be undertaken at step. As described above, in the sweep of the airways, sensorof catheteris inserted into the airways and a point cloud of position data is generated. At step, the point cloud of data is matched to the internal features of theD model and the coordinate system of radiographic imaging deviceis registered to the electromagnetic field coordinate system of the EM field output by the EM field generator. Once registered navigation of the catheter, either manually or robotically can be undertaken at step. Once proximate a target, a second CBCT scan may be undertaken at step. Reviewing the slice images of the CBCT scan, at stepthe positions of the distal end of the catheterand the target can be marked. At step, based on the marked positions of the distal portion of the catheterand the target and offset can be calculated. Since the position of the target is unlikely to have moved significantly during the procedure, this offset is substantially and indication of error in the detected position of the sensorat the distal end of the catheterin the EM filed. With this offset calculated, at stepa displayed position of the distal portion of the catheterin theD model can be updated to accurately depict the relative position of the catheter and the target in theD model, and in other views provided by the user interface of applicationdescribed herein above. The combination of stepsandare a local registration of the CBCT and the EM field coordinate systems and again provide greater accuracy as may be desired when performing a procedure at stepsuch as microwave ablation or diagnostics such as biopsy of a lesion. If further movement of the catheter is desired further navigation can be undertaken at step, and the method can revert back to stepto update the local registration.

10 FIG. 1000 1002 100 40 70 1004 20 1006 81 80 provides yet a further method in accordance with the disclosure. In method, at steppre-procedure planning (as described above) is undertaken utilizing a pre-procedure CT scan. At some time after the pre-procedure planning, systemis initialized, this may entail placement of the patient on the operating tableand initializing of the tracking systemand other steps described above. Once the patient is in position, at stepthe radiological imaging deviceis employed to acquire a CBCT scan of a relevant portion of the patient (e.g., the lungs). At step, he applicationoperating on computing device

81 20 80 1008 3 80 3 1010 94 96 1012 3 20 76 70 96 1014 1016 1018 96 1020 96 94 1022 96 3 receives the CBCT scan . For example, the applicationmay be retrieve the CBCT scan from a database in which the CBCT scan was stored following capture, or a user may direct the radiographic imaging deviceto output the CBCT scan directly to the computing device. At stepthe CBCT scan is registered to the pre-operative scan. A variety of means can be used for this registration, for example image orD model matching may be employed to substantially match the pre-procedure CT scan to the CBCT scan. This registration enables the transfer of a planned pathway and a target from the pre-procedure plan generated from the pre-procedure CT scan to the CBCT scan. As a result of this registration, a user interface on the computing devicecan display a pathway through aD model and other views generated from the CBCT scan to the target which is also now presented in the CBCT scan images. At step, a sweep of the airways may be undertaken. As described above, in the sweep of the airways, sensorof catheteris inserted into the airways and a point cloud of position data is generated. At step, the point cloud of data is matched to the internal features of theD model generated from the CBCT scan and the coordinate system of radiographic imaging deviceis registered to the electromagnetic field coordinate system of the EM field output by the EM field generator(or other tracking systemdescribed herein). Once registered navigation of the catheter, either manually or robotically can be undertaken at step. Once proximate a target, a second CBCT scan may be undertaken at step. Reviewing slice images of the CBCT scan, at stepthe positions of the distal end of the catheterand the target can be marked. At step, based on the marked positions of the distal portion of the catheterand the target and offset can be calculated. This offset is used to update the detected position of the sensorin the EM field relative to the target. With this offset calculated, at stepa displayed position of the distal portion of the catheterin theD model generated

3 81 1020 1022 1024 96 1026 1016 from the CBCT scan can be updated to accurately depict the relative position of the catheter and the target in theD model, as well as other views provided by the user interface of applicationdescribed herein above. The combination of stepsandare a local registration of the CBCT and the EM field coordinate systems and again provide greater accuracy as may be desired when performing a procedure at steptreatment such as microwave ablation or diagnostics such as biopsy of a lesion. If further movement of the catheteris desired further navigation can be undertaken at step, and the method can revert back to stepto update the local registration.

While several aspects of the disclosure have been shown in the drawings, it is not intended that the disclosure be limited thereto, as it is intended that the disclosure be as broad in scope as the art will allow and that the specification be read likewise. Therefore, the above description should not be construed as limiting, but merely as exemplifications of particular aspects.