Systems and Methods for Implementing a Zoom Feature Associated with an Imaging Device in an Imaging Space

This application claims priority to U.S. Provisional Patent Application No. 63/527,430, filed Jul. 18, 2023, the contents of which is hereby incorporated by reference in its entirety.

A computer-assisted surgical system that employs robotic and/or teleoperation technology typically includes a stereoscopic image viewer configured to provide, for display to a surgeon, images of an imaging space (e.g., a surgical space) as captured by an imaging device such as an endoscope. While the surgeon's eyes are positioned in front of viewing lenses of the stereoscopic image viewer, the surgeon may view the images of the surgical space while remotely manipulating one or more surgical instruments located within the surgical space. The surgical instruments are attached to one or more manipulator arms of a surgical instrument manipulating system included as part of the computer-assisted surgical system.

In addition to remotely manipulating the surgical instruments, the surgeon may remotely manipulate the imaging device, which is attached to one of the manipulating arms, to change the position and/or view of the imaging space captured by the imaging device. For example, the surgeon may move the imaging device closer to an object within the imaging space to get a closer view of the object. However, moving the imaging device too close to the object may cause damage to the object and/or result in reduced image quality of images captured by the imaging device. For example, moving the imaging device too close to tissue within the imaging space may result in burning of the tissue, fogging of the imaging device, and/or splatter on the imaging device, which may require removal of the imaging device for cleaning or replacement. It may be possible for a surgeon to manually activate/deactivate a zoom function of an imaging device during a procedure to zoom in on an object and possibly avoid such negative effects. However, such manual activation/deactivation of a zoom function undesirably increases the complexity of a procedure and results in additional workload for the surgeon. Accordingly, there remains room to improve the implementation of a zoom function of an imaging device used in an imaging space.

An example system comprises a memory storing instructions; and a processor communicatively coupled to the memory and configured to execute the instructions to perform a process comprising receiving an instruction to move an imaging device, which is located within an imaging space and coupled to a robotic manipulating arm of a computer-assisted surgical system, towards an object in the imaging space, determining, based on the instruction, that the imaging device is located at a position that is at or near a threshold distance from the object in the imaging space, preventing the imaging device from moving closer to the object than the threshold distance, and automatically activating, in response to the preventing of the imaging device from moving closer to the object than the threshold distance, an image zoom feature to zoom in a view of the imaging space.

An example computer program product embodied in a non-transitory computer readable storage medium comprises computer instructions for performing a process comprising receiving an instruction to move an imaging device, which is located within an imaging space and coupled to a robotic manipulating arm of a computer-assisted surgical system, towards an object in the imaging space, determining, based on the instruction, that the imaging device is located at a position that is at or near a threshold distance from the object in the imaging space, preventing the imaging device from moving closer to the object than the threshold distance, and automatically activating, in response to preventing the imaging device from moving closer to the object than the threshold distance, an image zoom feature to zoom in a view of the imaging space.

An additional example computer program product comprises instructions which, when executed by a computer, cause the computer to perform a process comprising receiving an instruction to move an imaging device, which is located within an imaging space and coupled to a robotic manipulating arm of a computer-assisted surgical system, towards an object in the imaging space, determining, based on the instruction, that the imaging device is located at a position that is at or near a threshold distance from the object in the imaging space, preventing the imaging device from moving closer to the object than the threshold distance, and automatically activating, in response to preventing the imaging device from moving closer to the object than the threshold distance, an image zoom feature to zoom in a view of the imaging space.

An example method comprises receiving, by an automatic zoom system, an instruction to move an imaging device, which is located within an imaging space and coupled to a robotic manipulating arm of a computer-assisted surgical system, towards an object in the imaging space, determining, by the automatic zoom system and based on the instruction, that the imaging device is located at a position that is at or near a threshold distance from the object in the imaging space, preventing, by the automatic zoom system, the imaging device from moving closer to the object than the threshold distance, and automatically activating, by the automatic zoom system and in response to preventing the imaging device from moving closer to the object than the threshold distance, an image zoom feature to zoom in a view of the imaging space.

Systems and methods for implementing a zoom feature associated with an imaging device in an imaging space are described herein. As will be described in more detail below, an illustrative system includes a memory that stores instructions and a processor communicatively connected to the memory. The processor is configured to execute the instructions to perform a process comprising receiving an instruction to move an imaging device, which is located within an imaging space and coupled to a robotic manipulating arm of a computer-assisted surgical system, towards an object in the imaging space, determining, based on the instruction, that the imaging device is located at a position that is at or near a threshold distance from the object in the imaging space, preventing the imaging device from moving closer to the object than the threshold distance, and automatically activating, in response to preventing the imaging device from moving closer to the object than the threshold distance, an image zoom feature to zoom in a view of the imaging space.

Various advantages and benefits are associated with systems and methods described herein. For example, systems and methods such as those described herein may facilitate quick and/or convenient implementation of a zoom feature associated with an imaging device. Instead of requiring a user (e.g., a surgeon) to manually activate/deactivate a zoom feature, systems and methods such as those described herein may facilitate activating a zoom feature automatically based on a predefined condition being satisfied. In so doing, systems and methods such as those described herein may simplify procedures performed within the imaging space and/or improve usability of a computer-assisted surgical system. In addition, systems and methods such as those described herein may facilitate minimizing or preventing negative effects (e.g., tissue damage, fogging, splatter, etc.) that may otherwise occur if an imaging device is moved too close to an object (e.g., tissue) in an imaging space. These and other benefits that may be realized by the systems and methods described herein will be evident from the disclosure that follows.

Example systems described herein may be configured to operate as part of or in conjunction with a plurality of different types of computer-assisted surgical systems. The different types of computer-assisted surgical systems may include any type of computer-assisted surgical system as may serve a particular implementation, such as a computer-assisted surgical system designed for use in minimally-invasive medical procedures, for example. In certain examples, a type of computer-assisted surgical system may include a system in which one or more surgical devices (e.g., surgical instruments) are manually (e.g., laparoscopically) controlled by a user. In certain examples, a type of computer-assisted surgical system may include a robotic surgical system configured to facilitate operation one or more smart instruments (e.g., smart sub-surface imaging devices) that may be manually and/or robotically controlled by a user. In certain implementations, the plurality of different types of computer-assisted surgical systems may be of different types at least because they include different types of surgical instrument manipulating systems. For example, a first computer-assisted surgical system may include a first type of surgical instrument manipulating system, a second computer-assisted surgical system may include a second type of surgical instrument manipulating system, and a third computer-assisted surgical system may include a third type of surgical instrument manipulating system.

1 FIG. Each type of surgical instrument manipulating system may have a different architecture (e.g., a manipulator arm architecture), have a different kinematic profile, and/or operate according to different configuration parameters. An illustrative computer-assisted surgical system with a first type of surgical instrument manipulating system will now be described with reference to. The described computer-assisted surgical system is illustrative and not limiting. Systems such as those described herein may operate as part of or in conjunction with the described computer-assisted surgical system and/or any other suitable computer-assisted surgical system.

1 FIG. 100 100 100 102 102 104 106 illustrates an example computer-assisted surgical system(“surgical system”). As shown, surgical systemmay include a surgical instrument manipulating system(“manipulating system”), a user control system, and an auxiliary systemcommunicatively coupled one to another.

100 108 110 1 110 2 110 3 110 4 110 Surgical systemmay be utilized by a surgical team to perform a computer-assisted surgical procedure on a patient. As shown, the surgical team may include a surgeon-, an assistant-, a nurse-, and an anesthesiologist-, all of whom may be collectively referred to as “surgical team members.” Additional or alternative surgical team members may be present during a surgical session as may serve a particular implementation.

1 FIG. 1 FIG. 100 100 100 Whileillustrates an ongoing minimally invasive surgical procedure, surgical systemmay similarly be used to perform open surgical procedures or other types of surgical procedures that may similarly benefit from the accuracy and convenience of surgical system. Additionally, it will be understood that the surgical session throughout which surgical systemmay be employed may not only include an operative phase of a surgical procedure, as is illustrated in, but may also include preoperative, postoperative, and/or other suitable phases of the surgical procedure. A surgical procedure may include any procedure in which manual and/or instrumental techniques (e.g., teleoperated instrumental techniques) are used on a patient to investigate, diagnose, or treat a physical condition of the patient. Additionally, a surgical procedure may include any procedure that is not performed on a live patient, such as a calibration procedure, a simulated training procedure, and an experimental or research procedure.

1 FIG. 102 112 112 1 112 4 112 1 108 108 As shown in, surgical instrument manipulating systemmay include a plurality of manipulator arms(e.g., manipulator arms-through-) to which a plurality of robotic surgical instruments (“robotic instruments”) (not shown) may be coupled. As used herein, a “robotic instrument” refers to any instrument that may be directly attached to (e.g., plugged into, fixedly coupled to, mated to, etc.) a manipulator arm (e.g., manipulator arm-) such that movement of the manipulator arm directly causes movement of the instrument. Each robotic instrument may be implemented by any suitable therapeutic instrument (e.g., a tool having tissue-interaction functions), imaging device (e.g., an endoscope), diagnostic instrument, or the like that may be used for a computer-assisted surgical procedure (e.g., by being at least partially inserted into patientand manipulated to perform a computer-assisted surgical procedure on patient). In some examples, one or more of the robotic instruments may include force-sensing and/or other sensing capabilities.

1 FIG. 112 102 112 102 112 102 112 In the example shown in, manipulator armsof manipulating systemare attached on a distal end of an overhead boom that extends horizontally. However, manipulator armsmay have other configurations in certain implementations. In addition, while manipulating systemis depicted and described herein as including four manipulator arms, it will be recognized that manipulating systemmay include only a single manipulator armor any other number of manipulator arms as may serve a particular implementation.

112 112 100 Manipulator armsand/or robotic instruments attached to manipulator armsmay include one or more displacement transducers, orientational sensors, and/or positional sensors (hereinafter “surgical system sensors”) used to generate raw (e.g., uncorrected) kinematics information. One or more components of surgical systemmay be configured to use the kinematics information to track (e.g., determine positions of) and/or control the robotic instruments.

112 112 112 1 112 1 112 1 112 1 112 1 112 1 112 112 112 1 112 1 In addition, manipulator armsmay each include or otherwise be associated with a plurality of motors that control movement of manipulator armsand/or the surgical instruments attached thereto. For example, manipulator arm-may include or otherwise be associated with a first internal motor (not explicitly shown) configured to yaw manipulator arm-about a yaw axis. In like manner, manipulator arm-may be associated with a second internal motor (not explicitly shown) configured to drive and pitch manipulator arm-about a pitch axis. Likewise, manipulator arm-may be associated with a third internal motor (not explicitly shown) configured to slide manipulator arm-along insertion axis. Manipulator armsmay each include a drive train system driven by one or more of these motors in order to control the pivoting of manipulator armsin any manner as may serve a particular implementation. As such, if a robotic instrument attached, for example, to manipulator arm-is to be mechanically moved, one or more of the motors coupled to the drive train may be energized to move manipulator arm-.

112 100 Robotic instruments attached to manipulator armsmay each be positioned in an imaging space. An “imaging space” as used herein may refer to any space or location where an imaging operation may be performed by an imaging device such as described herein. In certain examples, an imaging space may correspond to a surgical space. A “surgical space” may, in certain examples, be entirely disposed within a patient and may include an area within the patient at or near where a surgical procedure is planned to be performed, is being performed, or has been performed. For example, for a minimally invasive surgical procedure being performed on tissue internal to a patient, the surgical space may include the tissue, anatomy underlying the tissue, as well as space around the tissue where, for example, robotic instruments and/or other instruments being used to perform the surgical procedure are located. In other examples, a surgical space may be at least partially disposed external to the patient at or near where a surgical procedure is planned to be performed, is being performed, or has been performed on the patient. For instance, surgical systemmay be used to perform an open surgical procedure such that part of the surgical space (e.g., tissue being operated on) is internal to the patient while another part of the surgical space (e.g., a space around the tissue where one or more instruments may be disposed) is external to the patient. A robotic instrument may be referred to as being positioned or located at or within a surgical space when at least a portion of the robotic instrument (e.g., a distal portion of the robotic instrument) is located within the surgical space. Example imaging spaces and/or images of imaging spaces will be described herein.

104 110 1 112 112 110 1 104 112 104 110 1 108 104 108 110 1 110 1 112 User control systemmay be configured to facilitate control by surgeon-of manipulator armsand robotic instruments attached to manipulator arms. For example, surgeon-may interact with user control systemto remotely move, manipulate, or otherwise teleoperate manipulator armsand the robotic instruments. To this end, user control systemmay provide surgeon-with one or more images (e.g., high-definition three-dimensional (3D) images) of a surgical space associated with patientas captured by an imaging device. In certain examples, user control systemmay include a stereoscopic image viewer having two displays where stereoscopic images (e.g., 3D images) of a surgical space associated with patientand generated by a stereoscopic imaging system may be viewed by surgeon-. Surgeon-may utilize the images to perform one or more procedures with one or more robotic instruments attached to manipulator arms.

104 110 1 110 1 110 1 To facilitate control of robotic instruments, user control systemmay include a set of master controls (not shown). These master controls may be manipulated by surgeon-to control movement of robotic instruments (e.g., by utilizing robotic and/or teleoperation technology). The master controls may be configured to detect a wide variety of hand, wrist, and finger movements by surgeon-. In this manner, surgeon-may intuitively perform a surgical procedure using one or more robotic instruments.

104 110 1 100 110 1 104 100 100 112 104 110 1 User control systemmay further be configured to facilitate control by surgeon-of other components of surgical system. For example, surgeon-may interact with user control systemto change a configuration or operating mode of surgical system, to change a display mode of surgical system, to generate additional control signals used to control surgical instruments attached to manipulator arms, to facilitate switching control from one robotic instrument to another, to facilitate interaction with other instruments and/or objects within the surgical space, or to perform any other suitable operation. To this end, user control systemmay also include one or more input devices (e.g., foot pedals, buttons, switches, etc.) configured to receive input from surgeon-.

106 100 106 102 104 100 104 102 106 106 102 112 Auxiliary systemmay include one or more computing devices configured to perform primary processing operations of surgical system. The one or more computing devices included in auxiliary systemmay control and/or coordinate operations performed by various other components (e.g., manipulating systemand/or user control system) of surgical system. For example, a computing device included in user control systemmay transmit instructions to manipulating systemby way of the one or more computing devices included in auxiliary system. As another example, auxiliary systemmay receive, from manipulating system, and process image data representative of images captured by an imaging device attached to one of manipulator arms.

106 110 110 1 104 106 114 108 114 114 110 100 In some examples, auxiliary systemmay be configured to present visual content to surgical team memberswho may not have access to the images provided to surgeon-at user control system. To this end, auxiliary systemmay include a display monitorconfigured to display one or more user interfaces, such as images (e.g., 2D images) of the surgical space, information associated with patientand/or the surgical procedure, and/or any other visual content as may serve a particular implementation. For example, display monitormay display images of the surgical space together with additional content (e.g., representations of target objects, graphical content, contextual information, etc.) concurrently displayed with the images. In some embodiments, display monitoris implemented by a touchscreen display with which surgical team membersmay interact (e.g., by way of touch gestures) to provide user input to surgical system.

102 104 106 102 104 106 116 102 104 106 1 FIG. Manipulating system, user control system, and auxiliary systemmay be communicatively coupled one to another in any suitable manner. For example, as shown in, manipulating system, user control system, and auxiliary systemmay be communicatively coupled by way of control lines, which may represent any wired or wireless communication link as may serve a particular implementation. To this end, manipulating system, user control system, and auxiliary systemmay each include one or more wired or wireless communication interfaces, such as one or more local area network interfaces, Wi-Fi network interfaces, cellular interfaces, etc.

2 FIG. 2 FIG. 200 200 200 202 204 202 204 202 204 202 204 shows an example automatic zoom systemthat may be implemented according to principles described herein to implement a zoom feature associated with an imaging device. As shown in, automatic zoom system(e.g., system) may include, without limitation, a memoryand a processorselectively and communicatively coupled to one another. Memoryand processormay each include or be implemented by hardware and/or software components (e.g., processors, memories, communication interfaces, instructions stored in memory for execution by the processors, etc.). In some examples, memoryand processormay be implemented by a single device (e.g., a single computing device). In certain alternate examples memoryand processormay be distributed between multiple devices and/or multiple locations as may serve a particular implementation.

202 204 202 206 204 206 Memorymay maintain (e.g., store) executable data used by processorto perform any of the operations described herein. For example, memorymay store instructionsthat may be executed by processorto perform any of the operations described herein. Instructionsmay be implemented by any suitable application, software, code, and/or other executable data instance.

202 204 202 Memorymay also maintain any data received, generated, managed, used, and/or transmitted by processor. For example, memorymay maintain any suitable data associated with implementing an automatic zoom feature. Such data may include, but is not limited to, data associated with depth map information associated with an imaging space, movement data (e.g., kinematics data for imaging devices and/or manipulator arms, etc.), object data, procedure data, image data (e.g., endoscopic images) of an imaging space, data defining guidance content associated with an image zoom feature, predefined threshold distance data, user interface content (e.g., graphical objects, notifications, etc.), and/or any other suitable data.

204 206 202 204 Processormay be configured to perform (e.g., execute instructionsstored in memory) various processing operations associated with implementing a zoom feature associated with an imaging device. For example, processormay automatically activate an image zoom feature to zoom in a view of an imaging space.

204 As used herein, the expression “automatically” means that an operation (e.g., an operation activating a zoom feature) or series of operations are performed without requiring further input from a user. For example, a determination that a user is moving or intends to move an imaging device closer to an object than a threshold distance may trigger automatic control of a computer-assisted surgical system to prevent the imaging device from moving closer to the object than the threshold distance and automatic activation of an image zoom feature to zoom in a view of the imaging space. These and other operations that may be performed by processorare described herein.

3 3 FIGS.A-B 300 300 1 300 4 300 1 302 304 304 304 304 304 302 304 illustrate example views(e.g., views-through-) of an imaging space that may be implemented according to principles described herein. As shown in view-, an imaging deviceis provided within the imaging space in relation to an object. Objectmay represent any suitable type of object that may be located in an imaging space. For example, objectmay represent a physical object such as anatomical tissue, an instrument, or any other suitable object that may be located in the imaging space. In certain examples, objectmay represent an organ or a portion of an organ located in an imaging space. The distance between objectand imaging devicemay be measured to the surface of object.

302 302 302 304 302 302 Imaging devicemay correspond to any suitable type of imaging device that may be located within an imaging space. For example, imaging devicemay correspond to an endoscope in certain examples. Imaging devicemay capture one or more images in the imaging space. Any robotic instruments and/or objectthat are within a field of view of imaging devicemay be depicted in the image(s) captured by imaging device.

302 302 Imaging devicemay include any suitable number or type of sensors as may serve a particular implementation. For example, imaging devicemay include one or more cameras, depth sensors, capacitive sensors, ultrasonic sensors, optical time of flight (TOF) depth sensing sensors, and/or any other suitable type of sensor.

302 302 302 302 302 302 302 The sensors that may be included as part of imaging devicemay be configured in any suitable manner and may be located on any suitable portion of imaging device. For example, imaging devicemay include an image sensor at a distal end of imaging device. Additionally or alternatively, imaging devicemay include an image sensor at a proximal end of imaging device. In such examples, light collected at the distal end of imaging devicemay be transmitted to the image sensor at the proximal end by way of a light guide or in any other suitable manner.

302 302 302 Imaging devicemay provide data representing visible light data of an imaging space. For example, imaging devicemay capture visible light images of the imaging space that represent visible light sensed by imaging device. Visible light images may include images that use any suitable color and/or grayscale palette to represent a visible light-based view of the imaging space.

302 302 In certain additional or alternative examples, imaging devicemay provide data representing non-visible light data. In such examples, imaging devicemay include one or more infrared sensors and/or near infrared sensors.

302 302 302 302 302 Imaging devicemay also provide data representing depth data of an imaging space or data that may be processed to derive depth data of the imaging space. For example, imaging devicemay capture images of the imaging space that represent depth sensed by imaging device. Alternatively, imaging devicemay capture images of the imaging space that may be processed to derive depth data of the imaging space. The depth information may be represented as depth images (e.g., depth map images obtained using a Z-buffer that indicates distance from imaging deviceto each pixel point on an image of an imaging space), which may be configured to visually indicate depths of objects in the imaging space in any suitable way, such as by using different greyscale values to represent different depth values.

302 302 302 304 Images captured by imaging deviceand/or derived from images captured by imaging device(e.g., visible light images and depth images) may be used to facilitate detecting a position of imaging devicein relation to objectand/or determining when to automatically implement a zoom feature, such as described herein.

302 302 302 304 In certain examples, imaging devicemay additionally or alternatively include any suitable other type of distance/proximity sensor(s). For example, imaging devicemay include one or more capacitive sensors, ultrasonic sensors, optical time of flight (TOF) depth sensing sensors, and/or any other suitable type of sensor to determine the distance/proximity of imaging devicewith respect to object.

302 302 302 302 In certain examples, imaging devicemay include a plurality of lenses configured to facilitate an optical zoom function of imaging device. Imaging devicemay include any suitable number and/or configuration of lenses as may serve a particular implementation. For example, imaging devicemay include two lenses, three lenses, four lenses, and so forth.

3 FIG.A 3 3 FIGS.A andB 3 FIG.A 3 FIG.A 302 306 304 302 304 300 1 308 308 1 308 4 302 308 1 302 308 308 304 302 304 304 As shown in, imaging devicehas a field of viewof objectat the distance that imaging deviceis currently located from objectin view-. Images(e.g., images-through-) represent images captured by imaging deviceat the various positions depicted in. For example, image-inshows an image captured by imaging deviceat the position shown on the left side of. An “A” is shown in imagesfor illustrative purposes only to show how imagesof objectmay change based on the distance imaging deviceis from objectand/or the amount of zoom applied. It is understood that objectmay correspond to tissue or some other object in the imaging space that does not include a letter “A” or any other letter on the surface thereof.

300 2 302 304 308 2 302 304 302 302 As shown in view-, moving imaging devicerelatively closer to objectincreases the size of the “A” captured within image-. However, moving imaging devicecloser to objectmay cause, for example, tissue burning and/or may result in obscuring an image captured by imaging devicedue to fogging and/or splattering. In such examples, the tissue burning may be caused by illumination emitted at a distal end of imaging device.

200 302 302 310 302 304 304 302 304 304 3 FIG.B In view of this, systemmay implement one or more threshold distances that may be used in certain examples to determine when to either automatically activate or automatically deactivate an image zoom feature. In some implementations, a threshold distance may define a minimum safe distance that imaging device(e.g., a distal end of imaging device) may be positioned with respect to an object that does not result in one or more of the above-described negative effects. The threshold distance may correspond to any suitable distance from an object in an imaging space as may serve a particular implementation. In the example shown in, a threshold distanceis shown in dashed lines to illustrate a distance that a distal end of imaging devicemay stay away from objectto effectively capture images of objectand avoid causing one or more of the above-described negative effects associated with moving imaging deviceto close to object(e.g., cause damage to object).

200 200 200 In certain examples, the threshold distance used by systemmay be predefined. In certain alternative examples, systemmay select a threshold distance based on one or more factors associated with the imaging space. Systemmay use any suitable factor to select a threshold distance as may serve a particular implementation. For example, the one or more factors may include a type of procedure performed in the imaging space, attributes of (e.g., a type or model of, a resolution of, an illuminator of, heat generated by, etc.) an imaging device used, and/or attributes of (e.g., a type of) an object imaged in the imaging space. For example, a first threshold distance may be selected for a first type of procedure, a second threshold distance may be selected for a second type of procedure, and a third threshold distance may be selected for a third type of procedure. The first, second, and third threshold distances may each be different from one another. Additionally or alternatively, the one or more factors may include taking into consideration the best possible distance to view small tissue details, the best possible distance to avoid a fusing problem due to too strong parallax/stereo impression, and/or any other suitable factor.

200 200 In certain examples, systemmay facilitate a user selecting a threshold distance to use in a particular situation. In such examples, systemmay provide any suitable user interface to facilitate the user selecting the threshold distance.

200 200 200 In certain examples, systemmay be configured to dynamically change the threshold distance that is used during the course of a procedure performed in the imaging space. To illustrate, a first stage of the procedure may include capturing images of soft tissue where there is a relatively higher splatter risk. A second stage of the procedure may include capturing images of a bone of a subject where there is a relatively lower splatter risk. As such, it may be possible to move the imaging device closer to the bone of the subject than the soft tissue without resulting in splatter on the imaging device. Accordingly, systemmay select a first threshold distance to use during the first stage of the procedure and a second threshold distance to use during the second stage of the procedure. The first threshold distance may be relatively longer than the second threshold distance. Systemmay be configured to automatically switch from using the first threshold distance to using the second threshold distance upon initiation of the second stage of the surgical procedure.

200 200 302 200 302 200 302 302 200 302 In certain examples, systemmay be configured to implement any suitable object recognition algorithm to dynamically change the threshold distance that is used at a given time. For example, systemmay use an object recognition algorithm to determine that imaging deviceis currently capturing images of soft tissue. Based on such a determination, systemmay select a first threshold distance to use while capturing images of the soft tissue. The view of imaging devicemay then be adjusted and systemmay determine that imaging deviceis currently capturing images of bone or some other object that may not be as susceptible to burning and/or may not cause splatter on imaging device. As such, systemmay select a second threshold distance to use while imaging devicecaptures images of the bone.

300 4 200 302 304 310 200 306 312 308 4 110 1 302 302 304 302 304 310 3 FIG.B As shown in view-of, instead of systemmoving imaging devicecloser to objectthan threshold distance, systemmay automatically activate an image zoom feature to narrow field of viewto field of viewand result in a relatively larger “A” in image-. In so doing, the automatic zoom feature may provide an operator (e.g., surgeon-) of imaging devicewith an impression that imaging deviceis moving closer to objecteven though imaging devicedoes not actually move closer to objectthan threshold distance.

4 FIG. 400 200 204 402 200 302 110 1 illustrates a flow diagramdepicting various operations that may be performed by system(e.g., processor) to automatically implement a zoom feature such as described herein. At operation, systemmay receive an instruction to move an imaging device (e.g., imaging device) within an imaging space. The instruction may indicate that a user (e.g., surgeon-) is moving or intends to move the imaging device with respect to an object in the imaging space.

200 110 1 104 112 2 Systemmay receive the instruction in any suitable manner. For example, the instruction may be received by way of surgeon-interacting with master controls of user control systemto direct a robotic manipulating arm (e.g., robotic manipulating arm-) attached to the imaging device to move within the imaging space.

404 200 200 200 200 At operation, systemmay determine, based on the instruction, whether the imaging device is located at a position that is at or near a threshold distance from an object. Systemmay determine whether the imaging device at or near the threshold distance in any suitable manner. For example, systemmay access depth data that indicates how far the imaging device is from the object. Based on the depth data, systemmay determine that the imaging device would be closer to the object than the threshold distance if the imaging device is moved to within the imaging space based on the instruction.

In certain examples, the determining that the imaging device is located at the position may include determining that a distal end of the imaging device is located at the position. In such examples, the distal end of the imaging device may include an illumination device that generates heat and may cause damage to the object if the distal end is moved closer to the object than the threshold distance.

200 In certain examples, systemmay additionally or alternatively determine whether the imaging device is at or near the threshold distance by using an auto focus function of the imaging device. The auto focus function of the imaging device may indicate how far the imaging device is from the object at any given time.

404 402 404 406 406 200 200 If the answer at operationis “NO,” the flow returns to operation. If the answer at operationis “YES,” the flow proceeds to operation. At operation, systemmay prevent the imaging device from moving closer to the object than the threshold distance. This may be accomplished in any suitable manner. For example, systemmay instruct motors of the manipulator arm to which the imaging device is attached to stop moving towards the object once the threshold distance is reached or once the imaging device is within a predefined distance from the threshold distance. In addition, the motors may be prevented in any suitable manner from moving the imaging device in the z-direction past the threshold distance. However, the user may still be able to move the imaging device in the x-direction, the y-direction, and/or may be able to pivot the imaging device in any suitable manner.

In certain examples, the preventing of the imaging device from moving closer to the object than the threshold distance may include preventing the distal end of the imaging device from moving closer to the object than the threshold distance.

200 200 104 In certain examples, systemmay be configured to provide feedback to inform a user of the computer-assisted surgical that the threshold distance has been reached. Such feedback may include any suitable feedback (e.g., any of visual, audible, and haptic feedback) and may be provided in any suitable manner. For example, systemmay instruct a computer-assisted surgical system to cause one of the master controls of user control systemto vibrate to inform the user when the threshold distance is reached.

408 200 At operation, systemmay automatically activate an image zoom feature associated with the imaging device to zoom in a view of the imaging space. With the image zoom feature activated, physical movement of the imaging device in the z-direction beyond the threshold distance and towards the object is replaced with zooming in on the object. This may give the impression to the user that the imaging device is moving closer to the object even though the imaging device does not physically move closer to the object than the threshold distance.

200 In certain examples, systemmay obtain motion data associated with motion of the imaging device before automatically activating the image zoom feature.

200 200 200 Such motion data may include any suitable data associated with movement of the imaging device that may be used to control the image zoom function. For example, the motion data may be based on kinematic data, a motion sensor (e.g., one or more accelerometers) in the imaging device, data derived from images captured by the imaging device, and/or any other suitable data. Systemmay control the image zoom function in any suitable manner based on the motion data. For example, systemmay determine a velocity of the imaging device while the image zoom function is active. Systemmay then control the zoom rate to mimic the effect of the imaging device moving at the same velocity.

200 200 In certain examples, the automatically activating of the image zoom feature may be imperceptible to a user. That is, the user may not be aware that the imaging device has physically stopped at or near the threshold distance and zoom is being applied. This may be accomplished in any suitable manner. For example, systemmay determine a velocity of a distal tip of the imaging device. Based on the determined velocity, systemmay determine a rate of change of zoom applied by the zoom feature to substantially match the perceptible output of moving the distal tip of the imaging device closer to the object at the determined velocity.

200 200 200 200 200 In certain examples, the view of the imaging space may include a view, from a virtual camera, of a 3D model of at least a portion of the imaging space. Such a 3D model may be generated in any suitable manner. For example, systemmay generate the 3D model in real time or near real time based on images captured by the imaging device. In examples where systemuses a real time or near real time 3D model (e.g., a simultaneous localization and mapping (SLAM) based 3D model) of the imaging space, systemmay determine a 3D trajectory of the imaging device. Once the threshold distance is reached, systemmay prevent the imaging device from moving closer to the object than the threshold distance by stopping the imaging device at a point along the 3D trajectory. The automatically activating of the zoom feature may include moving the virtual camera closer to the object than the imaging device. For example, systemmay move the virtual camera along that 3D trajectory (and at a similar velocity) past the point after stopping the imaging device at the point. In such examples, the virtual camera may produce a virtual image that mimics moving the imaging device closer to the object in a manner that is imperceptible to the user.

200 200 In certain alternative implementations, systemmay be configured to provide a notification to a user that the image zoom feature has been activated. Such a notification may be provided in any suitable manner. For example, systemmay provide an audio notification (e.g., a voice message saying “zoom feature activated”), a text notification (e.g., the text “zoom feature activated”), an augmented reality notification, an icon notification (e.g., a magnifying glass icon), and/or any other suitable notification.

200 200 Systemmay automatically activate the image zoom feature in any suitable manner. In certain examples, the automatically activating of the image zoom feature may include activating an optical zoom feature of the imaging device. In such examples, the imaging device may be configured with a plurality of lenses that may be configured to facilitate optically zooming in on an object in any suitable manner. For example, systemmay be configured to change focal lengths between lenses included in the plurality of lenses to narrow the field of view and zoom in a view of the imaging space.

200 200 200 In certain examples, the automatically activating of the image zoom feature may include activating a digital zoom feature of the imaging device. Systemmay implement a digital zoom feature in any suitable manner. For example, systemmay decrease the field of view of an image by cropping the image captured by the imaging device down to an area with the same aspect ratio of the original image and scaling the cropped image up to the dimensions of the original image. In such examples, the image zoom feature may include systemdigitally zooming in on a particular portion of an image captured by the imaging device.

200 200 200 200 200 200 200 In certain examples, with a digital zoom feature, systemmay be configured to select which portion of a captured image to zoom in on based on an input provided by a user. Systemmay be configured to detect any suitable user input as may serve a particular implementation. For example, systemmay be configured to track the gaze of one or more eyes of a user of a computer-assisted surgical system and may digitally zoom in on a portion of the image that the user is currently looking at. In such examples, systemmay be configured to implement any suitable gaze tracking methodology as may serve a particular implementation. Additionally or alternatively, systemmay select which portion of a captured image to zoom in on based on a trajectory of the imaging device. Additionally or alternatively, systemmay select which portion of a captured image to zoom in on based on a detected portion of interest in an image captured by the imaging device In certain alternative examples, the automatically activating of the image zoom feature may include activating both an optical zoom feature and a digital zoom feature. In such examples, systemmay first optically zoom in on the object and may then digitally zoom in on any suitable portion of the image captured by the imaging device.

200 While the zoom feature is active, systemmay dynamically adjust the amount of zoom provided at any given time based on instructions to move the imaging device in the z-direction, the x-direction, the y-direction, and/or based on pivoting movements that change the viewing direction of the imaging device.

200 200 In addition, while the image zoom feature is active, systemmay perform any suitable processing operation(s) on images captured by the imaging device to give the impression that the imaging device is closer to the object than the threshold distance. For example, systemmay crop an image captured by the imaging device, dynamically adjust a perspective of the image, dynamically adjust distortion of all or part of the image, apply three-dimensional (“3D”) texture mapping to the image, dynamically adjust the scale of a 3D model or 3D rendering that may be displayed together with the image, artificially brighten the image, and/or dynamically adjust a zoom ratio depending on how close the user would move the imaging device to the object.

200 200 200 200 200 200 In certain examples, systemmay determine a virtual zoomed in pivot point for the imaging device. Such a virtual zoomed in pivot point may be determined in any suitable manner. For example, a virtual zoomed in pivot point may be determined based on the position of the imaging device at or near the threshold distance. Any change in the physical position and/or orientation of the imaging device also changes the virtual zoomed in pivot point. Further, the movement of the virtual zoomed in pivot point while the image zoom feature is active may change and/or otherwise affect the kinematics of the manipulator arm attached to the imaging device. Accordingly, systemmay take into consideration the virtual zoomed in pivot point when zooming in and/or processing a zoomed in image. In addition, systemmay take into consideration the virtual zoomed in pivot point when the imaging device is physically moved and/or pivoted while the image zoom feature is activated. For example, when a user input is provided to move the imaging device in the x-direction and/or the y-direction, systemmay take into consideration the virtual zoomed in pivot point, such as by using the virtual zoomed in pivot point as an input when translating the user input into a movement and/or by applying the movement based on the virtual zoomed in pivot point. Similarly, when a user input is additionally or alternatively provided to rotate the imaging device along a yaw and/or pitch axis while the image zoom feature is active, systemmay consider and use the virtual zoomed in pivot point as an input when translating the user input into a rotating motion along the yaw and/or pitch axis. For example, the amount of yaw/pitch rotation needed at the virtual zoomed in pivot point may be greater than that needed at the physical pivot point where the imaging device is physically located. Similarly, when a user input is additionally or alternatively provided to pivot the imaging device while the image zoom feature is active, systemmay consider and use the virtual zoomed in pivot point as an input when translating the user input into a pivoting motion and/or by applying the pivot based on the virtual zoomed in pivot point. This may allow the virtual location of the virtual pivot point to change within the imaging space even though the physical position of the imaging device may not change and/or may allow actual movement of the imaging device to be based on a zoomed in viewpoint such that the virtual zoomed in viewpoint remains realistic to a user.

5 FIG. 5 FIG. 5 FIG. 500 302 304 310 502 302 302 504 506 302 504 200 506 506 302 304 200 506 502 shows an example of a viewof an imaging space while an image zoom feature is active. As shown in, imaging deviceis prevented from moving closer to objectthan threshold distance. A pivot pointcorresponds to an actual pivot point of imaging device. Imaging deviceis currently zoomed in by an amount associated with dotted line. A virtual zoomed in pivot pointcorresponds to a virtual pivot point of imaging devicebased on the amount of zoom associated with dotted line. Systemmay be configured to dynamically adjust the location of virtual zoomed in pivot pointand/or image processing associated with virtual zoomed in pivot pointin any suitable manner based on movement of imaging devicein relation to object. For example, systemmay adjust the viewing angle associated with virtual zoomed in pivot pointbased on imaging device pivoting in the direction of the arrows shown inwith respect to pivot point.

200 200 200 In certain examples, while the image zoom feature is active, systemmay calculate a virtual camera calibration (e.g., calibration of the focal length) from the perspective of the virtual zoomed in pivot point. Systemmay implement such a virtual camera calibration in any suitable manner to facilitate providing a zoomed in image to a user. For example, systemmay adjust the depth mapping of the imaging space, the rendering of user interface elements, and/or any other aspect based on the virtual camera calibration.

200 110 1 104 100 200 Systemmay display a zoomed in image to a user of a computer-assisted surgical system in any suitable manner. For example, the zoomed in image may be displayed to surgeon-by way of a stereoscopic image viewer of user control system. In certain examples, systemmay display the zoomed in image together with an image captured by the imaging device that is not zoomed in. This may be accomplished in any suitable manner. For example, systemmay display a first window that depicts the zoomed in image and a second relatively smaller window that depicts an image that is not zoomed in. In certain examples, the second relatively smaller window may be overlaid over the first window (e.g., at a corner of the first window).

410 200 At operation, systemmay receive an additional instruction to move the imaging device to an additional position within the imaging space.

412 200 412 410 412 414 200 200 200 200 414 410 At operation, systemmay determine whether the additional instruction will move the imaging device farther from the object than the threshold distance. If the answer at operationis “NO,” the flow returns to operation. If the answer at operationis “YES,” the flow proceeds to operationin which the image zoom feature is automatically deactivated. Systemmay automatically deactivate the image zoom feature in any suitable manner. For example, based on the instruction, systemmay automatically zoom out until no more zoom is implemented by system. After zooming out, systemmay deactivate the image zoom feature at operationwhen no more zoom is applied and/or when the imaging device begins moving toward the additional position associated with the additional instruction at operation.

200 200 In certain examples, systemmay implement a different threshold distance for deactivating an image zoom feature than is used for activating an image zoom feature. For example, systemmay use a first threshold distance to determine when to activate the image zoom feature and may use a second threshold distance, different than the first threshold distance, to determine when to deactivate the image zoom feature. In certain examples, the second threshold distance may be greater than the first threshold distance. Having the second threshold distance be greater than the first threshold distance may improve usability by preventing repeatedly and quickly switching back and forth between modes based on a single threshold boundary.

414 402 200 After operation, the flow may then return to operationin which systemwaits to receive an additional instruction associated with the movement of the imaging device in the imaging space.

200 402 414 Systemmay repeat operations-any suitable number of times during the course of a procedure performed in an imaging space to facilitate automatically activating and automatically deactivating an image zoom feature associated with an imaging device.

200 In certain examples, systemmay provide an option for a user to manually deactivate the image zoom feature of an imaging device. Such an option may provide the user with full control of the imaging device also at close distances from an object in an imaging space.

200 200 200 200 200 200 200 200 In certain alternative examples, systemmay automatically activate an image zoom feature based on a detected attribute of an object in an imaging space. Systemmay use any suitable attribute of an object to determine when to automatically activate an image zoom feature. For example, systemmay use the temperature of an object such as tissue as an attribute that may be used to determine when to automatically activate an image zoom feature in certain implementations. In such examples, systemmay detect, in any suitable manner, a surface temperature of the tissue that is being imaged by an imaging device. Systemmay determine whether the temperature is above a predefined threshold that may cause damage to the tissue. If the temperature is above the predefined threshold temperature, systemmay automatically activate the image zoom feature and automatically move the imaging device any suitable distance away from the tissue to thereby reduce the surface temperature. While the imaging device moves away from the tissue, systemmay gradually increase the zoom, which may give the impression to a user that the imaging device is not moving away from the tissue. In so doing, systemmay prevent or mitigate damage that may occur to the tissue due to the heat emitted from the imaging device while still providing an up-close field of view that may be desired by a user.

200 Although the preceding disclosure describes an automatic zoom feature that automatically zooms in a view of an imaging space, it is understood that concepts such as those described herein may also be applied in the opposite direction. For example, an automatic zoom feature may additionally or alternatively include automatically zooming out based on a condition being satisfied. For example, if a user would like a wider field of view, the user may move an imaging device away from an object in the imaging space. However, at some point, the imaging device may not be able to physically move farther backward (e.g., without exiting a body cavity and/or due to kinematic constraints). In such examples, systemmay determine in any suitable manner that the imaging device is approaching an additional threshold distance away from an object, may prevent the imaging device from moving past the additional threshold distance, and may automatically zoom out (e.g., digitally and/or optically) to widen the field of view without physically moving the imaging device past the additional threshold distance.

6 FIG. 6 FIG. 6 FIG. 6 FIG. 600 200 illustrates an example methodfor implementing a zoom feature associated with an imaging device in an imaging space. Whileillustrates example operations according to one embodiment, other embodiments may omit, add to, reorder, and/or modify any of the operations shown in. One or more of the operations shown inmay be performed by a system such as system, any components included therein, and/or any implementation thereof.

602 200 602 At operation, an automatic zoom system (e.g., automatic zoom system) may receive an instruction to move an imaging device, which is located within an imaging space and coupled to a robotic manipulating arm of a computer-assisted surgical system, towards an object in the imaging space. Operationmay be performed in any of the ways described herein.

604 604 At operation, the automatic zoom system may determine, based on the instruction, that the imaging device is located at a position that is at or near a predefined distance from the object in the imaging space. Operationmay be performed in any of the ways described herein.

606 606 At operation, the automatic zoom system may prevent the imaging device from moving closer to the object than the threshold distance. Operationmay be performed in any of the ways described herein.

608 608 At operation, the automatic zoom system may automatically activate, in response to the imaging device being prevented from moving closer to the object than the threshold distance, an image zoom feature to zoom in a view of the imaging space. Operationmay be performed in any of the ways described herein.

In some examples, a non-transitory computer-readable medium storing computer-readable instructions may be provided in accordance with the principles described herein. The instructions, when executed by a processor of a computing device, may direct the processor and/or computing device to perform one or more operations, including one or more of the operations described herein. Such instructions may be stored and/or transmitted using any of a variety of known computer-readable media.

A non-transitory computer-readable medium as referred to herein may include any non-transitory storage medium that participates in providing data (e.g., instructions) that may be read and/or executed by a computing device (e.g., by a processor of a computing device). For example, a non-transitory computer-readable medium may include, but is not limited to, any combination of non-volatile storage media and/or volatile storage media. Illustrative non-volatile storage media include, but are not limited to, read-only memory, flash memory, a solid-state drive, a magnetic storage device (e.g., a hard disk, a floppy disk, magnetic tape, etc.), ferroelectric random-access memory (“RAM”), and an optical disc (e.g., a compact disc, a digital video disc, a Blu-ray disc, etc.). Illustrative volatile storage media include, but are not limited to, RAM (e.g., dynamic RAM).

7 FIG. 7 FIG. 7 FIG. 7 FIG. 7 FIG. 700 700 702 704 706 708 710 700 700 illustrates an example computing devicethat may be specifically configured to perform one or more of the processes described herein. As shown in, computing devicemay include a communication interface, a processor, a storage device, and an input/output (“I/O”) modulecommunicatively connected one to another via a communication infrastructure. While an example computing deviceis shown in, the components illustrated inare not intended to be limiting. Additional or alternative components may be used in other embodiments. Components of computing deviceshown inwill now be described in additional detail.

702 702 Communication interfacemay be configured to communicate with one or more computing devices. Examples of communication interfaceinclude, without limitation, a wired network interface (such as a network interface card), a wireless network interface (such as a wireless network interface card), a modem, an audio/video connection, and any other suitable interface.

704 704 712 706 Processorgenerally represents any type or form of processing unit capable of processing data and/or interpreting, executing, and/or directing execution of one or more of the instructions, processes, and/or operations described herein. Processormay perform operations by executing computer-executable instructions(e.g., an application, software, code, and/or other executable data instance) stored in storage device.

706 706 706 712 704 706 706 Storage devicemay include one or more data storage media, devices, or configurations and may employ any type, form, and combination of data storage media and/or device. For example, storage devicemay include, but is not limited to, any combination of the non-volatile media and/or volatile media described herein. Electronic data, including data described herein, may be temporarily and/or permanently stored in storage device. For example, data representative of computer-executable instructionsconfigured to direct processorto perform any of the operations described herein may be stored within storage device. In some examples, data may be arranged in one or more databases residing within storage device.

708 708 708 I/O modulemay include one or more I/O modules configured to receive user input and provide user output. One or more I/O modules may be used to receive input for a single virtual experience. I/O modulemay include any hardware, firmware, software, or combination thereof supportive of input and output capabilities. For example, I/O modulemay include hardware and/or software for capturing user input, including, but not limited to, a keyboard or keypad, a touchscreen component (e.g., touchscreen display), a receiver (e.g., an RF or infrared receiver), motion sensors, and/or one or more input buttons.

708 708 I/O modulemay include one or more devices for presenting output to a user, including, but not limited to, a graphics engine, a display (e.g., a display screen), one or more output drivers (e.g., display drivers), one or more audio speakers, and one or more audio drivers. In certain embodiments, I/O moduleis configured to provide graphical data to a display for presentation to a user. The graphical data may be representative of one or more graphical user interfaces and/or any other graphical content as may serve a particular implementation.

700 202 706 204 704 In some examples, any of the systems, computing devices, and/or other components described herein may be implemented by computing device. For example, memorymay be implemented by storage device, and processormay be implemented by processor.

In the preceding description, various example embodiments have been described with reference to the accompanying drawings. It will, however, be evident that various modifications and changes may be made thereto, and additional embodiments may be implemented, without departing from the scope of the invention as set forth in the claims that follow. For example, certain features of one embodiment described herein may be combined with or substituted for features of another embodiment described herein. The description and drawings are accordingly to be regarded in an illustrative rather than a restrictive sense.