Training and Feedback for a Controller Workspace Boundary

This application is a continuation of co-pending U.S. patent application Ser. No. 18/442,253, filed Feb. 15, 2024, entitled “TRAINING AND FEEDBACK FOR A CONTROLLER WORKSPACE BOUNDARY,” which is a continuation of U.S. patent application Ser. No. 16/833,275, filed Mar. 27, 2020, entitled “TRAINING AND FEEDBACK FOR A CONTROLLER WORKSPACE BOUNDARY,” now U.S. Pat. No. 11,911,120, issued Feb. 27, 2024, all of which are incorporated herein by reference in their entirety.

Embodiments relating to surgical robotic systems, are disclosed. More particularly, embodiments related to training and feedback for a user operating an ungrounded controller in an invisible workspace, are disclosed.

Minimally-invasive surgery (MIS), such as endoscopic surgery, involves looking into a patient's body and performing surgery inside the body using endoscopes and other surgical tools. For example, laparoscopic surgery can use a laparoscope to access and view an abdominal cavity. Endoscopic surgery can be performed using manual tools and/or a surgical robotic system having robotically-assisted tools.

A surgical robotic system may be remotely operated by a surgeon to command a robotically-assisted tool located at an operating table. Such operation of a robotically-assisted tool remotely by a surgeon may be commonly referred to as teleoperation. For example, the surgeon may use a computer console located in the operating room, or it may be located in a different city, to command a robot to manipulate the surgical tool mounted on the operating table. The robotically-controlled surgical tool can be an endoscope mounted on a robotic arm. Accordingly, the surgical robotic system may be used by the remote surgeon to perform an endoscopic surgery.

The surgeon may provide input commands to the surgical robotic system, and one or more processors of the surgical robotic system can control system components in response to the input commands. For example, the surgeon may hold in her hand a user input device that she manipulates to generate control signals to cause motion of the surgical robotic system components, e.g., an actuator, a robotic arm, and/or a surgical tool of the robotic system.

In some aspects, the surgical robotic system components may be controlled using a user input device or controller that is ungrounded. In the case of ungrounded controllers, which are only usable within a defined workspace area, the user may need training before usage and feedback during usage to ensure the controllers are being used within an operable range. For example, since the ungrounded (e.g., mid-air) controllers have an invisible workspace the user needs to learn in what space she or he can use the controllers and in what space the controllers are inactive. Therefore, in one aspect, the instant disclosure is directed to different multimodal methods for training the user so that they can build a mental model of where they are allowed to move the controllers to. Different multimodal approaches are also provided to give real-time feedback during usage which will prevent the user from going out of the workspace.

Representatively, in one aspect, a multimodal process may include three categories: (1) Training, (2) Pre-Usage Reminder and (3) Feedback During Usage. For the Training, there may be a task that is specifically designed to learn the boundary of the workspace area. For example, there could be a display which shows an outer box representing the actual controller workspace, an inner box representing a warning zone, and spheres representing the controllers within the inner box. If the user moves the controllers into the warning zone, the inner box is shown pushed out and turns a different color to warn the user they may be reaching the workspace limit. If the user reaches the outer box with the controllers, the outer box may turn red and an alarm may sound. The user will be able to fully focus on this one task which will allow her to create a mental model of the workspace. As a Pre-Usage Reminder, the user may perform a quick task during a login sequence to set her or his mind back to the workable area. For the Feedback During Usage there may be GUI and multimodal feedback in a non-intrusive way to inform the user that she or he is approaching the workspace limit (e.g., warning displayed next to the screen). The feedback may be seen in the periphery without interfering with the task at hand. For the sake of simplicity, a box-shaped workspace may be assumed, but these concepts can be expanded to differently shaped workspaces such as a sphere or ellipsoid or any type of three-dimensional (3D) shape.

Representatively, in one aspect, a method of determining a location of a user input device of a surgical robotic system within a surgical workspace using a virtual workspace includes determining, by one or more processors communicatively coupled to a user input device, that a user is engaging with the user input device within a surgical workspace; in response to determining the user is engaging with the user input device, displaying a virtual user input device within a first virtual workspace boundary, the first virtual workspace boundary representing a first workspace limit within which the user input device is operable to control a surgical robotic instrument in a teleoperation mode, wherein at least a portion of the first virtual workspace boundary is operable to move in response to a movement of the user input device; displaying a second virtual workspace boundary that represents a second workspace limit beyond which the user input device is inoperable to control the surgical robotic instrument in the teleoperation mode; and determining, by one or more processors, a location of the user input device within the surgical workspace based on a proximity of the portion of the first virtual workspace boundary relative to the second virtual workspace boundary. In some aspects, the first virtual workspace boundary comprises a first three dimensional shape and the second virtual workspace boundary comprises a second three dimensional shape that encompasses the first three dimensional shape. In some aspects, an area between the first three dimensional shape and the second three dimensional shape defines a warning zone that indicates the user input device is nearing the second workspace limit. In still further aspects, the first virtual workspace boundary comprises a cube. The portion of the first virtual workspace boundary operable to move may be a side wall of the cube. In some aspects, determining the location may include detecting that the portion of the first virtual workspace boundary has moved closer to the second virtual workspace boundary, and the method may further include in response to the detecting, providing user feedback. The feedback may include changing a visual characteristic of the first virtual workspace boundary. Determining the location may include detecting that the portion of the first virtual workspace boundary intersects with the second virtual workspace boundary, and the method further includes in response to the detecting, providing user feedback. The feedback may include changing a visual characteristic of the first virtual workspace boundary or the second virtual workspace boundary. The feedback may include an audible alert or haptic feedback. The user input device may be an ungrounded user input device.

In another aspect, a method of determining a location of an ungrounded user input device of a surgical robotic system within a surgical workspace includes determining, by one or more processors communicatively coupled to the surgical robotic system, whether a user is engaging with an ungrounded user input device; in response to determining the user is engaging with the ungrounded user input device, determining, by one or more processors communicatively coupled to the surgical robotic system, whether the ungrounded user input device is within a warning zone of the surgical workspace; and in response to determining the user input device is within the warning zone, providing user feedback corresponding to a location of the user input device relative to the surgical workspace. The surgical workspace may include a workspace limit beyond which the ungrounded user input device is inoperable to control a surgical robotic instrument, and the warning zone comprises an area of the surgical workspace near the workspace limit. In some aspects, determining the ungrounded user input device is within a warning zone comprises: providing a first virtual three-dimensional boundary representing an inner boundary of the warning zone; providing a second virtual three-dimensional boundary representing a workspace limit beyond which the ungrounded user input device is inoperable to control a surgical robotic instrument, the second virtual three-dimensional boundary encompasses the first virtual three-dimensional boundary; and determining a location of the user input device within the surgical workspace corresponds to an area between the first virtual three-dimensional boundary and the second virtual three-dimensional boundary. In some aspects, providing user feedback comprises providing the user with information indicating the user input device is near a workspace limit of the surgical workspace. In other aspects, providing user feedback comprises providing the user with information indicating the user input device has exceeded a workspace limit of the surgical workspace. The surgical workspace may include a workspace limit defined by a plurality of sides, and providing user feedback comprises providing the user with information indicating which of the plurality of sides the user input device is closest to. The plurality of sides may define a workspace limit having a three-dimensional shape. The user input device may be a first user input device, and the surgical robotic system may further include a second user input device, and providing user feedback comprises providing the user with information indicating whether the first user input device or the second user input device is in the warning zone. The surgical workspace may include a three-dimensional shape and providing user feedback further comprises providing the user with information indicating which side of the three-dimensional shape the first user input device or the second user input device is closest to. The feedback may include a visual feedback on a display of the surgical robotic system. The visual feedback may include at least one icon, and a location of the at least one icon on the display indicates the location of the user input device relative to the surgical workspace. The visual feedback may include at least one icon, and a location of the at least one icon on the display indicates whether the user input device within the warning zone is a left user input device or a right user input device. The visual feedback may include at least one icon, and a size or shape of the at least one icon changes to indicate the location of the user input device relative to the surgical workspace.

In another aspect, a method of determining a position of a user input device of a surgical robotic system within a surgical workspace based on a virtual workspace, the method comprising: determining, by one or more processors communicatively coupled to the surgical robotic system, whether a user controlling the user input device is in at least one mode of operation selected from a training mode, a pre-usage reminder mode, and a feedback during usage mode; in response to determining the user is in a training mode, assigning a user input device training operation to be performed within a virtual workspace boundary corresponding to the surgical workspace; in response to determining the user is in a pre-usage reminder mode, assigning a user input device engagement operation to be performed within the virtual workspace boundary after at least one other engagement operation is performed; and in response to determining the user is in a feedback during usage mode, determining whether the user input device is within a warning zone of the surgical workspace, and upon determining the user input device is within the warning zone, providing user feedback. The training mode may occur prior to a teleoperation mode in which the user input device is operable to control a surgical robotic instrument, and the training operation comprises moving the user input device without exceeding the virtual workspace boundary. The pre-usage reminder mode may occur prior to a teleoperation mode in which the user input device is operable to control a surgical robotic instrument, and the at least one other engagement operation comprises positioning a chair of the surgical robotic system to a position suitable for teleoperation, and the engagement operation comprises moving the user input device to a side of the virtual workspace boundary. The feedback during usage mode may occur during a teleoperation mode in which the user input device is operable to control a surgical robotic instrument, and providing feedback comprises indicating to the user (1) the user input device is nearing a side of the surgical workspace, (2) which side of the surgical workspace the user input device is closest to or (3) whether the user input device is a left user input device or a right user input device.

The above summary does not include an exhaustive list of all aspects of the present invention. It is contemplated that the invention includes all systems and methods that can be practiced from all suitable combinations of the various aspects summarized above, as well as those disclosed in the Detailed Description below and particularly pointed out in the claims filed with the application. Such combinations have particular advantages not specifically recited in the above summary.

In various embodiments, description is made with reference to the figures. However, certain embodiments may be practiced without one or more of these specific details, or in combination with other known methods and configurations. In the following description, numerous specific details are set forth, such as specific configurations, dimensions, and processes, in order to provide a thorough understanding of the embodiments. In other instances, well-known processes and manufacturing techniques have not been described in particular detail in order to not unnecessarily obscure the description. Reference throughout this specification to “one embodiment,” “an embodiment,” or the like, means that a particular feature, structure, configuration, or characteristic described is included in at least one embodiment. Thus, the appearance of the phrase “one embodiment,” “an embodiment,” or the like, in various places throughout this specification are not necessarily referring to the same embodiment. Furthermore, the particular features, structures, configurations, or characteristics may be combined in any suitable manner in one or more embodiments.

In addition, the terminology used herein is for the purpose of describing particular aspects only and is not intended to be limiting of the invention. Spatially relative terms, such as “beneath”, “below”, “lower”, “above”, “upper”, and the like may be used herein for ease of description to describe one element's or feature's relationship to another element(s) or feature(s) as illustrated in the figures. It will be understood that the spatially relative terms are intended to encompass different orientations of the device in use or operation in addition to the orientation depicted in the figures. For example, if the device in the figures is turned over, elements described as “below” or “beneath” other elements or features would then be oriented “above” the other elements or features. Thus, the exemplary term “below” can encompass both an orientation of above and below. The device may be otherwise oriented (e.g., rotated 90 degrees or at other orientations) and the spatially relative descriptors used herein interpreted accordingly.

As used herein, the singular forms “a”, “an”, and “the” are intended to include the plural forms as well, unless the context indicates otherwise. It will be further understood that the terms “comprises” and/or “comprising” specify the presence of stated features, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and/or groups thereof.

The terms “or” and “and/or” as used herein are to be interpreted as inclusive or meaning any one or any combination. Therefore, “A, B or C” or “A, B and/or C” mean “any of the following: A; B; C; A and B; A and C; B and C; A, B and C.” An exception to this definition will occur only when a combination of elements, functions, steps or acts are in some way inherently mutually exclusive.

Moreover, the use of relative terms throughout the description may denote a relative position or direction. For example, “distal” may indicate a first direction away from a reference point, e.g., away from a user. Similarly, “proximal” may indicate a location in a second direction opposite to the first direction, e.g., toward the user. Such terms are provided to establish relative frames of reference, however, and are not intended to limit the use or orientation of any particular surgical robotic component to a specific configuration described in the various embodiments below.

Referring to, this is a pictorial view of an example surgical robotic systemin an operating arena. The surgical robotic systemincludes a user console, a control tower, and one or more surgical robots, including robotic armsat a surgical robotic platform, e.g., an operating table, a bed, etc. The systemcan incorporate any number of devices, tools, or accessories used to perform surgery on a patient. For example, the systemmay include one or more surgical toolsused to perform surgery. A surgical toolmay be an end effector that is attached to a distal end of a surgical arm, for executing a surgical procedure.

Each surgical toolmay be manipulated manually, robotically, or both, during the surgery. For example, the surgical toolmay be a tool used to enter, view, or manipulate an internal anatomy of the patient. In an embodiment, the surgical toolis a grasper that can grasp tissue of the patient. The surgical toolmay be controlled manually, by a bedside operator; or it may be controlled robotically, via actuated movement of the surgical robotic armto which it is attached. The surgical robotic armsare shown as a table-mounted system, but in other configurations the surgical robotic armsmay be mounted in a cart, ceiling or sidewall, or in another suitable structural support.

Generally, a remote operator, such as a surgeon or other operator, may use the user consoleto remotely manipulate the surgical robotic armsand/or the attached surgical tools, e.g., teleoperation. The user consolemay be located in the same operating room as the rest of the system, as shown in. In other environments however, the user consolemay be located in an adjacent or nearby room, or it may be at a remote location, e.g., in a different building, city, or country. The user consolemay comprise a seat, one or more user interface devices, for example, foot-operated controlsor handheld user input devices (UID), and at least one user displaythat is configured to display, for example, a view of the surgical site inside the patient. In the example user console, the remote operatoris sitting in the seatand viewing the user displaywhile manipulating a foot-operated controland a handheld UIDin order to remotely control the armsand the surgical tools(that are mounted on the distal ends of the arms).

In some variations, the bedside operatormay also operate the systemin an “over the bed” mode, in which the bedside operator(user) is now at a side of the patientand is simultaneously manipulating a robotically-driven tool (end effector as attached to the arm), e.g., with a handheld UIDheld in one hand, and a manual laparoscopic tool. For example, the bedside operator's left hand may be manipulating the handheld UIDto control a robotic component, while the bedside operator's right hand may be manipulating a manual laparoscopic tool. Thus, in these variations, the bedside operatormay perform both robotic-assisted minimally invasive surgery and manual laparoscopic surgery on the patient.

During an example procedure (surgery), the patientis prepped and draped in a sterile fashion to achieve anesthesia. Initial access to the surgical site may be performed manually while the arms of the robotic systemare in a stowed configuration or withdrawn configuration (to facilitate access to the surgical site). Once access is completed, initial positioning or preparation of the robotic systemincluding its armsmay be performed. Next, the surgery proceeds with the remote operatorat the user consoleutilizing the foot-operated controlsand the UIDsto manipulate the various end effectors and perhaps an imaging system, to perform the surgery. Manual assistance may also be provided at the procedure bed or table, by sterile-gowned bedside personnel, e.g., the bedside operatorwho may perform tasks such as retracting tissues, performing manual repositioning, and tool exchange upon one or more of the robotic arms. Non-sterile personnel may also be present to assist the remote operatorat the user console. When the procedure or surgery is completed, the systemand the user consolemay be configured or set in a state to facilitate post-operative procedures such as cleaning or sterilization and healthcare record entry or printout via the user console.

In one embodiment, the remote operatorholds and moves the UIDto provide an input command to move a robot arm actuatorin the robotic system. The UIDmay be communicatively coupled to the rest of the robotic system, e.g., via a console computer system. Representatively, in some embodiments, UIDmay be a portable handheld user input device or controller that is ungrounded with respect to another component of the surgical robotic system. For example, UIDmay be ungrounded while either tethered or untethered from the user console. The term “ungrounded” is intended to refer to implementations where, for example, both UIDs are neither mechanically nor kinematically constrained with respect to the user console. For example, a user may hold a UIDin a hand and move freely to any possible position and orientation within a workspace, only limited by, for example, a predetermined three-dimensional surgical workspace limit or boundary recognized by the system. Representatively, the system may include a tracking mechanism that tracks the location of the UIDwithin, and relative to, the surgical workspace limit or boundary. The UIDcan generate spatial state signals corresponding to movement of the UID, e.g., position and orientation of the handheld housing of the UID, and the spatial state signals may be input signals to control a motion of the robot arm actuator. The signals (e.g., tracking sensor signals, clutch signals or engage/disengage teleoperation mode signals) may be wirelessly communicated between UIDand the computer system. In addition, a power source, such as a rechargeable battery, may be stored within the housing of UIDso that it does not need to be mechanically connected to a power source, such as by a wire or cable. The robotic systemmay use control signals derived from the spatial state signals, to control proportional motion of the actuator. In one embodiment, a console processor of the console computer systemreceives the spatial state signals and generates the corresponding control signals. Based on these control signals, which control how the actuatoris energized to move a segment or link of the arm, the movement of a corresponding surgical tool that is attached to the arm may mimic the movement of the UID. Similarly, interaction between the remote operatorand the UIDcan generate for example a grip control signal that causes a jaw of a grasper of the surgical toolto close and grip the tissue of patient.

The surgical robotic systemmay include several UIDs, where respective control signals are generated for each UID that controls the actuators and the surgical tool (end effector) of a respective arm. For example, the remote operatormay move a first UID(e.g., a left UID) to control the motion of an actuatorthat is in a left robotic arm, where the actuator responds by moving linkages, gears, etc., in that arm. Similarly, movement of a second UID(e.g., a right UID) by the remote operatorcontrols the motion of another actuator, which in turn moves other linkages, gears, etc., of the robotic system. The robotic systemmay include a right armthat is secured to the bed or table to the right side of the patient, and a left armthat is at the left side of the patient. An actuatormay include one or more motors that are controlled so that they drive the rotation of a joint of the arm, to for example change, relative to the patient, an orientation of an endoscope or a grasper of the surgical toolthat is attached to that arm. Motion of several actuatorsin the same armcan be controlled by the spatial state signals generated from a particular UID. The UIDscan also control motion of respective surgical tool graspers. For example, each UIDcan generate a respective grip signal to control motion of an actuator, e.g., a linear actuator, that opens or closes jaws of the grasper at a distal end of surgical toolto grip tissue within patient.

In some aspects, the communication between the platformand the user consolemay be through a control tower, which may translate user commands that are received from the user console(and more particularly from the console computer system) into robotic control commands that are transmitted to the armson the robotic platform. The control towermay also transmit status and feedback from the platformback to the user console. The communication connections between the robotic platform, the user console, and the control towermay be via wired and/or wireless links, using any suitable ones of a variety of data communication protocols. Any wired connections may be optionally built into the floor and/or walls or ceiling of the operating room. The robotic systemmay provide video output to one or more displays, including displays within the operating room as well as remote displays that are accessible via the Internet or other networks. The video output or feed may also be encrypted to ensure privacy and all or portions of the video output may be saved to a server or electronic healthcare record system. It will be appreciated that the operating room scene inis illustrative and may not accurately represent certain medical practices.

As previously discussed, a UIDof the surgical robotic systemmay be used to control the actuators and the surgical tool (end effector) of a respective armonce certain operational conditions are met. Representatively, to control the actuators and/or surgical tool, the user may hold the UIDand perform at least one intentional action that indicates the user is ready to control the actuators and/or surgical tool in a teleoperation mode. The teleoperation mode is a mode of the surgical robotic system in which the user is actively controlling, or is able to actively control, the surgical robotic system components, e.g., an actuator, a robotic arm, a surgical tool and/or endoscope, for example, with a user input device or foot pedal. For example, the intentional action may be a gesture detected by the user input device, such as the user tapping on the user input device or squeezing the user input device. Alternatively, the action may be an action which results in the user input device being in a particular location, orientation and/or position, such as the user docking the user input device in a docking station or positioning the user input device in close proximity to the docking station. On the other hand, when the user is unable to actively control the surgical robotic system components, the system is considered to have exited the teleoperation mode, be out of the teleoperation mode, be in a non-teleoperation mode or have disengaged the teleoperation mode. For example, the system may be considered to have exited or disengaged the teleoperation mode or be in a non-teleoperation mode when (1) no user input is accepted by the surgical robotic system, (2) user input commands to a graphical user interface (GUI) of the system are accepted, but cannot control the associated surgical robotic system components or (3) the user is not yet able to control the surgical robotic components, but a sequence of intentional actions would cause the user to enter or engage teleoperation mode. Once in teleoperation mode, movement of the UIDby the user within a defined workspace area controls the actuators and/or surgical tool.

As previously discussed, when the UIDis an ungrounded (e.g., mid-air) controller, the workspace within which the UIDis able to control components of the system is an invisible surgical workspace the user needs to learn before she or he can use the UIDduring teleoperation. Therefore, in one aspect, the system may implement a multimodal process to help the user learn, understand and work within the invisible workspace. For example, the multimodal process may include three modes or categories: (1) Training, (2) Pre-Usage Reminder and (3) Feedback During Usage. The Training mode may occur prior to any surgical operations (e.g., prior to the teleoperation mode) and allow the user to build a mental model of where they are allowed to move the UID. The Pre-Usage Reminder mode may also occur prior to a surgical operation (e.g., immediately before the user engages in teleoperation mode) and may remind the user of the workspace boundary or limits prior to using the UID. The Feedback During Usage mode may occur during the teleoperation mode, for example, when the user is using the UIDto operate a surgical component, and gives real-time feedback about the location of the UID within the workspace. The desired mode (e.g., Training, Pre-Usage Reminder and/or Feedback During Usage) may be selected by the user, or may be automatically selected by the surgical robotic system.

provides a block diagram of an exemplary multi-modal process of a surgical robotic system. Representatively, the multi-modal processmay include detecting that the UID is engaged at operation. The engagement operationmay be any operation by the user that the surgical robotic system detects as the user engaging with, or otherwise holding, a UID (e.g., UID). For example, engagement of the UID may be detected by a sensor (e.g., a proximity or pressure sensor coupled to the UID) that detects when a user holds or grasps UIDwith a hand, performs a gesture with the UID, or some other predefined active operation by the user that the system recognizes as meaning the user is engaging with the UID. Once engagement is detected, a Training mode may be initiated at operation. The Training mode may be automatically initiated by the system once engagement of a UID is detected, or selected manually by a user. As previously discussed, the Training mode occurs prior to any surgical operations (e.g., prior to the teleoperation mode) and allows the user to build a mental model of where they are allowed to move the UID. The Training mode will be discussed in more detail in reference to-B andA-B.

The system then determines whether the Training mode has been completed by the user at operation. If the Training mode is not completed, the Training mode may continue, restart, repeat, etc., depending on where the user is at in the training operation. Once the Training mode is completed, the Pre-Usage Reminder mode is initiated at operation. As previously discussed, the Pre-Usage Reminder mode may also occur prior to a surgical operation, for example immediately before the user engages in teleoperation mode, and may remind the user prior to using the UIDof the workspace. The Pre-Usage Reminder mode will be discussed in more detail in reference to.

The system then determines whether the Pre-Usage Reminder mode is completed by the user at operation. If the Pre-Usage Reminder mode is not completed by the user, the system returns to operationuntil the Pre-Usage Reminder mode is completed. Once the Pre-Usage Reminder mode is completed, the Feedback During Usage mode is initiated at operation. The Feedback During Usage mode may occur during the teleoperation mode, for example, when the user is using the UIDto operate a surgical component, and gives real-time feedback about the location of the UID within the workspace. The Feedback During Usage mode will be discussed in more detail in reference to.

The system then determines at operationwhether a teleoperation mode is also engaged. If it is, the process returns to operationand the Feedback During Usage mode continues until the system determines teleoperation is no longer engaged. If teleoperation is no longer engaged, for example because the UID is no longer within the workspace limit, the process may return to the Pre-Usage Reminder mode at operation. The user will then continue in the Pre-Usage Reminder mode until the system determines it can proceed to the Feedback During Usage mode (e.g., the Reminder mode is completed and teleoperation is engaged). It should be understood that although the Training, Pre-Usage Reminder and Feedback During Usage modes are described one after the other in, it should be understood that other modes (e.g., disengagement of teleoperation, etc.) may occur between each of these modes and they do not necessarily have to occur during any particular time frame. For example, the Training modecould occur over a series of days, and once completed, need not occur again before the Pre-Usage Reminder mode.

Referring now to the various modes in more detail,provides a block diagram of an exemplary Training mode of a surgical robotic system. The Training mode(e.g., operationof) may include a task that is specifically designed to help the user learn the boundary of the workspace area. The user may be instructed or guided so they can fully focus on the designated task so that they can create a mental model of the workspace. Representatively, Training modemay begin when the training mode is selected at operation(e.g., by the user) and a user input device engagement operationis detected by the surgical robotic system. The engagement operationmay be any operation by the user that the surgical robotic system detects as the user engaging with, or otherwise holding, a UID (e.g., UID). For example, engagement of the UID may be detected by a sensor (e.g., a proximity or pressure sensor coupled to the UID) that detects when a user holds or grasps UIDwith a hand, performs a gesture with the UID, or some other predefined active operation by the user that the system recognizes as meaning the user is engaging with the UID. Once engagement is detected, a first virtual workspace boundary may be provided at operation. The first virtual workspace boundary may be displayed to the user and provide a visual representation of the real surgical workspace within which the UID should be manipulated in order to operate an associated surgical component (e.g., a surgical tool). In other words, as long as the operator is manipulating the UID within a real workspace area that corresponds to the virtual area defined by the first virtual workspace boundary, the user will be able to control an associated surgical component during a real surgical procedure. The movement of the real UID may be tracked by a tracking device of the surgical robotic system and the movement may be displayed as a corresponding virtual movement of a virtual UID relative to the first virtual workspace boundary. The user can therefore see whether they are within the workspace boundary or limit, or exceeding the boundary or limit, and begin to understand the workspace limits they should stay within during a real procedure. As previously mentioned, the first virtual workspace boundary may represent a desired area within which the UID should be during a surgical procedure. If the virtual UID exceeds the first virtual workspace boundary, the real UID may still be able to control an associated surgical component, but be considered within a “warning zone.” The “warning zone” may be an area of operation or workspace region that is considered too close to a boundary or limit where the UID is no longer allowed to control an associated component. In this aspect, the first virtual workspace boundary may also be referred to as a boundary for the warning zone.

To indicate to the user the workspace limit, boundary or region beyond which control of an associated surgical component by the UID is no longer allowed (e.g., the UID is too far outside of the workspace), the training operation also provides the user with a second virtual workspace boundary at operation. The second virtual workspace boundary may be displayed to the user and provide a visual representation of an area within which the UID must be manipulated in order to operate an associated surgical component (e.g., a surgical tool). If the movement of the UID exceeds the second virtual workspace boundary, the UID is considered to have exceeded the workspace limit or boundary and is prevented from controlling an associated surgical component by the system. In some cases, the teleoperation mode is disengaged in response to detecting that the workspace boundary or limit has been met or exceeded by the UID. In this aspect, the second virtual workspace boundary may be displayed to the user as a boundary or limit that is outside of the first virtual workspace boundary. For example, in some aspects, the first virtual workspace boundary and the second virtual workspace boundary may be displayed to the user as three-dimensional shapes. The three-dimensional shape of the second virtual workspace boundary may be larger (e.g., define a larger area) than that of the first virtual workspace boundary. In this aspect, the three-dimensional shapes may be displayed to the user as one inside of the other. For example, the three-dimensional shape of the first virtual workspace boundary may be displayed to the user as located within, or otherwise encompassed by, the three-dimensional shape of the second virtual workspace boundary. In this aspect, the area within (or at) the boundary of the inner three-dimensional shape (e.g., the first virtual workspace) may be understood by the user as the desired area of operation of the UID. The area or zone between the inner three-dimensional boundary (e.g., the first virtual workspace boundary) and the outer three-dimensional boundary (e.g., the second virtual workspace boundary) may be understood by the user as the warning zone or area indicating the user is approaching the operation limit of the UID. The area beyond (or at) the outer three-dimensional boundary (e.g., the second virtual workspace boundary) may be understood by the user as an area in which the UID has exceeded its operational limit and is therefore no longer able to control an associated surgical component. Accordingly, when the user moves the real UID, and this movement is shown as the virtual UID exceeding the inner three-dimensional boundary, approaching the outer three-dimensional boundary, or exceeding the outer three-dimensional boundary, the user begins to recognize and learn the operational boundaries of the real workspace.

In some aspects, a portion of the inner three-dimensional boundary follows, or otherwise moves in response to, the movement of the virtual UID. Accordingly, in such cases, a proximity of the first workspace boundary (e.g., three-dimensional shape) to the second workspace boundary (e.g., three dimensional shape) may be determined (e.g., measured) at operation. The proximity information determined at operationmay then be used to determine whether the UID is nearing the second workspace boundary at operation.

,,andschematically illustrate representative Training operations in accordance with. Representatively,andschematically illustrate exemplary three-dimensional shapes corresponding to the virtual workspace boundaries, and which are displayed to the user, as described in training operation.andillustrate cross-sectional side views ofandalong lines-′ and-′, respectively.

Referring now in more detail to,illustrates a training operationin which a first virtual workspace boundaryand a second virtual workspace boundaryare displayed to the user as three-dimensional cube or box-shaped workspaces. For example, the first virtual workspace boundarymay be an inner three-dimensional cube or box-shaped workspace and the second virtual workspace boundarymay be an outer three-dimensional cube or box-shaped workspace encompassing the inner cube or box-shaped workspace. The three-dimensional shapes may be a virtual workspace or area within the bounds of which the user can learn to operate the UID. The virtual workspace, in turn, represents the area of the real workspace boundary or limit within which the surgeon can use the real UID. The inner three-dimensional shape (e.g., boundary) may represent an inner boundary of a warning zone within the actual workspace (e.g., an area considered close to or near an actual workspace limit). The outer three-dimensional shape (e.g., boundary) may represent the boundary of the real workspace limit (e.g., the boundary beyond which the UID is no longer operable).

Although not shown, each of the three-dimensional cube or box-shaped workspaces may be considered bounded by six square faces, facets or sides, that are connected by lines as shown. For example, in operation, virtual UIDsA-B are displayed to the user as located within boundaryand/or boundary. At least one face or sideof the three-dimensional shape forming the first workspace boundaryis also displayed to the user. The sidemay be flexible such that it moves outward when a UID exceeds boundary. For example, as shown in operation, UIDA exceeds the boundary, while UIDB is within boundary. This outward movement of the sidein response to the movement of the UIDA signals to the user that the movement of the UIDA has exceeded boundary. In other words, the UIDA is now in the warning zone between boundaryand boundary. Since the movement of UIDA is mapped to the movement of the real UIDA within the real workspace, the user begins to learn the workspace boundaries and when they are nearing a workspace limit. In some aspects, the proximity of the boundaryto boundarycan therefore be used to determine the UID locations within the workspaces, and whether the UIDs are nearing the boundaries,.

Representatively,shows a cross-sectional side view of, along line-′. From this view, it can be seen that the movement of the real UIDA is causing virtual UIDA to exceed boundary, while the movement of UIDB is within the boundary, therefore UIDB is shown within boundary. The movement of UIDA pushes sideof boundaryoutward toward boundary. In other words, UIDA is now within the warning zonebetween boundaryand boundary. Since UIDA moves sidewhen it reaches or exceeds the boundary, the proximity of the siderelative to the sideof boundarycan be used to determine if the UID is nearing a workspace boundary or limit, and a location of the UID within the workspace in general. For example, when it is determined that a distance Dbetween sideand sidenear UIDA is less than a predetermined distance D(e.g., distance between sidesandwhen not pushed out by the UID), the system indicates to the user that the UID is nearing the boundary(e.g., within the warning zone).

In some aspects, additional alerts may be provided to inform the user that they are nearing or exceeding a workspace boundary. For example, in some aspects, a visual characteristic of one or more of boundaries,and/or UIDsA-B may change to alert the user that they are nearing or exceeding a workspace boundary. The visual characteristic may, for example, be a change in the color of one or more of the boundaries or UIDs when within a warning zone or workspace limit. For example, sideof boundarymay be a first color when it is in its normal position and not being pushed out by a UID. When a UID causes sideto move toward boundary(e.g., into the warning zone), sidemay turn a second color. Still further, when sideis pushed out so far that it intersects with boundary, it may turn a third color. For example, sidemay be black when in the normal position, turn orange when in the warning zone and turn red when it intersects boundary. In addition, the outer boundarymay turn a different color depending on the proximity of boundaryto boundary. For example, when sideof boundaryintersects with boundary, boundarymay change from a first color to a second color (e.g., from black to red). In still further aspects, the alert may be an audible alarm or haptic response that occurs when the user is nearing or exceeding a workspace limit.

illustrates an operationin which the UID is shown exceeding the workspace limit and therefore no longer operable. Representatively, as can be seen from, UIDA has exceeded boundaryand is within (or exceeding) boundary. This operation can be both visually displayed to the user as shown in, and determined by the system based on a proximity of the sideof boundaryrelative to the sideof boundaryas shown in. For example, when sideintersects with side, Dmay equal D, and operationdetermines that the boundaryhas been met or exceeded. Once boundaryis met or exceeded, UID is no longer within a workspace in which it is allowed to operate the associated surgical component, and the system alerts the user (e.g., visual color changes, audible alarm, haptic response, etc.). This, in turn, helps the user understand the corresponding real workspace boundaries within which the real UID must be used.

In some aspects, to further assist the user in learning the workspace limits, the system may assign the user with a task that asks the user to move one or more of the UIDs around without going out of the boundaryand/or boundary. The task could be in combination with a “pause” button/pedal. While the “pause” button/pedal is held, the representation of the UIDs on the display are not translated with the movement of the real UIDs. A user could then be asked to move the on-screen UIDs to a location shown on the display screen without hitting the workspace limits. The virtual workspace boundaries (e.g., boundaries,) would then be visible only when the user moves the UID within the warning zone or outside a workspace limit.

provides a block diagram of an exemplary Pre-Usage Reminder mode of a surgical robotic system. As previously discussed, the Pre-Usage Reminder mode or process(e.g., operationof) may be part of an initial engagement sequence that the operator may be asked to perform to remind them of the workspace limits prior to engagement of the teleoperation mode. For example, the Pre-Usage reminder mode or processmay occur when the user uses the surgical robotic system for the first time that day, for the first time after a long delay, or for the first time after any predetermined period of time. Representatively, processmay include an initial operation of detecting an engagement of a UID at operation. Engagement of the UID may be detected by any of the previously discussed operations, for example, proximity or pressure sensor detecting when a user holds or grasps UIDwith a hand, performs a gesture with the UID, or some other predefined active operation by the user that the system recognizes as meaning the user is engaging with the UID. Once engagement is detected, processmay further include detection of at least one pre-teleoperation mode engagement operation at operation. The at least one pre-teleoperation mode engagement operation may be, for example, the user positioning their chair to a position suitable for teleoperation, or detection of some other user or system characteristic that suggests the user is about to engage in teleoperation. Once a pre-teleoperation mode engagement operation is detected, the process includes detecting at least one pre-teleoperation mode workspace task at operation. The pre-teleoperation mode workspace task may be any task that reminds the user of the boundaries or limits of the workspace, and confirms to the system that the user remembers the boundaries or limits of the workspace. For example, the task could be requiring the user to hold the UID close to an edge or side of the workspace boundary or limit (e.g., close to a left edge or close to a right edge). If the user is able to move the UID within the virtual workspace close to the edge without exceeding the boundary or limit, the system determines the user remembers the boundaries or limits. If the user exceeds the edge of the boundary or limit, the system may require the user to complete one or more additional tasks until the user successfully completes the task(s). The one or more additional tasks may include, for example, moving the UID close to the same edge again, moving the UID close to another edge, or moving the UID close to multiple edges in sequence. Once the task(s) is completed successfully, the system determines that the user remembers the workspace boundaries or limits and is ready to operate the UID within the workspace at operation.

Once both the Training modeand Pre-Usage Reminder modeare completed, a Feedback During Usage mode may be initiated.provides a block diagram of an exemplary Feedback During Usage mode of a surgical robotic system. As previously discussed, the Feedback During Usage mode or process(e.g., operationof) may be a process which occurs while the user is operating the UID within the real workspace and provides real time feedback of the UID location relative to the workspace boundaries. Representatively, processmay include detecting engagement of a UID at operationand detecting engagement of teleoperation mode at operation. Once both of these operations are detected, the system knows that the user is now manipulating the UID within the workspace and intends to use it to control a surgical component (e.g., surgical tool or instrument). The processtherefore continues to monitor the UID location within the workspace and can detect if the UID is in a warning zone of the virtual workspace at operation. If the system detects that the UID is in a warning zone, and therefore nearing the workspace limit, the system may provide feedback to the user about a location of the UID at operation. Representatively, the feedback may be that a UID is in the warning zone, a distance of a UID to the workspace limit when within the warning zone, whether the UID nearing the workspace limit is a left or right UID, which edge or side of the workspace limit a UID is closest to, or which tool associated with the UID location is affected. This information about the UID relative to the workspace may be determined using the proximity information previously discussed in reference to. The feedback may be any one or more of the previously discussed feedback mechanisms, including a visual feedback, haptic feedback or audio feedback, or a combination of any one or more of a visual, haptic or audio feedback.

provide examples of the visual feedback that may be provided to the user during the Feedback During Usage mode. Representatively,illustrates a display screenthat the surgeon can use to view the movement of a surgical instrumentthey are controlling with the UID during a surgical procedure on a patient. To provide visual feedback to the user about the UID location relative to the workspace during the surgical procedure, an indicatormay also appear on the display screen. In some aspects, the indicatormay only appear when the UID is determined to be in a warning zone. The indicatormay be a text or icon that appears in one specific location that is known to the user, but does not interfere with the user's view of the surgical procedure displayed on the screen. Since the surgeon knows where to expect the indicator, as soon as it appears, the user immediately recognizes that a UID is within a warning zone of the workspace and therefore nearing a workspace limit. In some aspects, the indicatorindicates that at least one UID is within the warning zone and close to a workspace limit, but does not indicate additional details such as which UID (e.g., left or right UID) is in the warning zone, or which boundary the UID is closest to. Since the indicatordoes not, in this aspect, need to also indicate which UID is in the warning zone and/or a UID location relative to a boundary, the indicator may appear anywhere on the display screenthat can be viewed by the surgeon. For example, the indicatormay appear in a central area of the screen, for example, at the bottom of the screen in the center. The indicatormay also have any size, shape, color, or other visual characteristic that is recognized by the user as an indicator that a UID is in a warning zone.

In other aspects, the indicatormay, in addition to indicating to the user that the UID is in a warning zone, indicate to the user which UID is within the warning zone and/or which boundary of the warning zone the UID is closest to. For example,illustrates an indictorA that indicates to the surgeon which UID is approaching a workspace boundary or limit. Representatively, in, display screenis shown having a left side panelA and a right side panelB. The left side panelA may display information relating to a left UID (e.g., a UID held in the left hand), and the right side panelB may display information relating to a right UID (e.g., a UID held in the right hand). Accordingly, when the right UID is detected within the warning zone, an indicatorA will appear in the right side panelB, as shown in. In this aspect, the indicatorA is shown as a line along the right side panelB. The indicatorA could, however, be any type of shape, image, icon or the like that can be viewed by the user in the right side panelB. The surgeon will therefore understand that the UID held in his or her right hand is in the warning zone and nearing the workspace limit. Similarly, although not shown, if the left UID is detected in the warning zone, a similar warning indicator may appear in the left side panelB. The indicatorA may appear in the center of the left or right panelsA-B, or in any other region of the left or right panelsA-B that is easily viewed and understood by the surgeon to mean a left and/or right UID is within the warning zone. In addition, in some aspects, the display may also include an indicatorB including text which also informs the surgeon that the UID is approaching the workspace limit and/or which side of the workspace the UID is near. For example, the indicatorB may also appear in the right side panelB to indicate that the UID is nearing the right boundary of the workspace, and also include text stating that the UID is “Approaching the workspace limit.” Although both indicatorsA,B are shown in, it is contemplated that in some aspects, only one of indicatorsA,B may be displayed to indicate that a UID is approaching a workspace boundary or limit (i.e., in the warning zone), and which UID (right or left) is approaching the workspace boundary or limit.

In still further aspects, the feedback provided to the user may identify which edge or boundary a UID is closest to. Representatively,are schematic illustrations of indicators that may be used to inform the user which edge a UID is closest to. The indicatorsA-F may appear on the display screen, similar to the previously discussed visual indicators. In this aspect, however, the indicatorsA-F illustrates the boundaries or edges,,,of the workspace. Specifically, the workspace is represented by a top edge, a right side edge, a bottom side edgeand a left side edge. The edges,,,are understood by the surgeon as corresponding to the top, right, bottom and left sides of the actual workspace limit of the UID being manipulated by the surgeon. Although edges,,,are shown as straight edges (e.g., forming a square) it should be understood that the edges, boundaries and/or limits discussed herein may have other shapes and/or orientations relative to one another so long as they can be understood by the user as representing a workspace boundary or limit. For example, the edges,,,could be curved, or have any other shape. In addition, in some aspects only the edge or a portion of the edge that the UID is closest to could be shown.

The indicatorsA-F further illustrate to the surgeon a UID iconand position icon. The UID iconmay represent the UID within the workspace and may be any type of icon that the surgeon recognizes as representing a UID within a workspace. The position iconrepresents the side or edge of the workspace that the UID is closest to. In other words, the warning zone corresponding to, or otherwise defined by, that particular side or edge. The position iconmay appear between the UID iconand the workspace edge,,,that the UID is closest to. The position iconmay therefore appear when a UID is detected within a warning zone at a particular side or edge. If no UID is detected near a particular side or within a warning zone defined by a particular side or edge, the position iconmay disappear or not otherwise be displayed. If no position iconis present, the surgeon will understand this to mean that no UID is within a warning zone or otherwise close to a particular side or edge of the workspace. The UID iconmay remain stationary relative to the workspace edges,,,.

Referring now in more detail to,shows an indicatorA with the position icondisplayed near the bottom edgeof the workspace. Accordingly, the surgeon will understand that at least one UID being engaged by the surgeon is within the warning zone near the bottom edge of the real workspace.shows an indicatorB with the position iconnear the top edgeof the workspace. Based on the indicatorB shown in, the surgeon will understand that at least one UID being engaged by the surgeon is within the warning zone near the top edge of the real workspace.shows an indicatorC with the position iconnear the left edgeof the workspace. Based on the indicatorC shown in, the surgeon will understand that at least one UID being engaged by the surgeon is within the warning zone near the left edge of the real workspace.shows an indicatorD with the position iconnear the right edgeof the workspace. Based on the indicatorD shown in, the surgeon will understand that at least one UID being engaged by the surgeon is within the warning zone near the right edge of the real workspace.

andprovide additional indicators that illustrate to the surgeon whether they are nearing a front or back edge of the workspace. Representatively, similar to the indicators shown in, a position iconis used to indicate which edge the UID is closest to, except in this embodiment, the position iconappears in front of or behind the UID iconto indicate the location of the UID relative to the front or back edge. For example, indicatorE ofshows the position iconin front of the UID iconto indicate to the user that the UID is close to the front edge (farthest from the user) of the workspace. The indicatorF ofshows the position iconbehind the UID iconto indicate to the user that the UID is close to the back edge (closest to the user) of the workspace. It should be understood that while the position iconsare shown inas lines or crosses, any shape or size icon suitable for indicating a UID location relative to a boundary may be used.

In some aspects, in addition to indicating to the user that a UID is closest to a particular side, the system may also indicate visually to the user which UID (e.g., right or left UID) is closest to the particular side.provides a schematic illustration of one such indicator. Representatively,shows the previously discussed indicatorsD andF described in reference toand, displayed on the display screenthat is viewed by the user. In addition,illustrates whether the left or right UID is at the location represented by indicatorsD,F based on the position of the indicatorsD,F on the screen. In particular, the indicator may be positioned in the bottom left cornerA of the screen to indicate the left UID position and the indicator may be positioned in the bottom right cornerB of the screen to indicate the right UID position. For example, indicatorD is positioned at the bottom left cornerA of the display screento indicate to the user that the left UID is close to the right edge of the workspace. IndicatorF is positioned at the bottom right cornerB of the display screento indicate to the user that the right UID is close to the back edge of the workspace. Although indicatorsD andF are illustrated in, it should be understood that any one or more of indicatorsA-F described inmay be displayed in the cornersA-B to indicate whether a right or left UID is at the indicated location. Similar to the previously discussed feedback operations, if none of indicatorsA-F are displayed in the cornersA-B, this means that no UID is within a warning zone or otherwise approaching a workspace limit.

provides an additional visual feedback indicator that may be provided to the user during the Feedback During Usage mode to indicate to the surgeon whether they are nearing, meeting and/or exceeding a workspace limit. Representatively,illustrates a display screenthat the surgeon can use to view the movement of a surgical instrumentthey are controlling with the UID during a surgical procedure on a patient. The display screenfurther includes a left panelA and a right panelB that may be used to display an indicator. In some aspects, the panelsA andB may also correspond to a particular left or right UID. For example, the left panelA may correspond to a left UID and the right panelB may correspond to a right UID. Accordingly, if the indicatoris in the right panelB as shown, the user will understand that the indicatoris providing information about the right UID in the user's right hand. Similarly, if the indicatoris in the left penalA, the user will understand that the indicatoris providing information about the left UID in the user's left hand. Similar to the previously discussed indicators, indicatorindicates to the user whether they are nearing a workspace limit. Indicatorin this aspect, however, is shown as a series of concentric circles or rings. The intensity of the circles or ringschange based on a distance between the UID and the workspace limit or boundary. For example, the circles or ringsgrow outwardly to the right side of the displayas the UID gets closer to the workspace limit.shows the circles or ringsextending all the way to the right side of panelB. Accordingly, the user would understand this to mean that the UID in the user's right hand has reached the workspace limit, and is possibly exceeding the workspace limit or boundary. Since the UID is no longer within the workspace boundaries, the UID is no longer operable to control the surgical instrument. On the other hand, if only a single ringnear the left side of the panelB is shown, this indicates to the user that the user has moved the UID within the warning zone, but the UID is still relatively far from the workspace limit. If the user continues to move the UID in the same direction, however, the intensity (e.g., number) of the ringswill increase indicating to the user that they are getting closer to the workspace limit. The user can therefore adjust the movement of the UID to ensure they remain within the workspace limit.