Methods and systems for positioning robots and adjusting postures

This application is a continuation of International Application No. PCT/CN2022/092003, filed on May 10, 2022, which claims priority to Chinese Patent Application No. 202110505732.6, filed on May 10, 2021, titled “METHODS, APPARATUS, SYSTEMS, AND COMPUTER DEVICES FOR POSITIONING ROBOTS,” and Chinese Patent Application No. 202111400891.6, filed on Nov. 19, 2021, titled “METHODS, SYSTEMS, AND STORAGE MEDIA FOR ADJUSTING POSTURES OF CAMERAS AND SPATIAL REGISTRATION,” the entire contents of each of which are hereby incorporated by reference.

The present disclosure relates to the field of robots, and in particular, to methods and systems for positioning robots and adjusting postures.

In recent years, robots are widely used in the medical field, such as orthopedics, neurosurgery, thoracoabdominal interventional surgeries or treatment, etc. Generally speaking, a robot includes a robotic arm with a multi-degree-of-freedom structure, which includes a base joint where a base of the robotic arm is located and an end joint where a flange of the robotic arm is located. The flange of the robotic arm is fixedly connected with end tools, such as surgical tools (e.g., electrode needles, puncture needles, syringes, ablation needles, etc.).

When the robots are used, it is necessary to precisely position the robots and adjust postures of the robots, such that preoperative planning and/or surgical operations can be performed accurately.

One embodiment of the present disclosure provides a method for positioning a robot. The method may include obtaining a first image and a second image of a target object, the first image being captured using an image capturing apparatus, and the second image being captured using a medical imaging device; determining at least one target region corresponding to at least one target portion of the target object from the first image, wherein the at least one target portion is less affected by physiological motions than other portions; and determining positioning information of the robot based on the at least one target region and the second image.

One embodiment of the present disclosure provides a method for adjusting a posture of an image capturing apparatus. The method may include capturing a target image of a target object using the image capturing apparatus; determining at least one target feature point of the target object from the target image; determining at least one reference feature point corresponding to the at least one target feature point from a reference model of the target object, wherein the reference model corresponds to a target shooting angle; and determining a first target posture of the image capturing apparatus in a base coordinate system based on the at least one target feature point and the at least one reference feature point.

One embodiment of the present disclosure provides a system for positioning a robot. The system may include a storage device configured to store a computer instruction, and a processor connected to the storage device. When executing the computer instruction, the processor may cause the system to perform the following operations: obtaining a first image and a second image of a target object, the first image being captured using an image capturing apparatus, and the second image being captured using a medical imaging device; determining at least one target region corresponding to at least one target portion of the target object from the first image, wherein the at least one target portion is less affected by physiological motions than other portions; and determining positioning information of the robot based on the at least one target region and the second image.

In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the accompanying drawings to be used in the description of the embodiments will be briefly described below. Obviously, the accompanying drawings in the following description are only some examples or embodiments of the present disclosure, and that the present disclosure may be applied to other similar scenarios in accordance with these drawings without creative labor for those of ordinary skill in the art. Unless obviously obtained from the context or the context illustrates otherwise, the same numeral in the drawings refers to the same structure or operation.

It should be understood that “system,” “device,” “unit,” and/or “module” as used herein is a way to distinguish between different components, elements, parts, sections, or assemblies at different levels. However, these words may be replaced by other expressions if other words accomplish the same purpose.

As indicated in the present disclosure and in the claims, unless the context clearly suggests an exception, the words “one,” “a,” “a kind of,” and/or “the” do not refer specifically to the singular but may also include the plural. In general, the terms “including” and “comprising” suggest only the inclusion of clearly identified steps and elements, which do not constitute an exclusive list, and the method or device may also include other steps or elements.

The present disclosure uses flowcharts to illustrate the operations performed by the system according to some embodiments of the present disclosure. It should be understood that the operations described herein are not necessarily executed in a specific order. Instead, they may be executed in reverse order or simultaneously. Additionally, other operations may be added to these processes or certain steps may be removed.

is a schematic diagram illustrating an application scenario of an exemplary robotic control systemaccording to some embodiments of the present disclosure.

The robotic control systemmay be used for positioning a robot and adjusting a posture of the robot. As shown in, in some embodiments, the robotic control systemmay include a server, a medical imaging device, and an image capturing apparatus. The plurality of components of the robotic control systemmay be connected to each other via a network. For example, the serverand the medical imaging devicemay be connected or in a communication through a network. As another example, the serverand the image capturing apparatusmay be connected or in a communication through a network. In some embodiments, connections between the plurality of components of the robotic control systemmay be variable. For example, the medical imaging devicemay be directly connected to the image capturing apparatus.

The servermay be configured to process data or information received from at least one component (e.g., the medical imaging device, the image capturing apparatus) of the robotic control systemor an external data source (e.g., a cloud data center). For example, the servermay obtain a first image captured by the image capturing apparatusand a second image captured by the medical imaging device, and determine the positioning information of the robot based on the first image and the second image. As another example, the servermay capture a target image of a target object using the image capturing apparatus, and determine a first target posture of the image capturing apparatusin a base coordinate system. In some embodiments, the servermay be a single server or a server group. The server group may be centralized or distributed (e.g., the servermay be a distributed system). In some embodiments, the servermay be local or remote. In some embodiments, the servermay be implemented on a cloud platform or provided virtually. Merely by way of example, the cloud platform may include a private cloud, a public cloud, a hybrid cloud, a community cloud, a distributed cloud, an internal cloud, a multi-tiered cloud, or any combination thereof.

In some embodiments, the servermay include one or more components. As shown in, the servermay include one or more (only one shown in) processors, storages, transmission devices, and input/output devices. It is understood by those skilled in the art that the structure shown inis merely for purposes of illustration, and does not limit the structure of the server. For example, the servermay include more or fewer components than those shown in, or may have a configuration different from that shown in.

The processormay process data or information obtained from other devices or components of the system. The processormay execute program instructions based on the data, the information, and/or processing results, to perform one or more functions described in the present disclosure. In some embodiments, the processormay include one or more sub-processing devices (e.g., a single-core processing device or a multi-core multi-processor device). Merely by way of example, the processormay include a microprocessor unit (MPU), a central processing unit (CPU), an application-specific integrated circuit (ASIC), an application-specific instruction processor (ASIP), a graphics processing unit (GPU), a physics processing unit (PPU), a digital signal processor (DSP), a field-programmable gate array (FPGA), a programmable logic device (PLD), a controller, a microcontroller unit, a reduced instruction set computer (RISC), or the like, or any combination thereof. In some embodiments, the processormay be integrated or included in one or more other components (e.g., the medical imaging device, the image capturing apparatus, or other possible components) of the robotic control system.

The storagemay store data, instructions, and/or any other information. For example, the storagemay be configured to store a computer program such as a software program and module for an application, for example, a computer program corresponding to positioning methods and posture adjustment methods in the embodiment. The processormay perform various functional applications and data processing by executing the computer program stored in the storage, thereby implementing the methods described above. The storagemay include high-speed random-access memory and may further include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state storage. In some embodiments, the storagemay also include a remote storage configured relative to the processor. The remote storage may be connected to a terminal via a network. Examples of the network may include the Internet, intranets, local area networks, mobile communication networks, or any combination thereof. In some embodiments, the storagemay be implemented on a cloud platform.

The communication devicemay be configured to implement communication functions. For example, the communication devicemay be configured to receive or transmit data via a network. In some embodiments, the communication devicemay include a network interface controller (NIC) that can communicate with other network devices via a base station to communicate with the Internet. In some embodiments, the communication devicemay be a radio frequency (RF) module for wireless communication with the Internet.

The input/output devicemay be configured to input or output signals, data, or information. In some embodiments, the input/output devicemay facilitate communication between a user and the robotic control system. Exemplary input devices may include a keyboard, a mouse, a touch screen, a microphone, or the like, or any combination thereof. Exemplary output devices may include a display device, a speaker, a printer, a projector, or the like, or any combination thereof. Exemplary display devices may include a liquid crystal display (LCD), a light emitting diode (LED) display, a flat panel display, a curved display, a television, a cathode ray tube (CRT), or the like, or any combination thereof.

In some embodiments, the servermay be disposed at any location (e.g., a room where the robot is located, a room used for placing the server, etc.), as long as the location ensures that the serveris in a normal communication with the medical imaging deviceand the image capturing apparatus.

The medical imaging devicemay be configured to scan the target object in a detection region or a scanning region to obtain imaging data of the target object. In some embodiments, the target object may include a biological and/or a non-biological object. For example, the target object may be an organic and/or inorganic substance with or without life.

In some embodiments, the medical imaging devicemay be a non-invasive imaging device for diagnostic or research purposes. For example, the medical imaging devicemay include a single-model scanner and/or a multi-model scanner. The single-model scanner may include, for example, an ultrasound scanner, an X-ray scanner, a computed tomography (CT) scanner, a magnetic resonance imaging (MRI) scanner, an ultrasound examiner, a positron emission tomography (PET) scanner, an optical coherence tomography (OCT) scanner, an ultrasound (US) scanner, an intravascular ultrasound (IVUS) scanner, a near-infrared spectroscopy (NIRS) scanner, a far-infrared (FIR) scanner, or the like, or any combination thereof. The multi-model scanner may include, for example, an X-ray imaging-magnetic resonance imaging (X-ray-MRI) scanner, a positron emission tomography-X-ray imaging (PET-X-ray) scanner, a single-photon emission computed tomography-magnetic resonance imaging (SPECT-MRI) scanner, a positron emission tomography-computed tomography (PET-CT) scanner, a digital subtraction angiography-magnetic resonance imaging (DSA-MRI) scanner, or the like, or any combination thereof. The scanners are merely for purposes of illustration, and do not limit the scope of the present disclosure. Merely by way of example, the medical imaging devicemay include a CT scanner.

The image capturing apparatusmay be configured to capture image data (e.g., the first image, the target image) of the target object. Exemplary image capturing apparatus may include a camera, an optical sensor, a radar sensor, a structured light scanner, or the like, or any combination thereof. For example, the image capturing apparatusmay include a device capable of capturing optical data of the target object, such as, the camera (e.g., a depth camera, a stereo triangulation camera, etc.), the optical sensor (e.g., a red-green-blue-depth (RGB-D) sensor, etc.), etc. As another example, the image capturing apparatusmay include a device capable of capturing point cloud data of the target object, such as, a laser imaging device (e.g., a time-of-flight (TOF) laser capture device, a point laser capture device, etc.), etc. The point cloud data may include a plurality of data points, wherein each of the plurality of data points may represent a physical point on a body surface of the target object, and one or more feature values (e.g., feature values related to a position and/or a composition) of the physical point may be used to describe the target object. The point cloud data may be used to reconstruct an image of the target object. As still another example, the image capturing apparatusmay include a device capable of obtaining location data and/or depth data of the target object, such as, a structured light scanner, a TOF device, a light triangulation device, a stereo matching device, or the like, or any combination thereof. The location data and/or the depth data obtained by the image capturing apparatusmay be used to reconstruct the image of the target object.

In some embodiments, the image capturing apparatusmay be installed on the robot in a detachable or non-detachable connection manner. For example, the image capturing apparatusmay be detachably disposed to an end terminal of a robotic arm of the robot. In some embodiments, the image capturing apparatusmay be installed at a location outside the robot using a detachable or non-detachable connection manner. For example, the image capturing apparatusmay be disposed at a fixed location in the room where the robot is located.

In some embodiments, a corresponding relationship between the image capturing apparatusand the robot may be determined based on a position of the image capturing apparatus, a position of the robot, and a calibration parameter (e.g., a size, a capturing angle) of the image capturing apparatus. For example, a mapping relationship (i.e., a first transformation relationship) between a first coordinate system corresponding to the image capturing apparatusand a second coordinate system corresponding to the robot may be determined.

It should be noted that the descriptions are provided for the purposes of illustration, and are not intended to limit the scope of the present disclosure. For persons having ordinary skills in the art, various variations and modifications may be conducted under the teaching of the present disclosure. The features, structures, methods, and other features of the exemplary embodiments described in the present disclosure can be combined in various manners to obtain additional and/or alternative exemplary embodiments. For example, the image capturing apparatusmay include a plurality of image capturing apparatus.

In some embodiments, as shown in, the robot control systemmay also include a robot.

The robotmay perform a corresponding operation based on an instruction. For example, the robotmay perform a movement operation (e.g., translation, rotation, etc.) based on a movement instruction. Exemplary robots may include a surgical robot, a rehabilitation robot, a bio-robot, a remote rendering robot, a follow-along robot, a disinfection robot, or the like, or any combination thereof.

Merely by way of example, the robotmay include a multi-degree-of-freedom robotic arm. The multi-degree-of-freedom robotic arm may include a base joint where a base of the robotic arm is located and an end joint where a flange of the robotic arm is located. The flange of the robotic arm is fixedly connected with end tools, such as surgical tools (e.g., electrode needles, puncture needles, syringes, ablation needles, etc.).

is a block diagram illustrating an exemplary processoraccording to some embodiments of the present disclosure. The processormay include an obtaining module, a determination module, and a positioning module.

The obtaining modulemay be configured to obtain a first image and a second image of a target object. The first image may be captured using an image capturing apparatus, and the second image may be captured using a medical imaging device. More descriptions regarding the obtaining the first image and the second image may be found in elsewhere in the present disclosure. See, e.g., operationinand relevant descriptions thereof.

The determination modulemay be configured to determine at least one target region corresponding to at least one target portion of the target object from the first image. The at least one target portion may be less affected by physiological motions than other portions. More descriptions regarding the determination of the at least one target region may be found in elsewhere in the present disclosure. See, e.g., operationinand relevant descriptions thereof.

The positioning modulemay be configured to determine positioning information of a robot based on the at least one target region and the second image. The positioning information refers to location information of the robot or a specific component (e.g., an end terminal of a robotic arm for mounting a surgical instrument) thereof. In some embodiments, the positioning modulemay obtain a first transformation relationship between a first coordinate system corresponding to the image capturing apparatus and a second coordinate system corresponding to the robot. The positioning modulemay further determine a second transformation relationship between the first coordinate system and a third coordinate system corresponding to the medical imaging device based on a registration relationship between the at least one target region and the second image. The positioning modulemay determine the positioning information of the robot based on the first transformation relationship and the second transformation relationship. More descriptions regarding the determination of the positioning information of the robot may be found in elsewhere in the present disclosure. See, e.g., operationinand relevant descriptions thereof.

All or some of the modules of the robot control system described above may be implemented through a software, a hardware, or a combination thereof. These modules may be hardware components embedded in or separated from the processor of a computing device, or may be stored in a storage of a computing device in software form, so as to be retrieved by the processor to perform the operations corresponding to each module.

It should be noted that the descriptions of the robot control system and the modules thereof are provided for convenience of illustration, and are not intended to limit the scope of the present disclosure. It should be understood that those skilled in the art, having an understanding of the principles of the system, may arbitrarily combine the various modules or constitute subsystems connected to other modules without departing from the principles. For example, the obtaining module, the determination module, and the positioning moduledisclosed inmay be different modules in the same system, or may be a single module that performs the functions of the modules mentioned above. As another example, modules of the robot control system may share a storage module, or each module may have an own storage module. Such modifications may not depart from the scope of the present disclosure.

is a flowchart illustrating an exemplary processfor robot positioning according to some embodiments of the present disclosure. In some embodiments, the processmay be implemented by the robot control system. For example, the processmay be stored in a storage device (e.g., the storage) in the form of an instruction set (e.g., an application). In some embodiments, the processor(e.g., the one or more modules as shown in) may execute the instruction set and direct one or more components of the robot control systemto perform the process.

Robots are widely used in the medical field. To accurately control operations of the robots, it is necessary to position the robots. A marker-based positioning technique is commonly used to position robots. Taking neurosurgery as an example, markers need to be implanted in the skull of a patient or attached to the head of the patient, and medical scans are performed on the patient with the markers. Furthermore, corresponding position information of the markers in an image space and a physical space may be determined, thereby positioning the robot based on a corresponding relationship between the image space and the physical space. However, the markers usually cause additional harm to the patient. In addition, once there is a relative displacement between the markers and the head of the patient in the preoperative images, the accuracy of the robot positioning is reduced, thereby affecting preoperative planning or surgical operations. Therefore, it is necessary to provide an effective system and method for robot positioning. In some embodiments, the robot may be positioned by performing the following operations in the process.

In, the processor(e.g., the obtaining module) may obtain a first image and a second image of a target object. The first image may be obtained using an image capturing apparatus, and the second image may be obtained using a medical imaging device.

In some embodiments, the target object may include a biological object and/or a non-biological object. For example, the target object may be an organic and/or inorganic substance with or without life. As another example, the target object may include a specific part, organ, and/or tissue of a patient. Merely by way of example, in a scenario of neurosurgery, the target object may be the head or face of the patient.

The first image refers to an image obtained using the image capturing apparatus (e.g., the image capturing apparatus). The first image may include a three-dimensional (3D) image and/or a two-dimensional (2D) image. In some embodiments, the first image may include a depth image of the target object, which includes distance information from points on the surface of the target object to a reference point.

In some embodiments, the processormay obtain image data of the target object from the image capturing apparatus (e.g., the image capturing apparatus), and determine the first image of the target object based on the image data. For example, when the image capturing apparatus is a camera, the processormay obtain optical data of the target object from the camera, and determine the first image based on the optical data. As another example, when the image capturing apparatus is a laser imaging device, the processormay obtain point cloud data of the target object from the laser imaging device, and determine the first image based on the point cloud data. As still another example, when the image capturing apparatus is a depth camera, the processormay obtain depth data of the target object from the depth camera, and generate a depth image based on the depth data as the first image. In some embodiments, the processormay directly obtain the first image from the image capturing apparatus or a storage device (e.g., the storage).

In some embodiments, before the first image of the target object is captured using the image capturing apparatus, a surgical position of the target object may be determined based on preoperative planning. The target object may be fixed, and a posture of the image capturing apparatus may be adjusted such that the image capturing apparatus captures the target object from a target shooting angle and/or a target shooting height. For example, the posture of the image capturing apparatus may be adjusted such that the face of a patient is completely within a field of view of the image capturing apparatus, and the image capturing apparatus is aligned vertically with the face of the patient for imaging. More descriptions regarding the adjustment of the posture of the image capturing apparatus may be found in elsewhere in the present disclosure. See, e.g.,and relevant descriptions thereof.

The second image refers to a medical image captured using a medical imaging device (e.g., the medical imaging device). Merely by way of example, the medical imaging device may be a CT device. Correspondingly, the processormay obtain CT image data of the target object using the CT device, and reconstruct a CT image based on the CT image data. The processormay further obtain the second image by performing a 3D reconstruction on the CT image.

In some embodiments, the processormay directly obtain the second image of the target object from the medical imaging device (e.g., the medical imaging device). Alternatively, the processormay obtain the second image of the target object from a storage device (e.g., the storage) that stores the second image of the target object.

In some embodiments, the processormay first obtain a first initial image and/or a second initial image, wherein the first initial image is captured using the image capturing apparatus, and the second initial image is captured using the medical imaging device. The processormay generate the first image and/or the second image by processing the first initial image and/or the second initial image. Merely by way of example, the processormay obtain a full-body depth image and a full-body CT image of a patient. The processormay obtain the first image by segmenting a portion corresponding to the face of the patient from the full-body depth image. The processormay obtain a 3D reconstructed image by performing a 3D reconstruction on the full-body CT image, and obtain the second image by segmenting a portion corresponding to the face of the patient from the 3D reconstructed image.

In some embodiments, after the first image and the second image of the target object are obtained, the processormay perform a preprocessing operation (e.g., target region segmentation, dimension adjustment, image resampling, image normalization, etc.) on the first image and the second image. The processormay further perform other operations of the processon the preprocessed first image and the preprocessed second image. For purposes of illustration, the first image and the second image are taken as examples for describing the execution process of the process.

In, the processor(e.g., the determination module) may determine at least one target region corresponding to at least one target portion of the target object from the first image.

In some embodiments, the at least one target portion may be less affected by physiological motions than other portions. The physiological motions may include blinking, respiratory motions, cardiac motions, etc. Merely by way of example, the at least one target portion may be a static facial region. The static facial region refers to a region that is less affected by changes in facial expressions, such as, a region near a facial bone structure. In some embodiments, the processormay capture shape data of human faces under different facial expressions, obtain a region that is less affected by the changes in facial expressions by performing a statistical analysis on the shape data, and determine the region as the static facial region. In some embodiments, the processormay determine the static facial region using physiological structure information. For example, the processormay determine a region close to the facial bone structure as the static facial region. Exemplary static facial regions may include a forehead region, a nasal bridge region, etc.

A target region refers to a region corresponding to a target portion of the target object from the first image. In some embodiments, the processormay determine the at least one target region corresponding to the at least one target portion of the target object from the first image using an image recognition technique (e.g., a 3D image recognition model). Merely by way of example, the processormay input the first image into the 3D image recognition model, and the 3D image recognition model may segment the at least one target region from the first image. The 3D image recognition model may be obtained by training, based on a plurality of training samples, an initial model. Each of the plurality of training samples may include a sample first image of a sample object and a corresponding sample target region, wherein the sample first image is determined as a training input, and the corresponding sample target region is determined as a training label. In some embodiments, the processor(or other processing devices) may iteratively update the initial model based on the plurality of training samples until a specific condition is met (e.g., a loss function is less than a certain threshold, a certain count of training iterations is performed).

In some embodiments, the first image may be a 3D image (e.g., a 3D depth image). The processormay obtain a 2D reference image of the target object captured using the image capturing apparatus. The processormay determine at least one reference region corresponding to the at least one target portion based on the 2D reference image. Further, the processormay determine the at least one target region from the first image based on the at least one reference region. More descriptions regarding the determination of the at least one target region may be found in elsewhere in the present disclosure. See, e.g.,and relevant descriptions thereof.