Legal claims defining the scope of protection. Each claim is shown in both the original legal language and a plain English translation.
1. A method for moving a robotic manipulator in free space to avoid objects in a scene, based upon a prior 3D scene model corresponding to the scene in a prior state, the prior 3D scene model comprised of 3D object models corresponding to objects of the scene in the prior state, the method comprising the steps of: (a) receiving, by a processor, the current image of the scene, wherein the image of the scene is a range image; (b) initializing, by the processor, the 3D scene model to the prior 3D scene model, wherein each object p of the 3D scene model has a signed distance function; and (c) modifying the 3D scene model to be consistent with the image, by: (i) comparing, by the processor, data of the image with 3D object models of the 3D scene model resulting in associated data and unassociated data by, for each 3D object model p, finding locations in the image such that the value of the signed distance function of the 3D object model p applied to the location is less than or equal to a specified tolerance, and the value of the distance function for all other 3D object models q applied to the location is not less than the value of p's distance function, the data points of these locations being associated data of the 3D object model p, and the data points of any locations not associated with any 3D object model p of the 3D scene model being unassociated data; (ii) using, by the processor, the unassociated data to compute new 3D object models and adding the new 3D object models to the 3D scene model; and (iii) using, by the processor, the associated data to detect 3D object models in the prior 3D scene model that are absent and removing the absent 3D object models from the 3D scene model; (d) computing, by the processor, a trajectory that avoids the 3D object models of the 3D scene model; (e) directing, by the processor, the robotic manipulator to follow the computed trajectory in the scene; and (f) following, by the robotic manipulator, the trajectory directed by the processor.
Robotic manipulators operating in dynamic environments require real-time obstacle avoidance to navigate safely. Existing methods often rely on static scene models or limited sensor data, leading to collisions or inefficient paths. This invention addresses these limitations by dynamically updating a 3D scene model using range images to ensure accurate obstacle detection and path planning. The method begins by receiving a current range image of the scene and initializing a 3D scene model based on a prior state. Each object in the model is represented by a signed distance function. The system compares the range image data with the 3D object models to identify associated and unassociated data points. Associated data points correspond to locations where a specific object's distance function meets a tolerance threshold, while unassociated points indicate new or missing objects. The system then updates the 3D scene model by adding new object models for unassociated data and removing models for absent objects. Using the updated scene model, the system computes a collision-free trajectory for the robotic manipulator. The manipulator follows this trajectory, ensuring safe navigation. This approach improves obstacle avoidance by dynamically adapting to environmental changes, reducing the risk of collisions and enhancing path efficiency. The method is particularly useful in environments where objects may move or appear unpredictably.
2. A method for moving a robotic manipulator in free space to avoid objects in a scene, based upon a prior 3D scene model corresponding to the scene in a prior state, the prior 3D scene model comprised of 3D object models corresponding to objects of the scene in the prior state, the method comprising the steps of: (a) receiving, by a processor, the current image of the scene; (b) initializing, by the processor, the 3D scene model to the prior 3D scene model; and (c) modifying the 3D scene model to be consistent with the image, by: (i) comparing, by the processor, data of the image with 3D object models of the 3D scene model, resulting in associated data and unassociated data by: (A) rendering each 3D object model p of the 3D scene model, resulting in a rendered image for p; and (B) finding locations in the image such that a first difference between the value of the image data at a location and the value of the rendered image for 3D object model p at the location is less than or equal to a tolerance and for all other 3D object models q, a second difference between the value of the image data at the location and the value of the rendered image for 3D object model q at the location is not less than the first difference, the data points of these locations being associated data of the 3D object model p, and the data points of any locations not associated with any 3D object model of the 3D scene model being unassociated data; (ii) using, by the processor, the unassociated data to compute new 3D object models and adding the new 3D object models to the 3D scene model; and (iii) using, by the processor, the associated data to detect 3D object models in the prior 3D scene model that are absent and removing the absent 3D object models from the 3D scene model; (d) computing, by the processor, a trajectory that avoids the 3D object models of the 3D scene model; (e) directing, by the processor, the robotic manipulator to follow the computed trajectory in the scene; and (f) following, by the robotic manipulator, the trajectory directed by the processor.
This invention relates to robotic navigation in dynamic environments, specifically a method for moving a robotic manipulator to avoid obstacles using an updated 3D scene model. The system addresses the challenge of navigating in scenes where objects may have moved or changed since a prior 3D model was created. The method begins by receiving a current image of the scene and initializing a 3D scene model based on a previously generated 3D model of the scene. The system then updates this model by comparing the current image with rendered views of 3D object models from the prior model. For each object model, the system renders an image and identifies image locations where the difference between the rendered and current image data is minimized, associating those locations with the object. Unassociated data points, which do not match any existing object model, are used to generate new 3D object models representing newly detected obstacles. The system also removes object models from the prior scene that no longer appear in the current image. After updating the 3D scene model, the system computes a collision-free trajectory for the robotic manipulator and directs the manipulator to follow this path. This approach ensures the robot can navigate safely in environments where objects may have moved or changed since the last model update.
3. A method for moving a robotic manipulator in free space to avoid objects in a scene, based upon a prior 3D scene model corresponding to the scene in a prior state, the prior 3D scene model comprised of 3D object models corresponding to objects of the scene in the prior state, the method comprising the steps of: (a) receiving, by a processor, the current image of the scene; (b) initializing, by the processor, the 3D scene model to the prior 3D scene model; and (c) modifying the 3D scene model to be consistent with the image, by: (i) comparing, by the processor, data of the image with 3D object models of the 3D scene model, resulting in associated data and unassociated data; (ii) using, by the processor, the unassociated data to compute new 3D object models and adding the new 3D object models to the 3D scene model; and (iii) using, by the processor, the associated data to detect 3D object models in the prior 3D scene model that are absent by: (A) rendering the 3D scene model, resulting in a synthetic rendering; (B) using the synthetic rendering to compute synthetic data associated with 3D object models; (C) comparing the associated data with the synthetic data associated with objects to find 3D object models in the 3D scene model that do not appear in the image; and (D) determining that a given 3D object model q in the 3D scene model that does not appear in the image is absent from the scene by finding that there is no set of other 3D object models that may be computed from the image that occludes the 3D object model q; and removing the absent 3D object models from the 3D scene model; (d) computing, by the processor, a trajectory that avoids the 3D object models of the 3D scene model; (e) directing, by the processor, the robotic manipulator to follow the computed trajectory in the scene; and (f) following, by the robotic manipulator, the trajectory directed by the processor.
This invention relates to robotic navigation in dynamic environments, specifically a method for updating a 3D scene model to accurately reflect current conditions and enabling a robotic manipulator to avoid obstacles. The system addresses the challenge of navigating in scenes where objects may have moved or been removed since the last known 3D model was created. The method begins by receiving a current image of the scene and initializing the 3D scene model using a prior 3D model, which contains 3D representations of objects from a previous state. The system then updates this model by comparing image data with the existing 3D object models, identifying both associated (matching) and unassociated (new or changed) data. New 3D object models are generated from unassociated data and added to the scene model. For associated data, the system renders the current 3D scene model to produce a synthetic image, then compares this with the actual image to detect missing objects. If an object in the prior model does not appear in the current image and cannot be occluded by other objects, it is removed from the model. The updated 3D scene model is then used to compute a collision-free trajectory for the robotic manipulator, which follows this path to navigate the scene safely. This approach ensures real-time adaptation to environmental changes, improving robotic navigation accuracy in dynamic settings.
4. A method for moving a robotic manipulator in free space to avoid objects in a scene, based upon a prior 3D scene model corresponding to the scene in a prior state, the prior 3D scene model comprised of 3D object models corresponding to objects of the scene in the prior state, the method comprising the steps of: (a) receiving, by a processor, the current image of the scene; (b) initializing, by the processor, the 3D scene model to the prior 3D scene model; and (c) modifying the 3D scene model to be consistent with the image, by: (i) comparing, by the processor, data of the image with 3D object models of the 3D scene model, resulting in associated data and unassociated data, by: (A) processing the image to detect image features; (B) processing each 3D object model in the 3D scene model to detect model features that would be visible; (C) associating the image features to visible model features of a given 3D object model p when they match at the same or nearby locations and have no better match with visible model features from any other 3D object model q of the 3D scene model; (D) taking the image features associated with model features of 3D object model p as data associated with 3D object model p; (ii) using, by the processor, the unassociated data to compute new 3D object models and adding the new 3D object models to the 3D scene model; and (iii) using, by the processor, the associated data to detect 3D object models in the prior 3D scene model that are absent and removing the absent 3D object models from the 3D scene model; (d) computing, by the processor, a trajectory that avoids the 3D object models of the 3D scene model; (e) directing, by the processor, the robotic manipulator to follow the computed trajectory in the scene; and (f) following, by the robotic manipulator, the trajectory directed by the processor.
Robotic manipulators operate in dynamic environments where objects may move or change, requiring real-time navigation to avoid collisions. This invention addresses the challenge of updating a 3D scene model to reflect current conditions and planning a collision-free trajectory for a robotic manipulator. The method begins by receiving a current image of the scene and initializing a 3D scene model based on a prior state, which includes 3D object models of known objects. The system then modifies the model to align with the current image by comparing detected image features with visible features of the 3D object models. Features are associated when they match in location and have no better match with other objects, while unmatched features lead to the creation of new 3D object models. Absent objects from the prior model are removed. Using the updated scene model, the system computes a trajectory that avoids all detected objects and directs the robotic manipulator to follow it. This approach ensures accurate scene representation and safe navigation in dynamic environments.
5. A method for grasping an object in a scene by a robotic manipulator, based upon and a prior 3D scene model corresponding to the scene in a prior state, the prior 3D scene model comprised of 3D object models corresponding to objects of the scene in the prior state, the method comprising the steps of: (a) receiving, by a processor, the current image of the scene, wherein the image of the scene is a range image; (b) initializing, by the processor, the 3D scene model to the prior 3D scene model, wherein each object p of the 3D scene model has a signed distance function; and (c) modifying the 3D scene model to be consistent with the image, by: (i) comparing, by the processor, data of the image with 3D object models of the 3D scene model, resulting in associated data and unassociated data by, for each 3D object model p, finding locations in the image such that the value of the signed distance function of the 3D object model p applied to the location is less than or equal to a specified tolerance, and the value of the distance function for all other 3D object models q applied to the location is not less than the value of p's distance function, the data points of these locations being associated data of the 3D object model p, and the data points of any locations not associated with any 3D object model p of the 3D scene model being unassociated data; (ii) using, by the processor, the unassociated data to compute new 3D object models and adding the new 3D object models to the 3D scene model; and (iii) using, by the processor, the associated data to detect 3D object models in the prior 3D scene model that are absent and removing the absent 3D object models from the 3D scene model; (d) computing, by the processor, a grasp of one of the 3D object models of the 3D scene model; (e) directing, by the processor, the robotic manipulator to grasp the corresponding object in the scene using the computed grasp; and (f) grasping, by the robotic manipulator, the object directed by the processor using the grasp directed by the processor.
This invention relates to robotic manipulation, specifically methods for grasping objects in dynamic environments using updated 3D scene models. The problem addressed is the challenge of accurately grasping objects when the scene has changed since the last known 3D model, such as when objects have moved or new objects have appeared. The method updates a prior 3D scene model to reflect the current state of the scene, enabling precise grasping by a robotic manipulator. The process begins by receiving a current range image of the scene. The prior 3D scene model, which includes 3D object models of objects in their previous state, is initialized. Each object in the model has a signed distance function. The system then modifies the 3D scene model to match the current image by associating image data points with the closest 3D object models based on their signed distance functions. Data points not associated with any existing 3D object model are used to generate new 3D object models, which are added to the scene model. Meanwhile, 3D object models that no longer correspond to any image data are removed. After updating the scene model, the system computes a grasp for one of the objects, directs the robotic manipulator to execute the grasp, and the manipulator performs the grasping action. This approach ensures the robotic system operates with an accurate, up-to-date representation of the scene, improving grasping reliability in dynamic environments.
6. A method for grasping an object in a scene by a robotic manipulator, based upon and a prior 3D scene model corresponding to the scene in a prior state, the prior 3D scene model comprised of 3D object models corresponding to objects of the scene in the prior state, the method comprising the steps of: (a) receiving, by a processor, the current image of the scene; (b) initializing, by the processor, the 3D scene model to the prior 3D scene model; and (c) modifying the 3D scene model to be consistent with the image, by: (i) comparing, by the processor, data of the image with 3D object models of the 3D scene model, resulting in associated data and unassociated data by: (A) rendering each 3D object model p of the 3D scene model, resulting in a rendered image for p; and (B) finding locations in the image such that a first difference between the value of the image data at a location and the value of the rendered image for 3D object model p at the location is less than or equal to a tolerance and for all other 3D object models q, a second difference between the value of the image data at the location and the value of the rendered image for 3D object model q at the location is not less than the first difference, the data points of these locations being associated data of the 3D object model p, and the data points of any locations not associated with any 3D object model of the 3D scene model being unassociated data; (ii) using, by the processor, the unassociated data to compute new 3D object models and adding the new 3D object models to the 3D scene model; and (iii) using, by the processor, the associated data to detect 3D object models in the prior 3D scene model that are absent and removing the absent 3D object models from the 3D scene model; (d) computing, by the processor, a grasp of one of the 3D object models of the 3D scene model; (e) directing, by the processor, the robotic manipulator to grasp the corresponding object in the scene using the computed grasp; and (f) grasping, by the robotic manipulator, the object directed by the processor using the grasp directed by the processor.
This invention relates to robotic manipulation systems that use 3D scene modeling to grasp objects in dynamic environments. The problem addressed is the challenge of accurately updating a 3D scene model in real-time to reflect changes in the environment, such as the addition or removal of objects, to enable precise robotic grasping. The method begins by receiving a current image of the scene and initializing a 3D scene model based on a prior 3D model of the scene. The prior model includes 3D representations of objects from a previous state. The system then updates this model by comparing the current image with rendered versions of the 3D object models. For each object model, the system renders an image and identifies image locations where the difference between the rendered image and the current image is within a tolerance, while ensuring no other object model matches better. These matched locations are associated with the object, while unmatched data indicates new objects. The system then adds new 3D object models for unassociated data and removes models that no longer match the current scene. After updating the 3D scene model, the system computes a grasp for one of the objects and directs the robotic manipulator to execute the grasp. This approach ensures the robot operates with an accurate, up-to-date scene model, improving grasping reliability in dynamic environments.
7. A method for grasping an object in a scene by a robotic manipulator, based upon and a prior 3D scene model corresponding to the scene in a prior state, the prior 3D scene model comprised of 3D object models corresponding to objects of the scene in the prior state, the method comprising the steps of: (a) receiving, by a processor, the current image of the scene; (b) initializing, by the processor, the 3D scene model to the prior 3D scene model; and (c) modifying the 3D scene model to be consistent with the image, by: (i) comparing, by the processor, data of the image with 3D object models of the 3D scene model, resulting in associated data and unassociated data; (ii) using, by the processor, the unassociated data to compute new 3D object models and adding the new 3D object models to the 3D scene model; and (iii) using, by the processor, the associated data to detect 3D object models in the prior 3D scene model that are absent by: (A) rendering the 3D scene model, resulting in a synthetic rendering; (B) using the synthetic rendering to compute synthetic data associated with 3D object models; (C) comparing the associated data with the synthetic data associated with objects to find 3D object models in the 3D scene model that do not appear in the image; and (D) determining that a given 3D object model q in the 3D scene model that does not appear in the image is absent from the scene by finding that there is no set of other 3D object models that may be computed from the image that occludes the 3D object model q; and removing the absent 3D object models from the 3D scene model; (d) computing, by the processor, a grasp of one of the 3D object models of the 3D scene model; (e) directing, by the processor, the robotic manipulator to grasp the corresponding object in the scene using the computed grasp; and (f) grasping, by the robotic manipulator, the object directed by the processor using the grasp directed by the processor.
Robotic manipulators often struggle to accurately grasp objects in dynamic environments due to changes in scene composition between prior modeling and current execution. This invention addresses the problem by updating a 3D scene model in real-time using a current image of the scene, ensuring the model reflects the current state before grasping. The method initializes the 3D scene model with a prior version, which includes 3D object models of objects in the scene from a previous state. The system then processes the current image by comparing it with the 3D object models to identify associated data (matching objects) and unassociated data (new or changed objects). Unassociated data is used to compute and add new 3D object models to the scene model. For associated data, the system renders the 3D scene model to generate synthetic data, comparing this with the actual image data to detect missing objects. If an object in the prior model does not appear in the image and cannot be occluded by other objects, it is removed from the updated model. The system then computes a grasp for one of the remaining 3D object models and directs the robotic manipulator to execute the grasp, ensuring accurate interaction with the current scene state. This approach improves robotic grasping reliability in dynamic environments by dynamically updating scene models before execution.
8. A method for grasping an object in a scene by a robotic manipulator, based upon and a prior 3D scene model corresponding to the scene in a prior state, the prior 3D scene model comprised of 3D object models corresponding to objects of the scene in the prior state, the method comprising the steps of: (a) receiving, by a processor, the current image of the scene; (b) initializing, by the processor, the 3D scene model to the prior 3D scene model; and (c) modifying the 3D scene model to be consistent with the image, by: (i) comparing, by the processor, data of the image with 3D object models of the 3D scene model, resulting in associated data and unassociated data, by: (A) processing the image to detect image features; (B) processing each 3D object model in the 3D scene model to detect model features that would be visible; (C) associating the image features to visible model features of a given 3D object model p when they match at the same or nearby locations and have no better match with visible model features from any other 3D object model q of the 3D scene model; (D) taking the image features associated with model features of 3D object model p as data associated with 3D object model p; (ii) using, by the processor, the unassociated data to compute new 3D object models and adding the new 3D object models to the 3D scene model; and (iii) using, by the processor, the associated data to detect 3D object models in the prior 3D scene model that are absent and removing the absent 3D object models from the 3D scene model; (d) computing, by the processor, a grasp of one of the 3D object models of the 3D scene model; (e) directing, by the processor, the robotic manipulator to grasp the corresponding object in the scene using the computed grasp; and (f) grasping, by the robotic manipulator, the object directed by the processor using the grasp directed by the processor.
Robotic manipulation systems often struggle with accurately grasping objects in dynamic environments due to changes in object positions or the presence of previously unknown objects. This invention addresses the problem by updating a 3D scene model in real-time using a current image of the scene, allowing a robotic manipulator to grasp objects reliably. The method begins by receiving a current image of the scene and initializing a 3D scene model based on a prior 3D model of the scene. The prior model includes 3D object models representing objects in their previous state. The system then modifies the 3D scene model to align with the current image by comparing image features with visible features of the 3D object models. Image features are matched to visible model features of a given object if they align spatially and have no better match with features from other objects. Associated data (matched features) are linked to their corresponding 3D object models, while unassociated data (unmatched features) are used to generate new 3D object models, which are added to the scene model. Additionally, the system identifies and removes any 3D object models from the prior model that are no longer present in the current scene. Once the updated 3D scene model is generated, the system computes a grasp for one of the objects and directs the robotic manipulator to execute the grasp. The manipulator then grasps the object as directed. This approach ensures the robotic system operates with an accurate, up-to-date representation of the scene, improving grasping success in dynamic environments.
9. A system for moving a robotic manipulator in free space to avoid objects in a scene, based upon a prior 3D scene model corresponding to the scene in a prior state, the prior 3D scene model comprised of 3D object models corresponding to objects of the scene in the prior state, the system comprising: (a) a processor configured to: (i) receive the current image of the scene, wherein the image of the scene is a range image; (ii) initialize the 3D scene model to the prior 3D scene model wherein each object p of the 3D scene model has a signed distance function; and (iii) modify the 3D scene model to be consistent with the image, by: (A) comparing data of the image with the 3D object models of the 3D scene model, resulting in associated data and unassociated data by, for each 3D object model p, finding locations in the image such that the value of the signed distance function of the 3D object model p applied to the location is less than or equal to a specified tolerance, and the value of the distance function for all other 3D object models q applied to the location is not less than the value of p's distance function, the data points of these locations being associated data of the 3D object model p, and the data points of any locations not associated with any 3D object model p of the 3D scene model being unassociated data; (B) using the unassociated data to compute new 3D object models and adding the new 3D object models to the 3D scene model; (C) using the associated data to detect 3D object models in the prior 3D scene model that are absent and removing the absent 3D object models from the 3D scene model; (iv) compute a trajectory that avoids the 3D object models of the 3D scene model; and (v) direct the robotic manipulator to follow the computed trajectory in the scene; and (b) the robotic manipulator configured to follow a trajectory as directed by the processor.
This invention relates to robotic navigation systems that use 3D scene modeling to avoid obstacles in dynamic environments. The system addresses the challenge of updating a pre-existing 3D scene model in real-time to account for changes in the environment, ensuring safe and efficient movement of a robotic manipulator. The system processes a current range image of the scene and initializes a 3D scene model based on a prior state, where each object in the model is represented by a signed distance function. The processor compares the image data with the 3D object models to identify associated and unassociated data points. Associated data points are those where a specific object's signed distance function meets a tolerance threshold, while unassociated data points belong to no existing model. The system then updates the 3D scene model by adding new 3D object models derived from unassociated data and removing models that no longer correspond to detected objects. Using the updated model, the processor computes a collision-free trajectory for the robotic manipulator and directs its movement accordingly. The manipulator follows the computed path, ensuring safe navigation in dynamic environments. This approach improves robotic autonomy by dynamically adapting to environmental changes.
10. A system for moving a robotic manipulator in free space to avoid objects in a scene, based upon a prior 3D scene model corresponding to the scene in a prior state, the prior 3D scene model comprised of 3D object models corresponding to objects of the scene in the prior state, the system comprising: (a) a processor configured to: (i) receive the current image of the scene; (ii) initialize the 3D scene model to the prior 3D scene model; and (iii) modify the 3D scene model to be consistent with the image, by: (A) comparing data of the image with the 3D object models of the 3D scene model, resulting in associated data and unassociated data by: (1) rendering each 3D object model p of the 3D scene model, resulting in a rendered image for p; and (2) finding locations in the image such that a first difference between the value of the image data at a location and the value of the rendered image for 3D object model p at the location is less than or equal to a tolerance and for all other 3D object models q, a second difference between the value of the image data at the location and the value of the rendered image for 3D object model q at the location is not less than the first difference, the data points of these locations being associated data of the 3D object model p, and the data points of any locations not associated with any 3D object model of the 3D scene model being unassociated data; (B) using the unassociated data to compute new 3D object models and adding the new 3D object models to the 3D scene model; (C) using the associated data to detect 3D object models in the prior 3D scene model that are absent and removing the absent 3D object models from the 3D scene model; (iv) compute a trajectory that avoids the 3D object models of the 3D scene model; and (v) direct the robotic manipulator to follow the computed trajectory in the scene; and (b) the robotic manipulator configured to follow a trajectory as directed by the processor.
The system enables a robotic manipulator to navigate in free space while avoiding obstacles by updating a 3D scene model in real-time using current image data. The technology addresses the challenge of dynamic environments where objects may have moved or appeared since the last known scene model was created. The system initializes a 3D scene model based on a prior state of the environment, which includes 3D representations of objects. A processor receives a current image of the scene and updates the 3D scene model to reflect changes. This involves comparing the image data with rendered views of the 3D object models to identify associated data (matching locations) and unassociated data (unmatched locations). The processor then modifies the 3D scene model by adding new 3D object models for unassociated data and removing models that no longer match the image. The updated scene model is used to compute a collision-free trajectory for the robotic manipulator, which follows the trajectory to navigate the environment safely. The system ensures real-time adaptation to environmental changes, improving robotic navigation in dynamic settings.
11. A system for moving a robotic manipulator in free space to avoid objects in a scene, based upon a prior 3D scene model corresponding to the scene in a prior state, the prior 3D scene model comprised of 3D object models corresponding to objects of the scene in the prior state, the system comprising: (a) a processor configured to: (i) receive the current image of the scene; (ii) initialize the 3D scene model to the prior 3D scene model; and (iii) modify the 3D scene model to be consistent with the image, by: (A) comparing data of the image with the 3D object models of the 3D scene model, resulting in associated data and unassociated data; (B) using the unassociated data to compute new 3D object models and adding the new 3D object models to the 3D scene model; (C) using the associated data to detect 3D object models in the prior 3D scene model that are absent by: (1) rendering the 3D scene model, resulting in a synthetic rendering; (2) using the synthetic rendering to compute synthetic data associated with 3D object models; (3) comparing the associated data with the synthetic data associated with objects to find 3D object models in the 3D scene model that do not appear in the image; and (4) determining that a given 3D object model q in the 3D scene model that does not appear in the image is absent from the scene by finding that there is no set of other 3D object models that may be computed from the image that occludes the 3D object model q; and removing the absent 3D object models from the 3D scene model; (iv) compute a trajectory that avoids the 3D object models of the 3D scene model; and (v) direct the robotic manipulator to follow the computed trajectory in the scene; and (b) the robotic manipulator configured to follow a trajectory as directed by the processor.
This invention relates to robotic navigation systems that use 3D scene modeling to avoid obstacles in dynamic environments. The system addresses the challenge of updating a pre-existing 3D scene model to reflect real-time changes in the environment, ensuring accurate obstacle avoidance for a robotic manipulator. The system begins with a prior 3D scene model containing 3D object models of objects in the scene. A processor receives a current image of the scene and initializes the 3D scene model to this prior state. The model is then updated by comparing image data with the 3D object models, separating associated and unassociated data. Unassociated data is used to compute new 3D object models, which are added to the scene model. Associated data is used to detect missing objects by rendering the 3D scene model and comparing synthetic data with the image data. If an object in the prior model does not appear in the image and cannot be occluded by other objects, it is removed from the model. The system then computes a collision-free trajectory for the robotic manipulator, which follows the trajectory to navigate the scene. This approach ensures real-time adaptation to environmental changes, improving robotic manipulator safety and efficiency in dynamic environments.
12. A system for moving a robotic manipulator in free space to avoid objects in a scene, based upon a prior 3D scene model corresponding to the scene in a prior state, the prior 3D scene model comprised of 3D object models corresponding to objects of the scene in the prior state, the system comprising: (a) a processor configured to: (i) receive the current image of the scene; (ii) initialize the 3D scene model to the prior 3D scene model; and (iii) modify the 3D scene model to be consistent with the image, by: (A) comparing data of the image with the 3D object models of the 3D scene model, resulting in associated data and unassociated data, by: (1) processing the image to detect image features; (2) processing each 3D object model in the 3D scene model to detect model features that would be visible; (3) associating the image features to visible model features of a given 3D object model p when they match at the same or nearby locations and have no better match with visible model features from any other 3D object model q of the 3D scene model; (4) taking the image features associated with model features of 3D object model p as data associated with 3D object model p; (B) using the unassociated data to compute new 3D object models and adding the new 3D object models to the 3D scene model; (C) using the associated data to detect 3D object models in the prior 3D scene model that are absent and removing the absent 3D object models from the 3D scene model; (iv) compute a trajectory that avoids the 3D object models of the 3D scene model; and (v) direct the robotic manipulator to follow the computed trajectory in the scene; and (b) the robotic manipulator configured to follow a trajectory as directed by the processor.
This invention relates to robotic navigation systems that use 3D scene models to avoid obstacles in dynamic environments. The system addresses the challenge of updating a pre-existing 3D scene model to reflect real-time changes in the environment, ensuring a robotic manipulator can navigate safely without colliding with objects. The system processes a current image of the scene and initializes a 3D scene model based on a prior 3D scene model, which includes 3D object models representing objects from a previous state. The system then modifies the 3D scene model to align with the current image by comparing image features with visible features of the 3D object models. Image features are matched to model features of a given 3D object model if they align spatially and have no better match with features from other models. Associated data (matched features) are used to refine the 3D object models, while unassociated data (unmatched features) are used to detect and add new objects to the scene model. The system also identifies and removes 3D object models that are no longer present in the scene. After updating the 3D scene model, the system computes a collision-free trajectory for the robotic manipulator and directs it to follow this path. The robotic manipulator executes the trajectory as instructed by the processor, ensuring safe navigation in dynamic environments. This approach improves robotic autonomy by dynamically adapting to environmental changes.
13. A system for grasping an object in a scene by a robotic manipulator, based upon a prior 3D scene model corresponding to the scene in a prior state, the prior 3D scene model comprised of 3D object models corresponding to objects of the scene in the prior state, the system comprising: (a) a processor configured to: (i) receive the current image of the scene, wherein the image of the scene is a range image; (ii) initialize the 3D scene model to the prior 3D scene model, wherein each object p of the 3D scene model has a signed distance function; and (iii) modify the 3D scene model to be consistent with the image, by: (A) comparing data of the image with the 3D object models of the 3D scene model, resulting in associated data and unassociated data by, for each 3D object model p, finding locations in the image such that the value of the signed distance function of the 3D object model p applied to the location is less than or equal to a specified tolerance, and the value of the distance function for all other 3D object models q applied to the location is not less than the value of p's distance function, the data points of these locations being associated data of the 3D object model p, and the data points of any locations not associated with any 3D object model p of the 3D scene model being unassociated data; (B) using the unassociated data to compute new 3D object models and adding the new 3D object models to the 3D scene model; (C) using the associated data to detect 3D object models in the prior 3D scene model that are absent and removing the absent 3D object models from the 3D scene model; (iv) compute a grasp of one of the 3D object models of the 3D scene model; and (v) direct the robotic manipulator to grasp the corresponding object in the scene using the computed grasp; and (b) the robotic manipulator configured to grasp an object as directed by the processor.
The system enables a robotic manipulator to grasp objects in a scene by updating a prior 3D scene model with current sensor data. The prior 3D scene model includes 3D object models representing objects in the scene from a previous state. The system receives a current range image of the scene and initializes the 3D scene model using the prior model, where each object in the model is represented by a signed distance function. The system then modifies the 3D scene model to align with the current image by associating image data points with the closest matching 3D object models. This involves comparing the image data with the 3D object models to determine which data points correspond to each object, based on the signed distance function. Unassociated data points, which do not match any existing 3D object model, are used to generate new 3D object models, which are added to the scene model. Additionally, the system identifies and removes any 3D object models from the prior scene model that are no longer present in the current image. After updating the 3D scene model, the system computes a grasp for one of the objects and directs the robotic manipulator to execute the grasp. The robotic manipulator then physically grasps the object as instructed. This approach ensures accurate object recognition and manipulation by dynamically updating the scene model with real-time sensor data.
14. A system for grasping an object in a scene by a robotic manipulator, based upon a prior 3D scene model corresponding to the scene in a prior state, the prior 3D scene model comprised of 3D object models corresponding to objects of the scene in the prior state, the system comprising: (a) a processor configured to: (i) receive the current image of the scene; (ii) initialize the 3D scene model to the prior 3D scene model; and (iii) modify the 3D scene model to be consistent with the image, by: (A) comparing data of the image with the 3D object models of the 3D scene model, resulting in associated data and unassociated data by: (1) rendering each 3D object model p of the 3D scene model, resulting in a rendered image for p; and (2) finding locations in the image such that a first difference between the value of the image data at a location and the value of the rendered image for 3D object model p at the location is less than or equal to a tolerance and for all other 3D object models q, a second difference between the value of the image data at the location and the value of the rendered image for 3D object model q at the location is not less than the first difference, the data points of these locations being associated data of the 3D object model p, and the data points of any locations not associated with any 3D object model of the 3D scene model being unassociated data; (B) using the unassociated data to compute new 3D object models and adding the new 3D object models to the 3D scene model; (C) using the associated data to detect 3D object models in the prior 3D scene model that are absent and removing the absent 3D object models from the 3D scene model; (iv) compute a grasp of one of the 3D object models of the 3D scene model; and (v) direct the robotic manipulator to grasp the corresponding object in the scene using the computed grasp; and (b) the robotic manipulator configured to grasp an object as directed by the processor.
A system for robotic object grasping updates a 3D scene model using a current image of the scene to accurately identify and grasp objects. The system starts with a prior 3D scene model containing 3D object models of objects in the scene from a previous state. A processor receives a current image of the scene and initializes the 3D scene model to the prior state. The processor then modifies the 3D scene model to align with the current image by comparing image data with the 3D object models. Each 3D object model is rendered to produce a rendered image, and the system identifies locations in the current image where the difference between the image data and the rendered image for a specific 3D object model is minimized compared to other models. These locations are associated with the corresponding 3D object model, while unassociated data points are those not linked to any model. The system uses unassociated data to compute new 3D object models, adding them to the scene model, and removes any 3D object models from the prior model that are no longer present in the current image. The processor then computes a grasp for one of the 3D object models and directs the robotic manipulator to grasp the corresponding object in the scene. The robotic manipulator executes the grasp as directed. This approach ensures accurate object recognition and grasping by dynamically updating the scene model based on real-time visual input.
15. A system for grasping an object in a scene by a robotic manipulator, based upon a prior 3D scene model corresponding to the scene in a prior state, the prior 3D scene model comprised of 3D object models corresponding to objects of the scene in the prior state, the system comprising: (a) a processor configured to: (i) receive the current image of the scene; (ii) initialize the 3D scene model to the prior 3D scene model; and (iii) modify the 3D scene model to be consistent with the image, by: (A) comparing data of the image with the 3D object models of the 3D scene model, resulting in associated data and unassociated data; (B) using the unassociated data to compute new 3D object models and adding the new 3D object models to the 3D scene model; (C) using the associated data to detect 3D object models in the prior 3D scene model that are absent and removing the absent 3D object models from the 3D scene model by: (1) rendering the 3D scene model, resulting in a synthetic rendering; (2) using the synthetic rendering to compute synthetic data associated with 3D object models; (3) comparing the associated data with the synthetic data associated with objects to find 3D object models in the 3D scene model that do not appear in the image; and (4) determining that a given 3D object model q in the 3D scene model that does not appear in the image is absent from the scene by finding that there is no set of other 3D object models that may be computed from the image that occludes the 3D object model q; (iv) compute a grasp of one of the 3D object models of the 3D scene model; and (v) direct the robotic manipulator to grasp the corresponding object in the scene using the computed grasp; and (b) the robotic manipulator configured to grasp an object as directed by the processor.
The system enables a robotic manipulator to grasp objects in a scene by updating a prior 3D scene model based on a current image of the scene. The prior 3D scene model includes 3D object models representing objects in the scene at an earlier time. The system initializes the 3D scene model with the prior model and modifies it to match the current image. It compares image data with the 3D object models, identifying associated and unassociated data. Unassociated data is used to compute and add new 3D object models to the scene model. Associated data is used to detect and remove 3D object models that are no longer present in the scene. This involves rendering the 3D scene model to generate synthetic data, comparing it with the associated data to identify missing objects, and confirming their absence by ensuring no other objects in the image could occlude them. The system then computes a grasp for one of the remaining 3D object models and directs the robotic manipulator to execute the grasp. The manipulator performs the grasping action as instructed. This approach ensures the robotic system operates with an accurate, up-to-date 3D scene model for reliable object grasping.
16. A system for grasping an object in a scene by a robotic manipulator, based upon a prior 3D scene model corresponding to the scene in a prior state, the prior 3D scene model comprised of 3D object models corresponding to objects of the scene in the prior state, the system comprising: (a) a processor configured to: (i) receive the current image of the scene; (ii) initialize the 3D scene model to the prior 3D scene model; and (iii) modify the 3D scene model to be consistent with the image, by: (A) comparing data of the image with the 3D object models of the 3D scene model, resulting in associated data and unassociated data by: (1) processing the image to detect image features; (2) processing each 3D object model in the 3D scene model to detect model features that would be visible; (3) associating the image features to visible model features of a given 3D object model p when they match at the same or nearby locations and have no better match with visible model features from any other 3D object model q of the 3D scene model; (4) taking the image features associated with model features of 3D object model p as data associated with 3D object model p; (B) using the unassociated data to compute new 3D object models and adding the new 3D object models to the 3D scene model; (C) using the associated data to detect 3D object models in the prior 3D scene model that are absent and removing the absent 3D object models from the 3D scene model; (iv) compute a grasp of one of the 3D object models of the 3D scene model; and (v) direct the robotic manipulator to grasp the corresponding object in the scene using the computed grasp; and (b) the robotic manipulator configured to grasp an object as directed by the processor.
The system is designed for robotic manipulation in dynamic environments where objects may move or change. The challenge is accurately updating a 3D scene model in real-time to reflect current conditions, enabling a robotic manipulator to grasp objects reliably. The system uses a prior 3D scene model, which includes 3D object models representing objects in a previous state. A processor receives a current image of the scene and initializes the 3D scene model to this prior state. It then updates the model by comparing image features from the current image with visible features of the 3D object models. Features are matched when they align spatially and have no better match with other objects. Associated data updates existing object models, while unassociated data is used to detect and add new objects. The system also removes objects from the model if they are no longer present. After updating the model, the processor computes a grasp for one of the objects and directs the robotic manipulator to execute it. The manipulator then grasps the object as instructed. This approach ensures the robot operates with an accurate, up-to-date scene representation, improving grasping success in dynamic environments.
Unknown
May 12, 2020
Browse 5M+ US patents with plain-English claim translations and AI-generated analysis.