Mixed reality assisted spatial programming of robotic systems

1. A computer-based method for spatial programming of a robotic device, the method comprising: selecting, by a mixed reality tool, entities of a mixed reality environment for simulation as simulated entities; selecting, by the mixed reality tool, an object related to one or more interactive tasks for the robotic device; determining, by a skill engine, a spatial location of the object including Cartesian coordinates and orientation coordinates of the object; executing an application program to operate the robotic device using the spatial location; simulating, in-a the mixed reality environment, execution of the one or more interactive tasks by the robotic device on the object related to a skill set, the execution based on initial parameters; receiving system data from the mixed reality environment comprising virtual data related to characteristics of the simulated entities and real data from one or more real components of the mixed reality environment; processing the received system data for various simulations of the mixed reality environment in each of a set of modes, including: a real-real mode in which only the real components interface the application program, a virtual-virtual mode in which one or more simulated entities respond to the application program and simulate activity or operations tracked by the mixed reality tool to produce virtual data inputs to the skill engine, and a real-virtual mode in which the skill engine processes the system data for one or more of the real components simultaneously with data for one or more of the simulated entities; and iteratively performing the following steps until a convergence of parameters is achieved for execution of the one or more tasks within spatial tolerance thresholds: modifying, by the mixed reality tool, the initial parameters in response to gesture inputs received from a user; and repeating the simulating of the execution of the one or more tasks using the modified parameters.

2. The method of claim 1 , further comprising: receiving an indication of an acceptable simulated execution of the one or more tasks based on the convergence of parameters; mapping the skill set to the parameters corresponding to the indication; and uploading the mapping to a database.

3. The method of claim 1 , further comprising: displaying an indication to confirm selection of the object.

4. The method of claim 1 , wherein the skill set includes a task of grasping the object, and the simulating comprises: setting a grasp pose for a gripper of the robotic device relative to the Cartesian coordinates and the orientation coordinates of the object, wherein the grasp pose includes relative Cartesian coordinates and orientation coordinates; and displaying a virtual representation of the robotic device performing a simulated grasp of the object based on the grasp pose in the mixed reality environment.

5. The method of claim 1 , wherein the skill set includes a task of placing the object, and the simulating comprises: selecting a target location related to placement of the object; displaying a virtual marker at the target location; determining Cartesian coordinates and orientation coordinates of the target location; setting a release pose for a gripper of the robotic device relative to the Cartesian coordinates and the orientation coordinates of the target location, wherein the grasp release pose includes relative Cartesian coordinates and orientation coordinates; and displaying a virtual representation of the robotic device performing a simulated placement of the object onto the target location based on the release pose in the mixed reality environment.

6. The method of claim 5 , further comprising: displaying an indication to confirm selection of the target location.

7. The method of claim 5 , wherein the skill set includes an obstacle avoidance task for a pathway the object, and the simulating comprises: identifying any obstacles between the location of the object and the target location; setting a pathway between the location of the object and the target location that avoids contact with the identified obstacles; and displaying a virtual representation of the robotic device performing a simulated movement of the object along the pathway in the mixed reality environment.

8. The method of claim 1 , wherein the mixed reality environment comprises one or more of the robotic device, the object, and the target being simulated by a virtual representation.

9. A system comprising: a mixed reality tool configured to select entities of a mixed reality environment for simulation as simulated entities, and select an object related to one or more interactive tasks for the robotic device; a skill engine for an automation system, the skill engine comprising an application program comprising a first module configured to determine a spatial location of the object including Cartesian coordinates and orientation coordinates of the object, and a second module configured to execute an application program to operate the robotic device using the spatial location; and a simulator configured to: simulate, in the mixed reality environment, execution of the one or more tasks by the robotic device on the object related to a skill set, receive system data from the mixed reality environment comprising virtual data related to characteristics of the simulated entities and real data from one or more real components of the mixed reality environment, and process the received system data for various simulations of the mixed reality environment in each of a set of modes, including: a real-real mode in which only the real components interface the application program, a virtual-virtual mode in which one or more simulated entities respond to the application program and simulate activity or operations tracked by the mixed reality tool to produce virtual data inputs to the skill engine, and a real-virtual mode in which the skill engine processes the system data for one or more of the real components simultaneously with data for one or more of the simulated entities; wherein the mixed reality tool is further configured to: modify the initial parameters in response to instructions gesture inputs received from a user; wherein the simulator is further configured to repeat the simulating of the execution of the one or more tasks using the modified parameters; and wherein the parameters are modified and the execution of the one or more tasks are repeated iteratively until a convergence of parameters is achieved for execution of the one or more tasks within spatial tolerance thresholds.

10. The system of claim 9 , wherein the mixed reality tool is further configured to receive an indication of an acceptable simulated execution of the one or more tasks based on the convergence of parameters; wherein the skill engine is further configured to map the skill set to the parameters corresponding to the indication and to upload the mapping to a database.

11. The system of claim 9 , wherein the mixed reality tool is further configured to display an indication to confirm selection of the object.

12. The system of claim 9 , wherein the skill set includes a task of grasping the object, and the simulator is further configured to: set a grasp pose for a gripper of the robotic device relative to the Cartesian coordinates and the orientation coordinates of the object, wherein the grasp pose includes relative Cartesian coordinates and orientation coordinates; and the mixed reality tool is further configured to display a virtual representation of the robotic device performing a simulated grasp of the object based on the grasp pose in the mixed reality environment.

13. The system of claim 9 , wherein the skill set includes a task of placing the object, and the simulator is further configured to: select a target location related to placement of the object; render a display of a virtual marker at the target location; determine Cartesian coordinates and orientation coordinates of the target location; set a release pose for a gripper of the robotic device relative to the Cartesian coordinates and the orientation coordinates of the target location, wherein the grasp pose includes relative Cartesian coordinates and orientation coordinates; and render a display of a virtual representation of the robotic device performing a simulated placement of the object onto the target location based on the release pose in the mixed reality environment.

14. The system of claim 13 , wherein the mixed reality tool is further configured to display an indication to confirm selection of the target location.

15. The system of claim 13 , wherein the skill set includes an obstacle avoidance task for a pathway the object, and the simulator is further configured to: identify any obstacles between the location of the object and the target location; set a pathway between the location of the object and the target location that avoids contact with the identified obstacles; and render a display of a virtual representation of the robotic device performing a simulated movement of the object along the pathway in the mixed reality environment.

16. The system of claim 9 , wherein the mixed reality environment comprises one or more of the robotic device, the object, and the target being simulated by a virtual representation.