Safety features for a robot assembly and method of same

This U.S. National Stage Patent Application claims the benefit of PCT International Patent Application Serial No. PCT/US2022/018921 filed Mar. 4, 2022 entitled “SAFETY FEATURES FOR A ROBOT ASSEMBLY AND METHOD OF SAME” which claims the benefit of and priority to U.S. Provisional Patent Application Ser. No. 63/156,434, filed Mar. 4, 2021 and titled “Safety Features for a Robot Assembly and Method of Same,” the entire disclosures of which are hereby incorporated by reference.

The present invention relates to robots in manufacturing environment. More particularly, the present invention relates to safety features for robots used in a manufacturing environment.

This section provides background information related to the present disclosure which is not necessarily prior art.

As productivity and efficiency are the goals in any manufacturing environment, robots are being utilized, now more than ever, for a greater number of tasks. Conventional robots are known to weld, assemble, and move workpieces between locations, whereas humans typically perform more complicated tasks that require recognition of several types of objects. While there is typically some overlap between an area in which a robot is operating and an area in which a human is working, robots have traditionally operated within the confines of highly-controlled environments. In highly controlled environments, robots can be programed to execute repeated and preprogramed tasks with limited risks of damaging nearby environments. However, with improvements in technology, modern robots are now being programmed to perform more and more advanced tasks similar to those of a human. With these advanced tasks, robots are now being placed in close confines with humans and with higher degrees of freedom than ever before. The major concern of introducing robots to a manufacturing environment where humans are working is safety. Specifically, the concern is that a robot's movement and a human's movement will intersect; causing a collision that ultimately harms the human or damages the robot. The main goal of using robots is productivity that, in theory, will recoup the large initial cost of buying the robots. Productivity requires these robots to perform tasks quickly, but the faster a robot performs a task, the higher the magnitude of potential harm to both the robot and the human during a collision. As is often times the case, an attempt to create a safer setting for humans detrimentally affects productivity.

Various strategies have been employed to create safer manufacturing environments. One strategy is to create a physical barrier around an area in which the robot operates so that a human must remain a safe distance away. This method, while simple, requires additional space in the manufacturing environment and is generally over restrictive, particularly for robots that move between locations.

All of these strategies slow productivity and are not guaranteed to avoid collision with a human, particularly a human that is moving. Moreover, even when a human is standing in one place, their upper body or a carried object will move as required for that given task creating an area of risk beyond what is anticipated. Accordingly, there exists a need to advance the art such that neither safety nor productivity must suffer.

According to one aspect of the invention, a method for operating a robot assembly to prevent collision with a human in a manufacturing setting is provided. The method includes the steps of: providing a robot assembly with a positional apparatus having at least one tag; performing predefined movements with the robot assembly associated with an assigned task; and detecting the at least one tag on the positional apparatus with a detector and generating a movement model of the robot assembly.

In one aspect, the method includes replacing the positional apparatus with a robotic hand configured to perform tasks associated with the movement model.

In one aspect, the positional apparatus includes a plurality of sides with each side including at least one tag.

In one aspect, each of the sides includes at least two distinct tags.

In one aspect, the method includes capturing a plurality of images of the robot assembly during the assigned task.

In one aspect, the method includes extrapolating 3D information from the plurality of images.

In one aspect, the method includes generating posture nodes associated with a human.

In one aspect, the method includes determining a range of motion of the human.

In one aspect, the method includes detecting a human approaching the robot assembly during performance of the predetermined task.

In one aspect, the method includes stopping or changing the performance of the predefined task in response to detecting the human approaching.

In another aspect, a vision processing method is provided, including the steps of: capturing a plurality of images of a robot assembly having a plurality of tags coupled to the robot assembly; processing the plurality of images and determining a hazard present within a working environment; extrapolating 3D information from the plurality of images; and continuously capturing additional images and, in response thereto, modifying a predetermined movement of the robot assembly.

In one aspect, the method includes modeling human posture and generating a plurality of posture nodes.

In one aspect, the method includes capturing images of a human working in close proximity with the robot assembly.

In one aspect, the method includes extrapolating 3D information from the images of the human working and determining the hazard is present.

In another aspect, a method of calibrating a robot assembly is provided, the method including: generating operating instructions for a robot assembly corresponding to a predetermined task and waypoint; moving the robot according to the operating instructions and the predetermined task and waypoint; detecting a plurality of locations of a positional apparatus of the robot assembly; extrapolating locations of the positional apparatus after detecting the plurality of locations; after extrapolating the locations, detecting precise locations of the positional apparatus; comparing the precise locations with the extrapolated locations and generating pose nodes; accumulating pose nodes for a plurality of positions of the robot assembly and generating a movement model of the robot assembly.

In one aspect, the method includes generating bypass information differing from the movement model for avoiding an object entering the waypoint.

In one aspect, the method includes departing from the movement model according to the bypass information in response to detecting a hazard.

In one aspect, the method includes identifying a variance between the extrapolated locations and the precise locations and reducing the variance by modifying the extrapolated location with the precise locations.

In one aspect, the method includes predicting an object overlapping with the robot assembly during performance of the predetermined task and initiating safety protocols in response thereto.

In another aspect, a system for operating a robot assembly is provided, the system including at least one computing device having at least one processor and at least one memory having instructions stored thereon that, when executed by the at least one processor, cause the at least one processor to perform the methods described above and herein.

Further areas of applicability will become apparent from the description provided herein. The description and specific examples in this summary are intended for purposes of illustration only and are not intended to limit the scope of the present disclosure.

Example embodiments will now be described more fully with reference to the accompanying drawings. In general, the subject embodiments are directed to a robot assembly and a method of operation. However, the example embodiments are only provided so that this disclosure will be thorough, and will fully convey the scope to those who are skilled in the art. Numerous specific details are set forth such as examples of specific components, devices, and methods, to provide a thorough understanding of embodiments of the present disclosure. It will be apparent to those skilled in the art that specific details need not be employed, that example embodiments may be embodied in many different forms and that neither should be construed to limit the scope of the disclosure. In some example embodiments, well-known processes, well-known device structures, and well-known technologies are not described in detail.

Referring to the Figures, wherein like numerals indicate corresponding parts throughout the views and charts, the robot assembly and the method of operation are intended for preventing collision between robots and environments, such as human workers, while maintaining robot efficiency and productivity in a manufacturing environment.

With Reference initially to, a schematic view of a safeguard systemis provided, wherein a robot assemblyreceives continued feedback during operation. The safeguard systemincludes a series of sensorsthat detect one or more environmental hazards within a working environment. For example, the hazards within a working environmentmay include moving and non-moving hazards, such as humans, other robots, environmental infrastructures, and other hazards. A motion plannerincludes a predetermined or preprogramed travel path instruction that may be stored locally at the robot assembly or remotely. A safeguard modulemay bypass the travel path and/or motion plannerinstructions in the event of detecting the one or more hazards within a working environment. The robot assemblymay include a controllerfor receiving information from the sensors, the motion planner, and the safeguard modulefor preventing movement of the robot assemblyupon a sensed or detected hazard within a working environment. For example, the robot assemblymay move in accordance with travel path instructions provided by the motion planer, during movement, the sensorsmay sense or detect one or more hazards within a working environmentthat pose a risk of collision if the robot assemblycontinues along the travel path. Upon a threat of collision, the safeguard moduleconstrains continued movement along the travel path. The constraint on movement may include generating and selecting a bypass travel path. The safeguard modulemay work in conjunction with the controller.

is a schematic view of a computing systemof the robot assembly. The computing systemincludes a plurality of sensors, such as camerasthat have been synchronized and synchronously transmit data to a computing device. The cameras may include Blackfly S cameras transmitting data via a network, such as a local area network consisting of one or more Ethernet cables. The networkmay include a Power over Ethernet (PoE) configuration including a master POE switch. Data from the camerasmay then be transmitted to the computing device. The camerasmay be synchronized via a master clockthat connects to the POE switchand the cameras. For example, the master clockmay include a Raspberry Pi device that is triggered via general purpose input/output (GPIO) between the camerasand Raspberry Pi device. The computing devicemay include a controllerthat includes a processorand a memory. The memorymay include instructions that, when executed by the processor, cause the processorto perform various functionalities, for example, movement of the robot assembly. It should also be appreciated that the controllermay also include a processorand a memory. The memorymay include instructions that, when executed by the processor, cause the processorto perform various functionalities, for example, movement of the robot assembly. The computing deviceand controllerthereof may be configured to provide instructions to more than one robot assembly. In some embodiments, the memoryandmay in RAM, ROM, solid state, or any other type of storage. In some embodiments, the memoryandmay include GigE Vision and other protocols for transmitting one or more images from the cameras. In some embodiments, each cameraincludes an ROS driver. In some embodiments, connection between camerasthe master clock, the robot, the master switch, and the computing devicemay be wireless via known technologies such as Bluetooth.

is a schematic view of a vision processing methodof the robot assembly. At, the vision processing methodincludes capturing an at least one image. For example, the camerasmay capture a series of BayerB8 images. At, the images are transmitted from the camerasto the computing devicewhereat they are converted via instructions executed saved on memoryand carried out by the processor. For example, the images may be converted into a RGB format and published with additional camera information, such as time stamp, originating camera, etc. The image may then be undistorted atif necessary. The undistorted image can then be processed at. For example, the computing devicemay process the images and determine the presence of any hazards within a working environment. In some embodiments, each or select camerasmay include RGB cameras the steps,, andmay further include extrapolating 3D information from a series of 2D images for one or more of the cameras. For example, the memoryandmay include multi-view stereo (MVS) or structure through motion algorithms carried out by one or more of the processorsand.

is a perspective view of the robot assembly, the robot assemblyincludes a baseand at least one armextending from the baseto an attachment locationfor a robotic hand. The handperforms certain operations via instructions from one or more of the processorsand. At least one jointis located along the armfor articulating the handalong three dimensional routes. The baseinclude moveable elements, such as wheels or tracks, that permit the robot assemblyto move along travel paths between locations.

With continued reference to, the robot assemblyincludes a calibration systemincluding a positional apparatusfor connection to the attachment locationduring a calibration process. The positional apparatusincludes a plurality of sides, wherein each sideincludes at least one tagwith distinguishing features. In some embodiments, the positional apparatusis cube-shaped and at least five of the sidesA-E include more than one tag. For example, each of the sidesA-E may include four tags located in each quadrant of each of the sides. The positional apparatusand/or spacing between tagsmay correspond to the size of the handin its most expanded operational position, e.g., when fingers/clamps have been extended. In some embodiments, the tagsmay be configured as April tags. One or more tag readersmay be configured to detect a location of each of the tags. The location detection of the tagsmay be primarily or entirely for calibration before usage of the robot assemblyin a manufacturing environment.

A calibration methodis proved inin accordance with the details provided in. The calibration methodincludes generating operation instructionsfor a robot assembly. For example, the operating instructionsmay include one or more locations (e.g., waypoints) wherein certain tasks will be completed, such as organizing, sorting, etc. The robot assemblyis then instructedto travel in accordance with the operation instruction. For example, the robot assemblyis instructed to travel between one or more locations to perform various tasks. Once the robot assemblyis located in a location to perform an assigned task, the robot assemblymoves in accordance with the assigned task wherein the positional apparatusis detected. In some embodiments, the calibration methodinclude enough camerasat locations that allow each of the tagsto be captured by more than one camera, such as at least three cameras. At, the images obtained by the camerasare extrapolated into 3D information in accordance with the vision processing methodof. Based on the extrapolated information, an extrapolated location of the positional apparatusmay be generated. The extrapolated location of the positional apparatusmay include extrapolated locations of one or more tags. After generation of the 3D model, the one or more tagsmay be detected via the one or more tag readersfor precise location of the positional apparatusand one or more tags. The precise locational information may then be compared to the extrapolated locational information to generate pose nodes corresponding to the position of the robot assembly. These nodes can be accumulated for various positions of the robot assembly to generate a model at. Atthe methodmay include tagging nodes and saving them in memoryor. The variances between extrapolated locational information can then be reduced via a modification of the extrapolated locational information based on the precise locational information. With continued reference to step, a robot assembly activity profile of positional details vs time can be developed. The robot assembly activity profile may correspond to an activity at a given location and a given robot assembly task for establishing predictive locational information for the robot assembly. As such, objects entering a location that is predicted to have overlapping robot assembly movement can initiate safety moduleprotocols to provide bypass information to the robot assembly.

In some embodiments, the methodutilizes one positional apparatusand four cameras. During method, an estimate is provided of Tend/tag as X, and Tworld/cam i as Y. Thus estimates may be generated for robot forward kinematics

and camera observation

These estimates may be optimzed via SE(3) transformation parameterizations and smooth manifold optimizations with the Levenberg-Marquardt algorithm.

provide a systemfor human posture generation. The systemincudes modeling the human body with a plurality of posture nodes. The posture nodesmay include nodes that correspond to the joints and the extension of various human appendages. In some embodiments, these posture nodesmay be used in the vision processing method. For example, the memoryandmay include instructions for generating the human posture with a plurality of human posture nodes. For example, the camerasmay capture images of a human working in close proximity to a robot assemblyor a location with a robot assembly activity profile. The method may include extrapolating 3D information from a series of 2D images of the human (or posture nodes) using extra multi-view stereo (MVS) or structure through motion algorithms carried out by one or more of the processorsand.

The posture nodesmay further be used to identify a human in a relaxed state, for example, standing upright arms resting against their body. However, based on the human's designated work task, their arms or legs may have to extend outwardly away from their body, or in some cases, the human may need to move an associated work object. These task oriented movements will cause the posture nodesto extend outwardly. Distance from each identifiable posture nodeof a human may be used to determine a range of motion, such that an identifiable threats of overlap and be predicted early and the system described herein can generate a bypass route.

With reference now to, a methodof operating a robot assembly in a hazardous environment is provided. At, the methodincludes providing a robot assembly with a positional apparatus. At, the methodincludes performing predetermined movements with the robot assembly associated with an assigned task. At, the methodincludes capturing a plurality of images of the robot assembly as the robot assembly executes the assigned task. At, the methodincludes extrapolating 3D information from the plurality of images. For example, determining a location of one or more tags on the positional apparatus. At, the methodincludes detecting one or more tags on the positional apparatus with at least one detector. At, the methodincludes determining variances between the 3D information and the detected tag information. At, the methodincludes saving the 3D information, the detected tag information, and the variance of a series of movements associated with the assigned tasks to generate a task profile associated with the robot assembly. At, the methodincludes generating posture nodes associated with a human. At, the methodincludes replacing the positional apparatus with a robotic hand. At, the methodincludes performing the assigned task with the robot assembly. At, the methodincludes continuously capturing a plurality of images of the robot assembly as executes the assigned task. At, the methodincludes extrapolating 3D information from the plurality of images. At, in response to a recognition of a human approaching the robot assembly and the predetermined movement locations, stopping or changing the predetermined movements of the robot assembly based on predicted extensions of the human using posture nodes. At, in response to a recognition of a human crossing a predetermined movement location, stopping or changing the predetermined movements of the robot assembly.

provide additional details of the robot calibration method. With initial reference to, the methodmay include using multiple cameras and solving for AX=YB individually to generate an 3D extrapolated location of the robot assembly, such as the hand, base, or arm.provides a pose-graph generation step, including two states xi and xj of the robot assemblywith a relative factor measurement zij. An predetermined path and or bypass path may be generated based on optimal states of xi and xj via the provided posteriori algorithm.provides additional steps of the pose-graph generation step including generating a model wherein G=(X,Z). The factor X is a set of poses at each state that are vertices of G. The factor Z is a set of all factors which are edges of G. The pose-graph generation can thus be used to estimate a series of states (e.g., location) of the robot assembly.

The system, modules, assemblies, methods and/or processes described above, and steps thereof, may be realized in hardware, software or any combination of hardware and software suitable for a particular application. The hardware may include a general purpose computer and/or dedicated computing device or specific computing device or particular aspect or component of a specific computing device. The processes may be realized in one or more microprocessors, microcontrollers, embedded microcontrollers, programmable digital signal processors or other programmable device, along with internal and/or external memory. The processes and modules may also, or alternatively, be embodied in an application specific integrated circuit, a programmable gate array, programmable array logic, or any other device or combination of devices that may be configured to process electronic signals. It will further be appreciated that one or more of the processes may be realized as a computer executable code capable of being executed on a machine readable medium.

The computer executable code may be created using a structured programming language such as C, an object oriented programming language such as C++, or any other high-level or low-level programming language (including assembly languages, hardware description languages, and database programming languages and technologies) that may be stored, compiled or interpreted to run on one of the above devices, as well as heterogeneous combinations of processors, processor architectures, or combinations of different hardware and software, or any other machine capable of executing program instructions.

Thus, in one aspect, each method described above and combinations thereof may be embodied in computer executable code that, when executing on one or more computing devices, performs the steps thereof. In another aspect, the methods may be embodied in systems that perform the steps thereof, and may be distributed across devices in a number of ways, or all of the functionality may be integrated into a dedicated, standalone device or other hardware. In another aspect, the means for performing the steps associated with the processes described above may include any of the hardware and/or software described above. All such permutations and combinations are intended to fall within the scope of the present disclosure.

It should be appreciated that the foregoing description of the embodiments has been provided for purposes of illustration. In other words, the subject disclosure it is not intended to be exhaustive or to limit the disclosure. Individual elements or features of a particular embodiment are generally not limited to that particular embodiment, but, where applicable, are interchangeable and can be used in a selected embodiment, even if not specifically shown or described. The same may also be varies in many ways. Such variations are not to be regarded as a departure from the disclosure, and all such modifications are intended to be included within the scope of disclosure.