Augmented reality system and method for conveying to a human operator information associated with human-imperceptible indicia within an operating environment of a robot

This application claims the benefit of U.S. Provisional Patent Application 63/048,123, filed Jul. 4, 2020, and entitled, “Augmented Reality System and Method for Conveying To a Human Operator Information Associated With Human-Imperceptible Indicia within an Operating Environment of a Robot,” which is incorporated by reference herein in its entirety.

Having a robot traverse a space and interact with objects can be a difficult task. The robot must be able to distinguish obstacles which could damage the robot, or which can be damaged by the robot. In addition, the robot must be able to identify objects with which it must interact in order to carry out a task and/or follow instructions. Furthermore, some objects can present dangers or difficulties to interaction, such as the difference between a freezer and a refrigerator door, or which knob on a stove corresponds to which burner, and others.

In some operating scenarios, an autonomous robot can operate within an operating environment. In other operating scenarios, control of the robot or robotic device can be performed at least in part by a human operator, such with a remote control device, with a tele-operated control system (e.g., a master/slave system), or a combination of these. This is typically done where it is desired to maintain at least some control over the movements of the robot rather than the robot being fully autonomous. Within these operating scenarios, there may be instances where the human operator is located in a remote location from the robot where direct, real-world viewing of the operating environment (the environment in which the robot resides and is operating) by the human operator is not possible or desired. In these cases, a human operator can utilize an augmented reality system as part of an augmented reality platform, operable with the robot, where the human operator is able to view the operating environment in which the robot resides using an electronic display (e.g., a headset, one or more screens, etc.). The electronic display can be electronically connected or interfaced with the robot to receive sensed information (e.g., still or video images from one or more cameras on the robot), and/or sensed information directly from a sensor within the operating environment (e.g., one or more cameras mounted or otherwise present within the operating environment). The augmented reality system facilitates remote, electronically transmitted real-world human operator viewing of the operating environment and any desired control over the robot within the operating environment without the operator being physically present within the operating environment. The augmented reality system can be employed to oversee an autonomous robot operating in a remote environment, or to facilitate at least partial human operator control (i.e., tele-operated control or operation) of the robot in the remote environment, depending upon the type of robot and/or the level of human control to be associated with the robot.

While augmented reality systems can provide certain advantages when a robot is being remotely operated (tele-operated), there are conditions where at least partial human operator control of the robot or fully autonomous operation of the robot (with a human operator performing oversight) within the operating environment falls short of needed or desired performance, namely because some conventional augmented reality systems are incapable of conveying enough information to the human operator to overcome some, if not all, of the performance deficiencies that can be caused by inherent difficulties associated with one or more of the objects within the operating environment. For example, a fully autonomous robot may not be able to adequately perform certain tasks, or interact with one or more objects within the operating environment as a result of inadequate programming, complexity of the task or interaction requirements, or for other reasons. In another example, a robot that is at least partially controlled by a human operator may encounter certain tasks, or may be required to interact with one or more objects, that the human operator is unfamiliar with, or that are complex in nature, or that for any other reason prohibit or limit the ability of the human operator to complete the task(s) or interact with the object. In these situations, the human operator using some of the conventional augmented reality systems will likely be unable to enhance the performance of the robot simply by being able to view the real-world view of the operating environment due to the limits of the information conveyed to the human operator through the augmented reality system.

Of course, some augmented reality systems utilize object recognition or other machine learning methodologies to try and teach the robot how to interact with certain objects, and then convey associated information to human operators in the form of computer generated imagery based on the results of the object recognition process, but these methodologies can be extremely costly and can employ complex algorithms that may be insufficient for interaction with some objects within an operating environment, such as those that are more complex or difficult for a robot to interact with.

Reference will now be made to the exemplary embodiments illustrated, and specific language will be used herein to describe the same. It will nevertheless be understood that no limitation of the scope of the invention is thereby intended.

An initial overview of technology embodiments is provided below and then specific technology embodiments are described in further detail later. This initial summary is intended to aid readers in understanding the technology more quickly but is not intended to identify key features or essential features of the technology nor is it intended to limit the scope of the claimed subject matter.

The present disclosure sets forth a robotic system comprising a robot; human-imperceptible indicia associated with an object within an environment in which the robot operates, the human-imperceptible indicia facilitating access to interaction information; at least one sensor operable to sense the human-imperceptible indicia and the interaction information; an augmented reality system comprising a computer, the augmented reality system in communication with the robot, and operable to convey human-understandable information corresponding to or otherwise associated with the interaction information to a human operator; and interaction information stored in a database, the interaction information being associated with the linking information and pertaining to a predetermined intended interaction of the robot with the object, the interaction information being operable to facilitate interaction with the object by the robot in accordance with the predetermined intended interaction.

In one example, the augmented reality system can further comprise a translation module operable to translate the interaction information into the human-understandable information, and to transmit the human-understandable information to the operator.

The present disclosure also sets forth a robotic system comprising a robot; human-imperceptible indicia associated with an object within an environment in which the robot operates, the human-imperceptible indicia comprising interaction information directly thereon pertaining to a predetermined intended interaction of the robot with the object, the interaction information being operable to facilitate interaction with the object by the robot in accordance with the predetermined intended interaction; at least one sensor operable to sense the human-imperceptible indicia and the interaction information; and an augmented reality system comprising a computer, the augmented reality system in communication with the robot, and operable to convey human-understandable information corresponding to or otherwise associated with the interaction information to a human operator.

In one example, the robot can comprise a camera, and the augmented reality system can further comprise a visual display viewable by a human operator, the augmented reality system being operable to display, at least in part, an electronically transmitted real-world view of the operating environment on the visual display as captured by the camera. The robot can utilize the camera to transmit the real-world view of the operating environment to the augmented reality system that can be combined with the human-understandable information to provide augmented reality operation of the robot.

The present disclosure further sets forth a method for deploying a robot, as part of an augmented reality platform, within a machine navigable environment, the method comprising identifying a desired space in which the robot will operate; selecting one or more objects that are or will be located in the space; marking the one or more objects with human-imperceptible indicia being machine-readable, the human-imperceptible indicia comprising at least one of interaction information or linking information that links to a database comprising interaction information; introducing the robot into the space, the robot being capable of movement within the space; operating an augmented reality system in communication with the robot, the augmented reality system comprising a computer, and operable to convey human-understandable information to a human operator, the human-understandable information corresponding to or otherwise being associated with at least one of the interaction information or the linking information; sensing the human-imperceptible indicia; conveying the human-understandable information to a human operator; and facilitating interaction with the one or more objects by the robot in accordance with the human-understandable information.

The system and method can comprise marking or identifying an object to be perceptible (i.e., machine-readable) to a robot or the augmented reality system, while being invisible or substantially invisible to humans. Such marking can facilitate identification, interaction and navigation by the robot, or a human operator overseeing or at least partially operating the robot using the augmented reality system and human-readable interaction instructions as conveyed to the human operator through the augmented reality system. The markings can create a machine or robot navigable environment (an operating environment) for the robot, whether autonomously operated, or operated at least in part by the human operator (such as via a tele-operated control system operable with the augmented reality system to facilitate at least partial control of the robot). By way of example, such an object can be an appliance, such as a refrigerator, with which the robot approaches and interacts in order to complete a task, such as to retrieve an item (e.g., a beverage). Thus, the appliance or refrigerator can have a marking, label, tag (i.e., indicia) that is machine-readable, and that can identify the appliance as a refrigerator, and thus the object with which the machine or robot will interact to retrieve the item, as well as that can provide information about the refrigerator, such as how to interact with it, its status, and any other information relevant to and pertaining to the refrigerator or how the robot can interact with the refrigerator. Obviously an operating environment containing an appliance is not intended to be limiting in any way as a robot can be placed within any operating environment in which there are other types of objects. Such environments and associated objects can be dangerous or impractical for humans, such as a hostile operating environment with hostile objects, a hazardous operating environment with hazardous objects, and others. As used herein, the term “indicia” refers to one or more than one “indicium” that is associated with an object or otherwise present within the operating environment. The indicia can be contained in or on a tag or other marker placed on the object, printed or otherwise attached to or associated with the object, supported on a structure within the operating environment, etc. The indicia can include various information, such as information that identifies the parts of the object, such as the handle of the refrigerator, again with which the robot will interact. Multiple different markings, labels, tags or other indicia can be used to distinguish the multiple different parts of the object. In keeping with the refrigerator example, this can include the refrigerator handle, the freezer handle, the ice maker dispenser, the water dispenser, different shelves or bins, etc. Furthermore, the indicia can also indicate information pertaining to a status of an object or objects, direction (travel direction), magnitude, operating or other interaction instructions, information related to or ancillary to the object, or any other information or parameters that may be useful in terms of how the robot should or could interact with the object, whether operating autonomously, or being operated at least in part by a human operator, such as via a tele-operated control system and the augmented reality system. Using the example of the refrigerator, the indicia can indicate which direction the door opens, how much force should be exerted to open the door, the degree of rotation to be carried out, instructions regarding removing items or moving items to a different location within the refrigerator, the make/model of the refrigerator, how long the refrigerator has been in use, instructions for repair or maintenance, and any other associated information.

The machine readable indicia can comprise symbols, codes, wording, etc. that can be perceived and interpreted by the robot or the augmented reality system, but that are not visible to the unaided human eye. In one example, the robot or the augmented reality system can utilize a camera with an image sensor to see the indicia. In addition, the indicia can be invisible or substantially invisible to the unaided human eye so that such indicia does not create an unpleasant environment for humans, and remains aesthetically pleasing to humans. For example, the indicia can reflect UV light, while the image sensor can be capable of detecting such UV light. Thus, the indicia can be perceived by the robot or the augmented reality system, while not interfering with the aesthetics of the environment.

The machine readable indicia associated with an object can comprise indicia carried by or on the object itself, where the indicia comprises all or a part of the information that facilitates identification and interaction with the object within the operating environment. In another example, the machine readable indicia associated with an object can comprise indicia carried by the object itself, where the indicia comprises linking information that facilitates a link to a database where interaction information pertaining to the object can be accessed by the robot, or by one or more computer systems as part of the augmented reality system that facilitates identification and interaction with the object within the operating environment.

As used herein, the term “substantially” refers to the complete or nearly complete extent or degree of an action, characteristic, property, state, structure, item, or result. For example, an object that is “substantially” enclosed would mean that the object is either completely enclosed or nearly completely enclosed. The exact allowable degree of deviation from absolute completeness may in some cases depend on the specific context. However, generally speaking the nearness of completion will be so as to have the same overall result as if absolute and total completion were obtained. The use of “substantially” is equally applicable when used in a negative connotation to refer to the complete or near complete lack of an action, characteristic, property, state, structure, item, or result.

As used herein, “adjacent” refers to the proximity of two structures or elements. Particularly, elements that are identified as being “adjacent” may be either abutting or connected. Such elements may also be near or close to each other without necessarily contacting each other. The exact degree of proximity may in some cases depend on the specific context.

As used herein, “visible” refers to visible to the unaided human eye and visible with light in the visible spectrum, with a wavelength of approximately 400 to 750 nanometers.

As used herein, “indicia” refers to both indicia and indicium, unless specified otherwise.

illustrates a schematic block diagram of a robotic system in accordance with an example of the present disclosure.illustrates a schematic perspective view of what a specific robotic systemrepresentative of the robotic system ofmay comprise. With reference to, the robotic systemcan comprise a robotconfigured to navigate and interact within a spacewithin an operating environment, the spacehaving any number of exemplary objects (e.g., objects-, with “n” representing any number) (e.g., see specifically objects-in) marked with machine-readable, but human-imperceptible, indicia, therein, defining a machine-navigable operating environment through which the robotcan navigate and interact, in accordance with an example.

The robotcan be mobile and self-propelled, such as with power-actuated joints, a propulsion system (e.g., wheels and a motor) and other components and systems. The robotcan comprise any type of robot, such as a tele-operated robot, a humanoid robot, or any other as understood by those of skill in the art. The robotcan have one or more arms with one or more end effectors in the form of grippers or fingers, or the like, for grasping or engaging objects. The robotcan comprise one or more sensors or sensor devices, including a camera. In one example, the robotcan be at least partially tele-operated by a human operator. Indeed, the robotic systemcan further comprise a remote or tele-operated control system(e.g., a master control system) in communication with the robot, such as via network, wherein the tele-operated control systemprovides the human operator with the ability to control, to at least some extent, movements, locomotion, sensor deployment and other functions of the robot. This can be done in close proximity to the robot, or from a remote location. In one example, the tele-operated control systemcan comprise a wearable master control system comprising an exoskeleton that can be donned by the human operator (e.g., see). The tele-operated control systemcan comprise various sensors, communications components/capabilities, and any other components or systems supported thereon (see) designed and configured to facilitate at least partial remote operation (tele-operation) of the robot.

The robotic systemcan further comprise at least one sensor operable to sense the human-imperceptible indicia, including any identification or interaction (including linking) information carried or supported on the one or more objects. In one example, the sensor can comprise the camerawith an image sensor as supported on the robot, wherein the image sensor is operable to view or sense objects and the environment. In one aspect, the cameraor image sensor can be operable in the UV spectrum, and can be capable or viewing or sensing UV light. In addition, the robotcan have a UV light sourcecapable of emitting UV light to illuminate and reflect off of the objects and environment. In one aspect, the UV light sourcecan emit, and the UV cameraor image sensor thereof can detect, near UV light (NUV) with a wavelength between 200 to 380 nanometers.

The robotic systemcan further comprise an augmented reality systemin communication with the robot, or the tele-operated control system, or both of these. The augmented reality systemcan be operable to convey (e.g., display) to an operator operating all or part of the robotic systeman electronically transmitted real-world view of at least a portion of the spaceof the operating environment. The real-world view can be generated from still or video images captured by the cameraor other sensors of the robot. The augmented reality systemcan further be operable to convey human-understandable information, to the human operator of the robotic system. “Human-understandable information” refers to information that a human understands and that corresponds to certain types or forms of one or more indiciaand/or interaction information associated with the indicia, which cannot initially be perceived by a human, such as because the one or more indicia are machine readable. As is discussed below, some indiciacan comprise human-understandable information that can be directly conveyed to the human operator, while some indiciacan comprise, or facilitate access to, at least one of machine readable interaction information that must be translated into human-understandable information prior to conveying it to the human operator in order to be understood by the human operator. The robotcan utilize a cameraor other sensor to transmit the real-world view of the operating environment to the augmented reality systemthat can be combined with the human-understandable information to provide augmented reality operation of the robot.

The augmented reality systemcan comprise a computer having all of the necessary processors, memory, I/O communication devices needed to facilitate the augmented reality. The augmented reality systemcan further comprise one or more operator or visual displays, in connection with the computer, the visual displaybeing viewable by the human operator, wherein the augmented reality systemcan display, at least in part, the electronically transmitted real-world view of the spaceand any human-understandable information. The one or more displayscan comprise one or more screens or monitors, a head-mounted display (HMD), or any other type of display viewable by the human operator or any combination of these. The augmented reality systemcan further comprise audible or haptic devices, such as sensors and transmission devices, which allow the human operator to receive human-understandable information in the form of audible sounds or haptic feedback.

The augmented reality systemcan be connected to the tele-operated control system, thus allowing the human operator to at least partially control the robotusing augmented reality, including using any conveyed human-understandable information, as will be discussed in more detail below.

In one example, the image sensor(s)operable to sense the human-imperceptible indicia, including any interaction, or linking or other information, can be supported within the augmented reality systemsuch as on at least one of the computer or the head-mounted displayof the augmented reality system. The image sensorcan be in communication with the sensor device(s)(e.g., camera) on the robot, which can be configured to transmit image data (e.g., still or video image data) to the image sensoras acquired by the camerafrom within the spaceof the operating environment. In another example, the image sensoroperable to sense the human-imperceptible indicia can be supported on the robot(e.g., the sensor can comprise the cameraand a suitable image sensor), as discussed above. In either scenario, the augmented reality systemcan be configured to display the indicia, or the information provided or facilitated by (e.g., linked to) the indicia, to the human operator in the form of human-understandable information, which, in one example, can be information that pertains to a predetermined intended interaction of the robot with the one or more objects. However, this is not intended to be limiting in any way as the human-understandable information can comprise any information for any purpose, such as that designed to assist the operator in operating the robotic system. Indeed, the human operator can utilize the human-understandable information obtained or derived from the human-imperceptible indiciaand conveyed in augmented reality to the human operator for a variety of reasons, such as to more efficiently and/or expertly operate the robotas it is caused to interact with the one or more objects within the space, to supervise or oversee and verify proper operation of the robot, for training of the robot, and others. In the example where the robotcomprises an autonomous robot, the augmented reality systemcan enable a human operator to oversee the movements of the robotand how the robotinteracts with the objects within the spaceby being able to view the human-understandable information that is associated with the indicia, which indiciais associated with the objectswithin the space. This information can be used to make various corrections in the robotic system as needed. In the example where the robotis at least partially operated or controlled by a human operator, such as via a master control system, the augmented reality systemcan enable the human operator to control one or more movements of the robotas it interacts with the objectswithin the spaceby viewing the human-understandable information associated with the one or more indicia associated with the objectswithin the space. With the human-understandable information being conveyed to the human operator from the image sensorsensing the indicia, the human operator can carry out instructions related to interacting with the object(s), note status or other parameters of the objects(s), perform maintenance or other functions on the object(s), manipulate the object(s)in a directed way, and/or perform any other function or task that can be made possible or more efficient with the human-understandable information. In addition, safety can be enhanced as the human operator is better equipped to interact with the object(s)in an intended manner. Moreover, as mentioned, training of the robotcan be facilitated by the augmented reality systemas the robotlearns how to interact with objects in a more efficient manner.

As the one or more indiciaon the object(s)is human-imperceptible (not viewable by a human in the visible spectrum), the augmented reality systemcan be connected to the sensorand the robotto facilitate receipt of the information associated with the indiciaand conveyance of such information to the human operator. In one example, the indiciacan comprise information that would otherwise be readable by the human operator were the indiciaviewable in the viewable spectrum. In other words, the indiciacan comprise human-understandable information that the augmented reality system can directly convey to the human operator, such as via the operator display(s), upon the sensor that is operable with the augmented reality system sensing (e.g., machine reading) the indicia.

In another example, the indiciacan comprise interaction information (including any linking information) that can be sensed by the sensor(i.e. machine read), but that must be translated into human-understandable information that can be displayed to the human operator. As such, the augmented reality systemcan further comprise a translation module or translation systemoperable to translate the interaction information (including any linking information) associated with the indiciainto human-understandable information, and to transmit or convey the now human-understandable information to the human operator, such as via the operator display(s)or an audible or haptic device. The translation modulecan comprise translation software comprising one or more translation algorithms that are able to facilitate the translation of the interaction information into the human-understandable information. For example, the human-imperceptible indiciacan comprise machine-readable indicia, such as a quick-response or QR code, containing or associated with certain interaction information and/or linking information, and the computer of the augmented reality systemcan be programmed, using the translation software executing a suitable translation algorithm (e.g., a QR code algorithm), to effectuate or carry out the translation of such interaction information into human-understandable information. This translation can comprise accessing a database of preprogrammed user-understandable information linked with, mapped to, or otherwise associated with the interaction information on the indicia. For example, the translation modulecan execute the translation algorithm, part of which can include instructions to access a database of human-understandable information that is linked with or mapped to or otherwise associated with the interaction information on the indiciasensed or machine read by the image sensor. Once retrieved, the information can be conveyed (i.e., transmitted) to the operator in conjunction with the real-world view made viewable to the operator by the augmented reality system. The human-understandable information can be different from the information that the robotmay utilize upon sensing the indiciaand reading the interaction information. For example, as discussed herein, the robotcan receive interaction information in the form of computerized instructions or other information executable by the robotin terms of how the robotis to interact with the object upon which the indicium or indiciais supported, such as the amount of force to use, the types and degree of movements to make (e.g. the degrees of rotation of one or more joints to undergo), etc. On the other hand, the translation modulecan facilitate translation of the interaction information into human-understandable information that is essentially equivalent in terms of how the robotis to perform and/or what the robotis to do as it pertains to interacting with the object(s) in support of the indicia, but that an operator can understand so as to effectuate the same or more efficient interaction of the robotwith the object. In other words, the human-understandable information translated by the translation moduleand conveyed to the operator will likely be more high-level information due to a human's ability to process complex tasks from simple instructions (e.g., go to and open the refrigerator door and retrieve a soda type of drink from the top shelf). The indiciathemselves can comprise an identifier in addition to the interaction information (including any linking information). The indicia identifiers, the interaction information, or a combination of these, can be linked or otherwise associated with human-understandable information stored in a database and properly mapped, such that the appropriate human-understandable information can be retrieved and conveyed to the operator upon a specific indicium or indicia being machine read by the image sensor. The human-understandable information can comprise as much detailed information as needed or desired that relates to the object or interaction with the object that the operator would need to be informed sufficiently as to the purpose of and function of that particular indicium or indicia, to assist the robotwith interaction with or otherwise performing a function related to the object, or to carry out a task associated with the object. It is noted that the operator display(s)or other devices (e.g., HMDor monitor(s)) can receive the human-understandable information from the robotor from the computer of the augmented reality system, depending upon the location of the sensor.

The augmented reality systemcan be configured to display on the operator display(s)(e.g., the HMDor one or more screens/monitors) the human-understandable information in addition to, and/or in synchronization with, the electronically transmitted real-world view. As such, the augmented reality systemcan link the human-understandable information to the indiciafrom which it is derived to properly and timely convey (i.e., electronically transmit) the human-understandable information to the operator (specifically to a display viewable by the operator) upon the indiciabeing sensed by the image sensor(s)(and translated if need be). In one example, the human-understandable information can be in the form of computer-generated imagery (e.g., text, pictures, and other graphics) superimposed over the electronically transmitted real-world view viewable by the operator. In another example, the human-understandable information can comprise audible or haptic messages conveyed to the operator, which messages can be linked to the indicia from which they are derived, and conveyed in association with the real-world view viewable by the operator. The computer-generated imagery can be representative of at least some of the interaction or other information associated with the indiciaas obtained by the sensorsensing the one or more indiciaon the one or more objectswithin the space.

There are several recognized advantages of the augmented reality system discussed herein. One such advantage is that the robot can be caused to perform complex tasks and other functions within the operating environment that it may not be able to otherwise perform. Utilizing the interaction information associated with the indiciapresent within the operating environment, and conveying (i.e., transmitting) this to an operator in the form of human-understandable information can overcome some, if not all, of the performance deficiencies that can be caused by inherent difficulties associated with one or more of the objects within the operating environment. For example, a partial or fully autonomous robot that otherwise may not be able to adequately perform certain tasks, or interact with one or more objects within the operating environment as a result of inadequate programming, complexity of the task or interaction requirements, or for other reasons, can now receive assistance from the operator utilizing the augmented reality system and the human-understandable information to perform the task, or interact appropriately and more efficiently with the object due to the interaction information associated with the indiciawithin the operating environment being conveyed to the operator in the form of human-understandable information. In another example, a robot that is at least partially controlled by a human operator and that may encounter certain tasks, or that may be required to interact with one or more objects, that the human operator is unfamiliar with, or that are complex in nature, or that for any other reason may otherwise prohibit or limit the ability of the human operator to complete the task(s) or interact with the object, can be made more functional and capable due to the interaction information associated with the indicia in the operating environment being conveyed to the operator in the form of human-understandable information as provided by the augmented reality system. With the augmented reality system disclosed herein, the human operator will be able to enhance the performance of the robot by being able to view the real-world view of the operating environment in conjunction with the human-understandable information that is derived from the indicia and the interaction information associated with the indicia, and that is conveyed to the operator through the augmented reality system.

Another advantage is that the augmented reality system discussed herein provides a much more efficient, effective and economical alternative to conventional augmented reality systems that utilize object recognition or other machine learning methodologies to try and teach the robot how to interact with certain objects, and then convey associated information to human operators in the form of computer generated imagery based on the results of the object recognition process, which methodologies can be extremely costly and can employ complex algorithms that may be insufficient for interaction with some objects within an operating environment, such as those that are more complex or difficult for a robot to interact with.

The objects (e.g., objects,, and) can vary depending upon the operating environment or space. In one aspect, the environment or spacecan be a home or kitchen therein. Thus, the objects can comprise built-in appliances, such as a refrigerator, a rangeor stove and cooktop, a microwave oven, a dishwasher, a trash compactor, etc. In another aspect, the objects can comprise countertop appliances, such as a blender, a rice-cooker, a vegetable steamer, a crockpot, a pressure cooker, etc. In addition, the objects can have an interface, such as a handle, knob, button, actuator, etc. In addition, the objects can be or can represent a physical boundary or barrier to movement or action of the robot, such as walls, counters or islands, etc. In addition, the objects can be or can represent a physical boundary or barrier subject to damage of the object by the robot, such as an outdoor window, an oven window, a microwave window, a mirror, a glass door of a cabinet, etc. In addition, the objects can be or can represent a physical boundary or barrier hazardous to the robot, such as stairs, machinery, etc. Furthermore, the objects can be or can represent a physical boundary or barrier subject to frequent movement, such as a chair, a door, etc. The objects can have exterior, outfacing surfaces, represented byof the refrigerator, that face outwardly and into the space. The spacecan comprise any type of space, such as within a commercial, government, or residential structure, any type of space within an indoor or outdoor setting, and the objects can comprise any type of objects. As such, those specifically identified and discussed herein are not intended to be limiting in any way.

The indiciacan be disposed on and carried by the object, such as on the exterior, outfacing surface, etc. In one aspect, the indiciacan comprise information pertaining to the object, to the interaction of the robot with the object, the environment, etc. For example, the indicia cancan identify the object, a location of an interface of the object, indicate the physical boundary or barrier thereof, indicate the position of the indicia within a series of indicia that comprise or that are associated with a sequence of tasks, or can comprise certain interaction information directly thereon, or that comprises database-related or linking information that facilitates access to such interaction information, such as by providing a link, opening or causing to be opened, or other access initiating indicia to a computer database comprising the information, wherein the robot, or an operator through the augmented reality system, can interact with the computer database to obtain the information designed to assist the robotin how to interact with the object, its environment, etc. This is explained in further detail below.

The phrase “interaction information,” as used herein, can mean information conveyed to or retrievable (e.g., through database-related or linking information that facilitates access to such interaction information, such as by providing a link, opening or causing to be opened, or other access initiating indicia to a computer database comprising the information) by the robotor the augmented reality systemfacilitated by reading of the marking indiciathat can be needed by or useful to the robotto interact with an object in one or more ways, wherein the interaction information facilitates efficient, accurate and useful interaction with the object by the robot. More specifically, the interaction information can comprise information that pertains to a predetermined intended interaction of the robotwith an object, and any other type of information readily apparent to those skilled in the art upon reading the present disclosure that could be useful in carrying out predetermined and intended robot interaction with the object in one or more ways. Interaction information can comprise, but is not limited to, information pertaining to the identification of the object (e.g., what the object is, the make or model of the object, etc.), information about the properties of the object (e.g., information about the material makeup, size, shape, configuration, orientation, weight, component parts, surface properties, material properties, etc. of the object), computer readable instructions regarding specific ways the robot can manipulate the object, or a component part or element of the object (e.g., task-based instructions or algorithms relating to the object and how it can be interacted with (e.g., if this task, then these instructions), information pertaining to the status of the object, information pertaining to how the robotcan interact with the object (e.g., direction, magnitude, time duration, etc.), relative information as it pertains to the object (e.g., information about the object relative to its environment (e.g., distance from a wall, location where it is to be moved to, etc.) or one or more other objects, information about its location (e.g., coordinates based on GPS or other navigational information), etc.), information about tasks that utilize multiple marking indicia (e.g., sequence instructions that can include identification of the sequence, the position of each indicia within a series of indicia, what the next sequential marking indicia is and where to go to find the next sequential marking indicia, etc.). The nature, amount and detail of the interaction information can vary and can depend upon a variety of things, such as what the robotalready knows, how the robotis operated, such as whether the robotis intended to function autonomously, semi-autonomously, or under complete or partial control via the remote or tele-operation control system(e.g., a master control system) in conjunction with the augmented reality system, etc. The interaction information described and identified herein is not intended to be limiting in any way. Indeed, those skilled in the art will recognize other interaction information that can be made available to the robot, or to the augmented reality system, where the interaction information is displayed in the form of human-understandable information, via the indicia. It is contemplated that such interaction information can be any that may be needed or desired for the robot, the human operator, or a combination of these to interact with any given object, and as such, not all possible scenarios can be identified and discussed herein. In one aspect, the indiciacan comprise interaction information (e.g., identification information, interaction instructions, etc.) in the form of machine language that is recognizable and understood by the robotor computer processor thereof, or by the augmented reality systemor computer processor(s) thereof, and that facilitates interaction of the robotwith one or more objects within the space. In another aspect, the indiciacan comprise linking information (e.g., codes, numbers, etc.) in the form of machine language that is recognizable by the robotor the augmented reality system, or computer processors thereof, wherein the linking information facilitates access (e.g., through a computer network) to a data repository containing the interaction information. The data repository can comprise a look-up table or library. Thus, the look-up table or library can comprise codes or numbers associated with interaction information. In one aspect, the data repository and the interaction information can be internal to the robotor part of the augmented reality system. In another aspect, a server and data repository and the interaction information can be external to the robotor the augmented reality system, and with which the robotor the augmented reality systemcan communicate during operation via a computer network or cloud-based system accessible by the robotor the augmented reality system.

As indicated above, the spaceand the operating environment can comprise any type in any location in which it may be desirable to operate one or more robots. In one example, the environment or spacecan be an office. Thus, the objects can comprise walls, cubicles, desks, copiers, etc. In another example, the environment or spacecan be an industrial or manufacturing warehouse, plant, shipyard, etc. Moreover, the objects can comprise products, machinery, or any other objects. Essentially, it is contemplated herein that the labels with indiciacan be utilized in any conceivable environment in which a robot could operate, and that they can be utilized on any objects or items or structural elements within such environments, and that they can comprise any type of information of or pertaining to the environment, the objects, the interaction of these with one another, etc.

depicts a schematic front view of a marker in the form of a label comprising indiciaconfigured to be applied to an object in accordance with an example. As such, the marker can comprise a physical medium or carrier operable to support or carry thereon the marking indicia. For sake of clarity, the label with indiciais shown as being visible, although such indiciacan be configured to be invisible to the unaided human eye (human-imperceptible) and substantially transparent to visible light.depicts a schematic front view of the label and indiciasupported thereon, but illustrated in such a way to indicate invisibility to the unaided human eye, and substantial transparency to visible light. The indiciais machine-readable. With reference to, thus, the indiciacan include symbols or patternsor both that can be perceived by the sensor (e.g., the cameraor image sensor (or other sensor or detector) on the robot, or the sensor as supported within the augmented reality system), and recognized by the robot. In one aspect, the indiciacan be or can comprise a bar code that is linear or one-dimensional with a series of parallel lines with a variable thickness or spacing to provide a unique code, or a two-dimensional MaxiCode or QR code with a pattern of rectangular or square dots to provide a unique code, etc. In another aspect, the indicia can include symbols, letters, and even arrowsor the like. In another aspect, the indiciacan be passive, as opposed to an active identifier, such as a radio frequency identification (RFID) device or tag.

In addition, the indiciacan be human-imperceptible or invisible or substantially invisible to the unaided human eye, as indicated in. In one aspect, the indiciacan be invisible in the visible spectrum, and thus does not substantially reflect visible light. In one aspect, the indiciacan be transparent or translucent, so that visible light substantially passes therethrough. In one aspect, the indiciacan be ultraviolet and can reflect ultraviolet light, either from the surrounding light sources, such as the sun, or as illuminated from the UV light sourceof the robot. Thus, the indiciacan be visible in the UV spectrum. Therefore, the indiciacan be applied to an object and appear to the human eye as translucent or clear, or transparent to visible light, and thus can be substantially imperceptible to the unaided human eye to maintain the aesthetic appearance of the object. But the indiciacan be visible, such as in the UV spectrum, to the cameraof the robotupon UV light being caused to be incident on the indicia.

The indiciacan be visible outside of the visual spectrum. In one aspect, the indiciacan fluoresce when exposed to light within certain spectrums (e.g., UV light), or be visible with infrared or IR (near, mid and/or far). For example, the object can be marked with indicia that is IR reflective and visible in the IR spectrum; and the robot can have a camera with an image sensor operable in the IR spectrum and an IR light source. In another aspect, the indiciacan be radio frequencies or sound in wavelengths outside of the human audible ranges. For example, the object can be marked with indicia that is RF reflective; and the robot can have a sensor operable in the RF spectrum. As another example, the robot can touch the object with an ultrasonic emitter and receive a characteristic return signal, or emit a sound that causes an internal mechanism to resonate at a given frequency providing identification. Thus, the object can be marked with indicia that emits ultrasonic wavelengths, and the robotor the augmented reality systemcan have a sensor operable in the ultrasonic spectrum. Again, the indiciaand/or the interaction information associated therewith can be translated or interpreted into human-understandable information by the augmented reality systemin order to be displayed to and understood by the human operator.

Referring again to, the object, such as a window of the range, or a window of the microwave, or the outside window, can have a transparent portion. The indiciacan be disposed on the transparent portion, and can be transparent, as described above, so that the indicia does not interfere with the operation of the object or transparent portion thereof. Thus, the indiciaapplied to the windowor transparent portion of the rangeor microwavedoes not impede viewing through the window or transparent portion.

depicts a schematic cross-sectional side view of the label with indiciafor marking an object. In one aspect, the indiciacan be disposed over a finished surfaceof the object. In one aspect, the indiciacan be supported on a removable label or a removable appliqueremovably disposed on or applied to an outermost finished surfaceof the object. For example the indiciacan be printed on the label or the applique. The appliquecan adhere to the finished surfaceof the object, such as with a releasable or permanent adhesive. Thus, the indiciacan be applied to the objectsin an existing space, and after manufacture and transportation of the objects to the space. The indiciaor labelscan be applied to an existing space and the objects therein.

depicts a schematic cross-sectional side view of the label or indiciafor marking an object. In one aspect, the indiciacan be disposed under an outermost finished surfaceof the object. For example, the indiciacan be disposed directly on the object, such as by printing or painting, and then the outermost finished surface, such as a clear coat or the like, can be applied over the indicia. Thus, the indiciacan be or can comprise a laminate with an outer layerdisposed over the indicia. In one aspect, the outer layercan be transparent or translucent to visible light, such as a clear coat. In another aspect, the outer layercan be opaque to visible light. In another aspect, the outer layercan be transparent or translucent to UV light. Thus, the indiciacan be visible to the robot or the augmented reality system or camera or image sensor thereof, even when covered by the finished outer layerof the object. Thus, the aesthetic appearance of the object can be maintained. In addition, the indiciacan be applied to the objects during manufacture and at the place of manufacture.

In another example, the indiciacan be applied directly to the object without the use of a carrier medium (e.g., such as a label). In one example, the indiciacan be applied by printing onto the object using a suitable ink. Many different types of inks are available, such as those that fluoresce at specific wavelengths.

In another aspect, the indiciacan be applied or painted on during production of the object. In one aspect, the entire object can be painted with fluorescent paint or covering, and the indiciacan be formed by a part that is unpainted or uncovered. In another aspect, the indiciaor object can comprise paint that becomes capable fluorescence when another chemical is applied to the paint.

In another aspect, fluorescent markers can be used. For example, the fluorescent markers can include fluorescently doped silicas and sol-gels, hydrophilic polymers (hydrogels), hydrophobic organic polymers, semiconducting polymer dots, quantum dots, carbon dots, other carbonaceous nanomaterials, upconversion NPs, noble metal NPs (mainly gold and silver), various other nanomaterials, and dendrimers.

In another aspect, the indiciacan be part of an attached piece. For example, the indiciacan be on a handle or trim that is mechanically fastened or chemically adhered to the object. As another example, the indiciacan be molded into a polymer of the handle, trim or other part.

depicts a schematic front view of an object, namely a refrigerator, in accordance with an example. With reference to, and with continued reference to, the objectcan have an interface, such as one or more handlesor one or more buttons, knobs or actuators. In addition, the objectcan have multiple labels or indicia. In one aspect, the indiciacan identify the objectas a refrigerator. In another aspect, other indiciaandcan identify a location of an interface, or the handleor the actuator, respectively. Thus, the objectcan have multiple different indicia,andon different parts of a single object, identifying the different partsandof the single object and information on how to interact with them. As described above, the indiciacan include an arrow() to indicate location of the interface. The indiciaandcan be disposed or located adjacent to the interface, or handleand actuator, of the object so that the arrowis proximal the interface. In addition, the indicia can include other information, or can facilitate access to other information (interaction information), such as a direction of operation (e.g. outwardly, inwardly, downward, to the right or to the left) of the interface or handle, or magnitude or limit of force required to operate the interface. For example, the indiciaincan indicate, or facilitate access to information that indicates, that the objectis the refrigerator, identify the location of the handle, that a force of 10-20 lbs. is required to open the door, that the force applied should not exceed 30 lbs., that the door opens to the right or the maximum rotation of the door is not to exceed 120 degrees, or both. Such information can reduce the calculation time and computational power required by the robot to complete a task, or the time and effort required by the human operator remotely controlling the robotto complete the task, and can reduce the need to learn certain aspects useful or required for interacting with the object(s) as compared to conventional robot control and machine learning systems.

The indicia,and, can be located or disposed inside a perimeteror lateral perimeter of the object. In another aspect, the indiciacan be disposed at the perimeteror lateral perimeter of the objectto help identify the boundaries of the object. The indiciacan substantially circumscribe the perimeter of the object.

depicts a schematic front view of an object, namely a doorway, with an interface, namely the openingtherethrough, in accordance with an example. With reference to, and with continued reference to, the indiciacan be disposed on a wallto identify the wall or the doorway, and to indicate the interface or openingthereof and other pertinent information. The indiciacan be located adjacent to or proximal to the openingat a perimeter thereof.

Similarly,depicts a schematic front view of an object, namely a door, with an interface, namely a doorknob, in accordance with an example. With reference to, and with continued reference to, the indiciacan be disposed on the doorto identify the door, and to indicate the interface or the doorknobthereof and other pertinent information. To do so, the indiciacan be located adjacent to or proximal to the interface or the doorknob, respectively. Again, the indiciacan also indicate, or can facilitate access to information that can indicate, the direction of operation of the object or door (e.g. inwardly, outwardly, to the left, to the right), the direction of operation of the interface or doorknob (e.g. clockwise or counter-clockwise), the force required to open the door or the doorknob, etc.

depicts a schematic view of a look-up tableor library of information corresponding to the indicia. With reference to, and with continued reference to, the look-up tableor library of information can be stored in the computer database, accessible by at least one of the robotor the augmented reality system via one or more ways as is known in the art. As indicated above, the indiciacan identify or facilitate identification of the object, a location of an interface of the object, indicate or facilitate indication of the physical boundary or barrier thereof, and can comprise or facilitate access to information pertaining to robot interaction (or interaction information or instructions) with the object.