Computer vision and artificial intelligence method to optimize overlay placement in extended reality

This application is a National stage of International Application No. PCT/IB2020/057085, filed Jul. 27, 2020, which is hereby incorporated by reference. This application is related to International Application No. PCT/IB2020/057084, titled PRIVATE SHARING OF LOCATION DATA FOR EXTENDED REALITY RENDERING, filed Jul. 27, 2020, which is hereby incorporated by reference.

Embodiments of the invention relate to the field of extended reality rendering; and more specifically, to a system and process for optimizing overlay placement in connection with extended reality rendering.

Augmented reality (AR) augments the real world and the physical objects in the real world by overlaying virtual content. This virtual content is often produced digitally and may incorporate sound, graphics, and video. For example, a shopper wearing augmented reality glasses while shopping in a supermarket might see nutritional information for each object as they place it in their shopping cart. The glasses augment reality with information.

Virtual reality (VR) uses digital technology to create an entirely simulated environment. Unlike AR, which augments reality, VR immerses users inside an entirely simulated experience. In a fully VR experience, all visuals and sounds are produced digitally and do not include input from the user's actual physical environment. For example, VR may be integrated into manufacturing where trainees practice building machinery in a virtual reality before starting on the real production line.

Mixed reality (MR) combines elements of both AR and VR. In the same vein as AR, MR environments overlay digital effects on top of the user's physical environment. MR also integrates additional, richer information about the user's physical environment such as depth, dimensionality, and surface textures. In MR environments, the end user experience more closely resembles the real world. As an example, consider two users hitting a MR tennis ball on a real-world tennis court. MR incorporates information about the hardness of the surface (grass versus clay), the direction and force the racket struck the ball, and the players' height. Augmented reality and mixed reality are often used to refer to the same idea. As used herein, “augmented reality” also refers to mixed reality.

Extended reality (XR) is an umbrella term referring to all real-and-virtual combined environments, such as AR, VR and MR. XR refers to a wide variety and vast number of levels in the reality-virtuality continuum of the perceived environment, consolidating AR, VR, MR and other types of environments (e.g., augmented virtuality, mediated reality, etc.) under one term.

An XR device is the device used as an interface for the user to perceive both virtual and/or real content in the context of extended reality. An XR device typically has a display that may be opaque and displays both the environment (real or virtual) and virtual content together (i.e., video see-through) or overlay virtual content through a semi-transparent display (optical see-through). The XR device may acquire information about the environment through the use of sensors (typically cameras and inertial sensors) to map the environment while simultaneously tracking the device's location within the environment.

Object recognition in extended reality is mostly used to detect real world objects and for triggering the digital content. For example, the consumer would look at a fashion magazine with augmented reality glasses and a video of a catwalk event would play in a video instantly. Note that sound, smell, and touch are also considered objects subject to object recognition. For example, a diaper ad could be displayed as the sound and perhaps when the mood of a crying baby is detected. Mood could be deducted from machine learning applied to sound data.

In one embodiment, a method of an electronic device in an edge cloud of a mobile network supports extended reality overlay placement for an object having a location in the real world. The method includes receiving a request from an application of a user equipment, the request including an object identifier for an object that is a target of an extended reality overlay, and determining a static motion probability field (S-MPF) for the object identifier.

In another embodiment, an electronic device in an edge cloud of the mobile network supports extended reality overlay placement for an object in the real world. The electronic device includes a non-transitory machine-readable medium having stored therein an extended reality overlay service, and a processor coupled to the non-transitory machine-readable medium. The processor executes the extended reality overlay service. The extended reality overlay service receives a request from an application of a user equipment. The request includes an object identifier for an object that is a target of an extended reality overlay, and determines a static motion probability field (S-MPF) for the object identifier.

The following description describes methods and apparatus for improving the operation of extended reality applications by assisting these applications in predicting the movement of objects. The methods and apparatus position motion probability fields (MPFs) in edge cloud resources and facilitate the retrieval and usage of these MPFs to enable the extended reality applications to improve rendering of digital overlays at the extended reality devices. In the following description, numerous specific details such as logic implementations, opcodes, means to specify operands, resource partitioning/sharing/duplication implementations, types and interrelationships of system components, and logic partitioning/integration choices are set forth in order to provide a more thorough understanding of the present invention. It will be appreciated, however, by one skilled in the art that the invention may be practiced without such specific details. In other instances, control structures, gate level circuits and full software instruction sequences have not been shown in detail in order not to obscure the invention. Those of ordinary skill in the art, with the included descriptions, will be able to implement appropriate functionality without undue experimentation.

References in the specification to “one embodiment,” “an embodiment,” “an example embodiment,” etc., indicate that the embodiment described may include a particular feature, structure, or characteristic, but every embodiment may not necessarily include the particular feature, structure, or characteristic. Moreover, such phrases are not necessarily referring to the same embodiment. Further, when a particular feature, structure, or characteristic is described in connection with an embodiment, it is submitted that it is within the knowledge of one skilled in the art to affect such feature, structure, or characteristic in connection with other embodiments whether or not explicitly described.

Bracketed text and blocks with dashed borders (e.g., large dashes, small dashes, dot-dash, and dots) may be used herein to illustrate optional operations that add additional features to embodiments of the invention. However, such notation should not be taken to mean that these are the only options or optional operations, and/or that blocks with solid borders are not optional in certain embodiments of the invention.

In the following description and claims, the terms “coupled” and “connected,” along with their derivatives, may be used. It should be understood that these terms are not intended as synonyms for each other. “Coupled” is used to indicate that two or more elements, which may or may not be in direct physical or electrical contact with each other, co-operate or interact with each other. “Connected” is used to indicate the establishment of communication between two or more elements that are coupled with each other.

An electronic device stores and transmits (internally and/or with other electronic devices over a network) code (which is composed of software instructions and which is sometimes referred to as computer program code or a computer program) and/or data using machine-readable media (also called computer-readable media), such as machine-readable storage media (e.g., magnetic disks, optical disks, solid state drives, read only memory (ROM), flash memory devices, phase change memory) and machine-readable transmission media (also called a carrier) (e.g., electrical, optical, radio, acoustical or other form of propagated signals—such as carrier waves, infrared signals). Thus, an electronic device (e.g., a computer) includes hardware and software, such as a set of one or more processors (e.g., wherein a processor is a microprocessor, controller, microcontroller, central processing unit, digital signal processor, application specific integrated circuit, field programmable gate array, other electronic circuitry, a combination of one or more of the preceding) coupled to one or more machine-readable storage media to store code for execution on the set of processors and/or to store data. For instance, an electronic device may include non-volatile memory containing the code since the non-volatile memory can persist code/data even when the electronic device is turned off (when power is removed), and while the electronic device is turned on that part of the code that is to be executed by the processor(s) of that electronic device is typically copied from the slower non-volatile memory into volatile memory (e.g., dynamic random access memory (DRAM), static random access memory (SRAM)) of that electronic device. Typical electronic devices also include a set of one or more physical network interface(s) (NI(s)) to establish network connections (to transmit and/or receive code and/or data using propagating signals) with other electronic devices. For example, the set of physical NIs (or the set of physical NI(s) in combination with the set of processors executing code) may perform any formatting, coding, or translating to allow the electronic device to send and receive data whether over a wired and/or a wireless connection. In some embodiments, a physical NI may comprise radio circuitry capable of receiving data from other electronic devices over a wireless connection and/or sending data out to other devices via a wireless connection. This radio circuitry may include transmitter(s), receiver(s), and/or transceiver(s) suitable for radiofrequency communication. The radio circuitry may convert digital data into a radio signal having the appropriate parameters (e.g., frequency, timing, channel, bandwidth, etc.). The radio signal may then be transmitted via antennas to the appropriate recipient(s). In some embodiments, the set of physical NI(s) may comprise network interface controller(s) (NICs), also known as a network interface card, network adapter, or local area network (LAN) adapter. The NIC(s) may facilitate in connecting the electronic device to other electronic devices allowing them to communicate via wire through plugging in a cable to a physical port connected to a NIC. One or more parts of an embodiment of the invention may be implemented using different combinations of software, firmware, and/or hardware.

A network device (ND) is an electronic device that communicatively interconnects other electronic devices on the network (e.g., other network devices, end-user devices). Some network devices are “multiple services network devices” that provide support for multiple networking functions (e.g., routing, bridging, switching, Layer 2 aggregation, session border control, Quality of Service, and/or subscriber management), and/or provide support for multiple application services (e.g., data, voice, and video).

The embodiments present a method and system to use spatial mapping and scene detection paired with computer vision and artificial intelligence to predict the location of visual and other overlays in extended reality. The embodiments utilize an architecture to benefit from improved latency and speed of 5Generation New Radio (5G NR) by the 3Generation Partnership Project (3GPP) to incorporate image, video, audio, and location data into the estimation of dynamic motion probability fields for overlay placement. The embodiments further introduce an architecture that uses 5G NR's improved latency and speed to allow extended reality devices to access static motion probability fields, stored in the edge cloud, to improve placement of visual overlays without using dynamically calculated data. The embodiments also introduce a means to store either the dynamic or static motion probability fields in the edge cloud. The embodiments can include methods to use artificial intelligence to dynamically learn from mistakes in overlay placement.

The embodiments overcome the problems of the existing art. 5G New Radio (NR) will support dozens of new use cases for consumers and enterprise. Supporting augmented reality (AR) and virtual reality (VR), herein referred to collectively as extended reality (XR) is one of 5G NR's key use cases. By laying critical information, insights, and predictions over an end user's field of vision, XR provides value for consumers and enterprise users and is forecast to grow rapidly. Widespread adoption of XR, and unlocking its commercial potential, requires addressing four key technical impediments. The first relates to streamlining XR devices' form factors. Currently, XR devices perform most of their computation and data processing on the XR device. To enable this computation, today's XR devices integrate powerful, battery-draining electronic circuits. XR devices also have large batteries, resulting in physically large and heavy product design. The second challenge is how to streamline these physical forms. To reduce the need for large circuit boards and batteries, XR devices must conduct ever more processing at the edge and/or in the cloud (hereinafter, “the edge cloud”). Thirdly, XR devices must be connected to mobile and/or Wi-Fi networks that are sufficiently fast to stream large amounts of data with minimal latency.

The fourth technical impediment is placing overlays on top of the correct object. While this problem can be addressed by using computer vision and object detection when objects are unique and stationary, it is a tremendous challenge when a scene has multiple, visually similar objects or objects that move. Moving objects could be located within space by having the object stream its location in the form of latitude, longitude, and altitude obtained via global positioning system (GPS). GPS solutions do not work well in indoor environments, but other ways of locating objects in indoor environments are possible. Radio positioning (e.g., Bluetooth beacons) or marker-based positioning technologies can also be used but requires specific infrastructure. Using visual identifiers (e.g., QR codes) can help locate specific objects relative to the camera position, but these are prone to occlusion. Moreover, these data sources are highly sensitive and lack controls to restrict or control accessing the location information that they produce, which has broad privacy implications.

These challenges can be addressed by XR devices having access to high speed, low latency mobile and/or Wi-Fi networks. Prior to the introduction of 5G NR, earlier mobile networks lacked the capability to directly address these four challenges. For instance, 4G Long Term Evolution's (LTE) upload and download speeds are insufficient to stream uncompressed high definition (HD) audio and video. While audio and video compression codecs allow these data to be streamed over 4G LTE, the loss in fidelity poses challenges for computer vision and artificial intelligence (AI). 4G LTE's latency, moreover, is too high to allow XR devices to push most data processing into the edge cloud. As even small changes in an end user's posture affect the placement of visual overlays, any perceptible lag will diminish the end user experience.

The speed and latency standards in 5G NR 3GPP (e.g., Release 15) solve the speed and latency impediments to widespread XR adoption. The embodiments build on these improvements to introduce a solution to the problem of placing visual overlays in an end user's field of vision. The embodiments build on an architecture that incorporates differential privacy to allow for time-delimited access to location information and location information purging. In some cases, even with 5G NR's low latency, the architectures of the mobile network or Wi-Fi network can be too slow to correctly place visual overlays. This problem is particularly acute when objects move quickly (for example, cars on a freeway) or with inconsistent patterns of movement (for example, athletes). The embodiments introduce an architecture and method to process semantic data (e.g., visual, auditory, location, and sensory data) in the edge cloud, and dynamically predict where an overlay should be placed.

is a diagram of a moving object that illustrates issues related to the extended reality overlays. The diagram shows the range for a visual overlay for a vehicle traveling at 108 kilometers an hour at different levels of latency. The vehicle in the top panel has an extended reality (XR) overlay situated directly above the vehicle. The XR application may consider this placement to be an ideal placement of an XR overlay for a vehicle (e.g., in this case the car has an ObjectID=A123456) that is to be rendered by an XR application when the user of the XR device is viewing the vehicle. While ideal placement might depend upon the object, XR application design, the end user, and the use case, the embodiments can accommodate any preferred or ideal position for the XR overlay. The bottom panel ofpanel shows the uncertainty range over where the ideal placement would be, given minimal changes in network latency. In the example, if there is 100 milliseconds (ms) of latency, then the XR overlay may be 3 meters off the ideal placement. If there is 200 ms of latency, then the displacement may be 6 meters. Similarly, the displacement is 9 meters and 12 meters for latency of 300 ms and 400 ms, respectively.

The embodiments provide a method and apparatus to solve the problem of correctly placing visual overlays in XR renderings by XR applications on XR devices. The embodiments improve overlay placement by integrating semantic understanding and scene detection into artificial intelligence (AI) and machine learning workflows. The embodiments assign a motion probability field (MPF) to each object. Each object is assigned a unique identifier (e.g., an ObjectID). Each MPF describes the probability a given object will move next over a space. An MPF integrates contextual information about the object (e.g., maximum speed, range of motion, physical size, and similar information). This type of MPF does not incorporate real-time conditions, thus, this MPF is static and used to reduce the potential range within a field of view for overlay placement. In the architecture of the embodiments, object placement algorithms can access an MPF as the algorithms run in the edge cloud.

The embodiments utilize an architecture that benefits from the next generation of low latency and high-speed networks (e.g. 5G NR and/or Wi-Fi) to localize physical items (hereinafter, “objects”) within a defined space in real-time. These physical items, i.e., objects, are associated with electronic devices capable of communicating with a mobile network to share information with other electronic devices connected to the mobile network. For example, a taxi can either have an embedded device or can associate with a mobile handset device of a driver. ‘Objects’ as discussed herein encompass both the physical object, e.g., a taxi, and the associated electronic device that enables its connectivity with the mobile network.

is a diagram of another example case illustrating the use of an MPF for predicting where to render an XR overlay. The example is a bird's eye view of a street scene where either a dynamic MPF (discussed below) or static MPF for a vehicle (ObjectID=A123456) is applied. The example shows the probability the vehicle will move forward (or potentially forward and to the side) more than it will move backwards. The MPF can correlate to a space around the object into which the object can possibly move. The area of the space around the object is divided into a set of three dimensional spaces.

The architecture can utilize a global shared real world mapping system. The real world mapping system can divide the real world into two or three dimensional spaces. In one embodiment, a cubic 1-meter grid over the surface of the planet is determined and each cubic meter is assigned a unique alphanumeric identifier referred to as a Cube Identifier (a CubeID) or similar location identifier (LocID). However, in other embodiments, any size, shape, or configuration of a real-world mapping can be utilized. The example of a cubic representation is described herein by way of example and not limitation.

Similarly, the use of CubeIDs or LocIDs are described herein as a mechanism to locate objects, however, one skilled in the art would understand that the principles, processes, and structures described herein are fully compatible with other demarcations of location such as latitude, longitude, and altitude. The CubeID is one type of LocID, which uniquely identifies a location (e.g., a cube) within a spatial mapping system that is correlated with the real world. Any type of location identifier can be used with the embodiments; the example of the CubeID is used herein by way of example and not limitation. Within the edge cloud, each object is assigned an alphanumeric ObjectID that uniquely identifies it. Each object then uploads its location and other data into the edge cloud, in response to changes in that data, at regular intervals, or in response to other events. In some embodiments, to improve efficiency, all location information is matched to the closest CubeID.

Returning to the illustrated example, the MPF correlates with the grid of cubes or similar spatial mapping to determine a probability that the object can move into nearby locations (e.g., cubes) in the subsequent given time frame. In this example, the object cannot go immediately to the right or backward, since it is currently moving toward the intersection. The MPF indicates probabilities that the object (i.e., the vehicle) will be in the illustrated locations (i.e., cubes) at the next timing interval. The grey shaded cubes indicate higher probability and the unshaded cubes lower probability.

Benefitting from the low latency and high speed of 5G NR, the architecture of the embodiments also allows for the dynamic estimation of MPFs (D-MPF). Unlike a static MPF, a D-MPF applies semantic understanding to real-time data (e.g., live audio and video and weather data) uploaded into an edge cloud environment. After estimation, D-MPFs can be integrated into AI and machine learning models to improve overlay placement prediction. Further, the low latency and high speed of 5G NR allow XR devices to access a dynamically predicted range of motion for each object with a visual overlay. After processing real-time semantic and scenario data uploaded from the XR device to the edge cloud, the embodiments assign each object with an overlay with a temporally and situationally dynamic overlay range. Finally, these fields are anonymized and stored in the edge cloud to improve estimation efficiency.

is a diagram of another example case where a vehicle is moving at 108 kilometers an hour. The diagram shows the value of dynamically processing semantic data to constrain the potential range of a visual overlay. The example shows how real-time contextual information can be used to improve placement of overlays. In this scenario, the presence of the second vehicle in front of the target vehicle narrows down the possible range where an overlay could be placed. This information is integrated into the real-time dynamic motion probability field (D-MPF) as described further herein. Any algorithms can be used to estimate an object's potential range of motion. These algorithms can utilize any available source of location information and object state information such as acceleration, range, and physical limitations of the object. To that end, the embodiments provide a method for gathering this contextual data and an architecture for processing the contextual data. The embodiments can be utilized with any static-MPF (S-MPF), D-MPF or similar structures. Similarly, the embodiments can work with any algorithms intended to estimate, generate, update, or otherwise update or maintain an S-MPF or a D-MPF using any available information.

Similarly, the embodiments can be utilized in combination with any computer vision, artificial intelligence, or other applications for semantic understanding of physical environments. Such applications can be related to measuring occlusion, object detection, and semantic mapping for a variety of use cases, including mapping indoor small cells and digital advertising in XR. On their own, these applications are insufficient for the use case described herein, as they neither integrate with a privacy-preserving architecture nor are they suitable for predicting overlays in XR. These applications, moreover, do not use 5G NR's latency and speed to allow for both static and dynamic mapping of probability fields.

The following features are missing from these applications: the ability to incorporate image, video, spatial, audio, and location data for purposes of improving visual overlay placement for XR purposes; the ability to process image, video, spatial, audio, and location data paired with spatial mapping and scene detection to derive static and/or dynamic motion probability fields; the ability to predict ideal location for visual overlay placement by the movement range of an object and in the dynamic case other objects in real time in two dimensions which are projected from a three-dimensional space to a two-dimensional plane based on perspective; integrating object detection for purposes of deriving a dynamic motion probability field, whereby the movement patterns and other contextual information is included and processed in the edge cloud; integrating pose (location and orientation) into the visual display in overlay placement for XR/VR, such that the predicted overlay location can be viewed in certain dimensions, and location can be observed differently from different perspectives; the ability to store predicted dynamic and/or static motion probability fields in the edge cloud for future use.

The embodiments address the deficiencies in the art. In this disclosure, we propose a method to estimate static motion probability fields (S-MPFs) and dynamic motion probability fields (D-MPFs) for an object in an XR application. In both cases, an S-MPF and/or D-MPF is used to improve the placement of overlays by an XR application. In the S-MPF case, the S-MPF is estimated using static features of the environment, such as building location information and other infrastructure location information. No real-time environmental information is used for static MPF estimation. In the D-MPF case, the D-MPF integrates information of a dynamic environment streamed real time from the XR device. After processing the real time audio, video, and other data types in the edge cloud, the resulting D-MPF updates in real time. This architecture uses 5G NR and/or Wi-Fi to minimize errors in overlay placement.

is a diagram of one embodiment of a motion probability field applied to an example vehicle in a street environment. When scanning a scene for purposes of placing a visual overlay in XR, the optimal overlay location estimation predicts a location a use case-defined distance away from the relevant object. The ideal location for an XR overlay is relative to an object, in terms of how far above, below, and how far away, and is likely to vary depending upon the use case, end user preference, and adaptation to factors such as occlusion, bright light, and similar environmental factors. Any method and system can be used for defining and implementing XR overlay placement relative to the object and the associated preferences and configuration.

In the architecture of the embodiments, every object is assigned an ObjectID that is associated with it, and the XR application is able to access a motion probability field (MPF) that is stored within the edge cloud for that object. Depending upon the use case, a global, national, or regional MPF is produced and geospatially relevant subsets from the master MPF are stored within the edge cloud. For purposes of network, machine learning computational efficiency and privacy, an object only has access to the subset of the MPF that is within an arbitrary distance (e.g., in meters) from the object. This arbitrary distance parameter will depend upon the use case, speed of the object, and network connectivity. Any method and system can be used for determining an optimal method for setting the arbitrary distance parameter and associated parameters.

An MPF can be defined, broadly, as a geospatially-defined vector field of penalty or reward scores for movement. In non-technical terms, this means each pairwise combination of geospatial coordinate (latitude, longitude, and altitude) or similar location information (e.g., a LocID) and object type is assigned a likelihood of travel ranging from 0 to 1 or using a similar range. When 0, there is no probability a given object can move to that geospatial coordinate or location. For example, a vehicle cannot move hundreds of meters vertically above the street. Thus, locations above a certain height can be given a 0. When 1, there is a 100% probability a given object will move along the predicted range.

In the illustrated example, the slanted lines indicate different probabilities that the vehicle with ObjectID A123456 will be at these locations in the next time frame. In the example, the vehicle can be traveling at 100 kilometers an hour. The denser the lines, the more likely the MPF indicates it is that the vehicle will travel to that location in the next time frame. In the case where the MPF is dynamic, the D-MPF also integrates information about pedestrians and their current location into its estimation. The static case would not integrate such real time information and only uses information available in a spatial map or model as well as motion dynamics of the object receiving the overlay. Any method or process can be used to generate or predict an MPF (e.g., a S-MPF or D-MPF). XR application developers, mobile network operators, or others can specify preferred processes and methods for generating the MPFs. Example processes are described herein. Any combination or set of specific features can be included in the model to estimate either a D- or S-MPF. Once calculated, the resulting D-MPF or S-MPF can be included within the metadata associated with a particular environment XR and stored in the edge cloud.

The embodiments provide numerous advantages over the art. The embodiments improve end user experience in XR applications by using the low latency and speed of 5G NR to improve visual overlays. This improves experiences for XR application developers, end users, and XR device manufacturers. The embodiments provide a method to improve overlay placement for XR applications based on using the low latency of new generation networks, such as 5G NR to facilitate objects (e.g., Internet of Things (IoT)) sharing their location information in real time, storing location information using differential privacy such that data are only retained and shared according to end user preferences, assigning a unique identifier to each request for access to data retained within the edge cloud such that access is revoked once the transaction completes, measuring physical location in three dimensions such that overlay graphics can be correctly rendered on the XR device, purging data from the edge cloud such that no data with implications for privacy are retained longer than strictly necessary. The architecture of the embodiments has the following advantages: (1) speed, wherein 5G NR in combination with the edge cloud allows for XR devices to access motion probability fields stored in the edge cloud in the static case, and upload visual, audio, spatial, and location data in the dynamic case; (2) scalability, wherein the architecture is scalable since it is designed to operate on the edge cloud; (3) flexibility, wherein the architecture is flexible since any type of network connected XR device can access MPFs stored on the edge cloud and upload data for processing for D-MPFs; (4) context, where by incorporating location data information into spatial mapping and rendering, the architecture disclosed facilitates customized XR overlays for each end user; and (5) dimensionality, where by incorporating all three dimensions (height, width, and depth) into the object-geospatial location pairing, the embodiment supports three-dimensional rendering in XR.

is a diagram of one embodiment of the architecture of the edge cloud based XR overlay service. This diagram illustrates how an XR application on an XR deviceaccesses a motion probability field, for example, a static-motion probability field (S-MPF) and/or a dynamic motion probability field (D-MPF) stored in the edge cloudor a databaseassociated with the edge cloud. One such way an XR device can access a D-MPF or S-MPF is by generating a RequestID. Once a RequestID is generated, it contains information about the location of the XR device, for example geo-spatial coordinates. This information then allows the XR deviceto receive the relevant D-MPF or S-MPF from the edge cloudand/or database. The XR application at the XR devicecan then utilize the MPF in support of determining how to place visual overlays using any algorithm or placement process.

Prior to the architecture being utilized to provide an MPF, the objects connected with the mobile network and the edge cloudreport location information as well as location information retention limits. Each object connected to the mobile network of the edge cloudis assigned an ObjectID and broadcasts its location information to the edge cloud. The ObjectID can be assigned, generated, reset, or regenerated by the object, by the edge cloud, or a resource (e.g., database system) associated with the edge cloud. In some embodiments, the objects can share their location data at varying times. Objects can share location information at any frequency. For example, objects can share location information responsive to changes in location, at fixed intervals, continuously, or any variation or combination thereof. Instead, users or administrators associated with the objects can set the frequency and in some case can manually toggle location sharing on and off (e.g., via a graphical user interface (GUI) on the device or an application associated with the electronic device of the object). Once enabled, location data can be sent as a message, reported, streamed, pushed/pulled, or similarly transmitted via 5G NR, Wi-Fi, or technologies of earlier mobile network generations to the nearest edge cloud environment. The location information can be shared using any protocol or format. In one example embodiment, location information is shared in the form of an array containing the ObjectID, location (either as latitude, longitude and altitude, CubeID or other type of LocID), and any other fields that may be required to share additional information.

The location data arrays are processed in the edge cloudand retained in a database. If location data is shared in a raw format (e.g., latitude and longitude), these datapoints are assigned to their nearest LocID and the raw data are purged. The collected location information can be utilized at the edge cloudto update MPFs, in particular, D-MPFs. The location information provides information about the environment of the XR deviceand any object that is to have an XR overlay associated with it. As an object moves in space, new options for further movement become possible based on whether they are blocked by terrain, other objects, structures, or other obstacles. The continuously updating of local object location information enables the MPFs to accurately model the probabilities of the movement of the associated objects.

The edge cloudis a set of electronic devices and resources positioned in proximity to base stations, cells, towers, or similar transmission component in the radio access network (RAN) of the 5G NR mobile network, similar mobile network technologies, earlier mobile network technologies and any combination therewith. The edge cloud services and devices are administered by a mobile network operator (MNO) or similar entity. For sake of clarity and conciseness, the edge cloudservices and devices are represented as a single point of contact in the illustration. However, the edge cloudservices and devices are distributed over a wide area of a mobile network. Transitioning between connected edge cloud devices as an object or XR devicemoves through the area of coverage of the mobile network is transparent for purposes of the XR overlay service and is handled by the handover process of the mobile network.

Similarly, the database systemin which ObjectID and location information are stored can be co-located with edge cloud devices or resources or can have a low latency connection with the edge cloud devices or resources. The database systemcan have any format or organization such as an associative or object oriented database system or other type of database system that is capable of storing and associating ObjectIDs with the appropriate location information as it is reported by the objects to the edge cloud. Likewise, the MPFs stored in the databasecan be stored in any format or structure therein according to the organization and operation of the database. The databasecan be distributed over any number of locations and devices to ensure a low latency availability to the XR overlay services of the edge cloud.

The architecture of the embodiments enables applications, websites, and other services and tools to access MPFs retained in the edge cloudand databaseand utilize in their operations. Applications can be executed on any electronic device (e.g., on an XR device) that uses the mobile network to transmit requests for MPFs associated with objects to servers controlled by the MNO or the application or jointly administered with the application. In some embodiments, the applications (e.g., XR applications) running on an XR deviceoperate in conjunction with services and functions (e.g., servers) that execute in the edge cloudsuch that these services and functions are provided by any combination of the developer of the application, which is deployed to the edge cloud, and services and functions provided by the MNO, for example as part of XR overlay or other services. The XR overlay services, as used herein, refers to any combination of MNO and developer functions and services deployed to the edge cloudto support the functions described herein. An XR device, as used herein, refers to any electronic device that is utilized to render XR via a display, haptic feedback, or other output device and which can include cameras, sensors, positional devices, or other input devices to collect information about the real world around the electronic device.

The XR applications (e.g., at an XR device) can access specific MPFs associated with specific objects by generating a RequestID containing information about the physical location of the XR device. The XR deviceuploads the RequestID into the edge cloud. The RequestID is used to identify the relevant MPF or forwarded to a database to access the relevant MPF (e.g., a S-MPF or D-MPF stored within the database). The requested MPF is returned to the edge cloudby the databaseor retrieved from a local cache or electronic device of the edge cloud. The MPF is then returned by the edge cloudto the XR device.

The RequestID is used as part of a lookup process by the edge cloudor database. An XR application or related services or functions can send a RequestID that identifies the XR application or associated XR device to the XR overlay service, which includes a lookup or similar service or in other embodiments the lookup process or service is separate from the XR overlay service. The lookup process can identify MPFs that are associated with an ObjectID specified in the RequestID as well as MPFs that are relevant to the location of the XR device.

After the lookup process, the matching MPFs are returned to the XR devicevia the XR overlay services of the mobile network.

Each RequestID can includes specific fields including information about the application requesting the data, as well as, the ObjectID, the billing type for data transfers, the time limit (e.g., in hours) for data retention, a checksum, and similar data. An example of the format of the RequestID is shown below in Table I.

In the example RequestID, the first field (e.g., seven characters) uniquely identifies the application associated with or that generated the RequestID, the billing type indicates who pays for the data transfers, 6 Digit Prefix can be a grouping or type for the object, padding, or similar information, the ObjectID uniquely identifies the object whose MPF is to be retrieved, the time limit (e.g., in hours) is the amount of time the data related to the RequestID can be retained in a database or at the edge cloud, and the checksum is used to validate the RequestID. In this example, the resulting RequestID is ABCDEFG001000001ABC123456700020001, which uniquely identifies the requesting application. All data retained in the edge cloudrelated to this RequestID, and the ability of the XR application and/or deviceto access data in the edge cloud related to this RequestID are purged after the specified time period expires. Other information or formats can be used for RequestIDs. The format described herein above is provided by way of example and illustration and not limitation. Other fields and information can be included, and other types of data structures can be used to organize this data.