Systems and methods for receive-side customization of presentation of mixed media data

There has been an increasing reliance on mixed media applications, such as video conference applications and video chat applications, for communication between individuals and among groups of participants. Continued improvements to the user experience during operation of these applications are desirable.

Mixed media applications, such as video conference applications and video chat applications, often combine video, audio, images (such as profile pictures) and computer-generated images. Over time, these applications have increasingly been employed for communication between individuals and among groups. Many available applications focus on the transmit path, allowing a sender to replace a video with an avatar, for example. However, a technical problem is presented, in that receive-side users do not have the ability to manage the receive-side of the mixed media presentation.

Some solutions provide an ability on the receive-side to enable/disable all data streams, replacing them with profile pictures. However, this all-or-nothing solution can result in a negative receive-side user experience, cutting off valuable video information from all participants just to address a single participant. It is desirable to provide a more inclusive user experience. Receive-side users may have different sensitivities and/or sensory overload thresholds and could benefit from an ability to enable/disable individual streams or replace individual streams with avatars.

Aspects of the present disclosure provide a technical solution to this technical problem and other related enhancements, in the form of systems and methods for receive-side customization of presentation of mixed media data. Provided embodiments advantageously focus on the receive path where each user/participant can enable/disable and/or customize individual incoming streams. Embodiments enable all participating users (not just a host user) to customize their receive-side mixed media presentation with preferences for individual video streams. Embodiments enable an organization to restrict available avatars to a pre-approved list. Additionally, embodiments can be implemented in an existing server.

Embodiments may be detected by reviewing mixed media application documentation to look for instructions for disabling individual participants' video streams and enabling customized avatars, per participant, and by viewing options available during operation of a video conference application (including Web and native Operating System applications). The feature of enabling an organization to restrict available avatars to a pre-approved list can also be detected in the product literature or operation.

As used herein, the terms “processor unit,” “processing circuitry,” “processing unit,” or “processor” may refer to any device or portion of a device that processes electronic data from registers and/or memory to transform that electronic data into other electronic data that may be stored in registers and/or memory. A processor unit may be a system-on-a-chip (SOC), and/or include one or more digital signal processors (DSPs), application-specific integrated circuits (ASICs), central processing units (CPUs), graphics processing units (GPUs), general-purpose GPUs (GPGPUs), accelerated processing units (APUs), field-programmable gate arrays (FPGAs), neural network processing units (NPUs), data processor units (DPUs), accelerators (e.g., graphics accelerator, compression accelerator, artificial intelligence accelerator), controller cryptoprocessors (specialized processors that execute cryptographic algorithms within hardware), server processors, controllers, or any other suitable type of processor units. As such, the processor unit can be referred to as an XPU (or xPU).

As used herein, the term “module” may refer to any hardware, software, firmware, electronic control component, processing logic, and/or processor device, individually or in any combination. In various embodiments, a module is one or more of: an application specific integrated circuit (ASIC), a field-programmable gate-array (FPGA), an electronic circuit, a computer system comprising a processor (shared, dedicated, or group) and memory that executes one or more software or firmware programs, a combinational logic circuit, and/or other suitable components that provide the functionality attributed to the module.

For the sake of brevity, conventional techniques related to signal processing, data transmission, signaling, control, artificial intelligence (AI) models, machine learning models, image analysis, and other functional aspects of the systems (and the individual operating components of the systems) may not be described in detail herein. Furthermore, the connecting lines shown in the various figures contained herein are intended to represent example functional relationships and/or physical couplings between the various elements. It should be noted that many alternative or additional functional relationships or physical connections may be present in an embodiment of the present disclosure.

Embodiments of the present disclosure may be described herein in terms of functional and/or logical block components that perform different actions or tasks. It should be appreciated that such block components may be realized by any number of hardware, software, and/or firmware components configured to perform the specified functions. For example, an embodiment of the present disclosure may employ various integrated circuit components, e.g., memory elements, digital signal processing elements, logic elements, look-up tables, or the like, which may carry out a variety of functions under the control of one or more microprocessors or other control devices.

As mentioned, media applications that consume and process audio, video, and images, such as, video conference applications and video chat applications, can provide an improved user experience when using embodiments described herein. The technologically enhanced systems and methods for receive-side customization of presentation of mixed media data are described in more detail in connection with the figures below.

1 FIG. 100 1 106 106 is a schematic diagram depicting an example environmentin which embodiments may be implemented. A plurality of user devices, represented as user deviceto user device N (N>1), may be communicatively coupled to one another via a network. A variety of different transmission protocols and architectures may be utilized in the networkand in support of the bidirectional communication between the user devices, such as, but not limited to, wireless, WIFI, 2G, 3G, 4G, 5G, etc.

108 1 108 2 110 1 110 2 112 1 112 2 114 1 114 2 116 1 116 2 118 1 118 2 106 The user devices are capable of transmitting and receiving media data comprising at least audio, video, and images; and, the user devices are configured, such as, with an installed device application, to run at least one media application that consumes and processes audio, video, and images. Some example media applications include video conference applications and video chat applications. Accordingly, the user devices generally include at least a camera (-,-), a speaker-,-, a microphone-,-, a display-,-, a user input device-,-(e.g., a keyboard or touch screen), and a communication system-,-that supports communication via the network.

Although the user devices are drawn alike, in practice, they can be any combination of available computing devices that meet the above criteria. For example, the user devices can comprise any combination of laptop computers, desktop computers, kiosks, and cellular phones.

106 120 120 132 134 136 124 126 128 130 120 105 122 In various aspects of the disclosure, the networkincludes a cloud server. The servermay include at least one or more of an avatar database (ADB), avatar video-based generator (AVG), avatar audio-based generator (AAG), demultiplexing system (demux), video encoding system, encoding system, and communication system. Other components, not shown to avoid clutter, may also be included in the server. A system for receive-side customization of presentation of mixed media data, shown generally as system, includes control circuit.

122 120 105 105 In various embodiments, the control circuitis communicatively coupled to systems and components within the server, as illustrated. In operation, the systemmay receive mixed media data signals Rx, receive user preferences, process the data signals Rx with the user preferences, as described herein, generate therefrom a data output that is receive-side customized, and transmit the data output to the user's device. As may be appreciated, for multiple users, the systemis to concurrently perform these operations such that every user is receiving respective receive-side customized data.

1 FIG. 122 152 156 150 156 158 152 150 152 150 120 150 105 152 150 152 In various embodiments, as shown in, the control circuitis realized as an enhanced computer system, comprising computer readable storage device or media, memory, for storage of instructions, algorithms, and/or programs, such as programand a plurality of preprogrammed thresholds and parameters, the processorto execute the program, and input/output interface (I/O). The computer readable storage device or media, memory, may include volatile and nonvolatile storage in read-only memory (ROM), random-access memory (RAM), and keep-alive memory (KAM), for example. KAM is a persistent or non-volatile memory that may be used to store various operating variables while the processoris powered down. The memorymay be implemented using any of several known memory devices such as PROMs (programmable read-only memory), EPROMs (electrically PROM), EEPROMs (electrically erasable PROM), flash memory, or any other electric, magnetic, optical, or combination memory devices capable of storing data, some of which represent executable instructions, used by the processorin other aspects of serveroperation. In various embodiments, processoris to implement the system. The memorymay also be utilized by the processorto cache data, to temporarily store results of comparisons and analyses, and the like. Information in the memorymay be organized and/or imported from an external source during an initialization or installment operation in a method; it may also be programmed via a user I/O interface.

158 150 122 122 158 158 150 158 The input/output interface (I/O)may be operationally coupled to the processorvia a bus and enables intra-circuitcommunication as well as extra-circuitcommunication. The input/output interface (I/O)may include one or more wired and/or wireless network interfaces and can be implemented using any suitable method and apparatus. In various embodiments, the input/output interface (I/O)includes the hardware and software to support one or more communication protocols for wireless communication between the processorand external sources, such as satellites, processing systems in the cloud, communication towers and ground stations. In various embodiments, the input/output interface (I/O)supports communication with technicians, and/or one or more storage interfaces for direct connection to storage apparatuses.

105 150 156 105 105 150 130 105 122 105 During operation of the system, the processorloads and executes one or more algorithms, instructions, and rules embodied as program, and, as such, controls the general operation of the system. During operation of the system, the processormay receive data from external sources via the communication system. In various embodiments of the system, the control circuitmay: perform operations attributed to the systemin accordance with an algorithm; perform operations in accordance with state machine logic; and perform operations in accordance with logic in a programmable logic array.

105 122 156 105 105 150 While the exemplary embodiment of the systemis described in the context of the control circuitimplemented as a fully functioning enhanced computer system, those skilled in the art will recognize that the mechanisms of the present disclosure are capable of being distributed as computer-executable instructions or a computer program product (e.g., program) and predefined parameters. Such a program product may comprise an arrangement of instructions organized as multiple interdependent program code modules, each configured to achieve a separate process and/or perform a separate algorithmic operation, arranged to manage data flow through the system. The program code modules may each comprise an ordered listing of executable instructions or rules for implementing logical functions for the processes performed by the system. The instructions in the program code modules, when executed by a processor (e.g., processor), cause the processor to receive and process signals, and perform logic, calculations, methods and/or algorithms as described herein. Such a program product may take a variety of forms, and the present disclosure applies equally regardless of the type of computer-readable signal bearing media used to carry out the distribution.

1 FIG. 2 FIG. 3 FIG. 105 300 105 With continued reference to, exemplary application process modules of the systemare described in connection with, and an exemplary methodfor operating the systemis described in connection with.

102 In the following description, the first user deviceis designated as a reference device or receive-side device, to distinguish it from a plurality of other user's devices participating in a mixed media application or conference call. The receive-side user's preferences are referred to as either receive-side user preferences, or simply, the user preferences, as this is the focus of the disclosure. Those with skill in the art will recognize that an incoming mixed media data (Rxn) signal associated with another user may reflect send-side preferences that are overridden by receive-side user's preferences. It may be appreciated that, in operation, the techniques and methods described for the receive-side user may be employed for every user participating in the conference call or mixed media application.

2 FIG. 105 200 105 202 120 204 106 202 204 120 102 104 202 206 provides at least two different organizations for application process modules in the system. In a first organizational view shown in embodiment, the systemincludes operations performed by a server applicationmodule attributed to serverand operations performed by a device applicationmodule; the modules communicatively coupled together (e.g., via the network), as indicated. Either of the server applicationmodule and device applicationmodule may be part of a larger application, such as a video conference call application. In an application, each application module may be realized as one or more sub-modules, and the modules and sub-modules may be distributed among and between various serverand/or device systems and components. There are N user devices (i.e., first user devicethrough N user device) in operational communication with the server application, providing NRx data input. The N user devices are also referred to as external devices. The NRx are mixed media data signals.

200 201 222 202 120 204 120 106 246 106 1 FIG. In a second organizational view shown in embodiment, one or more of the application modulesthroughattributed to the server applicationmodule in the serverinmay instead be located at the receive-side device, and be part of the device applicationmodule, communicatively coupled to the server(e.g., via the network). In the second organizational view, the moduleswithin the dashed box may be omitted, as the transmission of the data streams Dn across the networkis not necessary.

300 300 105 300 300 300 1 2 FIGS.- 3 FIG. 3 FIG. For illustrative purposes, the following description of methodmay refer to elements mentioned above in connection with. In various embodiments, portions of methodmay be performed by different components of the described system. It should be appreciated that methodmay include any number of additional or alternative operations and tasks, the tasks shown inneed not be performed in the illustrated order, and methodmay be incorporated into a more comprehensive procedure or method, such as a video conference call application, having additional functionality not described in detail herein. Moreover, one or more of the tasks shown incould be omitted from an embodiment of the methodif the intended overall functionality remains intact.

208 120 206 302 208 210 210 A demultiplexing (demuxing) modulein the serverreceives NRx data input, which is a combined media stream (i.e., combined mixed media data signals) that includes video signals, audio signals, and images or profile pictures for N users (the N users do not include duplicate users) via their respective user devices. At, the demuxing modulesorts the NRx signal into its N constituent data streams, Rxn. In various embodiments, a receive-side customization modulereceives and operates on the Rxn. In other embodiments, the receive-side customization moduleis a means for receive-side customization.

304 210 204 304 105 212 105 At, the receive-side customization module, or means for receive-side customization, receives user preferences from the device applicationmodule at the receive-side user device. The user preference is to customize one or more of the Rxn signals. The user preferences include, for the N users, a respective preference UPn. Stated differently, at, the systemreceives user preferences (UPn) for n=1 to n=N. Obtaining the user preferencescan be achieved in a various ways. In a first example, at the start of a conference call, the receive-side user may be prompted or polled to provide user preferences N*UPn. Prompting may include generating a prompt on the reference device or displaying a graphical user interface (GUI) on the reference device that includes the N users and available customization options and determining user selections and responding to user selections based thereon. In a second example, receiving the N*UPn may be interrupt driven. In an interrupt driven scenario, the systemmay be set to a default, such as, passing all video streams Rxn, until and unless the receive-side user interrupts the conference call operations by asserting a user preference UPn to customize a Rxn.

214 216 222 210 222 222 222 222 204 Blocking—blocking (at blocking module) reflects a receive-side user preference to prevent the user's video stream from being displayed on the receive-side device. Blocking is achieved by not passing the video stream to the combining module. Instead, the modulepasses the user's audio signal and the user's image or profile picture as the user's data stream (Dn) to the combining module. Using a blocking preference, especially when applied to multiple data streams, can advantageously reduce power consumption on the receive-side device because blocking the device means that the received data stream doesn't get passed to the combining moduleand therefore less computational demands are made on the combining module. And, the combining modulecan be in the reference user's device, as part of the user's device application. 218 220 220 222 Passing—passing allows the user's video and audio stream to be displayed on the receive-side device. Passing is achieved (passing video module) by passing the user's video Rxn to a decoder moduleand from the decoder moduleto the combining module, as the user's data stream Dn. 234 232 120 228 232 234 232 234 222 Converting—calling avatar generators—This reflects a receive-side user preference to convert an incoming video signal to an avatar data stream Dn. Alternately, this may be a user preference to convert an incoming audio signal to an avatar data stream Dn. Additionally, an avatar data stream can be a video, a 3D object (gITF file format) or motion detector input. Accordingly, calling to avatar generators may be further subdivided into calling to an audio avatar generatorand calling to a video avatar generator. As mentioned, the avatar generators are available modules on-board the server; the avatar modules may employ artificial intelligence (AI) or other machine learning techniques. Calling to an avatar generator includes a call to the avatar libraryto apply the user's preferred avatar. The video avatar generatorconverts the user's video signal into an avatar video (e.g., by first mapping face, eyes, mouth, body, etc., and monitoring motion thereof), and the audio avatar generatorconverts the user's audio signal into an audio avatar data stream or video (e.g., by mapping sounds in the audio signal to mouth motions). It may be appreciated that the user's audio signal may come from an audio signal (such as, when a user has called in from a telephone and is not using a camera) or the user's audio signal may be parsed from the user's video signal. The output from the video avatar generatorand the output from the audio avatar generatoris passed as the user's data stream Dn to the combining module. 230 230 120 218 232 230 Filtering—Calling a distracting behavior module—In some scenarios, a user may be performing distracting behavior while in a video conference call. Some examples of distracting behavior include eating, exercising, and having other individuals in the background. The distracting behavior moduleis another example of an available module in the serverthat can be employed to execute an algorithm to remove the distracting behavior from video signals; therefore, it may operate on video signals from passing video moduleand video signals from the video avatar generator. Depending on the algorithm implemented by the distracting behavior module, the user's video may be replaced with the user's image or profile picture, the user's background may be blurred, or AI may be used to alter a portion of the user's video stream, such as, by rendering the user's arms/hands as being still. Non-limiting examples of the user preference optionsinclude:

306 The user preferences (UPn) for n=1 to n=N may optionally be stored at. Storing user preferences can be temporary, such as, for the duration of the video conference, or may embody default settings that the receive-side user returns to for multiple different video conferences.

308 105 105 222 At, the systemconverts the received data stream into a customized data stream Dn for the recipient device, wherein the customized data stream is responsive to received user preferences. E.g., responsive to receiving the user preferences N*UPn, the systemcreates respective data stream(s) (Dn) for n=1 to n=N, as described above. Customized data streams can include video, 3d object (glTF files), or motion vectors. Customized data streams are passed to the combining module, blocked data streams are not.

310 222 222 Atthe N data streams Dn which are receive-side customized are combined at the combining module. Note that power consumption at this stage may be reduced because blocked video streams are not processed in the combining module; this translates into less data and fewer operations to perform.

105 120 247 246 105 246 247 242 244 105 120 222 247 224 312 226 204 314 Depending on whether the systemis serverbased or receive-side device based, the customized data streams that make up created signalsmay or may not be encoded and transmitted (module) externally. If the systemis receive-side device based, the operations of moduleare omitted and created signalscan be composited at a graphics processing moduleonboard the receive-side device and displayed/emitted at the receive-side device by module. If the systemis serverbased, output from the combining module(created signals) may be encoded by the encoding moduleat. The encoded combined data stream may be transmitted by the transmitting moduleto the user's device applicationon receive-side device at.

314 304 310 314 247 After, the method may end or return to obtaining user preferencesand generating data streams Dn until the conference call ends. After eitheror, the receive-side device has obtained the created signals, comprising one or more converted data streams Dn, and can proceed to process and display/emit the receive-side customized data streams as described herein.

244 204 237 238 106 240 224 242 244 At the receive-side device, a displaying an audio emitting modulecan present the video conference call for the receive-side user. As may be appreciated, in support of the actions and tasks described herein, the device applicationmodule may also include a user input detection module, as may be associated with a keyboard or touchscreen, a transmit/receive moduleto communicate over the network, a decoding moduleassociated with the encoding moduleand the graphics processing (compositing) moduleto generate and drive the displaying an audio emitting module.

Thus, systems and methods for receive-side customization of presentation of mixed media data have been provided. Embodiments advantageously improve the user's experience during conference calls and other mixed media applications. Aspects of the disclosure allow users to reduce sensory information to improve their ability to focus, enable users to customize the presentation of media during mixed media applications, allow users to remove distractions from other participants' video streams, and can increase battery life by reducing power and network bandwidth by way of blocking video streams.

The following additional figures and description are intended to illustrate various contexts for usage and application of the present disclosure.

4 FIG. 400 402 408 410 412 414 400 402 105 400 Disclosed embodiments may be implemented in a compute node. In the simplified example depicted in, a compute nodeincludes a compute engine (referred to herein as “compute circuitry”), an input/output (I/O) subsystem, data storage, a communication circuitry subsystem, and, optionally, one or more peripheral devices. With respect to the present example, the compute nodeor compute circuitrymay perform the operations and tasks attributed to the system. In other examples, respective compute nodesmay include other or additional components, such as those typically found in a computer (e.g., a display, peripheral devices, etc.). Additionally, in some examples, one or more of the illustrative components may be incorporated in, or otherwise form a portion of, another component.

400 400 404 406 404 404 In some examples, the compute nodemay be embodied as a single device such as an integrated circuit, an embedded system, a field-programmable gate array (FPGA), a system-on-a-chip (SOC), or other integrated system or device. In the illustrative example, the compute nodeincludes or is embodied as a processorand a memory. The processormay be embodied as any type of processor capable of performing the functions described herein (e.g., executing compile functions and executing an application). For example, the processormay be embodied as a multi-core processor(s), a microcontroller, a processing unit, a specialized or special purpose processing unit, or other processor or processing/controlling circuit.

404 404 404 400 In some examples, the processormay be embodied as, include, or be coupled to an FPGA, an application specific integrated circuit (ASIC), reconfigurable hardware or hardware circuitry, or other specialized hardware to facilitate performance of the functions described herein. Also in some examples, the processormay be embodied as a specialized x-processing unit (xPU) also known as a data processing unit (DPU), infrastructure processing unit (IPU), or network processing unit (NPU). Such an xPU may be embodied as a standalone circuit or circuit package, integrated within an SOC, or integrated with networking circuitry (e.g., in a SmartNIC, or enhanced SmartNIC), acceleration circuitry, storage devices, or AI hardware (e.g., GPUs or programmed FPGAs). Such an xPU may be designed to receive programming to process one or more data streams and perform specific tasks and actions for the data streams (such as hosting microservices, performing service management or orchestration, organizing, or managing server or data center hardware, managing service meshes, or collecting and distributing telemetry), outside of the CPU or general-purpose processing hardware. However, it will be understood that a xPU, a SOC, a CPU, and other variations of the processormay work in coordination with each other to execute many types of operations and instructions within and on behalf of the compute node.

406 The memorymay be embodied as any type of volatile (e.g., dynamic random-access memory (DRAM), etc.) or non-volatile memory or data storage capable of performing the functions described herein. Volatile memory may be a storage media that requires power to maintain the state of data stored by the media. Non-limiting examples of volatile memory may include various types of random-access memory (RAM), such as DRAM or static random-access memory (SRAM). One type of DRAM that may be used in a memory module is synchronous dynamic random-access memory (SDRAM).

406 404 406 In an example, the memory device is a block addressable memory device, such as those based on NAND or NOR technologies. A memory device may also include a three-dimensional crosspoint memory device (e.g., Intel® 3D XPoint™ memory), or other byte addressable write-in-place nonvolatile memory devices. The memory device may refer to the die itself and/or to a packaged memory product. In some examples, 3D crosspoint memory (e.g., Intel® 3D XPoint™ memory) may comprise a transistor-less stackable cross point architecture in which memory cells sit at the intersection of word lines and bit lines and are individually addressable and in which bit storage is based on a change in bulk resistance. In some examples, all or a portion of the memorymay be integrated into the processor. The memorymay store various software and data used during operation such as one or more applications, data operated on by the application(s), libraries, and drivers.

402 400 408 402 404 406 402 408 408 404 406 402 402 The compute circuitryis communicatively coupled to other components of the compute nodevia the I/O subsystem, which may be embodied as circuitry and/or components to facilitate input/output operations with the compute circuitry(e.g., with the processorand/or the main memory) and other components of the compute circuitry. For example, the I/O subsystemmay be embodied as, or otherwise include, memory controller hubs, input/output control hubs, integrated sensor hubs, firmware devices, communication links (e.g., point-to-point links, bus links, wires, cables, light guides, printed circuit board traces, etc.), and/or other components and subsystems to facilitate the input/output operations. In some examples, the I/O subsystemmay form a portion of a system-on-a-chip (SoC) and be incorporated, along with one or more of the processor, the memory, and other components of the compute circuitry, into the compute circuitry.

410 410 410 410 400 The one or more illustrative data storage devicesmay be embodied as any type of devices configured for short-term or long-term storage of data such as, for example, memory devices and circuits, memory cards, hard disk drives, solid-state drives, or other data storage devices. Individual data storage devicesmay include a system partition that stores data and firmware code for the data storage device. Individual data storage devicesmay also include one or more operating system partitions that store data files and executables for operating systems depending on, for example, the type of compute node.

412 402 The communication subsystemmay be embodied as any communication circuit, device, transceiver circuit, or collection thereof, capable of enabling communications over a network between the compute circuitryand another computing device (e.g., an edge gateway of an implementing edge computing system).

412 412 412 412 412 412 The communication subsystemmay implement any of a number of wireless standards or protocols, including but not limited to Institute for Electrical and Electronic Engineers (IEEE) standards including Wi-Fi (IEEE 802.11 family), IEEE 802.16 standards (e.g., IEEE 802.16-2005 Amendment), Long-Term Evolution (LTE) project along with any amendments, updates, and/or revisions (e.g., advanced LTE project, ultra-mobile broadband (UMB) project (also referred to as “3GPP2”), etc.). IEEE 802.16 compatible Broadband Wireless Access (BWA) networks are generally referred to as WiMAX networks, an acronym that stands for Worldwide Interoperability for Microwave Access, which is a certification mark for products that pass conformity and interoperability tests for the IEEE 802.16 standards. The communication subsystemmay operate in accordance with a Global System for Mobile Communication (GSM), General Packet Radio Service (GPRS), Universal Mobile Telecommunications System (UMTS), High Speed Packet Access (HSPA), Evolved HSPA (E-HSPA), or LTE network. The communication subsystemmay operate in accordance with Enhanced Data for GSM Evolution (EDGE), GSM EDGE Radio Access Network (GERAN), Universal Terrestrial Radio Access Network (UTRAN), or Evolved UTRAN (E-UTRAN). The communication subsystemmay operate in accordance with Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Digital Enhanced Cordless Telecommunications (DECT), Evolution-Data Optimized (EV-DO), and derivatives thereof, as well as any other wireless protocols that are designated as 3G, 4G, 5G, and beyond. The communication subsystemmay operate in accordance with other wireless protocols in other embodiments. The communication subsystemmay include an antenna to facilitate wireless communications and/or to receive other wireless communications (such as AM or FM radio transmissions).

412 412 412 412 412 412 In some embodiments, the communication subsystemmay manage wired communications, such as electrical, optical, or any other suitable communication protocols (e.g., IEEE 802.3 Ethernet standards). As noted above, the communication subsystemmay include multiple communication components. For instance, a first communication subsystemmay be dedicated to shorter-range wireless communications such as Wi-Fi or Bluetooth, and a second communication subsystemmay be dedicated to longer-range wireless communications such as global positioning system (GPS), EDGE, GPRS, CDMA, WiMAX, LTE, EV-DO, or others. In some embodiments, a first communication subsystemmay be dedicated to wireless communications, and a second communication subsystemmay be dedicated to wired communications.

412 416 416 400 416 416 416 416 402 416 The illustrative communication subsystemincludes an optional network interface controller (NIC), which may also be referred to as a host fabric interface (HFI). The NICmay be embodied as one or more add-in-boards, daughter cards, network interface cards, controller chips, chipsets, or other devices that may be used by the compute nodeto connect with another computing device (e.g., an edge gateway node). In some examples, the NICmay be embodied as part of a system-on-a-chip (SoC) that includes one or more processors or included on a multichip package that also contains one or more processors. In some examples, the NICmay include a local processor (not shown) and/or a local memory (not shown) that are both local to the NIC. In such examples, the local processor of the NICmay be capable of performing one or more of the functions of the compute circuitrydescribed herein. Additionally, or alternatively, in such examples, the local memory of the NICmay be integrated into one or more components of the client compute node at the board level, socket level, chip level, and/or other levels.

400 414 414 400 400 Additionally, in some examples, a respective compute nodemay include one or more peripheral devices. Such peripheral devicesmay include any type of peripheral device found in a computing device or server such as audio input devices, a display, other input/output devices, interface devices, and/or other peripheral devices, depending on the particular type of the compute node. In further examples, the compute nodemay be embodied by a respective edge compute node (whether a client, gateway, or aggregation node) in an edge computing system or like forms of appliances, computers, subsystems, circuitry, or other components.

400 400 In other examples, the compute nodemay be embodied as any type of device or collection of devices capable of performing various compute functions. Respective compute nodesmay be embodied as a type of device, appliance, computer, or other “thing” capable of communicating with other compute nodes that may be edge, networking, or endpoint components. For example, a compute node may be embodied as a personal computer, server, smartphone, a mobile computing device, a smart appliance, smart camera, an in-vehicle compute system (e.g., a navigation system), a weatherproof or weather-sealed computing appliance, a self-contained device within an outer case, shell, etc., or other device or system capable of performing the described functions.

5 FIG. 502 504 512 514 512 514 502 504 508 510 512 514 500 512 516 502 512 514 illustrates a multi-processor environment in which embodiments may be implemented. Processor unitsandfurther comprise cache memoriesand, respectively. The cache memoriesandcan store data (e.g., instructions) utilized by one or more components of the processor unitsand, such as the processor coresand. The cache memoriesandcan be part of a memory hierarchy for the computing system. For example, the cache memoriescan locally store data that is also stored in a memoryto allow for faster access to the data by the processor unit. In some embodiments, the cache memoriesandcan comprise multiple cache levels, such as level 1 (L1), level 2 (L2), level 3 (L3), level 4 (L4) and/or other caches or cache levels. In some embodiments, one or more levels of cache memory (e.g., L2, L3, L4) can be shared among multiple cores in a processor unit or among multiple processor units in an integrated circuit component. In some embodiments, the last level of cache memory on an integrated circuit component can be referred to as a last level cache (LLC). One or more of the higher levels of cache levels (the smaller and faster caches) in the memory hierarchy can be located on the same integrated circuit die as a processor core and one or more of the lower cache levels (the larger and slower caches) can be located on an integrated circuit dies that are physically separate from the processor core integrated circuit dies.

500 500 Although the computing systemis shown with two processor units, the computing systemcan comprise any number of processor units. Further, a processor unit can comprise any number of processor cores. A processor unit can take various forms such as a central processing unit (CPU), a graphics processing unit (GPU), general-purpose GPU (GPGPU), accelerated processing unit (APU), field-programmable gate array (FPGA), neural network processing unit (NPU), data processor unit (DPU), accelerator (e.g., graphics accelerator, digital signal processor (DSP), compression accelerator, artificial intelligence (AI) accelerator), controller, or other types of processing units. As such, the processor unit can be referred to as an XPU (or xPU). Further, a processor unit can comprise one or more of these various types of processing units. In some embodiments, the computing system comprises one processor unit with multiple cores, and in other embodiments, the computing system comprises a single processor unit with a single core. As used herein, the terms “processor unit” and “processing unit” can refer to any processor, processor core, component, module, engine, circuitry, or any other processing element described or referenced herein.

500 In some embodiments, the computing systemcan comprise one or more processor units that are heterogeneous or asymmetric to another processor unit in the computing system. There can be a variety of differences between the processing units in a system in terms of a spectrum of metrics of merit including architectural, microarchitectural, thermal, power consumption characteristics, and the like. These differences can effectively manifest themselves as asymmetry and heterogeneity among the processor units in a system.

502 504 The processor unitsandcan be located in a single integrated circuit component (such as a multi-chip package (MCP) or multi-chip module (MCM)) or they can be located in separate integrated circuit components. An integrated circuit component comprising one or more processor units can comprise additional components, such as embedded DRAM, stacked high bandwidth memory (HBM), shared cache memories (e.g., L3, L4, LLC), input/output (I/O) controllers, or memory controllers. Any of the additional components can be located on the same integrated circuit die as a processor unit, or on one or more integrated circuit dies separate from the integrated circuit dies comprising the processor units. In some embodiments, these separate integrated circuit dies can be referred to as “chiplets”. In some embodiments where there is heterogeneity or asymmetry among processor units in a computing system, the heterogeneity or asymmetric can be among processor units located in the same integrated circuit component. In embodiments where an integrated circuit component comprises multiple integrated circuit dies, interconnections between dies can be provided by the package substrate, one or more silicon interposers, one or more silicon bridges embedded in the package substrate (such as Intel@embedded multi-die interconnect bridges (EMIBs)), or combinations thereof.

502 504 520 522 520 522 516 518 502 504 516 518 520 522 502 504 5 FIG. Processor unitsandfurther comprise memory controller logic (MC)and. As shown in, MCsandcontrol memoriesandcoupled to the processor unitsand, respectively. The memoriesandcan comprise various types of volatile memory (e.g., dynamic random-access memory (DRAM), static random-access memory (SRAM)) and/or non-volatile memory (e.g., flash memory, chalcogenide-based phase-change non-volatile memories), and comprise one or more layers of the memory hierarchy of the computing system. While MCsandare illustrated as being integrated into the processor unitsand, in alternative embodiments, the MCs can be external to a processor unit.

502 504 530 532 534 532 536 502 538 530 534 540 504 542 530 530 550 530 552 530 552 554 Processor unitsandare coupled to an Input/Output (I/O) subsystemvia point-to-point interconnectionsand. The point-to-point interconnectionconnects a point-to-point interfaceof the processor unitwith a point-to-point interfaceof the I/O subsystem, and the point-to-point interconnectionconnects a point-to-point interfaceof the processor unitwith a point-to-point interfaceof the I/O subsystem. Input/Output subsystemfurther includes an interfaceto couple the I/O subsystemto a graphics engine. The I/O subsystemand the graphics engineare coupled via a bus.

530 560 562 560 564 560 570 560 580 580 580 582 588 590 592 592 580 584 500 586 The Input/Output subsystemis further coupled to a first busvia an interface. The first buscan be a Peripheral Component Interconnect Express (PCIe) bus or any other type of bus. Various I/O devicescan be coupled to the first bus. A bus bridgecan couple the first busto a second bus. In some embodiments, the second buscan be a low pin count (LPC) bus. Various devices can be coupled to the second busincluding, for example, a keyboard/mouse, audio I/O devices, and a storage device, such as a hard disk drive, solid-state drive, or another storage device for storing computer-executable instructions (code)or data. The codecan comprise computer-executable instructions for performing methods described herein. Additional components that can be coupled to the second businclude communication device(s), which can provide for communication between the computing systemand one or more wired or wireless networks(e.g. Wi-Fi, cellular, or satellite networks) via one or more wired or wireless communication links (e.g., wire, cable, Ethernet connection, radio-frequency (RF) channel, infrared channel, Wi-Fi channel) using one or more communication standards (e.g., IEEE 802.11 standard and its supplements).

584 584 500 In embodiments where the communication devicessupport wireless communication, the communication devicescan comprise wireless communication components coupled to one or more antennas to support communication between the computing systemand external devices. The wireless communication components can support various wireless communication protocols and technologies such as Near Field Communication (NFC), IEEE 802.11 (Wi-Fi) variants, WiMax, Bluetooth, Zigbee, 4G Long Term Evolution (LTE), Code Division Multiplexing Access (CDMA), Universal Mobile Telecommunication System (UMTS) and Global System for Mobile Telecommunication (GSM), and 5G broadband cellular technologies. In addition, the wireless modems can support communication with one or more cellular networks for data and voice communications within a single cellular network, between cellular networks, or between the computing system and a public switched telephone network (PSTN).

500 500 512 514 516 518 590 594 596 500 586 500 500 The systemcan comprise removable memory such as flash memory cards (e.g., SD (Secure Digital) cards), memory sticks, Subscriber Identity Module (SIM) cards). The memory in system(including cachesand, memoriesand, and storage device) can store data and/or computer-executable instructions for executing an operating systemand application programs. Example data includes web pages, text messages, images, sound files, and video data biometric thresholds for particular users or other data sets to be sent to and/or received from one or more network servers or other devices by the systemvia the one or more wired or wireless networks, or for use by the system. The systemcan also have access to external memory or storage (not shown) such as external hard drives or cloud-based storage.

594 596 596 6 FIG. The operating system(also simplified to “OS” herein) can control the allocation and usage of the components illustrated inand support the one or more application programs. The application programscan include common computing system applications (e.g., email applications, calendars, contact managers, web browsers, messaging applications) as well as other computing applications.

594 596 594 594 In some embodiments, a hypervisor (or virtual machine manager) operates on the operating systemand the application programsoperate within one or more virtual machines operating on the hypervisor. In these embodiments, the hypervisor is a type-2 or hosted hypervisor as it is running on the operating system. In other hypervisor-based embodiments, the hypervisor is a type-1 or “bare-metal” hypervisor that runs directly on the platform resources of the computing systemwithout an intervening operating system layer.

596 596 596 594 500 594 594 In some embodiments, the applicationscan operate within one or more containers. A container is a running instance of a container image, which is a package of binary images for one or more of the applicationsand any libraries, configuration settings, and any other information that one or more applicationsneed for execution. A container image can conform to any container image format, such as Docker®, Appc, or LXC container image formats. In container-based embodiments, a container runtime engine, such as Docker Engine, LXU, or an open container initiative (OCI)-compatible container runtime (e.g., Railcar, CRI-O) operates on the operating system (or virtual machine monitor) to provide an interface between the containers and the operating system. An orchestrator can be responsible for management of the computing systemand various container-related tasks such as deploying container images to the computing system, monitoring the performance of deployed containers, and monitoring the utilization of the resources of the computing system.

500 598 500 500 The computing systemcan support various additional input devices, represented generally as user interfaces, such as a touchscreen, microphone, monoscopic camera, stereoscopic camera, trackball, touchpad, trackpad, proximity sensor, light sensor, electrocardiogram (ECG) sensor, PPG (photoplethysmogram) sensor, galvanic skin response sensor, and one or more output devices, such as one or more speakers or displays. Other possible input and output devices include piezoelectric and other haptic I/O devices. Any of the input or output devices can be internal to, external to, or removably attachable with the system. External input and output devices can communicate with the systemvia wired or wireless connections.

598 594 596 500 500 500 In addition, one or more of the user interfacesmay be natural user interfaces (NUIs). For example, the operating systemor applicationscan comprise speech recognition logic as part of a voice user interface that allows a user to operate the systemvia voice commands. Further, the computing systemcan comprise input devices and logic that allows a user to interact with computing the systemvia body, hand, or face gestures. For example, a user's hand gestures can be detected and interpreted to provide input to a gaming application.

564 500 The I/O devicescan include at least one input/output port comprising physical connectors (e.g., USB, IEEE 1394 (FireWire), Ethernet, RS-232), a power supply (e.g., battery), a global satellite navigation system (GNSS) receiver (e.g., GPS receiver); a gyroscope; an accelerometer; and/or a compass. A GNSS receiver can be coupled to a GNSS antenna. The computing systemcan further comprise one or more additional antennas coupled to one or more additional receivers, transmitters, and/or transceivers to enable additional functions.

594 594 In addition to those already discussed, integrated circuit components, integrated circuit constituent components, and other components in the computing systemcan communicate with interconnect technologies such as Intel® QuickPath Interconnect (QPI), Intel® Ultra Path Interconnect (UPI), Computer Express Link (CXL), cache coherent interconnect for accelerators (CCIX®), serializer/deserializer (SERDES), Nvidia® NVLink, ARM Infinity Link, Gen-Z, or Open Coherent Accelerator Processor Interface (OpenCAPI). Other interconnect technologies may be used and a computing systemmay utilize more or more interconnect technologies.

5 FIG. 5 FIG. 5 FIG. 502 504 552 It is to be understood thatillustrates only one example computing system architecture. Computing systems based on alternative architectures can be used to implement technologies described herein. For example, instead of the processorsandand the graphics enginebeing located on discrete integrated circuits, a computing system can comprise an SoC (system-on-a-chip) integrated circuit incorporating multiple processors, a graphics engine, and additional components. Further, a computing system can connect its constituent component via bus or point-to-point configurations different from that shown in. Moreover, the illustrated components inare not required or all-inclusive, as shown components can be removed and other components added in alternative embodiments.

6 FIG. 600 600 is a block diagram of an example processor unitto execute computer-executable instructions as part of implementing technologies described herein. The processor unitcan be a single-threaded core or a multithreaded core in that it may include more than one hardware thread context (or “logical processor”) per processor unit.

6 FIG. 610 600 610 610 615 600 also illustrates a memorycoupled to the processor unit. The memorycan be any memory described herein or any other memory known to those of skill in the art. The memorycan store computer-executable instructions(code) executable by the processor unit.

620 610 630 630 620 635 640 The processor unit comprises front-end logicthat receives instructions from the memory. An instruction can be processed by one or more decoders. The decodercan generate as its output a micro-operation such as a fixed width micro-operation in a predefined format, or generate other instructions, microinstructions, or control signals, which reflect the original code instruction. The front-end logicfurther comprises register renaming logicand scheduling logic, which generally allocate resources and queues operations corresponding to converting an instruction for execution.

600 650 665 1 665 650 670 675 600 675 The processor unitfurther comprises execution logic, which comprises one or more execution units (EUs)-through-N. Some processor unit embodiments can include a few execution units dedicated to specific functions or sets of functions. Other embodiments can include only one execution unit or one execution unit that can perform a particular function. The execution logicperforms the operations specified by code instructions. After completion of execution of the operations specified by the code instructions, back-end logicretires instructions using retirement logic. In some embodiments, the processor unitallows out of order execution but requires in-order retirement of instructions. Retirement logiccan take a variety of forms as known to those of skill in the art (e.g., re-order buffers or the like).

600 630 635 650 The processor unitis transformed during execution of instructions, at least in terms of the output generated by the decoder, hardware registers and tables utilized by the register renaming logic, and any registers (not shown) modified by the execution logic.

While at least one embodiment has been presented in the foregoing detailed description, it should be appreciated that a vast number of variations exist. It should also be appreciated that the disclosed embodiments are only examples, and are not intended to limit the scope, applicability, or configuration of the disclosure in any way. Rather, the foregoing detailed description will provide those skilled in the art with a convenient road map for implementing the disclosed aspects of the present disclosure. Various changes can be made in the function and arrangement of elements without departing from the scope of the disclosure as set forth in the appended claims and the legal equivalents thereof.

As used herein, a “computer,” “computing system,” or “compute device” refers to any of a variety of computing devices and includes systems comprising multiple discrete physical components capable of executing instructions. In some embodiments, the computing systems are located in a data center, such as an enterprise data center (e.g., a data center owned and operated by a company and typically located on company premises), managed services data center (e.g., a data center managed by a third party on behalf of a company), a collocated data center (e.g., a data center in which data center infrastructure is provided by the data center host and a company provides and manages their own data center components (servers, etc.)), cloud data center (e.g., a data center operated by a cloud services provider that host companies applications and data), and an edge data center (e.g., a data center, typically having a smaller footprint than other data center types, located close to the geographic area that it serves).

Thus, the term “computer-executable instruction” refers to instructions that can be executed by any computing system, device, or machine described or mentioned herein as well as any other computing system, device, or machine capable of executing instructions. As mentioned, any of the disclosed methods (or a portion thereof) can be implemented as computer-executable instructions or a computer program product. Such instructions can cause a computing system or one or more processor units capable of executing computer-executable instructions to perform any of the disclosed methods.

The computer-executable instructions or computer program products as well as any data created and/or used during implementation of the disclosed technologies can be stored on one or more tangible or non-transitory computer-readable storage media, such as volatile memory (e.g., DRAM, SRAM), non-volatile memory (e.g., flash memory, chalcogenide-based phase-change non-volatile memory) optical media discs (e.g., DVDs, CDs), and magnetic storage (e.g., magnetic tape storage, hard disk drives). Computer-readable storage media can be contained in computer-readable storage devices such as solid-state drives, USB flash drives, and memory modules. Alternatively, any of the methods disclosed herein (or a portion) thereof may be performed by hardware components comprising non-programmable circuitry. In some embodiments, any of the methods herein can be performed by a combination of non-programmable hardware components and one or more processing units executing computer-executable instructions stored on computer-readable storage media.

The computer-executable instructions can be part of, for example, an operating system of the computing system, an application stored locally to the computing system, or a remote application accessible to the computing system (e.g., via a web browser). Any of the methods described herein can be performed by computer-executable instructions performed by a single computing system or by one or more networked computing systems operating in a network environment. Computer-executable instructions and updates to the computer-executable instructions can be downloaded to a computing system from a remote server.

Further, it is to be understood that implementation of the disclosed technologies is not limited to any specific computer language or program. For instance, the disclosed technologies can be implemented by software written in C++, C#, Java, Perl, Python, JavaScript, Adobe Flash, C#, assembly language, or any other programming language. Likewise, the disclosed technologies are not limited to any computer system or type of hardware.

Furthermore, any of the software-based embodiments (comprising, for example, computer-executable instructions for causing a computer to perform any of the disclosed methods) can be uploaded, downloaded, or remotely accessed through a suitable communication means. Such suitable communication means include, for example, the Internet, the World Wide Web, an intranet, cable (including fiber optic cable), magnetic communications, electromagnetic communications (including RF, microwave, ultrasonic, and infrared communications), electronic communications, or other such communication means.

Additionally, theories of operation, scientific principles, or other theoretical descriptions presented herein in reference to the apparatuses or methods of this disclosure have been provided for the purposes of better understanding and are not intended to be limiting in scope. The apparatuses and methods in the appended claims are not limited to those apparatuses and methods that function in the manner described by such theories of operation.

As used herein, phrases such as “embodiments,” “an aspect of the present disclosure,” “various aspects of the present disclosure,” “some aspects of the present disclosure,” and the like, indicate that some aspects of the present disclosure may have some, all, or none of the features described for other aspects of the present disclosure. “First,” “second,” “third,” and the like describe a common object and indicate different instances of like objects being referred to; unless specifically stated, they do not imply a given sequence, either temporally or spatially, in ranking, or any other manner. In accordance with patent application parlance, “connected” indicates elements that are in direct physical or electrical contact with each other and “coupled” indicates elements that co-operate or interact with each other, coupled elements may or may not be in direct physical or electrical contact. Furthermore, the terms “comprising,” “including,” “having,” and the like, are utilized synonymously to denote non-exclusive inclusions.

As used in this application and the claims, a list of items joined by the term “at least one of” can mean any combination of the listed terms. For example, the phrase “at least one of A, B and C” can mean A; B; C; A and B; A and C; B and C; or A, B, and C. Similarly, as used in this application and the claims, a list of items joined by the term “one or more of” can mean any combination of the listed terms. For example, the phrase “one or more of A, B and C” can mean A; B; C; A and B; A and C; B and C; or A, B, and C.

The following Examples pertain to additional aspects of the present disclosure of technologies disclosed herein.

Example 1 is a system, comprising: a reference device; a control circuit communicatively coupled to the reference device, wherein the control circuit is to: receive a mixed media data signal from an external device; receive a user preference to customize the mixed media data signal for display on the reference device; create a customized data stream as a function of the user preference; display the customized data stream on the reference device.

Example 2 includes the subject matter of Example 1, wherein the control circuit is further to: generate a prompt on the reference device responsive to receiving the mixed media data signal, wherein the prompt enables a user to select from among block, pass, convert, and filter; detect a user selection responsive to the prompt, wherein the user selection is the user preference to customize the mixed media data signal.

Example 3 includes the subject matter of Example 2, wherein the user selection is convert, and the control circuit is further to create the customized data stream by converting the mixed media data signal to an avatar data stream.

Example 4 includes the subject matter of Example 2, wherein the user selection is filter, and the control circuit is further to create the customized data stream by processing the mixed media data signal with an algorithm to remove distracting behavior.

Example 5 includes the subject matter of Example 1 wherein the control circuit is further to: create the customized data stream on a cloud server; transmit the customized data stream to the reference device via a network.

Example 6 includes the subject matter of Example 1, wherein the control circuit is further to create the customized data stream on the reference device.

Example 7 includes the subject matter of Example 1 wherein the mixed media data signal further comprises an audio signal, and the control circuit is further to: determine that the user preference is to convert the audio signal into an audio avatar data stream; create the customized data stream as an audio avatar data stream.

Example 8 includes the subject matter of Example 1, wherein the mixed media data signal is one of a plurality of mixed media data signals, and the control circuit is further to: receive the plurality of mixed media data signals; receive, for the plurality of mixed media data signal, a respective user preference to customize the mixed media data signal for display on the reference device; create a respective plurality of customized data streams; display the respective plurality of customized data streams concurrently on the reference device.

Example 9 is a non-transitory computer-readable media comprising instructions that are, when executed by processing circuitry, to: receive a mixed media data signal from an external device; receive a user preference to customize the mixed media data signal for display on a reference device; create a customized data stream as a function of the user preference; display the customized data stream on the reference device.

Example 10 includes the subject matter of Example 9, wherein the instructions are further to: generate a prompt on the reference device responsive to receiving the mixed media data signal, wherein the prompt enables a user to select from among block, pass, convert, and filter; detect a user selection responsive to the prompt, wherein the user selection is the user preference to customize the mixed media data signal.

Example 11 includes the subject matter of Example 10, wherein the instructions are further to: determine that the user selection is convert; create the customized data stream by converting the mixed media data signal to an avatar data stream, responsive to determining that the user selection is convert.

Example 12 includes the subject matter of Example 10, wherein the instructions are further to: determine that the user selection is filter; create the customized data stream by processing the mixed media data signal with an algorithm to remove distracting behavior, responsive to determining that the user selection is filter.

Example 13 includes the subject matter of Example 9, wherein the instructions are further to: create the customized data stream on a cloud server; transmit the customized data stream to the reference device via a network.

Example 14 includes the subject matter of Example 9, wherein the instructions are further to create the customized data stream on the reference device.

Example 15 includes the subject matter of Example 9, wherein the mixed media data signal further comprises an audio signal, and the instructions are further to: determine that the user preference is to convert the audio signal into an audio avatar data stream; create the customized data stream as an audio avatar data stream.

Example 16 includes the subject matter of Example 9, wherein the mixed media data signal is one of a plurality of mixed media data signals, and the instructions are further to: receive the plurality of mixed media data signals; receive, for the plurality of mixed media data signal, a respective user preference to customize the mixed media data signal for display on the reference device; create a respective plurality of customized data streams; display the respective plurality of customized data streams concurrently on the reference device.

Example 17 is a method, comprising: receiving a mixed media data signal from an external device; receiving a user preference to customize the mixed media data signal for display on a reference device; creating a customized data stream as a function of the user preference; displaying the customized data stream on the reference device.

Example 18 includes the subject matter of Example 17, further comprising: generating a prompt on the reference device responsive to receiving the mixed media data signal, wherein the prompt enables a user to select from among block, pass, convert, and filter; detecting a user selection responsive to the prompt, wherein the user selection is the user preference to customize the mixed media data signal.

Example 19 includes the subject matter of Example 18, further comprising: determining that the user selection is convert; and creating the customized data stream by converting the mixed media data signal to an avatar data stream.

Example 20 includes the subject matter of Example 18, further comprising: determining that the user selection is filter; creating the customized data stream by processing the mixed media data signal with an algorithm to remove distracting behavior.