Video transport stream stability prediction

This patent application is a continuation of, and claims the benefit of priority to, U.S. application Ser. No. 17/899,460, filed on Aug. 30, 2022, which is a continuation of, and claims the benefit of priority to, U.S. application Ser. No. 17/124,372, filed on Dec. 16, 2020, now U.S. Pat. No. 11,438,218, the entireties of which are incorporated herein by reference.

The embodiments of the invention are related to the field of managing video and image content. More specifically, the embodiments of the invention relate to methods and systems for determining the stability of a video stream transport stability in real time.

Many technologies related to the delivery of video streams are defined by various standards such as the standards of digital video broadcasting (DVB). DVB standards are maintained by the DVB Project, an international industry consortium, and are published by a Joint Technical Committee (JTC) of the European Telecommunications Standards Institute (ETSI), European Committee for Electrotechnical Standardization (CENELEC) and European Broadcasting Union (EBU). Video streaming and similar technologies can be transmitted over a variety of media including satellite, cable, terrestrial television, microwave and similar media.

Standards, such as the DVB standards, for the distribution of video streams can define the physical layer and data link layer of the distribution system. Video data is transmitted in defined formats such as motion pictures experts group (MPEG) transport streams. Video streams can be encoded using various discrete cosine transform (DCT) based encoding standard, such as, H.26x, MPEG and similar encoding schemes. Audio for the video streams can be encoded using modified DCT (MDCT) stand, mis including advanced audio coding (AAC), Dolby Digital (AC-3), MP3 and similar encoding formats.

DVB and similar standards can also define standards for measurement of analysis for transport streams, such as ETSI technical report (TR) 101 290 (herein after “ETR 101”). Transport stream measurement and analysis standards such as ETR 101 can define how metrics and errors in the transport stream are reported and managed. However, these standards have limitations in the information provided and do not provide any predictive indication of video stream stability or characteristics.

In the following description, numerous specific details are set forth. However, it is understood that embodiments of the invention may be practiced without these specific details. In other instances, well-known circuits, structures and techniques have not been shown in detail in order not to obscure the understanding of this description. It will be appreciated, however, by one skilled in the art that the invention may be practiced without such specific details. Those of ordinary skill in the art, with the included descriptions, will be able to implement appropriate functionality without undue experimentation.

References in the specification to “one embodiment,” “an embodiment,” “an example embodiment,” etc., indicate that the embodiment described may include a particular feature, structure, or characteristic, but every embodiment may not necessarily include the particular feature, structure, or characteristic. Moreover, such phrases are not necessarily referring to the same embodiment. Further, when a particular feature, structure, or characteristic is described in connection with an embodiment, it is submitted that it is within the knowledge of one skilled in the art to effect such feature, structure, or characteristic in connection with other embodiments whether or not explicitly described.

Bracketed text and blocks with dashed borders (e.g., large dashes, small dashes, dot-dash, and dots) may be used herein to illustrate optional operations that add additional features to embodiments of the invention. However, such notation should not be taken to mean that these are the only options or optional operations, and/or that blocks with solid borders are not optional in certain embodiments of the invention.

In the following description and claims, the terms “coupled” and “connected,” along with their derivatives, may be used. It should be understood that these terms are not intended as synonyms for each other. “Coupled” is used to indicate that two or more elements, which may or may not be in direct physical or electrical contact with each other, co-operate or interact with each other. “Connected” is used to indicate the establishment of communication between two or more elements that are coupled with each other. A “set,” as used herein refers to any positive whole number of items including one item.

An electronic device stores and transmits (internally and/or with other electronic devices over a network) code (which is composed of software instructions and which is sometimes referred to as computer program code or a computer program) and/or data using machine-readable media (also called computer-readable media), such as machine-readable storage media (e.g., magnetic disks, optical disks, read only memory (ROM), flash memory devices, phase change memory) and machine-readable transmission media (also called a carrier) (e.g., electrical, optical, radio, acoustical or other form of propagated signals-such as carrier waves, infrared signals). Thus, an electronic device (e.g., a computer) includes hardware and software, such as a set of one or more processors coupled to one or more machine-readable storage media to store code for execution on the set of processors and/or to store data. For instance, an electronic device may include non-volatile memory containing the code since the non-volatile memory can persist code/data even when the electronic device is turned off (when power is removed), and while the electronic device is turned on that part of the code that is to be executed by the processor(s) of that electronic device is typically copied from the slower non-volatile memory into volatile memory (e.g., dynamic random access memory (DRAM), static random access memory (SRAM)) of that electronic device. Typical electronic devices also include a set or one or more physical network interface(s) to establish network connections (to transmit and/or receive code and/or data using propagating signals) with other electronic devices.

Video Streaming System in a Cloud Computing Environment

1 FIG.A 151 151 151 151 100 100 15 is a first embodiment of a video streaming system to support video stability prediction in a cloud computing environment. The cloud computing environment can include a set of cloud regionsA-C. A ‘set,’ as used herein can refer to any whole number of items including one item. These cloud regions can be different cloud computing environments provided by a same or different cloud computing provider. The cloud regionsA-C can have different geographic locations and electronic device components and resources that are interconnected via communication networks with the other cloud regionsA-C. Any number, configuration, and variety of cloud computing environments and regionsA-C can be utilized to provide the video streaming system. In some embodiments, some of the components of the video streaming systemare located in a set of electronic devices referred to as a point of presence (POP). The POPIA can be provided by an internet service provider or similar interface through which external resources and electronic devices can interact with the cloud computing environment.

100 197 153 157 159 197 197 130 130 130 100 197 130 190 197 In an example embodiment, the video streaming systemincludes a pre-processing stage, set of virtual clusters, a transcode cluster, and a publishing service cluster. The pre-processing stagecan be composed of any number and combination of electronic devices including servers, storage devices, networking devices and related devices that make up a cloud or similar distributed computing environment. The pre-processing stagecan support a set of processors that can operate on the input video sources. These processors can be software components that process the incoming video sourcesand are hosted by any type or distribution of electronic devices. The processors can perform tasks associated with each video sourceto process that video source and can be considered as a separate stage or can be combined with other stages of the video streaming system. Each processor of the stagecan perform a single task or multiple tasks and can operate in conjunction with other processes to pre-process the video sources. The processors can implement video stability prediction via video stability predictorsas well as perform any number of other processing tasks. Any number of the processors in the pre-processing stagecan support the video stability predictor function.

151 153 153 155 155 130 153 155 130 100 155 155 130 153 155 130 155 190 155 190 155 A pop/cloud regionA can include a set of virtual clusters. Each cluster can be composed of any number and combination of electronic devices including servers, storage devices, networking devices and related devices that make up a cloud computing environment. The virtual clusterscan support a set of video processors. These video processorscan software components that process the incoming video sourcesand are hosted by the virtual clusters. The video processorscan perform tasks associated with each video sourceto process that video source and can be considered a unit of work or a worker in the video streaming system. Each video processorcan perform a single task or multiple tasks and can operate in conjunction with other video processorsto process the video sources. Any number of virtual clusterscan manage any number of video processors, which together can process any number of video sources. Video processorscan implement video stability prediction via video stability predictorsas well as perform any number of other video processing tasks. While a single video processoris illustrated as providing a video stability predictorany number of the video processorscan support this function.

130 130 100 130 130 190 190 190 100 130 Video sourcescan be any type of video input in any format that is provided by any source device or set of devices. For example, video sources can be provided by a set of content providers, which are sets of devices operated by an entity that produces a video sourceto be processed by the video streaming system. In some cases, the video sourcescan be provided directly or indirectly by a set of capture devices such as live programming (e.g., sporting events). Any content provider, capture device, or similar set of devices providing the video sourcescan execute or be in communication with a device executing a video stability predictor. A video stability predictor, as further described herein, analyzes a video stream to predict near term future stability (e.g., potential video stream failures or interruptions). This video stability predictorprocess can be implemented at any number of points within the video streaming systemsuch that contextual information about the video can be utilized in making decisions and taking actions associated with the video source.

155 157 157 125 157 125 155 159 157 125 174 157 The output of the video processorscan be provided to a transcode cluster. The transcode clustercan further process the video sources to organize the video sources into a set of channels handled by associated channel nodes. The transcode clusterand channel nodescan combine video sources from the video processorsand encode the resulting video streams according to the configuration of the respective channels to be distributed to the publishing service cluster. A transcode clusteris a set of electronic device resources including servers, storage devices, and networking devices that support any number of channel nodes. Similarly, the publishing service cluster is a set of electronic device resources including servers, storage devices, and networking devices that support any number of video streams that are to be published to a content distribution networkor in some cases returned to an origin server (e.g., a video source provider or other entity that may publish the video stream). In some embodiments, the transcode clustercan output the video streams as segments and associated metadata. In some embodiments, the publishing service can format the video streams using video encoding formats M3u8, MPD, TS, MP4 and similar video encoding formats. As used herein video, audio/video, and similar terms are used interchangeably to include video formats that may also encompass audio aspects.

1 FIG.A 190 157 157 159 190 100 As shown in, a set of video stability predictorscan be executed at the transcode clusteror between the transcode clusterand the publishing service cluster. The video stability predictorsas described further herein, analyze incoming video stream information to generate a prediction of the future operation (i.e., stability) of the video stream. The prediction of future near term stability (e.g., in the range of milliseconds to hours) can be utilized to adjust the handling of the video stream in the video streaming platform. For example, if the stability of the video stream is predicted to have high latency or interruption, then the resources devoted to that video stream can be adjusted to maximize their use either by reassignment or by increasing resources to attempt to improve stability.

1 FIG.B 100 1 176 177 174 170 illustrates another embodiment of a video streaming system containing multiple video streaming platforms. A video streaming systemincludes multiple video streaming platforms represented by a streaming platformthrough a streaming platform N at referencesand, a content distribution network, and a streaming platform coordinator.

170 1 170 174 The streaming platform coordinatorcommunicates with all the video streaming platforms including the streaming platformsthrough N. The streaming platform coordinatorcoordinates processing of the media contents routed to the various videostreaming platforms. The processed media contents from the video sources are then published to the content distribution network.

It is to be noted that the various video streaming platforms and/or the streaming platform coordinator may be hosted by any one or more of various cloud computing providers. When two video streaming platforms are hosted by two different cloud computing providers, which generally offer computing resources with different characteristics, the two video streaming platforms are often referred to as heterogeneous video streaming platforms (versus homogenous video streaming platforms hosted by the same cloud computing providers). Cloud computing providers are building up their infrastructures at various geographic locations, and it is now practical for the video streaming system to utilize the cloud infrastructures concurrently at the various geographic locations and/or by different cloud computing providers.

190 190 1 190 174 Each video streaming platform may contain a video stability predictor, which is illustrated as a video stability predictorin streaming platforms-N, respectively. The video stability predictoris to generate a prediction for a set of video streams as to their stability that can be updated as the video stream is received from the video sources, caused by processing media workflows created for video sources in a video streaming platform as discussed in more details herein below, or in similar processing of the received media (i.e., video) as it is received from the video sources, handled by the streaming platforms, and forwarded via the content distribution network.

170 191 190 190 190 170 170 191 190 100 191 190 100 100 In some embodiments, the streaming platform coordinatormay contain a video stability managerthat manages the video stability predictorsincluding configuration of the video stability predictorsas well as training or metric collection for the video stability predictorsfrom processing media workflows in all video streaming platforms the streaming platform coordinatorinteracts with, and the streaming platform coordinatormay manage or coordinate notifying a feedback mechanism or corrective components to address any inaccuracies in the video stability prediction such that a retraining of the machine learning models associated with the video stability predictors can be initiated. The video stability managercan also coordinate resources related to the video stability predictorat different video streaming platforms. Where a feedback mechanism is available in the systemthe video stability manageror video stability predictorcan send notification of failures or context mismatches to the feedback mechanism that may report the issues component to retrain the machine learning models or to an administrator. In cases where corrective components are available in the system, then the reporting of issues with the video stability prediction can trigger machine learning model retraining and in some cases further processing to correct or ameliorate the identified video stability issues. The reporting of metrics related to video stability prediction can indicate the frequency, type, label, categorization, and similar information about the video stability prediction, identifying information about the media or video source and frames upon which the prediction has been generated as well as other relevant information for video stability management in the system.

Video Streaming Platform in a Cloud Computing Environment

100 176 100 130 132 174 130 132 190 176 190 190 176 130 132 1 FIG.B s A set of video streaming platforms is a main component of a video streaming systemas illustrated in. The video streaming platformsand the video streaming systemcan perform any number of operations to process any number of input video sources-and output via a content distribution networkto reach any number of consuming devices and software applications. The operations performed on the video sources-can include the content generation process implemented by video stability predictor. Each streaming platformcan execute any number of video stability predictorusing workers or similar mechanism as described further herein. The video stability predictorscan operate on a per streaming platform, per video source-or similar basis.

2 FIG. 200 170 200 200 200 The architecture of the video streaming platform and its operations are discussed in more detailed discussion with relation to the additional figures.illustrates a video streaming platform in a cloud computing environment according to one embodiment of the invention. A streaming platform(also referred to as a video streaming platform, and the two terms are used interchangeably in the specification) is a computing system, and it contains one or more machines including one or more server computers, gateways, routers, or other computing/networking electronic devices. A streaming platform coordinator (such as the streaming platform coordinator) manages operations of the streaming platform, yet some or all of the electronic devices within the streaming platformmay be owned by a third party such as a cloud computing provider discussed herein above. That is, a cloud computing environment operated by a cloud computing provider may host the streaming platform.

200 102 200 102 200 The streaming platformreceives its data flow input at a stream input interfacein one embodiment. For example, video sources to be processed by the streaming platformenters through the stream input interface. A video source contains one or more Internet Packet (IP) packet streams in one embodiment. The IP packet streams may contain one or more live video feeds. A live video feed may be video of a live event or live performance, or may be video of a prerecorded event being played back according to a schedule. The live video feed may be a video broadcasted over cable, satellite, or over-the-air. It is to be noted that the terms “video source,” “video stream,” and “video feed,” as used interchangeably herein, refer to the video and corresponding audio of the particular recorded event (e.g., TV show, live performance, sporting event, etc.), but also may include video only. Additionally, the video source (sometimes referred to as the video and audio streams) of the streaming platformmay contain only audio (e.g., an Internet radio stream). The video source may be a webcast of a television broadcast, such as of a sporting event, a live or recorded performance, a live or recorded news report, or the like. A live event may also have pre-recorded content intermingled with live media content, such as advertisements, which are played in between the live telecast. It should be noted that the embodiments of the invention described herein may also be used for streaming video-on-demand (VOD) and any other type or combination of pre-recorded audio/video content.

200 200 200 A video source may be “pushed” to the streaming platformwhere the video source is IP packet streams such as the Moving Picture Experts Group (MPEG)-transport streams (MPEG-TS). The IP packet streams logically flow to streaming platformfrom an external source thus the video source is referred to as being pushed to the streaming platform.

200 102 181 A video source may also be “pulled” by a processing unit (referred to as a worker) of streaming platform, where the worker runs one or more processing tasks. The worker may initiate a Transmission Control Protocol (TCP) connection to an external uniform resource identifier (URI) (an external uniform resource locator (URL) or an external uniform resource name (URN)), and after performing a protocol handshake, cause inbound IP packet streams to flow directly into the worker for one or more processing tasks without being processed by the optional stream input interfaceor the stream coordinator. The pull of video feeds may be implemented through the real time messaging protocol (RTMP), where the processing task includes a RTMP capture task.

102 200 200 102 180 180 181 182 The stream input interfaceis a logical input point for data flows into the streaming platform. It may not be present as a physical entity of the streaming platformin one embodiment. From the stream input interface, a video source becomes an incoming data flow. The incoming data flow contains data of one or more video and audio streams. In one embodiment, the incoming data flow is transmitted in user datagram protocol (UDP) packets. The incoming data flowmay optionally go to a stream coordinator, which converts unicast data flows into distributed data flows.

200 152 158 150 162 168 160 200 120 120 150 160 185 Workers may be organized as worker clusters in a streaming platform. In the streaming platform, workers-are in a primary worker cluster, which contains workers actively working on processing tasks. Workers-are in a backup worker cluster, which contains workers remains standby thus provides redundancy and robustness for the streaming platform. Workers perform tasks through coordination with one or more orchestrators, which may form an orchestrator cluster such as an orchestrator cluster. The orchestrator clusterinteracts with worker clusters-through one or more control flows, included in control and performance data flows.

120 122 124 126 200 126 122 124 122 124 120 120 122 124 120 170 1 FIG. The orchestrator clustercontains orchestrators-and an orchestrator databasethat stores data for operations of the orchestrators. The orchestrators may form load-balanced group within an orchestrator cluster, and the orchestrator cluster may be paired with another separately located orchestrator cluster (e.g., the other orchestrator cluster being at a different rack or even a different geographic location) for redundancy and robustness purpose too. An orchestrator creates a workflow for a video source in the streaming platform, and it may also host services responsible for work scheduling and overall system health monitoring and management. In some embodiments, the orchestrator databaseis optional. For example, each of the orchestrators-contain a distributed in-memory storage to store information for the operations by the orchestrator-and/or orchestrator cluster. In alternative, a database outside of the orchestrator clustermay store the information for the operations by the orchestrator-and/or orchestrator cluster(e.g., the database may be stored in a streaming platform coordinator such as the streaming platform coordinatorin).

182 184 184 109 200 102 109 200 Workers are coupled to one or more orchestrators, and the workers execute processing tasks on the distributed data flows. The data flows are processed and the workers produce output data flows. The output data flowsmay optionally transmit to a stream output interface, a logical output point for the data flows going out of the streaming platform. It is to be noted that both the stream input interfaceand the stream output interfacemay be integrated into parts of worker functions and they may not be individual physical units of the streaming platform.

112 Output data flows goes to video destinations, which contains one or more IP streams in one embodiment. The output data flows may be delivered to an ingest point of a content delivery network (CDN). A CDN is a system of computers networked together across the Internet that cooperates transparently to deliver content, and may include, for example, one or more origin content servers, web servers, cache servers, edge servers, etc. The output data flows may also be delivered to a video playback device directly. A single output data flow may be delivered to multiple destinations through multicast.

200 It is to be noted that both workers and orchestrators of the streaming platform may be implemented on cloud-hosted virtual machines (VMs). The VMs are parts of the cloud computing environment hosting the streaming platform and they reside on computing systems of the cloud computing environment. These computing systems are referred to as hosts of the workers and orchestrators in the streaming platform. The hosts are managed by a cloud provider and they may concurrently host applications other than the video streaming platform. Thus, the worker hosts are not dedicated to the streaming platform and they are allocated to the streaming platform as needed and according to coordination of the orchestrators.

120 191 190 191 190 200 185 191 190 200 200 191 120 191 122 124 191 126 It is to be noted that in some embodiments orchestrator clusteralso contains a content manageror video stability predictor. The content managermonitors the video stability predictorsin the streaming platformthrough collecting data from the workers (e.g., the performance data collected along with the control flows, as the control and performance data flows illustrated at reference) and receives updates on content generation as they occur. As content information is generated, the content managerand/or video stability predictorcan initiate a transfer of the content information to other components in the streaming platformthat may utilize or take action based on the context information (e.g., to an operator of the streaming platformand/or to a streaming platform coordinator). While the content manageris illustrated a standalone entity of the orchestrator cluster, the content generatormay be integrated with other entities such as orchestrators-. Additionally, a portion of the video stability managermay be within the orchestrator databasein one embodiment.

200 For the streaming platform, a graph of tasks is used to process a media workflow. A media workflow, also referred to as a workflow or channel (the terms workflow and channel are used interchangeably in the specification), represents a processing work flow that transforms an individual incoming data stream (e.g., a video source) into its configured output data stream(s), and it contains all of the necessary information used to create a directed task graph and to calculate the correct parameters for each task required in order to correctly transform the incoming data stream into the specified output data stream(s). During workflow creation, the orchestrator is responsible for compiling a channel definition (e.g., using the JavaScript Objection Notation (JSON) format) into a directed graph of tasks (referred to as a task graph) with associated configuration data and for assigning those tasks into logical groups (referred to as task groups) based on estimated resource requirements. The directed graph of tasks is a directed acyclic graph (DAG) of tasks for processing the video source. A DAG is a directed graph with no directed cycles. The directed graph is formed by a collection of nodes (also referred to as vertices) and directed edges, each edge connecting one node to another, such that there is no way to start at a node and follow a sequence of edges that eventually loops back to the node. Each node of the task graph represents a processing task, and each edge represents a data flow across two processing tasks and corresponding input and output of each processing task.

200 200 125 120 170 170 200 200 125 170 200 Mandatory parameters describing the type of the video source (e.g., MPEG-2, MPEG-4, H.265, and etc.), and location of the video source (e.g., ingest protocol, IP address, URI, and etc.). Indication of whether and how to enable subtitle processing and/or enable advertisement insertion processing for the video source. The desired video and audio transcoding operations (e.g., how many audio/video layers, the desired output characteristics for each such as video frame size/rate and bitrate, the relevant portion of the incoming data flow to use if applicable) for the video source. The desired contention protection operations for the published output (e.g., Microsoft© PlayReady, Adobe© Access DRM, AES-128 Encryption for HTTP live streaming, etc.). The desired publishing operations to output (e.g., which output format(s) such as HTTP live streaming (HLS), HTTP dynamic streaming (HDS), RTMP, or Microsoft® smooth streaming) to publish, and the destination(s) to send each output format. Overall, the streaming platformingests video sources, transcodes, and transforms the video sources into desired one or more formats for publication and then outputs the resulting video data. The video streaming platform is a distributed architecture using cloud resources, and it is a flexible, scalable, and efficient platform for video processing. The streaming platformreceives operator inputto the orchestrator cluster. The operational input may be from the streaming platform coordinator. The communication between the streaming platform coordinatorand the streaming platformmay include sending requests/confirmations from the streaming platform coordinator and updates/responds from the streaming platform. The operator inputmay also receive input from an operator separately from the streaming platform coordinator. The operator may receive input in the form of API calls. One of the requests from the streaming platform coordinator is a request to create a workflow for a video source in the streaming platform. The request (may be referred to as a channel creation request) may contain a variety of parameters describing the video source and the expected operations. For example, the request may contain at least one of the following:

120 110 200 184 Based on the request, the orchestrator clustercreates media workflows for video sources, utilizing directed graphs of tasks, and each of the so called task graphs is a directed acyclic graph (DAG) of tasks for processing the video source. Each task graph contains tasks to be performed by a worker of the streaming platform. The tasks are then assigned to workers for execution, and the results are included in the output data flows.

A media workflow contains a large number of tasks to be performed by a video streaming platform. An outside-in network management approach (e.g., SNMP), where the network management system can only collect performance data at a worker level, cannot provide efficient performance monitoring of the processing of the media workflow within the video streaming platform, let alone mitigate any macroblock detected with regard to the processing blocks in a timely fashion. For example, the worker is often implemented as a virtual machine in the video streaming platform, and using SNMP, an operator of the video streaming platform may determine a percentage of central processing unit (CPU) usage. The CPU usage may be too high (90%) for the worker, but without knowing the details of the processing of the media workflow, SNMP cannot determine the reason of the high CPU (e.g., it can be caused by malfunctioning of decoder, frame rate conversion, scaling, and/or video encoders), thus cannot provide effective mitigation

Operations of Video Stability Prediction in a Video Streaming Platform

The embodiments provide a process and system for video stability prediction. In particular, the embodiments provide transport stream stability prediction using temporal machine learning analysis based on priority faults reported in the video stream (e.g., faults of varying types and priority levels utilized in the MPEG transport stream (TS). The video stability predictor can perform the temporal analysis and prediction for each video transport stream managed by the video streaming platform. The video stability reliability can be determined using machine learning models with ETR 101 priority faults and seasonality patterns as features that are utilized by the machine learning models. The machine learning models can scale to predict video transport stream reliability over dynamic time windows in real time using long tail time series ETR 101 priority fault data.

The embodiments provide advantages over the prior art video stream monitoring processes and systems. Prior art monitoring systems use ETR 101 priority faults to monitor video transport streams. However, the priority faults only provide monitoring about current and past conditions and do not provide any analysis or prediction of the most probable failures in the near future. The embodiments provide such a prediction by analyzing historical transport streams priority faults and external seasonal stimuli that affect source reliability. Priority faults are recorded across time and the data by nature can be noisy. The number of priority faults provides a lot of data and finding patterns in the noisy data using manual statistical technologies is inefficient and difficult.

In the embodiments, the video stability predictor uses received priority faults, external seasonality stimuli, and video transport stream origin information to learn about the stability of the video transport stream over time. The embodiments derive statistical and stochastic features from priority faults. The video stability predictor uses feature parity, weight and efficacy to pick the most effective derived features to train a set of temporal machine learning models. These temporal machine learning models provide a probability score within a deviation margin across any dynamic time window. The embodiments further integrate stochastic thresholding to generate alerts on the probability score on a dynamic resolution window to provide video transport stream reliability prediction automatically across time.

The embodiments utilize machine learning to perform temporal analysis over the priority faults, seasonality stimuli, video transport stream origin, and similar features. In some embodiments, the machine learning is primarily based on priority faults. In other embodiments, the machine learning models incorporate weather, geographical origin of the source, internet paths taken, and similar features to make the machine learning model more robust. With more features, deep learning in the machine learning models increase the prediction time window and accuracy.

3 FIG. 100 100 301 is a flowchart of one embodiment of a process for predicting video stability. The process is illustrated at a high level to represent the overall function of the video stability prediction process in a video streaming system. The video streaming systemreceives media (e.g., a video source or stream) from a content producer or similar video source and selects a set of network packets from the session transporting the video source to be processed (Block). A sequential processing of the network packets is illustrated by way of example for clarity and conciseness, however, one skilled in the art would appreciate that aspects of the process could be performed in parallel or in alternate order. A network packet can be any type of data encapsulation using any format or protocol. In the example embodiments, the network packets are part of a DVB standardized process. However, the embodiments are also applicable to other similar technologies. The video stream received from content producers can be encoded in any format and can have any size or organization. In some cases, the video stream can be formatted to satisfy legal agreements between content producers and content distributors who stream the media on their platform (e.g., via a content delivery network (CDN)) and/or to meet the technical requirements of the distributors who stream the media on their platform.

303 Input network packets can be processed individually, in sets, batches, or similar groupings. A set of network performance metrics can be obtained for the session associated with the selected network packets (Block). Network performance metrics can be collected from the embedded information in the session, reports from downstream devices, network management software in the video streaming platform, or similar sources. Network performance metrics that are collected can include latency, packet loss, jitter, bandwidth utilization, and similar metrics. Each selected network packet can be inspected to retrieve priority fault errors (e.g., ETR 101 priority faults) or similar information. The priority faults are part of the video stream parameters (e.g., Continuity_count_error which is part of the MPEG-TS packet). The ETR 101 priority faults indicate various errors in the transport stream.

307 309 311 In some embodiments, a source or destination of the network packet is examined to determine a location or geo-reference associated with either or both of the source or destination (Block). IP addresses of the source or destination can be correlated with geographic locations based on IP address registrations and similar resources. Similarly, the geographic location of the path of the network packets, as well as the timing, date, and medium of the segments of the path can be determined is some embodiments (Block). This information can be correlated with weather patterns, network conditions, properties of the media, bandwidth allocation and similar information that may affect the transmission of the network packet along the path between the source and destination. In particular, any information that can affect the viability or network conditions of the path and the video stream can be collected. The collected data including the priority fault information can be collected as a time series of data for further analysis (Block). The time series can cover any window relative to the current network packet that is being analyzed. To prevent anomalous data from skewing predictions, any or all of the collected data can be smoothed by averaging, exponential smoothing, or similar techniques.

The video stability predictor can select a set of values of a set of features of the selected network packets consistent with the training of the machine learning model. Any type of machine learning model can be utilized. In some embodiments, a neural network (e.g., a deep learning neural network) or similar machine learning model can be utilized. The machine learning model can be trained on particular types of features and network packets (e.g., priority faults, network performance metrics, and similar features) at any level of granularity. Any provided information about a selected set of network packets can be utilized to select an appropriate trained machine learning model. In some embodiments, a special machine learning model for identifying types of network packets or for use with certain video sources can be utilized as a first step to select a more specifically trained machine learning model.

315 The trained machine learning model is provided the set of features and/or network packets as input (Block). Any number of network packets and any set of features can be selected for input to the trained machine learning model. Features that are selected and identified in the input frames can include priority faults, network performance metrics, source and destination locations, and similar information. The trained machine learning model generates video stability scores based on the input features. The input features can be selected for real time computation or extraction from the network packets and can be derived without significant computational latency such that they can be input into the machine learning model without significant delay to the handling of the network packets.

The video stability score can have a defined range (e.g., 0 to 100) that indicates the level of stability. Similarly, the probability or confidence score can also be generated that estimates the accuracy of the video stability score for the video stream. The video stability score can indicate a prediction of an immediate failure or issue (e.g., a 0 score) or can indicate stability over a prediction window (e.g., a 100 score) or variations thereof. Multiple video stability scores for different windows of time or different aspects of video stability can be generated by a given trained machine learning model or by application of a set of machine learning models.

In some embodiments, the probability or confidence score can be examined to determine whether the applied machine learning model requires retraining or whether a different machine learning model is to be applied. If the probability or confidence score of the video stability scores below a threshold level or outside or set of boundaries for acceptable probability or confidence levels, then the retraining or reselection of the applied machine learning model for generating the video stability scores can be triggered. In some embodiments, a confidence score that is out of bounds for a network packet, a set of network packet, an averaged probability or confidence score over many network packets, or similar metric can be utilized to determine whether the machine learning model is to be retrained or reselected. The retraining of the machine learning model can be similar to the initial training of the machine learning model as further described herein below. The retraining can either update an existing machine learning model or generate a new machine learning model. Retraining can shift the set of features selected for input into the machine learning model.

317 301 A check is made after the processing of each set of network packets to determine if there are more network packets to process for the video source (Block). As long as the video source is continuing to stream to the destination, then the processing of the video stream can continue (Block). If the video stream completes, then the process can end the video stability prediction process.

The video stability cores, probability or confidence scores, and similar information generated by the video stability predictor can be consumed by a video stability manager or other components of the video streaming platform. The information can be utilized to make decisions on video stream handling including reallocation of resources for a given video stream in response to identifying video stability issues in sets of network packets of the video streams. The video stability prediction process can be executed at any point between a video source and the publishing of the content into the content delivery network (CDN) including multiple instances of the video stability predictor that can use a shared set of machine learning models or separate instances and/or separately trained machine learning models to generate the video stability information at different points in the video streaming platform. In particular, the video stability prediction process can be employed at positions in the video streaming platform that can generate video stability information prior to the video processing of the video source to enable the optimal allocation of resource for the video source based on the video stability information prior to being assigned resources.

th The video stability prediction process can be used to identify video stability information for individual network packets or sets of network packets and/or to produce video stability information for a video stream in real time or in sufficient real time (e.g., within 1-200 ms). In some embodiments, the video stability prediction process can be applied to determine video stability information for every individual network packet of the video stream. However, in some other implementations, the method can be applied to assess every second/third/fourth/fifth/ . . . /nnetwork packet of the video stream or similarly spaced batches of network packets.

4 FIG. 100 100 is a flowchart of one embodiment of a process for training a machine learning model to perform video stability prediction. The illustrated process provides an example process for a video streaming system that includes machine learning training for video stability prediction. The process is illustrated at a high level to represent the overall function of the video streaming system. The video streaming system receives media (e.g., a video source or stream) from a content producer or similar video source. The training of the machine learning model can be implemented by the video stability predictor, video stability manager, or any other component such as a separate training component within the video streaming system.

100 401 The video streaming systemreceives media (e.g., a video source or stream) from a content producer or similar video source and selects a set of network packets from the session transporting the video source to be processed (Block). A sequential processing of the network packets is illustrated by way of example for clarity and conciseness, however, one skilled in the art would appreciate that aspects of the process could be performed in parallel or in alternate order. A network packet can be any type of data encapsulation using any format or protocol. In the example embodiments, the network packets are part of a DVB standardized process. However, the embodiments are also applicable to other similar technologies. The video stream received from content producers can be encoded in any format and can have any size or organization. In some cases, the video stream can be formatted to satisfy legal agreements between content producers and content distributors who stream the media on their platform (e.g., via a content delivery network (CDN)) and/or to meet the technical requirements of the distributors who stream the media on their platform.

403 Input network packets can be processed individually, in sets, batches, or similar groupings. A set of network performance metrics can be obtained for the session associated with the selected network packets (Block). Network performance metrics can be collected from the embedded information in the session, reports from downstream devices, network management software in the video streaming platform, or similar sources. Network performance metrics that are collected can include latency, packet loss, jitter, bandwidth utilization, and similar metrics. Each selected network packet can be inspected to retrieve priority fault errors (e.g., ETR 101 priority faults) or similar information. The ETR 101 priority faults are part of the packet header. The ETR 101 priority faults indicate various errors in the transport stream.

407 409 415 In some embodiments, a source or destination of the network packet is examined to determine a location or geo-reference associated with either or both of the source or destination (Block). IP addresses of the source or destination can be correlated with geographic locations based on IP address registrations and similar resources. Similarly, the geographic location of the path of the network packets, as well as the timing, date, and medium of the segments of the path can be determined is some embodiments (Block). This information can be correlated with weather patterns, network conditions, properties of the media, bandwidth allocation and similar information that may affect the transmission of the network packet along the path between the source and destination. In particular, any information that can affect the viability or network conditions of the path and the video stream can be collected. The collected data including the priority fault information can be collected as a time series of data for further analysis (Block). The time series can cover any window relative to the current network packet that is being analyzed. To prevent anomalous data from skewing predictions, any or all of the collected data can be smoothed by averaging, exponential smoothing, or similar techniques. For training purposes, the time series data can have a long history to better establish patterns for predicting video stability issues.

413 Once the time series data is collected and prepared, then a set of machine learning models can be trained against the time series data (Block). The training process can select a set of values of a set of features of the selected network packets which can be varied for the training of the set of machine learning models. Any type of machine learning model can be utilized. In some embodiments, a neural network (e.g., a deep learning neural network) or similar machine learning model can be utilized. Each machine learning model can be trained on a different set of features and/or network packets (e.g., priority faults, network performance metrics, and similar features) at any level of granularity. Any provided information about a selected set of network packets can be utilized to train a machine learning model.

Any number of network packets and any set of features can be selected for use in training each machine learning model. Features that are selected and identified for the network packets can include priority faults, network performance metrics, source and destination locations, and similar information. The trained machine learning model generates video stability scores based on the input features. The input features can be selected for real time computation or extraction from the network packets and can be derived without significant computational latency such that they can be input into the machine learning model without significant delay to the handling of the network packets.

The video stability score can have a defined range (e.g., 0 to 100) that indicates the level of stability. Similarly, the probability or confidence score can also be generated that estimates the accuracy of the video stability score for the video stream. The video stability score can indicate a prediction of an immediate failure or issue (e.g., a O score) or can indicate stability over a prediction window (e.g., a 100 score) or variations thereof. Multiple video stability scores for different windows of time or different aspects of video stability can be generated by a given trained machine learning model or by application of a set of machine learning models.

In some embodiments, the probability or confidence score can be examined to determine whether the applied machine learning model requires retraining or whether a different machine learning model is to be applied. If the probability or confidence score of the video stability scores below a threshold level or outside or set of boundary for acceptable probability or confidence levels, then the retraining or reselection of the applied machine learning model for generating the video stability scores can be triggered. In some embodiments, a confidence score that is out of bounds for a network packet, a set of network packet, an averaged probability or confidence score over many network packets, or similar metric can be utilized to determine whether the machine learning model is to be retrained or reselected. The retraining of the machine learning model can be similar to the initial training of the machine learning model as described above. The retraining can either update an existing machine learning model or generate a new machine learning model. Retraining can shift the set of features selected for input into the machine learning model.

415 After the set of machine learning models is trained a validation process can be performed (Block). The validation process can check whether each machine learning model is operating correctly. Similarly, the machine learning model accuracy can be tested such that those models that have the best performance can be utilized for generating video stability predictions.

The training process can be performed by the video stability predictor, video stability, manager, or specialized training components. The training process can be implemented in any portion of the video streaming platform.

th The training process can utilize each individual input network packet or sets of network packets. However, in some other implementations, the training method can be applied to assess every second/third/fourth/fifth/ . . . /nnetwork packet of the video stream or similarly spaced batches of network packets.

Electronic Devices Implementing Embodiments of the Invention

5 FIG. 500 590 500 500 is a block diagram illustrating an electronic device that may serve as a video stability predictor of a video streaming platform in a cloud computing environment according to one embodiment of the invention. The electronic device may be a computing device (e.g., a computer server) of a cloud computing environment). The systemmay represent the video stability predictorand or video stability manager described above performing any of the processes or methods for training machine learning models and applying the trained machine learning models to generate context information in real time in a video streaming system described above. The systemcan include many different components. These components can be implemented as integrated circuits (ICs), portions thereof, discrete electronic devices, or other modules adapted to a circuit board such as a motherboard or add-in card of a computing system, or as components otherwise incorporated within a chassis of the computing system. Note also that the systemis intended to show a high level view of many components of the computing system. However, it is to be understood that additional components may be present in certain implementations and furthermore, different arrangement of the components shown may occur in other implementations.

500 501 503 504 508 510 501 501 501 501 In one embodiment, the systemincludes a processor, memory, and optionally device units-that are interconnected via a bus or an interconnect. A processormay represent a single processor or multiple processors with a single processor core or multiple processor cores included therein. The processormay represent one or more general-purpose processors such as a microprocessor, a central processing unit (CPU), or processing device. More particularly, the processormay be a complex instruction set computing (CISC) microprocessor, reduced instruction set computing (RISC) microprocessor, very long instruction word (VLIW) microprocessor, or processor implementing other instruction sets, or processors implementing a combination of instruction sets. Processormay also be one or more special-purpose processors such as an application specific integrated circuit (ASIC), a cellular or baseband processor, a field programmable gate array (FPGA), a digital signal processor (DSP), a network processor, a graphics processor, a network processor, a communications processor, a cryptographic processor, a co-processor, an embedded processor, or any other type of logic capable of processing instructions.

501 503 503 503 501 503 501 The processormay communicate with the memory, which in an embodiment can be implemented via multiple memory devices to provide for a given amount of system memory. The memorymay include one or more volatile storage (or memory) devices such as random access memory (RAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), static RAM (SRAM), or other types of storage devices. The memorymay store information including sequences of instructions that are executed by the processor, or any other device units. For example, executable code and/or data of a variety of operating systems, device drivers, firmware (e.g., input output basic system or BIOS), and/or applications can be loaded in the memoryand executed by the processor. An operating system can be any kind of operating systems, such as, for example, Windows® operating system from Microsoft®, Mac OS®/iOS® from Apple, Android® from Google®, Linux®, Unix®, or other real-time or embedded operating systems such as VxWorks.

503 590 590 501 590 The memorycontains a video stability predictor, manager, or related components for training, which may contain instructions to perform the operations of these components as discussed herein above. The video stability predictorand related components may contain functional blocks that implement functions as described herein with relation to the video stability prediction process and related training processes discussed herein above. The processormay instantiate the video stability predictorand related components to perform operations to as discussed herein above.

500 504 508 504 505 506 507 508 505 500 The systemmay optionally further include input/output (VO) devices such as the device units-, including display control and/or display device unit, wireless transceiver(s), video VO device unit(s), audio VO device unit(s), and other VO device unitsas illustrated. The wireless transceivermay be a WiFi transceiver, an infrared transceiver, a Bluetooth transceiver, a WiMax transceiver, a wireless cellular telephony transceiver, a satellite transceiver (e.g., a global positioning system (GPS) transceiver), or other radio frequency (RF) transceivers, or a combination thereof. The systemmay also include an ultrasound device unit (not shown) for transmitting a conference session code.

506 507 508 508 510 500 The video VO device unitmay include an imaging processing subsystem (e.g., a camera), which may include an optical sensor, such as a charged coupled device (CCD) or a complementary metal-oxide semiconductor (CMOS) optical sensor, utilized to facilitate camera functions, such as recording photographs and video clips and conferencing. An audio VO device unitmay include a speaker and/or a microphone to facilitate voice-enabled functions, such as voice recognition, voice replication, digital recording, and/or telephony functions. Other optional devicesmay include a storage device (e.g., a hard drive, a flash memory device), universal serial bus (USB) port(s), parallel port(s), serial port(s), a printer, a network interface, a bus bridge (e.g., a PCI-PCI bridge), sensor(s) (e.g., a motion sensor such as an accelerometer, gyroscope, a magnetometer, a light sensor, compass, a proximity sensor, etc.), or a combination thereof. The optional device unitsmay further include certain sensors coupled to the interconnectvia a sensor hub (not shown), while other devices such as a keyboard or thermal sensor may be controlled by an embedded controller (not shown), dependent upon the specific configuration or design of the system.

500 500 170 190 500 2 FIG. 1 FIG. 3 4 FIGS.and The systemmay be coupled to an orchestrator in an orchestrator as illustrated in. Additionally, the systemmay be integrated within a streaming platform coordinator, similar to the video stability predictorillustrated in. The systemmay perform methods discussed herein above relating to.

500 Note that while the systemis illustrated with various components, it is not intended to represent any particular architecture or manner of interconnecting the components; as such details are not germane to embodiments of the present invention. It will also be appreciated that an electronic device having fewer components or perhaps more components may also be used with embodiments of the invention.

Some portions of the preceding detailed descriptions have been presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the ways used by those skilled in conferencing technology to most effectively convey the substance of their work to others skilled in the art. An algorithm is here, and generally, conceived to be a self-consistent sequence of operations leading to a desired result. The operations are those requiring physical manipulations of physical quantities.

It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless specifically stated otherwise as apparent from the above discussion, it is appreciated that throughout the description, discussions utilizing terms such as those set forth in the claims below, refer to the action and processes of a conference device, or similar electronic computing device, that manipulates and transforms data represented as physical (electronic) quantities within the conference device's registers and memories into other data similarly represented as physical quantities within the conference device's memories or registers or other such information storage, transmission or display devices.

It is to be noted that the operations of the flow diagrams are described with reference to the exemplary embodiment electronic devices. However, it should be understood that the operations of flow diagrams can be performed by embodiments of the invention other than those discussed with reference to the electronic devices, and the embodiments discussed with reference to the electronic devices can perform operations different than those discussed with reference to the flow diagrams.

While the flow diagrams in the figures herein above show a particular order of operations performed by certain embodiments of the invention, it should be understood that such order is exemplary (e.g., alternative embodiments may perform the operations in a different order, combine certain operations, overlap certain operations, etc.).

While the invention has been described in terms of several embodiments, those skilled in the art will recognize that the invention is not limited to the embodiments described, can be practiced with modification and alteration within the spirit and scope of the appended claims.