Selective user plane monitoring multiple monitoring probes when a serving gateway has multiple IP addresses

This application is a Continuation of U.S. patent application Ser. No. 15/808,762 filed Nov. 9, 2017, which claims priority to U.S. Patent Application Ser. No. 62/565,946 filed Sep. 29, 2017 which is incorporated herein by reference in its entirety.This application is a reissue of U.S. Pat. No. 11,228,929, issued on Dec. 28, 2021, which is a Continuation of U.S. patent application Ser. No. 15/808,762, filed Nov. 9, 2017, which claims priority to U.S. patent application Ser. No. 62/565,946, filed on Sep. 29, 2017, which patent and applications are incorporated herein by reference in their entirety.

The disclosed embodiments generally relate to monitoring network traffic data, and more particularly, to performing selective user plane monitoring using multiple monitoring probes when a serving gateway has multiple IP addresses.

As a result of advances in technology and enormous increases in the number of wireless device us, the size and complexity of wireless communications networks has greatly increased. One consequence of such increases in size and complexity is that the relative increase in operational and performance problems associated with communications networks has also increased. Reliability issues, such as dropped calls, lack of coverage, poor audio quality, and application failure often lead to user frustration and to increased costs. As new services are introduced that use even more complex technology, exercise different usage modalities, and place additional demands on networks, network performance continues to be a prime concern. In fact, quality of service often has a direct impact on a service provider's profitability. Therefore, improving quality of service is a top priority for service providers.

Network monitoring solutions are well known in the art and widely employed by service providers. Typical approach to network monitoring includes placing probes at various points in the network to determine if network elements are functioning according to specification. Sometimes referred to as “sniffers”, “log monitors” or “event monitors,” these monitoring systems are effective at identifying performance issues with a particular network element, but they often fail to capture problems that stem from the interfaces among network elements.

In a practical network monitoring implementation, however, not all packets that pass through a particular monitoring point are examined in detail, but rather they are sampled (for example, one in every thousand data packets that flow through a sampling point may be examined in detail). In such a scenario, because the various metrics for any given flow are now necessarily expressed as estimates with a mean and a variance, taking the maximum of the data counts will result in an upward or downward bias in the estimated metrics.

The solutions that are currently available can only monitor and diagnose subsets of the overall telecommunications system in limited ways rather than providing a holistic view of network and device performance that may be needed to efficiently identify and resolve quality issues. Accordingly, there is a further need in the art for a reliable and flexible network monitoring approach that processes all relevant data for all subscribers.

Certain aspects of the present disclosure relate to monitoring network traffic data.

In accordance with a purpose of the illustrated embodiments, in one aspect, a system for selective user plane monitoring in a wireless network includes a service gateway (SGW) having a plurality of units. Each unit has a unique IP address. The system further includes a network packet broker configured to receive a plurality of packets including user plane data from one or more tunnels created to enable direct transmission of the user plane packets from user equipment to the plurality of SGW units. The network packet broker is also configured to receive a plurality of packets including control plane data from one or more channels created to enable transmission of the control plane packets from a base transceiver station to the SGW. The system also includes a plurality of monitoring probes operatively coupled to the network packet broker. The monitoring probes are configured to generate a first plurality of metrics associated with the received control plane packets and configured to selectively generate a second plurality of metrics associated with the received user plane packets based on one or more identifiers. The network packet broker is configured to forward a plurality of user plane packets being processed by a particular SGW unit to a particular monitoring probe of the plurality of probes.

In another aspect, a method for selective user plane monitoring in a wireless network includes receiving, by a network packet broker, a plurality of packets including user plane data from one or more tunnels created to enable direct transmission of the user plane packets from user equipment to a service getaway (SGW) comprising a plurality of units. Each of the plurality of SGW units has a unique IP address. A plurality of packets including control plane data is received by the network packet broker from one or more channels created to enable transmission of the control plane packets from a base transceiver station to the plurality of units of the SGW. The received plurality of user plane packets and the received plurality of control plane packets are transmitted by the network packet broker to a plurality of monitoring probes operatively coupled to the network packet broker. Each subset of user plane packets representing packets being processed by a particular SGW unit is transmitted to a particular monitoring probe of the plurality of probes. A first plurality of metrics associated with the received control plane packets is generated by the plurality of monitoring probes. A second plurality of metrics associated with the received user plane packets is selectively generated by the plurality of monitoring probes based on one or more identifiers.

The illustrated embodiments are not limited in any way to what is illustrated as the illustrated embodiments described below are merely exemplary, which can be embodied in various forms, as appreciated by one skilled in the art. Therefore, it is to be understood that any structural and functional details disclosed herein are not to be interpreted as limiting, but merely as a basis for the claims and as a representation for teaching one skilled in the art to variously employ the discussed embodiments. Furthermore, the terms and phrases used herein are not intended to be limiting but rather to provide an understandable description of the illustrated embodiments.

Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Although any methods and materials similar or equivalent to those described herein can also be used in the practice or testing of the illustrated embodiments, exemplary methods and materials are now described.

It must be noted that as used herein and in the appended claims, the singular forms “a”, “an,” and “the” include plural referents unless the context clearly dictates otherwise. Thus, for example, reference to “a stimulus” includes a plurality of such stimuli and reference to “the signal” includes reference to one or more signals and equivalents thereof known to those skilled in the art, and so forth.

It is to be appreciated the illustrated embodiments discussed below are preferably a software algorithm, program or code residing on computer useable medium having control logic for enabling execution on a machine having a computer processor. The machine typically includes memory storage configured to provide output from execution of the computer algorithm or program.

As used herein, the term “software” is meant to be synonymous with any code or program that can be in a processor of a host computer, regardless of whether the implementation is in hardware, firmware or as a software computer product available on a disc, a memory storage device, or for download from a remote machine. The embodiments described herein include such software to implement the equations, relationships and algorithms described above. One skilled in the art will appreciate further features and advantages of the illustrated embodiments based on the above-described embodiments. Accordingly, the illustrated embodiments are not to be limited by what has been particularly shown and described, except as indicated by the appended claims.

In exemplary embodiments, a computer system component may constitute a “module” that is configured and operates to perform certain operations as described herein below. Accordingly, the term “module” should be understood to encompass a tangible entity, be that an entity that is physically constructed, permanently configured (e.g., hardwired) or temporarily configured (e.g. programmed) to operate in a certain manner and to perform certain operations described herein.

In the context of the present description, the terms “network” and “communication network” refer to the hardware and software connecting one or more communication elements including wireline networks, wireless networks, and/or combinations thereof.

is a diagram illustrating elements or components of an example operating environment in which embodiments of the present invention may be implemented. It is noted that the access network ofis merely one example of a possible access network, and embodiments may be implemented in any of various access networks, as desired.

As shown, the exemplary wireless networkincludes one or more eNodeBswhich communicate over a transmission medium (LTE Uu interface) with one or more user devices. User devices may be referred to herein as “user equipment” (UE). Thus, the user devicesare referred to as UEs or UE devices. UE devicesmay comprise any of various types of computer systems or devices which are mobile or portable and which perform wireless communications. Examples of UE devices include mobile telephones or smart phones (e.g., iPhone™, Android™-based phones), portable gaming devices (e.g., Nintendo DS™, Play Station Portable™, Gameboy Advance™, iPhone™), laptops, PDAs, portable Internet devices, music players, data storage devices, other handheld devices, as well as wearable devices such as wrist-watches, headphones, pendants, earpieces, etc. In general, the term “UE” or “UE device” can be broadly defined to encompass any electronic, computing, and/or telecommunications device (or combination of devices) which is easily transported by a user and capable of wireless communication.

In addition to one or more eNodeBs, the access networkmay include a mobility management entity (MME)and a serving gateway (SGW), among other components. Whiledepicts a couple of eNodeBsand a single MMEand SGW, in other implementations,may include more than two eNodeBs, as well as multiple MMEs, SGWs, etc.

eNodeBsmay include one or more devices (e.g., base stations) and other components and functionality that allow UEto wirelessly connect to access network. eNodeBsmay include or be associated with one or more cells. For example, each cell may include a radio frequency (RF) transceiver facing a particular direction. eNodeBsmay interface with access networkvia an interface referred to as an S1 interface, which may be split into a control plane S1-MME interfaceand a data plane S1-U interface. S1-MME interfacemay interface with MME device. S1-MME interfacemay be implemented, for example, with a protocol stack that includes a Network Access Server (NAS) protocol and/or Stream Control Transmission Protocol (SCTP). An S1-U interfacemay interface with SGWand may be implemented, for example, using a General Packet Radio Service Tunneling Protocol version 2 (GTPv2).

MMEmay implement control plane processing for access network. For example, MMEmay implement tracking and paging procedures for UE, may activate and deactivate bearers for UE, may authenticate a user of UE, and may interface to non-LTE radio access networks. A bearer may represent a logical channel with particular quality of service (QoS) requirements. MMEmay also select a particular SGWfor a particular UE. A particular MMEmay interface with other MMEs in access networkand may send and receive information associated with UEs, which may allow one MME device to take over control plane processing of UEs serviced by another MME, if the other MME becomes unavailable.

SGWroutes and forwards user data packets. SGWacts as the mobility anchor for the user plane during inter-eNodeBhandovers. As noted above, SGWand eNodeBscommunicate over one or more LTE S1-U linksa-c. SGWand MMEcommunicate over an LTE S11 link.

In general, SGWmay be implemented by a single server, multiple distributed servers, and multiple clustered servers. Optionally, some of the SGW server functions may be provided by a separate server or servers. In the configuration shown in, a blade server is employed to implement SGW. The blade server includes multiple server bladesthat are installed in a common rack or chassis, with each server bladefunctioning as a separate SGW, each preferably with its own processor(s), memory, network interfaces and its own IP addresses. In one implementation, SGWis configured to load balance user plane sessions across multiple internal SGW blades.

The UEtypically generates and transmits a request to create a session to the MMEover the S1-MME interface. In response to receiving the request, the MMEgenerates and transmits a create session request to the SGW. The MMEcommunicates the create session request to the SGWover S11 interfacethat is adapted to accommodate the request.

SGWreceives the create session request and selects one of SGWs among a plurality of internal SGWs. As noted above, each SGWmay be implemented by a separate server blade having its own unique IP address. SGWgenerates a create session response, which includes an IP address for the selected internal SGW. SGWforwards the response to the MME, which transmits the create session response including the IP address for the selected SGW blade to the e-NodeBover the S1-MME interface. The eNodeBcommunicates to the UEthat the attach procedure is completed. Thereafter, the eNodeBroutes packets between the UEand the IP address included in the received create session response via a corresponding S1a-U interface linka-c.

is a diagram illustrating selective user plane monitoring using a single monitoring probe when a serving gateway has a single IP address, according to an embodiment of the present invention. In this embodiment, SGWis implemented using a single server having a single IP address. For illustrative purposes only assume that the hardware capacity limit of the SGW(the maximum throughput in bps that this SGW can handle) is 80 Gbps. In this scenario, the S1-U interfacecarries the user plane traffic between the eNodeBand the SGW. In this embodiment, the user plane IP traffic is carried within encapsulated unidirectional tunnels (e.g., GTP-U) tunnels for specific user's flows. Each user flow has a unique tunnel identifier. The user plane IP traffic is embedded within transport layer addresses (i.e., source and destination IP addresses of the network devices). In one embodiment, the user plane IP traffic is carried within 8 links, each link having maximum throughput of 10 Gbps.

In addition to components-of the access network,includes components that facilitate monitoring of traffic in the wireless network.

More specifically, in this embodiment, one Network Packet Broker (“NPB”)is incorporated into the networkto facilitate processing of data packets and/or to route data packets to/from network monitoring tools. These monitoring tools may include, for example, network analysis tools, forensic tools, various network monitoring tools, firewalls, malware prevention tools, intrusion detection tools, etc. NPBrepresents hardware and/or software modules that perform, among other tasks, aggregation of monitored traffic from multiple links, filtering and grooming of traffic to relieve overburdened monitoring tools, load-balancing traffic across a pool of monitoring tools, and regeneration of traffic to multiple monitoring tools. In one embodiment, NPBuses a plurality of taps (not shown in) to capture traffic transmitted within the access network. The taps are connections to the various transmission links (e.g., GTP-U tunnels) and are designed to copy packets transmitted through the links and provide the copied packets to the network probe(through the NPB) so that the probecan monitor the performance of the network and the quality of the user experience by analyzing the packets. The taps, such as passive taps, filterable taps, and others, include, but are not limited to XFP ports, UTP ports, SFP ports, or other similar means of connecting to the S1a-U interface transmission linkand capturing the transmitted packets for analysis.

The NPBshown inincludes a plurality of input portsb-n configured to receive user plane data packetsfrom the plurality of tunnels that carry the user plane IP traffic. In addition, the NPBincludes a separate input porta configured to receive control plane data packetsfrom the S11 interface. In this embodiment each input porta-n has maximum throughput of 10 Gbps.

According to this embodiment, the NPBaggregates both the received user plane data packetsand all of the received control plane packetsand forwards the aggregated packets to monitoring tools via two output portsa andb. It is noted that each output porta-b has maximum throughput of 40 Gbps. In other words, the NPBsends all received monitoring load including all received user plane packetsand all received control plane packetsto monitoring tools.

Generally speaking, currently existing packet brokers can perform basic packet aggregation functions as well as various more advanced packet processing functions on the traffic received from taps, such as removing protocol headers, filtering/classifying packets based on configured rules, and so on. Packet brokers can then transmit the processed traffic to a number of analytic probes/tools, which can carry out various types of calculations and analyses on the traffic in accordance with the business goals/purposes of network (e.g., calculation of KPIs, detection of security threats/attacks in core network, generation of reports, etc.).

Typically, network implementations use dedicated networking hardware (e.g., hardware comprising custom application-specific integrated circuits (ASICs) and/or field-programmable gate arrays (FPGAs)) in order to implement packet brokers. For instance, according to one known approach, a packet broker can be implemented using a device comprising at least one line card that includes an ASIC or FPGA-based packet processor. When the packet broker device receives replicated traffic from taps, the packet broker processes the traffic using the packet processors based on rules that are programmed into hardware memory tables (e.g., content-addressable memory (CAM) tables) resident on the packet processors and/or line cards. The packet broker then forwards the processed traffic onward to probes/toolsfor analysis.

This hardware-based approach for implementing advanced functionality of a packet broker suffers from several drawbacks. First, the scalability of monitoring network—in other words, its ability to process increasing volumes of traffic from a core network—is necessarily constrained by the hardware capabilities of a packet broker. For example, if a packet broker is implemented using a chassis-based router that supports up to X line cards, the packet broker cannot scale to support traffic volumes that exceed the capabilities of those X line cards.

Second, since hardware-based packet brokers perform packet processing based on rules that are programmed into hardware memory tables, these packet brokers are generally limited to executing packet processing operations that conform to the semantics of such rules (i.e., attempt to match one or more header fields of an incoming packet and then perform a specified action if a match is made). While these operations can be chained by passing the output of one packet processor/line card to another, the order of the chaining is fixed in hardware. This rigid processing paradigm is problematic if more flexible, complex, and dynamic packet processing operations are desired.

Third, implementing feature changes or additions to a hardware-based packet broker can be costly and time-consuming due to the need to design and validate such changes at the hardware level. This, in turn, makes it more difficult to quickly iterate the packet broker in response to evolving customer needs/requirements and increases the cost of the device for those customers.

Advantageously, various embodiments of the present invention address the aforementioned drawbacks by moving advanced packet processing functions from the hardware-based NPBto a plurality of software-based monitoring tools. This significantly reduces the overall cost of the monitoring infrastructure, and at the same time provides more flexible and dynamic packet processing operations customized to serve specific needs of network operators. Thus, various embodiments described herein contemplate utilization of NPBswith hardware implementations limited to basic packet aggregation functions.

In the illustrated embodiment, a monitoring tool is implemented as a network probe. The network probemay be configured as hardware, software, firmware, or combination thereof, for monitoring data transmission through the S1a-U interfaceand the S11 interface. While generally the network probemay be deployed at any of a variety of locations or links of the networkto collect network data packets transmitted through the S1a-U interfaceand the S11 interface, in this case the network probesare connected to the network through the NPBwhich, as described in another embodiment below, may be configured to permit balancing of the load between multiple network probes. The network probemay be configured as a special purpose computing device or a software component (not limited to a single process) dedicated to monitoring data communicated via the network. Alternatively, the network probemay be a general purpose computing device with specialized software components installed thereon. In one embodiment, the network probeis a nGenius InfiniStream, available from NetScout Systems, Inc. of Westford, Mass.

In this example, after collecting the packets (user plane and control plane), the network probeselectively generates an Adaptive Service Intelligence (ASI) data set that can include key performance indicators and Adaptive Session Records (“ASRs”) as described in U.S. patent application Ser. No. 12/756,638 entitled “Real-Time Adaptive Processing of Network Data Packets for Analysis” and filed on Apr. 8, 2010. The methods and systems described in U.S. patent application Ser. No. 12/756,638 enable the network probeto analyze network performance and end user experience in real-time, without extensive network buffering and analysis.

In one embodiment, the network probeincludes two Network Interface Controller (NIC) portsa-b and has maximum throughput of at least 80 Gbps. In other words, the maximum throughput of the entire SGWis less than or equal to the maximum throughput of the network probe.

Furthermore, according to an embodiment of the present invention, the network monitoring probeis configured to generate basic KPIs and meta data metrics for all subscribers and for the entire control and user plane traffic and configured to selectively generate advanced KPIs and metadata only for selected subscribers on user plane by using internal filtering techniques. Basic KPIs and meta data metrics may include Key Traffic Indicators (KTIs), such as, but not limited to, volume, utilization rate, and throughput metrics. The advanced set of metrics typically provides additional assistance in troubleshooting and may include, but is not limited to, the following metrics: network errors, network congestion, jitter, latency, transaction response times, timeouts, as well as various quality of service characteristics (e.g., for video and Voice over IP traffic). Keeping track of these metrics typically requires more processing power, thus selective generation of such metrics facilitates better management of resources on each probe. Exact arrangement of various advanced metrics typically depends on the type of network traffic flows being analyzed. Keeping track of these metrics typically requires more processing power, in contrast to the basic KPIs and metadata metrics.

According to an embodiment of the present invention, a list of subscribers (e.g., a whitelist) that requires additional monitoring (advanced set of KPIs) may be either received by the network probefrom a different network management system or may be preconfigured at the network probe. Examples of subscriber entity identifiers include, but are not limited to, an international mobile equipment identifier (IMEI) and an international mobile subscriber identifier (IMSI). It should be noted that in some embodiments, the list may include a criteria by which subscriber entities may be identified, such as location, handset and application. For example, the provided list may include only application identifiers, in which case the network probeis configured to generate the advanced set of metrics for all subscribers using the specified application (e.g., Facebook). Location identifiers may indicate geographic locations associated with one of eNodeBs.

A further advantage of example embodiments is that the monitoring of a subscriber's media experience may be adjusted based on the subscriber's priority, class, subscription level, and so forth. Therefore, if a certain subset of subscribers is paying for a premium experience, it may be critically important to ensure that their service is not impacted by various traffic conditions as compared to those who are paying for a lower experience or are not paying at all.

is a diagram illustrating selective user plane monitoring using multiple monitoring probes when a serving gateway has multiple IP addresses, according to yet another embodiment of the present invention. In this embodiment, SGWis implemented using a blade server. In other words, SGWincludes multiple server bladesa-c that are installed in a common rack or chassis, with each server blade functioning as an internal SGW, each with its own IP address and each performing a subset of SGW functionality. In this embodiment, the user plane IP traffic transmitted between the SGWand the eNodeBsis carried within 3 different links, each link having maximum throughput of 10 Gbps.

As explained above in conjunction with, the SGW, in response to receiving a create session request message, selects one of the blades according to a predefined criteria and generates a create session response, which includes an IP address for the selected internal SGWa-c. The SGWforwards the response to the MME, which transmits the create session response including the IP address for the selected internal SGW blade to the e-NodeBover the S1-MME interface. Thereafter, the eNodeBroutes packets between a particular UEand the IP address included in the corresponding create session response via a corresponding S1a-U interface link. In other words, all data plane packets associated with a particular subscriber are routed to a dedicated internal SGWa-c.

The NPBshown inincludes input portsb-d configured to receive user plane data packetsfrom the plurality of S1-U interface links that carry the user plane IP traffic. In addition, the NPBincludes a separate input porta configured to receive control plane packetsfrom the S11 interface. In this embodiment each input porta-d also has maximum throughput of 10 Gbps.

According to this embodiment, the NPBaggregates both the received user plane data packetsand all control plane packetsfor SGW. However, in contrast to embodiment shown in, the NPBperforms load balancing between various network probesa-c. In one embodiment, the NPBseparates the user plane traffic by IP addresses of the internal SGW bladesa-c preferably using IP address filtering methods. In other words, the NPBmay route all data packets being handled by the first SGW bladea to a first network probea via output portsa andb, all data packets being handled by the second internal SGW bladeb to a second network probeb via output portsc andd and may route all data packets being handled by the third internal SGW bladec to a third network probec via output portse andf. It should be noted that the entire load of the aggregated control plane packetsis forwarded to all network probesa-c y the packet brokerfor correlation purposes. In other embodiments, other criteria may be utilized for load balancing the user plane traffic. In yet other embodiments, the NPBmay send the entire user plane traffic to each network probea-c.

According to an embodiment of the present invention, all the correlations performed collectively by the plurality of network probesa-c can be seen as a sequence of control plane/user plane and control plane/control plane correlations where the control plane/user plane correlation can use, for example, subscriber entity attributes and tunnel identifiers for correlation purposes. According to an embodiment of the present invention, whenever possible, the network probesa-c attempt to correlate the packets based on a tunnel identifier/subscriber entity identifier mapping that can be extracted from the control plane traffic. This correlation is possible only if the subscriber identifier is present in all packets being correlated. Advantageously, in this embodiment the network probesa-c receive only a subset of user plane data, but because they also receive all control plane packetscontaining mappings between subscriber identifiers and tunnel identifiers, there is guaranteed to be a match for control plane/user plane correlation purposes.

In this embodiment, the maximum throughput of each individual SGW bladea-c is less than or equal to the maximum throughput of the corresponding network probea-c. For example, if each SGW bladea-c has maximum throughput of 10 Gbps, the network probesa-c having maximum throughput of 80 Gbps can be used in this configuration. As noted above, each network probea-c processes user plane traffic for a specific SGW bladea-c and each network probea-c processes the entire control plane traffic.

In an alternative embodiment, each network probea-c may receive the entire user plane traffic and may perform internal load balancing by selecting a subset of the received user plane traffic using, for example, odd/even classification of packets (in case of a two probe configuration) or using conventional hashing techniques.

Furthermore, according to an embodiment of the present invention, each network monitoring probea-c is configured to generate basic KPIs and meta data metrics (e.g., KTIs) for the entire user plane traffic and is preferably configured to perform additional filtering and selectively generate advanced KPIs and metadata only for selected subscribers on user plane sessions, either by subscriber or by application, as described above. Advantageously, each network probea-c discards a portion of the user plane traffic that is not relevant to the generation of advanced KPIs and may also optionally discard a portion of the control plane traffic that is not relevant to the generation of advanced KPIs, as described below in conjunction with. It should be noted that in various embodiments each network probea-c may either have its own unique list of subscribers for selective generation of advanced metrics or may each have the same global list of subscribers. In some cases, subscriber's sessions may be handed over from one e-NodeBto another. As a result, subscriber's user plane traffic may be rerouted by the SGWto a different bladea-c. Thus, if the NPBseparates the user plane traffic by IP addresses of the internal SGW bladesa-c then a subset of subscriber's user plane data may be rerouted to a different network probea-c as well. In this scenario, a global list of subscribers shared by all network probesa-c enables complete analysis of the entire user plane data for that particular subscriber even if that data is spread out between the network probesa-c.