Technologies for network path and topology management

The present disclosure is generally related to edge computing, cloud computing, network communication, data centers, network topologies, traffic steering and/or shaping techniques, and communication system implementations, and in particular, to techniques for dynamically routing flows along various network paths in a network.

The number and variety of edge devices and Internet of Things (IoT) devices, and their application usages, have increased significantly in the last decade. These devices and their associated networks interact with one another in a wide variety of ways, and may be used in many different modes of operation. One example is Factory 4.0, in which manufacturers are integrating/enabling technologies including IoT devices such as sensors and actuators, cloud computing and analytics. Under these initiatives, a large factory may include thousands of connected devices. However, various challenges are presented by this technology.

The following embodiments generally relate to techniques for determining optimal routing paths for computing devices in a network, including selecting an optimal gateway among a number of available gateways.

As mentioned, the deployment of computer devices such as IoT devices in factories or other settings presents various challenges. For example, consider a factory extending over 100,000 square meters. While 4G networks can support a maximum of 100,000 devices per km, 5G networks will connect up to a million devices per km, or one device per square meter. This translates to a need to connect up to 100,000 devices, permitting companies to connect every sensor, actuator, or other types of devices in a factory.

Given the possible range of operation, it is desirable to bring a degree of prescriptive or opinionated management over how they are configured, deployed, and optimized. Generally, it is imperative to connect the various heterogeneous devices using well-defined interfaces and reliable communication means. To achieve this, the devices must be correctly configured and the operations scaled.

These edge and IoT devices are typically inexpensive, single purpose and simple devices that are manually selected and configured, and connected, using a variety of protocols, to each other and to a cloud/network for external connectivity. It is not realistic to expect these numerous simple devices to perform complex decisions in case of network failures. Worst case scenarios can result quickly if a critical operation signal either goes undelivered or is not received in a timely manner. Some examples include the catastrophic life or safety threats when communications do not transpire in a deterministic manner to/from medical cardiac devices such as pacemakers or when time-coordinated signals do not get delivered to defibrillators.

In additional to the example of a factory, the device can be deployed in other settings such as a home or vehicle. For example, a representative home edge/IoT network includes a personal subnetwork connecting devices such smart phones, smart watches, laptops, health or activity sensors, etc. A typical car subnetwork links Lidar, Sensors, gyroscopes, acceleration sensors, smart phones and/or other devices as applicable. When edge/IoT devices that belong to a particular subnetwork interact with other subnetworks and/or other external subnetworks, important concerns relating to finding best routing paths are raised. These include, for example, managing topology, minimizing energy, maximizing connectivity, achieving high operational efficiency, and so on. As connected devices are growing at a massive rate, different applications running inside devices may have different requirements (latency, throughput, energy, packet error budget etc.). In fact, a single application can have different types of flows and/or workloads, which need to be routed through a network efficiently in terms of resource consumption, delivery time, and/or the like. Accordingly, there is a need for a protocol to choose a best, efficient route and gateway for workloads to provide a seamless user experience.

Protocols and methods such as the Gossip Protocol, proactive, reactive, hybrid, etc., are based on the type of device/application and its requirements, e.g., latency sensitive, bandwidth sensitive, etc. However, although gateways or paths are assigned for devices or applications based on device traffic requirements/characteristics, multiple applications running on a device can act differently at times and can have different requirements such as different bit-rate requirements, quality of service (QoS) requirements, and/or the like. For example, an application may require a guaranteed bit rate (GBR) flow for critical/emergency situations, sending data at higher rates, and may require a non-GBR for other, non-sensitive operations. Thus, it may not always be efficient to statically associate a path and a gateway with a device or a particular application instance. This approach does not dynamically provide the best or optimal routing path and gateways according to changing network conditions and/or changing workloads in a single application or multiple applications running on a device at a particular time. Moreover, these approaches may send non-critical data on priority channels, causing congestion on high priority, high latency channels, resulting in the user paying more for the required services.

Further, as gateways are connected to different networks (e.g., private 5G, enterprise networks, public networks with different vendors, particular sliced networks, etc.) it is possible that at any particular time, network conditions of any gateway may deteriorate for various reasons (e.g., workload/increase, signal strength issue, etc.) and it may not be able to complete the workload requirements for critical data (e.g., requiring low latency and packet error rate). Devices in this case send data to a network without acknowledging or discovering dynamically changing network conditions, resulting in inefficiencies.

Moreover, as a gateway has pre-defined metering capabilities it's quite possible that sensitive/priority data will get dropped by metering logic, even though a device is sending high priority data through a gateway properly.

Finally, slicing is becoming popular in networks to address workload, but current routing protocols on the device side are usually not aware of it and do not use it for routing dynamically. Network slicing is a network architecture that enables the multiplexing of virtualized and independent logical networks on the same physical network infrastructure.

The techniques herein address the above and other issues. Various implementations discussed herein include gathering data regarding characteristics of a network, including gateways and network access nodes (NANs) in at least one radio access network (RAN). The NAN and/or RAN characteristics can include, e.g., supported frequency bands, communication protocols, signal-to-noise ratio, power, signal noise and quality, and network slicing information. The NAN and/or RAN characteristics can further indicate whether a network (or network vendor) is a standalone (SA) network (or SA network vendor) or a non-SA network (or non-SA network vendor). An SA network may be a network that can only be accessed by authorized users, and may include enterprise networks or privately maintained networks. In one aspect, the characteristics comprise parameters according to a Mobile Broadband Interface Model (MBIM) provided by the USB Implementers Forum (USB-IF). These parameters are defined by 3GPP specifications and referred to in the MBIM specification. The MBIM is a protocol by which USB hosts and mobile broadband devices can efficiently exchange control commands and data frames, and is designed for use with high speed mobile broadband modem devices (see e.g., “Universal Serial Bus Communications Class Subclass Specification for Mobile Broadband Interface Model”, USB Implementers Forum, Inc., Revision 1.0, Errata-1 (1 May 2013), “Universal Serial Bus MultiFlow Extension for Mobile Broadband Interface Model”, USB Implementers Forum, Inc., Revision 1.0 (23 Jul. 2017), MBIM Extensions Release number 2.0 and MBIM Extensions Release number 3.0, the contents of each of which are hereby incorporated by reference in their entireties). The characteristics can be stored in a remote or local database, for example, for use by devices in determining routing paths in the network.

For example, an application comprising multiple flows/workloads may run on a device. The device can determine requirements of each flow, access the database, and select a gateway to access the at least one RAN based on a comparison of the characteristics to the one or more respective requirements of the flow. These comparisons seeks to find a best match or alignment between the characteristics and the requirements for individual flows. The best or optimal match or alignment is used for traffic shaping, traffic steering, and/or traffic splitting purposes.

These and other advantages will be apparent in view of the following discussion.

depicts an example arrangement of devices and gateways in networks according to various embodiments. Different type of interconnected networks are depicted, including IoT networks NT1-NT5, a private 5G network RAN4 (with a shared RAN with a public land mobile network (PLMN)), a time-sensitive private network for Factory 4.0 RAN5, as well as several gateways G1-G5 and various devices D1-D23 that are in their respective domains. Here, any device can communicate with any gateway G1-G5 or other devices either directly or via other devices.

As examples, the networks NT1-NT5 can be IoT networks, edge clouds, fogs, or some other network arrangement or combinations thereof. Networks NT1-NT5 may also be considered to be subnetworks of an overall network. Each network has an associated gateway, e.g., G1-G5, and each gateway is in an associated RAN, e.g., RAN1-RAN5, respectively. Although not shown in this example, one or more of the gateways could be associated with more than one RAN, which may implement one or more Radio Access Technologies (RATs). The gateways G1-G5 are components in the network which connect the devices D1-D23 to other devices in remote or non-local subnetworks, and/or connect the devices D1-D23 to the cloud via a network (e.g., a cellular network). In one implementation, the gateways G1-G5 comprise cellular modems. Additionally or alternatively, the gateways G1-G5 include multiple RAT interfaces such as, for example, a cellular modem, a WLAN modem, a Bluetooth® module, etc. Any of the devices or gateways ofmay be the same or similar as the computing nodeof(discussed infra).

Each (sub)network includes various devices D1-D23. Connections in the system are denoted by a solid line for various protocols such as cellular (e.g., 3GPP 5G/NR, LTE, WiMAX, etc.), ZigBee, Bluetooth, Controller Area Network (CAN), Local Interconnect Network (LIN), Wireless Local Area Network (WLAN) (e.g., Wi-Fi, etc.), Constrained Application Protocol (CoAP), wired (e.g., Ethernet, etc.), and/or some other access technology such as any of those discussed herein. An example of a CAN is a network in an automobile. LIN is a serial network communication protocol used between components in vehicles.

Connections with a short-dashed line denote a non-GBR connection. Connections with a long-dashed line denote a GBR connection. Connections with a wide double arrow denote a non-3GPP connectivity, where 3GPP refers to the 3rd Generation Partnership Project. This includes standards organizations which develop protocols for mobile telecommunications including Global System for Mobile Communications (GSM) and related 2G and 2.5G standards, Universal Mobile Telecommunications System (UMTS) and related 3G standards, Long-Term Evolution (LTE) and related 4G standards, and 5G New Radio (NR) and related 5G standards. Non-3GPP technologies include Worldwide Interoperability for Microwave Access (WiMAX), CDMA2000 (Code Division Multiple Access), WLAN, fixed/wired networks, and/or any other suitable communication protocol/standard such as those discussed herein.

The connectionwith a wide line without arrows denote a wired path such as an Ethernet cable. The devices can communicate with other devices in the same network and in another network.

Specifically, NT3 includes devices D9, D13-D15, D17 and D18. D17 runs applications App1 and App2, for example. An associated or default gateway G3 of the network communicates in RAN3, which includes a cell tower/base station. A NAN(e.g., a router or other network element) may be associated with the RAN, but is connected by a wired pathto gateway G2. In each of the networks NT1-NT5, the devices D1-D23 are connected to one another by paths as indicated, where the pathrepresents a GBR connection between D17 and G3, a pathrepresents a non-GBR connection between D17 and G2, a pathrepresents a GBR connection between D17 and D1, and a pathrepresents a connection between App1 of D17 and D4.

In this example, G3 has E-SIM technology and hosts multiple Access Point Names (APNs) and bearers including GBR, non-GBR and delay-critical GBR bearers. An APN is the name of a gateway between a mobile network and another computer network, such as the public Internet. E-SIM technology refers to a reprogrammable SIM embedded in a gateway to provide built-in, stand-alone cellular connectivity. The E-SIM allows the gateway to compare cellular networks and select a desired service. A bearer is a communication pipe, tunnel, or data service that carries a data stream or flow(s) between two or more devices. For purposes of the present disclosure, a “bearer” may refer to any such pipe, tunnel, or data service provided by any type of network or RAT. A gateway and/or any other device depicted bycan have multiple bearers and can sometimes create a new bearer. A bearer can connect to any frequency or channel of a cell phone tower/base station at the hardware level.

NT1 includes devices D1-D6. The associated or default gateway G1 of the network communicates in RAN1, which includes a NAN, which in this example is a cellular tower/base station. UEmay be associated with the RAN1 and communicates using non-3GPP connectivity pathwith a tablet. The dashed line pathrepresents a GBR connection between D1 and G1. In this example, G1 hosts two bearers, namely GBR and non-GBR bearers. D4 communicates with D7 over a path, and D5 communicates with D11 over a path.

NT2 includes devices D7-D12. The associated or default gateway G2 of the network communicates in RAN2, which includes a cell tower/base station. The tabletmay be associated with the RAN and communicates using non-3GPP connectivity on a pathwith another tablet. The pathrepresents a non-GBR connection between D7 and G2. The pathrepresents a non-3GPP connection between D8 and G5. In this example, G2 hosts multiple bearers.

NT4 includes devices D13-D16 and D23. The associated or default gateway G4 of the network communicates in RAN4, which includes a cell tower/base station. The pathrepresents a non-GBR connection between D14 and G4. In this example, G4 is in a private 5G network with shared RAN with a PLMN network.

NT5 includes devices D18-D22. The associated or default gateway G5 of the network communicates in RAN5, which includes a cell tower/base station. The pathrepresents a non-GBR connection between D19 and G5. In this example, G5 is in a Factory 4.0 private 5G time-sensitive network such as those discussed in Institute of Electrical and Electronics Engineers (IEEE) “Standard for a Precision Clock Synchronization Protocol for Networked Measurement and Control Systems”, IEEE Std 1588-2019 (16 Jun. 2020) (“[IEEE1588]”) and “IEEE Standard for Local and Metropolitan Area Ntworks—Timing and Synchronization for Time-Sensitive Applications,” IEEE Std 802.1AS™-2020 (19 Jun. 2020) (“[IEEE802.1AS]”), the contents of each of which are hereby incorporated by reference in its entireties.

The system further includes a primary deviceand a database.

As a specific example, consider device D17. It is a device on which multiple applications (e.g., App1, App2 . . . ) are running, where each application can have multiple flows/workloads with their own respective requirements. A flow/workload can be a method, process or other task, for example.

In a setup or exploration phase, the primary devicedetects characteristics of the gateways, including their capabilities, and stores those capabilities in gateway profiles in the database. The databaseneed not be stored at the primary devicebut can be distributed to other locations in the system. In one example, each of the devices D1-D23 can have their own local version of the databasethat they individually access when attempting to dynamically adjust a flow path. Here, the characteristics stored in a local databaseat a first device may be different than those included in a local databaseat a second device. Additionally or alternatively, the devices may obtain gateway profiles and/or characteristics from the database, store those profiles/characteristics locally (e.g., using a suitable caching system), and later obtain new or updated profiles/characteristics for dynamic traffic steering or traffic splitting. Additionally or alternatively, the functions of the primary devicecould be incorporated into one or more of the devices D1-D23 in the networks.

The gateway profiles can be used by devices such as D17 to select a gateway that is a best match to one or more requirements of D17. For example, the requirements can be expressed in terms of a service level agreement (SLA) of the device. The SLA can encompass factors such as availability, jitter, latency, and/or other reliability metrics such as those discussed herein. For example, D17 may ordinarily request the default gateway of NT3, G3, to create one more bearers for GBR at some time. However, if at a given time, the profile indicates G3 is already at saturation point for GBR flows and therefore cannot accept anymore GBR flows, D17 can explore and find another path for its GBR needs. It may, for example, select to route flows via D1 and Gateway 1, on pathsand. For a non-GBR flow, D17 may connect to G2 via path. Here, App1 generates a GBR flow while App2 may generate both GBR and non-GBR flows. D17 and D1 are depicted by dashed line boxes since they are GBR devices for at least some of their applications or application flows.

A converse situation may exist for a device such as D7 in NT2. D7 is depicted by a dotted line box since it is a non-GBR device. D7 can have its own non-GBR flow fulfilled by G2, so it can choose any path for its flows by considering other parameters in the gateway profiles that are better aligned to its needs at a given time in terms of characteristics such as cost (e.g., in terms of resource usage or the like), power usage, and packet loss statistics.

Usually, the devices are attached to a particular path or gateway based on characteristics of each device, applications running at the device, and any communication protocol supported by the device. A device can support more than one communication protocol, such as Bluetooth, Wi-Fi, wired, or ZigBee. In embodiments, a device (or selection function) can choose network paths at more granular level. The choice can account for bearers, and/or flow requirements of an application, where multiple flows/bearers are flowing from single or multiple applications having respective requirements. Devices can dynamically select different paths or gateways for different flows/bearers to provide a seamless experience to an end user. A flow can provide or indicate its requirements, e.g., in terms of packet error rate, packet delay budget, priority, etc. The device or application can choose any profile dynamically according to requirements. Moreover, a device can activate or use more than one communication protocol to communicate with different device paths as per the flow/bearer requirements.

The techniques discussed herein provide an efficient and network-assisted device flow routing method to find the best or optimal gateways and to manage topology selection. While the illustration above used GBR as a discriminating feature for D17 to select gateway G1 through D1 at one time, in general, many different criteria can factor into choosing a hierarchical device path. Moreover, the techniques can choose the optimal flow routing based on a single objective/requirement or multiple objectives/requirements. A multi-objective satisfaction algorithm can capitalize on known parameters in the gateway profiles to achieve a best-fit routing on a dynamic basis for different workloads at different times. When there is a single objective, it easier to decide on an optimal path.

Various multi-objective satisfaction algorithms (sometimes referred to as “constraint satisfaction algorithms” or “constrain satisfaction problems”) can be used. The techniques herein provide a method/framework and required parameters which can be fed to a multi-modal objective satisfaction algorithm whose output can be applied for configuring a best path at the flow level. For example, a classical approach in solving multi-objective optimization problems is the weighted sum method, which has the advantage of identifying a single unique solution for actual implementation such as a particular network topology. In comparison, the Epsilon (ε)-constraint method generates multiple Pareto optimal solutions. Generally, when there is a problem to be solved, to turn it into a multi-objective satisfaction algorithm, one can define a variable set, a domain set and create a constraint set with variables and domains. Then, a computation or process is performed to find a solution, ideally an optimal one, answering the set of defined constraints setting the conditions that the variables must satisfy.

A primary provisioning device or service, such as the primary device, can dynamically configure gateways according to current workload conditions. Optionally, it may configure them statically if the policy specifies. The primary provisioning device will check a number of parameters (from possibly a rich list) to compute a best routing path based on traffic types that need to be supported. Every device will be running numbers of applications, with each application generating different type of workloads. Each workload has certain characteristics, such as priority, packet error budget, latency, energy, packet delay budget, maximum expected data burst, and various other requirements. Devices can be connected directly to a gateway, or to a gateway though one or more other devices. Similarly, a flow can traverse various devices (or device groups) and can have multiple paths to reach to the same gateway. Based on the characteristics of flow and other factors, it is proposed to choose best appropriate paths and gateways.

The techniques provide a number of advantages, including the freedom to dynamically choose a best routing path/gateway at a more granular level (e.g., at the bearer/flow level) and therefore provide a better quality of service to the end user.

Additionally, an administrator may select different profiles and choose each to apply dynamically at the device/application/flow level to satisfy dynamic end user quality requirements. As a first example, if due to network congestion, a packet error budget or latency cannot be matched for ‘X’ number of packets, then to compensate for that, the application flow can be channeled through another path which provides better latency and has a gateway that is less prone to packet errors for the next ‘Y’ packets, or for an interval that is calculated. As a second example, suppose it is desired to reduce power consumption. This can be achieved by choosing a designated low energy path (and gateways) at a certain time (e.g., 11 pm) when workload requirements are less stringent.

A further advantage is that, if an existing gateway is not performing up to the mark due to network congestion or other issues, other gateways can be dynamically selected.

The device flow routing techniques are network assisted and efficient. The techniques are designed to allow applications to achieve the best topology without having to be burdened with network management, and without having to be locked into static or inflexible choices. The techniques achieve dynamic topology management through the selection of hierarchical device paths and gateways based on a variety of parameters which include, but are not limited to: network parameters such as cell capacities, signal strengths, power, signal-to-noise ratio (SNR), available bands to select from and their characteristics, best-match frequencies, slicing information; application or operations parameters such as packet delay budgets, device/gateway metering capabilities, energy efficiency, Packet Data Protocol (PDP) context sharing; performance and SLA considerations such as latency requirements and jitter tolerance, error tolerance, whether GBR, non-GBR or delay-critical GBR is supported and so forth. Many other parameters can be accommodated with evolution of usages over time, as the techniques are not dependent upon any given set of parameters.

illustrates a topology in which various devices are connected to each other using a set of protocols. For example, in a home, a network device can connect using Wi-Fi, Bluetooth, ZigBee, etc. In a car network, a device could be connected through a CAN/LIN bus and through various gateways, for example. Gateways could also be connected using non-3GPP access/wired protocols or by other means. As mentioned, D1 and D17 are GBR devices that generate a guaranteed bit rate flow; and the network is expected to fulfill latency, packet error rate and other requirements for this type of flow.

During discovery, some or all capabilities of each gateway are extracted and saved in a profile in the database. In one example, the databasestores a first profile for gateway G1, a second profile for gateway G2, and so forth. Additionally or alternatively, the databasemay store a single profile for all of the gateways, where a gateway identifier is used as an index for the capabilities of each gateway. Using the profile, a device can consider various characteristics/parameters among the profile parameters in a gateway profile in order to identify an optimal gateway. Here, the optimal gateway may be a gateway whose characteristics fit or fulfill the requirements of a subject device, application, flow, or workload at least in comparison with the characteristics of other gateways under consideration. For example, device D17, which may be a GBR device, can consider various characteristics/parameters among the profile parameters in order to identify the gateway that can optimally fulfil its SLA goals or needs (or the SLAs of an application, flow, or workload). In one example, D17 asks G3 to create one more bearer for a GBR flow. In another example, D17 has to choose G1 through D1 since G1 is already serving a GBR flow for D1. Similarly, D7 (or an application or flow operating at D7) has a non-GBR requirement. D7 is generating non-GBR flows which can be fulfilled by G2, so the best match paths can be any that happen to be optimal and such a choice can be dynamic, based on the volumes of traffic that are already utilizing available traffic capacities and/or other parameters, metrics, measurements, or characteristics.provide details of obtaining profiles andprovide details of the use of other dynamic criteria for determining an optimal routing path.

depicts a flowchart of an example process for creating a database of NAN and/or RAN characteristics of NANs in RANs according to various embodiments. At step, a primary devicediscovers gateways in communication with base stations in at least one RAN. Stepinclude determining characteristics of the base stations/gateways. Stepincludes saving the characteristics in a database. At step, the database is accessible to device to select a gateway. Seefor further details.

depicts a flowchart of an example process for using a database of characteristics of base stations in radio access networks (RANs) to select a gateway and a routing path according to various embodiments. At step, a device D1-D23 begins an application flow/workload having one or more requirements. Stepincludes accessing a database with characteristics of base stations/gateways of radio access networks (RANs). Stepincludes selecting a gateway based on a comparison of the characteristic to the one or more requirements. Stepincludes communicating data via the selected gateway. Seefor further details.

depicts a flowchart of an example implementation of the process ofaccording to various embodiments. At step, a new gateway is discovered. Stepincludes checking vendors available from a pre-configured plan. These can be, e.g., vendors associated with RANs/cell towers. Stepincludes getting registration information and a registered class for every Access Point Name (APN) in a profile. In one example, stepcan involve obtaining parameters according to a Mobile Broadband Interface Model (MBIM) such as depicted in step

Specifically, MBIMDataClass5G_NSA indicates whether the 5G Non-standalone (NSA) model of deployment is supported, where 5G services are provided without an end-to-end 5G network. Instead, the network will rely on some previous generation (e.g., 4G LTE) infrastructure or a mix of 4G LTE and 5G.

MBIMDataClass5G_SA indicates whether the 5G Standalone (SA) model of deployment is supported, where 5G services are provided through an end-to-end 5G network to ensure that the high performance features of 5G, including high-speed data and ultra-low latency, are delivered.

MBIMDataClassNone indicates whether a MBIM_DATA_CLASS is supported.

MBIMDataClassGPRS indicates whether General Packet Radio Service (GPRS) is supported. This is packet-based 2G wireless communication service.

MBIMDataClassEDGE indicates whether Enhanced Data GSM Evolution (EDGE) is supported. This is another 2G technology which is slightly faster than GPRS.

MBIMDataClassUMTS indicates whether the Universal Mobile Telecommunications System (UMTS) is supported.

MBIMDataClassHSDPA indicates whether High-Speed Packet Access (HSPA) is supported.