System and Methods for Detecting Network Failure

1. A method of identifying network failure comprising: periodically transmitting diagnostic messages to a plurality of predetermined routing points along a path to a destination; transmitting, if the diagnostic messages indicate a problem with intermediate nodes along the path, a series of path verification messages, each of the path verification messages including a command operable to direct an intermediate node to transmit a further message to a successive intermediate node in the path, receive the result from the further message, and report the result as a command response, the result indicative of reachability of the successive intermediate node; repeating the transmission of path verification messages to successive nodes along the path to the node indicating the problem; analyzing the received command responses from the successive path verification messages to identify problems; and determining an alternate route based on the analyzing to bypass the intermediate node identified as a source of the indicated problem.

2. The method of claim 1 wherein analyzing further comprises analyzing the received command responses from the path verification messages to identify the first intermediate node for which the command response indicated a problem and the previous intermediate nodes for which the command response to the diagnostic message indicates normal operation.

3. The method of claim 2 wherein the intermediate nodes denote segments, the segments further comprising a local segment between the customer device and a core network, a core network segment representing a plurality of provider devices, and a remote segment between the core network and the destination, tracking further comprising identifying the segment from which the response emanates.

4. The method of claim 3 wherein the segments further comprise a first segment from a customer edge router to an intermediate network to a remote edge router, a second segment between provider edge routers, and a third segment from a provider edge router to a remote customer edge router.

5. A method of identifying network failure comprising: periodically transmitting diagnostic messages to a plurality of predetermined routing points along a path to a destination; transmitting, if the diagnostic messages indicate a problem with intermediate nodes along the path, a series of path verification messages, each of the path verification messages including a command operable to direct an intermediate node to transmit a further message to a successive intermediate node in the path, receive the result from the further message, and report the result as a command response, the result indicative of reachability of the successive intermediate node; repeating the transmission of path verification messages to successive nodes along the path to the node indicating the problem; analyzing the received command responses from the successive path verification messages to identify problems; determining an alternate route based on the analyzing to bypass the intermediate node identified as a source of the indicated problem; wherein transmitting further comprises: transmitting a plurality of path verification messages to a plurality of predetermined network points according to a diagnostic protocol; receiving command responses corresponding to the transmitted path verification messages, the command responses including a test result according to the diagnostic protocol; and tracking the command responses received from each of the plurality of path verification messages transmitted along a path from a source to a destination.

6. The method of claim 5 wherein analyzing further comprises identifying receipt and non-receipt, where the receipt includes an indication of accessible paths from the predetermined network point sending the message and non-receipt indicates an interceding failure according to the diagnostic logic.

7. The method of claim 6 further comprising identifying, based on the location and nature of the network failure, network points at which to alter traffic.

8. The method of claim 7 further comprising: identifying important prefixes corresponding to network routing points having substantial logistic routing value; and transmitting the diagnostic messages to the important prefixes.

9. The method of claim 8 further comprising determining active prefixes, the active prefixes indicative of a substantial volume of routing traffic during a previous threshold timing window.

10. The method of claim 9 wherein the substantial volume of routing traffic load is based on a predetermined minimum quantity of bytes transported and the important paths correspond to the number of alternative routing paths available.

11. The method of claim 10 wherein the path verification messages are probe messages according to the predetermined protocol, the probe messages include a test indicator and a destination indicator and concluding further comprises applying diagnostic logic according to the predetermined protocol.

12. The method of claim 11 further comprising staggering the diagnostic messages based upon a jitterable configurable timer driving an end to end path check.

13. The method of claim 12 wherein identifying the intermediate nodes further corresponds to identifying a network prefix corresponding to a network hop between a test initiator and a destination.

14. A method of identifying network failure comprising: periodically transmitting diagnostic messages to a plurality of predetermined routing points along a path to a destination; transmitting, if the diagnostic messages indicate a problem with intermediate nodes along the path, a series of path verification messages, each of the path verification messages including a command operable to direct an intermediate node to transmit a further message to a successive intermediate node in the path, receive the result from the further message, and report the result as a command response, the result indicative of reachability of the successive intermediate node; repeating the transmission of path verification messages to successive nodes along the path to the node indicating the problem; analyzing the received command responses from the successive path verification messages to identify problems, wherein analyzing further comprises identifying a forwarding plane error indicative of inability of message propagation along a purported optimal path, and determining an alternate route based on the analyzing to bypass the intermediate node identified as a source of the indicated problem, wherein determining comprises changing a control plane routing decision corresponding to the purported operational path.

15. The method of claim 14 wherein transmitting the path verification messages further comprises: transmitting a first path verification message; identifying non-receipt of a command response corresponding to the first path verification message; waiting a predetermined threshold, the predetermined threshold corresponding to a convergence time adapted to allow automatic routing table updates to compensate for erratic routes; and transmitting a second path verification message, wherein receipt of a command response to the second path verification message is indicative of a routing table change around the erratic route.

16. A method for locating network failures comprising: transmitting a plurality of path verification messages to a plurality of predetermined network points according to a diagnostic protocol; receiving command responses corresponding to the transmitted path verification messages, the responses including a test result according to the diagnostic protocol; tracking the command responses received from each of the plurality of path verification messages transmitted along a path from a source to a destination; and concluding, based on the receipt of responses from the predetermined network points, alternate routing paths for message traffic in the network.

17. The method of claim 16 wherein concluding based on the receipt of messages further comprises: determining, based on the test result received in the responses, whether to reroute traffic in the network; and locating, based on the receipt and non-receipt of responses from particular network points, an alternate path.

18. A method for locating a deficient network interconnection comprising: identifying a path from a data communication device to a remote network destination, the path further including a plurality of segments, each segment delimited by a hop; identifying the failure point comprising identifying a segment order defined by a path to the destination; iteratively transmitting a probe to each successive hop along the ordered path; concluding, if a probe response returns with respect to a particular hop, that the path is unobstructed up to the hop corresponding to the returned probe; concluding, if the probe response is not received for a particular probe, that an obstruction exists between the hop corresponding to the particular probe and previous hop; identifying, based on the hop corresponding to the concluded obstruction, an alternate path; and determining, based on the identified alternate path, whether to direct message traffic to the identified alternate path.

19. A method for network failure location identification comprising: enumerating a set of significant routes, the significant routes carrying a substantial traffic load over a critical path; identifying active routes from the significant routes based on recently carried traffic; determining, for each of the identified active routes, whether an unobstructed network path exists; applying, for each active route determined to have an obstruction, a path verification to identify a path segment corresponding to a point of obstruction, the path verification process further comprising: pinging each of a plurality of intermediate hops; identifying hops for which the ping response is deficient; repinging, if the ping response was deficient, the hop after waiting for a convergence threshold delay; concluding, if the response to the repining is received, a core network failure which has been rerouted around; and determining if the repinging response is not received, a failure at a point between the repinged hop and the previous hop.

20. A data communications device for identifying network failure comprising: a memory operable to store instructions and data; an execution unit coupled to the memory, the execution unit in communication with the data and responsive to the instructions; a network interface coupled to other data communications devices; a path verification processor in the execution unit operable to periodically transmit diagnostic messages, via the network interface, to a plurality of predetermined routing points along a path to a destination, and further operable to transmit, if the diagnostic message indicate a problem with intermediate nodes along the path, a series of path verification messages, each of the path verification messages including a command operable to: direct an intermediate node to transmit a further message to a successive intermediate node in the path; receive the result from the further message; and report the result as a command response, the result indicative of reachability of the successive intermediate node, the path verification processor further operable to: repeat the transmission of path verification messages to successive nodes along the path to the node indicating the problem; and analyze the received command responses from the successive path verification messages to identify problems; and routing logic in the memory and responsive to the path verification processor and operable determining an alternate route based on the analyzing to bypass the intermediate node identified as a source of the indicated problem.

21. The data communications device of claim 20 wherein the path verification processor is further operable to analyze the received command responses from the path verification messages to identify the first intermediate node for which the command response indicated a problem, and the previous intermediate nodes for which the command response to the diagnostic message indicates normal operation.

22. The data communications device of claim 21 wherein the intermediate nodes denote segments, the segments further comprising a local segment between the customer device and a core network, a core network segment representing a plurality of provider devices, and a remote segment between the core network and the destination, tracking further comprising identifying the segment from which the response emanates.

23. The method of claim 20 further comprising a network interface coupled to the path verification processor, the network interface operable to: transmitting a plurality of path verification messages to a plurality of predetermined network points according to a diagnostic protocol; receiving command responses corresponding to the transmitted path verification messages, the command responses including a test result according to the diagnostic protocol; and forward the command responses to the path verification processor for tracking the command responses received from each of the plurality of path verification messages transmitted along a path from a source to a destination.

24. The data communications device of claim 23 wherein the path verification processor is further operable to: identify important prefixes corresponding to network routing points having substantial logistic routing value; and transmit the diagnostic messages to the important prefixes.

25. The data communications device of claim 24 wherein the path verification processor is further operable to determine active prefixes, the active prefixes indicative of a substantial volume of routing traffic during a previous threshold timing window.

26. The data communications device of claim 25 wherein the path verification messages are probe messages according to the predetermined protocol, the probe messages include a test indicator and a destination indicator and concluding further comprises applying diagnostic logic according to the predetermined protocol.

27. The data communications device of claim 20 wherein the path verification processor is further operable to: identify a forwarding plane error indicative of inability of message propagation along a purported optimal path, and determine comprises changing a control plane routing decision corresponding to the purported operational path.

28. The data communications device of claim 27 wherein the path verification processor is further operable to: transmit a first path verification message; identify non-receipt of a command response corresponding to the first path verification message; wait a predetermined threshold, the predetermined threshold corresponding to a convergence time adapted to allow automatic routing table updates to compensate for erratic routes; and transmit a second path verification message, wherein receipt of a command response to the second path verification message is indicative of a routing table change around the erratic route.

29. A computer program product having a computer readable storage medium operable to store computer program logic embodied in computer program code, the computer program code executable with a processor to identify network failure, the computer readable storage medium comprising: computer program code executable to periodically transmit diagnostic messages to a plurality of predetermined routing points along a path to a destination; computer program code executable to transmit, if the diagnostic message indicate a problem with intermediate nodes along the path, a series of path verification messages, each of the path verification messages including a command operable to direct an intermediate node to transmit a further message to a successive intermediate node in the path, receive the result from the further message, and report the result as a command response, the result indicative of reachability of the successive intermediate node; computer program code executable to repeat the transmission of path verification messages to successive nodes along the path to the node indicating the problem; computer program code executable to analyze the received command responses from the successive path verification messages to identify problems; and computer program code executable to determine an alternate route based on the analyzing to bypass the intermediate node identified as a source of the indicated problem.