Detecting Probes and Scans Over High-Bandwidth, Long-Term, Incomplete Network Traffic Information Using Limited Memory

1. A method of detecting surveillance probes on a computer communications network, comprising: receiving a plurality of messages from a data sensor located at a network audit point, said data sensor sampling data packets on said computer communications network and outputting said messages, each of said messages describing an event occurring on said communications network; processing said messages to form extrapolated connection sessions from said sampled data packets from which to determine a connection source that initiated the connection session by clustering packets exchanged in respective directions over a connection between addresses associated with a connection identifier for said connection said addresses including an address of said connection source and a destination address, and clustering packets that are a) within a specified time period where the source and destination addresses are not predetermined, (b) have certain flags set, or c) have source and destination addresses that are not predetermined but have similar characteristics; and detecting a surveillance probe by: grouping said connection sessions into a plurality of groups of related connection source addresses; scoring each group based on at least a quantity of attack destinations; and generating an alert for each group whose score is greater than an empirically derived threshold.

2. The method of claim 1 , further comprising controlling false positive detections of a surveillance probe vs. false negative detections of a surveillance probe.

3. The method of claim 1 , further comprising generating a profile of surveillance activity, said profile of surveillance activity comprising one or more of the following: a breakdown of probes; a number of attackers; a number of attacks per unit time; a percentage of activity that constitutes malicious surveillance; a breakdown of source country frequencies; the most frequently-targeted network addresses; and temporal frequency trends of individual attackers.

4. The method of claim 1 , further comprising processing one or more of said detected surveillance probes to produce a detected surveillance scan, said processing of one or more of said detected surveillance probes to produce a detected surveillance scan comprising one or more of the following: modeling and detecting surveillance scans as a series of surveillance probes that originate from one or more source addresses and that are sent to one or more destination addresses; modeling and detecting surveillance scans performed by a particular source address by identifying a particular source address that sends more than a specified number of probes; modeling and detecting surveillance scans performed by a particular source address by identifying a source address that generates more than a specified number of probes within a specified time period; modeling and detecting surveillance scans performed by one source IP address by identifying a source address that sends probes to more than a specified number of destinations; modeling and detecting surveillance scans performed by a particular source address by identifying a source address that sends probes to a specified set of destinations; modeling and detecting surveillance scans performed by a particular source address by identifying a source address that sends probes to specified ports; and modeling and detecting surveillance scans performed by a particular source address by identifying a source address that sends probes to a number of destinations in excess of a specified limit within a specified time period.

5. The method of claim 4 , further comprising controlling false positive detections of a surveillance probe vs. false negative detections of a surveillance probe.

6. The method of claim 4 , further comprising generating a profile of surveillance activity, said profile of surveillance activity comprising one or more of the following: a breakdown of probes; a breakdown of scans; a number of attackers; a number of attacks per unit time; a percentage of activity that constitutes malicious surveillance; a breakdown of source country frequencies; the most frequently-targeted network addresses; and temporal frequency trends of individual attackers.

7. The method of claim 4 , further comprising processing one or more of said detected surveillance scans to detect a group of scanning hosts, said processing of one or more of said detected surveillance scans to detect a group of scanning hosts comprising: modeling and detecting scans distributed across a series of source addresses by grouping addresses, said grouping of addresses being performed by subtracting one address from another and placing the two addresses in the same group if the difference is less than a specified amount.

8. The method of claim 7 , further comprising controlling false positive detections of a surveillance probe vs. false negative detections of a surveillance probe.

9. The method of claim 7 , further comprising generating a profile of surveillance activity, said profile of surveillance activity comprising one or more of the following: a breakdown of probes; a breakdown of scans; a number of attackers; a number of attacks per unit time; a percentage of activity that constitutes malicious surveillance; a breakdown of source country frequencies; the most frequently-targeted network addresses; and temporal frequency trends of individual attackers.

10. The method of claim 4 further comprising the steps of: limiting the number of detected scans by reporting only source addresses that perform more than a specified number of probes within a specified time; and limiting the number of detected scans by reporting only source address groups that perform more than a specified number of probes within a specified time.

11. The method of claim 1 wherein the steps of processing said messages to form extrapolated connection sessions and detecting a surveillance probe further comprises at least one of the following steps: identifying packets that have a particular arrangement of flags set; identifying packets that have all flags set; identifying packets that have payloads smaller than a predetermined size; identifying packets to which there is no response.

12. The method of claim 1 wherein the steps of processing said messages to form extrapolated connection sessions and detecting a surveillance probe further comprises at least one of the following steps: identifying detected connections with fewer packets than a predetermined limit; identifying detected connections with packets that have traveled only from a source to a destination; identifying detected connections with packets that have traveled only from the destination to the source; and identifying detected connections with packets whose payloads are smaller than a predetermined limit.

13. A system for detecting surveillance probes on a computer communications network, comprising: a data sensor located at a network audit point adapted to sample data packets on said computer communications network and to output messages, each of said messages describing an event occurring on said communications network; and a processor that processes said messages to form extrapolated connection sessions from said sampled data packets from which to determine a connection source that initiated the connection session by clustering packets exchanged in respective directions over a connection between addresses associated with a connection identifier for said connection, said addresses including an address of said connection source and a destination address, and clustering packets that are a) within a specified time period where the source and destination addresses are not predetermined, (b) have certain flags set, or c) having have source and destination addresses that are not predetermined but have similar characteristics, and that detects a surveillance probe by grouping said connection sessions into a plurality of groups of related connection source addresses, scoring each group based on at least a quantity of attack destinations, and generating an alert for each group whose score is greater than an empirically derived threshold.

14. The system of claim 13 , wherein said processor further generates a profile of surveillance activity comprising one or more of the following: a breakdown of probes; a number of attackers; a number of attacks per unit time; a percentage of activity that constitutes malicious surveillance; a breakdown of source country frequencies; the most frequently-targeted network addresses; and temporal frequency trends of individual attackers.

15. The system of claim 13 , wherein said processor further processes one or more of said detected surveillance probes to produce a detected surveillance scan by performing one or more of the following steps: modeling and detecting surveillance scans as a series of surveillance probes that originate from one or more source addresses and that are sent to one or more destination addresses; modeling and detecting surveillance scans performed by a particular source address by identifying a particular source address that sends more than a specified number of probes; modeling and detecting surveillance scans performed by a particular source address by identifying a source address that generates more than a specified number of probes within a specified time period; modeling and detecting surveillance scans performed by one source IP address by identifying a source address that sends probes to more than a specified number of destinations; modeling and detecting surveillance scans performed by a particular source address by identifying a source address that sends probes to a specified set of destinations; modeling and detecting surveillance scans performed by a particular source address by identifying a source address that sends probes to specified ports; and modeling and detecting surveillance scans performed by a particular source address by identifying a source address that sends probes to a number of destinations in excess of a specified limit within a specified time period.

16. The system of claim 15 , wherein said processor further processes one or more of said detected surveillance scans to detect a group of scanning hosts by modeling and detecting scans distributed across a series of source addresses by grouping addresses, said grouping of addresses being performed by subtracting one address from another and placing the two addresses in the same group if the difference is less than a specified amount.

17. The system of claim 15 wherein the processor is further programmed to perform the steps of: limiting the number of detected scans by reporting only source addresses that perform more than a specified number of probes within a specified time; and limiting the number of detected scans by reporting only source address groups that perform more than a specified number of probes within a specified time.

18. The system of claim 13 wherein the processor is also programmed to perform at least one of the following steps: identifying packets that have a particular arrangement of flags set; identifying packets that have all flags set; identifying packets that have payloads smaller than a predetermined size; identifying packets to which there is no response.

19. The system of claim 13 wherein the processor is further programmed to perform at least one of the following steps: identifying detected connections with fewer packets than a predetermined limit; identifying detected connections with packets that have traveled only from a source to a destination; identifying detected connections with packets that have traveled only from the destination to the source; and identifying detected connections with packets whose payloads are smaller than a predetermined limit.