Automatic Detection of Malicious Packets in Ddos Attacks Using an Encoding Scheme

1. A method of detecting patterns in network traffic, the method comprising: receiving a plurality of packets of network traffic, each packet having data associated with respective fields of a set of fields; performing a frequency analysis per field of the plurality of packets as a function of frequency of an occurrence of same data in a corresponding field; selecting top values which are values associated with each field of the set of fields that satisfy a criterion as having occurred most frequently in the plurality of packets as a function of a result of the frequency analysis; assigning a bit encoding scheme that uses variable bit encoding to encode encoding each of the top values for each field that has a top value using a variable bit encoding scheme; encoding into a single value each packet of the plurality of packets based on a bitfield representation that uses the variable bit encoding scheme for values associated with each field that has a top value; storing each potential combination of fields of the set of fields being processed, with all bits set per field when the field is an active field and no bits set when the field is inactive; performing a bitwise an operation on each encoded packet with the stored potential combinations; sorting results of the bitwise operation based on a number of the active fields and a number of occurrences of each same result of the bitwise operation; and providing the results of the sorting to a mitigation device for determining whether an attack is underway or for filtering network traffic for mitigating an attack based on the sorted results of the operation.

2. The method of claim 1, wherein before encoding each packet, fields of the set of fields that are determined to be a variable based on the frequency analysis result are excluded from further analysis in which a top value occurs at least 100/N % times of a total T of the number of the total number of packets in the subset of received network traffic and wherein N is a configurable variable.

3. The method of claim 2, wherein a field of the set of fields is determined to be the variable when there are no top values determined for the field.

4. The method of claim 3, wherein the criterion as having occurred most frequently is satisfied when the value occurs in the field a threshold percentage of times relative to other values that occur in the field for the plurality of packets.

5. The method of claim 1, wherein when generating the single value per packet, the a bit size for each field is determined by the a cardinality for the field based on the frequency analysis.

6. The method of claim 1, wherein the a single value is generated for only those packets that have a top value determined for each of its fields.

7. The method of claim 1, wherein the bitwise operation is a logical AND operation.

8. The method of claim 1, further comprising: creating a sparse memory array; and storing results of the bitwise operation in the sparse memory array, the sparse memory array being sized as a function of a number of bits per packet in the a single value.

9. The method of claim 8, further comprising: decoding top entries of the sorted sparse memory array; and generating a filter that includes data from the decoded top entries.

10. The method of claim 9, further comprising filtering the network traffic using the filter.

11. The method of claim 1, further comprising selecting a template for mitigating the attack based on the results of the sorting.

12. The method of claim 1, further comprising: obtaining results of the sorting when an attack is not underway; and comparing the results of the sorting when an attack is not underway to results of the sorting during network operations to determine when an attack is underway.

13. The method of claim 1, wherein the plurality of packets of network traffic are streaming and the results of the sorting are updated and provided to the a mitigation device in real time.

14. The method of claim 1, further comprising aggregating results of the sorting from a plurality of different positions in the network, a plurality of different networks, and/or a plurality of different times.

15. A network monitoring system comprising: a memory; a processor disposed in communication with the memory, and configured to issue a plurality of instructions stored in the memory, wherein the instructions cause the processor to: receive a plurality of packets of network traffic, each packet of the plurality of packets having data associated with respective fields of a set of fields; perform a frequency analysis per field of the plurality of packets as a function of frequency of an occurrence of same data in a corresponding field; select top values which are values associated with each field of the set of fields that satisfy a criterion as having occurred most frequently in the plurality of packets as a function of a result of the frequency analysis; assign a bit encoding scheme that uses variable bit encoding to encode each of the top values for each field that has a top value using a variable bit encoding scheme; encode into a single value each packet of the plurality of packets based on a bitfield representation that uses the variable bit encoding scheme for values associated with each field that has a top value; store each potential combination of fields of the set of fields being processed, with all bits set per field when the field is an active field and no bits set when the field is inactive; perform a bitwise an operation on each encoded packet with the stored potential combinations; sort results of the bitwise operation based on a number of the active fields and a number of occurrences of each same result of the bitwise operation; and provide the results of the sorting to a mitigation device for determining whether an attack is underway or for filtering network traffic for mitigating an attack based on the sorted results of the operation.

16. The network monitoring system of claim 15, wherein the instructions further cause the processor to before encoding each packet, exclude fields of the set of fields that are determined to be a variable based on the frequency analysis result are excluded from further analysis in which a top value occurs at least 100/N % times of a total T of the number of the total number of packets in the subset of received network traffic and wherein N is a configurable variable.

17. The network monitoring system of claim 15, wherein generating the single value per packet, further comprises determining the the instructions further cause the processor to determine a bit size for each field by the a cardinality for the field based on the frequency analysis.

18. The network monitoring system of claim 15, wherein the network monitoring system is further configured and operable to the instructions cause the processor of the network monitoring system to: create a sparse memory array; and store results of the bitwise operation in the sparse memory array, the sparse memory array being sized as a function of a number of bits per packet in the a single value.

19. The network monitoring system of claim 18, wherein the network monitoring system is further configured and operable to the instructions cause the processor of the network monitoring system to: decode top entries of the sorted sparse memory array; and generate a filter that includes data from the decoded top entries.

20. The network monitoring system of claim 18, wherein the network monitoring system is further configured and operable to the instructions cause the processor of the network monitoring system to: filter the network traffic using the a filter.