Methods and Systems for Using Keywords Preprocessing, Boyer-Moore Analysis, and Hybrids Thereof, for Processing Regular Expressions in Intrusion-Prevention Systems

1. In an intrusion-prevention system for examining network traffic and identifying therein the presence of signature data patterns, a method comprising: providing a state-transition table representative of a predetermined data pattern, the state-transition table comprising a plurality of states, each state having a set of egress events, each egress event defining a transition from a current state to a next state; parsing the predetermined data pattern to identify a set of character strings therein; receiving a subject to be evaluated for the presence of the predetermined data pattern, and preprocessing the subject to find therein any instances of the identified character strings; populating a keyword table with a subset of the identified character strings, the subset consisting of those character strings found in the subject during preprocessing; while using the state-transition table to evaluate the subject for a presence of the predetermined data pattern, transitioning into a first state having a first one of the identified character strings as a first egress event thereof, the first egress event defining a transition from the first state to a second state; and responsive to transitioning into the first state, checking, by a processing unit, the keyword table for the first character string, and, responsive to finding the first character string in the keyword table, transitioning, by the processing unit, from the first state to the second state.

2. The method of claim 1 , wherein the state-transition table is representative of a state diagram, the state diagram representative of the predetermined data pattern.

3. The method of claim 1 , wherein the predetermined data pattern is representative of a regular expression.

4. The method of claim 1 , wherein each egress event is either a character class or a character string.

5. The method of claim 1 , wherein the identified set of character strings in the predetermined data pattern consists of those character strings in the predetermined data pattern that (a) include at least two distinct characters and (b) have a string length that is greater than a threshold number.

6. The method of claim 1 , wherein the subject comprises a payload of one or more packets.

7. The method of claim 1 , wherein the presence of the predetermined data pattern is indicative of a potential security threat.

8. The method of claim 1 , wherein preprocessing the subject comprises using a keyword-tree search.

9. The method of claim 1 , wherein preprocessing the subject comprises identifying positions in the subject where the instances of the identified character strings are located, the method further comprising populating the keyword table with the identified positions.

10. The method of claim 9 , further comprising calculating a first-state range, the first-state range being a range of positions in the subject in which to search for the presence of at least one of the first state's egress events, wherein: checking the keyword table for the first character string comprises checking the keyword table for an instance of the first character string at a position within the first-state range; and finding the first character string in the keyword table comprises finding in the keyword table an instance of the first character string at a position within the first-state range.

11. The method of claim 10 , wherein a cursor corresponds to a location in the subject that is currently being evaluated.

12. The method of claim 11 , wherein transitioning into the first state comprises transitioning from a previous state into the first state according to a previous-state egress event, wherein the previous state has an associated previous-state range in the subject, and wherein calculating the first-state range comprises: setting a start of the first-state range equal to the cursor; starting at the cursor, and extending no further than an end of the previous-state range, determining that the subject includes a number of consecutive instances of the previous-state egress event, the consecutive instances ending at a first position in the subject; and setting an end of the first-state range based on the first position.

13. The method of claim 12 , further comprising determining that the first state does not have a character-class loop transition.

14. The method of claim 12 , further comprising calculating the previous-state range.

15. The method of claim 11 , wherein calculating the first-state range comprises: determining that the first state has a character-class loop transition; setting a start of the first-state range equal to the cursor; starting at the cursor, determining that the subject includes a number of consecutive characters that satisfy the character-class loop transition, the consecutive instances ending at a first position in the subject; and setting an end of the first-state range based on the first position.

16. The method of claim 1 , wherein transitioning from one state to another state comprises recursively calling a state-search function.

17. An intrusion-prevention network device for examining network traffic and identifying therein the presence of signature data patterns, the network device comprising: a network interface; a processing unit; and data storage comprising: a state-transition table representative of a predetermined data pattern, the state-transition table comprising a plurality of states, each state having a set of egress events, each egress event defining a transition from a current state to a next state; and instructions executable by the processing unit to: parse the predetermined data pattern to identify a set of character strings therein; receive a subject to be evaluated for a presence of the predetermined data pattern, and preprocess the subject to find therein any instances of the identified character strings; populate a keyword table with a subset of the identified character strings, the subset consisting of those character strings found in the subject during preprocessing; while using the state-transition table to evaluate the subject for the presence of the predetermined data pattern, transition into a first state having a first one of the identified character strings as a first egress event thereof, the first egress event defining a transition from the first state to a second state; and responsive to transitioning into the first state, check the keyword table for the first character string, and, responsive to finding the first character string in the keyword table, transition from the first state to the second state.

18. In an intrusion-prevention system for examining network traffic and identifying therein the presence of signature data patterns, a method comprising: providing a state-transition table representative of a predetermined data pattern, the state-transition table comprising a plurality of states, each state having a set of egress events, each egress event defining a transition from a current state to a next state; receiving a subject to be evaluated for a presence of the predetermined data pattern; while using the state-transition table to evaluate the subject for the presence of the predetermined data pattern, transitioning into a first state having a first character string as a first egress event thereof, the first egress event defining a transition from the first state to a second state; and responsive to transitioning into the first state, performing, by a processing unit, a Boyer-Moore search for the first character string in the subject, and, responsive to the Boyer-Moore search determining that an instance of the first character string is present in the subject, transitioning, by the processing unit, from the first state to the second state; calculating a first-state range, the first-state range being a range of positions in the subject in which to search for the presence of at least one of the first state's egress events, wherein performing the Boyer-Moore search for the first character string in the subject comprises performing the Boyer-Moore search for the first character string in the first-state range; wherein a cursor corresponds to a location in the subject that is currently being evaluated; wherein transitioning into the first state comprises transitioning from a previous state into the first state according to a previous-state egress event, wherein the previous state has an associated previous-state range in the subject, and wherein calculating the first-state range comprises: setting a start of the first-state range equal to the cursor; starting at the cursor, and extending no further than an end of the previous-state range, determining that the subject includes a number of consecutive instances of the previous-state egress event, the consecutive instances ending at a first position in the subject; and setting an end of the first-state range based on the first position.

19. The method of claim 18 , further comprising determining that the first state does not have a character-class loop transition.

20. The method of claim 18 , further comprising calculating the previous-state range.

21. In an intrusion-prevention system for examining network traffic and identifying therein the presence of signature data patterns, a method comprising: providing a state-transition table representative of a predetermined data pattern, the state-transition table comprising a plurality of states, each state having a set of egress events, each egress event defining a transition from a current state to a next state; parsing the predetermined data pattern to identify a set of a first type of character strings therein; receiving a subject to be evaluated for a presence of the predetermined data pattern, and preprocessing the subject to find therein any instances of the identified character strings; populating a keyword table with a subset of the identified character strings, the subset consisting of those character strings found in the subject during preprocessing; while using the state-transition table to evaluate the subject for the presence of the predetermined data pattern, transitioning into a first state having a given character string as a first egress event thereof, the first egress event defining a transition from the first state to a second state; and responsive to transitioning into the first state, searching, by a processing unit, the subject for an instance of the given character string, and, responsive to determining that there is an instance of the given character string in the subject, transitioning from the first state to the second state, wherein, when the given character string is of the first type, searching, by the processing unit, the subject for an instance of the given character string comprises checking the keyword table for the given character string, and determining that there is an instance of the given character string in the subject comprises finding the given character string in the keyword table, and wherein, when the given character string is of a second type different from the first type, searching, by the processing unit, the subject for an instance of the given character string comprises performing a Boyer-Moore search for the given character string in the subject, and determining that there is an instance of the given character string in the subject comprises the Boyer-Moore search determining that an instance of the given character string is present in the subject.

22. The method of claim 21 , wherein the state-transition table is representative of a state diagram, the state diagram representative of the predetermined data pattern.

23. The method of claim 21 , wherein the predetermined data pattern is representative of a regular expression.

24. The method of claim 21 , wherein each egress event is either a character class or a character string.

25. The method of claim 21 , wherein: the first type of character string is defined by both (a) including at least two distinct characters and (b) having a string length greater than a threshold number, and the second type of character string is defined by either or both of (a) not including at least two distinct characters and (b) having a string length less than or equal to the threshold number.

26. The method of claim 21 , wherein the subject comprises a payload of one or more packets.

27. The method of claim 21 , wherein the presence of the predetermined data pattern is indicative of a potential security threat.

28. The method of claim 21 , wherein preprocessing the subject comprises using a keyword-tree search.

29. The method of claim 21 , wherein preprocessing the subject comprises identifying positions in the subject where the instances of the identified character strings are located, the method further comprising populating the keyword table with the identified positions.

30. The method of claim 29 , further comprising calculating a first-state range, the first-state range being a range of positions in the subject in which to search for the presence of at least one of the first state's egress events, wherein: checking the keyword table for the given character string comprises checking the keyword table for an instance of the given character string at a position within the first-state range; and finding the given character string in the keyword table comprises finding in the keyword table an instance of the given character string at a position within the first-state range; performing the Boyer-Moore search for the given character string in the subject comprises performing the Boyer-Moore search for the given character string in the first-state range; and the Boyer-Moore search determining that an instance of the given character string is present in the subject comprises the Boyer-Moore search determining that an instance of the given character string is present in the first-state range.

31. The method of claim 30 , wherein a cursor corresponds to a location in the subject that is currently being evaluated.

32. The method of claim 31 , wherein transitioning into the first state comprises transitioning from a previous state into the first state according to a previous-state egress event, wherein the previous state has an associated previous-state range in the subject, and wherein calculating the first-state range comprises: setting a start of the first-state range equal to the cursor; starting at the cursor, and extending no further than an end of the previous-state range, determining that the subject includes a number of consecutive instances of the previous-state egress event, the consecutive instances ending at a first position in the subject; and setting an end of the first-state range based on the first position.

33. The method of claim 32 , further comprising determining that the first state does not have a character-class loop transition.

34. The method of claim 32 , further comprising calculating the previous-state range.

35. The method of claim 31 , wherein calculating the first-state range comprises: determining that the first state has a character-class loop transition; setting a start of the first-state range equal to the cursor; starting at the cursor, determining that the subject includes a number of consecutive characters that satisfy the character-class loop transition, the consecutive instances ending at a first position in the subject; and setting an end of the first-state range based on the first position.

36. The method of claim 21 , wherein transitioning from one state to another state comprises recursively calling a state-search function.

37. An intrusion-prevention network device for examining network traffic and identifying therein the presence of signature data patterns, the network device comprising: a network interface; a processing unit; and data storage comprising: a state-transition table representative of a predetermined data pattern, the state-transition table comprising a plurality of states, each state having a set of egress events, each egress event defining a transition from a current state to a next state; and instructions executable by the processing unit to: parse the predetermined data pattern to identify a set of a first type of character strings therein; receive a subject to be evaluated for a presence of the predetermined data pattern, and preprocess the subject to find therein any instances of the identified character strings; populate a keyword table with a subset of the identified character strings, the subset consisting of those character strings found in the subject during preprocessing; while using the state-transition table to evaluate the subject for the presence of the predetermined data pattern, transition into a first state having a given character string as a first egress event thereof, the first egress event defining a transition from the first state to a second state; and responsive to transitioning into the first state, search the subject for an instance of the given character string, and, responsive to determining that there is an instance of the given character string in the subject, transition from the first state to the second state, wherein, when the given character string is of the first type, the instructions to search the subject for an instance of the given character string comprise instructions to check the keyword table for the given character string, and the instructions to determine that there is an instance of the given character string in the subject comprise instructions to find the given character string in the keyword table, and wherein, when the given character string is of a second type different from the first type, the instructions to search the subject for an instance of the given character string comprise instructions to perform a Boyer-Moore search for the given character string in the subject, and the instructions to determine that there is an instance of the given character string in the subject comprise instructions to determine from the Boyer-Moore that an instance of the given character string is present in the subject.