Graph-Based Network Security Threat Detection Across Time and Entities

1. A method comprising: accessing, from a data store, a relationship graph corresponding to a time range, the time range having a number of time units, the relationship graph having entities as nodes and relationships among the entities as links, the relationship graph reflecting a batch of events that occurred during the time range, wherein each event of the batch of events includes timestamped, raw machine data that reflects one or more of: (1) activity occurred in an information technology (IT) or a security technology environment, (2) a time at which the activity occurred, and (3) a number of entities associated with the activity; assigning the nodes in the relationship graph to groups based on event timestamps, each group corresponding to a time unit and including nodes associated with activities that occurred in the time unit; constructing links for nodes between different groups, each link representing a relationship between nodes as established by a respective activity recorded in the batch of events, each chain of linked nodes forming a component; computing a total interest score for each of the formed components, wherein the total interest score reflects a totality of interest generated from all nodes attached to a given link; adjusting the total interest score for each of the formed components based on comparing events underlying a component with a pattern of interest, wherein the pattern of interest identifies an expected temporal order and/or logical relationship in underlying events for such component to be of interest; and identifying a component for further security scrutiny based on the adjusted total interest score.

2. The method of claim 1 , wherein the plurality of events comprise events that have been earmarked as anomalies.

3. The method of claim 1 , wherein each node carries an anomaly score that is assigned from a previous data analytic stage.

4. The method of claim 1 , wherein the relationship graph is a subset of a composite relationship graph that includes edges representing a plurality of anomaly activities conducted by entities.

5. The method of claim 1 , further comprising: determining a group interest score for each of the groups, based on steps including: generating the group interest score based on a set of features from a respective group, wherein the set of features are identified by a predetermined list of features that are characteristic of the activities recorded in the events in the respective group.

6. The method of claim 1 , further comprising: determining a group interest score for each of the groups, based on steps including: generating the group interest score based on a set of features from a respective group, wherein the set of features are identified by a predetermined list of features that are characteristic of the activities recorded in the events in the respective group, wherein a feature in the set of features carries a different weight than another feature.

7. The method of claim 1 , further comprising: determining a group interest score for each of the groups, based on steps including: generating the group interest score based on a set of features from a respective group, wherein the set of features are identified by a predetermined list of features that are characteristic of the activities recorded in the events in the respective group, wherein the total interest score for a formed component factors in the group interest score of the groups to which the nodes in the formed component belong.

8. The method of claim 1 , further comprising: determining a group interest score for each of the groups, based on steps including: generating the group interest score based on a set of features from a respective group, wherein the set of features are identified by a predetermined list of features that are characteristic of the activities recorded in the events in the respective group; and ranking the number of groups based on their group interest scores, wherein only a predetermined number of top ranked groups are further processed for constructing links for nodes between different groups.

9. The method of claim 1 , further comprising: determining a group interest score for each of the groups, based on steps including: generating the group interest score based on a set of features from a respective group, wherein the set of features are identified by a predetermined list of features that are characteristic of the activities recorded in the events in the respective group; ranking the number of groups based on their group interest scores, wherein only a predetermined number of top ranked groups are further processed for constructing links for nodes between different groups; and performing clustering for the number of groups after normalizing values in the set of features in each group.

10. The method of claim 1 , further comprising: determining a link score for each link in the formed components.

11. The method of claim 1 , further comprising: determining a link score for each link in the formed components, wherein the link score is determined based on a number of common nodes between the groups with which the formed component is associated.

12. The method of claim 1 , further comprising: determining a link score for each link in the formed components, wherein the link score is determined based on a distance in time between the groups with which the formed component is associated.

13. The method of claim 1 , further comprising: determining a link score for each link in the formed components, wherein the link score is determined based on an anomaly score of each node in the formed component.

14. The method of claim 1 , further comprising: determining a link score for each link in the formed components, wherein the total interest score for the formed component factors in the link score of the link that connects the nodes in the formed component.

15. The method of claim 1 , further comprising: creating a new graph using the formed components, wherein the new graph includes the nodes with respective links and corresponding groups.

16. The method of claim 1 , further comprising: creating a new graph using the formed components, wherein the new graph includes the nodes with respective links and corresponding group, wherein the nodes in the new graph are coupled to underlying events so that, responsive to a request, the underlying events are produced as supporting evidence.

17. The method of claim 1 , further comprising: before assigning nodes to groups, filtering the nodes and links in the relationship graph by removing nodes that include a whitelisted entity.

18. The method of claim 1 , further comprising: before assigning nodes to groups, filtering the nodes and links in the relationship graph by removing nodes that include an entity having an exceeding number of anomaly links to other entities as compared to a threshold.

19. The method of claim 1 , wherein the total interest score for a formed component increases exponentially when the events underlying the formed component matches the pattern of interest.

20. The method of claim 1 , wherein the total interest score for a formed component increases exponentially when the events underlying the formed component matches the pattern of interest, wherein the total interest score for a formed component decreases exponentially when the events underlying the formed component mismatches the pattern of interest.

21. The method of claim 1 , wherein the pattern of interest includes definitions for a sequence and an anti-sequence associated with an anomaly.

22. The method of claim 1 , wherein the pattern of interest includes definitions for a sequence and an anti-sequence associated with an anomaly, wherein the total interest score for a formed component decreases exponentially when the events underlying the formed component matches the anti-sequence.

23. The method of claim 1 , wherein the pattern of interest includes a malware installation followed by a file transfer or a beaconing anomaly.

24. The method of claim 1 , further comprising: performing a network security related action on the identified component.

25. The method of claim 1 , wherein the entities are users, computing devices, or any combination thereof.

26. The method of claim 1 , wherein the time range is more than one day, and wherein the time unit is one day.

27. The method of claim 1 , wherein steps recited in the method are repeated at a predetermined periodicity.

28. The method of claim 1 , wherein steps recited in the method are performed by a batch analysis engine that is implemented using APACHE SPARK™, and wherein the data store is implemented using APACHE HADOOP™.

29. A computer system comprising: a processor; and a communication device, operatively coupled to the processor, through which to receive first event data indicative of computer network activity of an entity that is part of or interacts with a computer network and second event data indicative of additional computer network activity associated with the entity; wherein the processor is configured to perform steps including: accessing, from a data store, a relationship graph corresponding to a time range, the time range having a number of time units, the relationship graph having entities as nodes and relationships among the entities as links, the relationship graph reflecting a batch of events that occurred during the time range, wherein each event of the batch of events includes timestamped, raw machine data that reflects one or more of: (1) activity occurred in an information technology (IT) or a security technology environment, (2) a time at which the activity occurred, and (3) a number of entities associated with the activity; assigning the nodes in the relationship graph to groups based on event timestamps, each group corresponds to a time unit and including nodes associated with activities that occurred in the time unit; constructing links for nodes between different groups, each link representing a relationship between nodes as established by a respective activity recorded in the batch of events, each chain of linked nodes forming a component; computing a total interest score for each of formed components, wherein the total interest score reflects a totality of interest generated from all nodes attached to a given link; adjusting the total interest score for each of the formed components based on comparing events underlying a component with a pattern of interest, wherein the pattern of interest identifies an expected temporal order and/or logical relationship in underlying events for such component to be of interest; identifying a component for further security scrutiny based on the adjusted total interest score.

30. A non-transitory machine-readable storage medium for use in a processing system, the non-transitory machine-readable storage medium storing instructions, an execution of which in the processing system causes the processing system to perform operations comprising: accessing, from a data store, a relationship graph corresponding to a time range, the time range having a number of time units, the relationship graph having entities as nodes and relationships among the entities as links, the relationship graph reflecting a batch of events that occurred during the time range, wherein each event of the batch of events includes timestamped, raw machine data that reflects one or more of: (1) activity occurred in an information technology (IT) or a security technology environment, (2) a time at which the activity occurred, and (3) a number of entities associated with the activity; assigning the nodes in the relationship graph to groups based on event timestamps, each group corresponds to a time unit and including nodes associated with activities that occurred in the time unit; constructing links for nodes between different groups, each link representing a relationship between nodes as established by a respective activity recorded in the batch of events, each chain of linked nodes forming a component; computing a total interest score for each of formed components, wherein the total interest score reflects a totality of interest generated from all nodes attached to a given link; adjusting the total interest score for each of the formed components based on comparing events underlying a component with a pattern of interest, wherein the pattern of interest identifies an expected temporal order and/or logical relationship in underlying events for such component to be of interest; identifying a component for further security scrutiny based on the adjusted total interest score.