Blackbox Memory Monitoring with a Calling Context Memory Map and Semantic Extraction

1. A computer implemented method for memory monitoring of a program execution with a calling context memory map and semantic extraction, comprising the steps of: monitoring memory activities such as allocation and de-allocation of memory objects as a program is being executed by a calling context memory map generation; extracting by said monitoring allocated or de-allocated memory information from the program and calling context and recording memory related captured information into a memory object map called a calling context memory map, responsive to detected events during said step of monitoring memory activities; extracting semantic information from the memory object map, by a semantic extraction and the memory map is augmented with the extracted information in a semantic annotated memory map; sending by a monitoring agent queries to a semantic annotated memory map for detecting memory anomalies such as memory leaks or detection of bloated memory; and changing memory monitoring policies responsive to query results or monitoring logic.

2. The method of claim 1 , wherein said memory monitoring policies comprise that once types of memory objects of interest are selected, new policies are set to redefine scope of said monitoring of memory activities.

3. The computer implemented method of claim 1 , wherein the calling context memory map generation comprises, depending on the mechanism of calling context encoding, two usage scenarios, the first scenario being a calling context encoding mechanism that determines the encoding values statically in program analysis with the first scenario approach entailing the original program being transformed into another version with instrumented code that updates calling context IDs (CCID) determined at static analysis, and the second scenario being another calling context encoding approach which does not determine the encoding value beforehand and can generate unique encoding values on the fly without requiring the availability of a program control graph, application of the second scenario is associated with the calling context encoding occurring dynamically on the program's execution.

4. The computer implemented method of claim 1 , wherein the calling context memory map generation comprises that in order to generate a map of memory objects, the program's memory related operations should be captured with each process having its own memory map, and when a new process is forked or a process is killed, the corresponding map is created or destroyed, when a new memory object is allocated or de-allocated, such event is captured and the corresponding entry of the object is created in the map or removed from the map, and a difference from existing components entails use of calling context encoded values as identifiers of memory objects whose values are extracted from the program execution when the allocation events or de-allocation events happen and they are stored in the memory object entries in the memory map.

5. The computer implemented method of claim 1 , wherein the calling context memory map comprises how information of memory objects is stored and managed in the memory map, each memory object being represented as an N-tuple data structure that records properties of each object with S N being a starting address of the object, Z N being a size of the object and C M and Fo being encoded calling contexts at the object's allocation and deallocation.

6. The computer implemented method of claim 1 , wherein the calling context memory map comprises extra fields for detecting potential garbage objects with H N being a hash of content and D N being a dirty bit, the hash and dirty bit capable of indicating whether the objects have been modified and the hash and dirty bit values can be used to identify potential garbage objects by detecting the usage of objects.

7. The computer implemented method of claim 1 , wherein the calling context memory map comprises an encoded calling context with each object in the memory map being linkable to a list of functions involved in its lifetime events, such as the allocation and de-allocation, and other properties to check garbage objects for assisting memory analysis such as memory bloating and memory leakage.

8. The computer implemented method of claim 1 , wherein the sematic extraction comprises how the captured memory objects are analyzed and summarized so that the users can understand their details and let users understand data objects without source code or domain knowledge on the application so that they can select the data objects of their intereset for monitoring for the purpose of a blackbox monitoring which does not require the availability of source code of the monitored software in the program by constructing a mapping between calling context and the data type, wherein if the code for the program is available the mapping can be constructed by using source code analysis.

9. The computer implemented method of claim 1 , wherein the semantic extraction comprises memory objects in the calling context memory map being clustered based on their encoded calling context values, for each cluster, the memory values corresponding to the objects being collected and analyzed and then the memory values being collected from the monitored process' memory which can be done without slowing down the monitored process by using a dedicated monitoring thread different from the monitored program.

10. The computer implemented method of claim 1 , wherein the semantic extraction comprises that after the memory values of all objects are collected there is a comparing of values that correspond to a field, and various information regarding the values of objects are extracted, the extracted information being organized like a data type for each object and enabling an understanding of data.

11. The computer implemented method of claim 1 , wherein the semantic annotated memory map comprises a map that can support queries to retrieve semantic types and the memory objects of the types.

12. A system for memory monitoring of a program execution with a calling context memory map and semantic extraction, comprising: a processor; code executable by the processor for monitoring memory activities such as allocation and de-allocation of memory objects as the program is being executed by an execution and calling context memory map generation; code executable by the processor for extracting by said monitoring allocated or de-allocated memory information from the program and calling context and recording memory related captured information into a memory object map called a calling context memory map, responsive to detected events during said step of monitoring memory activities; code executable by the processor for extracting semantic information from the memory object map by a semantic extraction and the memory map being augmented with the extracted information in a semantic annotated memory map; code executable by the processor for sending by a monitoring agent queries to a semantic annotated memory map for detecting memory anomalies such as memory leaks or detection of bloated memory; and code executable by the processor for changing memory monitoring policies responsive to query results or monitoring logic.

13. The system of claim 12 , wherein the calling context memory map generation comprises, depending on the mechanism of calling context encoding, two usage scenarios, the first scenario being a calling context encoding mechanism that determines the encoding values statically in program analysis with the first scenario approach entailing the original program being transformed into another version with instrumented code that updates calling context IDs (CCID) determined at static analysis, and the second scenario being another calling context encoding approach which does not determine the encoding value beforehand and can generate unique encoding values on the fly without requiring the availability of a program control graph, application of the second scenario is associated with the calling context encoding occurring dynamically on the program's execution.

14. The system of claim 12 , wherein the calling context memory map generation comprises that in order to generate a map of memory objects, the program's memory related operations should be captured with each process having its own memory map, and when a new process is forked or a process is killed, the corresponding map is created or destroyed, when a new memory object is allocated or de-allocated, such event is captured and the corresponding entry of the object is created in the map or removed from the map, and a difference from existing components entails use of calling context encoded values as identifiers of memory objects whose values are extracted from the program execution when the allocation events or de-allocation events happen and they are stored in the memory object entries in the memory map.

15. The system of claim 12 , wherein the calling context memory map comprises how information of memory objects is stored and managed in the memory map, each memory object being represented as an N-tuple data structure that records properties of each object with S N being a starting address of the object, Z N being a size of the object and C M and Fo being encoded calling contexts at the object's allocation and deallocation.

16. The system of claim 12 , wherein the calling context memory map comprises extra fields for detecting potential garbage objects with H N being a hash of content and D N being a dirty bit, the hash and dirty bit capable of indicating whether the objects have been modified and the hash and dirty bit values can be used to identify potential garbage objects by detecting the usage of objects.

17. The system of claim 12 , wherein the calling context memory map comprises an encoded calling context with each object in the memory map being linkable to a list of functions involved in its lifetime events, such as the allocation and de-allocation, and other properties to check garbage objects for assisting memory analysis such as memory bloating and memory leakage.