Generic unpacking of program binaries

By hooking application programming interfaces in an execution environment, the return address for hooked application programming interface calls can be logged and used to determine when a packed binary has been unpacked. In one approach, memory allocations are detected and the return address is checked against the memory regions allocated. In another approach, the contents of memory at the return address in a pre-execution copy of the executable binary is compared with the contents of memory at the return address in the executing copy of the binary. This allows efficient detection of the completion of unpacking without knowledge of the unpacking technique. The unpacked binary may then be analyzed for possible malware.