Real-time shellcode detection and prevention

Abstract

Methods, storage systems and computer program products implement embodiments of the present invention for protecting a computing device, which includes a processor and a memory and is coupled to a storage device storing a set of one or more files. In embodiments of the present invention, a call to a specified function for execution by the processor is detected, and a stack trace for the call to the specified function is generated in the memory. Upon detecting, in the stack trace, a stack frame including a return address referencing a shellcode region in the memory, wherein the shellcode region includes executable code that was not loaded from any given file on the storage device, then the referenced executable code is compared to a list of malicious shellcode. Finally, a preventive action is initiated upon detecting a match between the referenced executable code and one of malicious shellcodes in the list.

The invention claimed is: 
     
       1. A method for protecting a computing device, which includes a processor and a memory and is coupled to a storage device storing a set of one or more files, the method comprising:
 detecting, by the processor, a call to a specified function for execution by the processor; 
 generating, in the memory, a stack trace for the call to the specified function; 
 detecting, in the stack trace, a stack frame comprising a return address referencing a page in the memory containing executable code, the stack frame including a flag indicating whether the executable code was loaded from a file on the storage device; 
 in response to the flag indicating that the executable code was not loaded from any file on the storage device, 
 comparing the referenced executable code to a list of malicious shellcode; and 
 initiating a preventive action upon detecting a match between the referenced executable code and one of malicious shellcodes in the list. 
 
     
     
       2. The method according to  claim 1 , wherein the referenced executable code executes in kernel mode. 
     
     
       3. The method according to  claim 1 , wherein the referenced executable code executes in user mode. 
     
     
       4. The method according to  claim 1 , wherein the shellcode comprises an application programming interface (API) hashing algorithm. 
     
     
       5. The method according to  claim 1 , wherein the list of malicious shellcode comprises a list of signatures, and wherein comparing the referenced executable code to the list of malicious shellcode comprises extracting one or more shellcode buffers from the referenced executable code, and comparing the one or more extracted shellcode buffers to the list of signatures. 
     
     
       6. The method according to  claim 5 , wherein a signature in the list comprises a wildcard. 
     
     
       7. The method according to  claim 1 , wherein the specified function loads an executable library to the memory. 
     
     
       8. The method according to  claim 1 , wherein the specified function creates a new thread in the memory. 
     
     
       9. The method according to  claim 1 , wherein the specified function creates a new process in the memory. 
     
     
       10. The method according to  claim 1 , wherein the specified function initiates a new connection. 
     
     
       11. The method according to  claim 1 , wherein the specified function listens for an incoming connection. 
     
     
       12. The method according to  claim 1 , wherein the specified function communicates with kernel drivers so as to provide a view of input data and output data. 
     
     
       13. The method according to  claim 1 , wherein the memory comprises a set of memory pages, and wherein the specified function allocates one or more of the memory pages to a process in the memory. 
     
     
       14. The method according to  claim 1 , wherein the memory comprises a set of memory addresses, and wherein the specified function changes a memory permission of a given memory address. 
     
     
       15. The method according to  claim 1 , wherein the specified function retrieves a buffer from an address in the memory. 
     
     
       16. The method according to  claim 1 , wherein the specified function writes a buffer to an address in the memory. 
     
     
       17. The method according to  claim 1 , wherein the specified function sets a thread context of a thread in the memory. 
     
     
       18. The method according to  claim 1 , wherein the specified function adds an asynchronous procedure call (APC) to an APC queue in a thread in the memory. 
     
     
       19. The method according to  claim 1 , wherein the specified function injects additional executable code into a process in the memory. 
     
     
       20. An apparatus for protecting a computing device, comprising:
 a memory; and 
 a processor coupled to a storage device storing a set of one or more files and configured:
 to detect a call to a specified function for execution by the processor, 
 to generate, in the memory a stack trace for the call to the specified function, 
 to detect, in the stack trace, a stack frame comprising a return address referencing a page in the memory containing executable code. the stack frame including a flag indicating whether the executable code was loaded from a file on the storage device, 
 to compare the referenced executable code to a list of malicious shellcode, in response to the flag indicating that the executable code was not loaded from any file on the storage device, and 
 to initiate a preventive action upon detecting a match between the referenced executable code and one of malicious shellcodes in the list. 
 
 
     
     
       21. A computer software product for protecting a computing device, which includes a processor and a memory and is coupled to a storage device storing a set of one or more files, the computer software product comprising a non-transitory computer-readable medium, in which program instructions are stored, which instructions, when read by a computer, cause the computer:
 to detect a call to a specified function for execution by the processor; 
 to generate, in the memory, a stack trace for the call to the specified function; 
 to detect, in the stack trace, a stack frame comprising a return address referencing a page in the memory containing executable code, the stack frame including a flag indicating whether the executable code was loaded from a file on the storage device; 
 to compare the referenced executable code to a list of malicious shellcode, in response to the flag indicating that the executable code was not loaded from any file on the storage device; and 
 to initiate a preventive action upon detecting a match between the referenced executable code and one of malicious shellcodes in the list.