Compiling executable code into a less-trusted address space

Abstract

Unsafe application programs that implement managed code can be executed in a secure fashion. In particular, an operating system can be configured to execute an application program in user mode, but handle managed code compilation through a type-safe JIT compiler operating in kernel mode. The operating system can also designate a single memory location to be accessed through multiple address spaces with different permission sets. An application program operating in user mode can be executed in the read/execute address space, while the JIT compiler operates in a read/write address space. When encountering one or more pointers to intermediate language code, the application runtime can send one or more compilation requests to a kernel mode security component, which validates the requests. If validated, the JIT compiler will compile the requested intermediate language code, and the application program can access the compiled code from a shared memory heap.

1 . In a computerized environment comprising a memory, as well as a JIT compiler and one or more application programs loaded in the memory, a method of executing managed code so that untrusted program code can be compiled and executed in a manner that does not threaten or otherwise compromise system security, comprising:
 executing an application program from a first address space set with a first set of permissions for accessing a shared memory heap;   receiving one or more requests from the application program to compile one or more sets of intermediate language instructions;   compiling the one or more sets of intermediate language instructions into newly compiled code using a JIT compiler running in a second address space that has a second set of permissions for accessing the shared memory heap; and   passing the newly compiled code to the shared memory heap, wherein the application program can retrieve and execute the newly compiled code from the first address space.   
   
   
       2 . The method as recited in  claim 1 , further comprising an act of, upon receiving an indication that the newly compiled code has been passed to the shared memory heap, switching from a kernel mode to a user mode level of operation. 
   
   
       3 . The method as recited in  claim 2 , further comprising an act of the application program retrieving the compiled code, and executing the compiled code from the first address space. 
   
   
       4 . The method as recited in  claim 1 , wherein the first address space is configured with read/execute permissions with respect to accessing the shared memory heap, such that no component operating in the first address space can write to the shared memory heap. 
   
   
       5 . The method as recited in  claim 1 , wherein the second address space is configured to access the memory heap with read/write permissions, such that no component operating in the second address space can execute code in the memory heap. 
   
   
       6 . The method as recited in  claim 1 , wherein the JIT compiler is operating in a higher-privilege mode, and the application program is running in a lower-privilege mode. 
   
   
       7 . The method as recited in  claim 1 , wherein the JIT compiler is constrained to execute within one or more type-safety restraints, but configured to accept and compile intermediate language code that is not type-safe. 
   
   
       8 . The method as recited in  claim 7 , wherein the JIT compiler performs the acts of:
 receiving one or more requests to perform a function that violates a security restraint for the JIT compiler; and   rejecting the one or more requests to perform the function, or discontinuing compiling the one or more sets of intermediate language instructions.   
   
   
       9 . The method as recited in  claim 1 , further comprising an act of, upon receiving the one or more requests from the application program, activating a kernel mode level of operation. 
   
   
       10 . The method as recited in  claim 9 , wherein the act of activating a kernel mode level of operation includes an act of initiating a kernel mode security component. 
   
   
       11 . The method as recited in  claim 10 , wherein the one or more requests from the application program are received by a kernel mode security component. 
   
   
       12 . The method as recited in  claim 11 , further comprising an act of the kernel mode security component validating the one or more requests from the application program. 
   
   
       13 . The method as recited in  claim 12 , wherein the act of validating the one or more requests comprises an act of determining whether a handle included in the one or more requests is valid. 
   
   
       14 . In a computerized environment comprising a storage, a JIT compiler, and one or more application programs loaded in memory, a method of generating computer executable program code in a manner that uses JIT compilation while avoiding security violations, comprising:
 receiving application program code that includes executable code and code to be compiled;   executing the executable code in a lower-privilege mode and in a first address space;   identifying one or more pointers in the executable code for at least some code to be compiled;   switching to a higher-privilege mode; and   compiling the at least some code in a different address space using a compiler operating in the higher-privilege mode.   
   
   
       15 . The method as recited in  claim 14 , wherein the application program code comprises part of a video game application that is received from the storage into a video game operating system. 
   
   
       16 . The method as recited in  claim 14 , wherein the compiler is a type-safe JIT compiler configured to handle only type-safe requests, but otherwise configured to compile type-safe or non-type-safe intermediate language code. 
   
   
       17 . The method as recited in  claim 14 , wherein the higher-privilege mode is a kernel mode level of operation, and the lower-privilege mode is a user level of operation. 
   
   
       18 . The method as recited in  claim 14 , wherein the first address space is configured to access a memory heap with read/execute permissions, and the second address space is configured to access the memory heap with read/write permissions. 
   
   
       19 . The method as recited in  claim 14 , further comprising the acts of:
 switching to the lower-privilege mode upon identifying that the at least some code has been compiled; and   executing the compiled at least some code in the first address space.   
   
   
       20 . In a computerized environment comprising a memory, a JIT compiler, and one or more application programs loaded in the memory, a computer program storage product having computer executable instructions stored thereon that, when executed, cause one or more processors to perform a method comprising:
 executing an application program from a first address space set with a first set of permissions for accessing a shared memory heap;   receiving one or more requests from the application program to compile one or more sets of intermediate language instructions;   compiling the one or more sets of intermediate language instructions into newly compiled code using a JIT compiler running in a second address space that has a second set of permissions for accessing the shared memory heap; and   passing the newly compiled code to the shared memory heap, wherein the application program can retrieve and execute the newly compiled code from the first address space.