Remote user interface updates using difference and motion encoding

Abstract

Frames of user interface graphical data can be remotely rendered at a client during a remote session with a server by providing graphical data commands to the client. A server identifies moving objects having a first position in the current frame and a second position in the new frame, generates motion hints for the moving objects, and reduces the motion hints based on collision detection, motion verification and other factors. Motion commands are generated for the reduced set of motion hints and applied to a copy of the current frame at the server. Differences between the modified current frame and the new frame are then encoded as delta commands. The server then sends the motion commands and delta commands to the client. The client receives and applies the commands to the current frame to render the new frame.

1 . A method for updating a user interface, comprising:
 establishing a remote session between a thin client and a server over a network;   providing current graphical frame data for a current frame to the thin client by the server;   identifying a new frame by the server to be rendered at the thin client;   encoding delta data associated with differences in pixel data between the new frame and the updated current frame; and   transmitting the set of delta data to the thin client.   
     
     
         2 . The method of  claim 1 , wherein the encoding includes:
 separately encoding delta data for each of two or more graphical data formats.   
     
     
         3 . The method of  claim 1 , wherein the encoding includes:
 generating a set of dirty rectangles indicating a change in pixel values between the current frame and new frame; and   encoding pixel delta data associated with pixels contained within the dirty rectangles.   
     
     
         4 . The method of  claim 3 , wherein the generating a set of dirty rectangles includes:
 identifying two or more dirty rectangles that overlap; and   replacing the two or more overlapping dirty rectangles with two or more non- overlapping dirty rectangles which encompass the same pixels in the new frame.   
     
     
         5 . The method of  claim 1 , wherein the encoding includes:
 accessing pixel data for the current frame;   modifying the pixel data for the current frame; and   encoding delta data associated with the difference between the modified pixel data for the current frame and the pixel data for the new frame.   
     
     
         6 . A system comprising:
 a processor; and   computer readable storage coupled to the processor, the computer readable storage having instructions stored thereon which, when executed on the processor, cause the processor to:   establish a remote session between a thin client and a server over a network;   provide current graphical frame data for a current frame to the thin client by the server;   identify a new frame by the server to be rendered at the thin client;   encode delta data associated with differences in pixel data between the new frame and the updated current frame; and   transmit the set of delta data to the thin client.   
     
     
         7 . The system of  claim 6 , wherein the instructions which cause the processor to encode cause the processor to:
 separately encode delta data for each of two or more graphical data formats.   
     
     
         8 . The system of  claim 6 , wherein the instructions which cause the processor to encode cause the processor to:
 generate a set of dirty rectangles indicating a change in pixel values between the current frame and new frame; and   encode pixel delta data associated with pixels contained within the dirty rectangles.   
     
     
         9 . The system of  claim 8 , wherein the instructions which cause the processor to generate a set of dirty rectangles cause the processor to:
 identify two or more dirty rectangles that overlap; and   replace the two or more overlapping dirty rectangles with two or more non- overlapping dirty rectangles which encompass the same pixels in the new frame.   
     
     
         10 . The system of  claim 6 , wherein the instructions which cause the processor to encode cause the processor to:
 access pixel data for the current frame;   modify the pixel data for the current frame; and   encode delta data associated with the difference between the modified pixel data for the current frame and the pixel data for the new frame.   
     
     
         11 . A computer readable storage device having instructions stored thereon which, when executed on a processor, cause the processor to perform:
 establishing a remote session between a thin client and a server over a network;   providing current graphical frame data for a current frame to the thin client by the server;   identifying a new frame by the server to be rendered at the thin client;   encoding delta data associated with differences in pixel data between the new frame and the updated current frame; and   transmitting the set of delta data to the thin client.   
     
     
         12 . The computer readable storage device of  claim 11 , wherein the encoding includes:
 separately encoding delta data for each of two or more graphical data formats.   
     
     
         13 . The computer readable storage device of  claim 11 , wherein the encoding includes:
 generating a set of dirty rectangles indicating a change in pixel values between the current frame and new frame; and   encoding pixel delta data associated with pixels contained within the dirty rectangles.   
     
     
         14 . The computer readable storage device of  claim 13 , wherein the generating a set of dirty rectangles includes:
 identifying two or more dirty rectangles that overlap; and   replacing the two or more overlapping dirty rectangles with two or more non-overlapping dirty rectangles which encompass the same pixels in the new frame.   
     
     
         15 . The computer readable storage device of  claim 11 , wherein the encoding includes:
 accessing pixel data for the current frame;   modifying the pixel data for the current frame; and   encoding delta data associated with the difference between the modified pixel data for the current frame and the pixel data for the new frame.