Streaming application environment with recovery of lost or delayed input events

Abstract

In a streaming application environment coupled to a remote device over a packet-switching network, correction events may be synthesized from input events received from the remote device and injected into a virtual streaming application to account for lost or delayed input events. In addition, trailing events may be reissued by a remote device in frames during which no new input events are generated.

What is claimed is: 
     
         1 . A method implemented by one or more processors of a device, the method comprising:
 receiving an output display representation from a remote streaming application environment coupled to the device over a packet-switched network, the output display representation generated by a virtual streaming application resident in the remote streaming application environment;   rendering a plurality of frames of the received output display representation on a display associated with the device;   in response to inputs generated on the device during rendering of the plurality of frames via an input component of the device, issuing a plurality of input events for communication over the packet-switched network to the remote streaming application environment; and   in response to a determination that no inputs are generated during a frame among the plurality of frames, reissuing a trailing event issued in a prior frame for communication over the packet-switched network to the remote streaming application environment,
 wherein the trailing event corresponds to a given input event, of the plurality of input events, issued in the prior frame before the determination that no inputs are generated during the frame. 
   
     
     
         2 . The method of  claim 1 , wherein each input event, of the plurality of input events, is associated with a corresponding timestamp, and wherein the trailing event is associated with a trailing event timestamp, and wherein the trailing event timestamp corresponds to the corresponding timestamp of the given input event for which the trailing event is reissued. 
     
     
         3 . The method of  claim 2 , wherein the trailing event is placed in a priority queue, along with the plurality of input events, as a last input event, of the plurality of input events, based on the trailing event timestamp corresponding to the corresponding timestamp of the given input event 
     
     
         4 . The method of  claim 3 , further comprising:
 rendering a plurality of additional frames of the received output display representation on the display associated with the device, the plurality of additional frames of the received output display representation being based on the priority queue.   
     
     
         5 . The method of  claim 2 , wherein each input, of the plurality of input events, is associated with one or more corresponding input event parameters. 
     
     
         6 . The method of  claim 5 , wherein the one or more corresponding input event parameters include corresponding input coordinates with respect the output display representation and/or corresponding input identifiers with respect to a type of input event. 
     
     
         7 . The method of  claim 6 , wherein the type of input event corresponds to an input start event or an input stop event. 
     
     
         8 . The method of  claim 1 , further comprising:
 subsequent to issuing the plurality of input events for communication over the packet-switched network to the remote streaming application environment, reissuing the plurality of input events for communication over the packet-switched network to the remote streaming application environment.   
     
     
         9 . The method of  claim 8 , wherein issuing each input event, of the plurality of input events, for communication over the packet-switched network to the remote streaming application environment comprises:
 issuing a first input event, of the plurality of input events; and   issuing a second input event, of the plurality of input events.   
     
     
         10 . The method of  claim 9 , wherein reissuing each input event, of the plurality of input events, for communication over the packet-switched network to the remote streaming application environment comprises:
 in response to determining that a first delay has lapsed, reissuing the first input event, of the plurality of input events; and   in response to determining that a second delay has lapsed, reissuing the second input event, of the plurality of input events.   
     
     
         11 . The method of  claim 1 , further comprising:
 in response to a determination that inputs are generated during the frame among the plurality of frames, continue issuing the plurality of input events for communication over the packet-switched network to the remote streaming application environment.   
     
     
         12 . A device comprising:
 an input component;   at least one processor; and   memory storing instructions that, when executed, cause the at least one processor to:
 receive an output display representation from a remote streaming application environment coupled to the device over a packet-switched network, the output display representation generated by a virtual streaming application resident in the remote streaming application environment; 
   render a plurality of frames of the received output display representation on a display component associated with the device;   in response to inputs generated on the device during rendering of the plurality of frames via the input component, issue a plurality of input events for communication over the packet-switched network to the remote streaming application environment; and   in response to a determination that no inputs are generated during a frame among the plurality of frames, reissue a trailing event issued in a prior frame for communication over the packet-switched network to the remote streaming application environment,
 wherein the trailing event corresponds to a given input event, of the plurality of input events, issued in the prior frame before the determination that no inputs are generated during the frame. 
   
     
     
         13 . The device of  claim 12 , wherein each input event, of the plurality of input events, is associated with a corresponding timestamp, wherein the trailing event is associated with a trailing event timestamp, and wherein the trailing event timestamp corresponds to the corresponding timestamp of the given input event for which the trailing event is reissued. 
     
     
         14 . The device of  claim 13 , wherein the trailing event is placed in a priority queue, along with the plurality of input events, as a last input event, of the plurality of input events, based on the trailing event timestamp corresponding to the corresponding timestamp of the given input event 
     
     
         15 . The device of  claim 14 , wherein the instructions further cause the at least one processor to:
 render a plurality of additional frames of the received output display representation on the display associated with the device, the plurality of additional frames of the received output display representation being based on the priority queue.   
     
     
         16 . The device of  claim 13 , wherein each input, of the plurality of input events, is associated with one or more corresponding input event parameters, wherein the one or more corresponding input event parameters include corresponding input coordinates with respect the output display representation and/or corresponding input identifiers with respect to a type of input event, and wherein the type of input event corresponds to an input start event or an input stop event. 
     
     
         17 . The device of  claim 12 , wherein the instructions further cause the at least one processor to:
 subsequent to issuing the plurality of input events for communication over the packet-switched network to the remote streaming application environment, reissue the plurality of input events for communication over the packet-switched network to the remote streaming application environment.   
     
     
         18 . The device of  claim 17 , wherein the instruction to issue each input event, of the plurality of input events, for communication over the packet-switched network to the remote streaming application environment comprise instructions to:
 issue a first input event, of the plurality of input events; and   issue a second input event, of the plurality of input events.   
     
     
         19 . The device of  claim 18 , wherein the instructions to reissue each input event, of the plurality of input events, for communication over the packet-switched network to the remote streaming application environment comprise instructions to:
 in response to determining that a first delay has lapsed, reissue the first input event, of the plurality of input events; and   in response to determining that a second delay has lapsed, reissue the second input event, of the plurality of input events.   
     
     
         20 . A device comprising:
 an input component;   a display component;   at least one processor; and   memory storing instructions that, when executed, cause the at least one processor to:
 receive an output display representation from a remote streaming application environment coupled to the device over a packet-switched network, the output display representation generated by a virtual streaming application resident in the remote streaming application environment; 
 render a plurality of frames of the received output display representation on the display component associated with the device; 
 in response to inputs generated on the device during rendering of the plurality of frames via the input component, issue a plurality of input events for communication over the packet-switched network to the remote streaming application environment; and 
 in response to a determination that no inputs are generated during a frame among the plurality of frames, reissue a trailing event issued in a prior frame for communication over the packet-switched network to the remote streaming application environment,
 wherein the trailing event corresponds to a given input event, of the plurality of input events, issued in the prior frame before the determination that no inputs are generated during the frame.