Network impairment simulation framework for verification of real time interactive media streaming systems

Abstract

A method performed at an electronic device includes: executing an application; establishing a session between the executing application, a remote server associated with the application, and a media device that transmits output from the session to an output device, wherein the output from the session includes output from the executing application; and presenting to users of one or more second electronic devices on which the application is not executing and/or installed a session identifier associated with the session, wherein the session identifier facilitates joining of the session by the second electronic devices and identifies a controller configuration implementable on each of the one or more second electronic devices for enabling the users of the one or more second electronic devices to interact with the session.

1 - 21 . (canceled) 
     
     
         22 . A method comprising:
 at a computing system having one or more processors and memory:
 encoding a video stream associated with one or more operations of a network-enabled test application; 
 simulating transmission of the encoded video stream in a simulated network for a specified network simulation scenario using a first set of dynamic network parameters, wherein simulating the transmission includes generating, based on the encoded video stream, an additional video stream that is distinct from the encoded video stream and that reflects simulated real time effects of one or more modeled network impairments on the encoded video stream; and 
 determining a plurality of time-varying quality metrics for the additional video stream. 
   
     
     
         23 . The method of  claim 22 , wherein simulating the transmission of the encoded video stream in the simulated network includes selecting the specified network simulation scenario from a plurality of predefined network simulation scenarios. 
     
     
         24 . The method of  claim 22 , wherein at least one dynamic network parameter of the first set of dynamic network parameters is set to a first level during a first time period and a second level different from the first level during a second time period. 
     
     
         25 . The method of  claim 24 , wherein an additional dynamic parameter of the first set of dynamic network parameters is set to a third level during the first and second time periods. 
     
     
         26 . The method of  claim 25 , wherein the at least one dynamic parameter and the additional dynamic parameter are each selected from a group that includes latency, jitter, packet loss, channel noise, and channel bandwidth. 
     
     
         27 . The method of  claim 22 , further comprising:
 determining, based on the plurality of time-varying quality metrics, a simulated user experience value associated with the network-enabled test application and the specified network simulation scenario.   
     
     
         28 . The method of  claim 27 , wherein determining the simulated user experience value includes:
 deriving one or more simulated quality of service values from the plurality of time-varying quality metrics;   comparing the one or more simulated quality of service values to one or more target quality of service values; and   determining the simulated user experience value based on the comparison.   
     
     
         29 . The method of  claim 27 , further comprising:
 displaying a rendering of the additional video stream; and   determining the simulated user experience value based on quality of the displayed rendering of the additional video stream.   
     
     
         30 . The method of  claim 22 , further comprising:
 determining, based on the plurality of time-varying quality metrics, one or more performance aspects of the network-enabled test application.   
     
     
         31 . The method of  claim 22 , wherein the simulated real time effects include a specific dropped frame or a specific lost packet. 
     
     
         32 . The method of  claim 22 , further comprising deriving the dynamic network parameters for one or more predefined network simulation scenarios by:
 transmitting a first video stream including a first portion of content associated with the network-enabled test application to a user device through a real world communication environment;   characterizing performance of the network-enabled test application in the real world communication environment at the user device to produce first performance data; and   generating a simulated network based on the first performance data to approximate the real world communication environment.   
     
     
         33 . The method of  claim 32 , wherein deriving the dynamic network parameters for the one or more predefined network simulation scenarios further comprises:
 transmitting a second video stream including the first portion of content to the user device through the simulated network;   characterizing performance of the network-enabled test application in the simulated network at the user device to produce second performance data; and   adjusting one or more dynamic network parameters for a network simulation scenario corresponding to the real world communication environment based on a comparison of the first performance data with the second performance data.   
     
     
         34 . A computing system comprising one or more processors and memory coupled to the one or more processors, the memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for:
 encoding a video stream associated one or more operations of a network-enabled test application;   simulating transmission of the encoded video stream in a simulated network for a specified network simulation scenario using a first set of dynamic network parameters, wherein simulating the transmission includes generating, based on the encoded video stream, an additional video stream that is distinct from the encoded video stream and that reflects simulated real time effects of one or more modeled network impairments on the encoded video stream; and   determining a plurality of time-varying quality metrics for the additional video stream.   
     
     
         35 . The computing system of  claim 34 , wherein simulating the transmission of the encoded video stream in the simulated network includes selecting the specified network simulation scenario from a plurality of predefined network simulation scenarios. 
     
     
         36 . The computing system of  claim 34 , wherein at least one dynamic network parameter of the first set of dynamic network parameters is set to a first level during a first time period and a second level different from the first level during a second time period. 
     
     
         37 . The computing system of  claim 36 , wherein an additional dynamic parameter of the first set of dynamic network parameters is set to a third level during the first and second time periods. 
     
     
         38 . The computing system of  claim 37 , wherein the at least one dynamic parameter and the additional dynamic parameter are each selected from a group that includes latency, jitter, packet loss, channel noise, and channel bandwidth. 
     
     
         39 . The computing system of  claim 34 , further comprising:
 determining, based on the plurality of time-varying quality metrics, a simulated user experience value associated with the network-enabled test application and the specified network simulation scenario.   
     
     
         40 . The computing system of  claim 39 , wherein determining the simulated user experience value includes:
 deriving one or more simulated quality of service values from the plurality of time-varying quality metrics;   comparing the one or more simulated quality of service values to one or more target quality of service values; and   determining the simulated user experience value based on the comparison.   
     
     
         41 . The computing system of  claim 39 , further comprising:
 displaying a rendering of the additional video stream; and   determining the simulated user experience value based on quality of the displayed rendering of the additional video stream.   
     
     
         42 . The computing system of  claim 34 , further comprising:
 determining, based on the plurality of time-varying quality metrics, one or more performance aspects of the network-enabled test application.   
     
     
         43 . The computing system of  claim 34 , wherein the simulated real time effects include a specific dropped frame or a specific lost packet. 
     
     
         44 . The computing system of  claim 34 , further comprising deriving the dynamic network parameters for one or more predefined network simulation scenarios by:
 transmitting a first video stream including a first portion of content associated with the network-enabled test application to a user device through a real world communication environment;   characterizing performance of the network-enabled test application in the real world communication environment at the user device to produce first performance data; and
 generating a simulated network based on the first performance data to approximate the real world communication environment. 
   
     
     
         45 . The computing system of  claim 44 , wherein deriving the dynamic network parameters for the one or more predefined network simulation scenarios further comprises:
 transmitting a second video stream including the first portion of content to the user device through the simulated network;   characterizing performance of the network-enabled test application in the simulated network at the user device to produce second performance data; and   adjusting one or more dynamic network parameters for a network simulation scenario corresponding to the real world communication environment based on a comparison of the first performance data with the second performance data.