System and method for low-latency communication over unreliable networks

Abstract

A method for low-latency communication from a first device to a second device over an unreliable network using at least one predictive machine learning model, characterized in that the method includes: representing at least one frame of time series data at the first device, wherein the at least one frame of time series data is a series of data points indexed in time order; recording at least one output stream, a metadata associated with the at least one output stream, and a plurality of external inputs from the first device in an interaction recorder of the second device, wherein the at least one output stream includes the at least one frame of time series data; segmenting a background area of an image into at least one background area stream, wherein the at least one background area stream is captured from a plurality of users; compressing at least one character centered portion of the image into a character focus stream for enabling an output image to be treated as two streams; training the at least one predictive machine learning model at the first device for predictive frame regeneration by providing the at least one output stream from the interaction recorder as an input; transmitting the results or interactions in time series to the second device, from the first device; detecting at least one lost frame of time series data using the at least one predictive machine learning model, at the second device; regenerating the at least one lost frame of the time series data at the second device using the at least one predictive machine learning model based on the at least one output stream to obtain at least one regenerated frame of time series data; and comparing an application stream from a stream of data obtained from the unreliable network with the at least one regenerated frame of time series data obtained from the at least one predictive machine learning model at the second device using a decision engine, wherein the application stream includes the at least one frame of time series data.

1 . A system for low-latency communication from a first device to a second device over an unreliable network using at least one predictive machine learning model, wherein the system, when in operation:
 represents at least one frame of time series data at the first device, wherein the time series data is a series of data points indexed in time order;   records at least one output stream, a metadata associated with the at least one output stream, and a plurality of external inputs from the first device in an interaction recorder of the second device, wherein the at least one output stream comprises the at least one frame of time series data;   segments an image into at least one background area stream and at least one character-centered portion, wherein the at least one background area stream is captured from a plurality of users;   compresses the at least one character-centered portion of the image into a character focus stream;   detects, at the second device, at least one lost frame of time series data using a frame of time series data from previously received frames;   trains the at least one predictive machine learning model at the first device for a predictive frame regeneration by providing the at least one output stream from the interaction recorder as an input;   transmits results of the training or interactions, between the first device and the second device, in a time series to the second device, from the first device;   regenerates the at least one lost frame of time series data, at the second device, using the at least one predictive machine learning model based on the at least one output stream to obtain at least one regenerated frame of time series data; and   combines an output stream from an application stream with the at least one regenerated frame of time series data obtained from the at least one predictive machine learning model, at the second device to obtain a modified output stream, wherein the application stream comprises the at least one frame of time series data.   
     
     
         2 . A system for low-latency communication from a first device to a second device over an unreliable network as claimed in  claim 1 , wherein the results or interactions in the time series comprises a state space representation or the modified output stream of the at least one frame of time series data, wherein the state space representation comprises interactions between the first device and the second device. 
     
     
         3 . A system for low-latency communication from a first device to a second device over an unreliable network as claimed in  claim 1 , wherein training of the at least one predictive machine learning model comprises generating a plurality of predictive machine learning models based on a number of frames in a sequence and the second device computing capability. 
     
     
         4 . A system for low-latency communication from a first device to a second device over an unreliable network as claimed in  claim 3 , the plurality of predictive machine learning models comprises a stream source classification model, wherein the stream source classification model is selected by identifying the at least one predictive machine learning model to be used when an input is not tagged as a particular type. 
     
     
         5 . A system for low-latency communication from a first device to a second device over an unreliable network as claimed in  claim 2 , wherein the system, when in operation, provides the state space representation and the interaction between the first device and the second device as an input for training the at least one predictive machine learning model and generates a plurality of predictive machine learning model based on the input. 
     
     
         6 . A system for low-latency communication from a first device to a second device over an unreliable network as claimed in  claim 1 , wherein the system, when in operation, selects a suitable predictive machine learning model for the predictive frame regeneration based on the second device's computing capability and a quality of the at least one regenerated frame of time series data. 
     
     
         7 . A system for low-latency communication from a first device to a second device over an unreliable network as claimed in  claim 1 , wherein the predictive frame regeneration comprises the at least one background area stream and the character focus stream. 
     
     
         8 . A system for low-latency communication from a first device to a second device over an unreliable network as claimed in  claim 1 , wherein the at least one lost frame of the time series data is detected using a frame loss indicator. 
     
     
         9 . A system for low-latency communication from a first device to a second device over an unreliable network as claimed in  claim 1 , wherein the system, when in operation, detects a packet lost in the at least one frame of time series data by a packet sequence number or by using a mean or a median an inter-arrival time. 
     
     
         10 . A system for low-latency communication from a first device to a second device over an unreliable network as claimed in  claim 1 , wherein the system, when in operation, calibrates an acoustic model with a decoder, wherein the acoustic model enables the decoder to regenerate the at least one lost frame of the time series data from a lost data. 
     
     
         11 . A system for low-latency communication from a first device to a second device over an unreliable network as claimed in  claim 1 , wherein the system, when in operation, produces fill-frames using a different number of input frames as an input vector to the at least one predictive machine learning model to generate an output frame, wherein the output frame is of a different frame size. 
     
     
         12 . A system for low-latency communication from a first device to a second device over an unreliable network as claimed in  claim 11 , wherein the fill frames in a frame queue are replaced by actual frames that arrive later for improving an accuracy of subsequent frames to be generated using real-time time series data. 
     
     
         13 . A system for low-latency communication from a first device to a second device over an unreliable network as claimed in  claim 1 , wherein the system, when in operation, trains the at least one predictive machine learning model with specific stream content. 
     
     
         14 . A system for low-latency communication from a first device to a second device over an unreliable network as claimed in  claim 13 , wherein the system when in operation, associates a bundle model with the specific stream content, different input frame sizes and output frames into one package based on the second device's computing capability. 
     
     
         15 . A system for low-latency communication from a first device to a second device over an unreliable network as claimed in  claim 1 , wherein the system, when in operation, generates a confidence score of a quality of the at least one regenerated frame of time series data regenerated by the at least one predictive machine learning model, wherein the at least one regenerated frame of time series data with high confidence score beyond a specified confidence threshold is reused in a subsequent fill-frame generation. 
     
     
         16 . A cloud game interactive system for low-latency communication from a game execution system and a video stream decoder over an unreliable network using at least one predictive machine learning model, wherein the game execution system and the video stream decoder comprises one or more processors, one or more non-transitory computer-readable mediums storing one or more sequences of instructions, which when executed by the one or more processors, cause:
 representing at least one frame of time series data at the game execution system, wherein the at least one frame of time series data is a series of data points indexed in time order;
 recording at least one output stream, a metadata associated with the at least one output stream, and a plurality of external inputs from the game execution system in an interaction recorder of the video stream decoder, wherein the at least one output stream comprises the at least one frame of time series data; 
 segmenting an image into at least one background area stream and at least one character-centered portion, wherein the at least one background area stream is captured from a plurality of users; 
 compressing the at least one character-centered portion of the image into a character focus stream; 
 detecting, at the video stream decoder, at least one lost frame of time series data using a frame of time series data from previously received frames; 
 training the at least one predictive machine learning model at the game execution system for a predictive frame regeneration by providing the at least one output stream from the interaction recorder or an instantaneous application state, comprising interactive application state-variables, as an input, wherein the interactive application state-variables describe a state space representation of a closed-loop control defined by a user's interaction with an application that drives creation of the audio and video; 
 transmitting results of the training or interactions, between the game execution system and the video stream decoder, in a time series to the video stream decoder, from the game execution system; 
 regenerating the at least one lost frame of time series data, at video stream decoder, using the at least one predictive machine learning model based on the at least one output stream to obtain at least one regenerated frame of time series data; and 
 combining an output stream from an application stream with the at least one regenerated frame of time series data obtained from the at least one predictive machine learning model, at the video stream decoder to obtain a modified output stream, wherein the application stream comprises the at least one frame of time series data. 
   
     
     
         17 . An adaptive model selection system, for low-latency communication from a first device to a second device over an unreliable network using at least one predictive machine learning model, wherein the first device and the second device comprises one or more processors, one or more non-transitory computer-readable mediums storing one or more sequences of instructions, which when executed by the one or more processors, cause:
 representing at least one frame of time series data at the first device, wherein the at least one frame of time series data is a series of data points indexed in time order;   recording at least one output stream, a metadata associated with the at least one output stream, and a plurality of external inputs from the first device in an interaction recorder of the second device, wherein the at least one output stream comprises the at least one frame of time series data;   segmenting an image into at least one background area stream and at least one character-centered portion, wherein the at least one background area stream is captured from a plurality of users;   compressing the at least one character-centered portion of the image into a character focus stream;   detecting, at the second device, at least one lost frame of time series data using a frame of time series data from previously received frames;   pushing, from the second device to the first device, a decoded frame of time series data;   selecting, at the first device, a best set of predictive model package to obtain bundle models, and sending, from the first device to the second device, the best set of predictive model package;   training the at least one predictive machine learning model, from the best set of predictive model package, at the first device for a predictive frame regeneration by providing the at least one output stream from the interaction recorder as an input;   transmitting results of the training or interactions, between the first device and the second device, in a time series to the second device, from the first device;   regenerating the at least one lost frame of time series data, at the second device, using the at least one predictive machine learning model based on the at least one output stream to obtain at least one regenerated frame of time series data; and   combining an output stream from an application stream with the at least one regenerated frame of time series data obtained from the at least one predictive machine learning model, at the second device to obtain a modified output stream, wherein the application stream comprises the at least one frame of time series data.   
     
     
         18 . An adaptive model selection system, for low-latency communication from a first device to a second device over an unreliable network using at least one predictive machine learning model as claimed in  claim 17 , wherein the one or more processors is further configured to train the at least one predictive machine learning model with specific stream content. 
     
     
         19 . An adaptive model selection system, for low-latency communication from a first device to a second device over an unreliable network using wherein at least one predictive machine learning model as claimed in  claim 18 , wherein the one or more processors is further configured to calibrate an acoustic model into a decoder, wherein the acoustic model enables the decoder to regenerate the at least one lost frame of the time series data from lost data. 
     
     
         20 . An adaptive model selection system, for low-latency communication from a first device to a second device over an unreliable network using wherein at least one predictive machine learning model as claimed in  claim 18 , wherein the one or more processors is further configured to produce fill-frames using a different number of input frames as an input vector to the at least one predictive machine learning model to generate an output frame, wherein the output frame is of a different frame size. 
     
     
         21 . An adaptive model selection system, for low-latency communication from a first device to a second device over an unreliable network using at least one predictive machine learning model as claimed in  claim 17 , wherein the one or more processors is further configured to generate a confidence score of a quality of the at least one regenerated frame of time series data regenerated by the at least one predictive machine learning model, wherein the at least one regenerated frame of time series data with high confidence score beyond a specified confidence threshold is reused in a subsequent fill-frame generation.