Feature prediction for efficient video processing

Abstract

A technique is described herein that interprets some frames in a stream of video content as key frames and other frames as predicted frames. The technique uses an image analysis system to produce feature information for each key frame. The technique uses a prediction model to produce feature information for each predicted frame. The prediction model operates on two inputs: (1) feature information that has been computed for an immediately-preceding frame; and (2) frame-change information. A motion-determining model produces the frame-change information by computing the change in video content between the current frame being predicted and the immediately-preceding frame. The technique reduces the amount of image-processing operations that are used to process the stream of video content compared to a base case of processing all of the frames using the image analysis system. As such, the technique uses less computing resources compared to the base case.

What is claimed is: 
     
       1. A method for processing a stream of video frames, comprising:
 obtaining a first frame of video content, the first frame being interpreted as a key frame; 
 converting the first frame into first feature information using an image analysis system; 
 caching the first feature information in a data store; 
 obtaining a second frame of video content, the second frame being interpreted as a predicted frame; 
 mapping the first frame and the second frame into first frame-change information using a motion-determining model, the first frame-change information expressing a change in video content from the first frame to the second frame; 
 converting the first frame-change information and the first feature information into second feature information using a prediction model; and 
 caching the second feature information in the data store, 
 the method interpreting every nth frame in the stream of video frames as a key frame, and frames between neighboring key frames as predicted frames, n being specified by a configuration parameter, and 
 the method switching between use of the image analysis system and use of the motion-determining model and the prediction model depending on whether a key frame is encountered or a predicted frame is encountered in the stream of video frames. 
 
     
     
       2. The method of  claim 1 , further comprising:
 obtaining a third frame of video content, the third frame being interpreted as another predicted frame; 
 mapping the second frame and the third frame into second frame-change information using the motion-determining model, the second frame-change information expressing a change in video content from the second frame to the third frame; 
 converting the second frame-change information and the second feature information into third feature information using the prediction model; and 
 caching the third feature information in the data store. 
 
     
     
       3. The method of  claim 1 , further comprising:
 obtaining a third frame of video content, the third frame being interpreted as another key frame; 
 converting the third frame into third feature information using the image analysis system; and 
 caching the third feature information in the data store. 
 
     
     
       4. The method of  claim 1 , further comprising converting each instance of feature information into output information using another image analysis system. 
     
     
       5. The method of  claim 1 , wherein the image analysis system includes a model that is trained independently of, and prior to, training the motion-determining model and the prediction model. 
     
     
       6. The method of  claim 1 , wherein the motion-determining model and the prediction model are trained by:
 using the image analysis system, which has already been trained, to produce instances of ground-truth feature information for a set of video frames; 
 using the motion-determining model and the prediction model to produce instances of predicted feature information for video frames in the set that are interpreted as predicted frames; 
 determining differences between the instances of ground-truth feature information and counterpart instances of predicted feature information; 
 adjusting weights of the motion-determining model and the prediction model to reduce the differences; and 
 repeating said using the image analysis system, said using the motion-determining model and the prediction model, said determining the differences, and said adjusting weights plural times until a training objective is achieved. 
 
     
     
       7. The method of  claim 1 , wherein the motion-determining model is implemented, at least in part, by a convolutional neural network. 
     
     
       8. The method of  claim 1 , wherein the prediction model is implemented, at least in part, by a convolutional neural network. 
     
     
       9. The method of  claim 8 , wherein the convolutional neural network of the prediction model includes a first path neural network that uses a first kernel size and a second path neural network that uses a second kernel size for processing last-cached feature information, wherein the second kernel size is larger than the first kernel size, an output of the first path neural network being combined with an output of the second path neural network. 
     
     
       10. The method of  claim 8 , wherein the convolutional neural network of the prediction model operates by:
 mapping the first feature information obtained from the data store into intermediary information using a first convolutional neural network; 
 combining the intermediary information with the first frame-change information to produce combined information; and 
 mapping the combined information into the second feature information using another convolutional neural network. 
 
     
     
       11. A computing system for processing a stream of video frames, comprising:
 an image analysis system for receiving video frames that are interpreted as key frames, and for converting the key frames into instances of key-frame feature information; 
 a prediction neural network for receiving video frames that are interpreted as predicted frames, and for converting the predicted frames, along with instances of frame-change information, into instances of predicted feature information; 
 a data store for storing the instances of key-frame feature information produced by the image analysis system and the predicted feature information produced by the prediction neural network; and 
 a motion-determining neural network for mapping pairs of consecutive video frames in the stream of video frames into the instances of the frame-change information 
 the processing interpreting every nth frame in the stream of video frames as a key frame, and frames between neighboring key frames as predicted frames, n being specified by a configuration parameter, and 
 the processing switching between use of the image analysis system and use of the motion-determining neural network and the prediction neural network depending on whether a key frame is encountered or a predicted frame is encountered in the stream of video frames. 
 
     
     
       12. The computing system of  claim 11 , wherein one particular pair of consecutive video frames includes a particular key frame and an immediately-following particular predicted frame. 
     
     
       13. The computing system of  claim 11 , wherein one particular pair of consecutive video frames includes a first predicted frame and an immediately-following second predicted frame. 
     
     
       14. The computing system of  claim 11 , wherein the image analysis system is a first image analysis system, and wherein the computing system includes a second image analysis system for converting the instances of the key-frame feature information and the instances of the predicted feature information into instances of output information. 
     
     
       15. The computing system of  claim 11 , wherein the motion-determining neural network includes, at least in part, a convolutional neural network. 
     
     
       16. The computing system of  claim 11 , wherein the prediction neural network includes, at least in part, a convolutional neural network. 
     
     
       17. The computing system of  claim 16 , wherein the convolutional neural network of the prediction neural network includes a first path neural network that uses a first kernel size and a second path neural network that uses a second kernel size, wherein the second kernel size is larger than the first kernel size. 
     
     
       18. A computer-readable storage medium for storing computer-readable instructions, the computer-readable instructions, when executed by one or more hardware processors, performing operations that comprise:
 obtaining a first part of a data item having a sequence of parts, the first part being interpreted as a key part; 
 converting the first part into first feature information using a data item analysis process; 
 caching the first feature information in a data store; 
 obtaining a second part of the data item, the second part being interpreted as a predicted part; 
 mapping the first part and the second part into first part-change information using a motion-determining model, the first part-change information expressing a change in the data item from the first part to the second part; 
 converting the first part-change information and the first feature information into second feature information using a prediction model; and 
 caching the second feature information in the data store, 
 the operations interpreting every nth part in the sequence of parts as a key part, and parts between neighboring key parts as predicted parts, n being specified by a configuration parameter, and 
 the operations switching between use of the image analysis system and use of the motion-determining model and the prediction model depending on whether a key part is encountered or a predicted part is encountered in the sequence of parts. 
 
     
     
       19. The computer-readable storage medium of  claim 18 , wherein the data item is video content, and wherein the first part and the second part are respectively a first frame and a second frame of the video content. 
     
     
       20. The method of  claim 1 , wherein the image analysis system consumes more computing resources for each frame it processes compared to the prediction system.