Apparatus and method for prediction of video frame based on deep learning

Abstract

An apparatus and a method of predicting a video frame are provided. The apparatus includes a level encoder configured to extract and learn at least one feature from a video frame, a feature learning unit configured to learn based on the at least one feature or transmit predicted feature data corresponding to the at least one feature, and a level decoder configured to obtain and learn a predicted video frame based on the predicted feature data.

What is claimed is: 
     
       1. An apparatus for predicting a video frame, the apparatus comprising:
 a level encoder configured to extract and learn at least one feature from a video frame; 
 a feature learning unit configured to learn based on the at least one feature or transmit predicted feature data corresponding to the at least one feature; and 
 a level decoder configured to obtain and learn a predicted video frame based on the predicted feature data, 
 wherein the level encoder receives first to (T−1)th video frames, respectively, and extracts at least one feature from each of the first to (T−1)th video frames, where “T” includes a natural number equal to or greater than 2, 
 wherein the feature learning unit is trained based on at least one feature extracted from each of the first to (T−1)th video frames, 
 wherein the level encoder receives the T-th video frame, 
 wherein the level decoder obtains a (T+1)th predicted video frame corresponding to the T th video frame, 
 wherein the level encoder receives the (T+1)th predicted video frame, and 
 wherein the level decoder obtains a (T+2)th predicted video frame corresponding to the (T+1)th predicted video frame. 
 
     
     
       2. The apparatus of  claim 1 , wherein the level encoder includes a first-level encoder to an N-th level encoder, and
 wherein each of the first-level encoder to the N-th level encoder extracts features of different levels from the video frame where “N” is a natural number equal to or greater than 2. 
 
     
     
       3. The apparatus of  claim 2 , wherein the feature learning unit includes a first feature learning unit to an N-th feature learning unit corresponding to each of the first-level encoder to the N-th level encoder, and
 wherein each of the first feature learning unit to the N-th feature learning unit receives each feature of the different levels, obtains and transmit predicted feature data corresponding to each feature of the different levels. 
 
     
     
       4. The apparatus of  claim 3 , wherein the level decoder includes a first-level decoder to an N-th level decoder corresponding to each of the first-level encoder to the N-th level encoder or corresponding to each of the first feature learning unit to the N-th feature learning unit, and
 wherein the first-level decoder to the N-th level decoder receive each of the predicted feature data, respectively, and generate a predicted video frame by using the predicted feature data. 
 
     
     
       5. The apparatus of  claim 1 , wherein at least one of the level encoder and the level decoder is based on at least one of a convolutional neural network (CNN), a deep neural network (DNN), a recurrent neural network (RNN), and a deep belief neural network (DBN). 
     
     
       6. The apparatus of  claim 1 , wherein the feature learning unit is based on a long short term memory (LSTM). 
     
     
       7. A method of predicting a video frame, the method comprising:
 extracting at least one feature from a video frame; 
 obtaining predicted feature data trained based on the at least one feature or corresponding to the at least one feature; and 
 obtaining a predicted video frame based on the predicted feature data, 
 wherein the extracting of the at least one feature from the video frame includes: 
 receiving first to (T−1)th video frames, respectively, and extracting at least one feature from each of the first to (T−1)th video frames, where “T” includes a natural number of 2 or more, 
 wherein the training based on the at least one feature includes: 
 training based on at least one feature extracted from each of the first to (T−1)th video frames, 
 wherein the extracting of the at least one feature from the video frame includes: 
 receiving a T-th video frame and extracting at least one feature from the T-th video frame, 
 wherein the obtaining of the predicted video frame based on the predicted feature data includes: 
 obtaining a (T+1) th predicted video frame corresponding to the T th video frame based on the predicted feature data. 
 
     
     
       8. The method of  claim 7 , wherein the extracting of the at least one feature from the video frame includes:
 extracting, by the first-level encoder to the N-th level encoder, features of different levels from each other from the video frame, where “N” includes a natural number equal to or greater than 2. 
 
     
     
       9. The method of  claim 8 , wherein the obtaining of the predicted feature data corresponding to the at least one feature includes:
 receiving, by a first feature learning unit to an N-th feature learning unit corresponding to the first-level encoder to the N-th level encoder, the features of the different levels, respectively; and 
 obtaining, by the first feature learning unit to the N-th feature learning unit, the predicted feature data corresponding to each of the features of the different levels, respectively, and transmitting the predicted feature data to a next frame processing process. 
 
     
     
       10. The method of  claim 9 , wherein the obtaining of the predicted video frame based on the predicted feature data includes:
 receiving each of the predicted feature data by a first-level decoder to an N-th level decoder, respectively; and 
 generating, by the first-level decoder to the N-th level decoder, the predicted video frame by using the predicted feature data, and 
 wherein the first-level decoder to the N-th level decoder correspond to each of the first-level encoder to the N-th level encoder, or corresponding to each of the first feature learning unit to the N-th feature learning unit. 
 
     
     
       11. The method of  claim 7 , further comprising:
 receiving the (T+1)th predicted video frame and obtaining a (T+2)th predicted video frame corresponding to the (T+1)th predicted video frame. 
 
     
     
       12. A method of predicting a video frame, the method comprising:
 extracting at least one feature from a video frame; 
 obtaining predicted feature data trained based on the at least one feature or corresponding to the at least one feature; and 
 obtaining a predicted video frame based on the predicted feature data, 
 wherein the extracting of the at least one feature from the video frame includes: 
 receiving first to (T−1)th video frames, respectively, and extracting at least one feature from each of the first to (T−1)th video frames, where “T” includes a natural number of 2 or more, and 
 wherein the training based on the at least one feature includes: 
 training based on at least one feature extracted from each of the first to (T−1)th video frames, 
 wherein the extracting of the at least one feature from the video frame includes: 
 extracting, by the first-level encoder to the N-th level encoder, features of different levels from each other from the video frame, where “N” includes a natural number equal to or greater than 2, 
 wherein the obtaining of the predicted feature data corresponding to the at least one feature includes: 
 receiving, by a first feature learning unit to an N-th feature learning unit corresponding to the first-level encoder to the N-th level encoder, the features of the different levels, respectively; and 
 obtaining, by the first feature learning unit to the N-th feature learning unit, the predicted feature data corresponding to each of the features of the different levels, respectively, and transmitting the predicted feature data to a next frame processing process, and 
 wherein the obtaining of the predicted video frame based on the predicted feature data includes: 
 receiving each of the predicted feature data by a first-level decoder to an N-th level decoder, respectively; and 
 generating, by the first-level decoder to the N-th level decoder, the predicted video frame by using the predicted feature data, and 
 wherein the first-level decoder to the N-th level decoder correspond to each of the first-level encoder to the N-th level encoder, or corresponding to each of the first feature learning unit to the N-th feature learning unit.