Video playback recognition apparatus and method

Abstract

A method includes processing live image data captured by a camera associated with a first device. The live image data is displayed by the first device. A video displayed by a second device is viewable within the live image data displayed by the first device. The live image data is processed to identify a spatial position of the video displayed with respect to the camera associated with the first device within the live image data, and to identify a known frame of the video. The method also includes causing augmented imagery to be displayed by the first device in a display area of the first device within which the live image data is displayed concurrently with the video displayed by the second device. The augmented imagery is displayed in the display area of the first device based on the identified spatial position, and in response to the known frame.

What is claimed is: 
     
         1 . A method, comprising:
 processing live image data captured by a camera associated with a first device, wherein the live image data is displayed by the first device, a video displayed by a second device is viewable within the live image data displayed by the first device, the live image data is processed to identify a spatial position of the video displayed with respect to the camera associated with the first device within the live image data captured by the camera, and the live image data is processed to identify a known frame of the video displayed by the second device; and   causing augmented imagery to be displayed by the first device in a display area of the first device within which the live image data is displayed concurrently with the video displayed by the second device, wherein the augmented imagery is displayed in the display area of the first device based on the identified spatial position of the video displayed with respect to the camera associated with the first device within the live image data captured by the camera, and in response to the known frame.   
     
     
         2 . The method of  claim 1 , wherein the augmented imagery is displayed by the first device so as to appear between the video displayed by the second device and the first device. 
     
     
         3 . The method of  claim 1 , wherein the augmented imagery is displayed by the first device so as to appear behind the video displayed by the second device such that the video displayed by the second device is between the augmented imagery and the first device. 
     
     
         4 . The method of  claim 1 , wherein the augmented imagery is displayed by the first device so as to appear as being on the video displayed by the second device. 
     
     
         5 . The method of  claim 1 , wherein the processing of the live image data to identify the spatial position of the video displayed with respect to the camera associated with the first device within the live image data captured by the camera, comprises:
 identifying one or more points in a candidate frame of the video displayed by the second device;   comparing the one or more points in the candidate frame to a plurality of points in a database of known points corresponding to known video clips comprising a plurality of known frames; and   identifying the known frame within the video displayed by the second device based on a matching of the one or more points in the candidate frame of the video displayed by the second device and the plurality of points in the database of known points corresponding to the known video clips comprising the plurality of known frames.   
     
     
         6 . The method of  claim 5 , further comprising:
 detecting a boundary of the candidate frame in the live image data captured by the camera;   cropping the candidate frame from the live image data captured by the camera to generate a cropped frame image; and   processing the cropped frame image to identify the known frame.   
     
     
         7 . The method of  claim 6 , further comprising:
 de-warping the cropped frame image before processing the cropped frame image.   
     
     
         8 . The method of  claim 1 , wherein the augmented imagery is further caused to be displayed in response to a playback position in the video displayed by the second device, wherein the playback position is based on the known frame. 
     
     
         9 . The method of  claim 8 , wherein
 the playback position is an estimated playback position in the video displayed by the second device,   the estimated playback position is based on a continuously transformed state of the video displayed by the second device,   the continuously transformed state corresponds to a time in the playback of the video displayed by the second device, and   the continuously transformed state is based on a plurality of past states in a table of past states ranging from an oldest past state to a newest past state.   
     
     
         10 . The method of  claim 9 , wherein the table of past states is maintained outside a neural network that generates the continuously transformed state and the estimated playback position, and the method further comprises:
 causing one or more of the plurality of past states in the table of past states to be supplied to the neural network for determining a latest state based on the known frame;   causing the neural network to generate the estimated playback position; and   causing the latest state to be added to the table of past states as the newest past state.   
     
     
         11 . An apparatus, comprising:
 a processor; and   a memory having instructions stored thereon that, when executed by the processor, cause the apparatus to:   process live image data captured by a camera associated with a first device, wherein the live image data is displayed by the first device, a video displayed by a second device is viewable within the live image data displayed by the first device, the live image data is processed to identify a spatial position of the video displayed with respect to the camera associated with the first device within the live image data captured by the camera, and the live image data is processed to identify a known frame of the video displayed by the second device; and   cause augmented imagery to be displayed by the first device in a display area of the first device within which the live image data is displayed concurrently with the video displayed by the second device, wherein the augmented imagery is displayed in the display area of the first device based on the identified spatial position of the video displayed with respect to the camera associated with the first device within the live image data captured by the camera, and in response to the known frame.   
     
     
         12 . The apparatus of  claim 11 , wherein the augmented imagery is displayed by the first device so as to appear between the video displayed by the second device and the first device. 
     
     
         13 . The apparatus of  claim 11 , wherein the augmented imagery is displayed by the first device so as to appear behind the video displayed by the second device such that the video displayed by the second device is between the augmented imagery and the first device. 
     
     
         14 . The apparatus of  claim 11 , wherein the augmented imagery is displayed by the first device so as to appear as being on the video displayed by the second device. 
     
     
         15 . The apparatus of  claim 11 , wherein to process the live image data to identify the spatial position of the video displayed with respect to the camera associated with the first device within the live image data captured by the camera, the apparatus is caused to:
 identify one or more points in a candidate frame of the video displayed by the second device;   compare the one or more points in the candidate frame to a plurality of points in a database of known points corresponding to known video clips comprising a plurality of known frames; and   identify the known frame within the video displayed by the second device based on a matching of the one or more points in the candidate frame of the video displayed by the second device and the plurality of points in the database of known points corresponding to the known video clips comprising the plurality of known frames.   
     
     
         16 . The apparatus of  claim 15 , wherein the apparatus is further caused to:
 detect a boundary of the candidate frame in the live image data captured by the camera;   crop the candidate frame from the live image data captured by the camera to generate a cropped frame image; and   process the cropped frame image to identify the known frame.   
     
     
         17 . The apparatus of  claim 16 , wherein the apparatus is further caused to:
 de-warp the cropped frame image before processing the cropped frame image.   
     
     
         18 . The apparatus of  claim 11 , wherein the augmented imagery is further caused to be displayed in response to a playback position in the video displayed by the second device, wherein the playback position is based on the known frame. 
     
     
         19 . The apparatus of  claim 18 , wherein
 the playback position is an estimated playback position in the video displayed by the second device,   the estimated playback position is based on a continuously transformed state of the video displayed by the second device,   the continuously transformed state corresponds to a time in the playback of the video displayed by the second device, and   the continuously transformed state is based on a plurality of past states in a table of past states ranging from an oldest past state to a newest past state.   
     
     
         20 . A non-transitory computer readable medium having instructions stored thereon that, when executed by a processor, cause an apparatus to:
 process live image data captured by a camera associated with a first device, wherein the live image data is displayed by the first device, a video displayed by a second device is viewable within the live image data displayed by the first device, the live image data is processed to identify a spatial position of the video displayed with respect to the camera associated with the first device within the live image data captured by the camera, and the live image data is processed to identify a known frame of the video displayed by the second device; and   cause augmented imagery to be displayed by the first device in a display area of the first device within which the live image data is displayed concurrently with the video displayed by the second device, wherein the augmented imagery is displayed in the display area of the first device based on the identified spatial position of the video displayed with respect to the camera associated with the first device within the live image data captured by the camera, and in response to the known frame.