Moment localization in media stream

Abstract

Various implementations of the subject matter relate to moment localization in media stream. In some implementations, a two-dimensional temporal feature map representing a plurality of moments within a media stream is extracted from the media stream, wherein the two-dimensional temporal feature map comprises a first dimension representing a start of a respective one of the plurality of moments and a second dimension representing an end of a respective one of the plurality of moments. A correlation between the plurality of moments and an action in the media stream is determined based on the two-dimensional temporal feature map.

The invention claimed is: 
     
       1. A computer-implemented method, comprising:
 extracting, from a media stream, a two-dimensional temporal feature map representing a plurality of moments within the media stream, wherein the two-dimensional temporal feature map comprises a first dimension representing a start of a respective one of the plurality of moments and a second dimension representing an end of a respective one of the plurality of moments; 
 encoding a sentence feature extracted from an input; 
 fusing the encoded sentence feature with the two-dimensional temporal feature map into a unified subspace as a fused two-dimensional temporal map; 
 applying a convolutional layer to the two-dimensional temporal feature map to obtain a further feature map having a same dimension as the two-dimensional temporal feature map, the convolutional layer comprises a dilated convolution and strides of the dilated convolution are configured to increase as lengths of the respective moments increase; 
 generating a temporal adjacent network using the fused two-dimensional temporal map and the further feature map; 
 determining, using the temporal adjacent network, a correlation between the plurality of moments and an action in the media stream; and 
 identifying a matching a set of candidate moments for the input using the temporal adjacent network. 
 
     
     
       2. The method of  claim 1 , wherein extracting the two-dimensional temporal feature map comprises:
 segmenting the media stream into a plurality of clips; 
 extracting features of respective ones of the plurality of clips to obtain a feature map of the media stream; and 
 extracting, from features of one or more clips corresponding to a moment of the plurality of moments in the feature map of the media stream, features of this moment as a part of the two-dimensional temporal feature map. 
 
     
     
       3. The method of  claim 1 , wherein determining the correlation comprises:
 sampling the plurality of moments at respective sample rates to determine a plurality of candidate moments, wherein the sample rates are adaptively adjusted based on lengths of respective ones of the plurality of moments; and 
 determining a correlation between the plurality of candidate moments and the action in the media stream. 
 
     
     
       4. The method of  claim 3 , wherein the sample rates are configured to decrease as the lengths of the respective moments increase. 
     
     
       5. The method of  claim 1 , wherein determining the correlation comprises:
 applying a convolutional layer to the two-dimensional temporal feature map to obtain a further feature map having a same dimension as the two-dimensional temporal feature map; and 
 determining, based on the further feature map, scores of correlation between the plurality of moments and the action in the media stream. 
 
     
     
       6. The method of  claim 1 , wherein determining the correlation comprises:
 in response to receiving a query for a particular action in the media stream, extracting a feature vector of the query; and 
 determining the correlation based on the feature vector of the query and the two-dimensional temporal feature map. 
 
     
     
       7. The method of  claim 6 , wherein determining the correlation comprises:
 fusing the feature vector of the query and the two-dimensional temporal feature map to generate a further two-dimensional temporal feature map having a same dimension as the two-dimensional temporal feature map; and 
 determining, based on the further two-dimensional temporal feature map, the correlation between the plurality of moments and the particular action. 
 
     
     
       8. The method of  claim 7 , wherein fusing the feature vector of the query and the two-dimensional temporal feature map comprises:
 generating the further two-dimensional temporal feature map by applying a Hadamard product to the feature vector of the query and the two-dimensional temporal feature map. 
 
     
     
       9. The method of  claim 6 , wherein the query comprises a natural language query. 
     
     
       10. The method of  claim 1 , wherein the media stream comprises an untrimmed media stream. 
     
     
       11. A device comprising:
 a processing unit; and 
 a memory coupled to the processing unit and having instructions stored thereon, the instructions, when executed by the processing unit, causing the device to perform acts comprising:
 extracting, from a media stream, a two-dimensional temporal feature map representing a plurality of moments within the media stream, wherein the two-dimensional temporal feature map comprises a first dimension representing a start of a respective one of the plurality of moments and a second dimension representing an end of a respective one of the plurality of moments; 
 encoding a sentence feature extracted from an input; 
 fusing the encoded sentence feature with the two-dimensional temporal feature map into a unified subspace as a fused two-dimensional temporal map; 
 applying a convolutional layer to the two-dimensional temporal feature map to obtain a further feature map having a same dimension as the two-dimensional temporal feature map, the convolutional layer comprises a dilated convolution and strides of the dilated convolution are configured to increase as lengths of the respective moments increase; 
 generating a temporal adjacent network using the fused two-dimensional temporal map and the further feature map; 
 determining, using the temporal adjacent network, a correlation between the plurality of moments and an action in the media stream; and 
 identifying a matching a set of candidate moments for the input using the temporal adjacent network. 
 
 
     
     
       12. The device of  claim 11 , wherein extracting the two-dimensional temporal feature map comprises:
 segmenting the media stream into a plurality of clips; 
 extracting features of respective ones of the plurality of clips to obtain a feature map of the media stream; and 
 extracting, from features of one or more clips corresponding to a moment of the plurality of moments in the feature map of the media stream, features of this moment as a part of the two-dimensional temporal feature map. 
 
     
     
       13. The device of  claim 11 , wherein determining the correlation comprises:
 sampling the plurality of moments at respective sample rates to determine a plurality of candidate moments, wherein the sample rates are adaptively adjusted based on lengths of respective ones of the plurality of moments; and 
 determining a correlation between the plurality of candidate moments and the action in the media stream. 
 
     
     
       14. At least one non-transitory machine-readable medium comprising computer-executable instructions which, when executed by a device, cause the device to perform operations to:
 extract, from a media stream, a two-dimensional temporal feature map representing a plurality of moments within the media stream, wherein the two-dimensional temporal feature map comprises a first dimension representing a start of a respective one of the plurality of moments and a second dimension representing an end of a respective one of the plurality of moments; 
 encode a sentence feature extracted from an input; 
 fuse the encoded sentence feature with the two-dimensional temporal feature map into a unified subspace as a fused two-dimensional temporal map; 
 apply a convolutional layer to the two-dimensional temporal feature map to obtain a further feature map having a same dimension as the two-dimensional temporal feature map, the convolutional layer comprises a dilated convolution and strides of the dilated convolution are configured to increase as lengths of the respective moments increase; 
 generate a temporal adjacent network using the fused two-dimensional temporal map and the further feature map; 
 determine, using the temporal adjacent network, a correlation between the plurality of moments and an action in the media stream; and 
 identify a matching a set of candidate moments for the input using the temporal adjacent network. 
 
     
     
       15. The at least one non-transitory machine-readable medium of  claim 14 , the instructions to extract the two-dimensional temporal feature map comprising instructions to:
 segment the media stream into a plurality of clips; 
 extract features of respective ones of the plurality of clips to obtain a feature map of the media stream; and 
 extract, from features of one or more clips corresponding to a moment of the plurality of moments in the feature map of the media stream, features of this moment as a part of the two-dimensional temporal feature map. 
 
     
     
       16. The at least one non-transitory machine-readable medium of  claim 14 , the instructions to determine the correlation comprising instructions to:
 sample the plurality of moments at respective sample rates to determine a plurality of candidate moments, wherein the sample rates are adaptively adjusted based on lengths of respective ones of the plurality of moments; and 
 determine a correlation between the plurality of candidate moments and the action in the media stream. 
 
     
     
       17. The at least one non-transitory machine-readable medium of  claim 16 , wherein the sample rates are configured to decrease as the lengths of the respective moments increase. 
     
     
       18. The at least one non-transitory machine-readable medium of  claim 14 , the instructions to determine the correlation comprising instructions to:
 applying a convolutional layer to the two-dimensional temporal feature map to obtain a further feature map having a same dimension as the two-dimensional temporal feature map; and 
 determining, based on the further feature map, scores of correlation between the plurality of moments and the action in the media stream. 
 
     
     
       19. The device of  claim 11 , wherein determining the correlation comprises:
 in response to receiving a query for a particular action in the media stream, extracting a feature vector of the query; and 
 determining the correlation based on the feature vector of the query and the two-dimensional temporal feature map. 
 
     
     
       20. The at least one non-transitory machine-readable medium of  claim 14 , the instructions to determine the correlation further comprising instructions to:
 in response to receipt of a query for a particular action in the media stream, extract a feature vector of the query; and 
 determine the correlation based on the feature vector of the query and the two-dimensional temporal feature map.