Retrieval augmented generation for videos

Abstract

Methods and systems for video analysis include pre-processing clips of an input video using a first vision model to generate respective first textual descriptions for the clips. A subset of the clips that are relevant to a query is selected based on the first textual descriptions. Additional textual descriptions are generated for the selected subset using a second vision model. The query is answered using the additional textual descriptions.

What is claimed is: 
     
         1 . A computer-implemented method for video analysis, comprising:
 pre-processing a plurality of clips of an input video using a first vision model to generate respective first textual descriptions for the plurality of clips;   selecting a subset of the plurality of clips that are relevant to a query based on the first textual descriptions;   generating additional textual descriptions for the selected subset using a second vision model; and   answering the query using the additional textual descriptions.   
     
     
         2 . The method of  claim 1 , wherein the second vision model has more parameters than the first model. 
     
     
         3 . The method of  claim 1 , wherein generating the additional textual descriptions is repeated until a language model is able to answer the query based on the additional textual descriptions. 
     
     
         4 . The method of  claim 3 , wherein each repetition of generating the additional textual descriptions instructs the second vision model to generate additional tokens that continue a previous output. 
     
     
         5 . The method of  claim 4 , wherein the instruction to generate additional tokens includes increasing a maximum output token limit. 
     
     
         6 . The method of  claim 3 , further comprising prompting the language model to determine whether the language model can answer the query using the first textual descriptions and the additional textual descriptions. 
     
     
         7 . The method of  claim 1 , wherein selecting the subset of the plurality of clips includes embedding the query and the first textual descriptions in a latent space and selecting the subset of the plurality of clips according to a similarity between the embedded query and the embedded first textual descriptions. 
     
     
         8 . The method of  claim 1 , further comprising grouping the plurality of clips according to image complexity. 
     
     
         9 . The method of  claim 1 , wherein pre-processing the plurality of clips includes identifying a domain of the input video and selecting the first vision model according to the domain. 
     
     
         10 . The method of  claim 9 , further comprising updating the domain based on the query. 
     
     
         11 . A system for video analysis, comprising:
 a hardware processor; and   a memory that stores a computer program which, when executed by the hardware processor, causes the hardware processor to:
 pre-process a plurality of clips of an input video using a first vision model to generate respective first textual descriptions for the plurality of clips; 
 select a subset of the plurality of clips that are relevant to a query based on the first textual descriptions; 
 generate additional textual descriptions for the selected subset using a second vision model; and 
 answer the query using the additional textual descriptions. 
   
     
     
         12 . The system of  claim 11 , wherein the second vision model has more parameters than the first model. 
     
     
         13 . The system of  claim 11 , wherein the generation of the additional textual descriptions is repeated until a language model is able to answer the query based on the additional textual descriptions. 
     
     
         14 . The system of  claim 13 , wherein each repetition of the generation of the additional textual descriptions instructs the second vision model to generate additional tokens that continue a previous output. 
     
     
         15 . The system of  claim 14 , wherein the instruction to generate additional tokens includes an increase to a maximum output token limit. 
     
     
         16 . The system of  claim 13 , wherein the computer program further causes the hardware processor to prompt the language model to determine whether the language model can answer the query using the first textual descriptions and the additional textual descriptions. 
     
     
         17 . The system of  claim 11 , wherein the selection of the subset of the plurality of clips includes an embedding of the query and the first textual descriptions in a latent space and selection of the subset of the plurality of clips according to a similarity between the embedded query and the embedded first textual descriptions. 
     
     
         18 . The system of  claim 11 , wherein the computer program further causes the hardware processor to group the plurality of clips according to image complexity. 
     
     
         19 . The system of  claim 11 , wherein the pre-processing of the plurality of clips includes identification of a domain of the input video and selecting the first vision model according to the domain. 
     
     
         20 . The system of  claim 19 , wherein the computer program further causes the hardware processor to update the domain based on the query.