Inferring emerging and evolving topics in streaming text

Abstract

A method, system and computer program product for inferring topic evolution and emergence in a set of documents. In one embodiment, the method comprises forming a group of matrices using text in the documents, and analyzing these matrices to identify evolving topics and emerging topics. The matrices includes a matrix X identifying a multitude of words in each of the documents, a matrix W identifying a multitude of topics in each of the documents, and a matrix H identifying a multitude of words for each of the multitude of topics. These matrices are analyzed to identify the evolving and emerging topics. In an embodiment, two forms of temporal regularizers are used to help identify the evolving and emerging topics. In another embodiment, a two stage approach involving detection and clustering is used to help identify the evolving and emerging topics.

What is claimed is: 
     
         1 . A method of inferring topic evolution and emergence in a multitude of documents, comprising:
 forming a group of matrices using text in the documents, said group of matrices including a first matrix X identifying a multitude of words in each of the documents, a second matrix W identifying a multitude of topics in each of the documents, and a third matrix H identifying a multitude of words for each of said multitude of topics; and   analyzing said group of matrices to identify a first group of said multitude of topics as evolving topics and a second group of said multitude of topics as emerging topics.   
     
     
         2 . The method according to  claim 1 , wherein said multitude of documents comprise a sequence of streaming documents, each of the documents being associated with a timepoint t, in a defined period of time T, and wherein:
 the forming the group of matrices using data in the documents includes:   forming a first sequence of matrices X(t), each of the matrices X(t) identifying a multitude of words in each of a set of the documents associated with the timepoints within a defined sliding window in the time period T;   forming a second sequence of matrices W(t), each of the matrices W(t) identifying a multitude of topics in said set of documents associated with the timepoints within said defined window; and   forming a third sequence of matrices H(t), each of the matrices H(t) identifying a multitude of words for each of the topics identified in an associated one of the matrices W(t); and   the analyzing the groups of matrices includes using a defined equation including the matrices X(t), W(t) and H(t), to identify the evolving and the emerging topics.   
     
     
         3 . The method according to  claim 2 , wherein:
 said defined equation includes a first regularizer μ to enforce smooth evolution of the evolving topics via constraints on an amount of drift allowed by the evolving topics, and a second regularizer Ω to apply a topic bandwidth for early detection of the emerging topics to extract smooth trends of candidate emerging topics; and   said defined equation is an objective function:   
       
         
           
             
               
                 ( 
                 
                   
                     W 
                     * 
                   
                   , 
                   
                     H 
                      
                     
                       ( 
                       t 
                       ) 
                     
                   
                 
                 ) 
               
               ≡ 
               
                 
                   
                     
                       arg 
                        
                       
                           
                       
                        
                       min 
                     
                     
                       W 
                       , 
                       H 
                     
                   
                    
                   
                     
                        
                       
                         
                           X 
                            
                           
                             ( 
                             
                               
                                 t 
                                 - 
                                 w 
                               
                               , 
                               t 
                             
                             ) 
                           
                         
                         - 
                         WH 
                       
                        
                     
                     fro 
                     2 
                   
                 
                 + 
                 
                   μΩ 
                    
                   
                     ( 
                     W 
                     ) 
                   
                 
               
             
           
         
       
     
     
         4 . The method according to  claim 2 , wherein:
 said defined equation includes solving an l 1  dictionary learning problem to identify evolving topics, and using a reconstruction error to identify novel documents;   the analyzing the group of matrices further includes clustering said novel documents to identify emerging topics; and   said defined equation is an objective function:
     W*, H *=argmin W,H   ∥X−WH∥   1   +λ∥W∥   1  such that  W, H≧ 0. 
   
     
     
         5 . The method according to  claim 1 , wherein the forming the group of matrices using text in the documents includes using the first matrix to form the second and third matrices. 
     
     
         6 . The method according to  claim 1 , wherein the analyzing the group of matrices includes using a defined equation, including the first, second and third matrices, to identify the evolving topics and the emerging topics. 
     
     
         7 . The method according to  claim 6 , wherein said defined equation includes a regularizer to enforce smooth evolution of the evolving topics via constraints on an amount of drift allowed by the evolving topics. 
     
     
         8 . The method according to  claim 6 , wherein:
 said defined equation includes a regularizer to apply a topic bandwidth for early detection of the emerging topics: and   the regularizer extracts smooth trends of candidate emerging topics.   
     
     
         9 . The method of  claim 6  wherein the analyzing the group of matrices further includes:
 using said defined equation to identify novel documents based on reconstruction error; and 
 clustering the novel documents to identify emerging topics and used for updating the evolving topics. 
 
     
     
         10 . The method according to  claim 9 , wherein:
 the using the defined equation includes applying a threshold on the reconstruction errors obtained from a solution of said defined equation to identify the novel documents: and   the clustering the novel documents includes using a given clustering algorithm to cluster the novel documents.