Apparatus and Method of Implementing Batch-Mode Active Learning for Technology-Assisted Review of Documents

Abstract

The present disclosure relates to the electronic document review field and, more particularly, to various apparatuses and methods of implementing batch-mode active learning for technology-assisted review (TAR) of documents (e.g., legal documents).

1 - 14 . (canceled) 
     
     
         15 . A method to implement a diversity sampler process to select new batches of unlabeled instances, comprising:
 by one or more computing devices:
 inserting an identified document that has a nearest absolute distance from a current version of a classification model M out of an unlabeled set of available documents D into a new batch of unlabeled instances B c ; 
 removing documents that have a cosine angle≥t with respect to the inserted document from an unlabeled set of available documents D; and 
   repeating the insert operation and the remove operation until a new batch of unlabeled instances B c  are selected.   
     
     
         16 . The method of  claim 15 , wherein the classification model M is a hyperplane. 
     
     
         17 . The method of  claim 15 , wherein the unlabeled set of available documents D are sorted in increasing order. 
     
     
         18 . The method of  claim 15 , wherein the unlabeled set of available documents D has a batch size k. 
     
     
         19 . The method of  claim 18 , wherein the one or more computing devices performs the insert operation and the remove operation until k documents are inserted into the new batch of unlabeled instances B c . 
     
     
         20 . The method of  claim 15 , wherein the diversity sampler process is implemented using a support vector machine (SVM). 
     
     
         21 . The method of  claim 15 , wherein the diversity sampler process is implemented in a technology-assisted document review. 
     
     
         22 . The method of  claim 15 , wherein the current version of the classification model M is created by:
 obtaining an unlabeled set of documents D;   obtaining a batch size k;   constructing a first batch of k documents D;   obtaining labels for the first batch of k documents D, wherein the labeled first batch of k documents D are referred to as training data documents; and   constructing the current version of the classification model M using the training documents.   
     
     
         23 . The method of  claim 22 , further comprising:
 obtaining labels for the new batch of unlabeled instances B c ; and   adding the labeled new batch of instances B c  to a current version of the training data documents referred to as extended training data documents D c .   
     
     
         24 . The method of  claim 23 , further comprising constructing an updated classification model M using the extended training data documents D c . 
     
     
         25 . The method of  claim 15 , wherein identified the document that has a nearest absolute distance from a current version of a classification model M out of an unlabeled set of available documents D comprises:
 obtaining the current version of a classification model M, the unlabeled set of available documents D, and the cosine similarity threshold t;   sorting the unlabeled set of available documents D based on each of the documents absolute distance from the current version of the classification model M to obtain sorted indices I for each document of the unlabeled set of available documents D; and   identifying a document having a nearest sorted index I[1] from the current version of the classification model M.   
     
     
         26 . The method of  claim 15 , further comprising obtaining sorted indices I of the sorted unlabeled set of available documents D that have a cosine angle≥t with respect to the inserted document I[1]. 
     
     
         27 . The method of  claim 26 , further comprising removing the documents with the obtained sorted indices I from the sorted unlabeled set of available documents D. 
     
     
         28 . A system configured to implement a diversity sampler process to select new batches of unlabeled instances, the apparatus comprising:
 a processor; and   a memory that stores processor-executable instructions, wherein the processor interfaces with the memory to execute the processor-executable instructions, whereby the system is operable to:
 insert an identified document that has a nearest absolute distance from a current version of a classification model M out of an unlabeled set of available documents D into a new batch of unlabeled instances B c ; 
 remove documents that have a cosine angle≥t with respect to the inserted document from an unlabeled set of available documents D; and 
 repeat the insert operation and the remove operation until a new batch of unlabeled instances B c  are selected. 
   
     
     
         29 . The system of  claim 28 , wherein the diversity sampler process is implemented using a support vector machine (SVM). 
     
     
         30 . The system of  claim 28 , wherein the current version of the classification model M is created by:
 obtaining an unlabeled set of documents D;   obtaining a batch size k;   constructing a first batch of k documents D;   obtaining labels for the first batch of k documents D, wherein the labeled first batch of k documents D are referred to as training data documents; and   constructing the current version of the classification model M using the training documents.   
     
     
         31 . The system of  claim 30 , wherein the system is further operable to:
 obtain labels for the new batch of unlabeled instances B c ; and   add the labeled new batch of instances B c  to a current version of the training data documents referred to as extended training data documents D c .   
     
     
         32 . The system of  claim 31 , wherein the system is further operable to construct an updated classification model M using the extended training data documents D c . 
     
     
         33 . A method to implement a biased probabilistic sampler process to select new batches of unlabeled instances, comprising:
 by one or more computing devices:
 inserting an identified document that has a nearest absolute distance from a current version of a classification model M out of an unlabeled set of available documents D into a new batch of unlabeled instances B c ; 
 removing documents that have a cosine angle≥t with respect to the inserted document from an unlabeled set of available documents D; and 
 repeating the insert operation and the remove operation until a new batch of unlabeled instances B c  are selected. 
   
     
     
         34 . The method of  claim 33 , wherein the unlabeled set of available documents D has a batch size k. 
     
     
         35 . The method of  claim 34 , wherein the one or more computing devices performs the insert operation and the remove operation until k documents are inserted into the new batch of unlabeled instances B c . 
     
     
         36 . The method of  claim 33 , wherein the current version of the classification model M is created by:
 obtaining an unlabeled set of documents D;   obtaining a batch size k;   constructing a first batch of k documents D;   obtaining labels for the first batch of k documents D, wherein the labeled first batch of k documents D are referred to as training data documents; and   constructing the current version of the classification model M using the training data documents.   
     
     
         37 . The method of  claim 36 , further comprising:
 obtaining labels for the new batch of unlabeled instances B c ; and   adding the labeled new batch of instances B c  to a current version of the training data documents referred to as extended training data documents D c .