Knowledge-based information retrieval system evaluation

Abstract

Embodiments provide a computer implemented method of evaluating one or more IR systems, the method including: providing, by a processor, a pre-indexed knowledge-based document to a pre-trained sentence identification model; identifying, by the sentence identification model, a predetermined number of query-worthy sentences from the pre-indexed knowledge-based document, wherein the query-worthy sentences are ranked based on a prediction probability value of each query-worthy sentence; providing, by the sentence identification model, the query-worthy sentences to a pre-trained query generation model; generating, by the query generation model, a query for each query-worthy sentence; and evaluating, by the processor, the one or more IR systems using the generated queries, wherein one or more searches are performed via the one or more IR systems, and the one or more searches are performed in a set of knowledge-based documents including the pre-indexed knowledge-based document.

What is claimed is: 
     
       1. A computer implemented method in a data processing system comprising a processor and a memory comprising instructions, which are executed by the processor to cause the processor to implement the method of evaluating one or more information retrieval (“IR”) systems, the method comprising:
 receiving, by the processor, ground truth, wherein the ground truth is collected through a first round of crowdsourcing tasks, wherein the ground truth comprises a plurality of second query-worthy sentences identified by one or more crowd workers and a plurality of second queries written in a natural language by the one or more crowd workers; 
 validating, by the processor, the ground truth through a second round of crowdsourcing tasks, wherein the validation comprises:
 comparing a plurality of validation sentences identified by the one or more crowd workers with the plurality of second query-worthy sentences, and 
 including one of the plurality of second query-worthy sentences in the ground truth when one of the plurality of validation sentences is consistent with a corresponding one of the plurality of second query-worthy sentences; 
 
 training, by the processor, a sentence identification model using the validated ground truth to identify the query-worthy sentences, wherein the ground truth includes a plurality of ground truth tuples, each ground truth tuple including a second knowledge-based document, the plurality of second query-worthy sentences identified from the second knowledge-based document, and the plurality of second queries, wherein each second query corresponds to a second query-worthy sentence; 
 providing, by the processor, a pre-indexed knowledge-based document to the trained sentence identification model, wherein the trained sentence identification model is a first machine learning model; 
 identifying, by the sentence identification model, a predetermined number of query-worthy sentences from the pre-indexed knowledge-based document, wherein the query-worthy sentences are ranked based on a prediction probability value of each query-worthy sentence, wherein each of the query-worthy sentences are a sentence that contains an answer in response to a natural language query; 
 training, by the processor, a query generation model using the validated ground truth to generate the query for each query-worthy sentence; 
 providing, by the sentence identification model, the query-worthy sentences to the trained query generation model, wherein the trained query generation model is a second machine learning model; 
 generating, by the query generation model, a query for each query-worthy sentence; and 
 evaluating, by the processor, the one or more IR systems using the generated queries, wherein one or more searches are performed via the one or more IR systems, and the one or more searches are performed in a set of knowledge-based documents including the pre-indexed knowledge-based document. 
 
     
     
       2. The method as recited in  claim 1 , wherein the sentence identification model is trained using one or more features including a location of each query-worthy sentence within the second knowledge-based document, a plurality of entities in each query-worthy sentence, a type of a knowledge concept, and a plurality of properties of the knowledge concept. 
     
     
       3. The method as recited in  claim 2 , wherein the plurality of properties of the knowledge concept are obtained via Freebase. 
     
     
       4. The method as recited in  claim 1 , wherein the query is generated using a plurality of pre-defined templates. 
     
     
       5. The method as recited in  claim 1 , wherein the query generation model is a sequence to sequence model. 
     
     
       6. The method as recited in  claim 1 , wherein the evaluation of the one or more IR systems comprises evaluating the one or more IR systems using one or more matrices including precision@1, recall@5, and recall@10. 
     
     
       7. A computer program product for evaluating one or more information retrieval (“IR”) systems, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a processor to cause the processor to:
 receive, by the processor, ground truth, wherein the ground truth is collected through a first round of crowdsourcing tasks, wherein the ground truth comprises a plurality of second query-worthy sentences identified by one or more crowd workers and a plurality of second queries written in a natural language by the one or more crowd workers; 
 validate, by the processor, the ground truth through a second round of crowdsourcing tasks, wherein the validation comprises:
 comparing a plurality of validation sentences identified by the one or more crowd workers with the plurality of second query-worthy sentences, and 
 including one of the plurality of second query-worthy sentences in the ground truth when one of the plurality of validation sentences is consistent with a corresponding one of the plurality of second query-worthy sentences; 
 
 train, by the processor, a sentence identification model using the validated ground truth to identify the query-worthy sentences, wherein the ground truth includes a plurality of ground truth tuples, each ground truth tuple including a second knowledge-based document, the plurality of second query-worthy sentences identified from the second knowledge-based document, and the plurality of second queries, wherein each second query corresponds to a second query-worthy sentence; 
 provide a pre-indexed knowledge-based document to the trained sentence identification model, wherein the trained sentence identification model is a first machine learning model; 
 identify, by the sentence identification model, a predetermined number of query-worthy sentences from the pre-indexed knowledge-based document, wherein the query-worthy sentences are ranked based on a prediction probability value of each query-worthy sentence, wherein each of the query-worthy sentences are a sentence that contains an answer in response to a natural language query; 
 train, by the processor, a query generation model using the validated ground truth to generate the query for each query-worthy sentence; 
 provide, by the sentence identification model, the query-worthy sentences to the trained query generation model, wherein the trained query generation model is a second machine learning model; 
 generate, by the query generation model, a query for each query-worthy sentence; and 
 evaluate the one or more IR systems using the generated queries, wherein one or more searches are performed via the one or more IR systems, and the one or more searches are performed in a set of knowledge-based documents including the pre-indexed knowledge-based document. 
 
     
     
       8. The computer program product as recited in  claim 7 , wherein the sentence identification model is trained using one or more features including a location of each query-worthy sentence within the second knowledge-based document, a plurality of named entities in each query-worthy sentence, a type of a knowledge concept, and a plurality of properties of the knowledge concept. 
     
     
       9. The computer program product as recited in  claim 8 , wherein the set of knowledge-based documents are all WIKI documents. 
     
     
       10. The computer program product as recited in  claim 8 , wherein features related to the plurality of named entities include the number of entities within each query-worthy sentence, entity types, local weighted entity importance, and global weighted entity importance. 
     
     
       11. A system for evaluating one or more information retrieval (“IR”) systems, comprising:
 a processor configured to:
 receive, by the processor, ground truth, wherein the ground truth is collected through a first round of crowdsourcing tasks, wherein the ground truth comprises a plurality of second query-worthy sentences identified by one or more crowd workers and a plurality of second queries written in a natural language by the one or more crowd workers; 
 validate, by the processor, the ground truth through a second round of crowdsourcing tasks, wherein the validation comprises:
 comparing a plurality of validation sentences identified by the one or more crowd workers with the plurality of second query-worthy sentences, and 
 including one of the plurality of second query-worthy sentences in the ground truth when one of the plurality of validation sentences is consistent with a corresponding one of the plurality of second query-worthy sentences; 
 
 train, by the processor, a sentence identification model using the validated ground truth to identify the query-worthy sentences, wherein the ground truth includes a plurality of ground truth tuples, each ground truth tuple including a second knowledge-based document, the plurality of second query-worthy sentences identified from the second knowledge-based document, and the plurality of second queries, wherein each second query corresponds to a second query-worthy sentence; 
 provide a pre-indexed knowledge-based document to the trained sentence identification model, wherein the trained sentence identification model is a first machine learning model; 
 identify, by the sentence identification model, a predetermined number of query-worthy sentences from the pre-indexed knowledge-based document, wherein the query-worthy sentences are ranked based on a prediction probability value of each query-worthy sentence, wherein each of the query-worthy sentences are a sentence that contains an answer in response to a natural language query; 
 train, by the processor, a query generation model using the validated ground truth to generate the query for each query-worthy sentence; 
 provide, by the sentence identification model, the query-worthy sentences to the trained query generation model, wherein the trained query generation model is a second machine learning model; 
 generate, by the query generation model, a query for each query-worthy sentence; and 
 evaluate the one or more IR systems using the generated queries, wherein one or more searches are performed via the one or more IR systems, and the one or more searches are performed in a set of knowledge-based documents including the pre-indexed knowledge-based document.