User Feedback in Semi-Automatic Question Answering Systems

Abstract

A system applies rules to a set of documents to generate codes, such as billing codes for use in medical billing. A human operator provides input specifying whether the generated codes are correct. Based on the input from the human operator, the system attempts to identify which clause(s) in the rules which were relied on to generate the particular code are correct and which such clause(s) are incorrect. The system then assigns praise to components of the system responsible for generating codes in the correct clauses, and assigns blame to components of the system responsible for generating codes in the incorrect clauses. Such blame and praise may then be used to determine whether particular code-generating components are insufficiently reliable. The system may disable, or take other remedial action in response to, insufficiently reliable code-generating components.

What is claimed is: 
     
         1 . A method performed by at least one computer processor executing computer program instructions tangibly stored on at least one non-transitory computer-readable medium,
 the method for use with a system including a data source and a first billing code,   the method comprising:   (A) receiving input from a user, wherein the input represents a verification status of the first billing code;   (B) applying first inverse logic to the input, the billing code, and a set of forward logic, to identify first and second concept extraction components; and   (C) applying reinforcement to the first and second concept extraction components, comprising:
 (E) (1) determining whether the verification status indicates that the first billing code is accurate; 
 (E) (2) if the verification status indicates that the first billing code is inaccurate, then applying negative reinforcement to the first and second concept extraction components, comprising apportioning the negative reinforcement between the first and second concept extraction components.