Constructed response scoring mechanism

Abstract

A system, method, and related techniques are disclosed for scoring user responses to constructed response test items. The system includes a scoring engine for receiving a user response to a test question and evaluating the response against a scoring rubric. The scoring rubric may include a binding stage, an assertion stage, and a scoring stage. Furthermore, the system includes a database for referencing elements used by the scoring engine which may comprise objects, object sets, attributes of objects, and transformations of any elements.

1 . A method of refining a scoring rubric, comprising:
 receiving a plurality of user responses to a test item;   identifying a set of responses from the plurality of user responses;   providing a score for each of the identified responses;   generating one or more rules for differentiating a correct score from an incorrect score;   modifying a scoring rubric based on the one or more rules; and   applying the modified scoring rubric to multiple user responses.   
     
     
         2 . The method of  claim 1 , wherein the set of responses are identified based on a random sample. 
     
     
         3 . The method of  claim 1 , further comprising
 identifying one or more scores of user responses that change using the modified scoring rubric; and   reviewing a sample of the user responses and changed scores.   
     
     
         4 . The method of  claim 1 , wherein the sample is a census. 
     
     
         5 . The method of  claim 1 , further comprising making a decision to at least one of accept and reject the modified scoring rubric based on the review of the sample. 
     
     
         6 . The method of  claim 5 , wherein the decision is based on the intentions of a third party.