System and method for computer based creation of tests formatted to facilitate computer based testing

Abstract

A computer-based test creation system and method for generating test items. The test creation process includes creating items in electronic form and allowing access by a plurality of reviewers via a network. The reviewers may review the item for content, fairness and format, and collaborate with each other before indicating that the item is acceptable for inclusion in a test. After the administration of the items to test takers, item performance may be evaluated on-line by reviewing electronic feedback from the administrations and creating electronic evaluation reports. Since the items are created in an electronic form, the items may further include new forms of stimuli such as graphics and multimedia.

1 . A method for generating a test item in electronic form using an automated test creation system, comprising the steps of: 
 electronically associating components of the test item into an extensible markup language based electronic item form using the test creation system:    storing the electronic item form in an item repository of the test creation system;    any of a plurality of test creators accessing the electronic item form from the item repository and utilizing the test creation system to review the content and format of the electronic item form and to update the content and format of the electronic item form as appropriate; and    storing the updated electronic item form in the item repository as a locked item when the content and format are deemed acceptable by the test creators for inclusion in a test, where a locked item is not to be further reviewed by any of said test creators.    
   
   
       2 . The method of  claim 1 , wherein the components of the test item include a written text representative of a test question and an answer key.  
   
   
       3 . The method of  claim 2  wherein the components of the test item further includes a stimulus to which the test questions refers.  
   
   
       4 . The method of  claim 1 , wherein the step of electronically associating components of the test item comprises the steps of: 
 providing an extensible markup language based editing program with identification information relating to a selected test program, a selected test section, a selected delivery mode, and a selected item type for which the electronic item form is being created;    retrieving an extensible markup language based item template based on the information so provided from a template storage facility of the test creation system;    completing the item by entering written text representative of a test question, an answer key, and a stimulus to which the test questions refers; and    releasing the item template so completed to the item repository where the written text, stimulus, and answer key are electronically associated as the electronic item form.    
   
   
       5 . The method of  claim 4 , wherein the stimulus comprises at least one of: written text, graphics, video, audio, audiovisual material, animation and scanned images.  
   
   
       6 . The method of  claim 5 , further comprising the steps of: 
 accessing an one-line source for locating stimulus material via a communications interface of the test creation system;    selecting certain stimulus material; and    importing the selected stimulus material into the item template as the stimulus being associated with the written text and answer key as the electronic item form.    
   
   
       7 . The method of  claim 4 , wherein the stimulus comprises graphics, and the method further comprises steps of: 
 accessing an object database of the test creation system having stored therein predefined graphical objects;    selecting one of the predefined graphical objects; and    importing the one selected graphical object into the item template as the stimulus being associated with the written text and answer key as the electronic item form.    
   
   
       8 . The method of  claim 4 , wherein the stimulus comprises graphics, and the method further comprises the steps of: 
 generating a graphical object using a drawing application supported by the test creation system; and    importing the graphical object into the item template a the stimulus being associated with the written text and answer key as the electronic item form.    
   
   
       9 . The method of  claim 1 , wherein the electronic item form is reviewed in said accessing and utilizing step in sequence for content, fairness, and form at by at least one reviewer for each of content, fairness, and format, respectively, the method further comprising the steps of: 
 when a content reviewer has completed the content review, automatically routing the electronic item form to a fairness reviewer;    when the fairness reviewer has completed the fairness review, automatically routing the electronic item form to a format reviewer; and    when the format reviewer has completed the format review, automatically routing the electronic item form to a final resolver for a final review to determine whether the content, fairness, and format are acceptable for test assembly.    
   
   
       10 . The method of  claim 9 , wherein a number of reviewers are assigned to review the content, fairness and format of the electronic item form, and wherein at least two of the reviewers electronically collaborate concerning at least one of the content, fairness, and format of the electronic item form.  
   
   
       11 . The method of  claim 9 , further comprising the step of generating a workflow log listing the progress of each electronic item form as it is reviewed by each of the reviewers.  
   
   
       12 . The method of  claim 1 , including the additional steps of reviewing statistical information obtained after an administration of said test item and, if said test item does not meet predetermined performance criteria, initiating an electronic evaluation report for said test item.  
   
   
       13 . The method of  claim 12 , including the additional step of providing a statistical flag, which, when set, automatically initiates said electronic evaluation report.  
   
   
       14 . The method of  claim 1 , further comprising the step of automatically initiating copyright approval for said test item before the test item is stored as a locked item by initiating a process of automatically initiating copyright approval for the test item during review of the electronic item form.  
   
   
       15 . The method of  claim 1 , including the additional steps of monitoring the progress and checking the status of electronic item forms in the test creation system by measuring metrics statistics and management statistics for said electronic item forms.  
   
   
       16 . The method of  claim 15 , comprising the further steps of calculating the metrics statistics by cycle time, quality, and count, and calculating the management statistics by said cycle time and said count, wherein said count measures the number of items at a particular stage in a particular subsystem, said cycle time measures a number of days a particular item has been at a particular stage in a particular subsystem of said test creation system, and said quality measures a percentage of at least one of locked items, pre-test items, items with problems, or specific item problems, and present in all problems out of an overall group of items that could potentially be designed as a locked item.  
   
   
       17 - 42 . (canceled)  
   
   
       43 . A method for generating a test item in electronic form using an automated test creation system, comprising the steps of: 
 generating an electronic item form formatted to facilitate computer based testing over a network;    storing the electronic item form in an item repository;    reviewing the content and format of the electronic item;    storing the updated electronic item form in the item repository as a locked item when the content and format are acceptable, where a locked item is not to be further reviewed;    transmitting the item over the network as part of a computer based test; and receiving statistical feedback on the item.    
   
   
       44 . (canceled)  
   
   
       45 . The method of  claim 43 , comprising transmitting the item over the Internet as part of a computer based test.  
   
   
       46 . The method of  claim 43 , comprising generating an electronic item form formatted using extensible markup language.