Feature extraction and machine learning for evaluation of media-rich coursework

Abstract

Conventional techniques for automatically evaluating and grading assignments are generally ill-suited to evaluation of coursework submitted in media-rich form. For courses whose subject includes programming, signal processing or other functionally expressed designs that operate on, or are used to produce media content, conventional techniques are also ill-suited. It has been discovered that media-rich, indeed even expressive, content can be accommodated as, or as derivatives of, coursework submissions using feature extraction and machine learning techniques. Accordingly, in on-line course offerings, even large numbers of students and student submissions may be accommodated in a scalable and uniform grading or scoring scheme. Instructors or curriculum designers may adaptively refine assignments or testing based on classifier feedback. Using developed techniques, it is possible to administer courses and automatically grade submitted work that takes the form of media encodings of artistic expression, computer programming and even signal processing to be applied to media content.

1 . (canceled) 
     
     
         2 . A method for use in connection with automated evaluation of coursework submissions, the method comprising:
 receiving from an instructor or curriculum designer a selection of exemplary media content to be used in evaluating the coursework submissions, the exemplary media content including a training set of examples each assigned at least one quality score by the instructor or curriculum designer;   accessing computer readable encodings of the exemplary media content that together constitute the training set and extracting from each instance of exemplary media content a first set of computationally defined features;   for each instance of exemplary media content, supplying a classifier with both the instructor or curriculum designer's assigned quality score and values for the computationally defined features extracted therefrom;   based on the supplied quality scores and extracted feature values, training the classifier, wherein the training includes updating internal states thereof;   accessing a computer readable encoding of media content that constitutes, or is derived from, the coursework submission, wherein the computer readable encoding of media content includes an audio signal encoding processed or rendered by software code included in the coursework submission, wherein the audio signal encoding is generated based on a reference audio signal;   extracting, from the computer readable encoding of media content, a second set of computationally defined features; and   applying the trained classifier to the extracted second set of computationally defined features and, based thereon, assigning a particular quality score to the coursework submission.   
     
     
         3 . The method of  claim 2 , further comprising:
 supplying plural additional classifiers with respective instructor or curriculum designer's assigned quality scores and values for computationally defined features extracted from respective instances of the exemplary media content;   training the additional classifiers; and   applying the trained additional classifiers.   
     
     
         4 . The method of  claim 2 ,
 wherein the quality score is, or is a component of, a grading scale for an assignment- or test question-type coursework submission.   
     
     
         5 . The method of  claim 2 ,
 wherein the software code is submitted in satisfaction of a programming assignment or test question, the software code executable to perform, or compilable to execute and perform, digital signal processing to produce output media content;   wherein the exemplary media content includes exemplary output media content produced using exemplary software codes; and   wherein the particular quality score assigned to the coursework submission is based on the applying of the classifier to the second set of computationally defined features extracted from the output media content produced by execution of the submitted software code.   
     
     
         6 . The method of  claim 5 ,
 wherein the software code coursework submission is executable to perform digital signal processing on input media content to produce the output media content; and   wherein the exemplary output media content is produced from the input media content using the exemplary software codes.   
     
     
         7 . The method of  claim 2 ,
 wherein for the first and second sets, at least some of the computationally defined features are selected or derived from:
 a root mean square energy value; 
 a number of zero crossings per frame; 
 a spectral flux; 
 a spectral centroid; 
 a spectral roll-off measure; 
 a spectral tilt; 
 a mel-frequency cepstral coefficients (MFCC) representation of short-term power spectrum; 
 a beat histogram; and/or 
 a multi-pitch histogram 
   computed over at least a portion of the audio signal encoding.   
     
     
         8 . The method of  claim 2 ,
 wherein the classifier implements an artificial neural network (NN), k-nearest neighbor (KNN), Gaussian mixture model (GMM), support vector machine (SVN) or other statistical classification technique.   
     
     
         9 . The method of  claim 2 , further comprising:
 iteratively refining the classifier training based on supply of successive instances of the exemplary media content to the classifier and updates to internal states thereof.   
     
     
         10 . The method of  claim 9 ,
 wherein the classifier is implemented using one or more logical binary decision trees, blackboard voting-type methods, or rule-based classification techniques.   
     
     
         11 . The method of  claim 9 , further comprising:
 continuing the iterative refining until an error metric based on a current state of classifier training falls below a predetermined or instructor or curriculum designer-defined threshold.   
     
     
         12 . The method of  claim 2 , further comprising:
 supplying the instructor or curriculum designer with an error metric based on a current state of classifier training.   
     
     
         13 . The method of  claim 2 , further comprising:
 supplying the instructor or curriculum designer with a coursework task recommendation based on a particular one or more of the computationally defined features that contribute most significantly to classifier performance against the training set.   
     
     
         14 . The method of  claim 2 ,
 wherein the first and second sets of computationally defined features are the same.   
     
     
         15 . The method of  claim 2 ,
 wherein the second set of computationally defined features includes a subset of the first set features selected based on contribution to classifier performance against the training set.   
     
     
         16 . The method of  claim 2 ,
 wherein the quality score is, or is a component of a grading scale for an assignment- or test question-type coursework submission.   
     
     
         17 . The method of  claim 2 , further comprising:
 receiving from the instructor or curriculum designer at least an initial definition of the first set of computationally defined features.   
     
     
         18 . A computational system including one or more operative computers programmed to perform:
 receiving from an instructor or curriculum designer a selection of exemplary media content to be used in evaluating the coursework submissions, the exemplary media content including a training set of examples each assigned at least one quality score by the instructor or curriculum designer;   accessing computer readable encodings of the exemplary media content that together constitute the training set and extracting from each instance of exemplary media content a first set of computationally defined features;   for each instance of exemplary media content, supplying a classifier with both the instructor or curriculum designer's assigned quality score and values for the computationally defined features extracted therefrom;   based on the supplied quality scores and extracted feature values, training the classifier, wherein the training includes updating internal states thereof;   accessing a computer readable encoding of media content that constitutes, or is derived from, the coursework submission, wherein the computer readable encoding of media content includes an audio signal encoding processed or rendered by software code included in the coursework submission, wherein the audio signal encoding is generated based on a reference audio signal;   extracting, from the computer readable encoding of media content, a second set of computationally defined features; and   applying the trained classifier to the extracted second set of computationally defined features and, based thereon, assigning a particular quality score to the coursework submission.   
     
     
         19 . The computational system of  claim 18  embodied, at least in part, as a network deployed coursework submission system, whereby a large and scalable plurality (>50) of geographically dispersed students may individually submit their respective coursework submissions in the form of computer readable information encodings. 
     
     
         20 . The computational system of  claim 19 , further comprising: a student authentication interface for associating a particular coursework submission with a particular one of the geographically dispersed students. 
     
     
         21 . A non-transient computer readable encoding of instructions executable on one or more operative computers to perform:
 receiving from an instructor or curriculum designer a selection of exemplary media content to be used in evaluating the coursework submissions, the exemplary media content including a training set of examples each assigned at least one quality score by the instructor or curriculum designer;   accessing computer readable encodings of the exemplary media content that together constitute the training set and extracting from each instance of exemplary media content a first set of computationally defined features;   for each instance of exemplary media content, supplying a classifier with both the instructor or curriculum designer's assigned quality score and values for the computationally defined features extracted therefrom;   based on the supplied quality scores and extracted feature values, training the classifier, wherein the training includes updating internal states thereof;   accessing a computer readable encoding of media content that constitutes, or is derived from, the coursework submission, wherein the computer readable encoding of media content includes an audio signal encoding processed or rendered by software code included in the coursework submission, wherein the audio signal encoding is generated based on a reference audio signal;   extracting, from the computer readable encoding of media content, a second set of computationally defined features; and   applying the trained classifier to the extracted second set of computationally defined features and, based thereon, assigning a particular quality score to the coursework submission.   
     
     
         22 . A coursework management system for automated evaluation of coursework submissions, the system comprising:
 an instructor or curriculum designer interface for selecting or receiving exemplary media content to be used in evaluating the coursework submissions, the exemplary media content including a training set of examples each assigned at least one quality score by the instructor or curriculum designer;   a training subsystem coupled and programmed to access computer readable encodings of the exemplary media content that together constitute the training set and to extracting from each instance of exemplary media content a first set of computationally defined features;   the training subsystem further programmed to, for each instance of exemplary media content, supply a classifier with both the instructor or curriculum designer's assigned quality score and values for the computationally defined features extracted therefrom, and to, based on the supplied quality scores and extracted feature values, train the classifier, wherein the training includes updating internal states thereof; and   a coursework evaluation deployment of the trained classifier coupled and programmed to access a computer readable encoding of media content that constitutes, or is derived from, the coursework submissions and to extract therefrom a second set of computationally defined features,   wherein the media content that constitutes, or is derived from, the coursework submission includes an audio signal generated based on a reference audio signal;   wherein the coursework evaluation deployment is configured to apply the trained classifier to the extracted second set of computationally defined features and, based thereon, assigns a particular quality score to the coursework submission.   
     
     
         23 . The coursework management system as in  claim 22 ,
 wherein the training subsystem supplies plural additional classifiers with respective instructor or curriculum designer's assigned quality scores and values for computationally defined features extracted from respective instances of the exemplary media content and trains the additional classifiers; and   wherein coursework evaluation deployment also applies the trained additional classifiers.   
     
     
         24 . The coursework management system as in  claim 22 , further comprising:
 an execution environment,   wherein the coursework submission includes software code submitted in satisfaction of a programming assignment or test question, the software code executable in the execution environment to perform, or compilable to execute in the execution environment and perform, digital signal processing to produce output media content;   wherein the exemplary media content includes the output media content produced using the submitted software code; and   wherein the particular quality score assigned to the coursework submission is based on the applying of the classifier to the second set of computationally defined features extracted from the output media content produced using the submitted software code.   
     
     
         25 . The coursework management system as in  claim 22 ,
 wherein the classifier implements an artificial neural network (NN), k-nearest neighbor (KNN), Gaussian mixture model (GMM), support vector machine (SVN) or other statistical classification technique.   
     
     
         26 . The coursework management system as in  claim 22 ,
 wherein the training subsystem allows the instructor or curriculum designer to iteratively refine the classifier training based on supply of successive instances of the exemplary media content to the classifier and updates to internal states thereof.   
     
     
         27 . The coursework management system as in  claim 22 ,
 wherein the classifier is implemented using one or more logical binary decision trees, blackboard voting-type methods, or rule-based classification techniques.