Multi-stage adaptive system for content moderation

Abstract

A toxicity moderation system has an input configured to receive speech from a speaker. The system includes a multi-stage toxicity machine learning system having a first stage and a second stage. The first stage is trained to analyze the received speech to determine whether a toxicity level of the speech meets a toxicity threshold. The first stage is also configured to filter-through, to the second stage, speech that meets the toxicity threshold, and is further configured to filter-out speech that does not meet the toxicity threshold.

1 . A moderation system for managing content, the system comprising:
 a plurality of successive stages arranged in series, each stage configured to receive input content and filter the input content to produce filtered content, a plurality of the stages each configured to forward the filtered content toward a successive stage; and   training logic operatively coupled with the stages, the training logic configured to use information relating to speech toxicity processing by a given subsequent stage to train speech toxicity processing of an earlier stage, the given subsequent stage receiving content derived directly from the earlier stage or from at least one stage between the given subsequent stage and the earlier stage.   
     
     
         2 . The system of  claim 1 , wherein the filtered content of each stage comprises a subset of the received input content. 
     
     
         3 . The system of  claim 1 , wherein each stage is configured to produce filtered content from input content to forward to a less efficient stage, a given less efficient stage being more powerful than a second more efficient stage. 
     
     
         4 . The system of  claim 1 , wherein at least one stage of the plurality of successive stages is configured to receive forwarded content from a prior stage and send forwarded content to a later stage. 
     
     
         5 . The system of  claim 1 , wherein the plurality of successive stages together have a maximum moderation capacity, one stage having the most efficient stage and having the highest percentage of the maximum moderation capacity. 
     
     
         6 . The system of  claim 1 , wherein a first and second stages execute on a user device, a third and fourth stage execute off-device, the first and second stages executing more moderation capacity than that of the third and fourth stages. 
     
     
         7 . The system of  claim 1 , further comprising a user interface to receive input from at least one stage and verify processing by one or more of the plurality of stages. 
     
     
         8 . The system of  claim 1 , wherein the training logic is executed as a computer program product comprising a tangible medium storing program code. 
     
     
         9 . A moderation system comprising:
 a plurality of successive stages arranged in series from most efficient stage to least efficient stage of the plurality of stages, each stage configured to produce forwarded content from input content to forward to a less efficient stage; and   training logic operatively coupled with the stages, the training logic configured to use information relating to processing by a given stage to train processing of a second stage that is adjacent and more efficient at processing than the given stage.   
     
     
         10 . The moderation system of  claim 9 , wherein at least one stage of the plurality of successive stages is configured to receive forwarded content from a prior stage and send forwarded content to a later stage. 
     
     
         11 . The moderation system of  claim 9 , wherein the plurality of successive stages together have a maximum moderation capacity, the most efficient stage having the highest percentage of the maximum moderation capacity. 
     
     
         12 . The moderation system of  claim 9 , wherein a first and second stages execute on a user device, a third and fourth stage executing off-device, the first and second stages executing more moderation capacity than that of the third and fourth stages. 
     
     
         13 . The moderation system of  claim 9 , further having a user interface to receive input from the least efficient stage and verify processing by one or more of the plurality of stages. 
     
     
         14 . The moderation system of  claim 9 , wherein the training logic is executed as a computer program product comprising a tangible medium storing program code. 
     
     
         15 . A computer program product for use on a computer system for training a multi-stage content analysis system, the computer program product comprising a tangible, non-transient computer usable medium having computer readable program code thereon, the computer readable program code comprising:
 program code for providing a multi-stage content analysis system, the system having a first stage and a second stage;   program code for training the first stage using a database having training data with positive and/or negative examples of training content for the first stage;   program code for receiving speech content;   program code for analyzing the speech content using the first stage to categorize the speech content as having first-stage positive speech content and/or first-stage negative speech content;   program code for analyzing the first-stage positive speech content using the second stage to categorize the first-stage positive speech content as having second-stage positive speech content and/or second-stage negative speech content;   program code for updating the database using the second-stage positive speech content and/or the second-stage negative speech content;   program code for discarding at least a portion of the first-stage negative speech content.   
     
     
         16 . The computer program product of  claim 15 , the program code comprising:
 program code for analyzing less than all of the first-stage negative speech content using the second stage to categorize the first-stage negative speech content as having second-stage positive speech content and/or second-stage negative speech content; and   program code for further updating the database using the second-stage positive speech content and/or the second-stage negative speech content.   
     
     
         17 . The computer program product of  claim 15 , the program code comprising:
 program code for using a database having training data with positive and/or negative examples of training content for the first stage;   program code for producing first-stage positive determinations (“S1-positive determinations”) associated with a portion of the speech content, and/or first-stage negative determinations (“S1-negative determinations”);   program code for analyzing the speech associated with the S1-positive determinations.   
     
     
         18 - 22 . (canceled).