Media classification using local and global audio features

Abstract

Methods, systems, and computer-readable media for media classification using local and global audio features are disclosed. A media classification system determines local features of an audio input using an audio event detector model that is trained to detect a plurality of audio event classes descriptive of objectionable content. The local features are extracted using maximum values of audio event scores for individual audio event classes. The media classification system determines one or more global features of the audio input using the audio event detector model. The global feature(s) are extracted using averaging of clip-level descriptors of a plurality of clips of the audio input. The media classification system determines a content-based rating for media comprising the audio input based (at least in part) on the local features of the audio input and based (at least in part) on the global feature(s) of the audio input.

What is claimed is: 
     
       1. A system, comprising:
 a media classification system comprising one or more processors and one or more memories to store computer-executable instructions that, when executed, cause the one or more processors to:
 determine a plurality of audio event classes descriptive of a plurality of categories of content including violent content; 
 train an audio event detector model to detect the plurality of audio event classes; 
 extract a plurality of local features of an audio input using the audio event detector model, wherein the audio input comprises an audio modality of a video input, wherein the plurality of local features are extracted using max-pooling of audio event scores for individual ones of the plurality of audio event classes; 
 extract one or more global features of the audio input using the audio event detector model, wherein the one or more global features are extracted using mean-pooling of a plurality of clip-level descriptors of a plurality of clips of the audio input, and wherein the one or more global features summarize one or more content descriptors including a violent content descriptor in the audio input; 
 train a media classifier model based at least in part on the plurality of local features of the audio input and based at least in part on the one or more global features of the audio input; and 
 assign a content-based rating to the video input using the media classifier model. 
 
 
     
     
       2. The system as recited in  claim 1 , wherein the one or more memories store additional computer-executable instructions that, when executed, cause the one or more processors to:
 determine one or more additional scores descriptive of violent content in a visual modality of the video input comprising the audio input, wherein the content-based rating for the video input is determined based at least in part on the one or more additional scores. 
 
     
     
       3. The system as recited in  claim 1 , wherein the one or more memories store additional computer-executable instructions that, when executed, cause the one or more processors to:
 determine one or more additional scores descriptive of violent content in a textual modality of the video input comprising the audio input, wherein the content-based rating for the video input is determined based at least in part on the one or more additional scores. 
 
     
     
       4. The system as recited in  claim 1 , wherein the audio event detector model is used to extract additional local and global features of a plurality of additional video inputs. 
     
     
       5. A method, comprising:
 determining, by a media classification system, a plurality of local features of an audio input using an audio event detector model, wherein the audio event detector model is trained to detect a plurality of audio event classes descriptive of objectionable content, wherein the plurality of local features are determined using one or more maximum values of audio event scores for individual ones of the plurality of audio event classes in the audio input; 
 determining, by the media classification system, one or more global features of the audio input using the audio event detector model, wherein the one or more global features are extracted using averaging of a plurality of clip-level descriptors of a plurality of clips of the audio input; and 
 determining, by the media classification system, a content-based rating for media comprising the audio input based at least in part on the plurality of local features of the audio input and based at least in part on the one or more global features of the audio input. 
 
     
     
       6. The method as recited in  claim 5 , further comprising:
 determining, by the media classification system, one or more additional scores descriptive of objectionable content in a visual modality of the media comprising the audio input, wherein the content-based rating for the media is determined based at least in part on the one or more additional scores. 
 
     
     
       7. The method as recited in  claim 5 , further comprising:
 determining, by the media classification system, one or more additional scores descriptive of objectionable content in a textual modality of the media comprising the audio input, wherein the content-based rating for the media is determined based at least in part on the one or more additional scores. 
 
     
     
       8. The method as recited in  claim 5 , wherein the media comprising the audio input comprises video, and wherein the objectionable content comprises violent content. 
     
     
       9. The method as recited in  claim 5 , wherein a violent content descriptor is determined based at least in part on the plurality of local features of the audio input and based at least in part on the one or more global features of the audio input, and wherein the content-based rating for the media is determined based at least in part on the violent content descriptor. 
     
     
       10. The method as recited in  claim 5 , wherein the plurality of local features are extracted using max-pooling of the audio event scores for the individual ones of the plurality of audio event classes. 
     
     
       11. The method as recited in  claim 5 , wherein the one or more global features are extracted using mean-pooling of the plurality of clip-level descriptors of the plurality of clips of the audio input. 
     
     
       12. The method as recited in  claim 5 , further comprising:
 determining, by the media classification system, a media classifier model based at least in part on the plurality of local features of the audio input and based at least in part on the one or more global features of the audio input, wherein the content-based rating is determined using the media classifier model. 
 
     
     
       13. The method as recited in  claim 5 , wherein the media classification system is implemented as a service of a cloud provider network, and wherein the service is used by a plurality of clients to determine content-based ratings for a plurality of media titles. 
     
     
       14. One or more non-transitory computer-readable storage media storing program instructions that, when executed on or across one or more processors, perform:
 determining, by a media classification system, a plurality of local features of an audio input using an audio event detector model, wherein the audio event detector model is trained to detect a plurality of audio event classes descriptive of violent content, wherein the plurality of local features are determined using max-pooling of audio event scores for individual ones of the plurality of audio event classes; 
 determining, by the media classification system, one or more global features of the audio input using the audio event detector model, wherein the one or more global features are determined using mean-pooling of a plurality of semantic embeddings of a plurality of portions of the audio input; and 
 determining, by the media classification system, a content-based rating for a video comprising the audio input based at least in part on the plurality of local features of the audio input and based at least in part on the one or more global features of the audio input. 
 
     
     
       15. The one or more non-transitory computer-readable storage media as recited in  claim 14 , further comprising additional program instructions that, when executed on or across the one or more processors, perform:
 determining, by the media classification system, one or more additional scores descriptive of violent content in a visual modality of the video comprising the audio input, wherein the content-based rating for the video is determined based at least in part on the one or more additional scores. 
 
     
     
       16. The one or more non-transitory computer-readable storage media as recited in  claim 14 , further comprising additional program instructions that, when executed on or across the one or more processors, perform:
 determining, by the media classification system, one or more additional scores descriptive of violent content in a textual modality of the video comprising the audio input, wherein the content-based rating for the video is determined based at least in part on the one or more additional scores. 
 
     
     
       17. The one or more non-transitory computer-readable storage media as recited in  claim 14 , wherein a violent content descriptor is determined based at least in part on the plurality of local features of the audio input and based at least in part on the one or more global features of the audio input, and wherein the content-based rating for the video is determined based at least in part on the violent content descriptor. 
     
     
       18. The one or more non-transitory computer-readable storage media as recited in  claim 14 , further comprising additional program instructions that, when executed on or across the one or more processors, perform:
 training, by the media classification system, a media classifier model based at least in part on the plurality of local features of the audio input and based at least in part on the one or more global features of the audio input, wherein the content-based rating is determined using the media classifier model. 
 
     
     
       19. The one or more non-transitory computer-readable storage media as recited in  claim 14 , wherein the media classification system is implemented as a service of a cloud provider network, and wherein the service is offered to a plurality of clients. 
     
     
       20. The one or more non-transitory computer-readable storage media as recited in  claim 14 , wherein the audio event detector model comprises one or more machine learning models.