Audiovisual capture and sharing framework with coordinated, user-selectable audio and video effects filters

Abstract

Coordinated audio and video filter pairs are applied to enhance artistic and emotional content of audiovisual performances. Such filter pairs, when applied in audio and video processing pipelines of an audiovisual application hosted on a portable computing device (such as a mobile phone or media player, a computing pad or tablet, a game controller or a personal digital assistant or book reader) can allow user selection of effects that enhance both audio and video coordinated therewith. Coordinated audio and video are captured, filtered and rendered at the portable computing device using camera and microphone interfaces, using digital signal processing software executable on a processor and using storage, speaker and display devices of, or interoperable with, the device. By providing audiovisual capture and personalization on an intimate handheld device, social interactions and postings of a type made popular by modern social networking platforms can now be extended to audiovisual content.

1 . (canceled) 
     
     
         2 . A method, comprising:
 processing, in an audio pipeline, an audio stream to generate a plurality of audio segments corresponding to respective subphrase portions of a phrase template for a target song;   processing, in a video pipeline, a video stream corresponding to the audio stream to generate a plurality of video segments corresponding to the audio segments;   assigning coordinated audio and video effect filters to the audio segments and corresponding video segments; the mapped video segments; and   automatically generating, in the audio pipeline, a musical accompaniment for vocals in the audio stream as specified by the audio filter in the audio pipeline.   
     
     
         3 . The method of  claim 2 , wherein the video effect filter is applied to the video segments based on audio features extracted in the audio pipeline. 
     
     
         4 . The method of  claim 3 , wherein the audio features include melody pitches, and wherein the musical accompaniment is generated based on a selection of chords that are harmonies of the melody pitches. 
     
     
         5 . The method of  claim 2 ,
 wherein the audio stream includes vocals temporally synchronized with the video stream, and   wherein the audio segments are delimited in the audio pipeline based on onsets detected in the vocals.   
     
     
         6 . The method of  claim 2 , further comprising:
 in the audio pipeline, temporally aligning successive ones of the audio segments with respective pulses of a rhythmic skeleton for the target song, and temporally adjusting at least some of the temporally aligned segments,   in the video pipeline, temporally aligning and adjusting the respective video segments in correspondence with the audio segmentation aligning and adjusting.   
     
     
         7 . The method of  claim 2 , further comprising:
 using, in the audio pipeline, one or more temporally localizable features extracted in the video pipeline.   
     
     
         8 . The method of  claim 2 , further comprising:
 applying artistically consistent effects to the audio and video streams.   
     
     
         9 . A non-transitory machine-readable medium comprising a plurality of machine-readable instructions which, when executed by one or more processors, are adapted to cause the one or more processors to perform a method comprising:
 processing, in an audio pipeline, an audio stream to generate a plurality of audio segments corresponding to respective subphrase portions of a phrase template for a target song;   processing, in a video pipeline, a video stream corresponding to the audio stream to generate a plurality of video segments corresponding to the audio segments;   assigning coordinated audio and video effect filters to the audio segments and corresponding video segments; the mapped video segments; and   automatically generating, in the audio pipeline, a musical accompaniment for vocals in the audio stream as specified by the audio filter in the audio pipeline.   
     
     
         10 . The non-transitory machine-readable medium of  claim 9 , wherein the video effect filter is applied to the video segments based on audio features extracted in the audio pipeline. 
     
     
         11 . The non-transitory machine-readable medium of  claim 10 , wherein the audio features include melody pitches, and wherein the musical accompaniment is generated based on a selection of chords that are harmonies of the melody pitches. 
     
     
         12 . The non-transitory machine-readable medium of  claim 9 ,
 wherein the audio stream includes vocals temporally synchronized with the video stream, and   wherein the audio segments are delimited in the audio pipeline based on onsets detected in the vocals.   
     
     
         13 . The non-transitory machine-readable medium of  claim 9 , wherein the method further comprises:
 in the audio pipeline, temporally aligning successive ones of the audio segments with respective pulses of a rhythmic skeleton for the target song, and temporally adjusting at least some of the temporally aligned segments,   in the video pipeline, temporally aligning and adjusting the respective video segments in correspondence with the audio segmentation aligning and adjusting.   
     
     
         14 . The non-transitory machine-readable medium of  claim 9 , wherein the method further comprises:
 using, in the audio pipeline, one or more temporally localizable features extracted in the video pipeline.   
     
     
         15 . The non-transitory machine-readable medium of  claim 9 , wherein the method further comprises:
 applying artistically consistent effects to the audio and video streams.   
     
     
         16 . A system, comprising:
 a non-transitory memory; and   one or more hardware processors coupled to the non-transitory memory and configured to read instructions from the non-transitory memory to cause the system to perform a method comprising:   processing, in an audio pipeline, an audio stream to generate a plurality of audio segments corresponding to respective subphrase portions of a phrase template for a target song;   processing, in a video pipeline, a video stream corresponding to the audio stream to generate a plurality of video segments corresponding to the audio segments;   assigning coordinated audio and video effect filters to the audio segments and corresponding video segments; the mapped video segments; and   automatically generating, in the audio pipeline, a musical accompaniment for vocals in the audio stream as specified by the audio filter in the audio pipeline.   
     
     
         17 . The system of  claim 16 , wherein the video effect filter is applied to the mapped video segments based on audio features extracted in the audio pipeline. 
     
     
         18 . The system of  claim 17 , wherein the audio features include melody pitches, and wherein the musical accompaniment is generated based on a selection of chords that are harmonies of the melody pitches. 
     
     
         19 . The system of  claim 16 ,
 wherein the audio stream includes vocals temporally synchronized with the video stream, and   wherein the audio segments are delimited in the audio pipeline based on onsets detected in the vocals.   
     
     
         20 . The system of  claim 16 , wherein the method further comprises:
 in the audio pipeline, temporally aligning successive ones of the audio segments with respective pulses of a rhythmic skeleton for the target song, and temporally adjusting at least some of the temporally aligned segments,   in the video pipeline, temporally aligning and adjusting the respective video segments in correspondence with the audio segmentation aligning and adjusting.   
     
     
         21 . The system of  claim 16 , wherein the method further comprises:
 using, in the audio pipeline, one or more temporally localizable features extracted in the video pipeline.