Synthesizing audio for synchronous communication

Abstract

A computer-implemented method includes receiving a first audio stream of a performance associated with a first client device. The method further includes during a time window of the performance, wherein the time window is less than a total time of the performance: generating a synthesized first audio stream that predicts a future of the performance based on audio features of the first audio stream and mixing the synthesized first audio stream and a second audio stream associated with a second client device to form a combined audio stream that synchronizes the synthesized first audio stream and the second audio stream, where the time window is advanced and the generating and the mixing are repeated until the performance is complete.

What is claimed is: 
     
       1. A computer-implemented method comprising:
 receiving a first audio stream of a performance associated with a first client device; and 
 during a time window of the performance, wherein the time window is less than a total time of the performance:
 generating a synthesized first audio stream that predicts a future of the performance based on audio features of the first audio stream; and 
 mixing the synthesized first audio stream and a second audio stream associated with a second client device to form a combined audio stream that synchronizes the synthesized first audio stream and the second audio stream; 
 
 wherein the time window is advanced and the generating and the mixing are repeated until the performance is complete. 
 
     
     
       2. The method of  claim 1 , further comprising:
 responsive to receiving the first audio stream, determining a performance identifier for the performance associated with the first audio stream; and 
 receiving a reference audio based on the performance identifier. 
 
     
     
       3. The method of  claim 2 , wherein:
 generating the synthesized first audio stream includes determining a time offset between the first audio stream and the reference audio; and 
 the time offset occurs when the first audio stream has a different starting point from the reference audio and generating the synthesized first audio stream is further based on the time offset. 
 
     
     
       4. The method of  claim 2 , wherein:
 generating the synthesized first audio stream includes determining a rate of the first audio stream as compared to a rate of the reference audio; and 
 generating the synthesized first audio stream is further based on the rate of the first audio stream as compared to the rate of the reference audio. 
 
     
     
       5. The method of  claim 1 , wherein the audio features of the first audio stream are selected from the group of pitch, rate, phase, or combinations thereof. 
     
     
       6. The method of  claim 1 , wherein the audio features of the first audio stream include one or more speaker identifiers detected in the first audio stream. 
     
     
       7. The method of  claim 1 , further comprising:
 determining that a time difference between the first audio stream and the second audio stream exceeds a threshold time difference; and 
 generating graphical data for displaying a user interface that includes user guidance for the performance and a moving indicator that prompts a performer associated with the second client device to perform in a way that reduces the time difference between the first audio stream and the second audio stream. 
 
     
     
       8. The method of  claim 1 , further comprising modifying the combined audio stream to be consistent with acoustics of an environment where the second client device is located. 
     
     
       9. The method of  claim 1 , wherein generating the synthesized first audio stream includes:
 identifying that a portion of the performance was skipped in the first audio stream; and 
 synthesizing the first audio stream to correct for the portion of the performance that was skipped. 
 
     
     
       10. The method of  claim 1 , further comprising synchronizing the combined audio stream to match actions of performers that are displayed graphically. 
     
     
       11. A device comprising:
 a processor; and 
 a memory coupled to the processor, with instructions stored thereon that, when executed by the processor, cause the processor to perform operations comprising:
 receiving a first audio stream of a performance associated with a first client device; and 
 during a time window of the performance, wherein the time window is less than a total time of the performance:
 generating a synthesized first audio stream that predicts a future of the performance based on audio features of the first audio stream; and 
 mixing the synthesized first audio stream and a second audio stream associated with a second client device to form a combined audio stream that synchronizes the synthesized first audio stream and the second audio stream; 
 
 wherein the time window is advanced and the generating and the mixing are repeated until the performance is complete. 
 
 
     
     
       12. The device of  claim 11 , wherein:
 responsive to receiving the first audio stream, determining a performance identifier for the performance associated with the first audio stream; and 
 receiving a reference audio based on the performance identifier. 
 
     
     
       13. The device of  claim 12 , wherein generating the synthesized first audio stream includes:
 generating the synthesized first audio stream includes determining a time offset between the first audio stream and the reference audio; and 
 the time offset occurs when the first audio stream has a different starting point from the reference audio and generating the synthesized first audio stream is further based on the time offset. 
 
     
     
       14. The device of  claim 12 , wherein:
 generating the synthesized first audio stream includes determining a rate of the first audio stream as compared to a rate of the reference audio; and 
 generating the synthesized first audio stream is further based on the rate of the first audio stream as compared to the rate of the reference audio. 
 
     
     
       15. The device of  claim 11 , wherein the audio features of the first audio stream are selected from the group of pitch, rate, phase, or combinations thereof. 
     
     
       16. A non-transitory computer-readable medium with instructions stored thereon that, when executed by one or more computers, cause the one or more computers to perform operations, the operations comprising:
 receiving a first audio stream of a performance associated with a first client device; and 
 during a time window of the performance, wherein the time window is less than a total time of the performance:
 generating a synthesized first audio stream that predicts a future of the performance based on audio features of the first audio stream; and 
 mixing the synthesized first audio stream and a second audio stream associated with a second client device to form a combined audio stream that synchronizes the synthesized first audio stream and the second audio stream; 
 
 wherein the time window is advanced and the generating and the mixing are repeated until the performance is complete. 
 
     
     
       17. The computer-readable medium of  claim 16 , wherein:
 responsive to receiving the first audio stream, determining a performance identifier for the performance associated with the first audio stream; and 
 receiving a reference audio based on the performance identifier. 
 
     
     
       18. The computer-readable medium of  claim 17 , wherein:
 generating the synthesized first audio stream includes determining a time offset between the first audio stream and the reference audio; and 
 the time offset occurs when the first audio stream has a different starting point from the reference audio and generating the synthesized first audio stream is further based on the time offset. 
 
     
     
       19. The computer-readable medium of  claim 17 , wherein:
 generating the synthesized first audio stream includes determining a rate of the first audio stream as compared to a rate of the reference audio; and 
 generating the synthesized first audio stream is further based on the rate of the first audio stream as compared to the rate of the reference audio. 
 
     
     
       20. The computer-readable medium of  claim 16 , wherein the audio features of the first audio stream are selected from the group of pitch, rate, phase, or combinations thereof.