Method and apparatus for implementing virtual performance partner

Abstract

A method and apparatus for implementing a virtual performance partner are provided. The method includes collecting audio frame data performed by a performer; and for each piece of current audio frame data collected, performing: converting the piece of current audio frame data collected into a current digital score, matching the current digital score with a range of digital scores in a repertoire, and determining a matching digital score in the range of digital scores that matches the current digital score; positioning a position of the matching digital score in the repertoire, and determining a start time of playing a cooperation part of music in a next bar of the matching digital score in the repertoire for a performance partner.

What is claimed is: 
     
         1 . A method for providing a virtual performance partner, the method comprising:
 collecting audio frame data performed by a performer; and   for each piece of current audio frame data collected, performing:   converting the piece of current audio frame data collected into a current digital score;   matching the current digital score with a range of digital scores in a repertoire;   determining a matching digital score in the range of digital scores that matches the current digital score;   positioning a position of the matching digital score in the repertoire;   determining a start time of playing a cooperation part of music in a next bar of the matching digital score in the repertoire for a performance partner;   determining a performance error between the performer and the performance partner based on a performance time of the current digital score and a performance time of the matching digital score; and   adjusting a playing speed of the performance partner for the cooperation part in the repertoire based on the performance error.   
     
     
         2 . The method of  claim 1 , further comprising determining, based on a position of the current digital score, the range within which a next audio frame is matched. 
     
     
         3 . The method of  claim 1 , wherein the determining the start time of playing the cooperation part of music in the next bar of the matching digital score in the repertoire for the performance partner comprises:
 determining a performance speed of the performer based on the position of the matching digital score and positions of matching digital scores corresponding to first N pieces of audio frame data in the current audio frame data;   identifying the performance speed as a reference playing speed of the repertoire; and   determining a start time of playing a next bar of music of the matching digital score in the repertoire for the performance partner based on the reference playing speed.   
     
     
         4 . The method of  claim 3 , wherein the adjusting the playing speed of the performance partner for the repertoire based on the performance error comprises:
 based on the performance error being less than one beat, adjusting the playing speed of the performance partner within a current bar of music according to the performance error based on the reference playing speed, to make the performance partner consistent with the performer in a performance end time of the current bar of music; and   based on the performance error being greater than one beat, pausing playing, by the performance partner, at the current bar, and playing a next bar of music based on a playing time of the next bar of music.   
     
     
         5 . The method of  claim 1 , further comprising:
 based on repeated segments being contained in the repertoire, receiving an inputted set performance segment, and identifying the performance segment as the range.   
     
     
         6 . The method of  claim 1 , further comprising, based on the repertoire starting from a performance of the performance partner, playing, by the performance partner, a part of the repertoire prior to the performance of the performer based on a set playing speed. 
     
     
         7 . The method of  claim 1 , further comprising:
 based on the repertoire transitioning from a solo part of the performer to a performance part of the performance partner:   based on a performance speed of the performer changing, starting to play, by the performance partner, the repertoire based on a performance speed at an end of the solo part; and   based on the performance speed of the performer staying constant, starting to play, by the performance partner, the repertoire according to a set playing speed.   
     
     
         8 . The method of  claim 1 , wherein the performance partner ends the playing of the repertoire based on the current digital score not being matched successfully within a first set time. 
     
     
         9 . The method of  claim 1 , wherein the converting the piece of current audio frame data collected into the current digital score comprises:
 processing the piece of current audio frame data collected using a pre-trained neural network model; and   outputting the current digital score corresponding to the piece of current audio frame data collected, and   wherein the current digital score is represented using a binary saliency map, and the pre-trained neural network model is trained using a binary classification cross entropy loss function.   
     
     
         10 . The method of  claim 1 , wherein the matching is implemented using a neural-network processor. 
     
     
         11 . The method of  claim 1 , further comprising:
 determining a current scene based on the position determined by the positioning; and   synthesizing, corresponding to the current scene, a virtual performance animation corresponding to the current scene using an avatar pre-selected by the performer.   
     
     
         12 . The method of  claim 1 , wherein based on there being a plurality of performance users,
 the performer is a preset performance user among the plurality of performance users; and   based on the matching being unsuccessful within a preset time, the performer is switched to a next preset performance user among the plurality of performance users.   
     
     
         13 . The method of  claim 11 , wherein based on there being a plurality of performance users, an avatar pre-selected by each user is stored; and
 based on the virtual performance animation being displayed, a virtual performance animation synthesized using an avatar pre-selected by a current performer is displayed, and based on the performer being switched, the virtual performance animation is switched to a virtual performance animation synthesized using an avatar pre-selected by a performer switched to; or, avatars pre-selected by all the performance users are displayed simultaneously, and a desired virtual performance animation is synthesized.   
     
     
         14 . The method of  claim 11 , wherein the synthesizing, corresponding to the current scene, the virtual performance animation corresponding to the current scene using the avatar pre-selected by the performer comprises:
 pre-setting an animation switching position in the repertoire, and based on a performance progress of the repertoire by the performance partner reaching the animation switching position, changing the virtual performance animation; and/or   based on the current digital score not being matched successfully and/or the performance error corresponding to the current digital score being greater than a set threshold, changing an avatar preset by the performer into a preset action, and synthesizing the virtual performance animation.   
     
     
         15 . The method of  claim 14 , wherein the animation switching position is a position of switching between different musical instruments within the cooperation part in the repertoire, and
 wherein the changing the virtual performance animation comprises displaying a virtual performance animation preset corresponding to a performance of a musical instrument switched to corresponding to the switching position between the different musical instruments.   
     
     
         16 . An apparatus for implementing a virtual performance partner, the apparatus comprising:
 one or more processors configured to:   collect audio frame data performed by a performer;   convert, for each piece of current audio frame data collected, the piece of current audio frame data collected into a current digital score,   match the current digital score with a range of digital scores in a repertoire,   determine a matching digital score in the range of digital scores that matches the current digital score,   position, for each piece of the current audio frame data collected, a position of the matching digital score in the repertoire,   determine a start time of playing a cooperation part of music in a next bar of the matching digital score in the repertoire for a performance partner,   determine, for each piece of the current audio frame data collected, a performance error between the performer and the performance partner based on a performance time of the current digital score and a performance time of the matching digital score, and   adjust a playing speed of the performance partner for the cooperation part in the repertoire based on the performance error.   
     
     
         17 . A method for providing a virtual performance partner, the method comprising:
 receiving a current digital score corresponding to audio frame data;   matching the current digital score with a range of digital scores in a repertoire;   determining a matching digital score in the range of digital scores based on matching the current digital score;   identifying a position of the matching digital score in the repertoire;   identifying a start time of playing a cooperation part of music in a next bar of the matching digital score in the repertoire for a performance partner;   determining a performance error between the performer and the performance partner based on a performance time of the current digital score and a performance time of the matching digital score; and   adjusting a playing speed of the performance partner for the cooperation part in the repertoire based on the performance error.   
     
     
         18 . The method of  claim 17 , further comprising determining, based on a position of the current digital score, the range within which a next audio frame is matched. 
     
     
         19 . The method of  claim 17 , further comprising:
 based on the repertoire starting from a performance of the performance partner, playing, by the performance partner, a part of the repertoire prior to the performance of the performer based on a set playing speed.   
     
     
         20 . The method of  claim 17 , wherein the performance partner ends the playing of the repertoire based on the current digital score not being matched successfully within a first set time.