Ai based remixing of music: timbre transformation and matching of mixed audio data

Abstract

The present invention provides a method for processing audio data, comprising the steps of providing input audio data containing a mixture of audio data including first audio data of a first musical timbre and second audio data of a second musical timbre different from said first musical timbre, decomposing the input audio data to provide decomposed data representative of the first audio data, transforming the decomposed data to obtain third audio data.

1 . A method for processing audio data, comprising the steps of:
 providing input audio data containing a mixture of audio data including first audio data of a first musical timbre and second audio data of a second musical timbre different from said first musical timbre;   decomposing the input audio data to provide decomposed data representative of the first audio data; and   transforming the decomposed data to obtain third audio data, wherein transforming the decomposed data includes at least one of:
 i. changing musical timbre such that the third audio data are of a third musical timbre different from the first musical timbre, or 
 ii. changing melody, such that the third audio data represents a melody different from that of the decomposed data. 
   
     
     
         2 . The method of  claim 1 , wherein transforming the decomposed data includes changing musical timbre and wherein the third audio data and the decomposed data represent the a same melody or represent no melody. 
     
     
         3 . The method of  claim 1 , wherein the third audio data and the decomposed data have at least one of equal key or equal time-dependent harmony. 
     
     
         4 . The method of  claim 1 , wherein the third audio data and the decomposed data have the same timbre. 
     
     
         5 . The method of  claim 1 , wherein changing melody includes:
 detecting a time dependent musical harmony of the input audio data or decomposed data; and   generating, based on the time dependent musical harmony, pitch data for a plurality of individual musical tones of the third audio data, which are to be played sequentially at respective predetermined points in times to generate a melody of the third audio data.   
     
     
         6 . The method of  claim 1 , further comprising detecting pitch data indicating musical pitches of the decomposed data or the first audio data. 
     
     
         7 . The method of  claim 1 , further comprising a step of converting the decomposed data into event message data formed by a plurality of event messages of musical tones, wherein each event message at least specifies pitch data and velocity data of a corresponding musical tone. 
     
     
         8 . The method of  claim 7 , wherein the step of transforming the decomposed data includes synthesizer-based or sample-based generation of audio signals based on the pitch data or the event message data. 
     
     
         9 . The method of  claim 1 , wherein the step of transforming the decomposed data involves processing of audio data obtained from the decomposed data within an artificial intelligence system. 
     
     
         10 . The method of  claim 1 , wherein the step of decomposing the input audio data involves processing of audio data obtained from the input audio data within an artificial intelligence system. 
     
     
         11 . The method of  claim 1 , wherein the step of decomposing the input audio data provides first decomposed data representative of the first audio data, and second decomposed data representative of the second audio data, and wherein the method further comprises a step of recombining audio data obtained from the third audio data with audio data obtained from the second decomposed data to obtain recombined audio data. 
     
     
         12 . The method of  claim 1 , wherein the step of decomposing the input audio data provides a plurality of sets of decomposed data, wherein each set of decomposed data represents audio data of a predetermined musical timbre, such that a sum of all sets of decomposed data represents audio data substantially equal to the input audio data. 
     
     
         13 . The method of  claim 1 , wherein the input audio data are provided in the form of at least one input track formed by a plurality of audio frames, the input track having an input track length and each audio frame having an audio frame length, and wherein the step of decomposing the input audio data comprises decomposing a plurality of input track segments each having a length smaller than the input track length and larger than the audio frame length. 
     
     
         14 . The method of  claim 13 , wherein decomposing the plurality of input track segments obtains a plurality of decomposed track segments;
 wherein transforming the decomposed data is based on the plurality of decomposed track segments to obtain a plurality of third audio track segments; and   wherein a first segment of the third audio track segments is obtained before a second segment of the input track segments is being decomposed.   
     
     
         15 . The method of  claim 1 , wherein obtaining at least a segment of the third audio data is completed within a processing time of less than about 10 seconds after providing the input audio data or a segment of the input audio data associated with the segment of the third audio data. 
     
     
         16 . The method of  claim 1 , wherein the step of decomposing the input audio data provides first decomposed data representative of the first audio data, and second decomposed data representative of the second audio data; and
 wherein the method further comprises a step of generating an output track which includes a first output track portion, which is obtained by recombining audio data obtained from the first decomposed data with audio data obtained from the second decomposed data or which includes the input audio data, and a second output track portion which is obtained by recombining audio data obtained from the third audio data with audio data obtained from the second decomposed data to obtain recombined audio data.   
     
     
         17 . The method of  claim 16 , wherein the output track includes a transition portion between the first output track portion and the second output track portion, wherein, within the transition portion, in a direction from the first output track portion towards the second output track portion, a first volume level associated to the audio data obtained from the first decomposed data decreases and a second volume level associated to the audio data obtained from the third audio data increases. 
     
     
         18 . The method of  claim 1 , wherein the input audio data are provided as a first input audio data containing a first piece of music; and
 wherein the method further comprises the steps of:
 providing second input audio data containing a second piece of music different from the first piece of music, 
 mixing audio data obtained from the first input audio data with audio data obtained from the second input audio data to obtain mixed audio data, and 
 playback of audio data obtained from the mixed audio data. 
   
     
     
         19 . The method of  claim 18 , wherein the second input audio data contains a mixture of audio data including fourth audio data of a fourth musical timbre and fifth audio data of a fifth musical timbre; and
 wherein the method further comprises the steps of:
 decomposing the first input audio data to provide first decomposed data representative of the first audio data, and second decomposed data representative of the second audio data, and 
 decomposing the second input audio data to provide fourth decomposed data representative of the fourth audio data, and fifth decomposed data representative of the fifth audio data. 
   
     
     
         20 . The method of  claim 18 , wherein the step of transforming the first decomposed data to obtain the third audio data includes changing the musical timbre such that the third musical timbre substantially equals a musical timbre of the second input audio data. 
     
     
         21 . The method of  claim 19 , further including transforming the fourth decomposed data to obtain sixth audio data. 
     
     
         22 . The method of  claim 21 , wherein the third audio data and the sixth audio data are of substantially the same timbre. 
     
     
         23 . The method of  claim 21 , wherein the method further comprises a step of generating an output track which includes a first output track portion obtained by recombining audio data obtained from the third audio data with audio data obtained from the second decomposed data, and a second output track portion obtained by recombining audio data obtained from the sixth audio data with audio data obtained from the fifth decomposed data. 
     
     
         24 . The method of  claim 1 , wherein the input audio data are obtained from mixing a plurality of sets of source audio data including the first audio data and the second audio data; and
 wherein the first audio data are generated by or recorded from a first source selected from a first musical instrument, a first software instrument, a first synthesizer and a first vocalist, and the second audio data are generated by or recorded from a source selected from a second musical instrument, a second software instrument, a second synthesizer and a second vocalist.   
     
     
         25 . A device for processing audio data, comprising
 an input unit configured to receive input audio data containing a mixture of audio data including first audio data of a first musical timbre and second audio data of a second musical timbre different from said first musical timbre;   a decomposition unit for decomposing the input audio data to provide decomposed data representative of the first audio data; and   a transforming unit for transforming the decomposed data to obtain third audio data, wherein the transforming unit includes at least one of:
 i. a timbre changing unit configured to change musical timbre such that the third audio data are of a third musical timbre different from the first musical timbre, or 
 ii. a melody changing unit configured to change melody such that the third audio data represent a melody different from that of the decomposed data. 
   
     
     
         26 . The device of  claim 25 , wherein the third audio data and the first audio data represent musical tones of the same melody, or wherein the third audio data and the first audio data have equal harmony. 
     
     
         27 . The device of  claim 25 , wherein the melody changing unit comprises:
 a harmony detection unit for detecting a time dependent musical harmony of the first audio data; and   a pitch data generating unit for generating pitch data for a plurality of individual musical tones of the third audio data, which are to be played sequentially at respective predetermined points in times to generate a melody of the third audio data.   
     
     
         28 . The device of  claim 27 , further comprising a pitch detection unit for detecting pitch data indicating a musical pitch of the decomposed data or the first audio data. 
     
     
         29 . The device of  claim 28 , further comprising a data conversion unit for converting the decomposed data into event message data formed by a plurality of event messages of musical tones, wherein each event message at least specifies pitch data and velocity data of a corresponding musical tone. 
     
     
         30 . The device of  claim 29 , further comprising at least one of a synthesizer unit for synthesizer-based generation of audio signals based on the pitch data or the event message data, or a sample player for sample-based generation of audio signals based on the pitch data or the event message data. 
     
     
         31 . The device of  claim 29 , wherein at least one of the transforming unit, the timbre changing unit, the melody changing unit, the harmony detection unit, the pitch detection unit, the pitch data generating unit, and the data conversion unit comprises an artificial intelligence system. 
     
     
         32 . The device of  claim 25 , wherein the decomposition unit comprises an artificial intelligence system. 
     
     
         33 . The device of  claim 25 , wherein the decomposition unit is configured to decompose the input audio data to provide first decomposed data representative of the first audio data, and second decomposed data representative of the second audio data; and
 wherein the device further comprises a recombination unit for recombining audio data obtained from the third audio data with audio data obtained from the second decomposed data to obtain recombined audio data.   
     
     
         34 . The device of  claim 25 , wherein the decomposition unit is configured to decompose the input audio data to provide a plurality of sets of decomposed data, wherein each set of decomposed data represents audio data of a predetermined musical timbre, such that a sum of all sets of decomposed data represents audio data substantially equal to the input audio data. 
     
     
         35 . The device of  claim 25 , wherein the input audio data contain an input track formed by plurality of audio frames, the input track having an input track length and each audio frame having an audio frame length, and wherein the decomposition unit is adapted to decompose the input audio data by decomposing a plurality of input track segments each having a length smaller than the input track length and larger than the audio frame length. 
     
     
         36 . The device of  claim 35 , wherein the decomposition unit is configured to decompose the plurality of input track segments to obtain a plurality of decomposed track segments;
 wherein the transforming unit is configured to generate the third audio data based on the plurality of decomposed track segments to obtain a plurality of third audio track segments; and   wherein the device is configured to obtain a first segment of the third audio track segments before a second segment of the input track segments is being decomposed.   
     
     
         37 . The device of  claim 36 , wherein the device is configured such that obtaining at least a segment of the third audio data is completed within a processing time of less than about 10 seconds after providing the input audio data or a segment of the input audio data associated to the segment of the third audio data. 
     
     
         38 . The device of  claim 25 , wherein the decomposition unit is configured to decompose the input audio data to provide first decomposed data representative of the first audio data, and second decomposed data representative of the second audio data;
 wherein the device further comprises an output unit generating an output track which includes a first output track portion, which is obtained by recombining audio data obtained from the first decomposed data with audio data obtained from the second decomposed data or which substantially includes the input audio data, and a second output track portion obtained by recombining audio data obtained from the third audio data with audio data obtained from the second decomposed data; and   wherein the device further comprises a user control unit for receiving a user control input determining a starting point or an end point of the first output track portion or the second output track portion.   
     
     
         39 . The device of  claim 38 , wherein the user control unit includes a crossfader for setting a ratio between a first volume level associated to the audio data obtained from the first decomposed data and a second volume level associated to the audio data obtained from the third audio data. 
     
     
         40 . The device of  claim 25 , wherein the input unit has a first input section for receiving first input audio data containing a first piece of music, and a second input section for receiving second input audio data containing a second piece of music different from the first piece of music; and
 wherein the device further comprises:
 a mixing unit configured for mixing audio data obtained from the first input audio data with audio data obtained from the second input audio data to obtain mixed audio data, and 
 a playback unit for playing audio data obtained from the mixed audio data.