Systems and methods for assisting a user in practicing a musical instrument

Abstract

The present disclosure is related to systems and methods for assisting a user in practicing a smart musical instrument. The method includes obtaining a real-time image of the user playing the smart musical instrument. The method also includes obtaining information associated with a current key being pressed and information associated with a plurality of reference fingerings from the smart musical instrument. The method also includes displaying the real-time image in real-time. The method also includes determining whether a current fingering associated with the current key being pressed matches with a reference fingering, of the plurality of reference fingerings, associated with the current key being pressed. In response to a determination that the current fingering associated with the current key being pressed does not match the reference fingering, the method further includes generating a reminder. The method still further includes displaying the reference fingering corresponding to the current fingering.

1 . An augmented reality (AR) based method, for assisting a user in practicing a smart musical instrument, implemented on a smart device, the method comprising:
 obtaining a real-time image of the user playing the smart musical instrument;   obtaining information associated with a current key being pressed and information associated with a plurality of reference fingerings;   determining, based on the real-time image, the information associated with the current key being pressed, and the information associated with the plurality of reference fingerings, whether a current fingering associated with the current key being pressed matches with a reference fingering, of the plurality of reference fingerings, associated with the current key being pressed;   in response to a determination that the current fingering associated with the current key being pressed does not match the reference fingering,   generating a reminder.   
     
     
         2 . The method of  claim 1 , further comprises:
 obtaining, from the smart musical instrument, a fingering prompt corresponding to each of the plurality of reference fingerings in real time; and displaying the fingering prompt for a next fingering corresponding to the current fingering.   
     
     
         3 . The method of  claim 1 , further comprises:
 in response to the determination that the current fingering associated with the current key being pressed does not match the reference fingering,   the generating a reminder is done via at least one picture of a virtual instrument keyboard, a fingering, a hand posture, or a text.   
     
     
         4 . The method of  claim 1 , further comprises:
 generating a reminder via at least one of a picture of a virtual instrument keyboard, a fingering, a hand posture, or a text relating to the reference fingering associated with the current key being pressed.   
     
     
         5 - 9 . (canceled) 
     
     
         10 . A smart device, comprising:
 at least one computer-readable storage medium storing a set of instructions; and   at least one processor configured to communicate with the at least one computer-readable storage medium, wherein when executing the set of instructions, the at least one processor is directed to perform a method including:   obtaining a real-time image of a user playing a smart musical instrument;   obtaining information associated with a current key being pressed and information associated with a plurality of reference fingerings;   determining, based on the real-time image, the information associated with the current key being pressed, and the information associated with the plurality of reference fingerings, whether a current fingering associated with the current key being pressed matches with a reference fingering, of the plurality of reference fingerings, associated with the current key being pressed;   in response to a determination that the current fingering associated with the current key being pressed does not match the reference fingering,   generating a reminder.   
     
     
         11 - 28 . (canceled) 
     
     
         29 . The method of  claim 1 , further comprises:
 displaying the real-time image in real-time.   
     
     
         30 . The method of  claim 1 , wherein:
 the reminder includes an error corresponding to the current fingering associated with the current key being pressed.   
     
     
         31 . The method of  claim 1 , wherein:
 in response to the determination that the current fingering associated with the current key being pressed does not match the reference fingering, displaying the reference fingering corresponding to the current fingering.   
     
     
         32 . The method of  claim 1 , wherein the obtaining information associated with a current key being pressed includes:
 identifying the current key being pressed from the image data.   
     
     
         33 . The method of  claim 1 , wherein the obtaining information associated with a current key being pressed includes:
 obtaining first performance data of the user from the smart musical instrument, wherein the first performance data reflects an operation of the smart musical instrument by the user and the first performance data is associated with a component of the smart musical instrument; and   determining the current key being pressed from the first performance data.   
     
     
         34 . The method of  claim 33 , wherein the component of the smart musical instrument includes at least one of a plurality of keys or a pedal, and the first performance data includes at least one of:
 an identity of the current key being pressed,   a time of the current key being pressed,   a time of the pedal being stepped on,   a musical note produced by the current key being pressed, and   a musical note produced by the pedal being stepped on.   
     
     
         35 . The method of  claim 33 , wherein the first performance data is acquired by one or more sensors installed in one or more components of the smart musical instrument. 
     
     
         36 . The method of  claim 1 , wherein the obtaining information associated with a plurality of reference fingerings includes:
 obtaining a specific music score designated by the user via a user interface on a smart device; and   retrieving the plurality of reference fingerings corresponding to the specific music score.   
     
     
         37 . The method of  claim 1 , further comprising:
 determining whether the current key being pressed matches with a reference key that should be pressed.   
     
     
         38 . The method of  claim 1 , further comprising:
 determining whether a current hand posture matches with a reference hand posture.   
     
     
         39 . The method of  claim 1 , further comprising:
 displaying the reminder or the reference fingering via a smart AR device.   
     
     
         40 . The method of  claim 39 , wherein the displaying the reminder or the reference fingering via a smart AR device includes:
 causing the smart AR device to display the reference fingering associated with the current key being pressed on a picture of a virtual instrument keyboard.   
     
     
         41 . The method of  claim 1 , wherein the determining whether a current fingering associated with the current key being pressed matches with a reference fingering, of the plurality of reference fingerings, associated with the current key being pressed includes:
 identifying the current fingering associated with the current key being pressed from the real-time image;   determining a similarity between the current fingering and the reference fingering corresponding to the current key; and   determining whether the current fingering associated with the current key being pressed matches with the reference fingering based on the similarity.   
     
     
         42 . The smart device of  claim 1 , wherein the method further includes:
 obtaining, from the smart musical instrument, a fingering prompt corresponding to each of the plurality of reference fingerings in real time; and   displaying the fingering prompt for a next fingering corresponding to the current fingering.   
     
     
         43 . A non-transitory computer readable medium, comprising at least one set of instructions for assisting a user in practicing a smart musical instrument, wherein when executed by one or more processors of a computing device, the at least one set of instructions causes the computing device to perform a method, the method comprising:
 obtaining a real-time image of a user playing a smart musical instrument;   obtaining information associated with a current key being pressed and information associated with a plurality of reference fingerings;   determining, based on the real-time image, the information associated with the current key being pressed, and the information associated with the plurality of reference fingerings, whether a current fingering associated with the current key being pressed matches with a reference fingering, of the plurality of reference fingerings, associated with the current key being pressed;   in response to a determination that the current fingering associated with the current key being pressed does not match the reference fingering, generating a reminder.