Methods and systems for generating sports analytics with a mobile device

Abstract

Methods and systems for real-time generation of ball shot analytics are disclosed. The methods and systems perform steps for ball and posture detection, ball and posture flow generation, shot event identification and classification, and shot analytics generation based on identified shot events and shooter posture flow. Embodiments of the present invention use computer vision techniques to enable a resource-limited mobile device such as a smartphone to conduct the aforementioned steps. Also disclosed are benefits of the new methods, and alternative embodiments of implementation.

What is claimed is: 
     
       1. A method for generating ball shot analytics, comprising:
 receiving an input video of a ball game from a camera on a single mobile computing device, and an input location of a shooter in a shooter identification frame of the input video; 
 detecting, using the single mobile computing device, a plurality of balls and a plurality of player postures from frames of the input video using one or more Convolutional Neural Network (CNN) modules; 
 generating, using the single mobile computing device, by grouping the detected plurality of balls and the detected plurality of player postures along a time line, one or more ball flows and one or more player posture flows, wherein each ball flow is a time-sequence of a ball's positions in the input video, and wherein each player posture flow is a time-sequence of a player's postures in the input video; 
 identifying, using the single mobile computing device, a generated player posture flow as a shooter posture flow, based on the input location of the shooter; 
 identifying, using the single mobile computing device, a generated ball flow as a related ball flow, wherein the related ball flow is related to the shooter posture flow; 
 determining, using the single mobile computing device and without user input, a ball-from-shooter time based on the related ball flow and the shooter posture flow; 
 determining, using the single mobile computing device and without user input, a shot event occurring before the ball-from-shooter time based on the shooter posture flow; and 
 generating, using the single mobile computing device, one or more shot analytics based on the shot event, the shooter posture flow, and the related ball flow. 
 
     
     
       2. The method of  claim 1 , wherein the generating of the one or more shot analytics comprises determining a shot type of the shot event. 
     
     
       3. The method of  claim 1 , wherein the ball game is a basketball game. 
     
     
       4. The method of  claim 1 , wherein the input video is streamed. 
     
     
       5. The method of  claim 1 , wherein the detecting of the plurality of balls and the plurality of player postures is applied on a skip frame basis. 
     
     
       6. The method of  claim 1 , wherein the detecting of the plurality of balls and the plurality of player postures is limited to an image area close to the shooter. 
     
     
       7. The method of  claim 1 , wherein the generating of the one or more ball flows and the one or more player posture flows comprises applying bipartite matching to the detected plurality of balls and the detected plurality of player postures, respectively, to existing ball flows and player posture flows, wherein the bipartite matching of a detected ball and an existing ball flow comprises computing a matching score between the detected ball and the existing ball flow, and wherein the computing of the matching score comprises:
 generating a predicted ball comprising a next ball location and a next ball size based on the existing ball flow; and 
 computing the matching score based on a location difference and a size difference between the predicted ball and the detected ball. 
 
     
     
       8. The method of  claim 1 , wherein the shooter posture flow is closest to the input location of the shooter in the shooter identification frame when compared to other player posture flows. 
     
     
       9. The method of  claim 1 ,
 wherein identifying the related ball flow comprises applying non-max-suppression to all generated ball flows, 
 wherein the related ball flow has a score against the shooter posture flow, 
 wherein the score is computed based on shooter movements, a distance to the shooter, and a confidence value, and 
 wherein the score is above a pre-defined threshold. 
 
     
     
       10. The method of  claim 1 , further comprising detecting a shot attempt within the shooter posture flow, by determining that the related ball flow is thrown from the shooter's upper body upward at a time instant within the shooter posture flow, wherein the determining of the ball-from-shooter time is by backtracking the related ball flow, in reverse time, from the shot attempt. 
     
     
       11. The method of  claim 1 , wherein the shot event occurs within a pre-defined time period before the ball-from-shooter time. 
     
     
       12. The method of  claim 1 , wherein the shot event is selected from the group consisting of dribble event, jump event, catch-ball event, ball-leave-hand event, and one-two leg jump, and wherein a shot type is selected from the group consisting of layup, regular shot, dribble-pull-up, off-the-move, and catch-and-shoot. 
     
     
       13. The method of  claim 1 , wherein the shot analytics is selected from the group consisting of release time, back angle, leg bend ratio, leg power, moving speed, moving direction, and height of jump. 
     
     
       14. The method of  claim 1 , wherein each CNN module has been trained using one or more prior input videos or one or more images. 
     
     
       15. A system for generating ball shot analytics, comprising:
 at least one processor on a single mobile computing device; and 
 a non-transitory physical medium for storing program code and accessible by the processor, the program code when executed by the processor causes the processor to:
 receive an input video of a ball game from a camera on the single mobile computing device, and an input location of a shooter in a shooter identification frame of the input video; 
 detect, using the single mobile computing device, a plurality of balls and a plurality of player postures from frames of the input video using one or more Convolutional Neural Network (CNN) modules; 
 generate, using the single mobile computing device, by grouping the detected plurality of balls and the detected plurality of player postures along a time line, one or more ball flows and one or more player posture flows, wherein each ball flow is a time-sequence of a ball's positions in the input video, and wherein each player posture flow is a time-sequence of a player's postures in the input video; 
 identify, using the single mobile computing device, a generated player posture flow as a shooter posture flow, based on the input location of the shooter; 
 identify, using the single mobile computing device, a generated ball flow as a related ball flow, wherein the related ball flow is related to the shooter posture flow; 
 determine, using the single mobile computing device and without user input, a ball-from-shooter time, based on the related ball flow and the shooter posture flow; 
 determine, using the single mobile computing device and without user input, a shot event occurring before the ball-from-shooter time, based on the shooter posture flow; and 
 generate, using the single mobile computing device, one or more shot analytics based on the shot event, the shooter posture flow, and the related ball flow. 
 
 
     
     
       16. A non-transitory physical medium for generating ball shot analytics, the non-transitory physical medium comprising program code stored thereon, the program code when executed by a processor causes the processor to:
 receive an input video of a ball game from a camera on a single mobile computing device, and an input location of a shooter in a shooter identification frame of the input video; 
 detect, using the single mobile computing device, a plurality of balls and a plurality of player postures from frames of the input video using one or more Convolutional Neural Network (CNN) modules; 
 generate, using the single mobile computing device, by grouping the detected plurality of balls and the detected plurality of player postures along a time line, one or more ball flows and one or more player posture flows, wherein each ball flow is a time-sequence of a ball's positions in the input video, and wherein each player posture flow is a time-sequence of a player's postures in the input video; 
 identify, using the single mobile computing device, a generated player posture flow as a shooter posture flow, based on the input location of the shooter; 
 identify using the single mobile computing device, a generated ball flow as a related ball flow, wherein the related ball flow is related to the shooter posture flow; 
 determine, using the single mobile computing device and without user input, a ball-from-shooter time, based on the related ball flow and the shooter posture flow; 
 determine, using the single mobile computing device and without user input, a shot event occurring before the ball-from-shooter time, based on the shooter posture flow; and 
 generate, using the single mobile computing device, one or more shot analytics based on the shot event, the shooter posture flow, and the related ball flow.