Methods and systems to track a moving objects trajectory using a single camera

Abstract

Systems and methods are described for generating a three-dimensional track a ball in a gaming environment from a single camera. In some examples, an input video including frames of a ball moving in a gaming environment recorded by a camera may be obtained, along with a camera projection matrix associated with at least one frame that maps a two-dimensional pixel space representation to a three-dimensional representation of the gaming environment. Candidate two-dimensional image locations of the ball across the plurality of frames may be identified using a neural network or a computer vision algorithm. An optimization algorithm may be performed that uses a 3D ball physics model, the camera projection matrix and a subset of the candidate two-dimensional image locations of the ball to generate a three-dimensional track of the ball in the gaming environment. The three-dimensional track of the ball may then be provided to a user device.

1 - 20 . (canceled) 
     
     
         21 . A computer-implemented method for determining a track of a ball moving in a gaming environment, the method comprising:
 obtaining an input video of the gaming environment recorded by a single camera of a user device, the input video comprising a plurality of frames of a ball being thrown;   generate a plurality of camera projection matrices, wherein individual camera projection matrices of the plurality of camera projection matrices are associated with individual fames of the plurality of frames of the input video and the gaming environment, wherein generating an individual camera projection matrix comprises identifying at least two points in an individual frame that have a fixed dimensional relationship with each other and correlating the at least two points between a two-dimensional pixel space representation and a three-dimensional physical representation;   identifying candidate two-dimensional image locations of the ball across the plurality of frames of the input video of the gaming environment using a neural network trained on past video inputs of the gaming environment;   performing an inlier detection algorithm to select a subset of the candidate two-dimensional image locations of the ball from the candidate two-dimensional image locations of the ball that fit a curve;   using the plurality of camera projection matrices and a subset of the candidate two-dimensional image locations of the ball forming the curve to estimate three-dimensional positions of the ball moving in the gaming environment; and   providing metrics of movement of the ball based on the three dimensional positions of the ball to the user device.   
     
     
         22 . The computer-implemented method of  claim 21 , wherein metrics comprise at least one of a maximum speed of the ball, spin of the ball, swing of the ball, or at least one height of the ball. 
     
     
         23 . The computer-implemented method of  claim 21 , wherein performing the inlier detection algorithm to select the subset of the candidate two-dimensional image locations of the ball from the candidate two-dimensional image locations of the ball that fit the curve further comprises:
 selecting the subset of the candidate two-dimensional image locations of the ball from the candidate two-dimensional image locations of the ball by removing erroneous candidate locations of the ball that are not within a threshold distance from the curve, wherein the curve is determined by fitting a plurality of randomly selected subsets of the candidate two-dimensional image locations of the ball.   
     
     
         24 . The computer-implemented method of  claim 21 , wherein the gaming environment comprises a cricket gaming environment, and wherein the at least two points correlate to at least one of a position of one or more stumps, a location of a bat contacting a ball, or a location of the ball bouncing on the ground. 
     
     
         25 . The computer-implemented method of  claim 21 , wherein the gaming environment comprises a baseball gaming environment, and wherein the at least two points correlate to at least one of a position relative to a pitcher's mound, a location of a bat contacting a ball, or a glove contacting the ball. 
     
     
         26 . A computer-implemented method, comprising:
 obtaining an input video of a ball moving in a gaming environment recorded by a camera, the input video comprising a plurality of frames;   obtaining a camera projection matrix associated with at least one frame of the plurality of frames of the input video and the gaming environment, the camera projection matrix mapping a two-dimensional pixel space representation to a three-dimensional representation of the gaming environment;   identifying candidate two-dimensional image locations of the ball across the plurality of frames of the input video of the gaming environment using at least one of a neural network or a computer vision algorithm;   using the camera projection matrix and at least a subset of the candidate two-dimensional image locations of the ball to generate three-dimensional locations of the ball in the gaming environment; and   provide metrics of movement of the ball in the gaming environment based on the three-dimensional locations of the ball to a user device.   
     
     
         27 . The computer-implemented method of  claim 26 , further comprising:
 selecting the subset of the candidate two-dimensional image locations of the ball from the candidate two-dimensional image locations of the ball by removing erroneous candidate locations of the ball.   
     
     
         28 . The computer-implemented method of  claim 26 , wherein obtaining the camera projection matrix further comprises:
 generating a plurality of camera projection matrices, wherein individual camera projection matrices of the plurality of camera projection matrices are associated with individual key frames of the plurality of frames of the input video and the gaming environment, wherein generating an individual camera projection matrix comprises identifying at least two points in an individual key frame that have a fixed dimensional relationship with each other and correlating the at least two points between a two-dimensional pixel space representation and a three-dimensional physical representation;   
     
     
         29 . The computer-implemented method of  claim 26 , wherein the gaming environment comprises a cricket gaming environment, a baseball gaming environment, a football gaming environment, a soccer gaming environment, or a basketball gaming environment. 
     
     
         30 . The computer-implemented method of  claim 26 , further comprising applying one or more constraints to movement of the ball to generate at least a subset of the three-dimensional locations of the ball in the gaming environment. 
     
     
         31 . The computer-implemented method of  claim 30 , wherein the one or more constraints further comprises at least one of:
 a location in a frame of the plurality of frames of where the ball is pitched from;   a sound captured by the video input that correlates to a specific action or event in the gaming environment;   deviation in the horizontal or vertical direction of the ball being less than a threshold for a specific gaming environment;   a speed of the ball being between a minimum speed and a maximum speed; or   a position of one or more of the players in the gaming environment.   
     
     
         32 . The computer-implemented method of  claim 26 , further comprising:
 providing the metrics overlaid onto a representation of movement of the ball in the gaming environment to a graphical user interface of the user device.   
     
     
         33 . The computer-implemented method of  claim 26 , wherein metrics comprise at least one of a maximum speed of the ball, spin of the ball, swing of the ball, or at least one height of the ball. 
     
     
         34 . The computer-implemented method of  claim 26 , wherein the gaming environment comprises a cricket gaming environment, and wherein the at least two points correlate to at least one of a position of one or more stumps, a location of a bat contacting a ball, or a location of the ball bouncing on the ground. 
     
     
         35 . The computer-implemented method of  claim 21 , wherein the gaming environment comprises a baseball gaming environment, and wherein the at least two points correlate to at least one of a position relative to a pitcher's mound, a location of a bat contacting a ball, or a glove contacting the ball. 
     
     
         36 . A method for determining a two-dimensional track of a ball moving in a gaming environment, the method comprising:
 obtaining an input video of the gaming environment recorded by a single camera of a user device, the input video comprising a plurality of frames of a ball being thrown;   generate a plurality of camera projection matrices, wherein individual camera projection matrices of the plurality of camera projection matrices are associated with individual fames of the plurality of frames of the input video and the gaming environment, wherein generating the camera projection matrices comprises identifying at least two points in an individual frame that have a fixed dimensional relationship with each other and correlating the at least two points between a two-dimensional pixel space representation and a three-dimensional physical representation;   identifying candidate two-dimensional image locations of the ball across the plurality of frames of the input video of the gaming environment using a neural network trained on past video inputs of the gaming environment;   performing an inlier detection algorithm on the candidate two-dimensional image locations to select a subset of the candidate two-dimensional image locations of the ball that are fit to a curve from the candidate two-dimensional image locations of the ball;   generating three-dimension ball locations from the subset of the candidate two-dimensional image locations of the ball forming the curve using at least one of the plurality of camera projection matrices based on the at least two points; and   providing metrics of movement of the ball based on the three-dimensional ball locations.   
     
     
         37 . The method of  claim 36 , wherein performing the inlier detection algorithm on the candidate two-dimensional image locations to select the subset of the candidate two-dimensional image locations of the ball that are fit to the curve further comprises using a 2D tracking algorithm. 
     
     
         38 . The method of  claim 36 , wherein the gaming environment comprises a cricket gaming environment, and wherein the at least two points correlate to at least one of a position of one or more stumps, a location of a bat contacting a ball, or a location of the ball bouncing on the ground. 
     
     
         39 . The method of  claim 36 , wherein the gaming environment comprises a baseball gaming environment, and wherein the at least two points correlate to at least one of a position relative to a pitcher's mound, a location of a bat contacting a ball, or a glove contacting the ball. 
     
     
         40 . The method of  claim 36 , wherein metrics comprise at least one of a maximum speed of the ball, spin of the ball, swing of the ball, or at least one height of the ball.