System and method for generating trackable video frames from broadcast video

Abstract

A system and method of generating trackable frames from a broadcast video feed are provided herein. A computing system retrieves a broadcast video feed for a sporting event. The broadcast video feed includes a plurality of video frames. The computing system generates a set of frames for classification using a principal component analysis model. The set of frames are a subset of the plurality of video frames. The computing system partitions each frame of the set of frames into a plurality of clusters. The computing system classifies each frame of the plurality of frames as trackable or untrackable. Trackable frames capture a unified view of the sporting event. The computing system compares each cluster to a predetermined threshold to determine whether each cluster comprises at least a threshold number of trackable frames. The computing system classifies each cluster that includes at least the threshold number of trackable frames as trackable.

What is claimed: 
     
       1. A method of calibrating a camera, comprising:
 identifying, by a computing system, a broadcast video feed for a sporting event, the broadcast video feed comprising a plurality of frames captured by a camera; 
 classifying, by a neural network of the computing system, each frame of the plurality of frames as trackable or untrackable, wherein trackable frames capture a unified view of the sporting event; 
 determining, by the computing system, a motion of the camera between successive trackable frames by:
 identifying objects contained in the trackable frames, 
 removing the objects from the trackable frames, and 
 determining a flow from a first trackable frame to a second trackable frame of the trackable frames; and 
 
 based on the determining, calibrating, by the computing system, the camera. 
 
     
     
       2. The method of  claim 1 , wherein classifying, by the computing system, each frame of the plurality of frames as trackable or untrackable comprises:
 training the neural network to classify a video frame as trackable or untrackable using a training set comprising a plurality of training video frames and a label associated with each training video frame, wherein the label is trackable or untrackable. 
 
     
     
       3. The method of  claim 2 , wherein each training video frame of the plurality of training video frames comprises a trackable/untrackable classification and an associated cluster number. 
     
     
       4. The method of  claim 1 , wherein determining the flow from the first trackable frame to the second trackable frame of the trackable frames comprises:
 identifying a flow field from the first trackable frame to the second trackable frame. 
 
     
     
       5. The method of  claim 4 , further comprising:
 generating a homography matrix for the first trackable frame and/or the second trackable frame using the flow field. 
 
     
     
       6. The method of  claim 1 , wherein removing the objects from the trackable frames comprises:
 detecting a first player of a plurality of players in the first trackable frame; and 
 using body post information for the first player to remove the first player from the first trackable frame. 
 
     
     
       7. The method of  claim 1 , further comprising:
 matching, by the computing system, the first trackable frame to a first playing surface template representing a first camera perspective of a playing surface; and 
 matching, by the computing system, the second trackable frame to a second playing surface template representing a second camera perspective of the playing surface. 
 
     
     
       8. A non-transitory computer readable medium including one or more sequences of instructions that, when executed by one or more processors, causes a computing system perform one or more operations comprising:
 identifying, by the computing system, a broadcast video feed for a sporting event, the broadcast video feed comprising a plurality of frames captured by a camera; 
 classifying, by a neural network of the computing system, each frame of the plurality of frames as trackable or untrackable, wherein trackable frames capture a unified view of the sporting event; 
 determining, by the computing system, a motion of the camera between successive trackable frames by:
 identifying objects contained in the trackable frames, 
 removing the objects from the trackable frames, and 
 determining a flow from a first trackable frame to a second trackable frame of the trackable frames; and 
 
 based on the determining, calibrating, by the computing system, the camera. 
 
     
     
       9. The non-transitory computer readable medium of  claim 8 , wherein classifying, by the computing system, each frame of the plurality of frames as trackable or untrackable comprises:
 training the neural network to classify a video frame as trackable or untrackable using a training set comprising a plurality of training video frames and a label associated with each training video frame, wherein the label is trackable or untrackable. 
 
     
     
       10. The non-transitory computer readable medium of  claim 9 , wherein each training video frame of the plurality of training video frames comprises a trackable/untrackable classification and an associated cluster number. 
     
     
       11. The non-transitory computer readable medium of  claim 8 , wherein determining the flow from the first trackable frame to the second trackable frame of the trackable frames comprises:
 identifying a flow field from the first trackable frame to the second trackable frame. 
 
     
     
       12. The non-transitory computer readable medium of  claim 11 , further comprising:
 generating a homography matrix for the first trackable frame and/or the second trackable frame using the flow field. 
 
     
     
       13. The non-transitory computer readable medium of  claim 8 , wherein removing the objects from the trackable frames comprises:
 detecting a first player of a plurality of players in the first trackable frame; and 
 using body post information for the first player to remove the first player from the first trackable frame. 
 
     
     
       14. The non-transitory computer readable medium of  claim 8 , further comprising:
 matching, by the computing system, the first trackable frame to a first playing surface template representing a first camera perspective of a playing surface; and 
 matching, by the computing system, the second trackable frame to a second playing surface template representing a second camera perspective of the playing surface. 
 
     
     
       15. A system, comprising:
 a processor; and 
 a memory having programming instructions stored thereon, which, when executed by the processor, causes the system to perform operations comprising:
 identifying a broadcast video feed for a sporting event, the broadcast video feed comprising a plurality of frames captured by a camera; 
 classifying, by a neural network, each frame of the plurality of frames as trackable or untrackable, wherein trackable frames capture a unified view of the sporting event; 
 determining a motion of the camera between successive trackable frames by:
 identifying objects contained in the trackable frames, 
 removing the objects from the trackable frames, and 
 determining a flow from a first trackable frame to a second trackable frame of the trackable frames; and 
 
 based on the determining, calibrating the camera. 
 
 
     
     
       16. The system of  claim 15 , wherein classifying each frame of the plurality of frames as trackable or untrackable comprises:
 training the neural network to classify a video frame as trackable or untrackable using a training set comprising a plurality of training video frames and a label associated with each training video frame, wherein the label is trackable or untrackable. 
 
     
     
       17. The system of  claim 15 , wherein determining the flow from the first trackable frame to the second trackable frame of the trackable frames comprises:
 identifying a flow field from the first trackable frame to the second trackable frame. 
 
     
     
       18. The system of  claim 17 , wherein the operations further comprise:
 generating a homography matrix for the first trackable frame and/or the second trackable frame using the flow field. 
 
     
     
       19. The system of  claim 15 , wherein removing the objects from the trackable frames comprises:
 detecting a first player of a plurality of players in the first trackable frame; and 
 using body post information for the first player to remove the first player from the first trackable frame. 
 
     
     
       20. The system of  claim 15 , wherein the operations further comprise:
 matching the first trackable frame to a first playing surface template representing a first camera perspective of a playing surface; and 
 matching the second trackable frame to a second playing surface template representing a second camera perspective of the playing surface.