Temporal Alignment of Captured Inertial Measurement Unit Trajectory and 2D Video For Golf Swing Analysis

Abstract

A method for temporal alignment of golf club inertial measurement data and two-dimensional video for golf club swing analysis is provided. The method includes capturing inertial measurement data of a golf club swing on an inertial measurement unit (IMU), and sending the inertial measurement data of the golf club swing from the inertial measurement unit to a computing device. The computing device is configured to calculate a time bias between the inertial measurement data of the golf club swing and a two-dimensional video of the golf club swing, based on the computing device detecting an impact frame in the two-dimensional video and the computing device aligning the detected impact frame to an impact time from the captured inertial measurement data of the golf club swing.

What is claimed is: 
     
         1 . A method for temporal alignment of golf club inertial measurement data and two-dimensional video for golf club swing analysis, comprising:
 capturing inertial measurement data of a golf club swing on an inertial measurement unit (IMU); and   sending the inertial measurement data of the golf club swing from the inertial measurement unit to a computing device, so that the computing device calculates a time bias between the inertial measurement data of the golf club swing and a two-dimensional video of the golf club swing, based on the computing device detecting an impact frame in the two-dimensional video and the computing device aligning the detected impact frame to an impact time from the captured inertial measurement data of the golf club swing.   
     
     
         2 . The method of  claim 1 , wherein the computing device detecting the impact frame is based on the computing device detecting a location of a golf ball in a frame in the two-dimensional video. 
     
     
         3 . The method of  claim 1 , wherein the time bias includes a time difference between a first frame of the two-dimensional video and a first frame of the inertial measurement data of the golf club swing. 
     
     
         4 . The method of  claim 1 , wherein the computing device overlays a projected golf club swing trajectory, based on the inertial measurement data of the golf club swing, onto the two-dimensional video of the golf club swing, aligned using the time bias. 
     
     
         5 . The method of  claim 1 , wherein:
 the sending the inertial measurement data of the golf club swing to the computing device is through a wireless connection; and   the computing device receives the two-dimensional video of the golf club swing from a mobile device through a further wireless connection.   
     
     
         6 . A method for temporal alignment of golf club inertial measurement data and two-dimensional video for golf club swing analysis, performed by a computing device, comprising:
 receiving captured inertial measurement data of a golf club swing from an inertial measurement unit (IMU) attached to a golf club;   receiving, from a device having a video camera, a two-dimensional video of the golf club swing;   detecting an impact frame showing an object impacted by the golf club in the two-dimensional video; and   calculating a time bias based on aligning the detected impact frame from the two-dimensional video and an impact time from the captured inertial measurement data of the golf club swing.   
     
     
         7 . The method of  claim 6 , further comprising:
 refining the time bias, by minimizing spatial alignment error of the captured inertial measurement data of the golf club swing and the two-dimensional video of the golf club swing.   
     
     
         8 . The method of  claim 6 , further comprising:
 detecting a location of the object in a frame in the two-dimensional video; and   searching backwards, relative to a time span of the two-dimensional video, through frames of the two-dimensional video for a frame in which the object appears at the detected location, to detect the impact frame.   
     
     
         9 . The method of  claim 6 , further comprising:
 determining a background image from a plurality of frames of the two-dimensional video;   subtracting the background image from a further plurality of frames of the two-dimensional video; and   analyzing intensity at a specific location in the further plurality of frames having the background image subtracted, the specific location corresponding to detection of the object in a still further plurality of frames of the two-dimensional video, wherein the detecting the impact frame is based on the analyzing the intensity.   
     
     
         10 . The method of  claim 6 , further comprising:
 verifying the time bias through analysis of intensity at a determined object location in frames of the two-dimensional video, to detect false impact frames in the two-dimensional video.   
     
     
         11 . The method of  claim 6 , further comprising:
 overlaying a projected golf club swing trajectory onto the two-dimensional video, based on temporal alignment of the inertial measurement data of the golf club swing and the two-dimensional video, using the time bias.   
     
     
         12 . The method of  claim 6 , wherein the device having the video camera is a mobile device, and further comprising:
 sending a video having a projected golf club trajectory aligned using the time bias and overlaid on the two-dimensional video of the golf club swing, to the mobile device for display on the mobile device.   
     
     
         13 . A tangible, non-transitory, computer-readable media having instructions thereupon which, when executed by a processor, cause the processor to perform a method comprising:
 receiving, from an inertial measurement unit (IMU), inertial measurement data of a golf club swing;   receiving a two-dimensional video of the golf club swing;   determining a location of an object in at least one frame in the two-dimensional video;   determining an impact frame, by searching backwards from a later frame to an earlier frame in the two-dimensional video for a frame in which brightness at the determined location relative to the frame exceeds a threshold;   aligning the detected impact frame from the two-dimensional video to an impact time from the inertial measurement data of the golf club swing;   determining a time bias between the frames of the two-dimensional video and frames of the inertial measurement data of the golf club swing based on the aligning; and   overlaying a projected golf club swing trajectory onto the two-dimensional video of the golf club swing, based on temporal alignment, with the time bias, of the inertial measurement data of the golf club swing and the two-dimensional video.   
     
     
         14 . The computer-readable media of  claim 13 , wherein the receiving or capturing a two-dimensional video comprises:
 receiving the two-dimensional video from a mobile device via a wireless connection.   
     
     
         15 . The computer-readable media of  claim 13 , wherein the method further comprises:
 performing spatial alignment of the inertial measurement data of the golf club swing and the two-dimensional video of the golf club swing, with the time bias for an initial temporal alignment;   shifting temporal alignment of the inertial measurement data of the golf club swing and the two-dimensional video forwards or backwards relative to the initial temporal alignment;   determining spatial alignment error for each adjustment in the temporal alignment; and   determining a temporal alignment having a minimum spatial alignment error, as a refined time bias.   
     
     
         16 . The computer-readable media of  claim 13 , wherein the method further comprises:
 determining a maximum intensity at the location of the object in frames of the two-dimensional video;   determining a minimum intensity at the location in frames of the two-dimensional video, wherein the minimum intensity occurs in a frame with the object absent from the frame; and   determining the threshold between the maximum intensity and the minimum intensity.   
     
     
         17 . The computer-readable media of  claim 13 , wherein the method further comprises:
 determining a temporal pattern in frames of the two-dimensional video, with intensity at the location in frames before the impact frame being greater than intensity at the location in frames after the impact frame; and   verifying selection of the impact frame, based on the temporal pattern.   
     
     
         18 . The computer-readable media of  claim 13 , wherein the method further comprises:
 determining a set of frames of the two-dimensional video in which the object does not appear;   determining an average background from the set of frames; and   forming a set of background subtracted images, based on subtracting the average background from frames of the two-dimensional video, wherein the determining the impact frame is based on the set of background subtracted images.   
     
     
         19 . The computer-readable media of  claim 13 , wherein the method further comprises:
 sending a video having the projected golf swing trajectory aligned, based on the time bias, and overlaid on the two-dimensional video of the golf club swing, to a mobile device for display.   
     
     
         20 . The computer-readable media of  claim 13 , wherein the method further comprises:
 analyzing intensity, at the location of the object determined in the at least one frame, in frames of the two-dimensional video; and   determining whether the impact frame is a false impact frame, based on the analyzing the intensity.