Image processing method and device for instant replay

Abstract

What is disclosed is a computer-implemented image-processing system and method for the automatic generation of video sequences that can be associated with a televised event. The methods can include the steps of: Defining a reference keyframe from a reference view from a source image sequence; From one or more keyframes, automatically computing one or more sets of virtual camera parameters; Generating a virtual camera flight path, which is described by a change of virtual camera parameters over time, and which defines a movement of a virtual camera and a corresponding change of a virtual view; and Rendering and storing a virtual video stream defined by the virtual camera flight path.

1. A computer-implemented image-processing method for instant replay for the automatic generation of video sequences, the method comprising the steps of, in an online phase of the method:
 providing or enabling at least one source camera that is adapted to provide a source image of a scene,
 wherein the pose and optical settings of the camera define a view of the camera, and 
 wherein the pose and optical settings are described by a set of camera parameters; 
 
 inputting a user input defining a reference keyframe from a reference view from the source image sequence, the reference keyframe being a video image from the point in time at which the user wishes the subsequently generated virtual replay to take place; 
 retrieving the reference keyframe from a recording of the source image sequence; and optionally retrieving one or more further keyframes from further views from recordings of further source image sequences of the scene, each further keyframe having been recorded at the same point in time as the reference keyframe, subsequently called keyframe point; 
 automatically computing one or more sets of virtual camera parameters, each set describing a virtual view of the scene by a virtual camera; 
 if more than one set of virtual camera parameters has been computed, selecting one of the sets for the subsequent processing steps, the virtual view defined by the selected set subsequently being called selected virtual view; or if only one set of virtual camera parameters has been computed, using the corresponding virtual view as the selected virtual view; 
 generating a virtual camera flight path, which is described by a change of virtual camera parameters over time, and which defines a movement of the virtual camera and a corresponding change of the virtual view
 from a first view to the selected virtual view, the first view being the reference view or, optionally, one of the further views, and then 
 from the selected virtual view to a second view, the second view being the reference view or, optionally, one of the further views; 
 
 rendering a virtual video stream defined by the virtual camera flight path; and 
 at least one of storing the virtual video stream in a computer-readable storage device and transmitting the virtual video stream over a communication link. 
 
     
     
       2. The method according to  claim 1 , wherein the step of automatically computing one or more sets of virtual camera parameters, comprises the steps of:
 automatically computing the position of objects in the scene; 
 determining a classification of the situation observed in the scene, in particular by retrieving a user input that specifies this classification; and 
 automatically computing at least one set of virtual camera parameters based on the position of objects in the scene and the classification of the situation. 
 
     
     
       3. The method according to  claim 1 , wherein the step of selecting one of the sets of virtual camera parameters defining the selected virtual view for the subsequent processing steps comprises the steps of:
 automatically generating, for each of the sets of virtual camera parameters, the corresponding virtual view and the corresponding virtual image of the scene, subsequently called virtual keyframe at this point in time; and 
 displaying the virtual keyframes on an image display device and inputting a user selection indicating one of the virtual keyframes, and using the associated virtual view as the selected virtual view for the subsequent processing steps. 
 
     
     
       4. The method according to  claim 2 , wherein the step of selecting one of the sets of virtual camera parameters defining the selected virtual view for the subsequent processing steps comprises the steps of:
 automatically generating, for each of the sets of virtual camera parameters, the corresponding virtual view and the corresponding virtual image of the scene, subsequently called virtual keyframe at this point in time; and 
 displaying the virtual keyframes on an image display device and inputting a user selection indicating one of the virtual keyframes, and using the associated virtual view as the selected virtual view for the subsequent processing steps. 
 
     
     
       5. The method according to  claim 1 , the method further comprising the following steps being executed in a preparatory phase preceding the online phase:
 creating a feature model for the detection and separation of foreground objects from a background; 
 determining, from multiple images obtained from a camera observing a scene with known features, fixed camera parameters that do not change during later video capturing; and 
 storing the feature model and the fixed camera parameters in a computer-readable storage means, to be retrieved and used by later image processing. 
 
     
     
       6. The method according to  claim 5 , wherein the feature model is a color model and comprises color models corresponding to different classes of foreground objects. 
     
     
       7. The method according to  claim 1 , the method further comprising the following steps being executed in a preparatory phase preceding the online phase:
 sampling a plurality of images from views covering a potential camera configuration space; 
 locating and identifying features in these images; 
 determining camera parameters corresponding to these views; 
 storing, as reference information, the features and feature location relative to the respective image; and 
 storing an association from the reference information to the camera parameters in a computer-readable storage means for later retrieval for fast camera calibration based on image features. 
 
     
     
       8. The method according to  claim 1 , the method further comprising the step of, in a preparatory phase, inputting information about playing directions of the teams and time information associated therewith, specifying the time during which the playing direction is valid. 
     
     
       9. The method according to  claim 1 , when at least two source image sequences from at least two source cameras are being processed, comprising the steps of:
 automatically identifying foreground objects extracted from one of the source image sequences with foreground objects of the one or more remaining source image sequences; and 
 automatically, without user interaction, determining the position of these foreground objects in the scene. 
 
     
     
       10. The method according to  claim 1 , wherein, in the step of generating a virtual camera flight path, the following steps are executed for automatically determining the first view and the second view:
 if only one source image sequence is available, choosing the first and the second view to be identical to the reference view; and 
 if two source image sequences are available, choosing at least one of the views to be from a source camera that is situated closer to the observed scene than the other source cameras. 
 
     
     
       11. The method according to  claim 1 , further comprising the following steps:
 designating one of the source cameras as being a leading camera; and 
 automatically selecting the source camera that is used for least one of the first and second views according to the viewing direction of the leading camera,
 wherein said selected source camera is preferably a camera that is located on the side of the leading camera towards which the leading camera is looking. 
 
 
     
     
       12. The method according to  claim 1 , wherein, for generating a virtual camera flight path, the following steps are executed:
 generating a set of virtual camera flight paths for different combinations of first view, virtual view, and second view; 
 displaying a graphical representation of each of the set of virtual camera flight paths on an image display device and inputting a user selection indicating one of the virtual camera flight paths; and then 
 using the selected virtual camera flight path for rendering the virtual video stream. 
 
     
     
       13. The method according to  claim 1 , wherein the step of generating a virtual camera flight path comprises the steps of:
 given the selected virtual view, the first view, and the second view, retrieving the sets of camera parameters describing these views; 
 interpolating the set of virtual camera parameters, starting with the values from the set of camera parameters corresponding to the first view, changing to the values of the set of camera parameters corresponding to the selected virtual view, and then changing to the values of the set of camera parameters corresponding to the second view; 
 verifying, for all or for a subset of the views along the virtual camera flight path, whether the respective image for the view comprises areas for which there is no image information from the available keyframes; 
 if the amount of missing image information in one or more of these views exceeds a predetermined limit, then, automatically or by means of a user interaction, determining one or more intermediate key views for which the amount of missing image information is smaller than for the views(s) with missing information, or zero, and computing the virtual camera flight path to pass through these key views. 
 
     
     
       14. The method according to  claim 12 , wherein the step of generating a virtual camera flight path comprises the step of:
 when interpolating the set of virtual camera parameters, maintaining in the sequence of views of the virtual flight path a smooth movement of a stationary or moving point of interest in the scene. 
 
     
     
       15. The method according to  claim 1 , the method further comprising the steps of:
 if no selected virtual view or no satisfying virtual camera flight path is determined by the virtual replay unit or chosen by the user, then:
 generating an enhanced still image from the reference keyframe or one of the further keyframes, and 
 subsequently using the enhanced still image in place of the virtual video stream. 
 
 
     
     
       16. The method according to  claim 1 , the method further comprising the steps of:
 generating a concatenated video sequence comprising, in this order:
 a first video sequence from a first source camera leading up to and ending with the keyframe in the first view, 
 the virtual video stream, starting with from the keyframe in the first view and ending with the keyframe in the second view, and 
 a second video sequence from a second source camera starting with the keyframe in the second view. 
 
 
     
     
       17. The method according to  claim 1 , the method being implemented by means of a video server interacting with a virtual replay unit, the video server being configured to record and store video input streams and being controllable to output video output streams made up of incoming and stored video input streams, the virtual replay unit being configured to generate, from the one or more keyframes corresponding to different views of the scene at the same point in time, the virtual video stream; and the method comprising the steps of:
 the video server transmitting to the virtual replay unit one or more keyframes, either by one of the video output streams or by a separate communication link; and 
 the virtual replay unit generating the virtual video stream. 
 
     
     
       18. The method according to  claim 16 , the method being implemented by means of a video server interacting with a virtual replay unit, the video server being configured to record and store video input streams and being controllable to output video output streams made up of incoming and stored video input streams, the virtual replay unit being configured to generate, from the one or more keyframes corresponding to different views of the scene at the same point in time, the virtual video stream; and the method comprising the steps of:
 the video server transmitting to the virtual replay unit one or more keyframes, either by one of the video output streams or by a separate communication link; and 
 the virtual replay unit generating the virtual video stream. 
 
     
     
       19. The method according to  claim 18 , further comprising the steps of:
 the virtual replay unit controlling the video server to transmit to the virtual replay unit the one or more keyframes (i.e., the reference keyframe and the optional further keyframes); and 
 only one of the following two groups of steps:
 the virtual replay unit, having computed the virtual video stream, controlling the video server to transmit to the virtual replay unit the first video sequence and the second video sequence, and the virtual replay unit concatenating the first video sequence, the virtual video stream and the second video sequence; or 
 the virtual replay unit, having computed the virtual video stream, transmitting the virtual video stream to the video server and controlling the video server to concatenate the first video sequence, the virtual video stream, and the second video sequence. 
 
 
     
     
       20. The method according to  claim 18 , further comprising the steps of:
 the video server transmitting to the virtual replay unit the one or more keyframes; 
 the video server controlling the virtual replay unit to compute the virtual video stream, preferably by triggering the virtual replay unit to do so; and 
 only one of the following two groups of steps:
 the video server transmitting to the virtual replay unit the first video sequence and the second video sequence, and the video server controlling the virtual replay unit to concatenate the first video sequence, the virtual video stream, and the second video sequence, or 
 the virtual replay unit, having computed the virtual video stream, transmitting the virtual video stream to the video server and controlling the video server to concatenate the first video sequence, the virtual video stream, and the second video sequence. 
 
 
     
     
       21. The method according to  claim 19 , further comprising the steps of:
 the video server transmitting to the virtual replay unit the one or more keyframes; 
 the video server controlling the virtual replay unit to compute the virtual video stream, preferably by triggering the virtual replay unit to do so; and 
 only one of the following two groups of steps:
 the video server transmitting to the virtual replay unit the first video sequence and the second video sequence, and the video server controlling the virtual replay unit to concatenate the first video sequence, the virtual video stream, and the second video sequence, or 
 the virtual replay unit, having computed the virtual video stream, transmitting the virtual video stream to the video server and controlling the video server to concatenate the first video sequence, the virtual video stream, and the second video sequence. 
 
 
     
     
       22. A virtual replay unit for image processing for instant replay, comprising:
 one or more programmable computer data-processing units, being programmed to:
 input one or more keyframes, 
 perform the steps of the method according to  claim 1 , and 
 output at least one virtual video stream generated according to the method claim steps performed. 
 
 
     
     
       23. A non-transitory, computer-readable medium, comprising computer-program code recorded thereon,
 wherein when said code is loaded onto a computer, said computer is caused to execute steps to perform the method according to  claim 1 .