Systems and methods for extracting foreground video

Abstract

A color image and a depth image of a live video are received. Each of the color image and the depth image are processed to identify the foreground and the background of the live video. The background of the live video is removed in order to create a foreground video that comprises the foreground of the live video. A control input may be received to control the embedding of the foreground video into a second background from a background feed. The background feed may also comprise virtual objects such that the foreground video may interact with the virtual objects.

What is claimed is: 
     
         1 . A method comprising:
 receiving, from at least one camera, pixel-depth data and pixel-color data of an image that comprises a plurality of image pixels;   generating a preliminary depth categorization of the image pixels, based at least in part on a comparison of the pixel-depth data with pixel-depth history data, the generated preliminary depth categorization preliminarily categorizing each image pixel as being a foreground pixel, a background pixel, or an unknown pixel;   generating a depth overlay at least part by overlaying the generated preliminary depth categorization on the pixel-color data;   extracting a color-image persona from the generated depth overlay at least in part by:   recategorizing each image pixel that was preliminarily categorized as being an unknown pixel to being either a foreground pixel or a background pixel; and   extracting as the color-image persona those pixels that were either preliminarily categorized or recategorized as being foreground pixels; and   outputting the extracted color-image persona.   
     
     
         2 . The method of  claim 1 , wherein:
 the at least one camera is at least one video camera; and   the image is a frame of video data.   
     
     
         3 . The method of  claim 1 , wherein the recategorizing comprises:
 if a resolution of the pixel-depth data is less than a resolution of the pixel-color data, then performing foreground region filling comprising categorizing at least one preliminary unknown pixel as a foreground pixel if the preliminary unknown pixel is surrounded by foreground pixels.   
     
     
         4 . The method of  claim 1 , wherein the recategorizing comprises recategorizing using a graph cut. 
     
     
         5 . The method of  claim 1 , further comprising:
 grouping neighboring image pixels having matching preliminary categorizations and similar depth values in the pixel-depth data into connected components.   
     
     
         6 . The method of  claim 5 , wherein the recategorizing comprises recategorizing pixels in connected unknown components that are in motion as being foreground pixels if a head of a user is detected in the moving connected unknown components. 
     
     
         7 . The method of  claim 5 , wherein the recategorizing comprises recategorizing pixels in connected background components that are in motion as being foreground pixels if a head of a user is detected in the moving connected background components. 
     
     
         8 . The method of  claim 1 , wherein the recategorizing comprises applying a median filter to smoothen foreground edges. 
     
     
         9 . The method of  claim 1 , further comprising:
 displaying the extracted color-image persona via a user interface.   
     
     
         10 . The method of  claim 1 , further comprising:
 embedding the extracted color-image persona in a second video stream.   
     
     
         11 . A system comprising:
 at least one computer; and   a non-transitory computer readable medium having stored thereon one or more programs, which when executed by the at least one computer, cause the at least one computer to carry out a set of functions, the set of functions comprising:   receiving, from at least one camera, pixel-depth data and pixel-color data of an image that comprises a plurality of image pixels;   generating a preliminary depth categorization of the image pixels, based at least in part on a comparison of the pixel-depth data with pixel-depth history data, the generated preliminary depth categorization preliminarily categorizing each image pixel as being a foreground pixel, a background pixel, or an unknown pixel;   generating a depth overlay at least part by overlaying the generated preliminary depth categorization on the pixel-color data;   extracting a color-image persona from the generated depth overlay at least in part by:   recategorizing each image pixel that was preliminarily categorized as being an unknown pixel to being either a foreground pixel or a background pixel; and   extracting as the color-image persona those pixels that were either preliminarily categorized or recategorized as being foreground pixels; and   outputting the extracted color-image persona.   
     
     
         12 . The system of  claim 11 , wherein:
 the at least one camera is at least one video camera; and   the image is a frame of video data.   
     
     
         13 . The system of  claim 11 , wherein the recategorizing comprises:
 if a resolution of the pixel-depth data is less than a resolution of the pixel-color data, then performing foreground region filling comprising categorizing at least one preliminary unknown pixel as a foreground pixel if the preliminary unknown pixel is surrounded by foreground pixels.   
     
     
         14 . The system of  claim 11 , wherein the recategorizing comprises recategorizing using a graph cut. 
     
     
         15 . The system of  claim 11 , the set of functions further comprising:
 grouping neighboring image pixels having matching preliminary categorizations and similar depth values in the pixel-depth data into connected components.   
     
     
         16 . The system of  claim 15 , wherein the recategorizing comprises recategorizing pixels in connected unknown components that are in motion as being foreground pixels if a head of a user is detected in the moving connected unknown components. 
     
     
         17 . The system of  claim 15 , wherein the recategorizing comprises recategorizing pixels in connected background components that are in motion as being foreground pixels if a head of a user is detected in the moving connected background components. 
     
     
         18 . The system of  claim 11 , wherein the recategorizing comprises applying a median filter to smoothen foreground edges. 
     
     
         19 . The system of  claim 11 , the set of functions further comprising:
 displaying the extracted color-image persona via a user interface.   
     
     
         20 . The system of  claim 11 , the set of functions further comprising:
 embedding the extracted color-image persona in a second video stream.