Image processing device and image processing method

Abstract

An edge image of the image frame is created after it is determined to start tracking. Meanwhile, a particle is distributed in a space of a coefficient set for each control point sequence when the control point sequence of a B-spline curve representing the shape of the target object is represented in a linear combination of control point sequence of a B-spline curve representing a plurality of reference shapes that are made available in advance. A particle is also distributed in the space of a shape-space vector, the likelihood of each particle is observed, and the probability density distribution is acquired. A curve obtained by weighting parameters by the probability density distribution and then averaging the weighted parameters is created as a tracking result.

1 . An image processing device comprising:
 a reference shape storage unit configured to store a plurality of parameters that define contour lines of a plurality of reference shapes; and   an object shape determination unit configured to represent and output, by defining a set of coefficients of each parameter in a linear combination of the plurality of parameters stored in the reference-shape storage unit, the shape of the contour line of an object in an image in the linear combination.   
     
     
         2 . The image processing device according to  claim 1  further comprising:
 an image acquisition unit configured to acquire a moving image stream including a first image frame and a second image frame in which the image of the object is captured, wherein 
 the object shape determination unit including: 
 a shape prediction unit that creates and eliminates, in a space of a coefficient set defined by the set of the coefficients, a particle used for a particle filter based on an estimated existence probability distribution of the object in the first image frame so as to cause the particle to make a transition based on a predetermined transition model, 
 an observation unit that observes the likelihood of each particle by computing a matching score between the contour line of the object in the second image frame and a candidate contour defined by the particle, and 
 a contour line acquisition unit that estimates the shape of the contour line of the object in the second image frame by computing the estimated existence probability distribution, in the space of the coefficient set, of the object in the second image frame based on the likelihood observed by the observation unit and then by weighting the set of the coefficients of each particle based on the estimated existence probability distribution. 
 
     
     
         3 . The image processing device according to  claim 1 , wherein the parameters that define the contour line are a control point sequence when the contour line is represented by a B-spline curve. 
     
     
         4 . The image processing device according to  claim 1 , wherein the parameters that define the contour line are a knot vector when the contour line is represented by a B-spline curve. 
     
     
         5 . The image processing device according to  claim 2  further comprising:
 a shape space vector prediction unit configured to create and eliminate, in a shape space vector space defined by a shape space vector that defines a translation amount, an amount of magnification, and the rotation angle of the contour line defined by each particle, the particle caused to make a transition by the shape prediction unit based on the estimated existence probability distribution of the object in the first image frame so as to cause the particle to make a transition based on a predetermined transition model, wherein 
 the observation unit observes the likelihood of the particle caused to make a transition by the shape space vector prediction unit, and wherein 
 the contour line acquisition unit estimates the translation amount, the amount of magnification, and the rotation angle of the object in the second image frame by further computing the estimated existence probability distribution, in the space of the shape space vector, of the object in the second image frame based on the likelihood observed by the observation unit and then by weighting the shape space vector of each particle based on the estimated existence probability distribution. 
 
     
     
         6 . The image processing device according to  claim 2 , wherein the shape prediction unit causes the particle, which is created and eliminated based on the estimated existence probability distribution of the object in the first image frame, to make a transition so as to form a Gaussian distribution around coordinates in the space of the coefficient set before the transition. 
     
     
         7 . The image processing device according to  claim 6 , wherein the shape prediction unit causes, when the shape defined by the particle is detected to be an intermediate shape between a first reference shape and a second reference shape based on the coordinates before the transition of the particle in the space of the coefficient set, the particle to make a transition so as to form a Gaussian distribution where the standard deviation of the distribution in the direction of a line connecting the coordinates representing the first reference shape and the coordinates representing the second reference shape in the space of the coefficient set is larger than those in the other directions. 
     
     
         8 . The image processing device according to  claim 2 , wherein the shape prediction unit distributes, when the shape defined by the particle is detected to be a reference shape based on the coordinates before the transition of the particle in the space of the coefficient set, the particle on the assumption that the probability of transition to other reference shapes, which are provided as a possible destination of a transition from the reference shape, is equal. 
     
     
         9 . An image processing method comprising:
 reading, from a memory device, a plurality of parameters that define contour lines of a plurality of reference shapes and defining a set of the coefficients of each parameter in a linear combination of the parameters; and   representing and outputting the contour line of an object in an image in the linear combination by using the defined set of the coefficients.   
     
     
         10 . The image processing method according to  claim 9  further comprising:
 acquiring a moving image stream including a first image frame and a second image frame in which the image of the object is captured and storing the moving image stream in memory, wherein, 
 the outputting of the contour line of the object includes: 
 predicting contour lines of the object in the second image frame based on the estimated existence probability distribution of the object in the first image frame in a space of the coefficient set defined by the set of the coefficients and acquiring the estimated existence probability distribution of the object in the second image frame by making a comparison with the contour line of the object in the second image frame read from the memory; and 
 estimating a contour line of the object in the second image frame based on the estimated existence probability distribution. 
 
     
     
         11 . A computer program embedded in a non-transitory computer-readable medium and that is comprising modules of:
 reading, from a memory device, a plurality of parameters that define contour lines of a plurality of reference shapes and defining a set of the coefficients of each parameter in a linear combination of the parameters; and   representing and outputting the contour line of an object in an image in the linear combination by using the defined set of the coefficients.   
     
     
         12 . The computer program according to  claim 11  that is embedded in a non-transitory computer-readable medium and that is comprising modules of:
 acquiring a moving image stream including a first image frame and a second image frame, in which the image of an object is captured, and storing the moving image stream in memory, 
 predicting contour lines of the object in the second image frame based on the estimated existence probability distribution of the object in the first image frame in a space of the coefficient set defined by the set of the coefficients and acquiring the estimated existence probability distribution of the object in the second image frame by making a comparison with the contour line of the object in the second image frame read from the memory; and 
 estimating a contour line of the object in the second image frame based on the estimated existence probability distribution. 
 
     
     
         13 . A non-transitory computer-readable recording medium having embodied thereon a computer program product comprising modules of:
 reading, from a memory device, a plurality of parameters that define contour lines of a plurality of reference shapes and defining a set of the coefficients of each parameter in a linear combination of the parameters; and   representing and outputting the contour line of an object in an image in the linear combination by using the defined set of the coefficients.   
     
     
         14 . The non-transitory computer-readable recording medium having embodied thereon a computer program product according to  claim 13  comprising modules of:
 acquiring a moving image stream including a first image frame and a second image frame in which the image of an object is captured and storing the moving image stream in memory, 
 predicting contour lines of the object in the second image frame based on the estimated existence probability distribution of the object in the first image frame in a space of the coefficient set defined by the set of the coefficients and acquiring the estimated existence probability distribution of the object in the second image frame by making a comparison with the contour line of the object in the second image frame read from the memory; and 
 estimating a contour line of the object in the second image frame based on the estimated existence probability distribution. 
 
     
     
         15 . An image processing device comprising:
 a contour line acquisition unit configured to acquire a contour line of an object from an image frame that constitutes a moving image;   a target part identification unit configured to identify the position of a predetermined part of the object based on the contour line of the object acquired by the contour line acquisition unit;   a processing unit configured to create a processed image in which predetermined processing is performed on an area, which is determined based on the position of the predetermined part identified by the target part identification unit, in the image frame; and   an output unit configured to output data of the moving image having, as the image frame, the processed image generated by the processing unit.   
     
     
         16 . The image processing device according to  claim 15 , wherein
 the target part identification unit acquires the time change of the inclination of the predetermined part from the time change of the contour line, and   the processing unit creates the processed image so that a result of the processing is time-variable in accordance with the time change of the inclination of the predetermined part.   
     
     
         17 . The image processing device according to  claim 15  further comprising:
 a storage unit configured to store data of a three-dimensional graphics image of clothing, wherein 
 the contour line acquisition unit represents the contour line of a human head, which is the object, by a Greek ohm-shaped template and acquires the contour line of the human head in a Greek ohm-shape, by matching against an edge image of an image frame, 
 the target part identification unit identifies a shoulder line in the contour line, and 
 the processing unit combines the image of the human body clothed with the clothing such that the shoulder line matches the shoulder line of the clothing selected by a user. 
 
     
     
         18 . The image processing device according to  claim 15  further comprising:
 a storage unit configured to store data of a three-dimensional graphics image of a fingernail with nail art, wherein 
 the contour line acquisition unit acquires the contour line of a human hand, which is the object, 
 the target part identification unit identifies an area of the fingernail based on the position of the fingertip in the contour line, and 
 the processing unit combines the image of the fingernail with nail art such that the image of the fingernail with nail art selected by a user is overlapped with the area of the fingernail. 
 
     
     
         19 . The image processing device according to  claim 18 , wherein the contour line acquisition unit acquires, by adjusting through, matching score computation, a set of coefficients occurring when a parameter of a B-spline curve that defines the contour line of the hand is represented in a linear combination of a parameter that defines the contour line of a plurality of reference shapes defined in advance. 
     
     
         20 . The image processing device according to  claim 19 , wherein the contour line acquisition unit represents the parameter of the B-spline curve in a linear combination of a parameter of respective B-spline curves of five reference shapes, in each of which any one of the five fingers is extended. 
     
     
         21 . The image processing device according to  claim 15 , wherein
 the contour line acquisition unit represents the contour line of a human head, which is the object, by a Greek ohm-shaped template and acquires the contour line of the human head in a Greek ohm-shape by matching against an edge image of an image frame,   the target part identification unit identifies, as the area of the head, an area surrounded by the Greek ohm-shaped line and by the segment of a line connecting the end points thereof, and   the processing unit performs mosaic processing on the area of the head.   
     
     
         22 . The image processing device according to  claim 15 , wherein the processing unit combines, in an area outside the contour of the object determined based on the position of the predetermined part identified by the target part identification unit, an image displaying information regarding the object. 
     
     
         23 . The image processing device according to  claim 22 , wherein
 the contour line acquisition unit represents the contour line of a human head, which is the object, by a Greek ohm-shaped template and acquires the contour line of the human head in a Greek ohm-shape by matching against an edge image of an image frame,   the target part identification unit identifies both the position of a predetermined point on the contour line of the head and the apparent size of the head, and   the processing unit combines an image of an information tag having a line pointing to the predetermined point and a figure, in which the information is displayed, in a size proportional to the apparent size of the head.   
     
     
         24 . The image processing device according to  claim 23 , wherein the processing unit does not combine an information tag having a size below a lower limit or above an upper limit, which are set for the size of the information tag in advance. 
     
     
         25 . The image processing device according to  claim 15 , wherein
 the target part identification unit detects, based on the time change of the contour line of the object, that at least a part of a first part is hidden behind a second part, and   the processing unit performs, when combining images corresponding to the first part and the second part, respectively, as the process, a hidden-surface removal process so that at least a part of the image corresponding to the first part is hidden by the second part.   
     
     
         26 . An image processing method comprising:
 reading an image frame constituting a moving image stored in a memory and acquiring a contour line of an object;   identifying an area of the object or of a predetermined part of the object based on the contour line; and   creating a processed image in which predetermined processing is performed on an area determined by the identified area in the image frame.   
     
     
         27 . A computer program embedded in a non-transitory computer-readable medium and that is comprising modules of:
 reading an image frame constituting a moving image stored in a memory and acquiring a contour line of an object;   identifying an area of the object or of a predetermined part of the object based on the contour line; and   creating a processed image in which predetermined processing is performed on an area determined by the identified area in the image frame.   
     
     
         28 . A non-transitory computer-readable recording medium having embodied thereon a computer program product comprising modules of:
 reading an image frame constituting a moving image stored in a memory and acquiring a contour line of an object;   identifying an area of the object or of a predetermined part of the object based on the contour line; and   creating a processed image in which predetermined processing is performed on an area determined by the identified area in the image frame.