System and method of image augmentation

Abstract

A method of image augmentation for an image of a book includes capturing an image of the book, detecting at least a portion of at least one fiduciary marker of the book within the image, estimating placement of the book's spine based upon the detected portion of the fiduciary marker, hypothesising possible positions for edges of a rigid leaf being turned in the book based upon estimated placement of the spine, processing the book image to identify edges within the image, comparing elements of the identified edges with the hypothesized positions for edges of the rigid leaf, selecting one of the hypothesized positions that best coincides with the compared elements of the processed image as representative of the position of the rigid leaf being turned in the book, and augmenting the book image with a virtual graphic element arranged in accordance with the selected representative position of the rigid leaf.

We claim: 
     
       1. A method of image augmentation for an image of a book, the book having at least one substantially rigid leaf, the method comprising the steps of:
 capturing an image of the book; 
 detecting at least a portion of at least one fiduciary marker of the book within the image; 
 estimating a placement of a spine of the book in the image based upon the detected at least the portion of the at least one fiduciary marker; 
 hypothesising a plurality of possible positions for edges of a rigid leaf being turned in the book based upon the estimated placement of the spine of the book; 
 processing the image of the book to identify edges within the image; 
 comparing elements of the identified edges of the processed image with the hypothesised plurality of possible positions for edges of the rigid leaf; 
 selecting one of the hypothesised plurality of possible positions for edges of the rigid leaf that best coincides with the compared elements of the processed image as representative of the position of the rigid leaf being turned in the book; and 
 augmenting the image of the book with a virtual graphic element arranged in accordance with the selected representative position of the rigid leaf. 
 
     
     
       2. A method according to  claim 1 , in which the step of hypothesising the plurality of possible positions for edges of the rigid leaf comprises generating one hypothesis for edges of the rigid leaf per unit of rotation of the leaf about the spine, where a unit lies in the range between 0.1 degree and 10 degrees. 
     
     
       3. A method according to  claim 1  in which the step of processing the image of the book to identify edges within the image uses the Canny edge detector. 
     
     
       4. A method according to  claim 1 , in which the step of comparing elements of the identified edges of the processed image with the hypothesised plurality of possible edge positions comprises the step of:
 scoring a candidate hypothesised edge responsive to the relative angle between the candidate hypothesised edge and an estimated orientation of the element of the identified edge. 
 
     
     
       5. A method according to  claim 4  in which scoring a candidate hypothesised edge is further responsive to its proximity to an identified edge element. 
     
     
       6. A method according to  claim 4 , in which the step of scoring a candidate hypothesised edge comprises the step of:
 estimating the orientation of identified edge elements based upon their response to a horizontal Sobel operator and a vertical Sobel operator. 
 
     
     
       7. A method according to  claim 4 , in which the step of selecting a hypothesis comprises the step of:
 summing the scores for each candidate hypothesised possible position for edges of the rigid leaf and selecting the candidate hypothesis with the highest score as representative of the position of the rigid leaf. 
 
     
     
       8. A method according to  claim 1 , in which the captured image is a stereoscopic image comprising a first monoscopic image and second monoscopic image, and the step of augmenting the image of the book comprises arranging a virtual graphic element in the second monoscopic image responsive to a displacement between elements of the first monoscopic image and second monoscopic image. 
     
     
       9. A tangible, non-transitory computer program product comprising a storage medium on which is stored computer readable program code, the program code, when executed by a processor, cause the processor to perform a method of image augmentation for an image of a book, the book having at least one substantially rigid leaf, the method comprising the steps of:
 capturing an image of the book; 
 detecting at least a portion of at least one fiduciary marker of the book within the image; 
 estimating a placement of a spine of the book in the image based upon the detected at least the portion of the at least one fiduciary marker; 
 hypothesising a plurality of possible positions for edges of a rigid leaf being turned in the book based upon the estimated placement of the spine of the book; 
 processing the image of the book to identify edges within the image; 
 comparing elements of the identified edges of the processed image with the hypothesised plurality of possible positions for edges of the rigid leaf; 
 selecting one of the hypothesised plurality of possible positions for edges of the rigid leaf that best coincides with the compared elements of the processed image as representative of the position of the rigid leaf being turned in the book; and 
 augmenting the image of the book with a virtual graphic element arranged in accordance with the selected representative position of the rigid leaf. 
 
     
     
       10. An entertainment device comprising:
 an image input operable to receive a captured image of a book; 
 a marking detector operable to detect at least a portion of at least one fiduciary marker of the book within the image; 
 a placement estimator operable to estimate a placement of a spine of the book in the image based upon the detected at least the portion of the at least one fiduciary marker; 
 a processor arranged in operation to hypothesise a plurality of possible positions for edges of a rigid leaf being turned in the book based upon the estimated placement of the spine of the book; 
 a image processor operable to process the image of the book to identify edges within the image; 
 a comparator operable to compare elements of the identified edges of the processed image with the hypothesised plurality of possible positions for edges of the rigid leaf; 
 a selector operable to select one of the hypothesised plurality of possible positions for edges of the rigid leaf that best coincides with the compared elements of the processed image as the hypothesis representative of the position of the rigid leaf being turned in the book; and 
 an image augmentation processor operable to augment the image of the book with a virtual graphic element arranged in accordance with the selected representative position of the rigid leaf. 
 
     
     
       11. An entertainment device according to  claim 10 , in which the processor arranged in operation to hypothesise is operable to generate one hypothesis per unit of rotation of a leaf about the spine, where a unit lies in the range between 0.1 degree and 10 degrees. 
     
     
       12. An entertainment device according to  claim 10 , in which the comparator is operable to score a candidate hypothesised edge responsive to the relative angle between the candidate hypothesised edge and an estimated orientation of the identified edge element. 
     
     
       13. An entertainment device according to  claim 12  in which the comparator is operable to score a candidate hypothesised edge further responsive to its proximity to an identified edge element. 
     
     
       14. An entertainment device according to  claim 12 , in which the comparator approximates pixel positions of the candidate hypothesised edge within the processed image using a line drawing algorithm. 
     
     
       15. An entertainment device according to  claim 10 , in which the captured image is a stereoscopic image comprising a first monoscopic image and second monoscopic image, and the image augmentation processor is responsive to a displacement between elements of the first monoscopic image and second monoscopic image to generate a second virtual graphic element to augment the second monoscopic image.