Book scanning using machine-trained model

Abstract

This application discloses a technology for flattening a photographed page of a book and straightening texts therein. The technology uses one or more mathematical models to represent a curved shape of the photographed page with certain parameters. The technology also uses one or more photographic image processing techniques to dewarp the photographed page using the parameters of the curved shape. The technology uses one or more additional parameters that represent certain features of the photographed page to dewarp the photographed page.

1 - 19 . (canceled) 
     
     
         20 . A method of preparing an input-output data pair for training a machine-trainable model, the method comprising:
 providing a markings page as part of a bound book, wherein the markings page comprises a plurality of predetermined markings that are added to the bound book for training the machine-trainable model;   providing, for at least part of the plurality of predetermined markings, a 2-dimensional (2D) location of each predetermined marking on the markings page at an unbound state thereof;   capturing, using a camera, a photographic image of the markings page in an open state thereof in the bound book in which the markings page is curled from the unbound state, wherein the markings page appearing on the photographic image is distorted from the unbound state at least due to curling of the markings page in the open state at the time of capturing the photographic image;   processing the photographic image to obtain distorted 2D locations of at least part of the plurality of predetermined markings on the photographic image; and   computing a set of parameters representing distortion of the markings page on the photographic image,   wherein the set of parameters comprises a first subset of parameters that relates to curling of the markings page in the open state from the unbound state, wherein computing the first subset of parameters involves a process of iteration comprising:
 assigning values to the first subset of parameters, 
 simulating, using the assigned values to the first subset of parameters, warping of the markings page from the unbound state and/or unwarping of the markings page from the photographic image, 
 determining if at least one of the simulated warping and the simulated unwarping using the assigned values corresponds to an amount of curling of the markings page, 
 when determined that at least one of the simulated warping and the simulated unwarping does not corresponds to the amount of curling of the markings page, assigning updated values to the first subset of parameters such that at least one of the simulated warping and the simulated unwarping using the updated values would likely be more corresponding to the amount of curling than the at least one of the previously simulated warping and the previously simulated unwarping, 
 subsequently determining that at least one of the simulated warping and the simulated unwarping using further updated values corresponds to the amount of curling of the markings page, and 
 associating the further updated values as the first subset of parameters with the photographic image of the markings page or a modified version of the photographic image such that the photographic image of the markings page or the modified version is part of an input of the input-output data pair and the further updated values are part of an output of the input-output data pair. 
   
     
     
         21 . The method of  claim 20 , wherein the process of iteration comprises:
 assigning the values to the first subset of parameters,   simulating, using the assigned values to the first subset of parameters, warping of the markings page from the unbound state,   determining if the simulated warping using the assigned values conforms to the curling of the markings page in the open state from the unbound state,   when determined that the simulated warping does not conform to the curling of the markings page, assigning updated values to the first subset of parameters such that the simulated warping using the updated values would likely be more conforming to the curling than the previously simulated warping,   subsequently determining that the simulated warping using further updated values conform to the curling of the markings page, and   associating the further updated values as the first subset of parameters with the photographic image of the markings page or a modified version of the photographic image such that the photographic image of the markings page or the modified version is part of an input of the input-output data pair and the further updated values are part of an output of the input-output data pair.   
     
     
         22 . The method of  claim 20 , wherein the process of iteration comprises:
 assigning the values to the first subset of parameters,   simulating, using the assigned values to the first subset of parameters, unwarping of the markings page from the photographic image,   determining if the simulated unwarping using the assigned values conforms to the markings page at the unbound state,   when determined that the simulated unwarping does not conform to the markings page at the unbound state, assigning updated values to the first subset of parameters such that the simulated unwarping using the updated values would likely be more conforming to the marking page at the unbound state than the previously simulated unwarping,   subsequently determining that the simulated unwarping using further updated values conform to the markings page at the unbound state, and   associating the further updated values as the first subset of parameters with the photographic image of the markings page or a modified version of the photographic image such that the photographic image of the markings page or the modified version is part of an input of the input-output data pair and the further updated values are part of an output of the input-output data pair.   
     
     
         23 . The method of  claim 20 , wherein the markings page appearing on the photographic image is distorted from the unbound state due to a 3-dimensional (3D) camera location relative to the markings page in addition to the curling of the markings page in the open state at the time of capturing the photographic image, wherein the set of parameters further comprises a second subset of parameters relating to the 3D camera location relative to the markings page at the time of capturing the photographic image. 
     
     
         24 . The method of  claim 20 , wherein the markings page appearing on the photographic image is distorted from the unbound state due to a 3D camera orientation relative to the markings page at the time of capturing the photographic image in addition to the curling of the markings page in the open state at the time of capturing the photographic image, wherein the set of parameters further comprises a third subset of parameters relating to the 3D camera orientation of the camera relative to the markings page at the time of capturing the photographic image. 
     
     
         25 . The method of  claim 20 , wherein the markings page appearing on the photographic image is distorted from the unbound state due to a 3-dimensional (3D) camera location relative to the markings page and due to a 3D camera orientation relative to the markings page at the time of capturing the photographic image in addition to the curling of the markings page in the open state at the time of capturing the photographic image, wherein the set of parameters further comprises a second subset of parameters relating to the 3D camera location relative to the markings page at the time of capturing the photographic image and a third subset of parameters relating to the 3D camera orientation of the camera relative to the markings page at the time of capturing the photographic image. 
     
     
         26 . The method of  claim 20 , wherein the markings page at the unbound state is flat or substantially flat. 
     
     
         27 . The method of  claim 20 , wherein the first subset of parameters includes two parameters representing a Bezier Curve. 
     
     
         28 . The method of  claim 20 , wherein the set of parameters further comprises at least one page size parameter representing a size of the markings page in a flattened image that would be obtained by unwarping the photographic image of the markings page or a modified version of photographic image. 
     
     
         29 . The method of  claim 28 , wherein the at least one page size parameter represents a width of the markings page in the flattened image relative to a width of the flattened image. 
     
     
         30 . The method of  claim 20 , wherein, in computing the distorted 2D location for each of the plurality of predetermined markings on the distorted image, the distorted image corresponds to a photographic image of the markings page taken by a pinhole camera located at the 3D camera location relative to the markings page according to assigned value(s),
 wherein the pinhole camera has an intrinsic parameter matrix of   
       
         
           
             
               
                 [ 
                 
                   
                     
                       f 
                     
                     
                       0 
                     
                     
                       
                         x 
                         o 
                       
                     
                   
                   
                     
                       0 
                     
                     
                       f 
                     
                     
                       
                         y 
                         o 
                       
                     
                   
                   
                     
                       0 
                     
                     
                       0 
                     
                     
                       1 
                     
                   
                 
                 ] 
               
               , 
             
           
         
         where f is a focal length of the pinhole camera, and x o  and y o  are offset parameters representing translation of the origin of imaging pixels of the pinhole camera relative to the pinhole of the pinhole camera. 
       
     
     
         31 . The method of  claim 30 , wherein at least one of the first subset of parameters defining the 3D camera location represents the pinhole camera's translation along an optical axis of the pinhole camera relative to the markings page, and further represents the focal length f of the pinhole camera such that the model does not provide a separate parameter representing the focal length f other than the set of parameters. 
     
     
         32 . A method of preparing a machine-trained model, the method comprising:
 generating a plurality of input-output data pairs according to the method of  claim 20 ; and   training a machine-trainable model using the plurality of input-output data pairs to provide a machine-trained model such that the machine-trained model is configured to generate values for the set of parameters in response to an input of an image of an opened book page.   
     
     
         33 . A non-transitory storage medium storing a plurality of instructions executable by a computer, wherein the plurality of instructions, when executed, causes the computer to generate a plurality of input-output data pairs according to the method of  claim 32 .