Method and apparatus for providing tooth-brushing guide information using augmented reality

Abstract

A method of providing tooth-brushing guide information according to an embodiment of the present invention includes an operation in which a user terminal analyzes a captured image of a user, recognizes the face of the user, and identifies landmarks of the face; an operation in which the user terminal generates a personalized dental model of the user using the landmarks, an operation in which the user terminal provides a virtual dental image by augmented reality using the dental model; and an operation in which the user terminal provides tooth-brushing guide information using the dental model.

1 . A method of providing tooth-brushing guide information, the method comprising:
 by a camera of a user terminal, capturing an image of a user,   by a processor of the user terminal, analyzing the captured image of the user, recognizing a face of the user, and identifying landmarks of the face, wherein the landmarks comprise at least one selected from the group consisting of a left eye, a right eye, a left eyebrow, a right eyebrow, a left cheek, a right cheek, a mouth, a left ear, a right ear, corners of the mouth, a center of the mouth, and a jawline, of the user;   by the processor of the user terminal, estimating a position of each tooth of teeth of the user based on the identified landmarks, the captured image of the user, and anatomical information;   by the processor of the user terminal, generating a dental model of the user based on the estimated positions, and generating a virtual dental image based on the dental model;   by a display of the user terminal, displaying the generated virtual dental image through augmented reality;   by the processor of the user terminal, generating tooth-brushing guide information based on the dental model; and   by the display of the user terminal, displaying the generated tooth-brushing guide information.   
     
     
         2 . The method of  claim 1 , wherein the estimating comprises:
 further estimating a size and an arrangement position of each tooth of the teeth of the user based on the identified landmarks, the captured image of the user, and the anatomical information.   
     
     
         3 . The method of  claim 1 , wherein the estimating comprises:
 estimating that mandibular right back teeth are located at a first midpoint portion between a first position, at which the jawline and the right ear of the user meet, and a position of the center of the mouth of the user;   estimating that mandibular left back teeth are located at a second midpoint portion between a second position, at which the jawline and the left ear of the user meet, and the position of the center of the mouth of the user; and   estimating that front teeth are located at the position of the center of the mouth of the user.   
     
     
         4 . The method of  claim 1 , wherein the estimating comprises:
 estimating a size of a jaw of the user from the jawline of the user; and   estimating the size of the teeth of the user using the estimated size of the jaw.   
     
     
         5 . The method of  claim 1 , wherein the estimating comprises:
 estimating a shape of a jaw of the user from the jawline of the user; and   estimating an arrangement of the teeth of the user as any one of a square arch, an oval arch, and a triangular arch using the estimated shape of the jaw.   
     
     
         6 . The method of  claim 1 , wherein the generating of the dental model comprises:
 receiving personalized information including one or more of a date of birth, a gender, a height, a weight, a number of teeth, a position of teeth, a number of decayed teeth, and a position of decayed teeth from the user; and   modifying the dental model of the user using the received personalized information.   
     
     
         7 . The method of  claim 6 , wherein the modifying of the dental model of the user comprises:
 counting back a number of months and number of years from a date of birth of the user and determining whether the teeth of the user are permanent teeth on the basis of the counted number of months and number of years; and   limiting the number of teeth included in the dental model to 20 or less when the teeth of the user are deciduous teeth and to 28 or less when the teeth of the user are permanent teeth.   
     
     
         8 . The method of  claim 7 , wherein the modifying of the dental model of the user includes:
 counting back the number of months and number of years from the date of birth of the user and determining whether the teeth of the user are the permanent teeth on the basis of the counted number of months and number of years; and   modifying one or more of a size and a shape of the teeth included in the dental model according to a result of determining whether the teeth of the user are permanent teeth.   
     
     
         9 . The method of  claim 6 , wherein the modifying of the dental model of the user comprises:
 modifying one or more of the number, size, and shape of the teeth included in the dental model using one or more of the gender, height, age, and weight of the user.   
     
     
         10 . The method of  claim 1 , wherein the displaying the generated virtual dental image through augmented reality comprises:
 further displaying a virtual toothbrush image by augmented reality or displaying only the virtual toothbrush image instead of displaying the virtual dental image.   
     
     
         11 . The method of  claim 1 , wherein the displaying the generated virtual dental image through augmented reality comprises:
 further displaying a virtual toothbrush image by augmented reality or displaying only the virtual toothbrush image instead of displaying the virtual dental image.   
     
     
         12 . The method of  claim 1 , wherein the displaying the generated virtual dental image comprises:
 analyzing the captured image and estimating a hand of the user with which a toothbrush is held and an angle thereof, and   displaying a position and an angle of a virtual toothbrush image through augmented reality on the basis of the estimated hand and angle thereof.   
     
     
         13 . The method of  claim 1 , wherein the providing of the tooth-brushing guide information comprises:
 dividing the dental model into a predetermined number of parts and displaying a degree to which tooth-brushing is performed on each part using colors.   
     
     
         14 . The method of  claim 13 , wherein the displaying of the degree to which tooth-brushing is performed on each part using colors comprises:
 when the teeth of the user are deciduous teeth, dividing the dental model into a total of eight parts including front and back surfaces of maxillary left teeth, front and back surfaces of mandibular left teeth, front and back surfaces of maxillary right teeth, and front and back surfaces of mandibular right teeth.   
     
     
         15 . The method of  claim 13 , wherein the displaying of the degree to which tooth-brushing is performed on each part using colors comprises:
 when the teeth of the user are permanent teeth, dividing the dental model into a total of sixteen parts including front, top, and back surfaces of maxillary left back teeth, front, top, and back surfaces of mandibular left back teeth, front and back surfaces of maxillary front teeth, front and back surfaces of mandibular front teeth, front, top, and back surfaces of maxillary right back teeth, and front, top, and back surfaces of mandibular right back teeth.   
     
     
         16 . The method of  claim 1 , wherein the providing of the tooth-brushing guide information comprises:
 dividing the dental model into a predetermined number of parts and visually or aurally guiding one or more of a number of required tooth-brushing strokes, a direction of tooth-brushing, and duration of tooth-brushing for each part.   
     
     
         17 . The method of  claim 16 , wherein the visually or aurally guiding of one or more of the number of required tooth-brushing strokes, the direction of tooth-brushing, and the duration of tooth-brushing for each part comprises:
 when a first tooth has not yet erupted or has fallen out in the dental model, not guiding for a position corresponding to the first tooth.   
     
     
         18 . The method of  claim 16 , wherein the visually or aurally guiding of one or more of the number of required tooth-brushing strokes, the direction of tooth-brushing, and the duration of tooth-brushing for each part comprises:
 guiding horizontal tooth-brushing using a roll technique or a Fones technique when the user is younger than a predetermined age and guiding vertical tooth-brushing using a Bass technique or a modified Bass technique in other cases.   
     
     
         19 . A non-transitory computer readable storage medium storing instructions/program which, when executed by a hardware computer, cause the computer to carry out the method of  claim 1 . 
     
     
         20 . A method of providing tooth-brushing guide information, the method comprising:
 by a camera of a user terminal, capturing an image of a user,   by a processor of the user terminal, analyzing the captured image of the user, recognizing a face of the user, and identifying landmarks of the face;   by the processor of the user terminal, generating a dental model of the user based on the identified landmarks, and generating a virtual dental image based on the dental model;   by the processor of the user terminal, analyzing the captured image of the user, estimating a hand of the user with which a toothbrush is held and an angle thereof, and generating the virtual toothbrush image based on the estimated hand and angle;   by a display of the user terminal, displaying the generated virtual dental image and the generated virtual toothbrush image through augmented reality;   by the processor of the user terminal, generating tooth-brushing guide information based on the dental model; and   by the display of the user terminal, displaying the generated tooth-brushing guide information.