Method, system, and non-transitory computer-readable medium for analyzing facial features for augmented reality experiences of physical products in a messaging system

Abstract

The subject technology receives image data including a representation of a face of a user. The subject technology analyzes the image data to determine a set of characteristics of the representation of the face. The subject technology, based at least in part on the determined set of characteristics, selects a particular product and a set of media content associated with the particular product. The subject technology causes display, at a client device, at least one recommendation corresponding to the set of media content associated with the particular product.

What is claimed is: 
     
       1. A method, comprising:
 receiving, by an image data receiving module of a client device, image data including a representation of a face of a user; 
 receiving a first selection of a portion of the representation of the face of the user; 
 in response to the first selection of the portion of the representation of the face of the user, causing display of a plurality of selectable graphical items, each of the plurality of selectable graphical items corresponding to a particular state of the portion of the representation of the face of the user; 
 receiving a second selection of a particular selectable graphical item from the plurality of selectable graphical items; 
 analyzing, by a product identification module accessing a shared memory of the client device, the image data to determine a set of characteristics of the representation of the face; 
 based at least in part on the determined set of characteristics and the second selection of the particular selectable graphical item from the plurality of selectable graphical items, selecting a product and a set of media content associated with the product, the product comprising a particular product that, when utilized, addresses a facial condition included in the determined set of characteristics, wherein selecting the product and the set of media content comprises: 
 sending, by the client device to a product catalog system over the Internet, an indicator of the selected product, the product catalog system comprising a first processor for processing product metadata and a first memory for storing information related to second product metadata associated with the product metadata of the selected product, the information comprising at least a set of manufacturers of the selected product; 
 sending, by the product catalog system, a request message over the Internet to a product catalog service system for obtaining metadata related to a respective physical item associated with the selected product, the request message including a product identifier, and the product catalog system and the product catalog service system being different systems that are communicatively coupled over the Internet; 
 performing, by the product catalog service system, a query on a database over a network based on the product identifier in the request message for information related to the respective physical item: 
 in response to the request message, providing, by the product catalog service system, a response message to the product catalog system, the response message including the second product metadata based on the information related to the respective physical item from the database over the network: 
 receiving, by the client device from the product catalog system over the Internet, the second product metadata, the second product metadata including additional information related to the selected product, the additional information comprising information that the selected product addresses the facial condition; and 
 causing display, at the client device, of at least one recommendation corresponding to the set of media content associated with the product, the at least one recommendation facilitating identifying a set of products, including the product, that address at least one health condition, corresponding to the facial condition, through using the set of products by the user. 
 
     
     
       2. The method of  claim 1 ,
 wherein the product metadata comprises the product identifier, a product augmented reality experience identifier, and information related to the product corresponding to the product identifier: 
 wherein selecting the product and the set of media content further comprises: 
 performing, by the product catalog system, a lookup operation on a product table to retrieve the product metadata; and 
 storing, by the product catalog system, the information related to the respective physical item into the product table. 
 
     
     
       3. The method of  claim 1 , wherein the at least one recommendation comprises a suggested augmented content reality generator, media content, or additional textual information related to at least one characteristic from the determined set of characteristics. 
     
     
       4. The method of  claim 1 , wherein the determined set of characteristics includes a health-related characteristic identified in the representation of the face, and the health condition comprises at least one of dry skin, oily skin, combination skin, sunburns, acne, eczema, rosacea, ichthyosis, or dermatitis. 
     
     
       5. The method of  claim 1 , wherein the determined set of characteristics includes a shape of the face, or a skin tone of the face. 
     
     
       6. The method of  claim 1 , wherein analyzing the image data to determine the set of characteristics of the representation of the face further comprises:
 performing a facial feature detection process to detect a first facial feature; and 
 identifying a health-related characteristic associated with the detected first facial feature. 
 
     
     
       7. The method of  claim 6 , wherein the facial feature detection process comprises a machine learning algorithm. 
     
     
       8. The method of  claim 7 , wherein the machine learning algorithm is based on a neural network model. 
     
     
       9. The method of  claim 6 , further comprising:
 performing a second facial feature detection process to detect the first facial feature, the second facial feature detection process being different than the facial feature detection process. 
 
     
     
       10. The method of  claim 1 , wherein first selection of the portion of the representation of the face of the user corresponds to an eye of the face of the user, and
 wherein the determined set of characteristics comprises different types of eyelashes, the different type of eyelashes comprising dry eyelashes and weak eyelashes, 
 wherein the at least one recommendation comprises textual information related to addressing dry eyelashes. 
 
     
     
       11. A system comprising:
 a processor; and 
 a memory including instructions that, when executed by the processor, cause the processor to perform operations comprising: 
 receiving, by an image data receiving module of a client device, image data including a representation of a face of a user; 
 receiving a first selection of a portion of the representation of the face of the user; 
 in response to the first selection of the portion of the representation of the face of the user, causing display of a plurality of selectable graphical items, each of the plurality of selectable graphical items corresponding to a particular state of the portion of the representation of the face of the user; 
 receiving a second selection of a particular selectable graphical item from the plurality of selectable graphical items; 
 analyzing, by a product identification module accessing a shared memory of the client device, the image data to determine a set of characteristics of the representation of the face; 
 based at least in part on the determined set of characteristics and the second selection of the particular selectable graphical item from the plurality of selectable graphical items, selecting a product and a set of media content associated with the product, the product comprising a particular product that, when utilized, addresses a facial condition included in the determined set of characteristics, wherein selecting the product and the set of media content comprises: 
 sending, by the client device to a product catalog system over the Internet, an indicator of the selected product, the product catalog system comprising a first processor for processing product metadata and a first memory for storing information related to second product metadata associated with the product metadata of the selected product, the information comprising at least a set of manufacturers of the selected product; 
 sending, by the product catalog system, a request message over the Internet to a product catalog service system for obtaining metadata related to a respective physical item associated with the selected product, the request message including a product identifier, and the product catalog system and the product catalog service system being different systems that are communicatively coupled over the Internet: 
 performing, by the product catalog service system, a query on a database over a network based on the product identifier in the request message for information related to the respective physical item: 
 in response to the request message, providing, by the product catalog service system, a response message to the product catalog system, the response message including the second product metadata based on the information related to the respective physical item from the database over the network: 
 receiving, by the client device from the product catalog system over the Internet, the second product metadata, the second product metadata including additional information related to the selected product, the additional information comprising information that the selected product addresses the facial condition; and 
 causing display, at the client device, of at least one recommendation corresponding to the set of media content associated with the product, the at least one recommendation facilitating identifying a set of products, including the product, that address at least one health condition, corresponding to the facial condition, through using the set of products by the user. 
 
     
     
       12. The system of  claim 11 ,
 wherein the product metadata comprises the product identifier, a product augmented reality experience identifier, and information related to the product corresponding to the product identifier: 
 performing, by the product catalog system, a lookup operation on a product table to retrieve product metadata; and 
 storing, by the product catalog system, the information related to the respective physical item into the product table. 
 
     
     
       13. The system of  claim 11 , wherein the at least one recommendation comprises a suggested augmented content reality generator, media content, or additional textual information related to at least one characteristic from the determined set of characteristics. 
     
     
       14. The system of  claim 11 , wherein the determined set of characteristics includes a health-related characteristic identified in the representation of the face. 
     
     
       15. The system of  claim 11 , wherein the determined set of characteristics includes a shape of the face, or a skin tone of the face. 
     
     
       16. The system of  claim 11 , wherein analyzing the image data to determine the set of characteristics of the representation of the face further causes the processor to perform further operations comprising:
 performing a facial feature detection process to detect a first facial feature; and 
 identifying a health-related characteristic associated with the detected first facial feature. 
 
     
     
       17. The system of  claim 16 , wherein the facial feature detection process comprises a machine learning algorithm. 
     
     
       18. The system of  claim 17 , wherein the machine learning algorithm is based on a neural network model. 
     
     
       19. The system of  claim 16 , wherein the memory includes further instructions, which further cause the processor to perform further operations comprising:
 performing a second facial feature detection process to detect the first facial feature, the second facial feature detection process being different than the facial feature detection process. 
 
     
     
       20. A non-transitory computer-readable medium comprising instructions,
 which when executed by a computing device, cause the computing device to perform operations comprising: 
 receiving, by an image data receiving module of a client device, image data including a representation of a face of a user; 
 receiving a first selection of a portion of the representation of the face of the user; 
 in response to the first selection of the portion of the representation of the face of the user, causing display of a plurality of selectable graphical items, each of the plurality of selectable graphical items corresponding to a particular state of the portion of the representation of the face of the user; 
 receiving a second selection of a particular selectable graphical item from the plurality of selectable graphical items; 
 analyzing, by a product identification module accessing a shared memory of the client device, the image data to determine a set of characteristics of the representation of the face; 
 based at least in part on the determined set of characteristics and the second selection of the particular selectable graphical item from the plurality of selectable graphical items, selecting a product and a set of media content associated with the product, the product comprising a particular product that, when utilized, addresses a facial condition included in the determined set of characteristics, wherein selecting the product and the set of media content comprises: 
 sending, by the client device to a product catalog system over the Internet, an indicator of the selected product, the product catalog system comprising a first processor for processing product metadata and a first memory for storing information related to second product metadata associated with the product metadata of the selected product, the information comprising at least a set of manufacturers of the selected product; 
 sending, by the product catalog system, a request message over the Internet to a product catalog service system for obtaining metadata related to a respective physical item associated with the selected product, the request message including a product identifier, and the product catalog system and the product catalog service system being different systems that are communicatively coupled over the Internet: 
 performing, by the product catalog service system, a query on a database over a network based on the product identifier in the request message for information related to the respective physical item: 
 in response to the request message, providing, by the product catalog service system, a response message to the product catalog system, the response message including the second product metadata based on the information related to the respective physical item from the database over the network; 
 receiving, by the client device from the product catalog system over the Internet, the second product metadata, the second product metadata including additional information related to the selected product, the additional information comprising information that the selected product addresses the facial condition; and 
 causing display, at the client device, of at least one recommendation corresponding to the set of media content associated with the product, the at least one recommendation facilitating identifying a set of products, including the product, that address at least one health condition, corresponding to the facial condition, through using the set of products by the user.