Advertising evaluation using physical sensors

Abstract

Systems and techniques for displaying advertisements on a digital display and gathering impression and view data related to the advertisement that may be used to refine or score the advertisements for greater effectiveness. The impression and view data may be used to identify effective portions of advertisements and subsequently to train a machine learning model to predict impression data for advertisements that may be used to iteratively improve the advertisements.

The invention claimed is: 
     
       1. A system comprising:
 a display device configured to provide advertising information, the display device positioned within a retail facility; 
 a camera positioned within the retail facility, connected with the display device, and having a field of view comprising a region in front of the display device to capture image data of one or more customers with a line of sight to the display device; and 
 a computing device comprising:
 a processor; and 
 one or more non-transitory computer-readable medium having instructions stored thereon that, when executed by the processor, cause the processor to perform operations comprising:
 causing the display device to output content; 
 receiving image data from the camera associated with the display device; 
 determining viewing data describing whether a customer is looking towards the display device based on the image data; 
 determining a portion of the content viewed by the customer in response to determining that the customer is looking towards the display device; 
 using an object recognition algorithm, identifying a type of object featured in the portion of the content and located on a shelf within the retail facility; 
 generating, using one or more machine learning models trained to identify characteristics of types of objects, a content tag for the type of object featured in the portion of the content, the content tag describing the type of object; 
 determining an impression count for the portion of the content based on an aggregate of viewing data for portions of the content; 
 determining an impression ratio for the portion of the content by dividing the impression count for the portion by an average number of impressions over the content; 
 receiving purchase data for the object, the purchase data associated with a purchase of the object by the customer after viewing the portion of the content; and 
 generating an impression score for the content based on the impression ratio for the portion of the content, the content tag, and the purchase data. 
 
 
 
     
     
       2. The system of  claim 1 , wherein the impression score for the content comprises a set of scores associated with segments of the content, and wherein the operations further comprise:
 determining a subset of the content based on scores associated with the subset of the content exceeding a predetermined threshold; 
 generating second content by selecting the subset of the content; and 
 displaying the second content on the display device. 
 
     
     
       3. The system of  claim 1 , wherein identifying when the customer is looking towards the display device comprises:
 determining, based on a pose of the customer, that the customer is viewing the display device; and 
 determining the portion based on a first timestamp when the customer begins viewing the display device and a second timestamp when the customer stops viewing the display device. 
 
     
     
       4. The system of  claim 1 , wherein generating the impression score comprises associating the portion of the advertisement with the impression ratio and aggregating the impression ratio with a second impression ratio representing other interactions with the advertisement to identify an effective portion of the advertisement. 
     
     
       5. A method comprising:
 causing a display device to output content, the display device positioned within a retail facility; 
 receiving image data from a camera associated with the display device; 
 determining viewing data describing whether a customer is looking towards the display device based on the image data; 
 determining a portion of the content viewed by the customer in response to determining that the customer is looking towards the display device; 
 using an object recognition algorithm, identifying a type of object featured in the portion of the content and located within the retail facility; 
 generating, using one or more machine learning models trained to identify characteristics of types of objects, a content tag for the type of object featured in the portion of the content, the content tag describing the type of object; 
 determining an impression count for the portion of the content based on an aggregate of viewing data for portions of the content; 
 determining an impression ratio for the portion of the content by dividing the impression count for the portion by an average number of impressions over the content; 
 receiving purchase data for the object, the purchase data associated with a purchase of the object by the customer after viewing the portion of the content; and 
 generating an impression score for the content based on the impression ratio for the portion of the content, the content tag, and the purchase data. 
 
     
     
       6. The method of  claim 5 , wherein determining the portion comprises providing the image data to one or more machine learning models trained to identify when the customer is looking at the display device, wherein the image data comprises a representation of at least part of a body of the customer. 
     
     
       7. The method of  claim 5 , wherein the impression score for the content comprises a set of scores associated with segments of the content, and wherein the method further comprises displaying a subset of the content based at least in part on the set of scores. 
     
     
       8. The method of  claim 5 , wherein determining the portion of the content comprises:
 determining a crop of at least part of the customer using a first algorithm; and 
 determining, using a second algorithm trained to identify when a person is looking at the camera based on a pose of the customer and by providing the crop of the portion of the customer, when the customer is looking towards the display device. 
 
     
     
       9. The method of  claim 5 , further comprising determining demographic data for the customer, and wherein generating the impression score further comprises associating the impression score with the demographic data. 
     
     
       10. The method of  claim 5 , further comprising:
 training a machine learning algorithm with data comprising a plurality of content files labeled with impression score data; 
 receiving second content; and 
 determining, using the machine learning algorithm, a set of predicted impression scores for the second content. 
 
     
     
       11. The method of  claim 10 , further comprising:
 identifying a second portion of the second content associated with a predicted impression score of the set of predicted impression scores based on the predicted impression score being below a threshold; and 
 altering the second portion to increase the predicted impression score. 
 
     
     
       12. A non-transitory computer-readable medium having instructions stored thereon that, when executed by one or more processors, cause the one or more processors to perform operations comprising:
 causing a display device to output content, the display device positioned within a retail facility; 
 receiving image data from a camera associated with the display device; 
 determining viewing data describing whether a customer is looking towards the display device based on the image data; 
 determining a portion of the content viewed by the customer in response to determining that the customer is looking towards the display device; 
 using an object recognition algorithm, identifying a type of object featured in the portion of the content and located on a shelf within the retail facility; 
 generating, using one or more machine learning models trained to identify characteristics of types of objects, a content tag for the type of object featured in the portion of the content, the content tag describing the type of object; 
 determining an impression count for the portion of the content based on an aggregate of viewing data for portions of the content; 
 determining an impression ratio for the portion of the content by dividing the impression count for the portion by an average number of impressions over the content; 
 receiving purchase data for the object, the purchase data associated with a purchase of the object by the customer after viewing the portion of the content; and 
 generating an impression score for the content based on the impression ratio for the portion of the content, the content tag, and the purchase data. 
 
     
     
       13. The non-transitory computer-readable medium of  claim 12 , wherein the operations further comprise:
 training a machine learning algorithm with data comprising a plurality of content files labeled with impression score data; 
 receiving second content; and 
 determining, using the machine learning algorithm, a set of predicted impression scores for the second content. 
 
     
     
       14. The non-transitory computer-readable medium of  claim 13 , wherein the operations further comprise:
 identifying a second portion of the second content associated with a predicted impression score of the set of predicted impression scores based on the predicted impression score being below a threshold; and 
 altering the second portion to increase the predicted impression score. 
 
     
     
       15. The non-transitory computer-readable medium of  claim 12 , wherein determining the portion comprises providing the image data to one or more machine learning models trained to identify when the customer is looking at the camera, wherein the image data comprises a representation of at least part of a body of the customer. 
     
     
       16. The method of  claim 5 , wherein the retail facility comprises a cashier-less retail facility, and further comprising receiving interaction data for the object featured in the portion of the content, the interaction data associated with the customer picking up the object from the shelf, looking at the object on the shelf, or returning the object to the shelf. 
     
     
       17. The method of  claim 5 , further comprising based at least in part on determining the type of object featured in the portion of the content and using the object recognition algorithm, determining the type of object is located on a shelf within the retail facility. 
     
     
       18. The method of  claim 17 , further comprising determining interaction data representing a hand of the customer entering a plane of the shelf and interacting with the type of object located on the shelf, and wherein the purchase data is associated with the interaction data. 
     
     
       19. The non-transitory computer-readable medium of  claim 12 , wherein the operations further comprise:
 based at least in part on determining the type of object featured in the portion of the content and using the object recognition algorithm, determining the type of object is located on a shelf within the retail facility. 
 
     
     
       20. The non-transitory computer-readable medium of  claim 19 , wherein the operations further comprise:
 determining interaction data representing a hand of the customer entering a plane of the shelf and interacting with the type of object located on the shelf, and wherein the purchase data is associated with the interaction data.