Identifying objects of interest for handicapped individuals based on eye movement patterns

Abstract

In some embodiments, apparatuses and methods are provided to support handicapped and other customers in a retail store. Some embodiments, provide systems, comprising: a plurality of cameras positioned in a retail store; an image processing control circuit configured to determine a customer field of view; a product identification control circuit configured to identify a set of multiple products within the identified customer field of view, assign a viewing probability to each product without the first customer having to touch the first product; a product selection control circuit configured to determine a purchase probability of the first product; add the first product to a virtual electronic cart when the purchase probability of the first product exceeds a purchase probability threshold; and communicate instructions to control the worker personal computing device to instruct a worker to retrieve the first product.

What is claimed is: 
     
         1 . A system to support handicapped customers in a retail store, comprising:
 a plurality of cameras positioned and distributed along both sides of each aisle of multiple aisles of a retail store associated with a retailer, wherein each camera of the plurality of cameras is oriented to capture images of customers moving along a respective one of the multiple aisles;   an image processing control circuit communicatively coupled with each of the plurality of cameras, wherein the image processing control circuit is configured to process images from each of the plurality of cameras, and based on multiple images from a set of multiple cameras of the plurality of cameras determine a customer field of view of a first customer within a field of view of one or more of the cameras of the set of the multiple cameras;   a product identification control circuit communicatively coupled with the image processing control circuit, wherein the product identification control circuit is configured to identify, based on product location information, a set of multiple products within the identified customer field of view, assign a viewing probability to each product of the set of the multiple products as a function of a distance from a determined center of the customer field of view and identify a first product of the set of the multiple products that the first customer is considering purchasing, without the first customer having to touch or pick up the first product, as a function of the viewing probability and a customer purchase history;   a product selection control circuit communicatively coupled with the product identification control circuit, wherein the product selection control circuit is configured to: determine a purchase probability that the first product is a product the first customer intends to purchase as a function of the purchase history associated with the first customer and without the first customer having to touch or pick up the first product; add the first product to a virtual electronic cart as a purchase set of one or more products that the first customer is predicted to intend to purchase when the purchase probability of the first product exceeds a purchase probability threshold; and communicate instructions to a worker personal computing device and control the worker personal computing device to instruct a first worker to retrieve the first product for purchase by the first customer and to be supplied to the first customer.   
     
     
         2 . The system of  claim 1 , wherein the image processing control circuit, in determining the customer field of view of the first customer, is configured to track eye movement of the first customer over time and determine the field of view of the first customer as a function of the tracked eye movement over time. 
     
     
         3 . The system of  claim 1 , wherein the image processing control circuit is further configured to:
 determine a reliability confidence of each camera of the set of multiple cameras; and   wherein the image processing control circuit, in determining the customer field of view of a first customer, is configured to determine the customer field of view based on orientations of each camera of the set of the multiple cameras, the fields of view of each camera of the set of the multiple cameras and the reliability confidences of each camera of the set of the multiple cameras.   
     
     
         4 . The system of  claim 3 , wherein the image processing control circuit, in determining the reliability confidence of each camera of the set of multiple cameras, is configured to identify an orientation of the first customer relative to a camera field of view of each of the cameras of the set of multiple cameras, and assign the reliability confidence of each camera as a function of the orientation of the customer relative to the corresponding camera field of view. 
     
     
         5 . The system of  claim 4 , wherein the image processing control circuit is further configured to communicatively coupled with an optical head mounted display system that is at least temporarily associated with the first customer, wherein the image processing control circuit receives eye tracking information from the optical head mounted display system;
 wherein the image processing control circuit, in determining the customer field of view of the first customer, is configured to determine the customer field of view based on the orientations of each camera of the set of the multiple cameras, the fields of view of each camera of the set of the multiple cameras, the reliability confidences of each camera of the set of the multiple cameras and the eye tracking information relative to a location of the first customer.   
     
     
         6 . The system of  claim 3 , wherein the image processing control circuit is further configured to identify facial features of the first customer from one or more images from at least one of the first set of the multiple cameras, identify the first customer within a respective aisle of the multiple aisles based at least on the facial features, access a customer profile associated with the first customer based on the identification of the first customer, and access the customer purchase history through the customer profile associated with the first customer. 
     
     
         7 . The system of  claim 3 , wherein the image processing control circuit in determining the customer field of view is configured to determine, based on image processing of the multiple images from the set of multiple cameras, a head orientation of a head of the first customer and a facial orientation of a face of the first customer, and determine the customer field of view as a function of the head orientation and the facial orientation. 
     
     
         8 . The system of  claim 1 , wherein the product selection control circuit, in determining the purchase probability that the first customer intends to purchase the first product, obtains a duration of a gaze of the first customer on the first product, obtains estimated emotions of the first customer based on facial expressions occurring within a first threshold time of the duration of the gaze of the first customer on the first product, and obtains body language data occurring within a second threshold time of the duration of the gaze of the first customer on the first product; and
 determine the purchase probability as a function of the purchase history associated with the first customer, the duration of the gaze, the estimated emotions, and the body language.   
     
     
         9 . The system of  claim 8 , further comprising a model training control circuit configured to receive notification that the first customer declined to actually purchase the first product, and adjust a modeling relative to the first customer based at least on the received notification and the purchase probability of the first product. 
     
     
         10 . The system of  claim 1 , wherein the product selection control circuit is further configured to determine a quantity of the first product predicted to be purchased by the first customer based on the purchase history of the first customer, and in communicating the instructions to the worker personal computing device is further configured to instruct the quantity of the first product and control the worker personal computing device to instruct the first worker to retrieve the quantity of the first product for purchase by the first customer. 
     
     
         11 . A method of supporting handicapped customers in a retail store, comprising:
 processing, by an image processing control circuit, images from each of a plurality of cameras, wherein the plurality of cameras are positioned and distributed along both sides of each aisle of multiple aisles of a retail store associated with a retailer, wherein each camera of the plurality of cameras is oriented to capture images of customers moving along a respective one of the multiple aisles;   determining, based on multiple images from a set of multiple cameras of the plurality of cameras, a customer field of view of a first customer within a field of view of one or more of the cameras of the set of the multiple cameras;   identifying, by a product identification control circuit, a set of multiple products within the identified customer field of view based on product location information;   assigning a viewing probability to each product of the set of the multiple products as a function of a distance from a determined center of the customer field of view;   identifying a first product of the set of the multiple products that the first customer is considering purchasing, without the first customer having to touch or pick up the first product, as a function of the viewing probability and a customer purchase history;   determining, by a product selection control circuit, a purchase probability that the first product is a product the first customer intends to purchase as a function of the purchase history associated with the first customer and without the first customer having to touch or pick up the first product;   adding the first product to a virtual electronic cart as a purchase set of one or more products that the first customer is predicted to intend to purchase when the purchase probability of the first product exceeds a purchase probability threshold; and   communicating instructions to a worker personal computing device and controlling the worker personal computing device to instruct a first worker to retrieve the first product for purchase by the first customer and to be supplied to the first customer.   
     
     
         12 . The method of  claim 11 , wherein the determining the customer field of view of the first customer comprises tracking eye movement of the first customer over time; and
 determining the field of view of the first customer as a function of the tracked eye movement over time.   
     
     
         13 . The method of  claim 11 , further comprising:
 determining a reliability confidence of each camera of the set of multiple cameras; and   wherein the determining the customer field of view of a first customer comprises determining the customer field of view based on orientations of each camera of the set of the multiple cameras, the fields of view of each camera of the set of the multiple cameras and the reliability confidences of each camera of the set of the multiple cameras.   
     
     
         14 . The method of  claim 13 , wherein the determining the reliability confidence of each camera of the set of multiple cameras comprises:
 identifying an orientation of the first customer relative to a camera field of view of each of the cameras of the set of multiple cameras; and   assigning the reliability confidence of each camera as a function of the orientation of the customer relative to the corresponding camera field of view.   
     
     
         15 . The method of  claim 14 , further comprising:
 receiving eye tracking information from an optical head mounted display system that is at least temporarily associated with the first customer;   wherein the determining the customer field of view of the first customer comprises determining the customer field of view based on the orientations of each camera of the set of the multiple cameras, the fields of view of each camera of the set of the multiple cameras, the reliability confidences of each camera of the set of the multiple cameras and the eye tracking information relative to a location of the first customer.   
     
     
         16 . The method of  claim 13 , further comprising:
 identifying facial features of the first customer from one or more images from at least one of the first set of the multiple cameras;   identifying the first customer within a respective aisle of the multiple aisles based at least on the facial features; and   accessing a customer profile associated with the first customer based on the identification of the first customer; and   accessing the customer purchase history through the customer profile associated with the first customer.   
     
     
         17 . The method of  claim 13 , wherein the determining the customer field of view comprises:
 determining, based on image processing of the multiple images from the set of multiple cameras, a head orientation of a head of the first customer and a facial orientation of a face of the first customer, and determine the customer field of view as a function of the head orientation and the facial orientation.   
     
     
         18 . The method of  claim 11 , wherein the determining the purchase probability that the first customer intends to purchase the first product comprises:
 obtaining a duration of a gaze of the first customer on the first product;   obtaining estimated emotions of the first customer based on facial expressions occurring within a first threshold time of the duration of the gaze of the first customer on the first product;   obtaining body language data occurring within a second threshold time of the duration of the gaze of the first customer on the first product; and   determining the purchase probability as a function of the purchase history associated with the first customer, the duration of the gaze, the estimated emotions, and the body language.   
     
     
         19 . The method of  claim 18 , further comprising
 repeatedly training a product identification model over time, comprising receiving notification that the first customer declined to actually purchase the first product; and adjusting the model relative to the first customer based at least on the received notification and the purchase probability of the first product.   
     
     
         20 . The method of  claim 11 , further comprising:
 determining a quantity of the first product predicted to be purchased by the first customer based on the purchase history of the first customer; and   wherein the communicating the instructions to the worker personal computing device comprises communicating an instruction of a quantity of the first product and controlling the worker personal computing device to instruct the first worker to retrieve the quantity of the first product for purchase by the first customer.