Real-time recommendation monitoring dashboard

Abstract

Methods and systems for analyzing and evaluating item recommendations presented on a web site are disclosed. In one aspect, a user interface is generated for display on a website. Item recommendations personalized for an individual user are displayed within in item recommendation region of the website. Impression, clickstream, and sales data are received from user activity on the website. The impression, clickstream, and sales data are displayed on a dashboard of an administrator user interface. An administrator user can select visualizations to display on the dashboard using selectors for impressions, clicks, sales, users, and time. Two or more sets of data can be overlaid to illustrate relationships between user interactions with recommended items and actual sales of those items. The dashboard is dynamically updated in response to new data received from the website in real-time.

The invention claimed is: 
     
       1. A system for evaluating item recommendations on a website, the system comprising:
 an enterprise web server configured to generate and maintain the website accessible by a plurality of users, the enterprise web server comprising:
 a website server configured to generate and provide for display a first user interface to the plurality of users accessing the website on user computing devices associated with the plurality of users, the first user interface including an item recommendation region presenting one or more items based on customized item recommendations generated for each of the plurality of users; 
 a click analysis application configured to analyze data associated with interactions of the plurality of users with the website received from the website server, the data including one or more of impressions data, clickstream data, and sales data associated with each item of the one or more items presented within the item recommendation region across the plurality of users; and 
 a recommendation evaluation service comprising at least one processor and a memory coupled to the at least one processor and storing instructions that, when executed by the at least one processor, cause the at least one processor to:
 generate and provide for display a second user interface to an administrator different from the plurality of users on an administration computing device associated with the administrator, the second user interface including a graphical chart display area, a plurality of analysis parameter selectors, one or more time selectors, and an overlay selector; 
 receive selections of one or more of the plurality of analysis parameter selectors and at least one time selector for a period of time; 
 based on the selected one or more of the plurality of analysis parameter selectors, depict, within the graphical chart display area, a first visualization of a first set of numerical values for one or more of impressions or clicks for at least a subset of the one or more items presented within the item recommendation region across the plurality of users based on the one or more of the impressions data and the clickstream data collected over the selected period of time; 
 receive a selection of the overlay selector within the second user interface; 
 in response to receiving the selection of the overlay selector, display, within the graphical chart display area, an overlay of a second visualization of a second set of numerical values for sales of at least the subset of items based on the sales data collected over the selected period of time; and 
 in response to receiving new data associated with interactions of the plurality of users with the website, dynamically update the graphical chart display area to reflect updated numerical values within one or more of the first set of numerical values and the second set of numerical values. 
 
 
 
     
     
       2. The system of  claim 1 , wherein the instructions that, when executed by the at least one processor, further cause the at least one processor to:
 organize the impressions data, the clickstream data, and the sales data by one or more of item, time period, location, and user. 
 
     
     
       3. The system of  claim 1 , wherein the instructions that, when executed by the at least one processor, further cause the at least one processor to:
 in response to receiving at least one new selection of one or more of the plurality of analysis parameter selectors and the at least one time selector, updating the graphical chart display area to reflect updated numerical values within one or more of the first set of numerical values and the second set of numerical values. 
 
     
     
       4. The system of  claim 1 , wherein the instructions that, when executed by the at least one processor, further cause the at least one processor to:
 receive, from the administrator via the second user interface, a selection of at least the subset of items. 
 
     
     
       5. The system of  claim 1 , wherein the instructions that, when executed by the at least one processor, further cause the at least one processor to:
 automatically select at least the subset of items based on one or more criterion, including one or more of a number of impressions, a number clicks, or a number of sales associated with the subset of items for the selected period of time. 
 
     
     
       6. The system of  claim 1 , wherein the plurality of analysis parameter selectors include one or more of a display time selector, an impression selector, a click selector, a purchase selector and a user selector, and one or more of the plurality of analysis parameter selectors include associated scales. 
     
     
       7. The system of  claim 6 , wherein based on a selection of the display time selector, the first set of numerical values for the one or more of impressions or clicks in the first visualization include a total time each item of at least the subset of items is displayed in the item recommendation region within the selected period of time. 
     
     
       8. The system of  claim 6 , wherein based on a selection of the impression selector, the first set of numerical values for the one or more of impressions or clicks in the first visualization include a number of times each item of at least the subset of items is displayed in the item recommendation region for the selected period of time. 
     
     
       9. The system of  claim 6 , wherein based on a selection of the click selector, the first set of numerical values for the one or more of impressions or clicks in the first visualization include a number of times each item of at least the subset of items is clicked within the selected period of time. 
     
     
       10. The system of  claim 6 , wherein based on a selection of the purchase selector, the first set of numerical values for the one or more of impressions or clicks in the first visualization include a number of times each item of at least the subset of items is purchased within the selected period of time. 
     
     
       11. The system of  claim 6 , wherein based on a selection of the user selector, the first set of numerical values in the first visualization for the one or more of impressions or clicks include one or more of the impressions data or the clickstream data associated with one or more particular users selected from the plurality of users. 
     
     
       12. The system of  claim 1 , wherein:
 a subset of the plurality of users accessing the website are unknown users having no associated item preference data, 
 the one or more items presented within the item recommendation region of the first user interface provided for display to the subset of the plurality of users are based on customized item recommendations generated for each of the subset of the plurality of users based on a context in which each of the unknown users is accessing the website, and 
 the instructions that, when executed by the at least one processor, further cause the at least one processor to:
 evaluate the customized item recommendations generated for each of the subset of the plurality of users based on one or more of impressions data, clickstream data, and sales data associated with each item of the one or more items presented within the item recommendation region to the subset of users. 
 
 
     
     
       13. A system for evaluating item recommendations on a website, the system comprising:
 at least one processor; and 
 a memory coupled to the at least one processor and storing instructions that, when executed by the at least one processor, cause the at least one processor to:
 generate and provide for display a first user interface to a plurality of users accessing the website on user computing devices associated with the plurality of users, the first user interface including an item recommendation region presenting one or more items based on customized item recommendations generated for each of the plurality of users; 
 collect and analyze data associated with interactions of the plurality of users with the website, the data including one or more of impressions data, clickstream data, and sales data associated with each item of the one or more items presented within the item recommendation region across the plurality of users; 
 generate and provide for display a second user interface to an administrator different from the plurality of users on an administration computing device associated with the administrator, the second user interface including a graphical chart display area, a plurality of analysis parameter selectors, one or more time selectors, and an overlay selector; 
 receive selections of one or more of the plurality of analysis parameter selectors and at least one time selector for a period of time; 
 based on the one or more of the plurality of analysis parameter selectors selected, depict, within the graphical chart display area, a first visualization of a first set of numerical values for one or more of impressions or clicks for at least a subset of the one or more items presented within the item recommendation region across the plurality of users based on the one or more of the impressions data and the clickstream data collected over the selected period of time; 
 receive a selection of the overlay selector; 
 in response to receiving the selection of the overlay selector, display, within the graphical chart display area, an overlay of a second visualization of a second set of numerical values for sales of at least the subset of items based on the sales data collected over the selected period of time; and 
 in response to receiving new data associated with interactions of the plurality of users with the website, dynamically update the graphical chart display area to reflect updated numerical values within one or more of the first set of numerical values and the second set of numerical values. 
 
 
     
     
       14. The system of  claim 13 , wherein the at least one processor is further caused to:
 organize the impressions data, the clickstream data, and the sales data by one or more of item, period of time, location, and user. 
 
     
     
       15. The system of  claim 13 , wherein the at least one processor is further caused to:
 in response to receiving at least one new selection of one or more of the plurality of analysis parameter selectors and the at least one time selector, updating the graphical chart display area to reflect updated numerical values within one or more of the first set of numerical values and the second set of numerical values. 
 
     
     
       16. The system of  claim 13 , wherein based on the one or more of the plurality of analysis parameter selectors selected, the first set of numerical values for the one or more of impressions or clicks in the first visualization include, one or more of:
 a total time each item of at least the subset of items is displayed in the item recommendation region within the selected period of time; 
 a number of times each item of at least the subset of items is displayed in the item recommendation region for the selected period of time; 
 a number of times each item of at least the subset of items is clicked within the selected period of time; 
 a number of times each item of at least the subset of items is purchased within the selected period of time; and 
 one or more of the impressions data or the clickstream data associated with one or more particular users from the plurality of users. 
 
     
     
       17. The system of  claim 13 , wherein the at least one processor is further caused to one of:
 receive, from the administrator via the second user interface, a selection of at least the subset of items; or 
 automatically select at least the subset of items. 
 
     
     
       18. The system of  claim 13 , wherein the first visualization is a first chart type and the second visualization is a second chart type different from the first chart type. 
     
     
       19. The system of  claim 13 , wherein the overlay selector is a toggle that turns on and off display of the sales data in the second visualization. 
     
     
       20. A non-transitory computer storage medium storing instructions for evaluating item recommendations on a website, wherein the instructions, when executed by at least one processor, cause the at least one processor to:
 generate and provide for display a first user interface to a plurality of users accessing the website on user computing devices associated with the plurality of users, the first user interface including an item recommendation region presenting one or more items based on customized item recommendations generated for each of the plurality of users; 
 collect and analyze data associated with interactions of the plurality of users with the website, the data including one or more of impressions data, clickstream data, and sales data associated with each item of the one or more items presented within the item recommendation region across the plurality of users; 
 generate and provide for display a second user interface to an administrator different from the plurality of users on an administration computing device associated with the administrator, the second user interface including a graphical chart display area, a plurality of analysis parameter selectors, one or more time selectors, and an overlay selector; 
 receive selections of one or more of the plurality of analysis parameter selectors and at least one time selector for a period of time; 
 based on the one or more of the plurality of analysis parameter selectors selected, depict, within the graphical chart display area, a first visualization of a first set of numerical values for one or more of impressions or clicks for at least a subset of the one or more items presented within the item recommendation region across the plurality of users based on the one or more of the impressions data and the clickstream data collected over the selected period of time; 
 receive a selection of the overlay selector; 
 in response to receiving the selection of the overlay selector, display, within the graphical chart display area, an overlay of a second visualization of a second set of numerical values for sales of at least the subset of items based on the sales data collected over the selected period of time; and 
 in response to receiving new data associated with interactions of the plurality of users with the website, dynamically update the graphical chart display area to reflect updated numerical values within one or more of the first set of numerical values and the second set of numerical values.