Automatically obtaining real-time, geographically-relevant product information from heterogeneus sources

Abstract

Techniques for obtaining geographically-relevant product inventory information, in real-time, from heterogeneous data sources are described. Product inventory information, including the volume of available products in specific geographical locations, is obtained from at least three different sources. First, one or more data feeds may be received. Second, a data obtaining module uses one or more APIs to obtain product inventory information from one or more third-party inventory management systems. Finally, a structured data mining module uses a web crawler, at the direction of a crawler configuration, to systematically obtain product inventory information from various third-party websites. Accordingly, a user's search query is processed to provide geographically relevant product inventory information in near real time.

What is claimed is: 
     
         1 . A method comprising:
 providing a web-based crawler configuration application to client device;   receiving, via the web-based crawler configuration application, a selection of an element of a web page displayed at the client device;   identifying, in a source document of the web page, an element of product inventory information corresponding to the selected element of the web page; and   generating a crawler configuration file based on the identified element of product inventory information, the crawler configuration file configuring an operation of a crawler application.   
     
     
         2 . The method of  claim 1 , further comprising:
 executing a first set of instructions representing an instance of the crawler application, the instance of the crawler application configured to perform a set of operations specified in the crawler configuration file, the set of operations resulting in a retrieval of product inventory information for one or more products hosted at one or more web servers;   executing a second set of instructions representing an instance of a real-time product availability lookup (RTPAL) application, the RTPAL application to use one or more application programming interfaces (APIs) to request and receive product inventory information from one or more third-party network-connected inventory management systems;   enhancing the product inventory information from the crawler application with the product inventory information from the RTPAL application; and   storing the enhanced product inventory information in the database.   
     
     
         3 . The method of  claim 2 , further comprising:
 subsequent to receiving product inventory information as a result of executing either of the crawler application and the RTPAL application, determining that received product inventory information for a particular product does not specify a unique product identifier;   performing a matching operation by comparing various elements of information concerning the particular product with corresponding information from one or more known products to determine a product with which the received product inventory information for the particular product not specifying the unique product identifier best matches; and   storing the received product inventory information for the particular product not specifying the unique product identifier in the database.   
     
     
         4 . The method of  claim 2 , further comprising:
 storing product inventory information for a particular product received via the crawler application or the RTPAL application in a data cache with each cache entry having a cache key based on a zipcode for a location at which the particular product received via the crawler application or the RTPAL application is available, a product identifier, and a product variation identifier.   
     
     
         5 . The method of  claim 2 , further comprising:
 assigning to each product identified in the received product inventory information as a result of executing either of the crawler application and the RTPAL application one or more category identifiers for a particular category in a hierarchical category.   
     
     
         6 . The method of  claim 1 , further comprising:
 receiving a search query from a user, the search query including information identifying a desired geographical area; and   processing the search query to provide product inventory information for a particular product satisfying the search query, the product inventory information for the particular product satisfying the search query specifying one or more merchant stores in a geographical location satisfying the desired geographical area identified in the query, and indicating a quantity of the particular product satisfying the search query available at each of the one or more merchant stores.   
     
     
         7 . The method of  claim 1 , wherein the web-based crawler configuration application enables the user to specify one of a suite of web crawlers for use with a particular crawler configuration file generated by the web crawler configuration application. 
     
     
         8 . The method of  claim 1 , wherein the crawler configuration identifies one crawler from a plurality of crawlers to be used with the crawler configuration. 
     
     
         9 . The method of  claim 1 , wherein the web-based crawler configuration application enables the user to invoke a code editing application to specify customized code for obtaining a particular element of product inventory information, the customized code for inclusion in a crawler configuration file for use with a particular crawler. 
     
     
         10 . The method of  claim 1 , wherein the web-based crawler configuration application defines a set of selectors, each selector describing how to extract a single item of information for insertion into the database. 
     
     
         11 . A server comprising:
 one or more processors for executing one or more sets of instructions stored in a memory, the one or more set of instructions comprising:   providing a web-based crawler configuration application to client device;   receiving, via the web-based crawler configuration application, a selection of an element of a web page displayed at the client device;   identifying, in a source document of the web page, an element of product inventory information corresponding to the selected element of the web page; and   generating a crawler configuration file based on the identified element of product inventory information, the crawler configuration file configuring an operation of a crawler application.   
     
     
         12 . The server of  claim 11 , wherein the one or more sets of instructions further comprise:
 executing a first set of instructions representing an instance of the crawler application, the instance of the crawler application configured to perform a set of operations specified in the crawler configuration file, the set of operations resulting in a retrieval of product inventory information for one or more products hosted at one or more web servers;   executing a second set of instructions representing an instance of a real-time product availability lookup (RTPAL) application, the RTPAL application to use one or more application programming interfaces (APIs) to request and receive product inventory information from one or more third-party network-connected inventory management systems;   enhancing the product inventory information from the crawler application with the product inventory information from the RTPAL application; and   storing the enhanced product inventory information in the database.   
     
     
         13 . The server of  claim 12 , wherein the one or more sets of instructions further comprise:
 subsequent to receiving product inventory information as a result of executing either of the crawler application and the RTPAL application, determining that received product inventory information for a particular product does not specify a unique product identifier;   performing a matching operation by comparing various elements of information concerning the particular product with corresponding information from one or more known products to determine a product with which the received product inventory information for the particular product not specifying the unique product identifier best matches; and   storing the received product inventory information for the particular product not specifying the unique product identifier in the database.   
     
     
         14 . The server of  claim 12 , wherein the one or more sets of instructions further comprise:
 storing product inventory information for a particular product received via the crawler application or the RTPAL application in a data cache with each cache entry having a cache key based on a zipcode for a location at which the particular product received via the crawler application or the RTPAL application is available, a product identifier, and a product variation identifier.   
     
     
         15 . The server of  claim 11 , wherein the one or more sets of instructions further comprise:
 receiving a search query from a user, the search query including information identifying a desired geographical area; and   processing the search query to provide product inventory information for a particular product satisfying the search query, the product inventory information for the particular product satisfying the search query specifying one or more merchant stores in a geographical location satisfying the desired geographical area identified in the query, and indicating a quantity of the particular product satisfying the search query available at each of the one or more merchant stores.   
     
     
         16 . The server of  claim 11 , wherein the web-based crawler configuration application enables the user to specify one of a suite of web crawlers for use with a particular crawler configuration file generated by the web crawler configuration application. 
     
     
         17 . The server of  claim 11 , wherein the crawler configuration identifies one crawler from a plurality of crawlers to be used with the crawler configuration. 
     
     
         18 . The server of  claim 11 , wherein the web-based crawler configuration application enables the user to invoke a code editing application to specify customized code for obtaining a particular element of product inventory information, the customized code for inclusion in a crawler configuration file for use with a particular crawler. 
     
     
         19 . The server of  claim 11 , wherein the web-based crawler configuration application defines a set of selectors, each selector describing how to extract a single item of information for insertion into the database. 
     
     
         20 . A machine-readable storage medium storing instructions thereon, which, when executed by a processor of a server, will cause the server to perform a set of operations comprising:
 providing a web-based crawler configuration application to client device;   receiving, via the web-based crawler configuration application, a selection of an element of a web page displayed at the client device;   identifying, in a source document of the web page, an element of product inventory information corresponding to the selected element of the web page; and   generating a crawler configuration file based on the identified element of product inventory information, the crawler configuration file configuring an operation of a crawler application.