Video And Digital Multimedia Aggregator Remote Content Crawler

Abstract

A remote content crawler continually crawls a digital communication network looking for content to provide to a content aggregator, The content provided to the aggregator may be stored in a form of an entire content file. The content may include an entire movie, television program or electronic book. Alternatively, the content provided to the aggregator may be a reference to a content file that is stored at, or that will he available at one of the remote locations. The content may be a reference to a future, scheduled live sports event that will be made available to system users. The sports event may be provided for a one time fee, as part of a sports package, for which a fee is collected, or as a free event.

1 . An apparatus, comprising:
 an aggregator remote content database configured to store information including a catalog of a plurality of content stored in one or more remote databases and metadata associated with the plurality of content; and   a search engine server configured to perform steps including:
 generating index data as a result of indexing the plurality of content, a source of each piece of the plurality of content, and information about a content provider associated with each piece of the plurality of content; 
 routing the index data to the aggregate remote content database; 
 maintaining a network resource database including a plurality of addresses; 
 perform a search of the aggregator remote content database using search criteria; and 
 building a list identifying a subset of the plurality of content as a result of the search. 
   
     
     
         2 . The apparatus of  claim 1 , wherein the maintaining of the network resource database includes periodically updating the plurality of addresses. 
     
     
         3 . The apparatus of  claim 2 , wherein the updating of the plurality of addresses includes at least one of:
 querying a content provider database;   receiving updated content provider listings from a crawler that includes hyperlinked names that were previously undocumented;   downloading domain name records from public and private domain name registration databases;   synchronizing a local domain name service database with one or more remote domain name service databases, which are connected to the apparatus via a network; and   performing reverse domain name resolution.   
     
     
         4 . The apparatus of  claim 2 , wherein the updating of the plurality of addresses includes automatically deleting dead or expired addresses. 
     
     
         5 . The apparatus of  claim 1 , wherein the network resource database further includes additional data in association with each of the plurality of addresses, the additional data including a URL owner's identity or contact information. 
     
     
         6 . The apparatus of  claim 1 , wherein the network resource database further includes additional data in association with each of the plurality of addresses, the additional data including information indicating a type of content available at the associated address. 
     
     
         7 . The apparatus of  claim 1 , wherein the network resource database further includes additional data in association with each of the plurality of addresses, the additional data including expiration time of the associated address. 
     
     
         8 . The apparatus of  claim 1 , wherein the network resource database further includes additional data in association with each of the plurality of addresses, the additional data including subdomain names of the associated address that are to be excluded during crawling. 
     
     
         9 . The apparatus of  claim 1 , wherein the search engine server comprises a crawler configured to crawl the one or more remote databases using crawling criteria to identify the plurality of content that is indexed. 
     
     
         10 . The apparatus of  claim 9 , wherein at least a portion of the crawling criteria used for crawling is automatically generated. 
     
     
         11 . The apparatus of  claim 10 , wherein the automatically generated crawling criteria comprises crawling criteria generated by importing metadata schemes for one or more content formats and parsing metadata included within a criteria database. 
     
     
         12 . The apparatus of  claim 10 , wherein the automatically generated crawling criteria comprises crawling criteria generated by analyzing hypertext associated with hyperlinks. 
     
     
         13 . The apparatus of  claim 10 , wherein the automatically generated crawling criteria comprises crawling criteria generated by analyzing text proximate to a hyperlink. 
     
     
         14 . The apparatus of  claim 9 , wherein the crawling of the one or more remote databases is performed more frequently on those databases that are more frequently updated. 
     
     
         15 . The apparatus of  claim 1 , wherein the hierarchical list is ordered partly based on a ranking of the one or more remote databases at which the corresponding content is located. 
     
     
         16 . A method, comprising:
 maintaining a network resource database including a plurality of addresses;   indexing a plurality of content stored in one or more remote databases, a source of each piece of the plurality of content, and information about a content provider associated with each piece of the plurality of content to obtain index data;   storing the index data in an aggregate content database;   performing a search of the aggregator content database using search criteria; and   building a list identifying a subset of the plurality of content as a result of the search.   
     
     
         17 . The method of  claim 16 , further comprising:
 crawling the one or more remote databases using crawling criteria to identify the plurality of content.   
     
     
         18 . The method of  claim 17 , wherein at least a portion of the crawling criteria is automatically generated. 
     
     
         19 . A method, comprising:
 building and maintaining network resource data, wherein the network resource data comprises address data for content servers coupled to one or more communication networks;   building and maintaining content provider data, Wherein the content provider data comprises data related to potential providers of content on the one or more communication networks;   building and maintaining crawling criteria; and   crawling the one or more communication networks using the crawling criteria to search for content.   
     
     
         20 . The method of  claim 19 , wherein the building and maintaining of the content provider data comprises ranking content providers based on at least one of:
 quantity of available content;   a content provider's membership in a professional association;   amount of content requested and downloaded by users; and   content provider ratings determined based on user feedback.