System, apparatus, method, and computer program product for indexing a file

Abstract

A search engine manages the indexing of web page contents and accepts user selection criteria to find and report hits that meet the search criteria. The inventive search engine has an associated crawler function wherein display images of the web pages are rendered and stored as snapshots, preferably when the pages are indexed. The search engine reports search results by composing an html page with links to the corresponding page hits and containing snapshot reduced size graphic images showing the web pages as they appeared when fetched and stored as snapshots.

I claim: 
     
       1. A method for processing data files stored at distributed addresses on a data processing network, at least some of the data files having text and graphic content, the method comprising:
 analyzing at least a subset of the data files to produce a database of information characterizing aspects of the data files that tend to distinguish the data files from one another, and cross referencing said information to addresses of the data files;   generating an image of at least a portion of the subset of data files, and storing a graphic file of said image in a manner cross referenced to the addresses of the data files, whereby the graphic file represents an image of the data files at a time of generation;   receiving search queries and applying the search queries to the database for selecting a hit list from among the data files;   reporting the hit list in a search report including the addresses of each of the data files selected and the image corresponding to the data files in the hit list at the respective time of generation.   
     
     
       2. The method of  claim 1 , wherein the data files comprise hypertext markup language text and linked graphic format files, and wherein said analyzing comprises at least one of indexing the text and reviewing at least a portion of the data files for assignment of an arbitrary categorization. 
     
     
       3. The method of  claim 2 , wherein the data files comprise hypertext markup language text and linked graphic format files on one of an intranet and the World Wide Web, and wherein said generating comprises rendering an image corresponding to the data files according to a predetermined display configuration defining a default choice of at least one of a pixel display size, font type, color pallet, color resolution and use of colors. 
     
     
       4. The method of  claim 2 , wherein said analyzing and said generating are accomplished using at least two processes, one of said processes collecting the hypertext markup language text and the linked graphic format files and another of said processes rendering the graphic files as a presentation of respective said data files. 
     
     
       5. The method of  claim 4 , comprising a greater number of said processes collecting the files than a number of said processes rendering the presentation. 
     
     
       6. The method of  claim 4 , comprising storing in a buffer each of the files collected by said processes collecting the files, queuing the process for generating the image, and deleting the files in the buffer after generating the image. 
     
     
       7. The method of  claim 4 , comprising operating said process rendering the presentation file using at least part of a computer's display facility to produce a bitmap, and converting the bitmap into a graphic format file. 
     
     
       8. The method of  claim 4 , comprising operating said process rendering the presentation file by reducing a display size of the bitmap and converting the bitmap into a graphic format file. 
     
     
       9. The method of  claim 2 , wherein said reporting of the hit list comprises composing a hypertext report page containing selectable links for addressing corresponding said data files, and transmitting the hypertext report page to a user submitting a query, and wherein the report page additionally includes an image link addressing the graphic file for at least a portion of the hits. 
     
     
       10. The method of  claim 1 , further comprising stripping at least one variable aspect of the data files, said aspect comprising at least one of a time changing display feature, a user interactive feature and a nonvisual media feature. 
     
     
       11. A network search engine for managing user selection of information contained on data files stored at distributed network addresses on a global information processing network wherein distributed users have control over associated data files accessible by other users, each of said data files having at least some associated text and each of the data files having at least one mode of graphic presentation, comprising:
 a crawler having at least one processor operable to address and load successive data files comprising at least a subset of said data files stored at said distributed network addresses, the crawler being operable to produce and store a database of information characterizing aspects of the data files that tend to distinguish the data files from one another, cross referenced to addresses of the data files; and,   wherein the crawler is further operable to produce graphic image files representing at least some of the data files, the graphic image files each corresponding to content of corresponding said data files at a point in time, and wherein the crawler is operable to store the graphic image file so as to cross reference the graphic image file to the data files in the database.   
     
     
       12. The network search engine of  claim 11 , further comprising programmed processes operable to receive search queries from network users, to apply the search queries to the database for selecting a hit list from among the data files and to report the hit list in a search report including the addresses of each of the data files selected and the image corresponding to the data files in the hit list at the respective time of generation. 
     
     
       13. The network search engine of  claim 11 , wherein the data files comprise hypertext markup language text and linked graphic format files, and wherein said analyzing comprises at least one of: indexing the text for storing a text index cross referenced to network addresses of the data files; and reviewing at least a portion of the data files for assignment of an arbitrary categorization and for storing a categorization cross referenced to the network addresses of the data files. 
     
     
       14. The network search engine of  claim 11 , wherein the crawler operates at least two discrete processes for collecting the data files and files linked thereto, and for producing the graphic image files. 
     
     
       15. The network search engine of  claim 14 , wherein the discrete processes operate together on at least one processor, and wherein the processes for collecting are more numerous than at least one said process for producing the graphic image files. 
     
     
       16. The network search engine of  claim 15 , wherein the process for producing the graphic image file renders an image of the data files from downloaded copies of the hypertext markup language text and linked graphic format files, and converts a resulting display image file into the graphic format file. 
     
     
       17. The network search engine of  claim 16 , wherein the process for producing the graphic image file renders the image of the data files according to a configuration selected as a default configuration with respect to at least one of use of changing visual features, presentation of user interactive features, presentation of non-visual media, display pixel resolution, color pallette, color resolution, and use of colors. 
     
     
       18. The network search engine of  claim 17 , further comprising a programmed process for producing the graphic image file, which co-opts a display bitmap from a processor programmed to present the data files, and converts the bitmap to a compressed graphic format file stored on the search engine. 
     
     
       19. An improved Internet search engine for managing user search and selection of web pages stored at distributed systems coupled at network addresses to the Internet, the search engine having an associated web crawler operable to address and load successive web pages, and to index text data associated with said successive web pages so as to obtain parameter information that distinguishes at least groups of the web pages from one another, the crawler storing the parameter information and associated addresses of the web pages, and the search engine being operable responsive to user submitted search criteria to search the parameter information and to report at least the associated addresses of web pages that met the search criteria when indexed, wherein the improvement comprises:
 said crawler being operable in conjunction with obtaining the parameter information for at least a subset of said successive web pages to generate a graphic image file containing a visual image that is substantially identical to an appearance of said web pages, for display in a size proportionally smaller than said web pages; and   wherein the search engine is operable when reporting the associated addresses of web pages that met the search criteria to include a representation of the graphic image file in said proportionally smaller size.   
     
     
       20. The improved Internet search engine of  claim 19 , wherein the crawler generates the graphic image file with an appearance of the web pages according to a predetermined default display configuration of a browser. 
     
     
       21. The improved Internet search engine of  claim 20  wherein the predetermined default display configuration defines a selection of at least one of relative font size and type, colors and pixel aspect ratio. 
     
     
       22. The improved Internet search engine of  claim 21 , wherein the search engine reports to the user the associated addresses of the web pages that met the search criteria, in a form of hypertext source data containing URL links to said web pages, and wherein the graphic image file is displayed in association with a URL link to the web page represented by the graphic image file. 
     
     
       23. The improved Internet search engine of  claim 21 , wherein the graphic image file comprises a compressed pixel image of a bitmap corresponding to said web pages. 
     
     
       24. The improved Internet search engine of  claim 22 , wherein the graphic image file is transmitted as an image link in the hypertext source data to a file compressed by at least one of MIME, Binhex and Base64. 
     
     
       25. A system comprising:
 a fetching agent configured to receive a website file via at least one network interface, wherein the website file is associated with a web page;   a rendering agent configured to generate, based on the website file, a visual representation file that represents a rendered appearance of the web page that is substantially identical to an appearance of the web page and to compress the visual representation file of the web page into a reduced image file,
 wherein the reduced image file represents a reduced-size rendered appearance of the web page for display in a size proportionally smaller than the web page, and 
 wherein the rendering agent is further configured to limit a dynamic aspect of dynamic content in the website file to a static display, wherein the static display comprises an image from the dynamic content in the web page at a fixed time; 
   a memory, configured to store the reduced image file and at least one network address associated with a network location of the website file, wherein the memory is further configured to cross reference the reduced image file with the at least one network address; and   a first plurality of fetching agents and a second plurality of rendering agents, and wherein a ratio of the first plurality of fetching agents to the second plurality of rendering agents is modified during processing of website files to maintain a consumption of the memory within a range of fractions of a capacity of the memory.   
     
     
       26. The system of claim 25, wherein the fetching agent is further configured to utilize a plurality of concurrently active requests for web pages. 
     
     
       27. The system of claim 25, wherein the fetching agent is further configured to remove dynamic content from the website file. 
     
     
       28. The system of claim 25, wherein the web page includes nontext data. 
     
     
       29. The system of claim 25, further comprising:
 a search portal configured to:
 receive search criteria; and 
 generate a report based on the search criteria, the report comprising the at least one network address and one of:
 a link to the reduced image file, and 
 a copy of the reduced image file. 
 
   
     
     
       30. The system of claim 25, wherein the at least one network address comprises a uniform resource locator (URL) address of the web page. 
     
     
       31. The system of claim 25, wherein the static display comprises a frame of the dynamic content at the fixed time. 
     
     
       32. The system of claim 25, wherein the rendering agent is further configured to include an icon in the reduced image file indicating that the web page includes dynamic content. 
     
     
       33. The system of claim 32, wherein the icon indicates a type of the dynamic content. 
     
     
       34. A method comprising:
 receiving, at a computer, a website file via at least one network interface, wherein the website file is associated with a web page;   generating, at the computer, a visual representation file that represents a rendered appearance of the web page that is based on the website file, wherein the generating limits a dynamic aspect of dynamic content in the website file to a static display, wherein the static display comprises an image from the dynamic content in the web page at a fixed time;   compressing, at the computer, the visual representation file of the web page into a reduced image file, wherein the reduced image file represents a reduced-size rendered appearance of the web page;   storing, at the computer, the reduced image file and at least one network address associated with a network location of the website file, wherein storing the reduced image file and the at least one network address comprises cross referencing the reduced image file with the at least one network address; and   modifying a ratio of first plurality of fetching agents to second plurality of rendering agents during processing of website files to maintain a consumption of memory within a range of fractions of a capacity of the memory.   
     
     
       35. The method of claim 34, wherein the web page includes nontext data. 
     
     
       36. The method of claim 34, further comprising:
 receiving search criteria; and   generating a report based on the search criteria, the report comprising the at least one network address and one of:
 a link to the reduced image file, and 
 a copy of the reduced image file. 
   
     
     
       37. A non-transitory computer-readable storage medium having instructions stored thereon, the instructions comprising:
 instructions for receiving a website file via at least one network interface, wherein the website file is associated with a web page,   instructions for generating, based on the website file, a visual representation file that represents a rendered appearance of the web page, wherein the generating limits a dynamic aspect of dynamic content in the website file to a static display, and wherein the static display comprises an image from the dynamic content in the web page at a fixed time;   instructions for compressing the visual representation file of the web page into a reduced image file, wherein the reduced image file represents a reduced-size rendered appearance of the web page;   instructions for storing the reduced image file and at least one network address associated with a network location of the website file, wherein the instructions for storing the reduced image file comprise instructions for cross referencing the reduced image file with the at least one network address; and   instructions for modifying a ratio of first plurality of fetching agents to second plurality of rendering agents during processing of website files to maintain a consumption of memory within a range of fractions of a capacity of the memory.   
     
     
       38. The non-transitory computer-readable storage medium of claim 37, wherein the instructions for receiving, the instructions for generating, the instructions for compressing, and the instructions for storing are configured to operate on a single processor. 
     
     
       39. The non-transitory computer-readable storage medium of claim 37, wherein the instructions for receiving, the instructions for generating, the instructions for compressing, and the instructions for storing are configured to operate on multiple processors. 
     
     
       40. The non-transitory computer-readable storage medium of claim 39, wherein at least one of the multiple processors exclusively executes the instructions for receiving. 
     
     
       41. The non-transitory computer-readable storage medium of claim 37, wherein the web page includes nontext data. 
     
     
       42. The non-transitory computer-readable storage medium of claim 37, further comprising:
 instructions for receiving search criteria; and   instructions for generating a report based on the search criteria, the report comprising the at least one network address and one of:
 a link to the reduced image file, and 
 a copy of the reduced image file. 
   
     
     
       43. A non-transitory computer-readable storage medium having instructions stored thereon, the instructions comprising:
 instructions for receiving a file via at least one network interface, wherein the file includes formatting information;   instructions for generating, based on the formatting information, a visual representation file that represents a rendered appearance of the file, wherein the generating limits a dynamic aspect of dynamic content in the file to a static display, wherein the static display comprises an image from the dynamic content in a web page at a fixed time;   instructions for compressing the visual representation file into a reduced image file, wherein the reduced image file represents a reduced-size rendered appearance of the file;   instructions for storing the reduced image file and at least one network address associated with a network location of the file, wherein the instructions for storing the reduced image file comprise instructions for cross referencing the reduced image file with the at least one network address; and   instructions for modifying a ratio of first plurality of fetching agents to second plurality of rendering agents during processing of website files to maintain a consumption of memory within a range of fractions of a capacity of the memory.   
     
     
       44. The non-transitory computer-readable storage medium of claim 43, further comprising:
 instructions for receiving search criteria; and   instructions for generating a report based on the search criteria, the report comprising the at least one network address and one of:
 a link to the reduced image file, and 
 a copy of the reduced image file.