Interactive web crawler

Abstract

The claimed subject matter provides a system or method for web crawling hidden files. An exemplary method comprises loading a web page with a browser agent, and executing any dynamic elements hosted on the web page using the browser agent to insert pre-determined values. A list of form controls may be retrieved from the web page using the browser agent, and the controls may be analyzed using a driver component. Form control values may be sent from the driver component to the browser agent, and an event may be submitted to the web page by the browser agent or scripted content may be run to trigger operations on the web page corresponding to the form control values. A URL may be generated for various form control values using a generalizer.

What is claimed is: 
     
         1 . A method of web crawling hidden files, comprising:
 loading a web page with a browser agent;   executing any dynamic elements hosted on the web page using the browser agent to insert pre-determined values;   retrieving a list of form controls from the web page using the browser agent;   analyzing the form controls using a driver component of a crawler;   sending form control values from the driver component to the browser agent;   submitting an event to the web page by the browser agent or running any scripted content to trigger operations on the web page corresponding to the form control values; and   generating a URL for various form control values using a generalizer.  1