System and method for preventing automated crawler access to web-based data sources using a dynamic data transcoding scheme

Abstract

A protection system and associated method prevent the automatic crawler access to a company's web-based data, without impacting the ability of an interactive user, such as a consumer, to access the data and to conduct regular business transactions. In one embodiment, the protection system will not prevent the crawler from downloading data; rather, the data will be rendered non-extractable by the crawler. In another embodiment, the protection system will prevent crawler access to the data. To this end, the protection system uses any one or a combination of the following six transcoding techniques: Transcoding technique that changes the web page structure; transcoding technique that changes the web page content; transcoding technique that selectively changes web page variable names; transcoding technique that selectively converts text to images in the web page; transcoding technique that alters form values when executed; and/or transcoding technique that generates a substantial portion of, or the entire web page when executed.

1. A system for preventing automated crawler access to data from a network-based data source, comprising:
 a transcoding proxy for automatically permutating data retrieved from the data source, to render the data uninterpretable by the crawler, while allowing a browser to render data retrieved from the data source; 
 wherein the transcoding proxy utilizes a transcoding technique that dynamically changes a form structure; 
 wherein the transcoding technique changes the form structure by shifting the position of the data in the form; 
 wherein the transcoding technique shifts the position of form data by inserting any one or more of: a character, a string of characters, a space, non-textual data, a blank row, a blank column, and/or an empty table; and 
 wherein the transcoding technique shifts the position of form data by nesting the data in a table within a table. 
 
   
   
     2. The system according to  claim 1 ,
 wherein the transcoding proxy concurrently permits a browser to access the data source. 
 
   
   
     3. The system according to  claim 1 , wherein the transcoding proxy utilizes a transcoding technique that dynamically changes a form content. 
   
   
     4. The system according to  claim 1 , wherein the data retrieved from the data source include a variable with a name; and
 wherein the transcoding proxy utilizes a transcoding technique that dynamically changes the name of the variable. 
 
   
   
     5. The system according to  claim 1 , wherein the data retrieved from the data source include a textual content; and
 wherein the transcoding proxy utilizes a transcoding technique that dynamically changes the textual content into non-textual content. 
 
   
   
     6. The system according to  claim 5 , wherein the transcoding technique replaces the textual content by a corresponding image, in order to render selected terms difficult to be searched automatically by the crawler. 
   
   
     7. The system according to  claim 1 , wherein the data retrieved from the data source includes a form containing a variable with a value;
 wherein the transcoding proxy utilizes a transcoding technique that incorporates an executable application in the form; and 
 wherein when the executable application is implemented, it dynamically alter the value of the variable. 
 
   
   
     8. The system according to  claim 7 , wherein the variable includes a label associated with the variable;
 wherein the transcoding technique further transforms the label into a synthetic label; and 
 wherein the executable application recalculates the variable value in order to prevent a server from authenticating the crawler access to the data source. 
 
   
   
     9. The system according to  claim 1 , wherein the data retrieved from the data source is collected in a page;
 wherein the transcoding proxy utilizes a transcoding technique that incorporates an executable application in the page; and 
 wherein when the executable application is implemented, it dynamically generates the page and renders the data. 
 
   
   
     10. The system according to  claim 9 , wherein the transcoding technique renders the data using an HTML code. 
   
   
     11. A system for preventing automated crawler access to data from a network-based data source, comprising:
 a transcoding proxy for automatically permutating data retrieved from the data source, to render the data uninterpretable by the crawler, while allowing a browser to render data retrieved from the data source; 
 wherein the transcoding proxy utilizes a transcoding technique that dynamically changes a form content; and 
 wherein the transcoding technique changes the form content by adding one or more content inserts that are substantially imperceptible to a browser user, in order to render selected terms in the page content difficult to be searched automatically by the crawler. 
 
   
   
     12. The system according to  claim 11 , wherein the content inserts include any one or more of: a characters, a string of characters, non-textual data, and/or a string of images. 
   
   
     13. The system according to  claim 12 , wherein the transcoding technique calculates a synthetic name from a variable name, and replaces the variable name with the synthetic name, in order to render selected terms difficult to be searched automatically by the crawler. 
   
   
     14. The system according to  claim 13 , wherein the variable is contained in a data entry form;
 wherein the data entry form further includes a session ID; and 
 wherein the transcoding technique recalculates the variable name from the synthetic name to enable the data to be rendered by the browser. 
 
   
   
     15. A method for preventing automated crawler access to data from a network-based data source, comprising the steps of:
 automatically permutating data retrieved from the data source, to prevent a crawler from automatically accessing the data source by rendering the data uninterpretable by the crawler, while allowing a browser to render data retrieved from the data source; and 
 utilizing a transcoding technique that dynamically changes a form structure; 
 wherein the transcoding technique changes the form structure by shifting the position of the data in the form; and 
 wherein the transcoding technique shifts the position of form data by nesting the data in a table within a fable. 
 
   
   
     16. The method according to  claim 15 , wherein the data contain any one or more of: a variable with a name, a textual content, or a value;
 wherein the data are collected in one or more of: a form or a page; and 
 wherein automatically permuting data and preventing crawler access include using one or more of the following transcoding techniques: 
 a first transcoding technique that dynamically changes a form structure; 
 a second transcoding technique that dynamically changes a form content; 
 a third transcoding technique that dynamically changes the name of the variable; 
 a fourth transcoding technique that dynamically changes the textual content into a non-textual content; 
 a fifth transcoding technique that incorporates an executable application in the form, and wherein when the executable application is implemented, it dynamically alter the value of the for variable; and/or 
 a sixth transcoding technique that incorporates an executable application in the page, and wherein when the executable application is implemented, it dynamically generates the page and renders the data. 
 
   
   
     17. A system for preventing automated crawler access to data from a network-based data source, comprising:
 a transcoding proxy for automatically permutating data retrieved from the data source, to render the data uninterpretable by the crawler, while allowing a browser to render data retrieved from the data source; 
 wherein the transcoding proxy utilizes a transcoding technique that dynamically changes a form structure; 
 wherein the transcoding technique changes the form structure by shifting the position of the data in the form; and 
 wherein the transcoding technique shifts the position of form data by nesting the data in a table within a table. 
 
   
   
     18. A computer program product including a plurality of executable instruction codes stored on a computer readable medium, for preventing automated crawler access to data from a network-based data source, comprising:
 a first set of instruction codes for automatically permutating data retrieved from the data source, to render the data uninterpretable by the crawler, while allowing a browser to render data retrieved from the data source; 
 a second set of instruction codes for dynamically changing a form structure; 
 wherein the second set of instruction codes changes the form structure by shifting the position of the data in the form; and 
 wherein the second set of instruction codes shifts the position of form data by nesting the data in a table within a table.