Recovery of Information from Commercial Web Portals

Abstract

Novel tools and techniques for automated recovery of information from commercial web portals, including commercial web portals requiring credentials for access. In some instances images are captured and pushed to external processes for improving system performance. In some instances access to automated software agent remote control modules is balanced across a system comprising a plurality of processors hosting the automated software agent remote control modules. Some instances provide provisioning of credentials, in particular indicating credentials available for an unlimited or a select limited number of users and monitoring credential use of those credentials usable by a select number of users. Some instances provide cache management optimizes retrieval of data by external processes and ensures reliability of such data to reduce unnecessary web portal inquiries.

What is claimed is: 
     
         1 . A method of utilizing a computer cache to optimize obtaining time variable information from a commercial web portal using an automated software agent, the method comprising:
 accessing, with an automated software agent, a web portal containing time variable information;   obtaining, with the automated software agent, a set of elements of time variable information;   date stamping the set of time variable information;   storing the set of time variable information with the date stamp in a computer cache as a reference set of elements of time variable data;   accessing, with the automated software agent, the web portal containing the time variable information;   obtaining, with the automated software agent, a later set of elements of the time variable information;   date stamping the later set of elements of the time variable information;   comparing, with the automated software agent, the reference set of elements of time variable information and the later set of elements of the time variable information to identify any changes between common elements of the reference set of elements of time variable information and the later set of elements of the time variable information;   replacing the elements of the reference set of time variable information with common elements of the later set of the time variable information in the computer cache if there are differences between the common elements of the reference set of time variable information and the later set of the time variable information;   analyzing each element of the reference set of elements of time variable information and the corresponding element of the later set of elements of the time variable information as a function of the difference between date stamps to establish criteria for determining whether the automated software agent can rely on the reference set of data in the cache in a future query to obtain an element of time variable information or whether the element of time variable information must be accessed from the web portal; and   associating and storing the criteria with each element of the reference set of time variable data in a database.   
     
     
         2 . The method of  claim 1  wherein the step of analyzing each element of the reference set of elements further comprises storing information about comparisons in a database and evaluating information in this database to optimize criteria of data type and age for determining whether the automated software agent can rely on the reference set of data in the cache in future queries; and
 associating and storing the optimized criteria with each element of the reference set of time variable data in a database. 
 
     
     
         3 . The method of  claim 1  further comprising adding any element of the later set of time variable information not in the reference set of elements of time variable information to the reference set of time variable information. 
     
     
         4 . A computer system comprising:
 one or more processors;   a database operatively associated with at least one of the one or more processors;   a cache operatively associated with at least one of the one or more processors;   an automated software agent operatively associated with at least one of the set of processors;   a set of instructions executable by at least one of the one or more processors, to cause the at least one of the one or more processors to perform one or more operations, the set of instructions comprising:   instructions for accessing, with an automated software agent, a web portal containing time variable information;   instructions for obtaining, with the automated software agent, a set of elements of time variable information;   instructions for date stamping the set of time variable information;   instructions for storing the set of time variable information with the date stamp in the computer cache as a reference set of elements of time variable data;   instructions for accessing, with the automated software agent, the web portal containing the time variable information;   instructions for obtaining, with the automated software agent, a later set of elements of the time variable information;   instructions for date stamping the later set of elements of the time variable information;   instructions for comparing, with the automated software agent, the reference set of elements of time variable information and the later set of elements of the time variable information to identify any changes between common elements of the reference set of elements of time variable information and the later set of elements of the time variable information;   instructions for replacing the elements of the reference set of time variable information with common elements of the later set of the time variable information in the computer cache if there are differences between the common elements of the reference set of time dependent information and the later set of the time variable information;   instructions for analyzing each element of the reference set of elements of time variable information and the corresponding element of the later set of elements of the time variable information as a function of the difference between date stamps to establish criteria for determining whether the automated software agent can rely on the reference set of data in the cache in a future query to obtain an element of time variable information or whether the element of time variable information must be accessed from the web portal; and   instructions for associating and storing the criteria with each element of the reference set of time variable data in a database.   
     
     
         5 . An apparatus comprising:
 a computer readable medium having stored thereon a set of instructions executable by one or more computers to perform one or more operations, the set of instructions comprising:   instructions for accessing, with an automated software agent, a web portal containing time variable information;   instructions for obtaining, with the automated software agent, a set of elements of time variable information;   instructions for date stamping the set of time variable information;   instructions for storing the set of time variable information with the date stamp in a computer cache as a reference set of elements of time variable data;   instructions for accessing, with the automated software agent, the web portal containing the time variable information;   instructions for obtaining, with the automated software agent, a later set of elements of the time variable information;   instructions for date stamping the later set of elements of the time variable information;   instructions for comparing, with the automated software agent, the reference set of elements of time variable information and the later set of elements of the time variable information to identify any changes between common elements of the reference set of elements of time variable information and the later set of elements of the time variable information;   instructions for replacing the elements of the reference set of time variable information with common elements of the later set of the time variable information in the computer cache if there are differences between the common elements of the reference set of time dependent information and the later set of the time variable information;   instructions for analyzing each element of the reference set of elements of time variable information and the corresponding element of the later set of elements of the time variable information as a function of the difference between date stamps to establish criteria for determining whether the automated software agent can rely on the reference set of data in the cache in a future query to obtain an element of time variable information or whether the element of time variable information must be accessed from the web portal; and   instructions for associating and storing the criteria with each element of the reference set of time variable data in a database.