Automated financial data aggregation

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for data aggregation. The methods, systems, and apparatus include determining whether a site-specific script for extracting financial data from a particular financial institution website is available; in response to determining that a site-specific script for extracting financial data from the particular financial institution website is not available, generating a site map of web pages and web page segments in the financial institution website, wherein the site map is generated based on at least in part on a statistical analysis of web pages and web page segments that are not in the financial institution website; generating, based on the site map of the financial institution website, a site-specific script for extracting financial data from the financial institution website; and extracting, for one or more users, financial data from the particular financial institution website using the generated site-specific script.

What is claimed is: 
     
       1. A computer-implemented method, comprising:
 crawling a financial institution website to identify a plurality of web pages in the financial institution website; 
 for each webpage of at least some of the plurality of web pages, determining a category for the webpage from a plurality of web page categories by inputting the web page into a web page classifier that includes a first data model that has been trained using a machine learning technique based on a plurality of webpages that are not from the financial institution website, the plurality of web page categories including at least a login page, an accounts summary page, and an account page; 
 for each segment of at least some segments from the plurality of web pages, determining a category for the segment from a plurality of segment categories by inputting the segment into a segment classifier that includes a second data model that has been trained using a machine learning technique based on a plurality of segments that are not from the financial institution website, the plurality of segment categories including at least a login field, an accounts description table, and a transaction descriptions table; 
 generating, automatically and in the computer, a site-specific script for extracting financial data from the financial institution website based on the determined categories for the plurality of web pages and segments in the financial institution website; and 
 extracting, in the computer, for one or more users, financial data from the financial institution website using the generated site-specific script. 
 
     
     
       2. The method of  claim 1 , wherein the first data model implements a naive Bayes classification. 
     
     
       3. The method of  claim 1 , wherein determining the category for the webpage includes:
 determining a respective plurality of scores for the web page, each score in the plurality of scores indicating a confidence that the web page corresponds to a particular category; 
 determining for each web page whether a score in the respective plurality of scores satisfies a threshold; and 
 in response to determining that a score in the respective plurality of scores for the web page satisfies a threshold, associating the web page with a particular category corresponding to the score. 
 
     
     
       4. The method of  claim 3 , further comprising:
 in response to determining that no score in the respective plurality of scores for a web page satisfies a threshold, providing the web page to a user for manual categorization; and 
 associating the web page with a category specified by the user. 
 
     
     
       5. The method of  claim 1 , wherein the second data model implements a nave Bayes classification. 
     
     
       6. The method of  claim 1 , wherein determining the category for the segment includes:
 determining a respective plurality of scores for the segment, each score in the plurality of scores indicating a confidence that the segment corresponds to a particular category; 
 determining for each segment whether a score in the respective plurality of scores satisfies a threshold; and 
 in response to determining that a score in the respective plurality of scores for the segment satisfies a threshold, associating the segment with the a particular category corresponding to the score. 
 
     
     
       7. The method of  claim 6 , further comprising:
 in response to determining that no score in the respective plurality of scores for a segment satisfies a threshold, providing the segment to a user for manual categorization; and 
 associating the segment with a category specified by the user. 
 
     
     
       8. The method of  claim 1 , further comprising:
 prior to crawling the financial institution website determining, in the computer, that a site-specific script for extracting financial data from the financial institution website is not available. 
 
     
     
       9. A computer program product, comprising a non-transitory computer-readable medium encoded with instructions to cause a data processing apparatus to:
 crawl a financial institution website to identify a plurality of web pages in the financial institution website; 
 for each webpage of at least some of the plurality of web pages, determine a category for the webpage from a plurality of web page categories by inputting the web page into a web page classifier that includes a first data model that has been trained using a machine learning technique based on a plurality of webpages that are not from the financial institution website, the plurality of web page categories including at least a login page, an accounts summary page, and an account page; 
 for each segment of at least some segments from the plurality of web pages, determine a category for the segment from a plurality of segment categories by inputting the segment into a segment classifier that includes a second data model that has been trained using a machine learning technique based on a plurality of segments that are not from the financial institution website, the plurality of segment categories including at least a login field, an accounts description table, and a transaction descriptions table; 
 automatically generate a site-specific script for extracting financial data from the financial institution website based on the categories for the plurality of web pages and segments in the financial institution website; and 
 extracting, for one or more users, financial data from the financial institution website using the generated site-specific script. 
 
     
     
       10. The computer program product of  claim 9 , wherein the first data model implements a naive Bayes classification. 
     
     
       11. The computer program product of  claim 9 , wherein the instructions to determine the category for the webpage include instructions to:
 determine a respective plurality of scores for the web page, each score in the plurality of scores indicating a confidence that the web page corresponds to a particular category; 
 determine for each web page whether a score in the respective plurality of scores satisfies a threshold; and 
 in response to determining that a score in the respective plurality of scores for the web page satisfies a threshold, associate the web page with a particular category corresponding to the score. 
 
     
     
       12. The computer program product of  claim 11 , further comprising instructions to:
 in response to determining that no score in the respective plurality of scores for a web page satisfies a threshold, provide the web page to a user for manual categorization; and 
 associate the web page with a category specified by the user. 
 
     
     
       13. The computer program product of  claim 9 , wherein the second data model implements a naive Bayes classification. 
     
     
       14. The computer program product of  claim 9 , wherein the instructions to determine the category for the segment include instructions to:
 determine a respective plurality of scores for the segment, each score in the plurality of scores indicating a confidence that the segment corresponds to a particular category; 
 determine for each segment whether a score in the respective plurality of scores satisfies a threshold; and 
 in response to determining that a score in the respective plurality of scores for the segment satisfies a threshold, associate the segment with the a particular category corresponding to the score. 
 
     
     
       15. The computer program product of  claim 14 , further comprising instructions to:
 in response to determining that no score in the respective plurality of scores for a segment satisfies a threshold, provide the segment to a user for manual categorization; and 
 associate the segment with a category specified by the user. 
 
     
     
       16. The computer program product of  claim 9 , further comprising instructions to:
 prior to crawling the financial institution website, determine that a site-specific script for extracting financial data from the financial institution website is not available.