Stock Market Prediction Using Natural Language Processing

Abstract

A method of using natural language processing (NLP) techniques to extract information from online news feeds and then using the information so extracted to predict changes in stock prices or volatilities. These predictions can be used to make profitable trading strategies. Company names can be recognized and simple templates describing company actions can be automatically filled using parsing or pattern matching on words in or near the sentence containing the company name. These templates can be clustered into groups which are statistically correlated with changes in the stock prices. The system is composed of two parts: message understanding component that automatically fills in simple templates and a statistical correlation component that tests the correlation of these patterns to increases or decreases in the stock price. The methods can be applied to a broad range of text, including articles in online newspapers such as the Wall Street Journal, financial newsletters, radio & TV transcripts and annual reports. In an enhanced embodiment of the system statistical patterns in Internet usage data and Internet data such as newly released textual information on Web pages are further leveraged.

1 . A method of predicting investment behavior using trading software having a data analysis tool implementing natural language processing and an investment predictor implementing an investment prediction model, said trading software being executed by a processor so as to cause said processor to implement the steps of:
 said processor extracting information from news media relating to a particular investment using said natural language processing to parse or pattern match on words in said news media to identify natural language text describing activities or announcements of said particular investment that is in or near sentences containing a name of said particular investment and to automatically fill templates with said natural language text;   said processor using a clustering algorithm to cluster at least some of said templates into groups that are statistically correlated with changes in investment price of said particular investment;   determining a statistical significance of said changes in investment price of said particular investment based on information in said clustered templates; and   predicting changes in price of said particular investment based on new information about said particular investment if information of the type included in the new information has in the past caused a statistically significant change in the investment price in said particular investment.   
     
     
         2 . A method as in  claim 1 , said processor further using said clustering algorithm to cluster templates containing the extracted information of said particular investment with templates containing extracted information of another investment whose investment price has been shown to be similarly affected by information of the type included in said new information. 
     
     
         3 . A method as in  claim 1 , said processor further standardizing different references to the particular investment by different proper names, co-referencing when the particular investment is referred to by a pronoun, adding said new information to said template, and adding additional information about said particular investment to said template using databases and/or derived values. 
     
     
         4 . A method as in  claim 1 , said clustering algorithm further clustering templates containing information about different publicly traded companies into similar cluster groups, and said processor determining changes in investment price at different intervals for different companies in a cluster group in response to comparable information and estimating the statistical probability of a change in the investment price of said particular investment in response to certain new information statistically correlated to that in said comparable information. 
     
     
         5 . A method as in  claim 1 , comprising the further step of conducting a trade based on said predicted changes in investment price. 
     
     
         6 . A method as in  claim 1 , comprising the further steps of weighting textual attributes of said news media and of said natural language processing pattern matching said weighted textual attributes for current and previous news media to provide statistical feedback comparing current news media to previous new media to determine similarities between said current news media and said previous news media. 
     
     
         7 . A method as in  claim 1 , comprising the further step of weighting written advice from market experts in accordance with the historical behavior of the market with respect to that written advice. 
     
     
         8 . A method as in  claim 1 , wherein the investment is a stock.