NewsInn is an Artificial Intelligence Algorithm that parses the major news publications for news articles, groups them based on the main story, and presents them in order of importance.
To do a sentiment analysis on each article, the algorithm used a derivative of SentiWord, but it quickly became clear that the library was not created to be used in a news Environment. So a new library was developed, that would better suit the case it was being used in. It manages to provide a good estimate of the impact that the text of an article has.
Since the stock market is heavily affected by world problems, and therefore, world news, it is highly likely that a general sentiment analysis of the news would produce a result directly related to the general sentiment of the public on the world affairs. This would theoretically, in time, affect the stock market. The time-to market of world news we estimated to be of a few days, since only huge golbal events have the power to change the stock price in-day.
Therefore we started testing at a one-day offset, then went on for two, three, and even four days offset between the news data and the stock data. We only kept the data that provided the highest accuracy in predicting the stock market indexes flow.
ABOUT NEWSINN
Abbildung in dieser Leseprobe nicht enthalten
Description
NewsInn is an Artificial Intelligence Algorithm that parses the major news publications for news articles, groups them based on the main story, and presents them in order of importance. [1] It was design to become a news aggregator at first, but since important stories are covered be most publications, an Artificial Intelligence algorithm was implemented in order to group such articles.
Opinion Extraction
To do a sentiment analysis on each article, the algorithm used a derivative of SentiWord [2], but it quickly became clear that the library was not created to be used in a news Environment. So a new library was developed, that would better suit the case it was being used in. It manages to provide a good estimate of the impact that the text of an article has (see image 1).
Abbildung in dieser Leseprobe nicht enthalten
Image 1. An example of an article’s Sentiment Analysis Result. Partial screenshot of http://newsinn.org/display/mostnegativeweek/ on 2015/10/18
ABOUT STOCK INDEXES
Definition
A stock index or stock market index is a measurement of the value of a section of the stock market. It is computed from the prices of selected stocks (typically a weighted average). It is a tool used by investors and financial managers to describe the market, and to compare the return on specific investments. [3]
Theory
Since the stock market is heavily affected by world problems, and therefore, world news, it is highly likely that a general sentiment analysis of the news would produce a result directly related to the general sentiment of the public on the world affairs. This would theoretically, in time, affect the stock market. The time-to market of world news we estimated to be of a few days, since only huge global events have the power to change the stock price in-day.
Therefore we started testing at a one-day offset, then went on for two, three, and even four days offset between the news data and the stock data. We only kept the data that provided the highest accuracy in predicting the stock market indexes flow.
Data Collection
Taking into account the fact that the final version of the NewsInn A.I. Algorithm started gathering information, on the 14th of September, the NewsInn algorithm checked more than 24,000 articles against the Opinion Extraction library. We used this data to gather an average opinion score of the all the news written that day.
The opinion, however, was extracted from the conglomerated articles. We found no resemblance between the raw article opinion and the indexes. The explanation relies probably in the idea that people only register one event, even if, in some cases, it is detailed in more than 20 articles.
The Data extracted can be found below. Please note that week-end data was removed, as it is not highly accurate (contains less than half the number of the articles that a week day has), and the fact that Stock Markets are closed during week-ends.
Abbildung in dieser Leseprobe nicht enthalten
Table 1. Result of the Average Opinion of Conglomerated Articles (excluding week-ends)
We used the historical prices of Dow Jones Index , the S&P 500 Index and the NASDAQ Index for analysis inside of the graphs, while offsetting them by two days in the case of Dow Jones and NASDAQ, and one day for S&P 500, to accommodate the prediction time of the algorithm. [4]
We arrived at the following data, normalized for ease of use:
Abbildung in dieser Leseprobe nicht enthalten
Table 2. Table used to plot the graphs and extract data
Were data could not be found, we duplicated the last known data. That is, the last row of the table. But this row was not taken into account when doing the calculations. Please see below in blue (on the right).
RESULTS
Dow Jones Index
The prediction, two days in advance , for the Dow Jones Index had the highest degree of accuracy, with an average prediction rate of 69.44%. For 4 of the days, for unknown reasons, the prediction was offset by a day, so it could only predict the moving of the index in a one day interval. This was probably due to the refugee crisis.
For the offsetted case we divided the number of days correctly predicted by a factor of two.
Abbildung in dieser Leseprobe nicht enthalten
Image 2. Results of Dow Jones Prediction Two Days Ahead
Abbildung in dieser Leseprobe nicht enthalten
NASDAQ Index
The smallest amount of success in predicting the Stock Market Indexes was for NASDAQ. this index as well had an almost 100% rate of prediction for the last week. Since no data was available for the 23rd of October, the last day of the graph was not taken into consideration.
Abbildung in dieser Leseprobe nicht enthalten
Image 3. Results of NASDAQ Prediction Two Days Ahead
Abbildung in dieser Leseprobe nicht enthalten
S&P 500 Index
The S&P 500 index was predicted with an accuracy rate of 66.66%. That is 2 out of 3 predictions made by the algorithm were correct. As with the previous indexes, the last two weeks showed a very high accuracy, missing only two out of 16 days. The issue with this was that it could only predict the Stock Market fluctuation only a day ahead.
Abbildung in dieser Leseprobe nicht enthalten
Image 4. Results of Dow Jones Prediction Two Days Ahead
Abbildung in dieser Leseprobe nicht enthalten
CONCLUSIONS
Abbildung in dieser Leseprobe nicht enthalten
As predicted, Stock Markets are heavily influenced by the news, and therefore, a sentiment analysis on them can indeed predict the fluctuation with a medium degree of accuracy. For our experiment, the medium prediction was the following
Abbildung in dieser Leseprobe nicht enthalten
All indexes had an almost 100% rate of prediction for the last week. Since the algorithm is not yet in its final form, we can expect these numbers to go higher. It also helps the fact that the more data goes into the system, the more the NewsInn Algorithm learns, and the better associations it can do.
FURTHER RESEARCH
Abbildung in dieser Leseprobe nicht enthalten
In order to thoroughly test the Artificial Intelligence Algorithm, and its prediction on the stock market, we will need to wait some more time, so that it can collect more news data. Even if it was tested against 24,000 articles, we do not feel this has enough significance. We will keep monitoring and hope that, in six months’ time, NewsInn will have collected enough information to properly test it.
SOURCES
Abbildung in dieser Leseprobe nicht enthalten
[...]
[1] NewsInn Website http://newsinn.org/
[2] SentiWordNet http://sentiwordnet.isti.cnr.it/
[3] Wikipedia - Stock market index page https://en.wikipedia.org/wiki/Stock_market_index
Frequently asked questions
What is NewsInn?
NewsInn is an Artificial Intelligence Algorithm designed to parse major news publications, group articles based on the main story, and present them in order of importance. It started as a news aggregator but evolved to use AI to group related articles.
How does NewsInn extract opinions from news articles?
NewsInn utilizes a custom-developed library, a derivative of SentiWord, designed to analyze the sentiment within news articles. This library provides an estimate of the impact that the text of an article has.
What is a stock index?
A stock index, or stock market index, is a measurement of the value of a section of the stock market. It's computed from the prices of selected stocks, typically a weighted average, and is used by investors to describe the market and compare investment returns.
What is the theory behind using news sentiment to predict stock market movements?
The theory is that the stock market is heavily affected by world problems and news. A general sentiment analysis of the news would reflect public sentiment on world affairs, which could influence the stock market over time.
How was data collected for this analysis?
The NewsInn A.I. Algorithm gathered data from over 24,000 articles, extracting an average opinion score for each day's news. Historical prices of the Dow Jones Index, S&P 500 Index, and NASDAQ Index were used for analysis, offset by one to two days to account for the prediction time of the algorithm.
What stock indexes were analyzed in relation to news sentiment?
The analysis focused on the Dow Jones Index, S&P 500 Index, and NASDAQ Index.
What were the results of the Dow Jones Index prediction?
The Dow Jones Index prediction, with a two-day lead time, had the highest degree of accuracy, with an average prediction rate of 69.44%.
What were the results of the NASDAQ Index prediction?
The prediction for the NASDAQ Index showed the smallest amount of success compared to the other indexes. The algorithm had almost a 100% prediction rate for the last week of the tested period.
What were the results of the S&P 500 Index prediction?
The S&P 500 index was predicted with an accuracy rate of 66.66%. The last two weeks showed a very high accuracy, missing only two out of 16 days.
What were the overall conclusions of the study?
Stock markets are heavily influenced by the news. Sentiment analysis of news can predict market fluctuations with a medium degree of accuracy.
What further research is needed?
More data needs to be collected to thoroughly test the Artificial Intelligence Algorithm and its prediction accuracy on the stock market. The current data set of 24,000 articles is considered insufficient. Continued monitoring and data collection are necessary.
Where can I find the sources used in this document?
The sources are listed at the end of the document, including the NewsInn website, SentiWordNet, Wikipedia's Stock Market Index page, and Yahoo Finance Historical Quotes.
- Quote paper
- Radu Nicoara (Author), 2015, Using The NewsInn Algorithm To Predict Stock Market Movements Two Days In Advance, Munich, GRIN Verlag, https://www.grin.com/document/309225