TDWI | Training & Research | Business Intelligence, Analytics, Big Data, Data Warehousing

Analytics in the Stock Market: Made for Each Other

Despite the abundance of numerical patterns, the stock market is one of the more difficult predictive challenges.

• By Brian J. Dooley
• October 6, 2016

It's coming.

The ability to accurately predict stock market activity has been the holy grail of analytics and machine learning for decades. Algorithmic trading has already had an effect on the market, being directly responsible for the Flash Crash (a brief swing of over 1,000 points on May 6, 2010) in the Dow Jones Industrial Average.

Can Stock Markets Be Predicted with Big Data?

Stock exchanges are big piles of numbers, with large rewards for any edge in predicting movement. Big data and social media analytics, combined with emergent deep learning for pattern forecasting, are a natural fit. This could make it very difficult for any investor who lacks access to the latest software and systems, but do we really need to worry about that yet?

Yes and no. The stock market is tricky. It's one of the more difficult predictive challenges; there are inherent numerical patterns as well as a wide range of interactions with the outside world. As a market, it is continuously adjusting to the latest prediction theories in continuous coevolution.

On the other hand, "winning" doesn't require a great deal of accuracy or a high percentage of correct guesses. You only need to beat the odds. To make a profit, you would need to be slightly faster and slightly more accurate in determining trends for a given basket of stocks than most other investors.

Program trading has been around long enough to create serious issues for the market, leading to regulation affecting trading to this day. There have also been innumerable attempts to guess the markets based on the numbers alone, some of which have been moderately successful. The current round of predictive analytics has some major differences.

Modern Prediction Strategies

Today's stock market prediction strategies combine numeric pattern analysis with sentiment analysis from social networks. This opens the door to new predictive models based on tweets about companies and markets, news source analysis, and a mountain of unstructured data that could anticipate how people and markets are about to react.

If successful, this could provide an edge. In fact, it's similar to the type of analysis being used for fantasy sports.

Many people are looking at the problem. It is hugely complicated, and nobody really knows what type of sentiment analysis will work. An Israeli company, I Know First, is using artificial intelligence and machine learning to provide daily investment forecasts based on a combination of big data analysis, financial modeling, and sentiment analysis.

Other companies and projects are still trying to determine the best way to read the real world. For example, Newz Wiz has an alpha-stage project that creates visualizations based on social media feeds; others have used Google search terms, Wikipedia entries, or Twitter.

There are even psychological factors to consider -- some research has indicated that stock performance can be affected by whether the name of the stock is a pronounceable word!

No Easy Money

If anyone did figure out a foolproof prediction method, they would be unlikely to share what they were doing. For one thing, every other investor with the capability would immediately do the same thing and the stock market would instantly adjust. There would no longer be an advantage, and prediction would once again be impossible.

Are we there yet? Perhaps we will never know!