Advanced Analytics: A Look Back at 2017 and What's Ahead for 2018
Next year will be the year of the three A's: AI, automation, and advancing analytics skills.
- By Fern Halper
- December 19, 2017
In 2017, advanced analytics maintained its momentum in the enterprise. Open source technologies such as R and Python gained ground, machine learning continued to pique interest, and use of the cloud became more mainstream. TDWI expects these technologies to continue to grow in importance. We also anticipate other advanced analytics hotspots in 2018.
2017 Trends
Open source. Open source has become quite popular, especially for big data and data science, because it is a low-cost source community for innovation. This appeals to many data scientists and analytics application developers -- especially those who like to code.
R has been in use for several decades, but a few years back it exploded on the scene anew with the advent of data science. During 2017, R maintained its popularity, and Python, an interpreted, interactive, easy-to-read, object-oriented scripting language, really gained steam. Stack Overflow, a site that helps developers solve coding questions, saw more visits to its site for Python-tagged questions than Java in 2017. We saw increased interest in Python at TDWI as well.
Commercial software vendors also saw the writing on the wall and fully embraced open source, integrating Python and R into their environments. Vendors also released "data science workbenches" that provide open source environments for data scientists to develop models and build analytics applications.
Machine learning. Machine learning -- the science of getting computers to act without being explicitly programmed -- has been and will continue to be an extremely hot topic.
Although this technology has also been around for decades, it continues to receive renewed interest as data volumes increase. In particular, deep learning, which uses algorithms to learn functions that can classify complex patterns, is gaining steam because organizations are interested in using it to classify images and sound. Machine learning is an important component of AI (artificial intelligence) that we see as a hot spot in 2018 (see below).
The cloud as part of a data strategy. The cloud has been hyped as the go-to platform for analytics for years, but at TDWI we saw resistance to it from a number of quarters, mostly based on security concerns.
In 2017, however, organizations seemed to realize that the cloud could be especially useful for analyzing big data. Cloud data warehouses gained popularity, for instance. Organizations began to understand that it makes sense to analyze data where it lands, and with new data sources such as machine data, this is often in the cloud. As organizations look to integrate disparate data for analytics, the cloud is becoming part of their data strategy.
2018 Anticipated Hot Spots
The trends above will continue to be important in 2018. However, there are other emerging trends that are worth noting.
The growth of AI and AI applications. The idea that machines could act "intelligently" has been around since the ancient Greeks, but there has been no real consensus about what artificial intelligence actually means. Back in the 1950s, when John McCarthy of Dartmouth College coined the term, he described it as, "Making a machine behave in ways that would be called intelligent if a human were so behaving." There has been debate about its meaning ever since. One thing is clear, though -- "AI" has become the buzzword du jour, and it will continue to be in 2018. Many vendors are hyping AI capabilities whether they have them or not.
AI and its subcomponents (including machine learning, deep learning, and natural language processing) are being woven into the analytics arsenal in marketing, sales, and operations across industries to increase insights and take action. This trend will continue in 2018.
Additionally, research in AI will continue with new algorithm development. We will see new companies forming around AI and vertical applications built that have machine learning or other AI components woven into them. We will also continue to see the rise of personal assistants and (probably much to our dismay) chatbots becoming ubiquitous. The bar on what is considered AI will continue to rise.
Automating the analytics life cycle. The analytics life cycle starts with data ingestion and continues into deployment. In 2017, we heard a lot more from vendors about embedding advanced analytics across the entire analytics life cycle. In other words, vendors are using advanced technology to perform tasks that normally required human intervention.
For instance, machine learning is being used in data integration to automatically identify schemas or metadata. It is being used in data preparation for data cleansing and profiling or to suggest transformations. It is being used in visualization to suggest what to plot. It is being used in predictive analytics to automatically build a model. The idea is to make analytics "smarter" so that it is easier for everyone in the business to make use of analytics, continuing the democratization trend we've seen over the past few years. This trend made market noise in 2017 and will make bigger noise in 2018.
Advancing analytics skills/organizational processes. As the smart/automation/ intelligent app phenomenon mentioned above continues to march ahead, organizations will realize that their tools shouldn't be smarter than they are. It is one thing to automate the building of a predictive model when you're exploring your data. It is another thing entirely to put that model into production if you don't know what it actually means.
The automation phenomenon is in the early adoption stage, and as organizations continue to adopt it, they will realize that it can't stand alone. Perhaps this realization will occur when a model that was built automatically has issues or the marketing director can't explain his model to the head of the organization. Once it costs the company money, organizations will wake up to the fact that either the automatically built models need to explain their output better or people need the skills to interpret model output -- or both. In either case, 2018 should be a year of reckoning.
The Bottom Line
Much of what was old is new again, thanks to some new algorithm development and the processing power to make advanced analytics buzz. Advanced analytics in 2018 will build upon what we've seen hyped in 2017. TDWI will be following these trends.
About the Author
Fern Halper, Ph.D., is well known in the analytics community, having published hundreds of articles, research reports, speeches, webinars, and more on data mining and information technology over the past 20 years. Halper is also co-author of several “Dummies” books on cloud computing, hybrid cloud, and big data. She is VP and senior research director, advanced analytics at TDWI Research, focusing on predictive analytics, social media analysis, text analytics, cloud computing, and “big data” analytics approaches. She has been a partner at industry analyst firm Hurwitz & Associates and a lead analyst for Bell Labs. Her Ph.D. is from Texas A&M University. You can reach her at [email protected], on Twitter @fhalper, and on LinkedIn at linkedin.com/in/fbhalper.