Q&A: The Looming Big Data Skills Shortage
With big data everywhere, a skills shortage of workers with the necessary knowledge to work with it looms.
- By Linda L. Briggs
- April 23, 2013
Consulting firm McKinsey predicts a shortage by 2018 of 140,000 to 190,000 people with the deep analytical skills needed to work with big data. "I think it actually could be higher than that," says Dr. Betsy Page Sigman, a distinguished teaching professor at the McDonough School of Business at Georgetown University. In this interview, she talks about the coming talent shortage and what can be done to address it.
Sigman has also worked at George Mason University, the Bureau of the Census, The Institute for Research in the Social Sciences, Decision/Making/Information, and the Office of Research and Statistics of the Social Security Administration. Her teaching and research interests include e-commerce and social media, databases, and survey research.
She recently received the IBM Faculty Award and hosted a conference in Washington, DC in early April to discuss how schools should teach big data to students.
What's your goal in focusing on the topic of big data?
Sigman: I've always been interested in data; I've taught the database class here at Georgetown for years. I wanted to get more hands-on with big data, and I was lucky enough to get a faculty award from IBM. It's given me the time to get hands-on with some technologies around big data. I'm starting to develop a curriculum around it, along with our April conference.
Big data is a huge topic; instead of hashing over what it is, what it means, and so forth, I'm most concerned with educating the next generation about big data. Data is going to become bigger and bigger -- how are we equipped to deal with it? I'm spending time thinking about how to teach students about big data -- that's where I want to focus.
What kind of skills do you anticipate people needing as we move into working more and more with big data?
I have a list here: Statistics is one. Data visualization techniques is another -- that includes charting, graphing, and mapping. Simulation is another skill. Text analytics and voice analytics should be included, certainly, and social media analytics. That's just a start; I'm sure I'm leaving things out.
Looking at that list, there are plenty of areas in which people have expertise now.
Sure, some people have expertise, but it won't be nearly enough. McKinsey, the consulting firm, says there will be a shortage of talent necessary for organizations to take advantage of big data. By 2018, they say, the U.S. alone could face a shortage of 140,000 to 190,000 people with deep analytical skills -- I think the number actually could be higher than that -- as well as more that 1.5 million analysts and managers with the know-how to use big data and analytics correctly. I'd say we could go higher on that number also.
And remember, those numbers are only in the U.S. When we look at the business world from a global perspective, the deficit is even larger. It's huge.
What can companies do to address the deficit?
The good news is that [technology] companies ... are developing products that make it easier for academics to learn big data technologies, and then they can teach others. There's a lot that can be done even now.
Industry really needs to provide technologies to the business world to make it easier to use big data. They are certainly working on that, and making great strides. Data sets are just going to get bigger, so the technology needs to keep up, and it will.
How should companies get the expertise that they need now and will need even more in the future?
Some [of the more technical] companies can train their own workers, but many companies can't. They are going to have to go to companies that are good at training, and to look for outside help on how to set up systems properly, connect their data stores, and then use it all for analytics. A lot of companies are going to have to go somewhere for training. Many companies, of course, are using big data now, but more people are going to have to be trained.
Is it more a case of business people learning new technologies and understanding how to use analytics, or is it IT people acquiring additional business skills and knowledge?
It's going to be a combination. I see IT needing to work with business people in order to help them pull out the analytics they need. Business people need to understand something about analytics, and that will get better and easier as the technology gets better. They are both going to need to pull big roles.
What about the role of universities in preparing students? How well are universities in general preparing students for big data and data analysis?
Let me just say that there is a need for more. We can clearly see that there is going to be a shortage of people to fill slots in industry, and academia can play an important role. It's very important, though, that [institutions] maintain academic rigor in this area, something that unfortunately can get lost when teachers are evaluated not on how rigorously they teach statistics or analytics, but on student ratings -- that isn't always the best judge, of course. There's also the heavy emphasis on research by professors. None of that is bad, but I hope universities ... are careful to maintain the necessary discipline and rigor around these topics.
I also think it will become easier to teach this subject as more tools and training come out. Students may be able to get good training from MOOCs -- massive open online courses -- for example. We're getting ready to experiment with those here at Georgetown. That's exciting, because MOOCs can help the whole learning system.
Could institutions be doing a better job of training people in these areas?
Yes, I think anyone would say that -- colleges and universities could be doing a better job. Also, I think it will help that people on college campuses are getting excited about big data. People have a personal interest in it, and how it can make things more efficient and help people make better decisions -- they can do all kinds of things with it if they harness it correctly.
With all the attention that big data is getting -- and I have to say, it's a bit odd that everyone has grabbed onto this single term "big data" -- but it's still a good thing. It's bringing a lot of attention to [data management in general.]
Are there specific areas or industries that will be impacted most heavily by the coming skills shortage, or are already feeling its impact? In particular, I'm thinking of healthcare.
I'm thinking of healthcare, too. That is an area that is so huge and has so many things that can be discovered within its data. As we get better at harnessing that data and getting people running all kinds of analytics on health data, we're going to find out so much that we don't know now. Just think of all the variables that go into people's lives: the food they eat, the ways they exercise, the checkups they get, and how it all affects them. Imagine that we knew how all those things interact and could pull out some things from that. How much better off could we all be? We don't know a lot of those things yet, we just don't. We just know the tip of the iceberg.