TDWI | Training & Research | Business Intelligence, Analytics, Big Data, Data Warehousing

Analysts in Action: The Rise of the Intelligent Analyst

Far from being “crushed” by IT, intelligent analysts seem to be emerging stronger than ever in business.

• By Ted Cuzzillo
• March 2, 2011

[Editor's Note: BI This Week is pleased to welcome columnist Ted Cuzzillo to our newsletter.  Ted is a former contributor and we welcome his insights into the roles, responsibilities, and activities of today's BI business analyst.]

An IT worker said to a data analyst on the business side, "You're cowboying this in" -- referring to the analyst's round-up and inclusion of third-party data. The analyst explained that this data, running on the cheap PC he tried to hide under a desk, was the only way to answer his boss' questions. No dice, the IT man warned, "You'll be crushed."

The man who overheard that confrontation a year ago explains that to many data analysts, the hunt for good data is "Machiavellian." They'll use any means to get the data they need. Many see themselves up against an immoral foe, the IT department, whose stingy nature and slothful response seem more suited to government bureaucracy than modern American business.

IT's ferocious opposition to creative analysts and their "cowboy" data seemed typical two years ago when I first looked at the subject. Have things changed? In my informal survey, I found improvement: more respect, more access, and even enthusiasm from old foes.

In fact, the IT worker who threatened to "crush" the "cowboying" analyst has left. The analyst, still there, now has more demand for his analysis than he can handle.

Those who fight third-party data did so for good reasons: it creates more silos, and puts good data at risk of going bad. "But if that's what business wants," says one sales manager at a major manufacturer who asked for anonymity, "that's what business gets." Everyone, he says, knows business needs more control of its data.

Gaining that control, at least in the cases I heard, started with disruption. When senior data analyst Andy Cotgreave started at the University of Oxford, he was "naive enough to not know who I couldn't shout at." Over and over, he shouted, "This is just not good enough!"

"We were desperate," he said. Reports took six months and cost thousands of pounds each. Getting informaton fast enough for thoughtful, rapid-fire analysis, he said, involved subterfuge. "Someone would say, 'Wow, I finally got an account,' and the other person would say, 'Well, don't tell anyone.'"

From his background in software engineering, Cotgreave know that writing one's own queries wasn't so hard. He pushed, and a "big culture clash" ensued in which the university IT department refused direct access to data. He recalls one IT person telling him, "'Oh, you're a business user. You can't access tables.'"

Eventually, his sustained push for access started attitudes to change. "We established that we didn't break things," he said, "and we did reports really quickly."

Soon Cotgreave's department hired a few people with slightly greater database skills, which reinforced IT's growing trust and let the two sides talk to each other fluently.

The culture has now changed dramatically. "DBAs are falling over themselves to allow more access to data," he says, even to allowing business users to run mass updates. Cotgreave says, "That's a real indication that what we were doing disruptively three years ago has proven itself."

The two sides may live happily ever after at the University of Oxford, but I suspect that in most environments this is would be just a happy beginning. Business managers with weaker data know-how than Cotgreave will struggle with data quality, and they'll push back on IT. A new clash will ensue, with resolution and "best practices" years away.

It's easier where data access was never an issue and where intelligent analysis is recognized and valued. One analyst, who asked for anonymity, had met with skepticism outside his unit. Then last year he was promoted as part of a company-wide reorganization to "train analysts to analyze." Before, analysts he dismissed as "key punchers" took sales forecasts straight from sales people. Now these newly trained analysts have been charged with providing independent analyses for account managers -- and to do that they have to think. "You can't just say you missed your forecast," he says, "You have to say why intelligently."

What's driving all this? Has dawn broken on IT, showing them new light? Those close to the action explain it from different angles -- but everything comes down to one root: an ever greater need for speedy, intelligent interpretation of the data -- a need accelerated by the economic downturn.

Stephen McDaniel, founder of Freakalytics and longtime author on books on SAS, Tableau, and other tools, hears a consistent theme at his Tableau training sessions: "People have to rationalize spending," he said, and adhere to scientific methods. Data analysis, performed quickly and intelligently by those who know the business intimately, is more important than ever.

To others, the main driver seems to be maturing tools. A sales manager at a major U.S. manufacturer who asked for anonymity credits availability of the kind of tools that Gartner calls "challengers": QlikTech, Tibco Spotfire, and Tableau. These, he says, let users keep up with "the pulse of the business.

While the business intelligence elite will continue their conversation about data quality, the cloud, mobile BI, and other obsessions, a small crowd of intelligent, creative, collaborative analysts will stay focused on the real work. Far from being "crushed," these analysts seem to be finding their way to the top of the business intelligence heap, where they belong.

Ted Cuzzillo is a journalist and industry analyst focused on analysts’ tools and needs as well as the environments in which they work. You can contact him directly at [email protected]. If you’re a data analyst, he’d appreciate your participation in his survey; you’ll receive a free preview of his report when it's complete.