Plato at Work

From the NYTimes:

Mr. Brynjolfsson and his colleagues, Lorin Hitt, a professor at the Wharton School of the University of Pennsylvania, and Heekyung Kim, a graduate student at M.I.T., studied 179 large companies. Those that adopted “data-driven decision making” achieved productivity that was 5 to 6 percent higher than could be explained by other factors, including how much the companies invested in technology, the researchers said.

In the study, based on a survey and follow-up interviews, data-driven decision making was defined not only by collecting data, but also by how it is used — or not — in making crucial decisions, like whether to create a new product or service. The central distinction, according to Mr. Brynjolfsson, is between decisions based mainly on “data and analysis” and on the traditional management arts of “experience and intuition.”

A 5 percent increase in output and productivity, he says, is significant enough to separate winners from losers in most industries.

The companies that are guided by data analysis, Mr. Brynjolfsson says, are “harbingers of a trend in how managers make decisions.”

“And it has huge implications for competitiveness and growth,” he adds.

Data analysis is without a doubt the next wave, and it’s influence on productivity will be enormous. Google’s Chieff Economist Val Harian has been saying this for years:

I keep saying the sexy job in the next ten years will be statisticians. People think I’m joking, but who would’ve guessed that computer engineers would’ve been the sexy job of the 1990s? The ability to take data—to be able to understand it, to process it, to extract value from it, to visualize it, to communicate it—that’s going to be a hugely important skill in the next decades, not only at the professional level but even at the educational level for elementary school kids, for high school kids, for college kids. Because now we really do have essentially free and ubiquitous data. So the complimentary scarce factor is the ability to understand that data and extract value from it.

I think statisticians are part of it, but it’s just a part. You also want to be able to visualize the data, communicate the data, and utilize it effectively. But I do think those skills—of being able to access, understand, and communicate the insights you get from data analysis—are going to be extremely important. Managers need to be able to access and understand the data themselves.

I completely agree, and I wish wholeheartedly that my data analysis skills were better. Statistics as a field initially frustrated me, because the notion of confounding variables seemed to preclude the sure-fire validity of any conclusion that I could discover or read about. Once more, I couldn’t get past the randomness of chance; that is, whose to say that just because a coin flip is 50/50 that it will flip heads 50% of the time. It’s just not possible to know for sure (and this is partially why I’ve always had fears of lightning).

My own suspicions notwithstanding, the data revolution will affect us in even more profound ways than productivity increases. Healthcare will soon face a revolution as new statistical tools are able to boost the power of computers servers and assist scientists in analyzing the millions upon millions of biological and genomic tests. We will glean real cures to horrible diseases  and, perhaps more importantly, move pre-vaccination to the direct manipulation of DNA to move the probability of contracting specific diseases to almost worry-free levels.

This article on the upcoming bubble in social-media hints at the new trend I fully expect. After noting how the biggest jobs in Silicon Valley are going to mathematicians and statisticians able to effectively manipulate data for advertising, journalist Ashlee Vance explains how Jeff Hammerbacher, a former Facebook employee, moved from advertising to a startup focused on data analysis tools for scientific research. This is Mr. Hammerbacher’s startup, Cloudera:

Cloudera is essentially trying to build a type of operating system, à la Windows, for examining huge stockpiles of information. Where Windows manages the basic functions of a PC and its software, Cloudera’s technology helps companies break data into digestible chunks that can be spread across relatively cheap computers. Customers can then pose rapid-fire questions and receive answers. But instead of asking what a group of friends “like” the most on Facebook, the customers ask questions such as, “What gene do all these cancer patients share?”

Eric Schadt, the chief scientific officer at Pacific Biosciences, a maker of genome sequencing machines, says new-drug discovery and cancer cures depend on analytical tools. Companies using Pacific Bio’s machines will produce mountains of information every day as they sequence more and more people. Their goal: to map the complex interactions among genes, organs, and other body systems and raise questions about how the interactions result in certain illnesses—and cures. The scientists have struggled to build the analytical tools needed to perform this work and are looking to Silicon Valley for help. “It won’t be old school biologists that drive the next leaps in pharma,” says Schadt. “It will be guys like Jeff who understand what to do with big data.”

When the tech bubble pops in advertising, most talented folks will be moving into new data analysis applications for healthcare, the science, general business, and perhaps even consumer electronics (e.g. imagine walking into a room and knowing the probability that you’ll make a friend–it’s not that far off if you imagine the data of Facebook mashing with the in-the-works technology of Color).

It’s notable that Google’s first issue of Think Quarterly, it’s first official foray into unique digital media and content creation, is focused on the coming data revolution.

If you’re unsure about the power of data to communicate extraordinary qualitative messages, then just watch these two presentations by the phenomenally charismatic and insightful statistician Hans Rosling at TED.

Hans Rosling on the rise of Asia:

A second presentation on why the washing machine is the greatest invention of all time:

(Thanks to smemon87 for the image.)