Note: This summary was generated by AI and may contain inaccuracies.
I've been doing research on the practical implications of AI, and I just completed a big project with colleagues at HBS, MIT, and elsewhere. We went to Boston Consulting Group, one of the top three elite consulting companies. My students are desperate to get in there—really high-quality folks. Maybe you've worked with them before.
We conducted a simple experiment. Well, it was simple in theory but complex in practice. We took 8% of their global workforce and gave some of them access to GPT-4 for a set of 18 real consulting tasks, while others did not have access. They picked the tasks and ran the whole thing.
It turned out that for the people who were given access to GPT-4, there was a 40% improvement in the quality of their output, they worked 26% faster, and they completed 12.5% more work. A few things to note here: first of all, 40% is an insane number. That's crazy, right? Just to give you context, when steam power was introduced into factories in the early 1800s, it increased individual performance by 18 to 22%. So, this is a big number that we haven't really seen before.
Second, there wasn't anything special here. This was not a specialized version of GPT. This is the same thing you have access to, and this was without training—just using the system. Some people got a 10-minute introduction, and that was it. This is a very, very large change. We don't fully understand all of what it means, and again, no one's optimized for use yet, but it was already performing at this very high level. I think that creates an imperative to think about how we use it.
There's another piece that's really interesting: the bottom half of participants got the biggest gain. Now, they're all pretty good—they're all at BCG—but the participants in the bottom half of the skill gain got a 43% improvement. So, people using GPT-4 outperformed people who didn't use it, even if they were better-skilled at this task. The people in the top half of the skill distribution only got a 17% performance improvement. So, it acts as a leveler; it basically moved everybody up to the 80th percentile of performance. Again, we're not quite sure what to do with that, but it's a pretty big change. We're used to having wide-scale performance distribution changes, but it seems like that's not what's happening here. Instead, it's a booster; it sort of makes everybody good at work.
And this isn't all. It turns out it's good at a bunch of stuff we wouldn't have expected. These are results from a study that my colleagues at Wharton just did with a class that's fairly legendary on product design and development, written by Karl Ulrich, who literally wrote the main textbook on product development. We have a very good group of MBAs and undergrads who generate ideas in that class that often get venture funded. We decided to compare their ideas to GPT-4's ideas. GPT-4 absolutely smoked the students. Of the top 40 ideas rated by outsiders as having high willingness to pay, 35 of those came from the AI, and only five from the students in the class. So, it beat the students on innovation on every measure we have and basically maxed out every innovation score we've got. We can talk about some downsides of that, but it's important to note.
The last piece of data I want to show you is from four studies that all show the same thing: the impact of generative AI on work. All these studies show almost the same thing, which is a near-perfect correlation between how much your job overlaps with AI—not replaced by AI, but how much AI will have to do with your job or how much you'll have to use AI in your job—and how much you're paid, how much you're educated, and how creative your job is. The more creative your job, the more highly paid you are, the more educated you are, and the more you overlap with AI. There's pretty much complete agreement on this from studies by Goldman Sachs, OpenAI, McKinsey, and NYU. So, this is not something you can ignore.
Also, we've never seen disruption from the top before. Usually, automation happens for dangerous, dirty, repetitive work. It doesn't usually happen for highly intellectual work, but that's what's happening now.