- A pharma company stopped using Microsoft's Copilot AI tool, with an exec citing high cost and low value.
- The company used Office 365 Copilot for 500 staff and compared it to middle-school presentations.
- Microsoft is mulling repackaging 365 licenses to better monetize AI features amid record spending.
AI models and tools look great when researchers measure success with their own wonky benchmarks. When paying customers try this technology in the real world, things can get ugly.
That's what happened when an IT executive at a pharmaceutical company tried out Microsoft's Copilot AI features, according to a recent research note from Morgan Stanley.
The chief information officer of this pharma company paid extra to have 500 employees use Office 365 Copilot in the fourth quarter of 2023 and first quarter of 2024. This is Microsoft's much-heralded AI upgrade to its popular suite of productivity software.
After six months, the exec canceled the upgrade because the AI tools weren't good enough to be worth the extra money.
In fact, he compared the slide-generation capability of Microsoft's AI tools to "middle school presentations," according to a transcript of a call with the Morgan Stanley analysts that was included in their research note.
"The price is double," the executive, identified only as Greg, said. "And we really just do not see the value we're getting out of those tools worth double."
The E3 version of Microsoft's 365 software suite costs about $34 per user a month. Adding Copilot AI features costs another $30 per user a month. For 500 employees, that would roughly add an extra $180,000 a year.
These new Microsoft tools are considered some of the premier examples of powerful artificial intelligence in action in the real world. Investors have bid up Microsoft and other big tech shares massively, betting that this product and similar offerings will catch on with paying customers.
If a large pharma company can't see $180,000 of value in these tools, that's a problem that should worry the entire tech industry.
Legal issues with AI meeting summaries
The 365 Copilot AI feature that the IT executive found more compelling was the ability to archive and summarize video meetings on Microsoft's Teams app. But he said his legal team was wary of retaining meeting transcripts, so the pharma company didn't use that feature.
"What you're left with are tools that are along the lines of building ChatGPT instead of Word, which I think is marginally useful at best," the exec said. "Building a generative AI slide capability, which really is at the quality of middle school presentations at this point. And then Excel, which is, again, not really something that most people who use spreadsheets would think of using it."
He added that the company would consider adopting Office 365 Copilot for a subset of employees if it cost a fraction of its current price.
Microsoft must recoup big AI investments
The comments follow internal concerns at Microsoft about whether its AI services will create enough value to persuade corporate customers to pay more.
Wall Street is also wondering how Microsoft will recoup its massive AI spending. The company is amassing 1.8 million graphics processing units to build and run AI models and related products. It also has a plan to triple data-center capacity, mainly to handle AI workloads. Its capital expenditure hit a record $14 billion in the most recent quarter. A company spokesperson did not respond to a request for comment.
Microsoft is considering revamping the way it packages 365 software licenses like E3 to try to make more money from AI. The outcomes could include repackaging AI features into existing licenses or creating a more expensive bundle with Copilot AI capabilities, a person with direct knowledge of the plan said.
Do you work at Microsoft or have insight to share?
Contact Ashley Stewart via the encrypted messaging app Signal (+1-425-344-8242) or email (astewart@businessinsider.com). Use a nonwork device.
