There's no such thing as free AI

We already know that spending on artificial intelligence infrastructure, like data centers, has been exploding in recent years. But spending on AI usage is turning out to be an ever-bigger expense for many companies.

KPMG estimates businesses will nearly double the amount they spend on AI in the next year, an average of more than $200 million dollars.

“Even, like, a quarter, two quarters ago, nobody bothered about [large language model] consumption costs,” said Swami Chandrasekaran, head of AI and data labs at KPMG North America.

But this year, those AI consumption costs have been rising to levels that rival human labor costs.

A recent Goldman Sachs research survey of large companies found many are overrunning their AI budgets by orders of magnitude, and AI spending could equal engineers’ salaries in the near future.

The way AI is metered is with tokens, roughly equivalent to the output of a word or two.

“The way that I like to think about it is sort of like the number of turns your interaction with the AI sort of takes,” said Max Kan, a “tokenomics” analyst at SemiAnalysis.

For instance a basic conversation with a chatbot takes a handful of turns. “You ask the AI some question, it gives you an answer. Maybe you asked like, two or three more follow ups, and that's about it,” he said.

But new, more powerful AI agents might search the internet, write and test code, analyze hundreds of documents, talk to other agents and keep doing these things around the clock.

“It is easily like 100x, 1,000x, maybe even 10,000x more tokens,” said Kan.

The average cost per token has been going down, thanks to model and hardware efficiencies, and is predicted to drop by 90% by 2030 according to research from Gartner.

But more advanced models use more expensive tokens, and demand to use advanced features has been increasing faster than prices are declining.

“How much budgets will actually need to grow, how much folks can have to spend is an open question,” said Will Sommer, economist at Gartner. “But unconstrained deployments of these more advanced AI tools, you know, like agents, it's not a viable strategy for adoption.”

He said enterprises will likely end up limiting the use of the most advanced and expensive models.

But recently, some companies, like Meta, have reportedly been tokenmaxxing.

“Employees are basically being encouraged to consume as many AI tokens as possible,” said Daniel Newman, CEO at Futurum Group. “The idea is there's this direct correlation between token consumption and productivity.”

Newman said his own business has seen triple digit growth in its use of AI agents in the last several months.

“It's an absolute killer productivity tool,” he said. “But it's increasingly becoming more and more expensive to do these things properly.”

Partly that’s because some AI companies have pulled back on subsidizing the use of their tools said Ed Zitron, a tech critic who wrote about what he called the “subprime AI crisis.”

“So you will have to pay the actual cost of the AI you use” he said. “Now this is starting to worry people.”

Anthropic and Microsoft recently announced several changes to pricing, which could raises costs for heavy users.

“I don't think a lot of businesses are actually aware of how much this really costs,” said Zitron.

And most aren’t don’t have great systems in place to measure return on their investments according to Brian Jabarian, an economist at the University of Chicago who consults with companies on AI transformation.

“How do you prove, with hard coded data, that AI has been fruitful for your business?” he said. “The time for the bill is going to come.”

The bigger the AI bill, the bigger the benefits needed to justify it.