645 Thought Leadership

The Actual Cost of AI

What really happens when you hit enter, and why it isn't free.

For fifty years, it cost almost nothing to run software. Creating it was expensive, but running it was nearly free. That's why Software as a Service has become one of the best business models we've seen over the last two decades. It is why a company in a rented office could reach a billion people and why software margins are ~70% or more.

AI just broke that math, and in the process created what we think of as the Physics of Software.

Every single time you ask an AI a question, something physical happens. Billions of calculations happen on a chip. Huge amounts of data (e.g. gigabytes) move in and out of memory. Electricity turns into heat on the chips. It happens on every single use, every prompt, every user, every day, and it grows as usage grows. The answer that is quick and weightless on your screen has real world consequences.

As AI gets woven into the fabric of society, software reaches into the physical world, where it costs real energy and real money not just to build but to run it over and over again. For most of the software era, the interesting questions were about distribution, design, and network effects, because the cost of running the thing was fixed and nearly free after it was built. That no longer holds in AI software.

Every prompt and answer costs money

Software just got a cost of goods sold. "Cost of goods sold" is what a business spends to make each unit of what it sells. A songwriter has almost none: record the song once, and every download or stream after that is nearly pure profit. That is what software has been for fifty years: write it once, sell it a billion times, keep almost everything. A restaurant is the opposite. Every plate it serves costs the restaurant money to make: the ingredients, the gas on the stove, the cook's time. There is no version of a restaurant where the ten-thousandth meal is free.

AI turned software from the songwriter into the restaurant. Every answer is a fresh plate, cooked from scratch, with real ingredients and real energy behind it. The kitchen never stops, because the orders never stop. And that one change, from a product you make once to a product you cook to order, forever, rewrites the economics of the entire industry.

So who actually pays?

The good news is cheap intelligence may be the most powerful deflationary force in modern history. When the cost of thinking falls, whole categories of work that were too expensive to automate become possible, and new markets appear. That's a win. If founders and investors find a way to deliver intelligence as cheaply and reliably as electricity, then we will see significant advancements in medicine, education, science, and for small businesses that could never afford a research team.

The bad news is someone has to pay, and the bill is real. It helps to split it into two very different kinds of cost, because they require two different responses.

The first is permanent — the physics of energy and water, which can be engineered down but never financed away.

The second is transient — the investor subsidy keeping prices below their true cost, a familiar capital cycle that will eventually reprice.

Between these two costs, everyone is affected: the communities living next to data centers, the academic researchers being lured to private startup labs that have what feels like infinite compute that no university can match, and the user who inherits the bill when the subsidy runs out. Each of these deserves its own post. But for now, let's focus on the cost that is already being paid, and who ends up paying it.

What happens when you hit enter

When you type a question into your favorite AI (e.g. Claude, ChatGPT, Gemini, etc.) and press return, your sentence begins a physical and economic journey: prompt → token → inference → memory → electrons → dollars. Your words are converted into tokens; those tokens are processed through inference on chips; the model draws on memory and context to shape the answer; that computation consumes electricity; and the electricity plus hardware time becomes cost to power and cool the chips. Each link matters because each is where performance, latency, margin, and new startup opportunities exist. Every time you ask AI, someone pays. The winners in this physics race will be the ones who make that answer smaller and smaller, until intelligence is as cheap and ordinary as electricity.

A token is like a syllable

A foundational model does not read your sentence the way we humans do. It breaks it into tokens, which are small chunks of text converted into numbers that the model understands. A token is usually a piece of a word rather than a whole one. Common short words like "the" or "dog" are a single token. A longer or rarer word gets split: "recommendation" might become three tokens. A rough rule of thumb is that 1,000 tokens is about 750 English words.

Tokens are the units of everything that follows. The model reads your prompt as a string of tokens, and it writes its answer one token at a time, predicting the next token, then the next, then the next, until the thought is finished. This matters for a simple reason: tokens are the unit AI is priced in. When you hear that a model costs some number of dollars "per million tokens," that is the meter running. Every word in, and every word out, is on the bill.

Inference is like thinking — here is where the costs live

There are two completely different things an AI model does: training and inference.

The first is training — this is like sending the model to college, where there is a one-time cost. This is the months-long, hundreds-of-millions-of-dollars process of feeding the model enormous amounts of text so it learns the patterns of language. Training happens once, and it is expensive.

The second is inference — what the model does every time it actually responds to a prompt. Inference is the model using what it learned to answer your specific question. And here is the part that most people miss: to produce just one token of its answer, the model has to do billions of multiplications across all of its weights, or internal numbers. Generating a single token means effectively reading the entire model. Then it does the whole thing again for the next token. And again for the one after that.

So inference is not a one-time cost like training. It is a cost that repeats for every token, in every answer, for every user, every day. A model is trained once and runs billions of times. Nvidia CEO Jensen Huang has described this moment as the arrival of the inflection point of inference. The AI industry is moving from building models to operating them, and the economics increasingly belong to the systems that can serve intelligence quickly, reliably, and cheaply at scale.

Memory is where the model lives and where the conversation is held

For the model to do all that multiplying, its weights have to sit in fast memory, physically right next to the chip that does the math. And the serious constraint is not how fast the chip can calculate. It is how fast memory can feed the chip the numbers it needs.

Imagine a world-class chef who can chop faster than anyone. If the ingredients only reach the counter one at a time, down a long hallway, the knife speed doesn't matter anymore. The chef is standing there just waiting. Modern AI chips are the chef. They are extraordinary at arithmetic, and they frequently sit waiting on memory to send over the next batch of numbers. Over the last decade, raw computing power has grown far faster than memory bandwidth has, so the gap keeps getting longer.

There is a second kind of memory worth knowing about. As the model works through your conversation, it keeps a running set of notes on everything said so far. This is called the KV-cache, and you can think of it as the model's short-term memory of the chat. The longer the conversation, and the longer the context the model has to hold in mind, the bigger those notes grow, and the more memory has to be stored and re-read for every new token. This is part of why long documents and long back-and-forths cost more: the model is carrying more in its head.

Electrons are why it runs hot and power-hungry

Moving all of those weights and notes in and out of memory, billions of times per second, takes real electricity. Electricity becomes heat. Heat has to be removed by cooling that, in some situations, requires a lot of water. Cooling and computing together draw power from the grid, power that has to be generated somewhere. This is the least glamorous link in the chain and increasingly the most important one. Data centers are now a meaningful and fast-growing share of electricity demand, and inference, not training, is becoming the dominant part of it.

The economics yet to be priced in

Here is where it gets counterintuitive, and where we think most people draw exactly the wrong conclusion.

The cost of inference, measured per token, is falling extremely fast. Research from Epoch AI suggests that the cost of achieving a given level of capability is declining by roughly 5–10x per year, driven by improvements in algorithms, hardware, and competition among providers. But that does not solve the problem. It amplifies it.

As the cost per answer falls, the number of answers we demand explodes, and it explodes faster. Google's token volume illustrates the point. At Google I/O 2025, the company reported processing more than 480 trillion tokens per month, up from roughly 9.7 trillion a year earlier, a fifty-fold increase in just twelve months.

When something useful gets cheaper, we do not use less of it. We use dramatically more. Economists have known this pattern for more than a century. Microsoft CEO Satya Nadella captured it succinctly in January 2025 when he wrote: "Jevons paradox strikes again! As AI gets more efficient and accessible, we will see its use skyrocket, turning it into a commodity we just can't get enough of."

That is exactly what is happening with AI. The cost per token is collapsing, but demand is growing even faster. Efficiency is not reducing the need for infrastructure. It is creating more reasons to consume it.

So "cheaper" does not mean "cheap in total." It means the total bill keeps growing even as each unit gets less expensive. For a lot of AI companies, that bill is the dominant line in their costs. Where a traditional software business might keep 70 to 80 cents of every revenue dollar, many AI products are running at far thinner margins because the compute to serve each customer sits right in the cost of goods.

The reactive response to all of this is "buy more chips." But more of the same chips does not fix a problem that is increasingly about memory and power, not raw computing. You cannot brute-force your way past a hallway that is too long or a grid that cannot deliver more electrons.

Why this is the prize

We believe AI will be woven into the fabric of society: into how we work, learn, get care, and build. That is not wishful thinking; it is close to inevitable given how useful these systems already are. But it comes with a condition that does not get said often enough: for AI to reach everyone, the cost and energy of each answer have to fall by orders of magnitude beyond where they sit today.

This is what we are most excited about. Making intelligence cheap enough to run everywhere is the defining infrastructure project of this era, the way building the electrical grid was the defining project of the last one. Electricity did not change the world when it was invented; it changed the world when it became cheap, reliable, and ambient enough to put in every home and factory. AI is at the pre-grid stage. Whoever builds the layer that turns expensive, power-hungry intelligence into something as cheap and ordinary as flipping a switch will not be running a side business, they will own a market on the scale of the hyperscalers.

That is not a model problem; we have remarkable models. It is an infrastructure problem, a physics problem. And it is not solved yet. The founders attacking it will be the first to tell you how far there is to go, which is precisely why it is worth doing.

A whole new stack is being built

The good news is that the answer is not only better chips. Much of the solution is in software, in being thoughtful about how we run AI on the hardware we already have. And that is why an entire new infrastructure stack is forming underneath AI, layer by layer. This stack, not the models, is where we think an enormous amount of value will accrue. You can see it taking shape from the metal up, and we see massive opportunities in building here.

At the bottom of the stack are Power, Data Centers, and GPU Clouds. This is the physical foundation of AI: electricity, cooling, networking, racks, and real estate. Companies such as Crusoe, Applied Digital, Switch, Vantage, CoreWeave, and Lambda provide the infrastructure that turns power into computation. Every token ultimately runs through this layer, and every answer inherits its costs.

Above that sits Inference Silicon. These are chips designed specifically to run AI models efficiently. Companies such as Nvidia, Groq, Cerebras, d-Matrix, SambaNova, Etched, Tenstorrent, and Positron are building hardware optimized for inference, aiming to deliver more performance per watt and lower cost than traditional approaches.

The next layer is Inference Optimization. The goal here is simple: produce the same answer with less computation. Remembering that every answer is billions of math operations, this layer shrinks that number, getting the same result from fewer calculations (in industry shorthand, fewer FLOPs, or floating-point operations). Companies like Liquid AI, VSINC, Code Metal, OctoML (now part of NVIDIA), and Neural Magic (now part of Red Hat / IBM) compress, restructure, and streamline models so they run faster and more efficiently on existing hardware.

Then comes Memory Infrastructure, the bottleneck we walked through earlier. As models grow and conversations lengthen, memory increasingly determines performance. This layer manages model weights, KV-caches, prefix caches, and other forms of reusable computation. Companies like Tensormesh are attacking the problem directly, building AI-native caching systems that reuse work the model has already done instead of recomputing it from scratch, reducing both latency and cost.

Above memory sits Inference Serving. Together AI, Fireworks AI, Baseten, Modal, fal, Hyperbolic, Anyscale, and Inference.net take the complexity of deploying, scaling, monitoring, and operating models and package it into a simple API. Their job is to turn raw infrastructure into a reliable service while extracting as much useful work as possible from every chip through batching, autoscaling, observability, and operational tooling.

At the top is Inference Routing, the layer we find most interesting. Companies such as OpenRouter, Martian, Canyon Code, and Gimlet Labs are building systems that decide, moment by moment, which models and hardware should handle each request. That matters because an AI workload is rarely one uniform job. A request may contain portions that are compute-bound, memory-bound, network-bound, or latency-sensitive. No single piece of hardware is best at all of them.

What's next

For most of the software era, the physical cost of running a product was an afterthought, a rounding error you could ignore. That is no longer the case. Compute, memory, and energy, which make up the Physics of Software, are now first-class constraints, and the companies that master them will hold the high ground in the AI age.

The race everyone watches is for the cleverest model. We believe the race is about who can serve intelligence fast enough, cheaply enough, and efficiently enough to put it everywhere. A breakthrough model that costs too much to run stays a demo. The company that makes it run for a tenth of the price changes the adoption and productivity curves.