Why does AI cost so much to run?
Because every single response requires substantial computation on expensive, power-hungry hardware — and unlike most software, the cost does not fall as usage grows, which breaks the economics people expect from technology products.
The two cost centres:
Training. Building a large model requires enormous computation over weeks or months on thousands of specialised processors. It is a very large one-off cost, and it must be repeated for each new model.
Inference — running the model to answer queries. Each response involves an enormous number of calculations, and this is the cost that scales with users. For a widely used service, cumulative inference cost exceeds training cost substantially.
Why this is different from conventional software. A traditional application costs a great deal to build and almost nothing per additional user — which is why software businesses have such high margins and why free tiers are viable. AI inference has a real marginal cost per request, more like a manufacturing business than a software one. This is why free usage is capped, why heavy users are limited, and why pricing is metered.
What drives the cost per request:
Model size. More parameters means more computation per token.
Tokens processed, both input and output — long contexts and long answers cost proportionally more.
Hardware. Specialised accelerators are expensive and, during periods of high demand, supply-constrained.
Memory bandwidth, which is frequently the actual bottleneck rather than raw computation.
Electricity and cooling, which is why data centre energy demand has become a live policy issue.
What is being done to reduce it: quantisation, reducing numerical precision; distillation into smaller models; mixture-of-experts architectures activating only part of the model per token; caching repeated prefixes; batching requests together; and routing simple queries to smaller models.
Why prices have fallen sharply despite this: efficiency improvements, competition, and smaller models reaching capability that previously required large ones.
The environmental dimension — water for cooling and electricity demand — is increasingly part of the accounting.