Inference, explained

Every time you ask an AI model a question, watch a video that was generated by one, or get a transcript from a meeting tool, you've triggered an inference. The model isn't learning; it isn't getting smarter. It's just running the math that was set in stone the last time it was trained, and producing an output.

That distinction — training versus inference — is the most important framing in the AI compute conversation right now. Almost every major industry move in 2026 makes more sense once you have it.

Training vs inference (the short version)

Training is the learning phase. Lots of data, lots of repeated passes, weights getting updated. Compute-heavy, long-running, done occasionally — typically once per major model release. This is the part everybody saw coming. It's the reason GPU clusters got large enough to need their own substations.

Inference is the answering phase. The weights are fixed. Compute is spent on each request — prompt comes in, output comes out. This is the part that runs constantly, in production, every time anyone uses an AI product.

If a model takes three months to train and a fraction of a second to answer, the math seems obvious: training is where the cost is, right? It's the opposite. Once a model has millions of users sending billions of queries, the aggregate cost of inference dwarfs the one-time cost of training. AWS's CEO called inference "a new building block" of computing earlier this year. He wasn't being dramatic. He was looking at AWS's own bill.

What an inference chip actually is

Compared with training-focused hardware, inference-optimized chips usually emphasize:

• Low latency — quick response time per request. Critical for chat, voice, vision apps.
• High throughput — lots of tokens or requests per second. Critical for serving many users at once.
• Memory capacity + bandwidth — models are huge. Moving and reading weights matters more than raw FLOPS.
• Efficiency — better performance per watt and per dollar. Important at data-center scale and on-device.

Training hardware optimizes for peak compute and big multi-GPU scaling. Inference hardware optimizes for efficient serving — latency, throughput, memory bandwidth, cost per watt.

The hardware landscape

The chips and accelerators people usually mean when they say "inference":

• GPUs — NVIDIA H100/H200/L4, AMD MI series. General-purpose but very strong for inference. Most production inference still runs on GPUs.
• TPUs — Google's accelerators. Used for both training and inference. The Meta–Google TPU rental deal in February 2026 was the clearest signal yet that TPUs are a real alternative lane.
• Purpose-built inference accelerators — chips designed specifically for serving. Cerebras (powering OpenAI's Codex-Spark real-time coding mode), Groq, Axelera AI (which raised $250M in February for European inference chips), and others.
• Edge / phone NPUs — Apple Neural Engine, Qualcomm Hexagon, Intel "Panther Lake" (in the Razer Blade 16). For on-device inference. Privacy + offline + lower cloud cost.

Razer's Blade 16 refresh putting an Intel Panther Lake NPU on the spec sheet matters because it's a sign NPUs are finally a headline spec on consumer machines. The split between "local NPU tasks" and "cloud GPU tasks" will shape workflows, app design, and subscription pricing for years.

Why the inference economy is suddenly the whole story

Three things converged in the last twelve months that pulled inference from "implementation detail" to "main event":

1. Aggregate volume crossed an inflection. When OpenAI was the only frontier consumer assistant with millions of users, training dominated the cost story. Now ChatGPT, Claude, Gemini, Grok, Perplexity, Copilot, Doubao, and countless apps embedding their APIs are all serving billions of inference requests per day. The integral got large.

2. Power became the gating factor. Training a model uses a lot of power for a few months. Serving a model uses a lot of power forever. NVIDIA + Emerald AI's "flexible AI factories as grid assets" partnership is essentially an inference-economics partnership: data centers that can shape demand to grid availability are the only ones that can scale.

3. Cost-per-token became a competitive feature. Google's Gemini 3.1 Flash-Lite. OpenAI's GPT-5.3-Codex-Spark on Cerebras. Meta renting TPUs. Anthropic running Claude on AWS Trainium. Every one of these moves is about inference unit economics. The frontier model may set the ceiling on capability, but the inference tier sets the ceiling on adoption.

What "inference cost" actually breaks down into

When a CFO asks "why is this AI line item so big," the answer usually has four components:

• Latency budget — how fast does the response need to be? Faster = more reserved compute = more expensive.
• Throughput — how many requests per second can the system serve? Higher = more parallel hardware.
• Per-token economics — for text models, you're billed (internally or by a vendor) per input + output token. Long contexts get expensive fast.
• Memory and bandwidth — loading model weights and moving data fast enough to keep the chip busy. Often the actual bottleneck.

Product choices that look unrelated to compute are often actually about inference cost:

• Quantizing a model from 16-bit to 8-bit or 4-bit weights — half or quarter the memory cost, with some accuracy hit.
• Caching common prefixes ("system prompts").
• Batching requests together at the cost of slightly higher latency.
• Routing simple queries to a cheap model and reserving the expensive one for hard queries.
• Distillation — training a smaller model to mimic a larger one for serving.

If you've ever wondered why AI companies are so obsessed with "smaller, faster, cheaper" models when the headlines are about scale, this is why. The cost curve at inference is what decides whether a feature ships or gets cut.

Why this matters for the next year

Three predictions worth holding loosely:

1. The inference chip market will fragment. GPUs will stay dominant for the next year, but custom inference accelerators will keep eating share for specific workloads. The question isn't whether NVIDIA loses inference; it's how quickly purpose-built silicon takes the high-volume long tail.

2. On-device inference will grow faster than cloud inference. Not in absolute terms — cloud is still the bulk — but in product surface area. Apple, Qualcomm, Intel, and AMD are all betting that the next AI features are local-first. Privacy, offline reliability, and zero per-token cost are too compelling.

3. Inference economics will decide which AI companies survive. Training a frontier model is a one-time fundraising event. Serving it profitably is forever. The next round of consolidation in the AI industry will be driven less by "who has the best model" and more by "who can serve queries at a price the market will pay."

If you only remember one thing about inference: it's the part of the AI economy that runs every minute of every day, costs more in aggregate than training, and is the actual battleground for the next phase of competition. The headlines are about model launches. The earnings calls are about inference.