Token-Level Latency Drives ASIC Adoption

This is really a fight over who wins the decode step of inference, where users care about each next token arriving fast, not just about raw batch throughput. Cerebras, SambaNova, and Groq are all building around that pain point with chips and software tuned for steady low latency output, then packaging them as APIs, dedicated racks, and sovereign or on premises systems instead of selling only standalone accelerators.1 2 3 4

• Cerebras has already moved from selling a small number of $2M training systems into a cloud inference API, with usage based sales to customers like Perplexity, Notion, Windsurf, and Cognition. That shift matters because latency sensitive workloads create many more repeat calls than classic training buyers do.1 5
• SambaNova is taking a similar full stack route, but with more emphasis on mixed deployments. Its RDUs are designed to keep many models live on one node and switch between them in microseconds, which fits agent workflows that bounce between coding, reasoning, and voice models without constantly reloading weights.2
• The real pressure is not only Nvidia. Google already offers TPU serving with latency tuned configurations, Microsoft is pushing Maia as an internal inference system, and AWS positions Inferentia as custom inference silicon. That means specialists must prove clearly better speed, power, or deployment flexibility, not just non GPU branding.3 6 7 8

The next phase is likely a split market. Hyperscalers will absorb more routine inference inside their own clouds, while specialists win the slices where first token speed, token by token consistency, power limits, or data residency make generic GPU clusters too blunt and too expensive.2 3 6 7 8