OpenAI builds custom LLM inference stack

This reveals that OpenAI is no longer treating chips as a generic rented input, it is shaping silicon around the exact way ChatGPT and Codex serve tokens at scale. A chip built for LLM inference can be tuned for the memory movement, networking, and decode heavy serving path that dominates real products, instead of inheriting design tradeoffs from training clusters or older vision workloads. That matters because inference is where latency, power, and margin show up every day in user experience and gross profit.1 2 3 4

• Jalapeño is the first step in a multi generation OpenAI and Broadcom platform, with Broadcom handling chip implementation and Celestica building server systems. That points to tighter control of the full serving stack, from model kernels to rack level deployment, rather than simply buying off the shelf accelerators.1 2 5
• The contrast with Cerebras is useful. Cerebras gives OpenAI a latency first serving tier for the fastest coding and agent workloads, while Jalapeño is OpenAI trying to own that optimization curve itself. One is outsourced speed, the other is internalized infrastructure advantage.3 4 6
• This follows the same pattern seen across AI infrastructure, where the bottleneck has shifted from training a model once to serving it millions of times cheaply and fast. In practice, every token saved in power or every millisecond cut from response time compounds across ChatGPT, API traffic, and coding agents.2 5 6

The next phase is a stack split, with Nvidia GPUs remaining foundational, while custom inference silicon takes over the highest volume and most latency sensitive paths. If Jalapeño works at gigawatt scale, OpenAI gets lower serving cost, faster products, and a harder to copy infrastructure moat that reaches from model design into the data center itself.1 2 3 4