Prefill and Decode on Separate Chips

This architecture matters because it turns AI inference from a one chip problem into a traffic routing problem. Reading the whole prompt and generating tokens back to the user stress hardware in different ways, so splitting them lets General Compute pair a high throughput engine for prompt ingestion with a low latency engine for token generation, instead of paying GPU prices for both jobs on the same fleet.1 2 3 4

• Prefill and decode are different workloads in practice. Prefill is a big burst that digests a long context window. Decode is a long sequence of tiny next token steps. Etched and MatX both frame this split as one of the central design questions in modern inference systems.2 4
• The comparable pattern across the market is heterogeneous inference. AMD with Cerebras, Intel with SambaNova, Nvidia with Groq, and software layers like Gimlet Labs all route different stages to different hardware. General Compute is following that same best engine per stage logic without owning its own chip stack.2 3 4
• For customers, this changes what is being bought. A bare metal deployment can dedicate one box type to prompt processing and another to response generation, while an endpoint sale can hide that complexity and sell a tokens per second promise. That makes the product more like managed capacity than rented servers.1

The next step is a broader move toward multi chip inference fleets where software decides where each stage runs. If General Compute can make that routing reliable across Cerebras, SambaNova, Positron, and GPUs, it can compete on workload level economics and latency, not just on access to more Nvidia boxes.1 3 4