Inference as a Memory Movement Problem

The core bet is that frontier inference is becoming a memory movement problem, not just a math problem. Majestic Labs and Cerebras are both arguing that once a model is loaded, the slow part is often fetching weights and KV cache fast enough during decode, especially for long outputs and long context. That creates room for systems built around bandwidth, memory layout, and split prefill and decode serving, instead of generic GPU fleets alone.1 2 3 4 5

• Cerebras has already turned this thesis into a product shape. Its CS-3 is sold as the decode side of a disaggregated stack, paired with AWS Trainium for prefill, and distributed through AWS Marketplace and API endpoints. That makes the argument concrete for buyers, not just architectural.2 3 5
• Majestic Labs is pushing the same bottleneck from a different hardware angle. Prometheus uses one large shared memory pool, 8 TB to 128 TB, and standard LPDDR instead of fragmented GPU memory tiers and scarce HBM, aiming to reduce the overbuying of compute that happens when customers mainly need attached memory.4
• This is not only a startup talking point. Cloud and infrastructure operators increasingly describe decode as memory bandwidth bound, with compute and decode separated because one accelerator is often mismatched to both phases. The market is moving from one general pool of GPUs toward mixed fleets with specialized roles.2 6 7 8

Going forward, the winners in inference are likely to be the companies that package this hardware argument into an easy serving product. Cerebras is ahead because it already wraps the architecture in cloud distribution and pay per token access. Majestic Labs is heading toward the same end state, where the real prize is not a faster chip, but becoming the preferred decode layer in frontier AI stacks.2 3 4 5