Full-stack Operations Drive On-prem AI
Tenry Fu, CEO of Spectro Cloud, on why 60% of AI will be on-prem
The real value is shifting away from picking the cheapest model and toward operating a reliable private AI estate. A router can decide whether a prompt goes to Claude, an open model, or a local cluster, but enterprises still need someone to install the GPU stack, keep Kubernetes and model runtimes patched, and manage thousands of sites without breaking production. That is why gateway features spread fast, while full stack operations remain scarce.
-
Gateways are becoming table stakes because they solve an obvious surface problem, logging requests, setting routing rules, and steering traffic across providers. Cloudflare packages AI Gateway as centralized visibility and control, and Kong has expanded from API gateway into AI traffic management, showing how quickly routing is being absorbed into existing platforms.
-
What enterprises actually struggle with is day two operations. Spectro Cloud is built around fleet management once customers pass roughly 20 clusters, and in edge settings it can mean more than 10,000 clusters. That work includes rolling out OS, Kubernetes, networking, GPU software, inference engines, and model updates across distributed environments.
-
The deeper reason routing alone commoditizes is that multi model systems create infrastructure problems above and below the gateway. Above it, teams want governance, quota control, and billing. Below it, self hosted models are huge files that need specialized deployment, memory management, and update workflows. That makes the durable moat operational software, not simple request forwarding.
The market is heading toward bundled AI infrastructure stacks, where routing is one feature inside a larger control plane for hybrid inference. As more enterprise workloads move on prem and to the edge, winners will look less like standalone gateways and more like VMware for AI, software that keeps heterogeneous model fleets running safely, cheaply, and at scale.