VMware for On-Prem AI
Tenry Fu, CEO of Spectro Cloud, on why 60% of AI will be on-prem
The real prize is not another model router, it is becoming the operating system for thousands of messy AI sites. Once inference moves into company owned servers, stores, hospitals, factories, and air gapped facilities, someone has to install GPU drivers, Kubernetes, model runtimes, security policies, and updates everywhere. Spectro Cloud is trying to turn that work into one central control plane, which is closer to what VMware did for virtual machines than what an API gateway does for model calls.
-
The fragmentation is physical, not just software level. A chain with 5,000 locations or a defense fleet cannot treat each site like a normal cloud region. The winning product has to push a full approved stack remotely, keep running when links fail, and roll out updates in waves from one console.
-
Spectro Cloud goes broader than point products focused only on GPU scheduling or model switching. PaletteAI packages operating system, Kubernetes, networking, storage, GPU drivers, inference engines like vLLM, quotas, token metering, and even VMs through KubeVirt, so customers can manage old and new workloads in one place.
-
That makes the competitive set look more like VMware replacement and hybrid infrastructure than classic MLOps. Red Hat, Nutanix, SUSE Rancher, Azure Arc, and NVIDIA Run:ai each cover part of the stack, but the gap is a neutral layer that can manage bare metal, edge, cloud, and disconnected environments without forcing one cloud or hardware vendor.
If enterprise AI keeps shifting from rented APIs to owned inference, control of day two operations becomes the choke point. The companies that win will bundle deployment, upgrades, governance, and cost controls into a standard fleet layer, and that is where AI infrastructure starts to look less like cloud software and more like the next VMware category.