Tenry Fu, CEO of Spectro Cloud, on why 60% of AI will be on-prem

With agentic coding driving token consumption higher and enterprises increasingly weighing local inference against hyperscalers & frontier APIs, we reached out to Tenry Fu, co-founder & CEO of Spectro Cloud ($260M raised, Stripes), to understand what the enterprise AI infrastructure stack looks like as inference moves into production.

Key points from our conversation via Sacra AI:

You founded Spectro Cloud in 2019. What was your original insight—the thesis behind the company—and how did the market and early investors respond to it?

Before Spectro Cloud, I had another startup, CliQr Technologies, a multi-cloud management startup that was acquired by Cisco. To me, Spectro Cloud is a result of unfinished business. We really wanted to meet the need for applications to become portable across multiple environments. At the time of CliQr, the cloud wasn't fully ready for that yet, because every application was based on a VM, and a VM isn't truly portable across multiple clouds. By the time we started Spectro Cloud, with containers and Kubernetes, the groundwork was laid and it had become possible for us to solve the problem.

But for an enterprise to really adopt Kubernetes at scale across multiple environments still takes a lot of work, because Kubernetes just scratches the surface of the entire stack that’s required. There are so many different moving parts in the cloud native stack. That's why we started Spectro Cloud—to help enterprises not only manage Kubernetes with ease, but also do full-stack management at scale across multiple environments.

Great, so to sum up: Spectro Cloud started out helping enterprises manage Kubernetes fleets more easily. Is that fair?

Yes, but it's more than just Kubernetes itself. It's the full stack—OS, Kubernetes, storage, network, all the way up to the application, across multiple environments. With more ease and consistency in infrastructure management, businesses save time and costs, can redeploy talent better, and focus on application innovation. On the customer end, it also leads to more secure and easier scaling.

How did the market and your early investors respond to that pitch around Kubernetes at scale? Was it immediately clear to customers and investors that this was a problem that required a startup to attack, or was there skepticism early on?

At the beginning, we were maybe a little bit early—but as a startup, you always have to be a little bit early. We did get support from early investors and some organizations like T-Mobile and GE Healthcare that saw the value early on. We were tackling a production-at-scale problem. Only when a customer has Kubernetes and the entire full stack in production, with many clusters, do they feel the pain of managing one cluster at a time, and realize it just doesn't scale. Then they need a management platform like ours to help them manage their fleet of clusters more efficiently and across multiple environments.

When we started, the Kubernetes adoption curve was still at a relatively early stage, and that's why a lot of other Kubernetes companies were focused on Kubernetes distros, whereas we focused on multi-cluster management. Because the starting point is different, for many of those companies management is really an afterthought, whereas for us it was the starting point. That has become one of our key strengths.

Can you explain briefly why Kubernetes has been so important for scaling a modern technology-anchored organization today?

Kubernetes has become the de facto runtime for cloud native container workloads and also for AI workloads, managing all the GPU clusters. Even legacy workloads like virtual machines can become first-class citizens in Kubernetes and be managed directly by Kubernetes. So, it has become the de facto runtime for all workloads—VMs, containers, and AI—and it acts as an abstraction layer for compute, storage, network, and even GPU, so that all these different workloads can efficiently consume the resources without dealing with the lowest-level infrastructure.

Also, because of Kubernetes' declarative design principle, it makes life a lot easier for developers and DevOps, or platform engineering. Whenever an application or any of the many software components in a stack has an update, you literally just change the declarative model and the entire application is automatically updated by Kubernetes. If any component goes down at runtime, the Kubernetes reconciliation loop will automatically relaunch that particular component. All of this makes life a lot easier for DevOps.

At what scale does a solution like yours become relevant?

For fleet management at scale, customers start feeling the pain when they grow beyond about 20 clusters. They can no longer manage one cluster at a time. Instead they need a multi-cluster management platform.

We also have a lot of customers doing edge deployments, shifting compute resources closer to where the data is at the edge location. Edge is by definition very distributed. We have customers that need to deploy more than 10,000 clusters across all their different retail stores, for example. And hospitals, factories, and military organizations also need to manage that kind of scale. Managing one cluster at a time, or sending a field engineer to a retail store to manually do the deployment and upgrade, is just not feasible. So they have to use a solution like ours to solve the manage-at-scale problem.

How does AI play into the business? Is AI feeding the fragmentation of workloads and deployment environments, and has it been a tailwind for your business?

AI is a huge tailwind for us, and it's a natural evolution for us, because as I mentioned, Kubernetes is the de facto runtime for AI clusters managing GPU infrastructure.

But in the AI world the stack becomes even more complicated, because besides regular compute, storage, and network, you now also introduce GPUs and DPUs (Data Processing Units) for the GPU east-west traffic—the interconnect between different servers, within the rack or across the rack. That introduces a lot of new concepts into the infrastructure. On top of that, the pace of innovation in AI is even faster than it was in the cloud native world. Every day, every week, you see lots of new component updates. That makes full stack management even more important, so users can manage the entire stack as a whole and guarantee interlayer compatibility. The entire stack can be one-click deployed and one-click updated without the user worrying that upgrading one component will break the whole thing.

Can you quantify how meaningful AI is for your business today? You have some AI-specific products—what percentage of the business comes through them, and how far along are you in becoming an AI-centric business?

Our AI business started almost three years ago, at the ChatGPT moment when everyone started waking up to AI's impact. In the early days we worked very closely with NVIDIA to turn NVIDIA's reference architecture for the AI factory into a full stack solution. We helped a lot of the neoclouds and sovereign clouds so that once they had secured their data center, electricity and power, and racked and stacked their GPU infrastructure, we could very quickly turn that into an enterprise-grade AI factory. That became a big chunk of the growth business for us.

Starting this year, we also began to see enterprises adopting a lot of AI inferencing as part of their hybrid AI strategy, whether on-prem or hosted in a cloud or neocloud environment. A lot of enterprises are facing the same challenge: their token expense is out of control. They urgently need to lower their token cost, and at the same time have full visibility into token usage and quota control, and direct the right inference request to the right model to protect their data and their IP—data sovereignty.

All of this has become the growth engine for our business. Right now, combining the neocloud and sovereign cloud side of the business, enterprise AI inferencing and hybrid AI, and edge—with some inference at edge locations—all three together mean that probably more than 60% of our total revenue is from AI use cases.

You mentioned some of the customer types you work with—sovereigns, neoclouds, enterprises. Can you give us an idea of your customer mix, in terms of how large your customers are and which industries they cluster in?

Most of our customer base is Global 2000 large enterprises, the public sector, and service providers—three categories.

For enterprises, it's mostly large enterprises; we don't work with SMBs that much. For the public sector, it can be DoD and intelligence, or civilian. For service providers, it spans neoclouds and sovereign clouds to managed service providers and telcos. A lot of those are building GPU infrastructure and providing infrastructure services to external customers.

When you say managed service providers in this context, what do you mean exactly?

A lot of managed service providers now want to call themselves neoclouds as well. Traditionally they own their own data centers and provide managed services worldwide to enterprises, acting as a dedicated provider managing the enterprise's infrastructure.

A lot of our readers are relative outsiders to this category. If you were drawing a verbal market map, what are the main buckets at this layer of AI infrastructure, who do you see as the competition, and where are the dividing lines between companies?

I wouldn't really think about Kubernetes anymore, because Kubernetes is just one piece of the puzzle. In the AI world it's about full stack GPU infrastructure management. Within that category, you can provide GPU as a service, Kubernetes as a service, or inferencing as a service. In NVIDIA's terminology, this becomes the AI factory and the token factory.

It also enables enterprises to provide not only GPU infrastructure and inferencing service, but to do it in a multi-tenant fashion with security and isolation, to handle multi-model routing so they can direct the right request to the right inferencing service, and to handle quota control, billing, and metering.

In this particular category, hybrid AI, the main competitor is probably Red Hat OpenShift AI. Obviously in the cloud, everyone is trying to move up the stack to provide some inferencing service. But Red Hat is probably the only credible competitor in terms of hybrid AI infrastructure, because they can work both on-prem and in the cloud environment. For most enterprises that's very important. Just like in the cloud era, large enterprises eventually embraced multi-cloud and hybrid cloud. In the AI era, everyone will eventually become multi-model and hybrid AI.

So hybrid means using models via cloud APIs and also using models locally?

Yes. You start on-prem for some of your model inferencing, with all your data staying local and your agentic workloads staying local. Then on an as-needed basis you can burst capacity out to the cloud, and selectively you can also route some inference requests to a frontier model. I'd say it will be roughly 60% on-prem, 30% bursting to the cloud, and 10% to the frontier model. That's how most enterprises will eventually go. And of that 60% on-prem, maybe 30% will eventually be at the edge and 30% in the data center.

How important are open source and open weight models to a world where an enterprise runs 60% of its AI usage on-prem?

That's really the key. The pace of innovation in AI is amazing, and the gap between the frontier labs and open weight models is getting shorter and shorter—now we see a one—to two-month gap at most. The performance of some of these open source models is very close to the frontier models, but at a fraction of the cost. Especially if they're hosted on-prem, you can essentially run your tokens without any cost other than your hardware CapEx. That makes a lot of economic sense.

Eventually, many of the frontier models will almost become a commodity. In the cloud era, the cloud pretty much became a commodity, and most enterprises wanted to treat it as multi-cloud so they weren't locked into a single environment. Ultimately it became about driving the right workload to the right cloud based on location and other SLA requirements. In the AI era, it will eventually be like that as well. With multi-model, you can drive the right inference request to the right model. One model doesn't necessarily fit all. If you run everything through the frontier model, you have security concerns and cost concerns, and it's overkill for a lot of simple agentic tasks.

A lot of people hear that a majority of AI usage will be on-prem, that open source is closing the gap, and that frontier lab AI is commoditizing—and then they look at Anthropic growing revenue at four digits percentage-wise year over year. There's a cognitive dissonance there. How can both things be true at the same time?

That's exactly what we've seen since the end of last year. December was really the first time Anthropic had a breakthrough with Claude Code. It was no longer just an AI assistant for coding—now it's fully AI-driven, agentic, automated coding. With the adoption of that, every enterprise has seen their token expense grow exponentially.

Even for a startup like us, from January to June, in six months, we saw our monthly token expense grow 10x. It was completely out of control. So we started to look for alternative solutions, and it turns out some of the open source models, like GLM, perform very close to Anthropic's models, especially for coding and cybersecurity tasks. You can run that inference locally while still, on an as-needed basis, sending some requests out where it makes sense. For example, in coding, for planning and reasoning, going to the frontier model is still better. But for a lot of the implementation phase, doing it locally through these open source models at almost zero cost makes a lot of economic sense.

That gives an enterprise the best of both worlds: their token expense is under control, but at the same time they aren't stopping or slowing down AI innovation.

Let’s return to NVIDIA's terminology—which was it again—token factory or AI factory?

For inferencing, with token metering and governance, they call it a token factory.

Spectro talks a lot about supporting customers from "metal to token." Is it fair to say Spectro Cloud will allow any enterprise—from a neocloud to a non-AI-native enterprise—to build its own token factory, from the low level GPU layer all the way up to the token and inference level?

Correct. We enable them so that they don't need to worry about the complexity of the AI infrastructure and can focus more on their business applications: how to utilize AI to improve their business processes and impact their top line revenue.

We're hearing a lot recently about AI gateways—even Ramp has launched one. How is that different from what you do? How should people think about your product for routing and cost savings versus what a Ramp or another AI gateway or a multi-model router offers?

The AI gateway or multi-model router will very quickly become a commodity, because everyone realizes this is a big problem and everyone sees the market need for multi-model routing. But the router or the AI gateway by itself isn't enough. What most enterprises really need is a full stack turnkey solution.

You have local inferencing and the entire stack—OS, Kubernetes, storage, network, all the way to the inferencing engine—plus open source models optimized for the AI infrastructure the enterprise is using. You need to be able to support that and keep it up to date and secure across the whole stack: Kubernetes, the OS runtime, and the models. Then you combine that with a multi-model router with governance, visibility, and routing rules, all as a turnkey solution. That's really the key for enterprises.

And this isn't just for one cluster. To reduce their blast radius, they will eventually grow to multiple clusters with different models for different purposes, so they need to manage these at scale. We're very uniquely positioned because we can manage metal to token, and not only for a single server or single cluster but with fleet management at scale, sometimes more than 10,000 clusters. That enables enterprise customers to get to the business outcome very quickly, while we can grow as they grow—it future-proofs their scalability.

Zooming out a little, what do people most often get wrong about AI infrastructure trends today?

In the early days, there was a lot of prototyping, and people didn't realize how complicated the AI stack is for production usage. Pulling a model from Hugging Face and running it is relatively easy. But keeping the entire AI stack secure and up to date, keeping all the models up to date, and managing those at scale at production grade is really not easy.

The other thing I’d add—and this may be controversial—is that with AI, with Claude Code and all this agentic coding, we're starting to see more DIY efforts, and that’s not always going to work. Just this year we've seen a lot of SaaS companies affected because people think they can just use Claude Code, do some vibe coding, and try to replace an entire system. But in reality, a lot of software solutions exist today because of integration, governance, and understanding the enterprise business workflow. Those are more important than just coding things up and creating some business logic.

There will be a learning curve for enterprises around this. When they start to adopt agentic coding and try to do DIY 2.0, maybe some will succeed. But then they will eventually realize that beyond writing the code, day-two operations—keeping everything secure and updated, and maintenance—is a really big burden for an enterprise to handle by itself. Many will still go back to turnkey solutions.

If there's a service like AWS Bedrock and the other tools in the hyperscalers' stack offering this very deep layer of services, why isn't it just easier to do everything in the cloud? Can't the security and other issues be worked out? If an enterprise can offload that complexity, why wouldn't it, rather than managing hybrid AI itself, even with a tool like Spectro Cloud?

The cloud has almost unlimited scalability, which is great, and it supports a pay-as-you-go model, so you don't have to make a very heavy investment upfront. Those are very unique advantages.

However, enterprises still have a lot of data on-prem, and more and more data is generated at the edge. That's the real limitation of the cloud: for AI to be efficient, it needs to stay close to where the data is, so it can process the data faster and in near real time to make business decisions. Running hybrid AI or edge AI gives you a unique competitive advantage because you stay closer to where the data is and can make decisions faster.

The only challenge is that doing edge AI and hybrid AI on-prem isn't easy. That's why, using a solution like Spectro Cloud, we can help them solve the infrastructure management problem so they can realize the competitive advantage and value of edge AI or hybrid AI.

You mentioned that edge deployments are one area where the management burden really bites. How significant is edge AI to your business, and why is it becoming more important?

A lot of data is generated at the edge, and it's always better to shift left and have the compute infrastructure stay closer to where the data is so it can process data faster. We see a lot of retailers, restaurants, manufacturers, hospitals, logistics companies, and we see it on the defense side also—every moving object, an airplane, tank, submarine, ship, even satellites—they all have a lot of data. They have real-time data processing requirements, whether for latency or for user experience. You want all that data processed directly at the edge. That in turn requires full stack infrastructure management at the edge and the ability to run inference at the edge, so decisions can be made directly alongside where the data lives.

So your software sits at the edge and gives you visibility there?

Yes, and we manage it all together. We centrally manage all these edge locations so the user doesn't need to go to an edge location to manually do the deployment and updates. That also helps customers reduce a lot of the operating cost of managing edge infrastructure.

Do you think people underestimate the degree to which AI will happen at the edge over time?

Yes. Edge AI is still at a relatively early stage, because we're still in the AI infrastructure build-out phase. People are still putting a lot of GPU infrastructure into the neoclouds and hyperscalers, and now enterprises are starting to build out data centers for GPU infrastructure too.

But some specific verticals already see the big value of the edge—retailers, healthcare, oil and gas, manufacturers, defense. We'll start to see more and more of these industries wanting to move inferencing to the edge. That will be the overall trend. That's why, in our mind, hybrid AI and on-prem AI include both the on-prem data center *and* the on-prem edge. Eventually it will be fifty-fifty between those two.

What's the main bottleneck to AI happening at the edge right now?

Let me give you a concrete example. Retailers have a lot of video streaming at the edge, and they want to use video to check customer traffic. Let’s say a customer comes in—they may want to do facial recognition and then run an in-store promotion. Sending all that video streaming back to the corporate data center or the cloud means very long latency and is very expensive. The bandwidth and the latency really become a problem. But if you have edge infrastructure installed, you can process those video streams directly, do the facial recognition, and make the real-time decision for the in-store promotion. That's a much better user experience, at lower cost and with less latency. That's what makes all this edge-side AI innovation possible. Without that kind of edge infrastructure, relying on the corporate data center or cloud for everything is hard.

Is there also a significant limitation in terms of the edge compute power and hardware?

No, a lot of it is a management constraint, because people just don't think it's possible to manage the entire edge infrastructure stack centrally at scale.

The edge has very unique challenges. You can't expect a store manager to install the OS, Kubernetes, GPU drivers and everything themselves, but at the same time sending a field engineer to these edge locations is very difficult and expensive. So a centralized management system that can remotely do all of this without a field engineer is a must-have.

Also, the edge is by definition less secure out of the box than a corporate data center or cloud environment. The edge device can be stolen, for example. So how do you keep everything end-to-end secure, so people can't tamper with the data or peek into the data? That's also very important. And the edge environment may not be as reliable—the internet can go down, Wi-Fi can have problems, power can be lost, hardware can fail. How do you deal with all these disruptions and handle resiliency in these scenarios? These are very unique challenges in the edge environment, and our platform helps solve all of them.

You spoke earlier about how agentic workflows caused AI spend to skyrocket. Ten years ago there were similar conversations about AWS bills—"my AWS bill came and it was so high." Do you see more similarities or differences in how cloud spend and AI spend are thought about? How is this time different, or is it a replay of how people thought about cloud spend before?

To some extent, history repeats itself, just in a new way in the AI era. Now, even with all these neoclouds and frontier models, they eventually become a commodity and a utility. It becomes hard to differentiate on anything besides price performance. Ultimately users want to make decisions based on price, location, and SLAs. That's ultimately how enterprises adopted their multi-cloud or hybrid cloud strategy.

Now we see the same happening in the AI era. But AI is still at a very early stage right now, so people are still building out all this infrastructure. Not many people are thinking yet about building redundancy across multiple AI infrastructures and across multiple models. But as more workloads get into production for business applications with AI, that will become a requirement as well. That's the similarity to the multi-cloud and hybrid cloud movement.

If Spectro Cloud continues doing well, how does it change the industry and the world for its customers? What does it look like if everything goes right for the company in the next few years?

I came from VMware, and I always admired VMware's growth in my early days, although I'm not happy that it eventually ended up in Broadcom's hands.

We're very well positioned to become a VMware-like company in the AI era. Just as VMware managed virtualization and enabled the cloud to provide infrastructure services, we can own the hybrid AI software foundation stack. Whether a customer needs inferencing, an agent sandbox, or a RAG service, all of these AI foundation services can be provided by a solution like ours. Then we can help them manage all that infrastructure efficiently, whether it's in the data center, at the edge, or in a cloud environment. That will become the enabler for enterprises to become AI enterprises and embrace a hybrid AI strategy.

Cloudflare CEO Matthew Prince has framed the open question as who becomes "the VMware of AI," since nobody has done for GPUs what virtualization did for CPUs. Do you agree with that sentiment?

Yes, there's nothing out there quite like that. And this is beyond just having virtualized GPUs. What VMware really did was enable hybrid cloud as a foundational layer. We want to be the leader for hybrid AI.

This transcript is for information purposes only and does not constitute advice of any type or trade recommendation and should not form the basis of any investment decision. Sacra accepts no liability for the transcript or for any errors, omissions or inaccuracies in respect of it. The views of the experts expressed in the transcript are those of the experts and they are not endorsed by, nor do they represent the opinion of Sacra. Sacra reserves all copyright, intellectual property rights in the transcript. Any modification, copying, displaying, distributing, transmitting, publishing, licensing, creating derivative works from, or selling any transcript is strictly prohibited.