Orchestrating Heterogeneous Agent Compute
Gimlet Labs
This reveals that inference is turning into a scheduling problem, not just a model hosting problem. In an agent, retrieval, prefill, token generation, tool use, and response assembly have very different speed and memory needs, so pushing them all through one GPU leaves money and latency on the table. Gimlet is built to break that chain into smaller jobs, then match each one to the cheapest and fastest available hardware and runtime path.
-
This is a different bet from GPU first clouds like Fireworks AI, which mostly optimize how open models run on large GPU fleets through better batching, caching, and serving. Gimlet is trying to optimize one level deeper, at the workflow graph itself, including non GPU accelerators and private datacenter installs.
-
It is also the mirror image of Groq. Groq argues a purpose built chip plus cloud can win many inference workloads through deterministic low latency on one architecture. Gimlet starts from the opposite premise, that no single chip stays best across every step of an agent loop, especially as workloads mix long context, fast decode, and tool execution.
-
The product only matters once inference looks like production software, not a demo. Earlier AI infrastructure buyers often chose managed GPU clouds like CoreWeave because they wanted autoscaling, APIs, and reliability without building their own cluster. Gimlet extends that same operational convenience to a much more fragmented hardware environment.
As agents handle longer sessions, more tool calls, and more multimodal work, the winning inference layer will look less like a single endpoint and more like an operating system for mixed compute. That pushes the market toward companies that can orchestrate across chips, clouds, and private infrastructure, and it gives Gimlet a path to become the control plane sitting above increasingly fragmented AI hardware.