You moved your agent stack to small models to cut the inference bill, and the bill didn't move. That surprises people. It shouldn't. The money's never been in the calls. It's in the servers holding the models, and small models need servers too.
The cost was never in the calls
Most agent stacks now run four or five models under one roof: an embedder for retrieval, a reranker for precision, an extractor for entities, and a small LLM for the routine generation. The standard way to serve them is one server per model. vLLM holds the LLM, TEI holds the embedder, and everything else gets a hand-rolled FastAPI wrapper. Each of those servers reserves its own slice of GPU memory and holds that slice whether a request arrives or not. GPUs bill by the hour. Idle time costs what busy time costs, to the penny.
So serving five models the standard way means five reserved slices, mostly idle, all billing around the clock. Swap in smaller models and you've trimmed the compute per call and left the reserved hardware exactly where it was. That's the trap. The line item you were trying to cut lives one layer below the thing you changed.
The hardware doesn't get cheaper
And the hardware is the part that isn't getting cheaper. NVIDIA's RTX Pro 6000 went from around $8,565 at launch to $13,250 inside sixteen months, and there's no second vendor in the room to argue the price down. Every idle reserved GPU is that number, burning whether it's doing a lick of work or not.
One process, many models
The structural fix is to stop handing every model its own server. Serve them all from one process that loads and evicts models based on traffic. Superlinked went and open-sourced exactly that. SIE, the Superlinked Inference Engine, is an Apache 2.0 server that runs 85-plus models behind a single API.
Four calls cover the whole workflow. encode() returns vectors, score() returns relevance scores, extract() returns entity spans, and generate() runs small open LLMs. Under those four primitives SIE wraps PyTorch, SGLang, Flash Attention, and Apple MLX, and it'll pick the right engine per model on its own, so you're calling one uniform interface instead of maintaining four.
Where the bill moves
The memory management is what moves your bill. Models load lazily on first request and get evicted least-recently-used when the GPU fills. One card serves a rotating working set instead of sitting siloed behind a single model, and the five idle slices collapse into one shared, hot pool. Superlinked's own numbers put the saving near three-quarters against the one-server-per-model layout. The exact figure depends on your working set, but the direction isn't in doubt: you're done paying for a card to sit still.
Air-gapped, and yours to own
Two details carry SIE past the cost line. I'd start with the air-gapped one. It ships model-weight snapshots for offline and air-gapped deployment, so the whole server's running disconnected, with no phone-home and no license check sitting on the far end of a network you don't control. The second is reach. The same Docker image, Helm chart, and Terraform modules run from a laptop to a Kubernetes cluster, so a two-desk shop and a rack estate run the identical software, each sized to what it does. It's Apache 2.0. You own the serving layer the way you already own open weights, and no vendor's metering it or deprecating it out from under a workflow you depend on.
For a tribe holding its own data, a defense shop under ITAR, a firm holding privilege, a clinic holding PHI, that stack's the whole argument: the models run on hardware you control, served by software you own, in a room you can pull off the internet at the breaker.
What SIE isn't
It's not your heavyweight generation server. generate() runs small open LLMs; it doesn't stand in for vLLM feeding a 70-billion-parameter chat model under load. SIE's embeddings, reranking, and extraction first, with light generation riding alongside. The two belong in the same rack, not in place of each other.
The eviction trick carries its own bill. An evicted model reloads on its next request, and that first call pays the load latency while the weights come back onto the card. The saving holds when your working set fits the GPU and rotates through it. It thins if every request drags in a cold model nobody touched a minute ago. Size it to the traffic you've got, not the model catalog you admire.
And Apache 2.0 isn't a managed service. Somebody stands it up, watches it, patches it, and carries the pager when it falls over at 2 a.m. Open source moves that work in-house. It doesn't delete it.