The H100 Is the Faster Card. Run the Arithmetic Anyway.
For two years the default answer for on-premises inference was a datacenter Hopper card. The NVIDIA H100 was the part everybody benchmarked against, and for organizations that can't send data to the cloud (law firms, medical practices, defense contractors, tribal nations) the question was usually how many refurbished H100s to buy and how to cool them.
That question aged badly. Not because the H100 got worse, it didn't. It's that a new professional Blackwell card turned up at a fraction of the secondary-market price with more VRAM on it, and the two numbers that decide on-premises inference economics, cost per token and total cost, quit pointing at Hopper. So here's the reasoning with the math rebuilt from current specs, and it's laid out so you can run it yourself against whatever parts you're weighing. I'm not going to tell you what to buy at the end of it. That decision doesn't belong in a blog post, and I'd be lying if I pretended it did.
The Headline Numbers
Stripped down, per card:
| GPU | Memory | Memory Bandwidth | FP16 Tensor | FP8 Tensor | Power | Street Price |
|---|---|---|---|---|---|---|
| RTX PRO 6000 Blackwell Server Edition | 96GB GDDR7 ECC | 1,597 GB/s | 1,000 TFLOPS | 2 PFLOPS | up to 600W | ~$8,000 - $9,200 new |
| H100 80GB PCIe (Hopper) | 80GB HBM2e | 2,000 GB/s | ~756 - 1,000 TFLOPS | ~1.5 - 2 PFLOPS | 350W | ~$25,000 - $33,000 refurbished |
The H100 holds a raw bandwidth edge: 2,000 GB/s of HBM2e against 1,597 GB/s of GDDR7 on the Server Edition part. On that one dimension Hopper wins, period, and I'm not going to talk anybody out of it. Then look at the rest of the row. There's 96GB against 80GB, there's comparable power draw, there's a card that's brand new with a warranty behind it, and there's a sticker that's roughly a third to a quarter of what a secondary-market H100 costs (NVIDIA RTX PRO 6000 specifications). The spec sheet isn't the whole story and the price line's where this gets decided.
Why Bandwidth Sets the Pace for Token Generation
Here's the concept most spec-sheet comparisons skip, and it's the one that settles the purchase.
When a model generates text one token at a time (the decode phase) it's memory-bandwidth-bound, not compute-bound. To produce each token the GPU reads the active model weights out of VRAM, multiplies, and writes the result back. The multiplication's fast. It's moving the weights across the memory bus that takes the time, and that's true no matter whose logo is on the card.
Which gives you a clean back-of-envelope ceiling for single-stream decode:
tokens per second ≈ memory bandwidth ÷ active weight bytes, then multiply by a real-world efficiency factor of roughly 50 to 70 percent for kernel overhead, attention, and KV-cache traffic.
Learn that one line and you don't need me for this part. It'll work on any card with a published bandwidth figure, and that includes every part I've never had my hands on.
Take DeepSeek V4-Flash. It's a mixture-of-experts model: 284 billion total parameters, only about 13 billion active per token, roughly 4.6 percent of the weights. At FP8 those active weights come to about 13GB. At 1,597 GB/s that's 1,597 / 13 ≈ 123 tokens per second in theory. Apply the efficiency factor and you're at a realistic 60 to 90 tokens per second single-stream (estimate). That's faster than most people read.
Now run the same arithmetic on a dense 70B at FP8. The full ~70GB of weights moves for every single token: 1,597 / 70 ≈ 23 tokens per second per card (estimate). Split it across two cards with tensor parallelism and you're at roughly 40 to 45 tokens per second (estimate). That's the whole reason V4-Flash feels several times quicker than a dense 70B on identical silicon. It's moving a fraction of the bytes per token, and that's it, that's the trick.
Every figure up there is a conservative single-stream estimate. Batching concurrent requests raises aggregate throughput, because weights loaded once serve several sequences, but per-user speed stays governed by bandwidth. They're planning figures, not guarantees.
Where the H100's Bandwidth Edge Does and Doesn't Matter
The H100's 2,000 GB/s is about 25 percent higher than 1,597. By the roofline above that's roughly 25 percent more single-stream tokens per second for the same model at the same precision. Real, measurable, and I'd rather concede it in the first sentence than bury it in a footnote.
It also doesn't change the buying decision for most regulated deployments, and the reasons aren't complicated. V4-Flash already clears 60 to 90 tokens per second on the Blackwell part, which is well past reading speed, so a faster card mostly shortens a wait that wasn't long. The H100 PCIe part you can buy today is refurbished secondary-market stock at $25,000 to $33,000 a card, against $8,000 to $9,200 for a new card that somebody will still honor a claim on. And 96GB fits more model than 80GB does. So you're paying three to four times as much per card to recover 25 percent on a workload that was already fast enough, and you're handing back VRAM headroom and warranty coverage to do it.
A 25 percent margin is the kind of number that wins an argument on a slide and loses one in a building. For most buyers that trade doesn't close. For a few it does, and if yours is one of them the arithmetic above will say so louder than I can.
What Fits Where
VRAM's the gate, and it's arithmetic rather than opinion.
V4-Flash served at FP8 wants roughly 170GB once all 284B parameters are resident and the KV cache has somewhere to live. At full FP16 precision it wants 384GB. A dense 70B at FP8 is about 70GB, so it'll sit on a single 96GB card with room to spare and it runs quicker split across two with tensor parallelism. That's the arithmetic, all of it. Divide by whatever your card carries and you've sized the memory half of the problem in about ten seconds, no vendor required.
Blackwell professional cards also support up to four MIG instances each and they carry ECC memory, so a box gets partitioned for isolated workloads and a memory error gets corrected instead of quietly corrupting a result. For regulated data that correctness guarantee is the difference between an answer and an answer you can defend, not a bullet point.
The Cost-Per-Token Math
Here's the version that decides budgets. Depreciate whatever you end up buying over three years against a billion tokens a year, a reasonable figure for a medium-sized organization, and the per-token hardware cost is a rounding error.
Cloud inference from the major providers runs $0.003 to $0.01 per token on comparable models. Even after electricity, cooling, and staffing, inference on hardware you own comes in cheaper by two to three orders of magnitude. The capital cost's front-loaded; the unit cost is low and it stays low for the life of the box. That's the whole shape of the trade.
Now price the same capacity in refurbished H100s. Two H100 PCIe cards at $25,000 to $33,000 each is $50,000 to $66,000 in silicon alone, before chassis, cooling, and integration, for 160GB with nobody standing behind it. That same money in new Blackwell cards buys more VRAM, ECC, and a warranty you can call somebody about. The cost-per-token line moves the same direction: lower capital, more memory, same order-of-magnitude advantage over cloud.
What the Throughput Looks Like Against Real Load
Do this part with your own headcount, not mine. A firm with 30 attorneys running 10 queries each per day is 300 queries. At 500 tokens a response that's 150,000 generated tokens a day. At 60 to 90 tokens per second you're well under an hour of cumulative generation spread across a workday, and you've got idle headroom you'd have to work at to consume. Scale it to 300 attorneys and 1.5 million tokens a day and the arithmetic still isn't frightening for sequential and loosely concurrent load.
Concurrency is what bends that curve, along with full-precision residency and holding several large models at once. Every one of those is a fact about your own shop, they're knowable, and they're knowable well before anybody quotes you a thing.
Build vs. Buy
People ask whether they can assemble something cheaper out of consumer cards or scavenged datacenter hardware. You can. You usually shouldn't.
A stack of consumer RTX cards or a pallet of refurbished H100s gets you raw silicon and that's all it gets you. It isn't a working inference system yet. Then come the cooling, the networking, the driver matrices, the firmware tuning, the systems administration, and the total absence of anybody to call when a secondary-market card dies at two in the morning. By the time it's built and debugged the capital savings are gone and you've got an integration project instead of a tool. I've watched that happen to genuinely competent people.
The Honest Guidance
I don't have a recommended configuration for you, and that's because I haven't been in your building, not modesty. I don't know your concurrency, your document lengths, your precision requirements, or which department's workflow is going to turn out to be the real load. Neither do you, entirely, until somebody sits down next to the people doing the work and watches them do it. That visit comes before the parts list, not after. Every deployment I've seen go sideways got that order backwards.
What this post gives you is the arithmetic, because it's yours to keep and it works whether you ever call me or not. Bandwidth divided by active weight bytes, times a half to two-thirds. Model weights divided by VRAM. Street price divided by the tokens the thing'll really produce.
The H100 was the right tool for its moment, and the people who bought them then weren't wrong. For a new on-premises deployment in 2026, a new Blackwell professional card wins on cost per token and on total cost for the workloads regulated organizations run. Run the numbers and find out whether yours is the exception.