Choosing a GPU Cloud: Hyperscaler vs Specialist GPU Cloud vs Serverless Inference

Infrastructure AI Cost
5 min read

Once you have decided to grab GPUs directly (see the break-even math), the next question is where to rent them. The same H100 runs 2〜5x apart in hourly rate depending on where you rent it. The conclusion up front: if your infrastructure already lives in AWS and integration and regulation matter, go hyperscaler; if pure GPU cost is the goal, a specialist GPU cloud; if you would rather not manage GPUs at all, serverless inference. The GPU server’s structure is covered in What Makes a GPU Server Different; this post is the choice of where to source that GPU. Prices are as of mid-2026.

Three branches: differing management levels #

GPU sourcing splits into three branches by management level.

  • Hyperscalers (AWS, GCP, Azure): GPU instances (AWS P5 and so on) used inside the same account, network, and IAM as your other cloud resources. H100 per-GPU on-demand runs $4〜8 an hour at hyperscalers, and AWS P5 is about $3.9 per GPU after a mid-2025 cut.
  • Specialist GPU clouds (Lambda, RunPod, CoreWeave, and others): providers specialized in GPU rental. H100 runs $2〜3.3 on-demand (RunPod PCIe $1.99, Lambda $3.29), and $1〜2 on spot — the same GPU 2〜5x cheaper than a hyperscaler.
  • Serverless inference (Together, Fireworks, and others): not renting a GPU but paying per token for model inference. Call open models at $0.05〜7 per million tokens (usually $0.27〜3), with no GPU provisioning or management at all.

There is one way to read it: moving left adds integration and management and raises the price; moving right approaches pure GPU cost but adds things you must handle yourself.

Why the price gap is 2〜5x #

Hyperscalers cost several times more than specialist GPU clouds not because the GPU differs (it is the same H100) but because of what is bundled into the price. Hyperscaler rates fold in VPC networking within the same account, IAM, low-latency integration with other managed services (S3, RDS), and enterprise SLA and compliance. Specialist clouds strip that integration and focus on the GPU itself to lower cost.

So the decision narrows to “do you need that integration?” If training data already sits in S3 and the pipeline lives in AWS, the transfer cost, latency, and security review of pulling data out can offset a specialist cloud’s savings. Conversely, for experiments, training, or batch that need only an independent GPU, the specialist cloud’s cost advantage stands as-is.

Spot: below half, in exchange for interruptions #

Both specialist GPU clouds and hyperscalers offer spot (interruptible) rates. H100 spot drops to $1〜2 — below half of on-demand. The price is that it can be reclaimed at any time.

  • Fitting workloads: training that checkpoints as it runs, batch inference that resumes after interruption, and reproducible experiments. If reclaimed, grab another and continue.
  • Misfitting workloads: production inference endpoints that must respond continuously. If the GPU is reclaimed mid-request, the service breaks.

That is, the distinction from Training vs Inference Servers reappears here. Training and batch lower cost on spot; always-on inference buys stability with on-demand or reservations.

Serverless inference: remove GPU management entirely #

Serverless inference bills per token instead of per GPU-hour, removing the entire GPU-management burden of self-hosting. Pick a model, call an API, done — the provider handles scaling, availability, and GPU provisioning. What differs from a commercial API (OpenAI and the like) is that you can freely pick open models, usually cheaper.

The boundary follows the same logic as break-even. For low-utilization intermittent inference, serverless wins by using GPUs with no idle time; for throughput that saturates a GPU 24 hours, grabbing GPUs directly on a specialist cloud and running batch is cheaper per token. Batch inference is also about 50% off on serverless, filling the middle.

The selection order #

  1. Filter on regulation and integration first: if data cannot leave a specific cloud, or integration with an existing pipeline matters, go hyperscaler.
  2. Decide the intent to manage: without people or the will to operate GPUs directly, use serverless inference — per-token billing makes the GPU disappear.
  3. Pick the source by workload: to grab GPUs directly for cost, a specialist GPU cloud; for always-on production inference, on-demand or reserved; for training and batch, spot.
  4. Re-compare against serverless by utilization: intermittent inference goes to serverless, sustained saturation goes to direct sourcing per token. Apply the break-even math as-is.
  5. Confirm on total cost: add data transfer and integration cost to a specialist cloud’s low GPU rate, and integration savings to a hyperscaler’s high rate, then compare totals.

Summary #

  • GPU sourcing splits three ways by management level: hyperscaler (integrated, pricey), specialist GPU cloud (cost, self-managed), serverless inference (per-token, no management).
  • Hyperscalers cost 2〜5x more than specialist clouds on H100 not for the GPU but because integration and SLA are bundled in. Whether you need that integration is the criterion.
  • Spot is below half of on-demand but accepts interruptions. Training and batch on spot, always-on inference on on-demand or reserved.
  • Serverless inference removes GPU management in exchange for per-token billing. It favors intermittent inference; sustained saturation is cheaper via direct sourcing.
  • Confirm a specialist cloud’s price advantage on totals including data transfer and integration cost. A raw rate comparison is half a calculation.
X