Hosted API vs renting a GPU for a mid-size model
Updated · prices checked on that date
A mid-size open-weight model in 4-bit fits on one 48 GB GPU, so self-hosting is realistic. Whether it pays off depends on how many tokens you use every month.
Key points
- In 4-bit, a model of 30–40B parameters takes roughly 20–25 GB, so one 48 GB card is enough for a 64,000-token context.
- A rented L40S costs $0.79–$1.09 an hour on RunPod: about $577–$796 a month around the clock.
- At Dolphin LLM prices, $577 buys roughly 2.6B tokens a month in a typical 80/20 input/output mix.
- Self-hosting makes sense at steady, very high volume; below that the hosted API costs less and needs no upkeep.
What hardware such a model needs
In 4-bit, the weights of a model with 30–40B parameters take roughly 20–25 GB. On a 48 GB card that leaves roughly 20 GB for the KV cache, which is enough for a 64,000-token context with a few parallel requests. Serving a full 262,144-token window yourself needs a lot more cache memory than that, or fewer parallel requests. Longer contexts or more parallel users need more memory or more cards.
Rental prices
| GPU | Where | Per hour | Per month, 24/7 |
|---|---|---|---|
| L40S 48 GB | RunPod Community / Secure | $0.79 / $1.09 | $577 / $796 |
| L40S 48 GB | Vast.ai (median of 4 offers) | $0.80 | $584 |
| RTX A6000 48 GB | RunPod Community / Secure | $0.33 / $0.59 | $241 / $431 |
| A100 80 GB | RunPod Secure | $1.79 | $1,307 |
| H100 80 GB | RunPod Secure (PCIe) | $2.89 | $2,110 |
Monthly figures are hourly price × 730 hours, without storage, traffic or the time the card sits idle.
Break-even
A typical mix of 80% input and 20% output tokens costs $0.22 per 1M tokens on Dolphin LLM. So the monthly price of a rented card buys:
| Monthly budget | Equals on Dolphin LLM | Per day |
|---|---|---|
| $241 (RTX A6000, Community) | about 1.1B tokens | about 36M |
| $577 (L40S, Community) | about 2.6B tokens | about 87M |
| $796 (L40S, Secure) | about 3.6B tokens | about 120M |
If you use fewer tokens than this, the hosted API is cheaper. Above it, a rented card can win, provided it is busy most of the day.
Costs that are not on the price list
- Setup and updates: inference server, drivers, model downloads, monitoring.
- Idle time: a rented card costs the same at night when nobody sends requests.
- Peaks: one card serves a limited number of parallel requests; more users mean more cards.
Which one to choose
- Self-host if you have steady, very high volume, need full control of the stack or a context longer than 262,144 tokens.
- Use Dolphin LLM if your volume is below the break-even point, varies from day to day, or you do not want to run servers.
Sources
Prices change. Check each provider's page before you decide.
Questions and answers
Can I run a model of this size on my own computer?
Yes, with a GPU that has enough memory: in 4-bit such a model takes about 20–25 GB, so a 48 GB card is comfortable and a 24 GB card is too small for useful context.
Is the hosted API slower than a local GPU?
It adds network latency on top of generation time. Generation speed depends on the load on the shared servers; for chat and most batch work the difference is small.
More guides
Dolphin LLM vs the OpenAI APIDolphin LLM next to OpenAI GPT-6 Luna and GPT-6.1 Terra: price per 1M tokens, context window, data policy, billing and when each one fits.
Dolphin LLM vs mid-size open models on OpenRouterPrice per 1M tokens for a popular mid-size open-weight chat model on OpenRouter providers and DeepInfra next to Dolphin LLM, plus credit fees, context and when each is cheaper.