How to Run Dolphin LLM: Local Setup vs Hosted API

Decide first: what hardware do you have?

Memory decides everything. A language model is a very large set of numbers, and the whole set has to sit in fast memory while it generates. Quantization shrinks those numbers. A 4-bit quantized model takes roughly half a byte per parameter, so a model with 8 billion parameters lands near 5 GB, and one with 70 billion lands near 40 GB. Treat these as rough figures, since the exact size depends on the quantization format.

On top of the weights you need space for the context cache. It grows with the prompt length, so a long roleplay history or a big document costs more memory than a short question. Plan for a few extra gigabytes at modest context, more if you push toward long prompts.

• 8 GB laptop with no GPU: only small models, slow output.
• 16 to 32 GB Apple Silicon Mac: mid-size models run well because the CPU and GPU share memory.
• Single consumer GPU with 12 to 24 GB VRAM: mid-size models at good speed.
• Anything bigger: multiple GPUs or a rented server.

Method 1: Ollama in a few commands

Ollama is the shortest path. It downloads a quantized model, starts a background server and gives you a chat prompt. Install it from the official site for your operating system, then check which Dolphin tags the Ollama model library currently lists, because names and sizes change between releases.

Copy# replace <tag> with a Dolphin tag from the Ollama library
ollama pull <tag>
ollama run <tag>

The first command downloads the file, which can take a while on a slow connection. The second opens an interactive chat in the terminal. Ollama also serves an HTTP API on your machine, including an OpenAI-style route under /v1, so other programs can use it.

If generation is very slow, the model probably did not fit in VRAM and part of it runs on the CPU. Pick a smaller tag or a lower-bit quantization.

Method 2: llama.cpp for more control

llama.cpp is the C++ engine many other tools are built on. You download a GGUF file of the model, then start its server. This takes more steps than Ollama but lets you set the context length, the number of layers placed on the GPU, and the quantization you want.

Copy# after building llama.cpp and downloading a GGUF file
llama-server -m ./model.gguf -c 8192 -ngl 99 --port 8080

Here -c sets the context length, -ngl sets how many layers go to the GPU, and --port picks the port. Check the project's README for the flags in your version, as they evolve. A larger -c uses more memory, so raise it only when you need it.

For graphical use, front ends such as Open WebUI can talk to either server. Our note on Open WebUI API endpoints shows how to add a custom endpoint.

Calling your local server from code

Both tools can speak the OpenAI chat format, so you write the client once and switch the target with two settings. This script holds both targets side by side. The local one uses the port Ollama uses by default; change it if you started llama.cpp on 8080.

Copyfrom openai import OpenAI

TARGETS = {
    "local": {"base_url": "http://localhost:11434/v1", "key": "not-needed", "model": "your-local-tag"},
    "hosted": {"base_url": "https://api.dolphinllm.com/v1", "key": "YOUR_KEY", "model": "uncensored"},
}

t = TARGETS["hosted"]
client = OpenAI(base_url=t["base_url"], api_key=t["key"])

resp = client.chat.completions.create(
    model=t["model"],
    messages=[{"role": "user", "content": "Summarize what a GGUF file is."}],
)
print(resp.choices[0].message.content)

Local servers usually ignore the key, but the SDK requires a non-empty string. The model name for a local server is whatever tag or alias you loaded, while the hosted model ID is always uncensored.

Method 3: skip the setup and use a hosted endpoint

If the hardware math looks bad, or you want the same behavior on every machine, a hosted endpoint is simpler. This service provides one open-weight model, run by an independent provider and tuned to answer without content refusals for lawful adult use. It is not the official API of any model vendor, and it does not let you pick other models. What you get is a stable URL, a key, and billing by the token.

The steps are short. Sign up with Google or with email and password, open the API key page, and claim the $0.50 trial credit with one click. It needs no card, is valid for 7 days, and is limited to one per person. Then run the Python script above with the hosted target. If you prefer curl:

Copycurl https://api.dolphinllm.com/v1/chat/completions   -H "Authorization: Bearer YOUR_KEY"   -H "Content-Type: application/json"   -d '{"model":"uncensored","messages":[{"role":"user","content":"Hello"}],"stream":false}'

The context window is 100,000 tokens, which is more than most laptops can hold in cache comfortably. Prompts are not used for training.

Cost and effort compared

Local feels free, but it is not. You pay for hardware, electricity and your own time. Hosted costs money per request, but the numbers are small. At $0.15 per 1M input tokens and $0.50 per 1M output tokens, a chat turn with 3,000 input tokens and 700 output tokens costs:

• Input: 3,000 x $0.15 / 1,000,000 = $0.00045
• Output: 700 x $0.50 / 1,000,000 = $0.00035
• Total: $0.0008 per turn, so $0.80 per 1,000 turns

Even a heavy user sending 5,000 such turns in a month would spend about $4. A GPU that costs hundreds of dollars would need years of use to break even on that volume. The comparison flips when you have continuous high volume, need a specific fine-tune, or cannot send data off your machine.

Full price details, including the +5%, +10% and +15% top-up bonuses, are on the pricing page, and the documentation covers limits and parameters.

A simple rule for choosing

Run locally if you want to experiment with many fine-tunes, work offline, or keep every prompt on your own disk. Use the hosted endpoint if you are building something other people will use, if your hardware is modest, or if you want a long 100,000-token context without managing memory. Many developers do both: a local model for private tinkering, and a hosted endpoint for anything that needs to stay up. Because the client code is the same, switching costs one line.

More guides

• Dolphin LLM API: Local, Rented GPU or Hosted
• Free LLM API for Developers: Trial vs Pay as You Go