How to Run a Local LLM with Ollama on a VPS: Step by Step

If you want an AI assistant that runs without sending your data to a third-party company, one of the most practical options is running an open-source model on your own virtual server. In this post we walk through running a local LLM with Ollama on a VPS end to end: installation, choosing a model, server settings, building a custom assistant, using the API, a browser interface, and publishing it securely. We covered the basics like model size, RAM needs and CPU vs GPU in our how AI runs on a virtual server post, so here we jump straight into practice.

Why Run a Local LLM on Your Own Server?

• Privacy: Questions, documents and answers never leave your server.
• Fixed cost: No per-token billing; the server's monthly fee is known.
• Independence: You're not affected by a provider's price, quota or policy changes.
• Customization: You can tune the model with your own system instructions for your own app.

Prerequisites

A virtual server running Ubuntu 24.04, root access, and enough RAM for the model you choose. For a 7B–8B model we recommend at least 16 GB of RAM; since model files are several GB, an NVMe disk also cuts load times significantly.

1. Install Ollama

The official install script installs Ollama and sets it up as a systemd service that starts automatically:

curl -fsSL https://ollama.com/install.sh | sh
systemctl status ollama      # is the service running?
ollama --version

2. Choose the Right Model

Ollama's model library has hundreds of models. The table below lists example options to get started; model families are updated quickly, so check ollama.com/library for the current list and versions.

What Do the Quantization Tags Mean?

The tags at the end of a model name show the compression level. If you don't specify one, Ollama usually downloads a 4-bit version.

Let's download the models and try them:

ollama pull qwen2.5:7b
ollama pull nomic-embed-text
ollama list                  # downloaded models
ollama run qwen2.5:7b "Hi, briefly introduce yourself."
ollama ps                    # models loaded in memory

3. Configure the Server

We tune Ollama's behaviour with environment variables. Using an "override" file instead of editing the service file directly keeps your settings safe across updates:

sudo systemctl edit ollama

# add these lines to the file that opens and save:
[Service]
Environment="OLLAMA_HOST=127.0.0.1:11434"
Environment="OLLAMA_KEEP_ALIVE=30m"
Environment="OLLAMA_NUM_PARALLEL=2"
Environment="OLLAMA_MAX_LOADED_MODELS=1"

# then restart the service:
sudo systemctl daemon-reload
sudo systemctl restart ollama

4. Build Your Own Assistant with a Modelfile

A Modelfile lets you package a model with your own system instructions and settings, so you don't have to send the same instructions with every request. Create a file called Modelfile in your project folder:

FROM qwen2.5:7b
PARAMETER temperature 0.3
PARAMETER num_ctx 4096
SYSTEM """
You are a company's support assistant. Answer briefly and clearly. Don't guess about things you don't know.
"""

ollama create support-assistant -f Modelfile
ollama run support-assistant

num_ctx sets how much text the model can "remember" at once. Raising it lets it handle longer documents but also increases RAM use and slowdown; 4096 is a good start on CPU servers.

5. Use It From Your App via the API

Ollama provides a local HTTP API. With its native API:

curl http://127.0.0.1:11434/api/chat -d '{
  "model": "support-assistant",
  "messages": [{"role": "user", "content": "What is an NVMe drive?"}],
  "stream": false
}'

Even more practical: Ollama also offers an OpenAI-compatible endpoint. So you can point an app written with the OpenAI library at your own server just by changing the address:

pip install openai

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:11434/v1", api_key="ollama")
reply = client.chat.completions.create(
    model="support-assistant",
    messages=[{"role": "user", "content": "What is a virtual server?"}],
)
print(reply.choices[0].message.content)

If APIs are new to you, our What is an API post is a good place to start.

6. Use It From the Browser with Open WebUI

Not everyone on the team needs a terminal. Open WebUI is an open-source app that connects to Ollama and gives you a ChatGPT-like interface. You can run it with a single Docker command:

docker run -d --network=host \
  -e OLLAMA_BASE_URL=http://127.0.0.1:11434 \
  -v open-webui:/app/backend/data \
  --name open-webui --restart always \
  ghcr.io/open-webui/open-webui:main

Thanks to --network=host, Open WebUI can reach Ollama, which is only accessible from inside the server (127.0.0.1), and it runs on port 8080. The account you create on first launch becomes the admin account; you manage other users from there. If you're new to Docker, check out our What is Docker post.

7. Publish It Securely

Let's publish Open WebUI on your domain, over HTTPS, opening only the ports we need. First create and enable /etc/nginx/sites-available/ai:

server {
    listen 80;
    server_name ai.example.com;

    location / {
        proxy_pass http://127.0.0.1:8080;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_read_timeout 300s;
    }
}

Then get a free SSL certificate and set up the firewall:

sudo apt install -y nginx certbot python3-certbot-nginx
sudo certbot --nginx -d ai.example.com

sudo ufw allow OpenSSH
sudo ufw allow 'Nginx Full'
sudo ufw enable              # we don't open ports 11434 and 8080

With this setup only ports 22, 80 and 443 are open from outside; Ollama (11434) and Open WebUI (8080) are reachable only through Nginx over an encrypted connection.

If you need to expose the Ollama API to an app on another server, at least add username/password protection:

sudo apt install -y apache2-utils
sudo htpasswd -c /etc/nginx/.htpasswd apiuser

# inside the Nginx server block:
location /ollama/ {
    auth_basic "Ollama API";
    auth_basic_user_file /etc/nginx/.htpasswd;
    proxy_pass http://127.0.0.1:11434/;
}

How Fast Is It on a CPU?

The numbers vary with the CPU, core count, context length and concurrent requests. A few performance tips:

• Pick the smallest model that does the job; for classification and summarization, 3B models are often enough.
• Don't overdo num_ctx; long context hurts both RAM and speed.
• Keep the model in memory; use OLLAMA_KEEP_ALIVE to avoid reloading on every request.
• Avoid swapping; if the model doesn't fit in RAM and spills to disk, speed drops dramaticaly.

Common Problems

Which VECDC Plan?

VPS-2 is a good start for 3B models and embeddings, VPS-3 (16 GB) or VPS-4 (24 GB) for comfortable work with 7B–8B models, and VPS-5 (32 GB) if you'll run several models together. VECDC virtual servers come with NVMe storage and dedicated CPU from $6/month and are ready in about 5 minutes with automated setup. For Open WebUI, Developer VDS, delivered with Docker installed, makes life easier. If you want to integrate your local model with existing apps like your CRM or help desk, our AI service can help.

Frequently Asked Questions

In Short

Running a local LLM with Ollama on a VPS takes just a few commands: install it, pull a model, configure the settings, build your own assistant with a Modelfile, add a UI with Open WebUI and publish it securely with Nginx + SSL. Pick the right model size and even a server without a GPU gives you a useful assistant that keeps your data in-house. If you get stuck during setup, drop us a message, were happy to help.