When people think of AI, they usually picture huge data centers and expensive graphics cards. But there's a smaller side to it too: thanks to open-source models, running an AI model on your own server, even on a virtual server, is now possible. So how does AI run on a virtual server, which model fits which server, and does it actually make sense? In this post we cover all of it in plain English, with tables.
The Basics: What Happens When an AI Model Runs?
Chatty AIs like ChatGPT are really large language models (LLMs). A model is a big file made of billions of numbers (parameters). When you ask a question, the model does math with those numbers and generates the answer word by word (token by token). This is called inference.
Training a model takes truly massive hardware, but just running an already-trained model is a much more modest job. That's what we'll do on a virtual server: download a ready open-source model and run it.
Two resources decide how well it runs:
- Memory (RAM or GPU memory): The whole model has to fit in memory. If it doesn't, it either won't run at all or gets extremely slow.
- Compute (CPU or GPU): Decides how fast the answer is generated.
RAM Needs by Model Size
The good news: models are usually shrunk with quantization. A 4-bit (Q4) quantized model takes roughly 4 times less space than the original, and the quality stays good enough for most tasks. The values below are approximate, for 4-bit quantized models:
| Model size | Approx. RAM needed (Q4) | Good for |
|---|---|---|
| 1–3 billion parameters (1B–3B) | ~1–3 GB | Simple chat, classification, summarization, text cleanup |
| 7–8B | ~5–6 GB | General assistant, support bot, RAG |
| 13–14B | ~9–10 GB | More consistent answers, coding help |
| 30–34B | ~20–22 GB | More complex tasks |
| 70B | ~40–45 GB | Quality close to large models |
On top of the model itself, leave a few GB for the operating system, your app and the conversation history (context).
CPU or GPU?
This is the most common question. Short answer: GPUs are much faster, but small models run fine on CPU too.
| CPU (standard virtual server) | GPU server | |
|---|---|---|
| Speed (7–8B model) | A few words per second (~3–10 tokens/s) | Tens to hundreds of tokens/s |
| Suitable model size | 1B–14B (quantized) | All, including large models |
| Many users at once | Weak | Good |
| Cost | Low, from a few dollars a month | High |
| Good for | Testing, internal tools, low-traffic bots, embeddings | Live products, high traffic, large models |
The numbers vary with the CPU, model and settings, but this is the general picture. For an internal assistant used by one person, CPU can be enough; for a product serving thousands of users, you need a GPU.
What Can You Do With AI on a Virtual Server?
- Internal assistant: A chat assistant that runs without your data leaving the server.
- RAG (Q&A over documents): A bot that answers based on your company documents.
- Embeddings and search: Turning text into vectors for meaning-based search. Small models handle this fast, even on CPU.
- Automation: Background jobs like classifying emails, prioritizing support tickets or summarizing text.
- Development and testing: Trying out an AI integration before taking your app live.
Step by Step: Setting Up AI on a Virtual Server With Ollama
The easiest way to run a model on a server is Ollama. It handles downloading, running and serving models with a single tool. On a Linux virtual server, setup looks like this:
# 1) Install Ollama
curl -fsSL https://ollama.com/install.sh | sh
# 2) Download and run a small model
ollama run llama3.2:3b
# 3) See installed models
ollama listOllama also exposes an API in the background, so you can send requests to the model from your own app:
curl http://localhost:11434/api/generate -d '{
"model": "llama3.2:3b",
"prompt": "What is a virtual server? Explain in one sentence.",
"stream": false
}'If you like working with Docker, Ollama has an official Docker image too; pair it with an interface like Open WebUI and you get a ChatGPT-like screen in your browser. Developer VDS, delivered with Docker installed, is a quick starting point for this.
Which VECDC Package for Which Model?
VECDC virtual servers are CPU-based, without GPUs. The mapping below is a rough guide for 4-bit quantized models and a single user / low traffic:
| Package | Resources | Models that run comfortably | Suggested use |
|---|---|---|---|
| VPS-1 | 2 Core, 8 GB RAM | 1B–3B | Embeddings, classification, testing |
| VPS-2 / VPS-3 | 4 Core, 12–16 GB RAM | 3B–8B | Internal assistant, simple RAG, automation |
| VPS-4 | 6 Core, 24 GB RAM | 7B–14B | Better answers, small team use |
| VPS-5 | 8 Core, 32 GB RAM | 7B–14B (comfortably), larger ones slowly | Multiple models, RAG + embeddings together |
Every package has NVMe storage, which means model files (several GB) load from disk into memory quickly. If you're curious why NVMe matters, see our What is NVMe post. Our virtual server packages start at $6/month and are ready in about 5 minutes with automated setup.
Use an API or Run It on Your Own Server?
There are clear differences between ready-made AI APIs (services from the big providers) and running a model on your own server:
| Ready-made AI API | Model on your own server | |
|---|---|---|
| Setup | None, grab a key and go | Server and model setup needed |
| Model quality | Access to the strongest models | Limited to open-source models |
| Cost | Pay as you go (per token) | Fixed monthly server fee |
| Data privacy | Data goes to the provider | Data never leaves your server |
| Speed | Usually fast | Depends on hardware (slow on CPU) |
| Good for | Best quality, variable usage | Privacy-sensitive, steady and predictable usage |
Often the best answer is a mix: sensitive or high-volume simple jobs on your own server, complex jobs through an API.
A Few Performance Tips
- Pick the right size: Use the smallest model that's good enough for the job. A 3B model handles many classification tasks as well as a 70B one.
- Use quantized models: Q4 or Q5 models are a good starting point for the quality/speed balance.
- Don't overdo context length: Very long context hurts both RAM and speed.
- Avoid swapping: If the model doesn't fit in RAM and spills onto disk, speed drops dramaticaly; size your RAM accordingly.
Frequently Asked Questions
Can an AI model run without a GPU?
Yes. Quantized (e.g. 4-bit) open-source models between 1B and 14B run on CPU-based servers. They're slower than on a GPU, but fine for single-user assistants, embeddings, classification and automation.
How much RAM does an AI model need?
For 4-bit quantized models, roughly: 1–3 GB for 1–3B, 5–6 GB for 7–8B, 9–10 GB for 13–14B and around 40 GB for 70B. Leave a few extra GB for the operating system and your app.
What is Ollama?
Ollama is a free tool that lets you download and run open-source AI models with a single command. It also provides a local API so you can use the model from your own application.
Does running AI on my own server protect my data?
When the model runs on your own server, your questions and documents aren't sent to a third-party provider. The server and API port still need to be configured securely.
In Short
AI models run on virtual servers too; the trick is choosing the model size to match the server's RAM and your expectations. Small and mid-size open-source models are perfectly fine on a CPU-based virtual server for internal assistants, RAG and automation. Large models and high traffic need a GPU. If you'd like to talk about what fits your project, check out our AI service or drop us a message and lets take a look together.