Running an LLM locally does not require a complicated AI stack.

Ollama provides a simple way to download and run open models on Linux. Once installed, you can run models directly from the terminal and interact with them without sending your prompts to a third-party API.

This guide shows how to install Ollama on Ubuntu, download a model, run it, and use its local API.

What You Need

You need:

  • Ubuntu Linux
  • A user account with sudo access
  • Internet access for installing Ollama and downloading models
  • Enough RAM and disk space for the model you choose

The hardware requirement depends heavily on the model. Smaller models can run on systems with limited resources, while larger models require considerably more RAM or GPU memory.

Ollama supports both x86-64 and ARM64 Linux systems. (Ollama)

1. Install Ollama on Ubuntu

The official Linux installation method is:

curl -fsSL https://ollama.com/install.sh | sh

The installer downloads and installs Ollama on the system. (Ollama)

After installation, check that it is available:

ollama -v

You can also run:

ollama

This opens Ollama’s interactive terminal interface.

Ollama Linux Installation Documentation

2. Download an LLM

Ollama does not include a model with the basic installation. You choose which model you want to run.

For example:

ollama pull llama3.2

This downloads the selected model to your machine.

You can see the models already installed with:

ollama list

Ollama provides models from several model families, including Llama, Qwen, Gemma, DeepSeek, Mistral and others. (Ollama)

Browse the Ollama Model Library

3. Run the Model

Once the model has been downloaded, run it with:

ollama run llama3.2

You can now type a prompt:

Explain what SSH is in simple terms.

Ollama will generate the response locally.

You can exit the conversation with:

/bye

The exact command depends on the model and tag you choose.

For example:

ollama run qwen3

or:

ollama run gemma4

Check the Ollama model library for the available model names and variants.

4. Check Downloaded Models

To see your installed models:

ollama list

Example:

NAME          ID              SIZE
llama3.2      xxxxxxxx        ...
qwen3         xxxxxxxx        ...

To remove a model you no longer need:

ollama rm llama3.2

This is useful because model files can consume several gigabytes of disk space.

5. Check Whether Ollama Is Running

You can start Ollama manually with:

ollama serve

Then, from another terminal:

ollama -v

For a server installation, running Ollama as a systemd service is generally more convenient because it can start automatically when Ubuntu boots. The official documentation provides a systemd service configuration for this setup. (Ollama)

For example, after configuring the service:

sudo systemctl daemon-reload
sudo systemctl enable ollama
sudo systemctl start ollama

Check its status:

sudo systemctl status ollama

View the service logs with:

journalctl -u ollama

6. Use Ollama Through Its Local API

Ollama also exposes a local HTTP API.

By default, applications can communicate with Ollama through:

http://localhost:11434

For example:

curl http://localhost:11434/api/chat 
  -d '{
    "model": "llama3.2",
    "messages": [
      {
        "role": "user",
        "content": "Explain Linux permissions."
      }
    ]
  }'

This makes Ollama useful beyond the terminal. You can connect it to applications written in Python, JavaScript, or other languages.

The official documentation also provides Python and JavaScript libraries for working with Ollama. (Ollama)

Ollama API Documentation

7. Choosing a Model

There is no single model that is best for every machine.

A practical approach is to start with a smaller model and see how your hardware handles it.

For example:

ollama run llama3.2

You can then try another model:

ollama run qwen3

or:

ollama run gemma4

The Ollama library lists different parameter sizes for many model families. Smaller models generally require less memory, while larger models require significantly more resources. (Ollama)

Do not choose a model purely because it has more parameters. The model’s size, quantization, context length, workload, and available RAM/VRAM all affect whether it is practical to run.

8. CPU vs GPU

Ollama can run models without a dedicated GPU.

For example, Google’s Gemma documentation specifically describes using Ollama to run models on systems without a GPU, although performance depends on the hardware and model size. (Google AI for Developers)

A GPU can significantly improve inference performance when the required hardware and supported drivers are available.

For NVIDIA GPUs, Ollama’s Linux documentation provides CUDA setup information. AMD GPUs can use ROCm with supported hardware and drivers. (Ollama)

If you only have a CPU, start with a smaller model rather than downloading a very large one.

9. Where Are the Models Stored?

Ollama stores the downloaded model data locally.

This means model storage can become significant if you install many models.

You can check your available disk space with:

df -h

Before downloading a large model, make sure you have enough free storage.

You should also remember that the model’s displayed size is not the only resource involved. Running the model requires system memory as well.

10. Running a Custom Model

Ollama also supports creating custom models using a Modelfile.

A simple example:

FROM llama3.2

PARAMETER temperature 0.7

SYSTEM """
You are a helpful Linux assistant.
Give concise answers and include commands when appropriate.
"""

Save it as:

Modelfile

Then create the model:

ollama create linux-assistant -f Modelfile

Run it:

ollama run linux-assistant

Ollama’s Modelfile supports defining the base model and parameters such as temperature. (Ollama)

Ollama Modelfile Reference

11. Removing Ollama

If you decide that you no longer need Ollama, the official Linux documentation provides instructions for removing the service, binary, libraries, downloaded models, and Ollama user/group. (Ollama)

The important thing to remember is that deleting the Ollama program itself does not necessarily mean all downloaded model data has been removed.

Conclusion

Ollama makes running an open-weight LLM on Ubuntu relatively straightforward.

The basic workflow is:

# Install Ollama
curl -fsSL https://ollama.com/install.sh | sh

# Download a model
ollama pull llama3.2

# Run it
ollama run llama3.2

# List installed models
ollama list

From there, you can use Ollama from the terminal, expose its local API to applications, create custom models with Modelfile, and experiment with different open models.

The main limitation is hardware. Running an LLM locally does not eliminate the need for adequate RAM, storage, and compute. Start with a model that fits your machine instead of immediately choosing the largest available model.

Related DevStackIO Resources

Official Resources

Note: “open-weight” applies to the models, and model licenses vary. Check the license for the specific model before using it commercially. Ollama itself is released under the MIT License. (GitHub)

Related Tools

Try these companion tools on DevStackIO Tools:

  • unit converter — convert model sizes and VRAM measurements for LLM deployment.