How to Run Local AI Models Privately with Ollama on Linux

This guide shows how to run language models on your local Linux machine. You will keep your data private and avoid cloud service fees.

Prerequisites

Before you start, make sure that your system meets these requirements:

• A 64-bit Linux distribution (Ubuntu, Debian, or Fedora).

• At least 8 GB of RAM for small models.

• At least 16 GB of storage space for model weights.

Step 1: Install Ollama

Open your terminal. Run the official installation script with this command:

curl -fsSL https://ollama.com/install.sh | sh

The script downloads the binary files and installs the service.

After the installation finishes, verify that the Ollama service is active:

systemctl status ollama

If the service is not running, start it manually:

sudo systemctl start ollama

Step 2: Download and Run a Local Model

Ollama provides many open-source models. The Llama 3.2 model offers good performance for general tasks.

To download and run Llama 3.2, enter this command:

ollama run llama3.2

The terminal downloads the model files. When the download completes, a prompt appears.

Type your query directly into the terminal prompt. Press Enter to generate a response.

To exit the prompt, type /bye and press Enter.

Step 3: Test the API Endpoint

Ollama runs a local REST API on port 11434. You can send requests to this endpoint from local applications.

To test the API, run this curl command:

curl http://localhost:11434/api/generate -d '{
  "model": "llama3.2",
  "prompt": "Why is open source software important?"
}'

The API returns a JSON response stream with the generated text.

Step 4: Configure Ollama to Start on Boot

To run Ollama automatically when your server reboots, enable the systemd service:

sudo systemctl enable ollama

The service will now start automatically at system boot.

Conclusion

You now have a private AI model running locally on Linux. All data stays on your machine.