CUDA on Fedora: Set Up a Local AI Workstation with Ollama or llama.cpp in an Afternoon

By the end of this post your Fedora machine will run language models on its NVIDIA GPU, first through Ollama (the five-minute path) and then, if you want it, through a llama.cpp you compiled yourself with the CUDA toolkit. The cheapest hardware this works on: any PC with a GeForce card from 2017 or later, including a used RTX 3060 12 GB.

Honesty first. We verified every command below against the official documents on 2026-09-09, but we did not execute them on a Fedora machine: our own workstation runs Ubuntu 24.04 on ARM with CUDA 13. Where we show expected output, we say whose output it is.

What you need

• Fedora Workstation or a Fedora Atomic desktop (Silverblue, Kinoite) with an NVIDIA GPU. The driver instructions come from RPM Fusion’s NVIDIA how-to; the toolkit instructions from llama.cpp’s own Fedora guide and RPM Fusion’s CUDA how-to.
• Secure Boot either disabled, or your own signing key enrolled. RPM Fusion is explicit: with Secure Boot on “you must sign the nvidia kmod”.
• About 10 GB of disk for the toolkit and one model. 15 minutes for path A, another 30 to 45 for path B.
• No account, no API key. Everything is free software plus NVIDIA’s redistributable toolkit.

Step 1: Install the NVIDIA driver from RPM Fusion

Enable the RPM Fusion nonfree repository (their configuration page has the one-line dnf install for it), then:

sudo dnf update -y
sudo dnf install akmod-nvidia
sudo dnf install xorg-x11-drv-nvidia-cuda

akmod-nvidia builds the kernel module on your machine; xorg-x11-drv-nvidia-cuda adds the CUDA user-space libraries that Ollama and llama.cpp need. Then wait. RPM Fusion says the build can take up to five minutes, and gives the check:

modinfo -F version nvidia

A version string such as 595.58.03 means the module is ready. An error means it is still building or failed; reboot once before you debug. Cards older than 2017 need a legacy package instead (akmod-nvidia-580xx for GTX 800/900/10 series, akmod-nvidia-470xx for the 600/700 series).

One rule to remember from the RPM Fusion CUDA page: do not mix NVIDIA’s own nvidia-driver packages with RPM Fusion’s driver. Use RPM Fusion for the driver and NVIDIA’s repository only for the toolkit.

Step 2 (path A): Ollama on the GPU in five minutes

curl -fsSL https://ollama.com/install.sh | sh
ollama -v
ollama run llama3.1:8b "Summarise this in one sentence: Fedora installs the driver, Ollama does the rest."

The official Linux install page documents the script, ollama -v for the version, and journalctl -e -u ollama for logs. The line you want in the log mentions CUDA and your card:

journalctl -u ollama --no-pager | grep -i -E "cuda|nvidia" | tail -5

If Ollama reports it is running on CPU, the usual cause is that step 1 was not finished when the service started: sudo systemctl restart ollama after the kernel module is loaded. To update later, re-run the install script; to pin a version, prefix it with OLLAMA_VERSION=X.X.X.

For most small teams this is the finish line. Ollama exposes an API on port 11434 that Open WebUI, n8n and every OpenAI-compatible SDK can use.

Step 3 (path B): Install the CUDA toolkit

You only need this to compile llama.cpp yourself, or for PyTorch work. Two routes.

On Fedora Workstation (dnf5, Fedora 43 and newer), from the RPM Fusion CUDA page:

sudo dnf config-manager addrepo \
  --from-repofile=https://developer.download.nvidia.com/compute/cuda/repos/fedora43/$(uname -m)/cuda-fedora43.repo
sudo dnf -y install cuda-toolkit xorg-x11-drv-nvidia-cuda

If NVIDIA has no repository yet for your Fedora release, the page’s advice is to use the newest one they publish. If nvcc complains that your system gcc is too new, RPM Fusion points to the cuda-gcc and cuda-gcc-c++ packages from COPR and export HOST_COMPILER=cuda-g++.

On an Atomic desktop, the llama.cpp guide keeps the host clean and does it all in a toolbox:

toolbox create --image registry.fedoraproject.org/fedora-toolbox:41 --container fedora-toolbox-41-cuda
toolbox enter --container fedora-toolbox-41-cuda
sudo dnf distro-sync
sudo dnf install @c-development @development-tools cmake
sudo dnf config-manager addrepo \
  --from-repofile=https://developer.download.nvidia.com/compute/cuda/repos/fedora41/x86_64/cuda-fedora41.repo
ls -la /usr/lib64/libcuda.so.1

If that last ls shows the file, the host already supplies the driver library and you should only register the driver packages in the RPM database (sudo dnf download --destdir=/tmp/nvidia-driver-libs --resolve --arch x86_64 nvidia-driver-cuda nvidia-driver-libs nvidia-driver-cuda-libs nvidia-persistenced followed by sudo rpm --install --verbose --hash --justdb /tmp/nvidia-driver-libs/*). If it is missing, install those same four packages normally. Then:

sudo dnf install cuda
sudo sh -c 'echo "export PATH=\$PATH:/usr/local/cuda/bin" >> /etc/profile.d/cuda.sh'
sudo chmod +x /etc/profile.d/cuda.sh
source /etc/profile.d/cuda.sh
nvcc --version

The guide expects CUDA 12.8 or newer. On our Ubuntu box the same command prints Cuda compilation tools, release 13.0, V13.0.88, so you know the shape of a good answer. If the GPU disappears inside the toolbox after a driver update, the guide’s fix is podman container restart --all.

Step 4: Build llama.cpp with CUDA and prove the GPU is used

From the llama.cpp build guide:

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release
./build/bin/llama-cli -m models/Llama-3.1-8B-Instruct-Q4_K_M.gguf \
  -p "Hi, how are you?" -ngl 99

If nvcc cannot detect your card’s compute capability, pass it explicitly, for example -DCMAKE_CUDA_ARCHITECTURES="86;89" (8.6 is an RTX 3080 Ti or 3060, 8.9 an RTX 4090). The startup banner of llama-cli lists the CUDA device and how many layers were offloaded; -ngl 99 means all of them.

What to expect and what we measured

We have not benchmarked a Fedora box. For the card most readers will start with, third parties report the following, and on Fedora the driver is the same binary, so the numbers carry over.

Hardware	Model	Speed	Source
RTX 3060 12 GB (used, ~USD 250-278)	Llama 3.1 8B Q4_K_M	52 to 65 tok/s (tyolab 64.5, LocalScore 52.2); another guide reports 42 to 48	SpecPicks, CraftRigs, GPUDojo prices
Our NVIDIA GB10 workstation (Ubuntu 24.04, CUDA 13.0, driver 580.173.02)	Llama 3.1 8B Q4_K_M	37.8 tok/s, measured 2026-09-08	our own ollama run; a shared machine, not a recommendation

Two verification commands you should run on your own machine and paste into your notes: nvidia-smi (driver version, CUDA version, memory in use while a model is loaded) and ollama ps, which shows whether the model sits 100% GPU.

Where this fits and honest limits

A Fedora workstation with a 12 GB card runs 7B and 8B models fast enough for one team’s daily use: drafting, summarising, classification, code help. It replaces a per-token bill for that routine share of work and keeps the documents on the machine.

Limits: Fedora moves fast, and a kernel update can land before akmod-nvidia has a build for it; if the desktop falls back to the open driver after an update, wait for the akmod rebuild or boot the previous kernel. Secure Boot adds a signing step that this post does not cover. And a 12 GB card stops at about 14B parameters in Q4; for 70B-class models you need 40-plus GB across one or more GPUs.

Next steps

• Choose and install the rest of the stack: How to deploy AI locally in your business.
• Check whether the card pays for itself: Cloud vs local break-even in 15 minutes.
• Fit bigger models into 12 GB: Quantization explained.

Work with us

We set up local inference on the hardware clients already own before recommending anything new, and we leave a written runbook behind. If you want that done on your Fedora machines, get in touch or read how our consulting works.