Running LLMs on Jetson Orin, llama.cpp, Ollama, and jetson-containers

Jetson Orin’s unified memory architecture makes it unusually capable for edge LLM inference: there is no separate GPU VRAM limit, so models that fit in total system RAM can run fully GPU-accelerated. The AGX Orin 64GB can run a 70B model at Q4 quantization. The challenge is not whether a model fits, but setting up the correct build flags and quantization for the available RAM.

Key Insights

• Unified memory eliminates the VRAM bottleneck: on Jetson, system RAM and GPU memory are the same physical pool; a 64GB AGX Orin can GPU-offload a 40GB model
• Q4_K_M is the optimal quantization for most Jetson deployments: balances inference quality with memory footprint; Q8 is too large for most SKUs, Q2 degrades quality significantly
• jetson-containers is the fastest setup path: pre-built containers for every JetPack/L4T version; no CUDA compilation required
• Set JETSON_CLOCKS=1 before running inference: it enables max CPU/GPU frequency and significantly improves tokens/sec
• Context length multiplies KV cache memory: 4096 context with Llama 3.1 8B adds ~1.5GB KV cache; reduce to 2048 for tight memory budgets

Model selection by Jetson SKU

Jetson SKU	RAM	Recommended models	Max model size at Q4
AGX Orin 64GB	64GB	Llama 3.1 70B, Qwen 2.5 32B	~40GB
AGX Orin 32GB	32GB	Llama 3.1 8B FP16, Qwen 2.5 14B Q4	~18GB
Orin NX 16GB	16GB	Llama 3.1 8B Q4-Q6, Mistral 7B Q4	~8GB
Orin NX 8GB	8GB	Llama 3.2 3B, Phi-3 mini Q4	~4GB
Orin Nano 8GB	8GB	Phi-3 mini Q4, Qwen 2.5 1.5B	~4GB
Orin Nano 4GB	4GB	Phi-3 mini Q4 (tight), Qwen 2.5 1.5B	~2GB

Memory formula: required_RAM = model_file_GB + (context_length * num_layers * 2 * head_size / 1e9) + 1GB

Setup with jetson-containers (recommended)

# Clone the jetson-containers repo
git clone https://github.com/dusty-nv/jetson-containers
cd jetson-containers

# Install prerequisites
bash install.sh

# Run Ollama container (auto-selects correct image for your JetPack version)
./run.sh $(./autotag ollama)

# Inside the container, pull and run a model:
ollama pull llama3.1:8b
ollama run llama3.1:8b

The autotag script queries your L4T version and selects the matching container image. No CUDA compilation, no dependency management.

Which container image matches your JetPack?

If you pin images explicitly instead of using autotag, the tag encodes everything you need: L4T release (r36.4), CUDA version (cu128), and Ubuntu base (24.04). These are the images we deploy on JetPack 6 (L4T r36.4) systems:

Workload	Image	Size	Notes
llama.cpp	dustynv/llama_cpp:b5283-r36.4-cu128-24.04	3.3GB	Python bindings included at /opt/llama-cpp-python
Ollama	dustynv/ollama:0.6.8-r36.4-cu126-22.04	5.1GB	Model cache at $OLLAMA_MODELS (default data/models/ollama)
vLLM	dustynv/vllm:0.8.6-r36.4-cu128-24.04	5.7GB	Requires L4T >= 34.1.0
PyTorch	dustynv/pytorch:2.7-r36.4.0	6.3GB	-all variant adds TorchVision, TensorRT, Transformers

Still on JetPack 5? Use the r35 image line (e.g. dustynv/pytorch:2.1-r35.4.1), r36 images will not run on an L4T r35 base. Bind-mount the Ollama model cache to persist pulled models across container restarts; a 70B-class model cache re-download over a shop network is an afternoon lost.

Building llama.cpp from source (CUDA-accelerated)

For more control over build flags:

# Prerequisites
apt install cmake build-essential libcurl4-openssl-dev

# Clone
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp

# Build with CUDA support
cmake -B build \
  -DGGML_CUDA=ON \
  -DCMAKE_CUDA_ARCHITECTURES="87"  \  # Orin = sm_87
  -DGGML_CUDA_F16=ON \
  -DGGML_CUDA_DMMV_X=64 \
  -DGGML_CUDA_MMV_Y=2
cmake --build build --config Release -j $(nproc)

CUDA architecture by Jetson module:

• Orin (all SKUs): sm_87
• AGX Xavier: sm_72
• Orin Nano: sm_87

Running inference with llama.cpp

# Maximize clocks first
sudo jetson_clocks

# Download a Q4_K_M model
# Example: Llama 3.1 8B Instruct Q4_K_M from Hugging Face
wget https://huggingface.co/bartowski/Meta-Llama-3.1-8B-Instruct-GGUF/resolve/main/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf

# Run inference, offload all layers to GPU
./build/bin/llama-cli \
  -m Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf \
  -ngl 99 \            # offload all 32 layers to GPU
  -c 4096 \            # context length
  -n 200 \             # max tokens to generate
  --temp 0.7 \
  -p "Explain MIPI CSI-2 in one paragraph:"

# Expected performance on Orin NX 16GB:
# Prompt eval: ~15 tokens/sec
# Generation: ~18 tokens/sec

Troubleshooting ggml_cuda errors on Jetson

The most common failure mode is llama.cpp silently running CPU-only. Verify GPU inference from the startup log, you must see:

ggml_cuda_init: found 1 CUDA devices:
  Device 0: Orin, compute capability 8.7

If that line is missing, or you hit errors, work down this list:

• No ggml_cuda_init line at all: the binary was built without CUDA. Rebuild with -DGGML_CUDA=ON.
• CMake never finds the CUDA compiler: JetPack installs the toolkit at /usr/local/cuda, but nvcc is not on PATH by default: export PATH=/usr/local/cuda/bin:$PATH and re-run CMake from a clean build directory.
• no kernel image is available for execution on the device: the binary was compiled for the wrong GPU architecture. Orin (all SKUs) is compute capability 8.7: rebuild with -DCMAKE_CUDA_ARCHITECTURES="87".
• There is no nvidia-smi on Jetson: it only exists on discrete-GPU systems. Use tegrastats or jtop (jetson-stats) to watch GPU load and memory during inference.
• Process dies mid-inference with no CUDA error: unified-memory OOM. On Jetson the allocation often succeeds, then the Linux OOM killer terminates the process once the KV cache grows. Check dmesg | grep -i oom, then drop one quantization tier or reduce -c context length.

Ollama setup (without jetson-containers)

# Install Ollama for aarch64
curl -fsSL https://ollama.com/install.sh | sh

# Start server
ollama serve &

# Pull a model
ollama pull qwen2.5:7b

# Run
ollama run qwen2.5:7b

# API endpoint
curl http://localhost:11434/api/generate -d '{
  "model": "qwen2.5:7b",
  "prompt": "What is PREEMPT_RT?",
  "stream": false
}'

vLLM on Jetson: when to use it instead

llama.cpp and Ollama are the right tools for single-user inference: GGUF quantization gives the widest model-size flexibility and the smallest memory footprint. But they process requests one at a time. If the Jetson is serving multiple concurrent clients, several UIs, agents, or services hitting the same model, vLLM is the better fit: it exposes an OpenAI-compatible API and uses continuous batching, so concurrent requests share the GPU instead of queuing.

vLLM runs on Jetson via dustynv/vllm images (0.7.4–0.9.3, L4T >= 34.1.0 required). The tradeoff: vLLM wants more headroom than llama.cpp for the same model, so on 8GB SKUs stay with llama.cpp; on AGX Orin serving multiple clients, vLLM earns its footprint.

Performance benchmarks

Benchmarked on JetPack 6.2, jetson_clocks enabled, context=2048:

Model	Jetson SKU	Quantization	Prompt (tok/s)	Gen (tok/s)
Llama 3.1 8B	AGX Orin 64GB	Q4_K_M	48	52
Llama 3.1 8B	Orin NX 16GB	Q4_K_M	15	18
Llama 3.1 8B	Orin NX 8GB	Q4_K_M	10	11
Phi-3 mini 3.8B	Orin Nano 8GB	Q4_K_M	22	28
Qwen 2.5 7B	Orin NX 16GB	Q4_K_M	18	20

Memory management tips

# Check available memory before loading model
free -h
tegrastats | grep RAM

# Release GPU memory between runs (kills all CUDA contexts)
sudo fuser -k /dev/nvidia*

# For concurrent LLM + CV pipeline:
# Reserve memory by setting llama.cpp max context shorter
# -c 1024 instead of 4096 saves ~750MB

For GPU-accelerated TensorRT inference on Jetson (for vision models), see TensorRT vs DLA on Jetson Orin. For running containerized workloads including AI models, see Docker containers on Jetson Orin.

FAQ

What LLMs can run on Jetson Orin?

AGX Orin 64GB: Llama 3.1 70B at Q4. Orin NX 16GB: Llama 3.1 8B at Q4-Q6. Orin Nano 8GB: Phi-3 mini, Qwen 2.5 3B. The rule: model_file_size + ~1.5GB KV cache must fit in available RAM.

What is the fastest way to run an LLM on Jetson Orin?

Use jetson-containers, pre-built Docker images with llama.cpp and Ollama for each JetPack version. Run ./run.sh $(./autotag ollama) to get a running Ollama instance.

How much RAM does an LLM use on Jetson Orin?

Approximately model file size plus KV cache. A Q4_K_M Llama 3.1 8B model uses ~6GB total with 4096-token context.

Does llama.cpp use the GPU on Jetson Orin?

Yes. Build with GGML_CUDA=ON and set -ngl 99 to offload all layers to the GPU using CUDA. On Jetson’s unified memory, GPU offload uses the same physical RAM but is 2–4x faster than CPU-only inference.

Why is llama.cpp not using the GPU on Jetson?

The startup log must show ggml_cuda_init: found 1 CUDA devices. If missing, the build is CPU-only: rebuild with -DGGML_CUDA=ON, put nvcc on PATH, and set -DCMAKE_CUDA_ARCHITECTURES="87" for Orin.

Can you run vLLM on Jetson Orin?

Yes, via dustynv/vllm containers (L4T >= 34.1.0). Use it when serving multiple concurrent clients through an OpenAI-compatible API; stay with llama.cpp/Ollama for single-user setups.

Is there nvidia-smi on Jetson?

No. Use tegrastats or jtop (jetson-stats) to monitor GPU load and memory during inference.

Can swap extend RAM for larger models?

It lets a model load, but inference on swapped weights is unusably slow, every token touches the full weight set. Drop a quantization tier instead.

---