{"id":2848,"date":"2026-08-30T18:42:06","date_gmt":"2026-08-30T18:42:06","guid":{"rendered":"https:\/\/dmarketertayeeb.com\/blog\/run-local-llms-low-vram-llama-cpp-pytllm-swap-moe\/"},"modified":"2026-08-30T18:45:12","modified_gmt":"2026-08-30T18:45:12","slug":"run-local-llms-low-vram-llama-cpp-pytllm-swap-moe","status":"publish","type":"post","link":"https:\/\/dmarketertayeeb.com\/blog\/run-local-llms-low-vram-llama-cpp-pytllm-swap-moe\/","title":{"rendered":"How to Run Local LLMs on Low VRAM: llama.cpp, PyTLLM and Swap-MoE"},"content":{"rendered":"\n

Short answer:<\/strong> If you want to run an open model on a machine with limited VRAM or RAM, start with an official llama.cpp<\/a> build and a compatible GGUF quantization. Let the runtime fit layers automatically, keep the context and server slot count conservative, and measure the output on your hardware. PyTLLM (also branded TLLM) and Swap-MoE are interesting experimental approaches for models that exceed ordinary memory limits, but their headline model-size and speed claims are not independently validated. They should be treated as experiments, not as guaranteed replacements for a smaller model.<\/p>\n\n\n\n

This guide is a practical snapshot checked on 30 August 2026. It covers desktop and small-server inference, not model training. The earlier DMT overview of mobile and on-device AI<\/a> explains why local execution matters; this article focuses on the very different problem of fitting open models into constrained desktop memory.<\/p>\n\n\n\n

Choose the memory strategy before you choose a model<\/h2>\n\n\n\n

\u201cA model has 70 billion parameters\u201d is not a deployment plan. The important questions are where the weights live, how much of each layer must be resident, how much memory the key-value (KV) cache needs, and how much capacity remains for activations and the operating system.<\/p>\n\n\n\n

\n\n\n
Approach<\/th>What it changes<\/th>Best fit<\/th>Evidence and trade-off<\/th><\/tr>\n<\/thead>\n
llama.cpp + GGUF<\/td>Quantizes weights and can split work between CPU and GPU.<\/td>Most constrained desktops; CPU-only or mixed CPU\/GPU machines.<\/td>Strongest documented baseline. Quality, speed and supported operations vary by model, quantization and backend.<\/td><\/tr>\n
PyTLLM\/TLLM<\/td>Streams transformer layers and caches them across VRAM, pinned RAM and disk.<\/td>Python\/CUDA users willing to test very large checkpoints.<\/td>Promising repository implementation, but model\/VRAM figures are project claims with no independent comparative benchmark found.<\/td><\/tr>\n
Swap-MoE<\/td>Demand-pages routed MoE experts from an SSD and adds optional expert caching\/prefetch.<\/td>MoE experiments where the model file is larger than physical RAM.<\/td>Experimental patch tied to a specific llama.cpp commit. SSD latency, quality risk and maintenance burden are real.<\/td><\/tr>\n
MLX-LM<\/td>Uses Apple Silicon unified memory and MLX quantized models.<\/td>Macs with Apple Silicon.<\/td>Strong Apple-specific path. It is not a Windows\/NVIDIA replacement, and models larger than RAM can be slow.<\/td><\/tr>\n
Transformers + bitsandbytes<\/td>Loads 8-bit or 4-bit linear layers in a normal PyTorch stack.<\/td>NVIDIA\/compatible GPU users who can fit a quantized checkpoint.<\/td>Established quantization path, but it does not provide SSD expert paging and still needs runtime headroom.<\/td><\/tr>\n
vLLM CPU\/offload options<\/td>Moves selected weights or KV data to host memory for serving.<\/td>Higher-throughput servers with a fast CPU-GPU interconnect.<\/td>Useful server feature, not a magic fix for a 4 GB GPU or a slow laptop.<\/td><\/tr>\n<\/tbody>\n<\/table><\/figure>\n\n\n\n

Start with a memory budget, not a parameter-count headline<\/h2>\n\n\n\n

A useful planning estimate is peak memory = model weights + KV cache + activations\/workspace + runtime and OS headroom<\/code>. It is an estimate, not a sizing guarantee: tensor layouts, multimodal components, batching, context length, backend workspaces and allocator fragmentation all matter.<\/p>\n\n\n\n

Quantization changes the weight term. The official llama.cpp quantization guide gives this Llama 3.1 example:<\/p>\n\n\n\n

\n\n\n
Model<\/th>Original size<\/th>Q4_K_M size<\/th><\/tr>\n<\/thead>\n
8B<\/td>32.1 GB<\/td>4.9 GB<\/td><\/tr>\n
70B<\/td>280.9 GB<\/td>43.1 GB<\/td><\/tr>\n
405B<\/td>1,625.1 GB<\/td>249.1 GB<\/td><\/tr>\n<\/tbody>\n<\/table><\/figure>\n\n\n\n

Those are file-size examples, not the amount of free VRAM required and not a promise that an 8B Q4 model will fit in exactly 4.9 GB. A 70B Q4 file is still about 43.1 GB before the KV cache and runtime overhead. Conversely, a sparse MoE model may have a large total file but activate only part of its experts per token. That is why layer or expert streaming can change the capacity problem, while still leaving a latency and bandwidth problem.<\/p>\n\n\n\n

Do not forget the conversion machine. The same official guide says the current quantization process fully loads the model and needs enough RAM and disk for the original and intermediate files. A small target device does not imply that the quantization step can run on that same device.<\/p>\n\n\n\n

What changed in llama.cpp in August 2026<\/h2>\n\n\n\n

llama.cpp now has two release rhythms. Stable version v0.2.0<\/strong> was published on 21 August 2026 and its release notes describe the new vX.Y.Z<\/code> line as the slower-cadence choice for downstream distributors and casual users. The b[NUM]<\/code> tags are nightly\/development builds for users who need newer functionality and can absorb more change. For a reproducible how-to, pin a stable tag or a known commit; use a nightly only when a specific fix matters.<\/p>\n\n\n\n