How to Run Local LLMs on Low VRAM: llama.cpp, PyTLLM and Swap-MoE

Short answer: If you want to run an open model on a machine with limited VRAM or RAM, start with an official llama.cpp build and a compatible GGUF quantization. Let the runtime fit layers automatically, keep the context and server slot count conservative, and measure the output on your hardware. PyTLLM (also branded TLLM) and Swap-MoE are interesting experimental approaches for models that exceed ordinary memory limits, but their headline model-size and speed claims are not independently validated. They should be treated as experiments, not as guaranteed replacements for a smaller model.

This guide is a practical snapshot checked on 30 August 2026. It covers desktop and small-server inference, not model training. The earlier DMT overview of mobile and on-device AI explains why local execution matters; this article focuses on the very different problem of fitting open models into constrained desktop memory.

Choose the memory strategy before you choose a model

“A model has 70 billion parameters” is not a deployment plan. The important questions are where the weights live, how much of each layer must be resident, how much memory the key-value (KV) cache needs, and how much capacity remains for activations and the operating system.

Start with a memory budget, not a parameter-count headline

A useful planning estimate is peak memory = model weights + KV cache + activations/workspace + runtime and OS headroom. It is an estimate, not a sizing guarantee: tensor layouts, multimodal components, batching, context length, backend workspaces and allocator fragmentation all matter.

Quantization changes the weight term. The official llama.cpp quantization guide gives this Llama 3.1 example:

Those are file-size examples, not the amount of free VRAM required and not a promise that an 8B Q4 model will fit in exactly 4.9 GB. A 70B Q4 file is still about 43.1 GB before the KV cache and runtime overhead. Conversely, a sparse MoE model may have a large total file but activate only part of its experts per token. That is why layer or expert streaming can change the capacity problem, while still leaving a latency and bandwidth problem.

Do not forget the conversion machine. The same official guide says the current quantization process fully loads the model and needs enough RAM and disk for the original and intermediate files. A small target device does not imply that the quantization step can run on that same device.

What changed in llama.cpp in August 2026

llama.cpp now has two release rhythms. Stable version v0.2.0 was published on 21 August 2026 and its release notes describe the new vX.Y.Z line as the slower-cadence choice for downstream distributors and casual users. The b[NUM] tags are nightly/development builds for users who need newer functionality and can absorb more change. For a reproducible how-to, pin a stable tag or a known commit; use a nightly only when a specific fix matters.

• The observed latest nightly at the research cutoff was b10699, published on 30 August 2026. Its headline change improves how RPC asks backends about operations whose transient allocation can expand. That is a correctness and allocation detail, not a universal speed increase.
• b10684 improves SYCL --fit and --fit-target accounting for actual peak VRAM at a selected context. The release note’s test used an Intel Arc B70 and a particular Qwen quantization, so do not generalize its result.
• b10584 makes fit account for server streams and a second or draft model. This matters when speculative or draft execution would otherwise reserve the wrong context memory.
• b10594 avoids an unnecessary device-information loop that could create a CUDA context and allocate about 550 MB of VRAM when trace-level logging was not requested. It is a useful diagnostic footgun, not a rule for every GPU.
• b10677 fixes a Vulkan view-alias dependency issue that could silently alter greedy-decoding output for affected stateful models. Correctness checks still matter after a backend or build change.

The project also lists OpenVINO, Vulkan, SYCL, OpenCL, CUDA, HIP, Metal and other backends. The b10672 release updated OpenVINO to 2026.3.1 and added related model/NPU work, while the backend documentation still describes quantized validation as a work in progress. Pick a backend that your model and hardware actually support; the feature list is not a performance ranking.

The recommended baseline: llama.cpp plus GGUF

1. Pick a compatible quantized file

For a first test, use a model-specific GGUF that matches the model’s chat template and architecture. Q4_K_M is a sensible starting point because it is widely supported and balances size and quality reasonably well, but it is not automatically optimal. Test Q5 or Q6 when quality matters and memory permits; use more aggressive formats only when the capacity constraint leaves no practical alternative.

2. Build or install a known version

The official build guide provides a plain CPU build and backend-specific instructions. A simple source build is:

cmake -B build
cmake --build build --config Release

On Windows, use a Visual Studio developer PowerShell or another supported C++ toolchain. If you need CUDA, Metal, Vulkan, SYCL or OpenVINO, follow that backend’s build section instead of assuming the plain CPU binary contains it.

3. Let the runtime fit the first run

With a local file, begin conservatively:

llama-cli -m /models/model-Q4_K_M.gguf --n-gpu-layers auto --ctx-size 4096 --fit on

The current server documentation accepts auto or all for GPU layers and has --fit on enabled by default. If the GPU path is unstable, compare against a CPU-only smoke test:

llama-cli -m /models/model-Q4_K_M.gguf --n-gpu-layers 0 --ctx-size 4096

For a local OpenAI-compatible endpoint, start with one slot and a modest context:

llama-server -m /models/model-Q4_K_M.gguf --n-gpu-layers auto --ctx-size 4096 --parallel 1

Only after that works should you test a quantized KV cache such as --cache-type-k q8_0 --cache-type-v q8_0, and only where the selected backend supports it. A smaller KV representation can reduce memory pressure, but it is another quality and compatibility variable to measure.

4. Troubleshoot in the least destructive order

When fitting fails, lower --ctx-size first because KV memory grows with context. For a server, lower --parallel because each concurrent sequence needs cache capacity. Then reduce --n-gpu-layers as a last resort; the remaining layers run on the CPU and generation can become much slower. This order follows the current llama.cpp multi-GPU guidance and keeps the cause of each change visible.

Record the exact model filename, llama.cpp tag, backend, context, slot count and GPU-layer setting. “It ran” without those details is not a reproducible result.

PyTLLM/TLLM: layer streaming with a three-tier cache

PyTLLM presents a different answer to the same bottleneck. Its README says it constructs the Transformers model on the meta device, streams one layer to the GPU before it runs, evicts it afterward, and prefetches the next layer. Its cache can keep some layers in VRAM, spill others to pinned host RAM, and reread the remainder from disk.

The project’s model table claims examples such as Llama 3.x 70B in about 4 GB of VRAM, Llama 3.1 405B in about 8 GB, DeepSeek-V3 671B in about 12 GB and Kimi K3 2.8T under 4 GB. Those figures are clearly useful hypotheses for testing, but they are project claims. The repository was created on 24 August 2026, reports zero stars and no issues at this snapshot, and does not provide an independent comparative benchmark or a broad hardware matrix. Do not copy the table into a deployment promise.

A minimal experiment looks like this:

pip install pytllm

from pytllm import AutoModel

model = AutoModel.from_pretrained(
    "Qwen/Qwen3-32B",
    cache_layers=True,
    vram_cache_reserve_gb=2.0,
)

inputs = model.tokenizer(
    ["Summarize this document in five bullets."],
    return_tensors="pt",
    return_attention_mask=False,
    truncation=True,
    max_length=512,
    padding=False,
)
output = model.generate(
    inputs["input_ids"].cuda(),
    max_new_tokens=128,
    use_cache=True,
)
print(model.tokenizer.decode(output[0]))

Use the exact model documentation, not just the repository headline. The first run can split the original checkpoint into per-layer shards and temporarily need roughly twice the disk space. The README says compression disables the layered cache because compressed shards are decompressed on each load. Native dtype, Transformers version, CUDA/PyTorch compatibility, tokenizer behavior and disk capacity are all part of the test.

My recommendation is to use PyTLLM for an offline experiment when the normal Transformers path is already familiar and the disk/RAM budget is understood. Compare it with a smaller llama.cpp GGUF model on the same prompt. If the goal is a dependable team endpoint, wait for independent reproducibility, issue history and a documented upgrade path before making it the default.

Swap-MoE: when a sparse model is larger than physical RAM

Swap-MoE is not another model format or a drop-in binary. It is an MIT-licensed patch set against llama.cpp. Its core idea is to keep routed expert tensors memory-mapped on an SSD and let demand paging pull selected expert pages into RAM. Router tensors and shared experts stay resident because every token needs them.

The current README says the patch targets llama.cpp commit f5e85d43a from 28 August 2026. A simplified CPU-only build path is:

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
git checkout f5e85d43a
git apply /path/to/llama.cpp-expert-streaming.patch
cmake -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release

The patch adds several flags:

Start with the accuracy-preserving mechanism and measure:

llama-cli -m /models/DeepSeek-V4-Flash-0731-IQ1_M.gguf \
  --expert-streaming \
  --expert-prefetch \
  --ctx-size 2048 \
  -n 128

The repository’s README reports CPU-only measurements on one 13th-generation Intel Core i7, 16 GB of RAM and an NVMe SSD. It reports expert streaming at 7.01 tokens per second for Qwen3.5-35B-A3B Q4_K_M, 2.00 for Qwen3.5-122B-A10B Q2_K and 0.75 for MiniMax-2.7 IQ1_M. The linked eight-page report covers three models and says fixation produced visibly repetitive or incoherent output in several configurations even when the raw token rate looked acceptable. These figures are feasibility evidence from one machine, not a cross-hardware benchmark.

There is also a systems trade-off beyond speed. A 2025 IEEE Computer Architecture Letters analysis estimated that SSD-offloaded MoE inference can consume up to about 12 times the per-token generation energy of an HBM baseline under its assumptions. That study does not benchmark Swap-MoE, so it is not a verdict on this patch. It is a reason to describe SSD paging as a capacity escape hatch with a storage cost, not as free or green compute.

Use Swap-MoE only when you are comfortable pinning the upstream commit, rebuilding after upstream changes, keeping backups, checking output quality and accepting that a future llama.cpp update may require a new port. For a normal low-VRAM desktop, a smaller quantized model is usually the lower-risk choice.

Alternatives that may fit your hardware better

Apple Silicon: MLX-LM

MLX-LM is built for Apple Silicon and can use MLX’s unified memory, where CPU and GPU operations share the same memory pool. Its README documents quantization, rotating KV caches and a configurable prefill step size. It also warns that models larger than available RAM can be slow and that its large-model memory wiring requires macOS 15 or newer. If you have a Mac, test an MLX-community quantized model before forcing a CUDA-oriented workflow onto it.

Standard Transformers: bitsandbytes

The official Transformers bitsandbytes integration exposes 8-bit and 4-bit loading through BitsAndBytesConfig. This is a sensible choice when you need the Python Transformers ecosystem and the quantized checkpoint plus runtime fits in available memory. It is not an SSD expert-streaming system; lower-bit weights still leave KV, activations, framework overhead and model-specific requirements.

Serving hardware: vLLM CPU offload

vLLM’s current engine documentation exposes --cpu-offload-gb, KV offloading and asynchronous layer-group offload. The docs describe CPU offload as a per-GPU virtual memory extension and warn that weights move from CPU memory to GPU memory during forward passes, which requires a fast CPU-GPU interconnect. That can be useful on a server with the right bus and RAM. It is a poor default for a low-end laptop with a slow interconnect.

Research such as LLM in a flash and the SSD-offloading energy analysis is valuable for understanding windowing, contiguous reads, latency and energy. These papers are not installation guides. Keep research results separate from a supported consumer workflow.

Measure the setup honestly

Before comparing tools, define what “useful” means. The accepted-result evaluation method is a good reminder that a raw speed number is not the same as an accepted result.

1. Pin the model file, quantization, runtime version, backend, driver and operating-system build.
2. Use a fixed prompt set and generation length. Keep the seed and sampling settings fixed when the runtime permits it.
3. Measure cold start and warm decode separately. Record time to first token, prompt processing, decode tokens per second, peak VRAM, peak process RAM and available SSD space.
4. Run at least one quality check against a smaller known-good model or a full-memory reference. Look for repetition, truncated answers, wrong tool calls, broken formatting and loss of instruction following.
5. Change one variable at a time: context, slot count, GPU layers, quantization, KV type, prefetch or cache. Keep a short test log.
6. Stop if the machine begins swapping heavily, the SSD is nearly full, the output becomes incoherent or the backend reports numerical errors. “It eventually produced text” is not a production gate.

For a marketing team, also inspect the data path. Local weights do not automatically mean private processing if prompts are logged, a UI calls a remote service, plugins have network access or the model directory is shared. Document retention, access and failure behavior before using local inference for client material.

If you are evaluating a model because it is open-weight, separate the license, access and safety questions from the memory question. DMT’s open-weight model context is a useful adjacent read before a team treats “open” as a complete governance answer.

How this fits a practical marketing stack

Local inference is best framed as one component in a governed workflow, not as a universal replacement for hosted models. The 2026 AI in digital marketing overview provides the wider strategy context. A local model may be useful for offline classification, first-pass content grouping, private drafts or repetitive transformations when latency and quality are acceptable.

Use the AI automation workflow reference to map where a model belongs in an end-to-end process, then add explicit human review, source checks and rollback. A local model that saves API cost but creates a silent factual or brand error is not a saving.

The operational layer matters too. A technical SEO crawl and monitoring guide is useful for thinking about observability and crawl/render failure modes; apply the same discipline to model versioning, prompt fixtures, logs and endpoint health. If the workflow is agentic, compare it with the agentic workflow guide and keep tool permissions narrower than the model’s imagination.

A simple decision tree

Frequently asked questions

Can a 4 GB GPU run a 70B model?

It depends on the runtime, quantization, host RAM, context and what “run” means. The official llama.cpp example lists a 70B Llama 3.1 Q4_K_M file at 43.1 GB, so ordinary full-file loading is not a 4 GB-GPU scenario. PyTLLM’s README claims a 70B full-precision example at about 4 GB of VRAM through layer streaming, but that is an unverified project claim and still involves host/disk work. Treat it as a test hypothesis, not a guarantee.

Which quantization should I try first?

Start with a model-specific Q4_K_M GGUF, then compare Q5 or Q6 if quality matters and memory allows. Quantization tables and speed figures are format- and hardware-specific. Keep a known-good reference so a smaller file does not quietly become a lower-quality workflow.

Does SSD offload make large models fast?

No. It can make some models launch when resident memory would fail, but storage bandwidth and latency remain in the critical path. Swap-MoE’s own measurements show a wide range of token rates on one NVMe machine, while independent research warns about SSD energy cost. Capacity and throughput are different problems.

Is PyTLLM production-ready?

The repository is new and its headline VRAM table is self-reported. Test the exact model, Transformers version, GPU, RAM budget, disk capacity and output quality. Until independent benchmarks and a stable maintenance history exist, keep it in an experimental lane.

How do I avoid an out-of-memory error?

Lower context first, lower server parallelism next, and reduce GPU layers last. Then inspect KV-cache type, backend support and hidden multimodal or draft-model memory. Keep headroom for the operating system instead of filling every advertised byte.

Is Swap-MoE part of official llama.cpp?

No. It is a separate patch set tied to a specific upstream commit. Pin both repositories, retain the patch, test output quality and expect to re-port it after upstream loader or graph changes.

Sources and version notes

• llama.cpp repository and quantization guide.
• llama.cpp server options and multi-GPU and fit guidance.
• llama.cpp v0.2.0 stable release and b10699 nightly release, observed 30 August 2026.
• PyTLLM/TLLM repository, including its README, implementation and tests.
• Swap-MoE repository and its technical report.
• MLX-LM, Transformers bitsandbytes documentation and vLLM engine arguments.
• LLM in a flash and SSD Offloading for LLM Mixture-of-Experts Weights Considered Harmful in Energy Efficiency.

Bottom line: Use llama.cpp and a measured GGUF baseline first. Reach for MLX-LM or bitsandbytes when your hardware and software stack point there. Use PyTLLM or Swap-MoE when you deliberately want to test an experimental capacity escape hatch and are prepared to validate the result yourself.