Fitting a 35B model into 16 GB, then checking it gets things right
Four open models, tuned for a local Goose agent on one Radeon RX 9070 XT and then tested on real agent work. Qwen3.6 35B A3B won. It passed all 12 hard runs, and putting each part of it on the right side of the memory bus made long chats 6.5 times faster.
Last updated . See what changed.
The setup
A desktop PC running an AI agent entirely offline.
| CPU and RAM | AMD Ryzen 7 7700X, 30 GB |
|---|---|
| GPU | Radeon RX 9070 XT, 16 GB VRAM. The same card drives the monitors, so filling its VRAM takes the desktop down. |
| OS | Bluefin Linux (Fedora Atomic), GNOME on Wayland |
| Model server | LM Studio 0.4.26, llama.cpp runtime 2.64.0 (Vulkan) |
| Agent | Goose 1.56, with Playwright MCP for browser control |
| Jobs it needs to do | Code, web design, documents, web research, personal finance, GitHub and Cloudflare via MCP, computer control |
Web search, GitHub, Cloudflare and browser control come from Goose's extensions and MCP servers, not from the model. The model's job is to call those tools reliably and get the details right. It also needs vision for screen-based computer control.
Starting point: three stacked failures
Goose answered every message with null result from llama cpp. Each problem was hidden behind the one before it.
| Symptom | Cause | Fix |
|---|---|---|
missing tensor 'blk.64.ssm_conv1d.weight' | Goose's built-in llama.cpp couldn't read a newer hybrid GGUF | Run models in LM Studio |
| Config said LM Studio, chat still used the built-in backend | A Goose session keeps the provider it was created with | Start a new chat |
request (9501 tokens) exceeds the available context size (8192 tokens) | Goose sends its system prompt and every tool definition before you type | Load with a larger context (now 128K) |
Read the logs before changing settings. Each error message named its own cause.
Why mixture-of-experts models fit this card
An MoE model stores all its weights but only computes with a few of them per token.
| Dense (Qwen3.8 27B) | MoE (Qwen3.6 35B A3B) | |
|---|---|---|
| Weights stored | 27B | 35B |
| Weights used per token | all 27B | about 3B: a router picks a few experts |
| File at 4-bit | 16.8 GB | 22.1 GB |
| Past 16 GB of VRAM | every token waits on system RAM, so it slows sharply | rarely used experts sit in RAM, so it slows a little |
| Measured generation | 8.9 tok/s | 29.2 tok/s untuned, 39.3 tuned |
Per gigabyte, an MoE model is somewhat less capable than a dense one. But on a card the dense model doesn't fit, MoE wins clearly. The dense model was dropped.
Speed tuning
Most of the gain came from where the model's pieces sit in memory, not from exotic settings.
The big one: keep every layer on the GPU
Raising the context to 128K looked free on a 10K-token prompt. A 79K-token prompt showed the real cost: Qwen3.6's generation fell from 29 to 5.5 tok/s. The real launch arguments explained why. LM Studio's automatic placement put 22 whole layers on the GPU and 18 on the CPU, attention included. So the CPU was doing attention over the entire chat.
The fix is the standard one for MoE models: every layer on the GPU, with only some layers' experts kept in system RAM (llama.cpp's --n-cpu-moe, LM Studio's "MoE experts on CPU"). Attention stays on the GPU, and only the cheap, rarely used expert weights wait on RAM.
Every step of every sweep
Each sweep step changed one setting, reloaded the model and timed a ~10K-token prompt and a ~79K-token prompt. The highlighted row is the setting the picker chose: fastest long-chat generation with VRAM and pinned RAM both at or under 14 GB.
Other settings that mattered
- mmap off: 61–99% faster prompt reading on every model. With mmap on, RAM-resident weights are pages of the model file. With it off they're ordinary memory, which the GPU appears to read much faster. The speedup is measured; the explanation is likely but not proven.
- Physical batch (ubatch) 2048: about 2× faster prompt reading than the default 512, for 0.4 GB more VRAM.
- Q8 K/V cache: saves 1.2 GB of VRAM at roughly 3% speed, which buys headroom for the desktop.
- CPU threads, flash attention: no measurable difference.
Two memory limits, not one
VRAM is the obvious limit. The second is GTT: system RAM the GPU driver pins for the GPU, capped at about 15.6 GB here. Experts kept in RAM are pinned in GTT, so moving more of them off the GPU trades one limit for the other. With 80% of experts in RAM, GTT filled, the load failed and the kernel logged 2,832 Not enough memory errors in one burst.
Vulkan beat ROCm on every model
LM Studio's ROCm runtime was slower and used about 5 GB more VRAM than estimated, so far less of each model fit:
| Model | Best ROCm offload | ROCm read / gen | Vulkan read / gen |
|---|---|---|---|
| Gemma 4 26B A4B | 50% | 933 / 25.0 | 2,776 / 36.3 |
| Qwen3.6 35B A3B | 20% | 476 / 21.2 | 1,118 / 29.2 |
| Qwen3.8 27B | none fit | – | 629 / 8.9 |
Every ROCm run also logged thousands of init_user_pages: Failed to get user pages: -1 errors. ROCm was asking for write access to the read-only memory-mapped model file. Turning mmap off removed the errors but didn't change the speed. The server log also showed missing hipBLASLt kernels for this GPU (gfx1201) in LM Studio's ROCm bundle. That's LM Studio's to fix.
A second look found that most of this gap was LM Studio's packaging, not ROCm. See ROCm, revisited.
GLM-4.7-Flash: a runtime limit
GLM's attention cache was far larger than LM Studio estimated. Its long-prompt reading stuck near 110 tok/s at every placement and batch size, 16× slower than Gemma. That pattern points to the runtime, not the settings: LM Studio's Vulkan build likely lacks a fast path for GLM's attention type. That's inferred from the measurements, not confirmed.
Quality: does it get the job right?
Speed says nothing about accuracy. Each tuned model ran real Goose tasks headless, scored automatically and then checked by hand.
Six easy tasks barely separated the models (Qwen3.6 12/12, Gemma 11/12). So a harder set followed, with traps, conflicting sources and hidden checks. Each model ran each task twice, with at most 40 tool steps and 25 minutes per run.
| Task | What makes it hard |
|---|---|
| H1 Code | Implement a spec across three modules; 10 hidden tests the model never sees |
| H2 Research | Ten facts about three npm packages across several pages, with sources |
| H3 Finance | Balance transfer vs personal loan: fees, a 0% promo, a grossed-up loan amount, APR vs APY |
| H4 Document | A brief from notes, emails and a policy, where a later email changes the date and an owner |
| H5 Web | A pricing page whose monthly/yearly toggle must actually work, plus accessibility and phone layout |
| H6 Agent | Flag the right 4 orders among 40 across four pages, past decoys and confirmation dialogs |
What the failures were
- H6 measures step efficiency as much as accuracy. Only Qwen3.6 finished within 40 steps (28 and 36). Gemma reused stale page references. Flint saved every page snapshot to a file and read it back, two steps per look; it had flagged 3 of the 4 right orders when it ran out. In daily use Goose would simply ask to continue.
- Gemma's H5 failure was a real bug: an inverted condition (
nextIsYearly = !isCurrentlyMonthly) meant the toggle could never switch. It rewrote the file 12 times instead of debugging, and timed out. - GLM's H3 error was consistent: both runs totalled 30 full $300 payments ($9,000) instead of $8,815.01, missing the smaller final payment.
Temperature matters
Qwen3.6 runs at its publisher's precise-coding temperature of 0.6, while Gemma ran at Google's general default of 1.0. Rerun at 0.6, Temperature fixed Gemma's coding reliability but not its step efficiency.
The scorers needed checking too
Before any model ran, every scorer was tested against a correct answer and a deliberately wrong one. That caught two wrong expected values in the hidden tests, which would have failed a perfect model. After the runs, three correct answers turned out to be marked down: two briefs that noted a handover in the owner column, and a loan answer that said "cheapest" instead of "cheaper". They were re-scored.
Reading every brief and loan explanation by hand also found slips the scorers passed. Qwen3.6 had two: a brief said someone was "out through mid-November" when the email gave no return date, and one loan breakdown didn't add up despite correct totals. Gemma and Flint had none.
Read the actual answer before trusting either a pass or a fail.
ROCm, revisited
Outside LM Studio, ROCm works. It still loses to Vulkan where it counts, and now we know why.
ROCm is AMD's answer to NVIDIA's CUDA, so it ought to win on AMD hardware. The first round said otherwise, so round two took LM Studio out of the picture: the same model and the same tuned settings, run in llama.cpp's own containers with llama-bench.
What was wrong inside LM Studio
- A missing math library. LM Studio's ROCm 7.2.4 bundle has no hipBLASLt kernels for the RX 9070 XT (gfx1201), so every load errors and falls back. This is an open bug (#1234), unanswered for eleven months.
- A 4.66 GB buffer for the vision encoder. On ROCm, loading Qwen3.6's image encoder tried to reserve 4.66 GB of VRAM. That was the "about 5 GB extra" from round one, and why so few layers fit.
- An unfair first test. Round one moved whole layers off the GPU instead of only the experts.
- A trap on any desktop Ryzen. ROCm counts the CPU's built-in graphics (gfx1036) as a second GPU with 15.6 GB of "VRAM", which is really system RAM, and llama.cpp splits the model across both.
HIP_VISIBLE_DEVICES=0fixes it.
Upstream containers, same model and settings
| Backend | Read, empty | Write, empty | Read at 78K | Write at 78K | Pinned RAM |
|---|---|---|---|---|---|
| Vulkan (RADV, Mesa 26.2.5) | 2,189 | 41.0 | 1,150 | 36.5 | 12.3 GB |
| ROCm 7.2.1 | 2,231 | 41.1 | 1,367 | 26.9 | 0.1 GB |
| ROCm 7.2.1, hipBLASLt off | 2,763 | 41.7 | 1,245 | 25.6 | 0.1 GB |
| ROCm 7.2.1, one hardware queue | 2,355 | 41.6 | 1,344 | 26.3 | 0.1 GB |
| ROCm 10.3 nightly (TheRock, via Lemonade) | 2,764 | 34.0 | 1,622 | 27.6 | 0.3 GB |
| ROCm 7.2.1, hipBLASLt off, f16 cache | 2,567 | 42.9 | 1,513 | 32.8 | 0.4 GB |
Why ROCm still trails
ROCm matches CUDA's programming interface, not its performance. llama.cpp's ROCm backend is its NVIDIA code, automatically translated and tuned mainly for NVIDIA and AMD's data-centre chips. Consumer RDNA 4 only got official ROCm support this year. Vulkan on Linux, by contrast, rides on Mesa's RADV driver and ACO compiler, which have had years of gaming-driven tuning on exactly these cards.
The two kinds of work favour different strengths. Writing a reply is limited by memory bandwidth, where Vulkan's kernels get closer to the card's 640 GB/s. Reading a prompt is limited by maths, where ROCm's matrix libraries pull ahead. That is exactly the pattern above: ROCm reads 20–40% faster, and Vulkan writes faster in long chats, which is where an agent spends its time.
The one setting that mattered was the cache type. ROCm reads a Q8 cache slowly over long contexts, and switching to f16 took long-chat writing from 25.6 to 32.8 tok/s. The remaining 10% gap is llama.cpp's ROCm attention code, which no setting fixes. The BIOS was checked too: EXPO already runs the RAM at DDR5-6000, Resizable BAR is on, and the B650 board's graphics slot is PCIe 4.0 by design.
Vulkan for running models; ROCm for training them, which is something Vulkan can't do.
Leaving LM Studio
Same engine, fewer layers in between: llama.cpp's own server as a Podman Quadlet.
LM Studio's Vulkan runtime works fine, but it trails upstream llama.cpp by weeks and hides launch settings behind per-model JSON with traps (one missing key silently undid the expert placement). Upstream Vulkan matched it on speed (36.5 vs 35.8 tok/s at 78K), so moving costs nothing.
The replacement is llama-server in router mode, running as a systemd user service through Podman's Quadlet. One server lists several models, each with its own tuned flags in an INI file, loads whichever one Goose asks for by name, keeps only one in memory, and puts it to sleep after 10 idle minutes. Asleep, it hands back its VRAM and pinned RAM (measured: VRAM back to 1.0 GB, pinned RAM to 0.2 GB), so games get the card without anyone remembering to stop anything. A cold load takes about 12 seconds.
Bluefin's own docs point to llmman for this. It runs the same upstream containers, but its build command reads a whole model file into memory, then a second copy, before writing it. Packaging the 21 GB model needed about 42 GB of RAM, and the kernel stopped it at 24.6 GB of the 30 available. The streaming code to fix it already exists in its source, unused.
Same model, two servers, and an Unsloth quant
With the new server in place, the 12 hard-task runs were repeated for the same Qwen3.6 file and for Unsloth's "Dynamic" quant of the same model (UD-Q4_K_XL, a similar size), which promises more quality per gigabyte.
| Run | Points | Median time per task | Median tokens written |
|---|---|---|---|
| LM Studio, Q4_K_M (Oct 24) | 24/24 | 102 s | 3,066 |
| llama-server, Q4_K_M | 24/24 | 154 s | 4,890 |
| llama-server, Unsloth UD-Q4_K_XL | 22/24 | 108 s | 3,566 |
- The move kept accuracy. The same file scored 24/24 on the new server. It wrote about 60% more on its median task, mostly extra thinking, so tasks took longer. Twelve runs is too few to tell whether that is the server or chance.
- The Unsloth quant made one real mistake. In one loan comparison its script left out the interest after the 0% period, giving 28 payments and $8,242.68 instead of 30 and $8,815.01. The answer text even mentioned the interest it had dropped. One miss in twelve isn't proof it's worse, but it gives no reason to switch.
- A pass can hide a sloppy summary. In two order-review runs the model flagged exactly the right orders in the app, then listed extra order numbers in its final message.
Final setup
Qwen3.6 35B A3B, served by upstream llama.cpp (Vulkan) in a Podman Quadlet: 128K context, every layer on the GPU, 25 layers' experts in system RAM.
| Where | Setting | Value | Why |
|---|---|---|---|
| Server | Engine | ghcr.io/ggml-org/llama.cpp:server-vulkan, pinned build | Same speed as LM Studio, newer builds, exact flags |
| Server | Mode | router, one model at a time, sleep after 600 s idle | Loads on demand; frees the GPU for games |
| Server | GPU passed in | card1 and renderD128 only | Keeps it off the Ryzen's built-in graphics |
| Model | n-gpu-layers / n-cpu-moe | 99 / 25 | Attention on the GPU, rarely used experts in RAM; 6.5× faster long chats |
| Model | load-mode | none (no mmap) | 61–99% faster prompt reading |
| Model | ctx-size / parallel | 131072 / 1 | Long chats, one user |
| Model | K and V cache | q8_0 | 1.2 GB less VRAM, no speed cost on Vulkan |
| Model | ubatch-size | 2048 | About 2× faster prompt reading |
| Model | Sampling | temp 0.6, top-k 20, top-p 0.95, min-p 0 | Qwen's precise-coding preset |
| Goose | Provider | custom llama_server provider → 127.0.0.1:8080 | OpenAI-compatible; 131072-token limit per model |
| Goose | Extensions | Developer, Todo, Memory, Analyze, Computer Controller, Extension Manager, Playwright | Fewer tools to choose from |
| Desktop | Control | llm status | unload | off | on | logs | llm off before gaming guarantees nothing loads mid-game |
| GNOME | Automatic suspend | off | Suspending with a model loaded crashed the desktop |
The model's entry in ~/.config/llama/models.ini (shared settings like the cache type sit in a [*] section):
[qwen3.6-35b-a3b]
model = /models/lmstudio/lmstudio-community/Qwen3.6-35B-A3B-GGUF/Qwen3.6-35B-A3B-Q4_K_M.gguf
mmproj = /models/lmstudio/lmstudio-community/Qwen3.6-35B-A3B-GGUF/mmproj-Qwen3.6-35B-A3B-BF16.gguf
n-cpu-moe = 25
ubatch-size = 2048
temp = 0.6
top-k = 20
top-p = 0.95
min-p = 0The other models
- Recursive Flint: as accurate as Qwen3.6 on everything except step efficiency, and the most careful with details, but about 2.7× slower per task. A good second opinion for code when time doesn't matter.
- Gemma 4 26B A4B QAT: the fastest, and clean on details. Set its temperature to 0.6 for coding and agent work.
- GLM-4.7-Flash: not recommended on this machine. It's slow at every setting and got a loan total wrong twice.
Lessons
Things that cost time, so you don't have to repeat them.
- Read the server's real launch arguments.
/proc/<pid>/cmdlineshowed settings the UI implied were applied but weren't. - Benchmark long prompts. 128K context looked free at 10K tokens and cost 80% of generation speed at 79K.
- Never run two model servers on a GPU that drives your screen. A leftover ROCm server kept 13.4 GB and the next load crashed GNOME.
- Watch GTT as well as VRAM. Moving experts to RAM trades one limit for the other.
- Block suspend during long GPU jobs. "Performance" power mode doesn't stop the idle timer. Use
systemd-inhibit --what=sleep:idle. - Journal every step. A
STARTwith noENDmarks exactly where a crash happened. - Use the publisher's sampling settings. Gemma at temperature 1.0 made a logic bug it didn't make at 0.6.
- Test the tests. Scorers had wrong expected values and false negatives; both were caught only by checking them.
- On a desktop Ryzen, hide the built-in graphics from ROCm with
HIP_VISIBLE_DEVICES=0, pass it/dev/kfdplus all of/dev/dri, and use an f16 cache. - Separate the app from the engine before blaming the engine. "ROCm is slow" was mostly "LM Studio's ROCm package is broken".
- Cap memory on anything that touches a 20 GB file. A packaging tool that read the model into RAM took the terminal down with it.
systemd-run --user -p MemoryMax=…confines the damage. - Small things:
lms chatprints stats to stderr; a 300 KB prompt is over Linux's 128 KB argument limit, so pipe it on stdin; Playwright MCP blocksfile://, so serve test pages over local HTTP.
How it was run
A small Python harness, standard library only, running unattended as systemd user services.
- Speed sweeps write each setting into LM Studio's per-model settings file, reload, verify the launch arguments, then time two prompts (two measured runs after a warm-up). A random string at the start of each prompt stops llama.cpp reusing its cache.
- Crash safety: kill leftover servers and require VRAM under 1.5 GB before each step. Watchdogs kill the server above 15.0 GB VRAM or 14.8 GB GTT. The run stops if the kernel logs GPU memory errors, and the original settings file is restored if the run is killed. The watchdogs stopped 15 steps before they could take the desktop down: 7 in the ROCm tests and 8 in the tuning sweeps.
- Settings picker: fastest long-chat generation with ≥ 2 GB VRAM headroom. It applies the publisher's sampling preset and confirms a flag-free load reproduces the tested arguments.
- Evaluation:
goose runper task and model, with session exports kept for hand review. Scorers check code with hidden tests, numbers against computed truth, and pages in a real browser. - Backend benchmarks:
llama-benchin llama.cpp's own containers, one at a time, with the same VRAM, pinned-RAM and kernel-error watchdogs.
The charts and sweep tables are generated from the run journals, and all the raw numbers are in data/results.json. The harness lives in a private repository.
Results log
What was tested and found, newest first.
- Moved off LM Studio to llama-server in a Podman Quadlet (router mode, idle sleep); same 24/24. Unsloth UD-Q4_K_XL scored 22/24. ROCm re-tested outside LM Studio and re-tuned; Vulkan stays.
- Hard tasks complete for all four tuned models; Gemma retested at temperature 0.6. Qwen3.6 confirmed as the daily model.
- 128K tuning sweeps for Qwen3.6, Gemma 4, Flint and GLM-4.7-Flash. Easy-task evaluation. Qwen3.8 dropped.
- ROCm vs Vulkan comparison with a crash-safe harness; mmap-off discovery.
- Fixed Goose: moved to LM Studio, new session, larger context.
Next up: more runs per task so small differences become meaningful, an 80-step H6 rerun, and a fresh ROCm check when llama.cpp's RDNA 4 attention code improves.
Sources
- LM Studio model catalog: sizes, capabilities and MoE parameter counts
- Unsloth run guides: Qwen3.6, Gemma 4, Flint, GLM-4.7-Flash: sampling, memory needs, known issues
- Recursive Flint model card, LM Studio GLM-4.7 page
- llama-server README: router mode, model presets, sleeping on idle
- ROCm #6203 (hipBLASLt slowdown on gfx1201) and TheRock #9228 (random slow state,
GPU_MAX_HW_QUEUES=1) - lemonade-sdk/llamacpp-rocm: nightly llama.cpp builds on TheRock
- Vulkan beats ROCm on RDNA 4: bandwidth efficiency measurements
- Bluefin AI docs and Unsloth on AMD
- All benchmark and evaluation figures were measured on the hardware above, Oct 24–25, 2026.
Changelog
Changes to this page, newest first.
- Added "ROCm, revisited" and "Leaving LM Studio" with the Unsloth quant comparison; rewrote the final setup for the llama-server Quadlet.