Every question I send to Claude/chatGPT is metered, so I ration the long, dull jobs: reading a hundred papers, trying fifty framings of an idea, grinding on code until a test passes. A local model makes those jobs free to attempt, and everything stays on my own disk.
Now, I have a 125-billion-parameter model running on a (relatively) small computer under my desk. If I am to believe th the coding and agent benchmarks Alibaba published, it beats Claude Opus 4.6, which was one of the strongest models of early 2026. I'm getting about 38 tokens a second and costs absolutely nothing (well, I paid for the machine).
The machine and the model
Quick recap first: the machine in question here is the Framework Desktop I built in August, with AMD's Ryzen AI Max+ 395. Its 128 GB of memory is shared by the CPU and the built-in GPU, and the GPU can address about 124 GB of it. Consumer graphics cards top out at 24 or 32 GB. Bandwidth is modest, around 256 GB/s, so this is a capacity machine, not a speed machine.
The model is Qwen3.8-Flash-Next, an open-weight preview of the architecture Alibaba is building Qwen4 on. It has 125 billion parameters but reads only about 6 billion per token: a shared trunk plus 10 of its 512 experts. On a bandwidth-limited machine (which this is), bytes read per token set the speed. At 4-bit precision the weights take 93.7 GB, so one copy fits with little to spare.
Claude Code, running on the machine, did the install from a short brief. The engine is drluoto's Strix Halo branch of llama.cpp, and the 4-bit weights are Unsloth's.
What it's like
- One conversation: about 38 tokens a second (41 on short answers).
- Three conversations at once: 13–15 tokens a second each, about 42 combined.
- Memory used: about 109 of 128 GB.
Parallel conversations share one memory bus, so they add almost no throughput.
Artificial Analysis, which runs the same tests on every model, scores it 40 on its current index (and this is an index where Gemini 3.1 Pro Preview scores 30). Alibaba's own numbers put it ahead of Opus 4.6 on agentic coding (62.5 to 53.4 on SWE-bench Pro), though behind it on the hardest reasoning exams.
Three big caveats here:
- My copy is compressed to 4 bits, which costs some quality on hard problems
- It is "wordy", and at 38 tokens a second the extra words take real time
- Its factual recall is poor: on Artificial Analysis's knowledge test it scores −12, meaning more wrong answers than right ones. So the first thing I added was web search, through the MCP support built into llama.cpp's chat page. With search, it looks facts up instead of guessing.
What I'll do with it
The plan is a division of labor. The local model is slower and weaker than Claude, but it can run all night for free!
Claude is better (naturally), but every token is paid for.
So, I'm thinking ... they can share a folder. Claude writes each task and a way to check the result. The local model does the volume. In the morning, Claude reads short summaries and tells me what deserves my attention ... this is the general plan, need time to play around and see what works.
If it works out, lots of experiments await (reading/analyzing, a lot of web-crawling type stuff that I've never bothered to try out because it'd be too token-heavy).
But first, I'll spend a few evenings "learning the model's limits":
- how speed holds up as a conversation grows
- how long it takes to read a long document
- where its tool use breaks
- and also, how it compares with Claude on my own tasks (gotta come up with a good way to test it)
Stay tuned for more on this!