Strata on a Mac Studio M5 Ultra (96 GB)

Strata made headlines with an unusual claim: running Qwen3.8-Flash-Next, a 125-billion-parameter AI model, on a consumer PC with just 12 GB of VRAM. The minimum requirements were a 12 GB RTX GPU and 32 GB of system RAM. In one widely shared demo, an RTX 5070 with 64 GB of RAM achieved 94 tokens per second, on a regular gaming PC, not an expensive AI server.

The secret is how Strata manages memory. Qwen3.8-Flash-Next uses a mixture-of-experts (MoE) architecture with 24,576 experts, but only about 10 are activated per token. Instead of loading the entire model into VRAM, Strata distributes the workload across three tiers:

• GPU (VRAM): Keeps frequently used experts and handles attention and model state. More VRAM means more experts can stay cached.
• System RAM: Holds the complete set of expert weights, making less frequently used experts available when needed.
• SSD: Stores a 28.8 GB n-gram lookup table, reading only the required entries instead of loading the whole table into memory.

On a Mac Studio with 96 GB of unified memory, things work differently. Apple Silicon shares memory between the CPU and GPU, so there's no separate VRAM-to-RAM transfer. With Metal and memory mapping, the full expert set can remain in unified memory, provided it fits alongside the runtime's other memory requirements.

Strata also uses Multi-Token Prediction (MTP) to speed up generation. It predicts several tokens ahead, then lets the full model verify them together in one pass. This improves generation speed while preserving the target model's output distribution when verification is implemented correctly.

My Mac Studio M5 Ultra with 96 GB of unified memory just arrived, and I want it to be a headless inference machine: no monitor, no keyboard, no seat at the desk. I reach it over SSH, and it answers on an HTTP port, an OpenAI-compatible API and an Anthropic API serving the model to every machine in the house. Since I am setting it up anyway, I may as well document it.

Prerequisites

Turn on SSH. System Settings → General → Sharing → Remote Login, or from the shell:

sudo systemsetup -setremotelogin on

Set a hostname.

sudo scutil --set ComputerName "mac-studio"
sudo scutil --set LocalHostName "mac-studio"
sudo scutil --set HostName "mac-studio"

From here on, I connect as ssh tpham@mac-studio.

Stay awake. System Settings → Energy → automatic startup after a power failure, then:

sudo pmset -a sleep 0
sudo pmset -a autorestart 1

Command Line Tools.

xcode-select --install

The setup

Unlock the GPU memory

Here is the one macOS quirk I had to know before serving anything. MacOS does not let the GPU use all of the unified memory. By default it caps the GPU's wired memory at roughly 2/3 of total RAM; on this 96 GB machine that is about 78 GB, and the rest is off-limits to Metal no matter how much the model needs.

I raise the cap to 88 GB:

sudo sysctl iogpu.wired_limit_mb=90112

That leaves ~8 GB for macOS itself and gives the model room to stay fully resident, with headroom for a larger context window. The setting resets on reboot, so I load it at boot with a LaunchDaemon:

sudo tee /Library/LaunchDaemons/com.local.iogpu-limit.plist > /dev/null <<'EOF'
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>Label</key><string>com.local.iogpu-limit</string>
  <key>ProgramArguments</key>
  <array>
    <string>/usr/sbin/sysctl</string>
    <string>iogpu.wired_limit_mb=90112</string>
  </array>
  <key>RunAtLoad</key><true/>
</dict>
</plist>
EOF

Then set the ownership and permissions a LaunchDaemon requires, validate the file, and load it right away:

# Set correct ownership and permissions
sudo chown root:wheel /Library/LaunchDaemons/com.local.iogpu-limit.plist
sudo chmod 644 /Library/LaunchDaemons/com.local.iogpu-limit.plist

# Validate the configuration
sudo plutil -lint /Library/LaunchDaemons/com.local.iogpu-limit.plist

# Load the LaunchDaemon immediately
sudo launchctl bootstrap system /Library/LaunchDaemons/com.local.iogpu-limit.plist

plutil -lint should print OK, and since RunAtLoad fires on bootstrap, sysctl iogpu.wired_limit_mb should now print 90112 without a reboot.

Install and launch

The setup guide offers three paths: the app, the terminal, and a source build, all running the same engine and models. For a headless server, the terminal is the right one. Over SSH:

curl -fsSL https://github.com/Niko1221/Strata/releases/latest/download/install.sh | sh

strata launch

strata launch is the one command that does everything. It asks one question at a time (model, context, image input, who may connect), then downloads the signed engine component and the model shards and starts serving. On this machine it recommended the IQ3_S quant, and I set the context to 128K.

Then it offers to tune the engine on the machine itself: a quick pass of about 20 minutes, or a thorough pass of up to 2 hours. Strata measures the engine settings on this Mac and keeps the fastest, so the KV cache format, the MTP draft and speculation window, the prefill chunks, and the expert cache all end up tailored to this machine rather than left at generic defaults. I gave it the quick pass; the tuning runs once and the result is kept, and a model tuned before for the same choices is not measured again unless I ask for --retune.

Access from the network

Since this is a headless server, I picked My local network on the Access screen ("other computers at home or the office"). Strata then turns the API key on and makes one for me, because the server now reaches beyond this computer; clients send it as their API key. To see which key it picked, run these commands in another terminal while Strata is running:

strata launch --list-presets                       # the preset names on this machine
strata launch --show-key --preset "IQ3_S 128K"     # the key for that preset

The server then answers on the LAN at the machine's IP or hostname, for me mac-studio:

Without a key the server still starts, but it warns that anyone who can reach the port can use the model, so I kept the key Strata generated.

Why IQ3_S

At 96 GB, Strata recommends the IQ3_S quant, a build by ISTA-DASLab, the Deep Algorithms and Systems Lab at the Institute of Science and Technology Austria. Their builds are non-uniform: instead of one quant type for every weight (like a plain Q4_K_M), a search assigns each of the model's 352 tensors its own type, spending precision where the model is sensitive and saving it where it is not. The method is described in their GSQ and RCO papers; the output is still a plain GGUF that llama.cpp, Ollama, LM Studio, and Strata load unmodified.

The unusual part is the result. Their release claims the IQ3_S build matches, and on some benchmarks exceeds, the BF16 model it was quantized from:

A 3.5-bit quant that is statistically indistinguishable from the 354 GB model it was made from (and nominally ahead on two of three reasoning benchmarks) means quantization at this size is close to free. (The model card itself is careful: results slightly above 100% are within benchmark noise, so I read them as parity, not improvement.)

Why this quant and not a bigger one comes down to what stays resident. Strata's Metal engine keeps the model's main file mapped and GPU-resident, so the question is what fits inside the memory I unlocked above. Since this is the 96 GB M5 Ultra, I would rather run the smaller 3.5-bit quant, which matches the base model and leaves room for more context, than Unsloth's higher-quality UD-Q4_K_XL, which needs ~91 GB and would have to stream experts from the SSD at any bigger context. IQ3_S needs ~63 GB with the KV cache, buffers, and the draft layer, and stays fully in memory.

The download is two shards: 54.8 GB of transformer weights plus the 28.8 GB n-gram table, plus ~5 GB for the MTP draft layer. I budget about 80 GB of free disk on the internal SSD. Only the first shard has to be resident; the n-gram table is read sparsely and lives happily on the SSD.

The strata commands