Notes · VulRunWG
Big models, no CUDA: what fits, and why
· written by the developer of VulRunWG
People keep asking whether you can run large AI models without NVIDIA’s CUDA, and on AMD graphics at all. VulRunWG does it every day, and has run on AMD graphics since July 2026. This note shows what fits where, how a model bigger than the graphics card still runs, and where the real limit is. Every number names the machine and the date.
Development record: the figures below describe the dated builds in the measurement notes. They are not a current download size, hardware guarantee or release certification. Current runtime information.
Not a wrapper
Most local AI tools are built on llama.cpp or on PyTorch with CUDA. VulRunWG is built on neither. It has its own engine: about 200 GPU programs written for it in WGSL, run through Vulkan, which a current NVIDIA or AMD driver already includes. There is no CUDA, ROCm or PyTorch to install, which is why the September 26 development archive was about 38 MB, excluding models. A current public package size is not established here.
That is checked, not just claimed. One test rigs llama.cpp to crash the moment anything touches it, then loads a model and generates. The tested generation path completes without triggering the trap. That checks implementation independence, not the factual accuracy of its answers. A second check walks everything the running engine loads and must find no PyTorch, ONNX Runtime or llama.cpp.
A 30B model in an 8 GB budget
Mixture-of-experts models, such as Qwen3-30B-A3B, have 30 billion parameters, but only a few “experts” are used for each word. VulRunWG keeps the experts in use on the graphics card and leaves the rest in the PC’s memory until they are needed. The card holds a fraction of the model, and the room it saves goes to a longer conversation.
| Qwen3-30B-A3B (Q3_K_S) | on the card | conversation | speed |
|---|---|---|---|
| Automatic, 16 GB card | 11.1 GB | 10,511 tokens | 20.4 tok/s |
| Held to an 8 GB card’s budget | 5.9 GB | 43,951 tokens | 18.5 tok/s |
RTX 5080, 26 September 2026, with the card budget forced for the second row. Answers were correct in every row. The whole model on the card would need 13.0 GB and leave room for only 256 tokens of conversation.
On a laptop with no graphics card
The Radeon 780M is the graphics built into AMD laptop processors, with 4 GB taken from system memory. No ROCm, no extra drivers: the same VulRunWG download that runs on an RTX 5080.
| Radeon 780M | result |
|---|---|
| Chat, Qwen2.5 1.5B (Windows 11) | 29.6 tok/s |
| Read a 728-token prompt, 1.5B (Windows 11) | 1.6 s |
| Chat, Qwen2.5 3B (Linux) | 13.8 tok/s |
| A minute of speech to text, Whisper (Linux) | 3.0 s |
| Qwen2.5 7B | refused: too big for 4 GB |
Windows 11, 30 September 2026. Linux Mint with Mesa 25.2.8, 28–30 September 2026.
Against llama.cpp, same card, same file
| Qwen2.5 7B Q4_K_M, RTX 5080 | VulRunWG | llama.cpp |
|---|---|---|
| Writing the answer | 150.4 tok/s | 159.0 tok/s |
28 September 2026. llama.cpp build b11146 with its Vulkan backend, run with its own benchmark tool, which carries less overhead than VulRunWG’s figure. Reading a long prompt is still about 1.7× slower than llama.cpp; that is the biggest remaining gap.
Where the limit is
The limit is memory, not a number written into the software. VulRunWG works out what a model needs before loading it, and refuses with a plain message rather than crashing the graphics driver. In practice:
- A mixture-of-experts model can be bigger than the graphics card, as above. It runs slower when experts have to be fetched, and the PC needs enough memory for the rest of the model.
- An ordinary (dense) model has to fit on the card. On the 4 GB 780M that means up to 3B, on an 8 GB card up to 7B, and on a 16 GB card up to 14B.
- One model runs on one graphics card. Splitting a single model across two cards is not built yet.
- The September conformance record covers 15 model files from seven families: Qwen 2.5 and 3, Llama 3, Mistral, Phi-3, Gemma 2 and 3, and SmolLM2. Loading a model is not proof that it passed those checks; historical results do not certify a later build.
What hasn’t been tested
- AMD Radeon RX graphics cards. Only the 780M processor graphics has been measured so far.
- Intel graphics.
- Linux beyond one laptop.
- Handhelds such as the ROG Ally or Steam Deck. We don’t own one.
Current availability
VulRunWG remains in development, with no approved public download offered here. The policy is free personal, educational, nonprofit, workplace and local developer use; final public agreements still await approval. You can ask about the project. The full tables are on the measurements page, with every date, setting and method in the measurement notes.