Vulkan backend much easier on the CPU and GPU memory than CUDA.
On linux and compiled my own llama.cpp with CUDA support, top would always show one pegged CPU core at 100% when running Qwen3.5-9B-GGUF:Q4_K_M on my potato like RTX A2000 12GB. Also, nvidia-smi would show 11GB+ of GPU memory usage. Speed is ~30 tokens per second. My system fans would spin up when this single core gets pegged which was annoying to listen to. Decided to compile llama.cpp again with Vulkan backend to see if anything would be different. Well it was a big difference when using the exact same model Now, top is only showing one CPU core at about 30% usage and nvidia-smi is only showing 7.2GB of GPU memory usage. Speed is the same at ~30 tokens per second. No longer have my system fan spinning up when running inferencing. Just curious why the GPU memory footprint is lower and CPU
Could not retrieve the full article text.
Read on Reddit r/LocalLLaMA →Reddit r/LocalLLaMA
https://www.reddit.com/r/LocalLLaMA/comments/1saizts/vulkan_backend_much_easier_on_the_cpu_and_gpu/Sign in to highlight and annotate this article
Conversation starters
Daily AI Digest
Get the top 5 AI stories delivered to your inbox every morning.
More about
llamamodelllama.cpp
Running OpenClaw with Gemma 4 TurboQuant on MacAir 16GB
Hi guys, We’ve implemented a one-click app for OpenClaw with Local Models built in. It includes TurboQuant caching, a large context window, and proper tool calling. It runs on mid-range devices. Free and Open source. The biggest challenge was enabling a local agentic model to run on average hardware like a Mac Mini or MacBook Air. Small models work well on these devices, but agents require more sophisticated models like QWEN or GLM. OpenClaw adds a large context to each request, which caused the MacBook Air to struggle with processing. This became possible with TurboQuant cache compression, even on 16gb memory. We found llama.cpp TurboQuant implementation by Tom Turney. However, it didn’t work properly with agentic tool calling in many cases with QWEN, so we had to patch it. Even then, the
AI As Co- Collaberator
I’ve long been thinking on the idea of AIs as co-collaborators on projects. My line of reasoning typically involves theoretical arguments and such, where you present an idea and you present it in such a way that the AI is encouraged to contemplate the idea alongside you.This is akin to being a senior researcher and inviting other researchers to work alongside you. Sometimes you just need more hands in a lab but sometimes you want more minds picking away at the idea. And so in this endeavor I have worked on the idea of how to conceptualize AI as a co-collaborator not just as an information deliverer or a giant calculator. Now some of this is in general just in the AI’s general ability to be generative on certain topics. AI, as large language models, work by breaking down conversations into
Stop Explaining Your Codebase to Your AI Every Time
Every conversation with your AI starts the same way. "I'm building a Rails app, deployed on Hetzner, using SQLite..." You've typed this a hundred times. Your AI is smart. But it has no memory. Every chat starts from zero. Your project context, your conventions, your past decisions — gone. What if your AI already knew all of that? Here are five notes that make that happen. 1. Your stack, saved once Write one note with your tech stack, deployment setup, and conventions. Now every conversation starts with context. Now ask: "Write a background job that syncs user data to Stripe." Your AI reads the note. It knows it's Rails, knows you use Solid Queue, knows your conventions. No preamble needed. 2. Error fixes you'll hit again You spend 45 minutes debugging a Kamal deploy. You find the fix. A we
Knowledge Map
Connected Articles — Knowledge Graph
This article is connected to other articles through shared AI topics and tags.
Discussion
Sign in to join the discussion
No comments yet — be the first to share your thoughts!