Backstory
I currently have a Radeon R9700 AI 32GB in my LLM lab. It was a bit expensive, but I was tired of the tighter and tighter limits Anthropic was pushing on their standard tier, and I was even hitting the limits of free models on OpenRouter when I would get deep into code generation. I don’t write any of my own code since I am not, nor do I want to be, a developer. So I can tax a model pretty hard with all of my constant little fix requests.
Open models have become very good at development, and my setup has been working well for what I need, but if 32GB of vRAM is good, why not more??
Adding a second GPU to Lemonade Server
I use Lemonade Server for my local inference mostly because AMD develops it and its very easy to use.
I picked up a used Sapphire Radeon 7900XT for a decent price from a friend. I had used a 7900XT for local LLM inference before, and it was fast. The memory bandwidth is around 800 GB/s, whereas the R9700 is around 600 GB/s, but I had decided trading some speed for lots of vRAM capacity was worth it.
Now with the 7900XT in the mix I have 52GB of vRAM in the mix. We’re getting closer to 8-bit model territory now!
Configuration
After installation of the second GPU, I had to edit Lemonade’s systemctl start file and export GGLM_VK_VISIBLE_DEVICES so Vulkan used my two dedicated GPUs only, and not the internal one as well.
After restarting the service, I loaded a large, 8-bit quantization Qwen 3.8 model and asked it to build me a flappy-bird type game in python.
amd-smi shows my R9700 running at around 55% memory utilization and my 7900XT running at around 26%. (Which means more room for even larger models!)
After it was done writing the code, the system averaged about 38 tokens per second with an 88% draft acceptance. I was getting about 40-42 tokens per second with the single R9700, so not a bad trade-off in speed for all of the extra vRAM for larger models.
Worth it?
From a cost to performance ratio…no. From a privacy, security, and learning standpoint…absolutely. Open LLMs are edging closer and closer to ‘good enough’ quality for home and personal use vs data-gobbling, energy-hungry frontier models that get more and more expensive by the month.
I can definitely see a day where we run small, efficient models on our local machines to get work done, and leave the frontier models to the corporations. If the company’s providing those models are even still around in a couple of years…