Best budget GPU for local AI and LLM training?

Question

What is currently the best value GPU for running local LLMs and some basic finetuning without breaking the bank? Ive been a hardware nerd for years but the transition from gaming specs to AI specs is tripping me up. My old 3060 just isnt cutting it anymore with 12GB VRAM when I try to push higher context windows. I have about $450 saved up and I want to get something that wont choke on Llama 3 or Mistral. I saw the 4060 Ti 16GB but everyone says the memory bandwidth is trash for this stuff. Is there a better option under $500 or should I just look for a used 3090 on eBay even though my PSU might explode? Need to decide by Saturday...

ServerAdmin24_7 · Accepted Answer

Ngl, the NVIDIA GeForce RTX 4060 Ti 16GB is kinda a trap for AI work. That 128-bit bus is a massive bottleneck for inference speeds once you actually utilize the VRAM. If you want to run Llama 3 70B or do any real finetuning, you really need the 24GB buffer to avoid constant OOM errors. I would suggest hunting for a used NVIDIA GeForce RTX 3090 24GB, but you have to be very careful with the power draw and hardware health:

Make sure your PSU is a Tier A unit at 850W minimum to handle those massive power spikes.

Check the VRAM temps because the memory chips on the back of the 3090 PCB run incredibly hot during long training runs.

Use separate PCIe power cables for each input, dont daisy chain them or youll risk melting the connectors. I managed to find one for around your budget recently and the 936 GB/s bandwidth makes a huge difference for token generation speeds. Just keep an eye on those transients and maybe undervolt it slightly to stay safe...

dsoqkrtsry · Answer

I'm satisfied with the NVIDIA GeForce RTX 3090 24GB GDDR6X. The NVIDIA GeForce RTX 4060 Ti 16GB GDDR6 draws less power, but its narrow memory bus limits inference speed significantly.