Best GPU for running local AI models?

Question

Ive been running LLMs locally for a while now using my old 3060 12GB, but Im running into major vram bottlenecks lately especially since Ive started experimenting more with some of the newer 30b and 70b quantized models using llama.cpp and kobold. The 12gb just cuts it too close when I try to load anything decent while keeping context windows open. Im based in Chicago and trying to upgrade before my next project starts next month, and ive got around a $800 to $1000 budget to work with. Should I hunt down a used 3090 just for that sweet 24gb of VRAM or would I be better off grabbing something else? What are people actually using these days for decent token generation speeds without breaking the bank...

ropkwxnpsk · Accepted Answer

Hunting down a used NVIDIA GeForce RTX 3090 24GB is honestly your best move for that budget. I made the same switch last month and the extra VRAM completely solves context window bottlenecks for 70b models.

Check local marketplaces in Chicago carefully to avoid mining cards that were abused.

Ensure your power supply can handle the 350W draw. Totally worth it imo.

yxwmikkjgt · Answer

Just catching up on this thread. You definitely need to look at the NVIDIA GeForce RTX 3090 24GB if you want to run those larger models smoothly without offloading to CPU!

Bandwidth is roughly 936 GB/s

Full 24GB GDDR6X buffer

Fits right under your grand if you hunt used I swapped mine in last month and token speeds on 70b quants jumped like crazy. Absolute game changer for local work.

lsesqqqjjh · Answer

Honestly grab a NVIDIA GeForce RTX 3090 24GB if you can find one near your budget, but make sure your PSU has enough headroom. In my experience running 70b models, that memory bandwidth and capacity are total gamechangers. Just keep an eye on temps because those VRAM modules get toasty.