Used GPU recommendations for AI under $500?

Question

I’m looking to pick up a used GPU specifically for AI/ML work (mostly local LLM inference + some light fine-tuning) and I’m trying to stay under $500. I’m a bit overwhelmed by the used market—some cards have great raw performance but not much VRAM, and I’m not sure what matters more for real-world AI workloads. Ideally I’d like 12GB+ VRAM, decent CUDA support, and something that won’t be a nightmare with drivers on Windows/Linux. Are there specific used models (and minimum VRAM) you’d recommend in this price range, and any “avoid at all costs” picks?

dqmysijmqg · Accepted Answer

Interested in this too

Kroumkacr · Answer

Hey, i feel u — used GPUs for LLM stuff is a mess cuz raw TFLOPs lie and VRAM is the real “can I even load the model??” gate.

A couple quick Qs before I steer you:
- What’s the biggest model you actually wanna run locally (like 7B/13B/34B), and are you okay with quantization?
- Are you doing fine-tuning (even light LoRA) or mostly inference… and on Windows, Linux, or dual-boot?

In my experience, the “pick A vs B vs C” usually becomes:
- More VRAM (slower) vs less VRAM (faster)
- Newer CUDA features vs older but cheap
- One big card vs future multi-GPU

Answer those and I’ll narrow it down without sending you into driver hell lol

hfvsehemin · Answer

TL;DR from this thread: reply #1 nailed it — raw compute is nice, but VRAM is literally the “can it load at all?” gate, so you gotta anchor on the biggest model you wanna run.

My own story: I grabbed a used card a while back for local inference and thought “more FPS-ish performance = better”… nah lol. I hit out-of-memory instantly, then spent a dumb amount of time juggling quant levels and context length.

Stuff I learned the hard way:
- VRAM > everything for LLM comfort (context + batch size + finetune headroom)
- Watch memory *bandwidth* too, not just cores
- Driver pain is real: stick to boring/stable CUDA setups, dont chase weird OEM cards
- Test immediately + stress it, used mining cards can be flaky!!!

Anyway, figure out your target model size first, then buy around that. gl!