Story time: I was on a 12GB card and hit the exact same wall… batch size bumps would just OOM, and yeah gradient accumulation “works” but iteration time gets sooo painful. I ended up comparing (A) more VRAM vs (B) more raw speed vs (C) “balanced” and honestly VRAM changed my day-to-day way more than compute. Like, I stopped babysitting microbatches, could keep activations around, and training felt less like fighting the runtime. Speed upgrades were nice for throughput, but if the model doesnt fit cleanly, it’s kinda pointless.

Cost-wise I also learned the hidden budget stuff: power/thermals, and whether your projects are fp16/bf16 friendly (AMP saved me a ton). Oh and keeping an eye on VRAM fragmentation + using checkpointing selectively helped too. Anyway… been there, i get it. gl!

I went through this last year… started on a 12GB card too and it was *always* VRAM, not compute. Once I moved to a bigger VRAM single-GPU setup, I stopped doing all the annoying hacks (accumulation, tiny batches) and iteration speed felt way better, even if raw TFLOPs werent night/day.

Quick Qs so I’m not guessing:
- Are you mostly training (mixed precision, grad checkpointing) or mostly local inference w/ bigger context?
- What models/sizes are you bumping into VRAM on (like “7B-ish” vs “big CV”)?

That detail changes whether you’ll feel the pain in activation memory vs weights, you know…

Oh man, been there. I started on an RTX 3060 12GB too and once my PyTorch stuff grew, VRAM was the bottleneck way more than “speed” (batch size, bigger models, fewer hacks). For your situation I’d prioritize jumping to a card with a lot more VRAM, even if it’s not the absolute fastest. My training got way less annoying once I stopped living on gradient accumulation and constant OOM tweaks (at least thats what worked for me). Good luck!!

Honestly, looking at the market right now, you’re basically stuck between the "new" path and the "prosumer" used path. If you stay with NVIDIA—which you probably should for PyTorch compatibility—the NVIDIA GeForce RTX 4070 Ti Super is basically the floor now at 16GB, but that 192-bit or 256-bit bus feels a bit weird for heavy training. On the other hand, a used NVIDIA GeForce RTX 3090 fits perfectly in your $700-1200 budget and gives you that 24GB sweet spot—plus the memory bandwidth is way higher than the newer mid-range cards which realy helps with throughput. I’ve seen people mention the AMD Radeon RX 7900 XTX because it has 24GB for a great price, but honestly, unless you want to fight with ROCm drivers and potential library issues all day, it might be a bit of a dead end for a serious PyTorch workflow. I'd definately lean towards the 3090 if you can find a clean one, just because the CUDA ecosystem is so much more stable for AI stuff right now!!

Honestly, I’m gonna have to disagree a bit with the used 3090 route. I’m a total stickler for reliability, and buying a used card for AI work feels like a gamble... you never know if it was running 24/7 in a dusty basement. If you want something that won’t die on you mid-training, I’d suggest the NVIDIA GeForce RTX 4080 Super 16GB. I know people suggest AMD cards like the AMD Radeon RX 7900 XTX 24GB because they have more VRAM for the money, but for PyTorch training, sticking with NVIDIA is basically mandatory because of how much easier CUDA makes your life. Going with a new 4080 Super gives you that 16GB ceiling, which is a nice jump from 12GB, but more importantly, you get a warranty and much better power efficiency. The 40-series architecture just runs cooler and more stable during long training sessions. 16GB might feel like a compromise compared to 24GB, but a stable 16GB is way better than a dead 24GB card with no warranty, ngl.

Exactly what I was thinking