Which GPU is best for training large machine learning models?

Question

What GPU should I actually get if I want to train like really big machine learning models because I am totally lost and my project is due so soon? Sorry if this is a really basic question but I honestly have no idea what I am doing and every time I try to google it I just get more confused by all the tech jargon. I am a student in Seattle and I have this massive final project due in less than three weeks where I have to train a model on thousands of pages of text data—it is basically all my journals from the last ten years—and my poor laptop literally crashed and gave me a blue screen yesterday because it got way too hot.

I have like maybe 800 dollars to spend on this which I know probably isnt a lot in the tech world but it is a ton for me. When I look at the professional stuff like those A100 things people talk about they cost like 10,000 dollars which is insane?? I just need something that wont give me that out of memory error every five seconds. Someone told me I need a lot of VRAM but I dont even know what that stands for or how much is enough for a large model. Is 8gb okay or do I need like 24?

I am looking at things like the RTX 3060 or maybe a 4070 but then some guy on a forum said those are bad for deep learning and now I am just paralyzed and dont want to waste my money on the wrong thing. I need to order something by tomorrow so it gets here for the weekend or I am going to be so far behind I might actually fail this semester. Is there a specific card that everyone uses when they are just starting out but still want to do the heavy lifting stuff? I'm really panicking here...

Ppppzz · Accepted Answer

Coming back to this... I totally get the panic! You definitely need to prioritize VRAM above everything else for those journals. I highly recommend looking at the NVIDIA GeForce RTX 4060 Ti 16GB GDDR6. It is a fantastic choice because 16GB gives you enough breathing room to avoid those annoying memory errors, and it fits well within your 800 dollar budget! I love it because it runs cool and quiet, so you wont have to worry about your computer crashing from heat again. It is basically the most efficient card for students right now who need that extra memory without breaking the bank. If you want to be super methodical about your setup, make sure you also check your power supply unit. A lot of people forget that! If you decide to go with a used NVIDIA GeForce RTX 3090 24GB GDDR6X, you really need a high-quality Corsair RM850x 850W PSU to keep everything stable during long training runs. Quick tip: definitely look into using mixed precision training. It basically doubles your effective memory and speeds up training like crazy without losing much accuracy! It is amazing for text-based models. Also, if the hardware shipping is too slow, jump on Google Colab Pro for a month. It is a fantastic backup plan while you wait for the mail! You have totally got this!

takechartiy · Answer

Look, I been there. I spent three days trying to get a NLP project running on an old 8GB card back in the day and I almost cried when the Out of Memory error popped up at 3 AM right before my deadline. It was a total nightmare. VRAM basically stands for Video RAM, and it's where your model lives while training. If it's too small, the whole thing just stops. For your 800 dollar budget, you have a few real paths:

The gold standard for home ML is the NVIDIA GeForce RTX 3090 24GB GDDR6X. You'll have to find one used, but that 24GB of VRAM lets you run the big stuff without it blowing up.

If you want something brand new, get the NVIDIA GeForce RTX 4060 Ti 16GB GDDR6. It's not as fast as the 3090, but 16GB is way better than 8GB for training.

The NVIDIA GeForce RTX 3060 12GB GDDR6 is a decent budget pick if you want to save cash, but since you have the budget, I'd go for more VRAM. Honestly, dont even look at the 4070 for this because it usually only has 12GB and costs way more than the 3060. That extra VRAM makes all the difference for massive text projects.

egfdptfjjg · Answer

Honestly, 12GB cards are unfortunately quite disappointing for training transformers on large datasets. I had major issues with the NVIDIA GeForce RTX 4070 12GB hitting memory limits way too fast. Within your 800 dollar budget, you really only have two solid options:

NVIDIA GeForce RTX 3090 24GB (try finding one used for the VRAM)

NVIDIA GeForce RTX 4060 Ti 16GB Anything with less than 16GB is just gonna keep crashing your project during training.