Which GPU is best for local AI and LLM training?

Question

honestly fed up with my old 3060 crashing mid-training every single night its just not cutting it anymore but im so pumped to upgrade my home setup in denver this month.

i have a 1500 dollar budget so which gpu is actually the best for local ai and llm training?

fhnutsxjhr · Accepted Answer

totally agree that 24gb is the sweet spot for vram. i'm super happy with the NVIDIA GeForce RTX 3090 Founders Edition 24GB i picked up. honestly tho, make sure your psu can handle the transient power spikes during training. i'm using a Corsair RM1000e 1000W 80 Plus Gold and it works well. having that extra wattage headrooms really important for stability during long finetuning sessions so your system doesnt just shut off.

hjglotqkxq · Answer

Adding my two cents! i switched from a messy multi-gpu rig to a single NVIDIA GeForce RTX 4090 24GB and the speed jump was amazing!

NVIDIA GeForce RTX 4090 24GB: insane tensor core performance and fp8 support. it totally crushes training tasks!

NVIDIA GeForce RTX 4080 Super 16GB: super fast but that 16gb vram is a huge bottleneck for llms. if you find a 4090 deal for 1500, grab it immediately!

mkirgrkgrf · Answer

i upgraded to a used EVGA GeForce RTX 3090 FTW3 Ultra 24GB recently and honestly im super satisfied. for local ai training, you really need that 24gb vram more than anything else... works well for me and havent had a single crash since switching. you could also look for a NVIDIA GeForce RTX 4090 24GB GDDR6X if you find a good deal in denver but it might be slightly over budget.