Watch out for the “it loads in 4-bit so I’m safe” trap — you’ll still OOM from activations + optimizer states once you crank seq length, batch, and LoRA rank. Been there, same mood lol.

For your situation, I’d target ~12–16GB VRAM for comfy 7B LoRA at 512–1024 tokens (mixed precision, small micro-batch + grad accum). You can sometimes squeak by on ~8–10GB if you keep batch=1, lower rank, use checkpointing, but it’s kinda misery. For 13B, honestly 20–24GB feels “stable” for 1024 tokens; ~16GB can work at 512 tokens with aggressive tricks.

Biggest VRAM knobs: sequence length (huge), micro-batch size, gradient checkpointing (saves a ton), LoRA rank/target modules, and whether you use an 8-bit optimizer. Good luck, cheers

Been using this for years, no complaints

Basically, everyone here is spot on about activations being the real bottleneck rather than just the model weights. If you look at the math, a 7B model in 4-bit (QLoRA) only takes about 5GB for weights, but your optimizer states and gradients for even a modest rank LoRA will eat several more GBs. The real VRAM killer is the KV cache and activations at 1024 sequence length. If you go with something like the NVIDIA GeForce RTX 4090 24GB, you get access to Flash Attention 2 and Xformers, which are way more memory efficient for those longer sequences compared to older tech. Comparing brands, while the AMD Radeon RX 7900 XTX 24GB has the raw VRAM for a better price, the software ecosystem for bitsandbytes and PEFT is just much more mature on the NVIDIA side. If you are serious about 13B models, dont settle for less than 24GB. Youll still need gradient checkpointing turned on to keep from hitting OOM, but it makes the process actually usable instead of a constant headache. Tbh, 24GB is the sweet spot for 7B/13B LoRAs right now.

Warning: don’t size VRAM based on “it loads in 4-bit so I’m fine” — training OOMs are usually activations + seq length + batch, not just weights.

TL;DR: for stable LoRA, I’d target ~16GB for 7B at 512 tokens, and ~24GB if you want 1024 tokens without babysitting. For 13B, think ~24GB for 512, and ~32GB (or more) for 1024 if you want it comfy.

Biggest VRAM knobs: sequence length (killer), micro-batch size, gradient checkpointing (huge win, slower), optimizer choice (Adam eats VRAM), and whether you train in fp16/bf16. 4/8-bit helps weights a lot, but it won’t magically save you from long-context activations. What’s your LoRA rank + target modules?

Honestly i was just looking at the market prices today and it is CRAZY how much of a premium you pay just to get that extra vram for training - i am still pretty new to the local lora scene myself but i have been tracking the differences between the green and red teams for a while now - basically it feels like nvidia has such a lock on the software side with cuda that even if you find a cheaper amd card with way more memory for the money you might end up fighting with drivers or libraries just to get it running. I have been trying to figure out if it is worth the headache to switch brands just to afford the 24gb or more - are you looking to buy something new for this or are you hunting on the used market? Also what is your actual budget for the hardware because that really dictates whether you can even aim for those higher 13b models comfortably tbh.