How can I fine-tune a model on a free GPU?
LoRA and QLoRA: train small matrices beside frozen weights · Builder · 5 min · training · after RAG vs fine-tuning, Model memory: parameters × bytes, plus the KV cache
LoRA fine-tunes a model by freezing its weights and training two small matrices beside each one, about 0.5% of the numbers. QLoRA also stores the frozen weights in 4 bits, so a 7B model trains in about 6 GB.
1 Guess first
2 Look
Fine-tuning Qwen2.5 7B, LoRA style
1 / 4rank r:q_proj in one block: 12,845,056 numbers stay frozen. Rank 16: B 3584 × 16 + A 16 × 3584 = 114,688 trained (0.89% of W).
Qwen2.5 7B shapes from its safetensors headers (Hugging Face) · memory = arithmetic, activations not counted · 2026-10-23
One weight matrix stays frozen. LoRA trains two thin strips beside it: at rank 16, 0.89% of its size.
Fine-tuning does not have to touch the model: freeze every weight, train thin strips beside them, and 122 GB of training memory becomes about 6 GB.
LoRA (low-rank adaptation) keeps each weight matrix W frozen and learns a correction B·A next to it. A is r × in, B is out × r, so a matrix adds r × (in + out) numbers. B starts at zero, so training begins from the original model. QLoRA also stores the frozen weights in 4 bits.
4 You are the model
You have one free 16 GB T4. For four Qwen2.5 sizes, add up the memory and pick the most precise method that still leaves 2 GB free: full fine-tune, LoRA, QLoRA, or no fit.
one free 16 GB T4 · keep 2 GB for activations and CUDA (a rough allowance) · pick the most precise method that fits
run 1 of 4
Fine-tune Qwen2.5 3B (3.09 billion parameters)
memory = model states only, from the formulas in the lesson · Qwen2.5 shapes from Hugging Face · 2026-10-23
Full fine-tune, LoRA, QLoRA, or no fit: which one for this model?5 Check
Fine-tune for habits: a format, a tone, a task done your way, from a few hundred good examples. Facts that change belong in the prompt (RAG), not the weights. Training a small model on a big model’s answers is distillation, and LoRA is a cheap way to run it. QLoRA’s authors trained a 65B model on one 48 GB GPU.
7 Say it back
8 Mission · 5 min
In a free Kaggle notebook (phone-verified), pick “GPU T4 x2” and run !nvidia-smi. Then build Qwen2.5-7B with empty weights (accelerate’s init_empty_weights and from_config: nothing downloads), wrap it in peft’s LoraConfig(r=16, target_modules=["q_proj", "v_proj"]) and call print_trainable_parameters(). Does it print 5,046,272? Which fits one card: 122, 15.9 or 6.2 GB?
Three quick ones
Drawn at random from this lesson's pool; different next time.
Done?
Go deeper
Sources
- Read with scripts/explain-data/lora.ts on 2026-10-23 from Hugging Face config.json and safetensors headers (tensor shapes, BF16): Qwen/Qwen2.5-7B (commit d149729) 28 blocks; q_proj [3584, 3584], k_proj and v_proj [512, 3584], o_proj [3584, 3584], gate_proj and up_proj [18944, 3584], down_proj [3584, 18944]; 7,615,616,512 parameters (index total_size 15,231,233,024 bytes / 2), 6,525,288,448 in the 7 projections, 1,090,328,064 else. Qwen2.5 0.5B 494,032,768; 3B 3,085,938,688; 32B 32,763,876,352
- LoRA counts: r × (in + out) per targeted matrix, summed over every block. Rank 16 on q_proj and v_proj: 5,046,272 (0.066%); on all 7 projections: 40,370,176 (0.53%); rank 64 on all 7: 161,480,704 (2.1%)
- Memory (model states, GB = 10⁹ bytes, activations not counted): full fine-tune = parameters × 16 bytes; LoRA = parameters × 2 bytes + trained × 16 bytes; QLoRA = projection weights × 4.127 bits + the rest × 2 bytes + trained × 16 bytes. Qwen2.5 7B, rank 16 on all 7: 121.8, 15.9, 6.2 GB. The 2 GB left for activations and CUDA is a rough allowance, not a measurement
- https://arxiv.org/abs/2106.09685 (Hu et al., LoRA, checked 2026-10-23): “freezes the pre-trained model weights and injects trainable rank decomposition matrices into each layer”; “We use a random Gaussian initialization for A and zero for B, so ΔW = BA is zero at the beginning of training”; merged W = W0 + BA gives “no additional inference latency”
- https://arxiv.org/abs/2305.14314 (Dettmers et al., QLoRA, checked 2026-10-23): “finetune a 65B parameter model on a single 48GB GPU”; double quantization reduces the constants per parameter “from 32/64 = 0.5 bits, to 8/64 + 32/(64·256) = 0.127 bits”, so NF4 storage is 4.127 bits
- https://arxiv.org/abs/1910.02054 (Rajbhandari et al., ZeRO, checked 2026-10-23): mixed-precision Adam holds fp16 parameters and gradients (2Ψ and 2Ψ bytes) plus an fp32 copy of the parameters, momentum and variance (4Ψ, 4Ψ and 4Ψ bytes): 16 bytes per trained parameter
- https://huggingface.co/docs/transformers/main/en/quantization/bitsandbytes (checked 2026-10-23): 4-bit loading uses “Linear4bit layers that replace standard PyTorch linear layers”; transformers quantizers/base.py get_keys_to_not_convert: “we may want to keep the lm_head in full precision”
- https://www.nvidia.com/en-us/data-center/tesla-t4/ (checked 2026-10-22): GPU memory 16 GB GDDR6; aifree.dev catalog kaggle-free-gpu-tpu (verified 2026-10-12): up to 30 hours/week GPU, “Accelerators: NVIDIA Tesla P100 (16GB), 2x NVIDIA T4, or TPU v3-8”, phone verification required
Published Oct 23, 2026 · updated Oct 23, 2026