Gradient Accumulation
term_id: gradient_accumulation
Category: training_techniques
Definition
This optimization strategy allows deep learning models to be trained with effective batch sizes larger than what fits into GPU memory. By accumulating gradients from several mini-batches and performing a weight update only after the accumulated steps, developers can maintain stable training dynamics associated with large batches without requiring proportional hardware resources. It is particularly useful for fine-tuning large language models on consumer-grade hardware.
Summary
Gradient accumulation is a technique that simulates larger batch sizes by summing gradients over multiple forward/backward passes before updating weights.
Key Concepts
- Batch Size Simulation
- Memory Optimization
- Stochastic Gradient Descent
- Weight Update
Use Cases
- Fine-tuning large models
- Training on limited VRAM
- Stabilizing loss convergence