Mixed Precision Training

term_id: mixed_precision_training

Category: training_techniques

Definition

Mixed Precision Training (MPT) combines half-precision (FP16) and full-precision (FP32) data types during neural network training. By using FP16 for most operations, MPT reduces memory footprint and increases computational speed on modern GPUs with tensor cores. To maintain numerical stability, critical updates are performed in FP32. This technique allows for larger batch sizes and faster convergence without sacrificing model accuracy, making it essential for training large-scale deep learning models efficiently.

Summary

A training technique that uses both 16-bit and 32-bit floating-point numbers to accelerate computation and reduce memory usage.

Key Concepts

  • FP16
  • FP32
  • Tensor Cores
  • Numerical Stability

Use Cases

  • Large model training
  • GPU acceleration
  • Memory-constrained environments

Code Example

1
2
3
4
5
6
7

import torch
import torch.cuda.amp as amp

# Example snippet showing automatic mixed precision context
with amp.autocast():
    output = model(input)
    loss = criterion(output, target)

Related