Batch Normalization

term_id: batch_normalization

Category: basic_concepts

Definition

This method adjusts and scales activations to have zero mean and unit variance within each mini-batch during training. It reduces internal covariate shift, allowing for higher learning rates and faster convergence. By adding learnable scale and shift parameters, it maintains the network’s representational power while mitigating issues caused by varying input distributions, making deep network training more robust and efficient.

Summary

Batch normalization is a technique that normalizes layer inputs across a mini-batch to stabilize and accelerate neural network training.

Key Concepts

  • Internal covariate shift
  • Mini-batch statistics
  • Gradient stabilization
  • Regularization effect

Use Cases

  • Deep Neural Networks
  • Convolutional Neural Networks
  • Training optimization

Code Example

1
2
3
4
5
6

import torch.nn as nn
layer = nn.Sequential(
    nn.Linear(10, 20),
    nn.BatchNorm1d(20),
    nn.ReLU()
)

Related