ReLU

term_id: relu

Category: basic_concepts

Definition

ReLU is widely used in deep learning neural networks due to its computational efficiency and ability to mitigate the vanishing gradient problem. Mathematically defined as f(x) = max(0, x), it introduces non-linearity into the model without saturating neurons for positive inputs. Despite potential issues like dying ReLUs, it remains a standard choice for hidden layers in convolutional and fully connected networks.

Summary

Rectified Linear Unit is an activation function that outputs the input directly if positive, otherwise zero.

Key Concepts

  • Non-linearity
  • Activation Function
  • Vanishing Gradient
  • Piecewise Linear

Use Cases

  • Hidden layers in CNNs
  • Deep Feedforward Networks
  • Image Recognition Models

Code Example

1
2

import torch.nn as nn
activation = nn.ReLU()

Related