Activation Function

term_id: activation_function

Category: basic_concepts

Definition

An activation function introduces non-linearity into a neural network, allowing it to learn complex patterns and relationships within data. Without these functions, a multi-layered network would behave like a single linear regression model, severely limiting its expressive power. Common examples include ReLU, Sigmoid, and Tanh. They decide whether a neuron should be activated or not by calculating a weighted sum and possibly adding a bias, effectively filtering signals to propagate only significant information through the network layers during forward propagation.

Summary

A mathematical equation that determines the output of a neural network node based on its input signal.

Key Concepts

  • Non-linearity
  • Gradient Descent
  • Neuron Activation
  • Backpropagation

Use Cases

  • Enabling deep neural networks to classify images
  • Facilitating natural language processing tasks
  • Improving convergence speed in training generative models

Code Example

1
2
3

import torch.nn as nn
relu = nn.ReLU()
output = relu(input_tensor)

Related