Activation functions play a crucial role in deep learning models. They are the mathematical functions that transform the input signals of a neuron into its output signals. Without activation functions, deep learning models would be unable to learn the complex and non-linear patterns that are present in real-world data.

In this article, we will explore what activation functions are, how they work, and why they are important in deep learning models.

What is an Activation Function?

The activation function occurs in the neurons of a neural network and is applied to the weighted sum of input values of the neuron. Because the activation function is non-linear, the perceptron can also learn non-linear correlations.

Thus, the neural networks get the property to learn and map complex relationships. Without the non-linear function, only linear dependencies between the weighted input values and the output values could be created. Then, however, one could also use linear regression. The processes within a perceptron are briefly described in the following.

The perceptron has several inputs at which it receives numerical information, i.e. numerical values. Depending on the application, the number of inputs may differ. The inputs have different weights that indicate how influential the inputs are to the eventual output. During the learning process, the weights are changed to produce the best possible results.

The neuron itself then forms the sum of the input values multiplied by the weights of the inputs. This weighted sum is passed on to the so-called activation function. In the simplest form of a neuron, there are exactly two outputs, so only binary outputs can be predicted, for example, “Yes” or “No” or “Active” or “Inactive”, etc.

If the neuron has binary output values, an activation function is used whose values also lie between 0 and 1. Thus, the output values result directly from the use of the function.

How do Activation Functions Work?

Activation functions introduce non-linearity into the neural network, allowing it to learn complex patterns in the data. This is important because real-world data is rarely linearly separable. In other words, there are usually no straight lines that can separate the different classes in the data. Activation functions allow the neural network to learn non-linear decision boundaries that can separate the different classes.

Activation functions also help to address the vanishing gradient problem, which can occur in deep neural networks. The vanishing gradient problem occurs when the gradients become very small as they are propagated through the network, making it difficult to update the weights of the earlier layers. The ReLU function, for example, helps to address this problem by mapping all negative values to 0, which ensures that the gradients are not too small.

Why are Activation Functions Important?

Activation functions are important in deep learning models because they allow the neural network to learn complex and non-linear patterns in the data. Without activation functions, neural networks would be limited to learning only linear patterns, which are often insufficient for real-world problems.

Activation functions also help to address the vanishing gradient problem, which can occur in deep neural networks. By introducing non-linearity into the network and mapping negative values to 0, activation functions ensure that the gradients do not become too small as they are propagated through the network.

What are the most-used Activation Functions?

Several activation functions are commonly used in deep learning models. In the following sections, we introduce the most commonly used.

Binary Step Function

The binary step function is the simplest type of activation function. It is a threshold function that takes an input and returns a binary output of either 0 or 1. The binary step function is defined as follows:

• If the input is less than zero, the function returns a value of 0.
• If the input is greater than or equal to zero, the function returns a value of 1.

The binary step function is useful for binary classification problems where the output needs to be either 0 or 1. However, it is not commonly used in more complex neural network models because it has several limitations, including:

• It is not continuous, which makes it difficult to use in gradient-based optimization algorithms.
• It is not differentiable, which means that it cannot be used with backpropagation, a popular optimization algorithm for neural networks.
• It can only output two values, which limits its usefulness in more complex classification problems where multiple classes are involved.

Despite these limitations, the binary step function can still be useful in certain situations, particularly in the early stages of model development when a simple and interpretable activation function is desired.

Softmax Function

The softmax is a mathematical function that takes a vector as input and converts its individual values into probabilities, depending on their size. A high numerical value leads to a high probability in the resulting vector.

In words, each value of the vector is divided by the sum of all values of the initial vector and stored in the new vector. In purely mathematical terms, this formula then looks like this:

\(\) \[\sigma (x)_{j} = \frac{e^{z_{j}}}{\sum_{k=1}^{K} e^{z_{k}}} \text{for } j = 1, …, K.\]

With a concrete example, the operation of the Softmax function becomes clearer:

\(\) \[\begin{pmatrix}1 \\ 2 \\3 \end{pmatrix} \underrightarrow{Softmax} \begin{pmatrix}\frac{1}{1 + 2 + 3} \\ \frac{2}{1 + 2 + 3} \\ \frac{3}{1 + 2 + 3} \end{pmatrix} = \begin{pmatrix} 0.166 \\ 0.33 \\ 0.5 \end{pmatrix} \]

The positive feature of this function is that it ensures that the sum of the output values is always less than or equal to 1. This is very advantageous especially in the probability calculation because it is guaranteed that no added probability can come out greater than 1.

ReLU Function

The Rectified Linear Unit (ReLU for short) is a linear activation function that was introduced to solve the vanishing gradient problem and has become increasingly popular in applications in recent years. In short, it keeps positive values and sets negative input values equal to zero. Mathematically, this is expressed by the following term:

\(\) \[ f(x) = \begin{cases}
x & \text{if x ≥ 0}\\
0 & \text{if x < 0}
\end{cases} \]

The ReLU activation function has gained acceptance primarily because of the following advantages:

• Easy calculation: Compared to the other options, the ReLU function is very easy to calculate and thus saves a lot of computing power, especially for large networks. This translates into either lower cost or lower training time.
• No vanishing gradient problem: Due to the linear structure, the asymptotic points that are parallel to the x-axis do not exist. As a result, the gradient is not vanishing and the error passes through all layers even for large networks. This ensures that the network actually learns structures and significantly accelerates the learning process.
• Better results for new model architectures: Compared to the other activation functions, ReLU can set values equal to zero, namely as soon as they are negative. In contrast, for the sigmoid, softmax, and tanh functions, the values only approach zero asymptotically but never become equal to zero. However, this leads to problems in newer models, such as autoencoders when creating deep fakes, since real zeros are needed in the so-called code layer to achieve good results.

However, there are also problems with this simple activation function. Because negative values are consistently set equal to zero, it can happen that individual neurons also have a weighting equal to zero, since they make no contribution to the learning process and thus “die off”. For individual neurons, this may not be a problem at first, but it has been shown that in some cases even 20 – 50 % of the neurons can “die” due to ReLU.

What are the characteristics of an Activation Function?

Activation functions are an essential component of artificial neural networks. They are used to introduce non-linearity into the model’s output, which allows it to learn complex relationships in the data. Here are some key characteristics of activation functions:

1. Non-linearity: Activation functions are non-linear by nature. This non-linearity is crucial in enabling neural networks to learn and model non-linear relationships within data.
2. Range: The range of values that a function can produce is an essential consideration when selecting a function for a neural network. The output of the function must be bounded within a specific range to ensure the stability of the neural network.
3. Differentiability: Differentiability is a critical characteristic of activation functions. It enables the use of backpropagation, which is the primary method used to train neural networks.
4. Monotonicity: Monotonicity refers to the direction of the slope of the activation function. A monotonically increasing function produces outputs that are always increasing, while a monotonically decreasing function produces outputs that are always decreasing.
5. Continuity: These functions must be continuous to ensure that small changes in the input data produce small changes in the output. This continuity is essential to the stability of the neural network.
6. Computational efficiency: The computational efficiency of an activation function is also important. It should be fast to compute to ensure that the neural network can train and make predictions in a reasonable amount of time.

Overall, these special functions play a crucial role in the performance of neural networks, and selecting the appropriate function for a particular task is an essential consideration.

This is what you should take with you

• Activation functions are non-linear mathematical functions that are applied to the input of a neuron to produce its output in a deep learning model.
• They are important because they allow the neural network to learn complex and non-linear patterns in the data and address the vanishing gradient problem.
• Commonly used activation functions include the sigmoid function, ReLU function, tanh function, and softmax function.
• The choice of activation function will depend on the specific problem and the nature of the data.
• Activation functions are a critical component of deep learning models and are essential for achieving high accuracy in complex tasks.

Other Articles on the Topic of Activation Function

• Here you can find an overview of the Activation Functions in TensorFlow.