Blog
Common Activation and Loss Functions
Introduction
In my previous post on deep learning, I discussed how to create feed-forward neural network layers, and how create models of arbitrary size with those layers.
I also mention that a variety of activation functions can be used in the hidden layers of a neural network, and various loss functions can be used depending on the desired goals for a model.
In this post, I will go over some common activation and loss functions and derive their local gradients.
Common Activation Functions
Sigmoid Function
The sigmoid function takes in real-valued input \(x\) and returns a real-valued output in the range \([0, 1]\).
This activation function is most often used in output layers for binary classification models, although it could technically be used in hidden layers as well.
\[
\begin{aligned}
\sigma(x) &= \frac{1}{1 + e^{-x}} \\
\frac{\partial \sigma(x)}{\partial x} &= \frac{\partial}{\partial x} (1+e^{-x})^{-1}\\
&= -(1+e^{-x})^{-2} \frac{\partial}{\partial x} (1 + e^{-x})\\
&= -(1+e^{-x})^{-2} \frac{\partial}{\partial x} e^{-x}\\
&= -(1+e^{-x})^{-2} e^{-x} \frac{\partial}{\partial x} -x\\
&= (1+e^{-x})^{-2} e^{-x}\\
&= \frac{e^{-x}}{(1+e^{-x})^{2}}\\
&= \frac{1}{1+e^{-x}} \frac{e^{-x}}{1+e^{-x}}\\
&= \frac{1}{1+e^{-x}} \frac{1 + e^{-x} - 1}{1+e^{-x}}\\
&= \frac{1}{1+e^{-x}} \left[\frac{1 + e^{-x}}{1+e^{-x}} - \frac{1}{1+e^{-x}}\right]\\
&= \frac{1}{1+e^{-x}} \left[1 - \frac{1}{1+e^{-x}}\right]\\
&= \sigma(x) [1 - \sigma(x)]\\
\end{aligned}
\]
Tanh Function
The tanh function takes in real-valued input \(x\) and returns a real-valued output in the range \([-1, 1]\).
This activation function is most often seen as an activation function for hidden layers.
\[
\begin{aligned}
\tanh(x) &= \frac{\sinh(x)}{\cosh(x)}\\
&= \frac{e^x - e^{-x}}{e^x + e^{-x}} \\
\frac{\partial \tanh(x)}{\partial x} &=
\frac{
\left[\frac{\partial}{\partial x} (e^x - e^{-x})\right]
(e^x + e^{-x})
-
(e^x - e^{-x})
\left[\frac{\partial}{\partial x} (e^x + e^{-x})\right]
}{
(e^x + e^{-x})^2
}\\
&=
\frac{
(e^x + e^{-x})^2 - (e^x - e^{-x})^2
}{
(e^x + e^{-x})^2
}\\
&= 1 -
\frac{
(e^x - e^{-x})^2
}{
(e^x + e^{-x})^2
}\\
&= 1 - \tanh^2(x)\\
\end{aligned}
\]
Rectified Linear Activation Function (ReLU)
The ReLU function takes in real-valued input \(x\) and returns real-valued output in the range \([0, \infty)\).
The ReLU function is most often seen as an activation function for hidden layers.
It should be noted that formally, the derivative for the ReLU function at 0 is undefined.
However, in practice, it is often set to 0 when \(x\) is 0.
\[
\begin{aligned}
r(x) &=
\begin{cases}
x & x > 0\\
0 & x < 0
\end{cases}\\
\frac{\partial r(x)}{\partial x} &=
\begin{cases}
1 & x > 0\\
0 & x < 0\\
\end{cases}\\
\end{aligned}
\]
Softmax Function
The softmax function takes in a vector of real-values and returns a vector of real values, each in the range \([0, 1]\).
The vector \(\vec{s(x)}\) obtained from running vector \(x\) through the softmax function always sums to 1.
Due to this, the softmax function is most often used as the activation function for output layers in multinomial classification problems.
\[
\begin{aligned}
s(x_i) &= \frac{e^{x_i}}{\sum^K_{k=1} e^{x_k}} \\
\frac{\partial s(x_j)}{\partial x_i} &= \frac{\partial}{\partial x_i} \frac{e^{x_j}}{\sum^K_{k=1} e^{x_k}}\\
&= \frac{
\left(\frac{\partial}{\partial x_i} e^{x_j}\right)\sum^K_{k=1}e^{x_k} -
e^{x_j}\left(\frac{\partial}{\partial x_i}\sum^K_{k=1}e^{x_k}\right)
}{
(\sum^K_{k=1}e^{x_k})^2
}\\
&=
\begin{cases}
\frac{
\left(\frac{\partial}{\partial x_i} e^{x_j}\right)\sum^K_{k=1}e^{x_k} -
e^{x_j}\left(\frac{\partial}{\partial x_i}\sum^K_{k=1}e^{x_k}\right)
}{
(\sum^K_{k=1}e^{x_k})^2
} & i=j\\
\frac{
\left(\frac{\partial}{\partial x_i} e^{x_j}\right)\sum^K_{k=1}e^{x_k} -
e^{x_j}\left(\frac{\partial}{\partial x_i}\sum^K_{k=1}e^{x_k}\right)
}{
(\sum^K_{k=1}e^{x_k})^2
} & i\neq j\\
\end{cases}\\
&=
\begin{cases}
\frac{
e^{x_i}\sum^K_{k=1}e^{x_k} -
e^{x_i}e^{x_i}
}{
(\sum^K_{k=1}e^{x_k})^2
} & i=j\\
\frac{
0\sum^K_{k=1}e^{x_k} -
e^{x_j}e^{x_i}
}{
(\sum^K_{k=1}e^{x_k})^2
} & i\neq j\\
\end{cases}\\
&=
\begin{cases}
\frac{
e^{x_i}(\sum^K_{k=1}e^{x_k} - e^{x_i})
}{
(\sum^K_{k=1}e^{x_k})^2
} & i=j\\
\frac{
e^{x_i}(-e^{x_j})
}{
(\sum^K_{k=1}e^{x_k})^2
} & i\neq j\\
\end{cases}\\
&=
\begin{cases}
s(x_i)(1 - s(x_j))
& i=j\\
s(x_i)(0 - s(x_j))
& i\neq j\\
\end{cases}\\
&= s(x_i)(\delta_{i=j} - s(x_j))\\
\end{aligned}
\]
Common Loss Functions
Binary Cross-Entropy
The binary cross-entropy loss function is used for models where the desired output is a binary probability.
This is necessary in binary classification models.
\[
\begin{aligned}
L_{\text{BCE}}(y, \hat{y}) &= -\left[y\log \hat{y} + (1-y)\log(1-\hat{y})\right]\\
\frac{\partial L}{\partial \hat{y}} &=
\frac{\partial}{\partial \hat{y}} -\left[y\log \hat{y} + (1-y)\log(1-\hat{y})\right]\\
&= -\left[
\frac{\partial}{\partial \hat{y}} y\log \hat{y} +
\frac{\partial}{\partial \hat{y}}(1-y)\log(1-\hat{y})
\right]\\
&= -\left[ \frac{y}{\hat{y}} - \frac{1-y}{1-\hat{y}} \right]\\
&= \frac{1-y}{1-\hat{y}} - \frac{y}{\hat{y}}\\
\end{aligned}
\]
Cross-Entropy
The cross-entropy function is used for models where the desired output is a probability distribution over \(K\) possible classes.
This is necessary in multinomial classification models.
\[
\begin{aligned}
L_{CE}(y, \hat{y}) &= - \left[ \sum^{K}_{k=1} y_k \log(\hat{y}_k) \right] \\
\frac{\partial L}{\partial \hat{y}_i} &=
\frac{\partial}{\partial \hat{y}_i} - \left[ \sum^{K}_{k=1} y_k \log(\hat{y}_k) \right]\\
&= -\left[ \frac{\partial}{\partial \hat{y_i}} y_i \log(\hat{y}_i) \right]\\
&= -\frac{y_i}{\hat{y}_i}\\
\end{aligned}
\]
Mean Squared Error
The mean squared error loss function is used for models where the desired output is a real number.
This is necessary for regression models.
\[
\begin{aligned}
L_{\text{MSE}}(y,\hat{y}) &= (y - \hat{y})^2\\
\frac{\partial L}{\partial \hat{y}} &= \frac{\partial}{\partial \hat{y}} (y - \hat{y})^2\\
&= 2(y-\hat{y})\frac{\partial}{\partial \hat{y}} (y - \hat{y})\\
&= 2(\hat{y}-y)\\
&\propto \hat{y}-y\\
\end{aligned}
\]