Blog

Logistic Regression

Introduction

Logistic regression is a parametric classification method that is used to estimate binary class output given real-valued input vectors. In this post, cross entropy loss is used as the loss function and gradient descent (or batched gradient descent) is used to learn parameters.

The computation graph below shows how logistic regression works. The dot product of each input, a vector \(\vec{x}\) of size \(D\), and weight vector \(\vec{w}\) transposed is taken, resulting in \(z\). \(z\) is put through the sigmoid function, and the loss is calculated using the mean squared error on label \(y\) and \(\sigma(z)\).

The bias term \(\beta\) is ignored for the purposes of this post, but can easily be appended to weight vector \(\vec{w}\) after appending a \(1\) to an input vector \(\vec{x}\).

png

Learning

Given a data set \(X\) of size \(N\) with \(D\) dimensions, parameters \(w\) must be learned that minimize our loss function \(L_{CE}(y, \hat{y})\). The weight vector is learned using gradient descent. The derivation for term \(\frac{\partial L}{\partial w}\) in the weight update is displayed in the derivations section of this post.

\[ \begin{aligned} \sigma(z) &= \frac{1}{1 + e^{-z}} & \text{[Logistic function]}\\ L_{\text{CE}}(y, \hat{y}) &= -\left[ y\log \hat{y} + (1-y)\log(1-\hat{y}) \right]& \text{[Cross entropy loss]}\\ w_i &= w_i - \alpha \frac{\partial L}{\partial w_i} & \text{[Weight update]}\\ &= w_i - \alpha \left[x_i (\sigma(z) - y)\right] &\\ w_i &= w_i - \alpha \frac{1}{B}\sum^B_{j=1}x_{j,i}(\sigma(z)-y) & \text{[Batch weight update]}\\ \end{aligned} \]

Code

Code for a logistic regression classifier is shown in the block below

from typing import List
from tqdm import trange
import torch


def ErrorRate(y: torch.Tensor, yhat: torch.Tensor) -> float:
    """ Calculate error rate (1 - accuracy)

    Args:
        y: true labels
        yhat: predicted labels

    Returns:
        error rate
    """

    return torch.sum((y != yhat).float()) / y.shape[0]


class LogisticRegressionClassifier:

    def __init__(self) -> None:
        """ Instantiate logistic regression classifier
        """

        self.w = None
        self.calcError = ErrorRate


    def fit(self, x, y, alpha=1e-4, epochs=1000, batch=32) -> None:
        """ Fit logistic regression classifier to dataset

        Args:
            x: input data
            y: input labels
            alpha: alpha parameter for weight update
            epochs: number of epochs to train
            batch: size of batches for training
        """

        self.w = torch.rand((1, x.shape[1]))

        epochs = trange(epochs, desc='Error')
        for epoch in epochs:

            start, end = 0, batch
            for b in range((x.shape[0]//batch)+1):
                hx = self.probs(x[start:end])
                dw = self.calcGradient(x[start:end], y[start:end], hx)
                self.w = self.w - (alpha * dw)
                start += batch
                end += batch

            hx = self.predict(x)
            error = self.calcError(y, hx)
            epochs.set_description('Err: %.4f' % error)


    def probs(self, x: torch.Tensor) -> torch.Tensor:
        """ Determine probability of label being 1

        Args:
            x: input data

        Returns:
            probability for each member of input
        """

        hx = 1 / (1 + torch.exp(-torch.einsum('ij,kj->i', x, self.w)))[:, None]
        return hx


    def predict(self, x: torch.Tensor) -> torch.Tensor:
        """ Predict labels

        Args:
            x: input data

        Returns:
            labels for each member of input
        """

        hx = self.probs(x)
        hx = (hx >= 0.5).float()
        return hx


    def calcGradient(self, x: torch.Tensor, y: torch.Tensor, hx: torch.Tensor) -> torch.Tensor:
        """ Calculate weight gradient

        Args:
            x: input data
            y: input labels
            hx: predicted probabilities

        Returns:
            tensor of gradient values the same size as weights
        """

        return torch.sum(x * (hx - y), dim=0) / x.shape[0]

Derivations

Derivative of loss function \(L\) with respect to sigmoid output \(a=\sigma(z)\):

\[ \begin{aligned} \frac{\partial L}{\partial a} &= \frac{\partial}{\partial a} - \left[y\log a + (1-y) \log (1-a)\right]\\ &= - \left[ y\frac{\partial}{\partial a}\log a + (1-y)\frac{\partial}{\partial a}\log (1-a) \right]\\ &= - \left[ \frac{y}{a}\frac{\partial}{\partial a}a + \frac{(1-y)}{(1-a)}\frac{\partial}{\partial a}(1-a) \right]\\ &= -\left[\frac{y}{a} - \frac{(1-y)}{(1-a)}\right] \end{aligned} \]

Derivative of sigmoid function \(\sigma(z)\) with respect to sigmoid input \(z\):

\[ \begin{aligned} \frac{\partial \sigma(z)}{\partial z} &= \frac{\partial}{\partial z}\frac{1}{1 + e^{-z}}\\ &= \frac{\partial}{\partial z}(1 + e^{-z})^{-1}\\ &= -(1 + e^{-z})^{-2}\frac{\partial}{\partial z}(1 + e^{-z})\\ &= -(1 + e^{-z})^{-2}\frac{\partial}{\partial z}(e^{-z})\\ &= \frac{-e^{-z}}{(1 + e^{-z})^{2}}\frac{\partial}{\partial z}(-z)\\ &= \frac{e^{-z}}{(1 + e^{-z})^{2}}\\ &= \frac{e^{-z}}{1 + e^{-z}} \frac{1}{1 + e^{-z}}\\ &= \left( \frac{1 + e^{-z}}{1 + e^{-z}} - \frac{1}{1 + e^{-z}} \right)\frac{1}{(1 + e^{-z})}\\ &= (1 - \sigma(z))\sigma(z)\\ \end{aligned} \]

Derivative of linear combination \(z\) with respect to weight \(w_i\):

\[ \begin{aligned} \frac{\partial z}{\partial w_i} &= \frac{\partial}{\partial w_i} \sum^D_{j=1}w_j \times x_j\\ &= x_i \end{aligned} \]

Derivative of loss function \(L\) with respect to weight \(w_i\):

\[ \begin{aligned} \frac{\partial L}{\partial w_i} &= \frac{\partial z}{\partial w_i} \frac{\partial \sigma(z)}{\partial z} \frac{\partial L}{\partial \sigma(z)}\\ &= x_i \left[(1-\sigma(z))\sigma(z)\right]\left[\frac{1-y}{1-\sigma(z)}-\frac{y}{\sigma(z)}\right]\\ &= x_i [(1-y)\sigma(z) - (1-\sigma(z))y] \\ &= x_i [\sigma(z)-y\sigma(z) - y + y\sigma(z)] \\ &= x_i [\sigma(z)- y] \\ \end{aligned} \]

Resources