Logistic Regression

Despite its name, logistic regression is a classification method. For a binary label y ∈ {0, 1} it models the probability that y = 1:

P(y = 1 | x) = σ(z), where z = w₀ + w₁x₁ + … + w_dx_d = wᵀx

and σ is the sigmoid (logistic) function

σ(z) = 1 / (1 + e^(−z))

The sigmoid function

σ squashes any real number into the interval (0, 1):

Useful facts:

• σ(0) = 0.5, σ(−z) = 1 − σ(z).
• Its derivative has a neat form: σ′(z) = σ(z)(1 − σ(z)), largest (0.25) at z = 0.
• Inverting it gives the log-odds (logit): z = ln( p / (1 − p) ). So logistic regression is a linear model for the log-odds: increasing xⱼ by 1 adds wⱼ to the log-odds, which multiplies the odds p/(1 − p) by e^(wⱼ).

Decision boundary

Predict class 1 when P(y = 1 | x) ≥ 0.5, that is, when σ(z) ≥ 0.5, which happens exactly when z = wᵀx ≥ 0. The decision boundary wᵀx = 0 is a hyperplane (a straight line in two dimensions), so logistic regression is a linear classifier. Using a threshold other than 0.5 moves the boundary parallel to itself.

Example. With w₀ = −6, w₁ = 2 for a single feature x (hours studied), P(pass) = σ(2x − 6). The boundary is 2x − 6 = 0, i.e. x = 3 hours. A student who studied 4 hours has z = 2 and P(pass) = σ(2) ≈ 0.881.

Training: log loss (cross-entropy)

Squared error would make the loss non-convex for logistic regression. Instead, the weights maximise the likelihood of the training labels, which is the same as minimising the log loss (binary cross-entropy):

L(w) = −Σᵢ [ yᵢ ln pᵢ + (1 − yᵢ) ln(1 − pᵢ) ], where pᵢ = σ(wᵀxᵢ)

• If yᵢ = 1 the term is −ln pᵢ: small when pᵢ is close to 1, huge when pᵢ is close to 0.
• If yᵢ = 0 the term is −ln(1 − pᵢ).
• L is convex in w, so gradient descent finds the global minimum.

The gradient has the same shape as in linear regression:

∇L = Σᵢ (pᵢ − yᵢ) xᵢ, so the update is w ← w − η Σᵢ (pᵢ − yᵢ) xᵢ.

There is no closed-form solution like the normal equations, so an iterative method is always used. If the training data is linearly separable, the likelihood keeps increasing as ‖w‖ grows, so the weights go to infinity unless the model is regularised.

import math

def sigmoid(z):
    return 1 / (1 + math.exp(-z))

# Hours studied -> passed (1) or failed (0)
hours = [0.5, 1.0, 1.5, 2.0, 2.5, 3.0, 3.5, 4.0, 4.5, 5.0]
passed = [0, 0, 0, 0, 1, 0, 1, 1, 1, 1]

w0, w1, eta = 0.0, 0.0, 0.1
for step in range(20000):
    g0 = sum(sigmoid(w0 + w1 * x) - y for x, y in zip(hours, passed))
    g1 = sum((sigmoid(w0 + w1 * x) - y) * x for x, y in zip(hours, passed))
    w0, w1 = w0 - eta * g0, w1 - eta * g1

loss = -sum(y * math.log(sigmoid(w0 + w1 * x)) + (1 - y) * math.log(1 - sigmoid(w0 + w1 * x))
            for x, y in zip(hours, passed))
print(f"w0 = {w0:.3f}, w1 = {w1:.3f}, log loss = {loss:.3f}")
print(f"decision boundary at x = {-w0 / w1:.3f} hours")
for x in [1, 2.75, 4]:
    print(f"P(pass | {x} hours) = {sigmoid(w0 + w1 * x):.3f}")
print(f"each extra hour multiplies the odds of passing by {math.exp(w1):.2f}")

w0 = -7.159, w1 = 2.603, log loss = 2.509
decision boundary at x = 2.750 hours
P(pass | 1 hours) = 0.010
P(pass | 2.75 hours) = 0.500
P(pass | 4 hours) = 0.963
each extra hour multiplies the odds of passing by 13.51

More than two classes: softmax

For K classes, multinomial logistic regression (softmax regression) keeps one weight vector wₖ per class and sets

P(y = k | x) = e^(zₖ) / Σⱼ e^(zⱼ), where zₖ = wₖᵀx.

The outputs are positive and sum to 1. With K = 2 it reduces to the sigmoid. Another approach is one-vs-rest: train K binary classifiers and pick the class whose classifier is most confident.

Practice questions