Linear Regression: OLS Method and Gradient Descent

Linear Regression: From Data to Optimized Model

Linear regression models linear relationships between predictors and a target variable. For simple linear regression, the equation is y = b₀ + b₁ × x, where x is the predictor, y is the response, b₀ is the intercept, and b₁ is the slope. The coefficients determine the line's position on the plane.

Data in tabular format with features and a response serves as the foundation. Example: the relationship between apartment price and number of rooms. Model quality depends on sample representativeness—the principle of "garbage in, garbage out" dictates the outcome.

Models are useful for predictions and identifying patterns. Transforming tabular data into an analytical expression allows extrapolation to new observations.

Ordinary Least Squares Method

OLS (Ordinary Least Squares) minimizes the sum of squared residuals: ∑(y_i - ŷ_i)². Analytical solution: b₁ = Cov(x, y) / Var(x), b₀ = ȳ - b₁ × x̄.

# Example coefficient calculation
import numpy as np
x = np.array([1, 2, 3, 4, 5])
y = np.array([2, 4, 5, 4, 5])
b1 = np.cov(x, y)[0,1] / np.var(x)
b0 = np.mean(y) - b1 * np.mean(x)
print(f'b0: {b0:.2f}, b1: {b1:.2f}')

This approach works for non-degenerate data but requires matrix inversion (XᵀX)⁻¹ in the multivariate case.

Model Quality Assessment

Visually: scatter plot of predictions vs. actual values, residual plots. Metrics:

• R² (coefficient of determination): proportion of explained variance, 0–1.
• RMSE: √(MSE), in response units.
• MAE: mean absolute error.
• MAPE/SMAPE: percentage errors.

Residual plots reveal heteroscedasticity or nonlinearity.

Data Splitting and Validation

Train/test split (80/20) prevents overfitting. Cross-validation (k-fold) averages metrics across folds.

def train_test_split(X, y, test_size=0.2):
    indices = np.arange(len(X))
    np.random.shuffle(indices)
    split = int(len(X) * (1 - test_size))
    return X[indices[:split]], X[indices[split:]], y[indices[:split]], y[indices[split:]]

Multivariate Regression

Extension to y = b₀ + b₁x₁ + ... + bₙxₙ. Matrix form: ŷ = Xβ, solution β = (XᵀX)⁻¹Xᵀy. Preprocessing:

• Normalization: (x - min)/(max - min).
• Standardization: (x - μ)/σ.
• One-hot encoding of categorical features.

Probabilistic Interpretation

Model assumes normal distribution of errors: ε ~ N(0, σ²). Prediction interval: *ŷ ± t SE √(1 + 1/n + (x - x̄)² / SXX)*. Maximum likelihood estimation (MLE) optimizes log-likelihood.

Outlier Handling

Detection methods:

• Mahalanobis distance: generalized distance in multivariate space.
• Cook's distance: influence of an observation on coefficients.
• LOF (Local Outlier Factor): local density.
• RANSAC: consensus of random subsets.

Removal or robust regressors (Huber loss).

Gradient Descent and Regularization

For large data: iterative optimization θ := θ - α ∇J(θ), where J is cost function.

• L1 (Lasso): |β|, sparsity.
• L2 (Ridge): β², shrinkage.

Hyperparameter λ tuned via CV.

def gradient_descent(X, y, lr=0.01, epochs=1000):
    theta = np.zeros(X.shape[1])
    for _ in range(epochs):
        pred = X @ theta
        grad = X.T @ (pred - y) / len(y)
        theta -= lr * grad
    return theta

Key Takeaways

• Linear regression is a fundamental tool for linear relationships, scalable to multivariate tasks.
• OLS provides an analytical solution; gradient descent is for large datasets.
• Metrics like R² and RMSE assess quality; residual plots diagnose issues.
• Regularization (L1/L2) combats overfitting and multicollinearity.
• Preprocessing (scaling, outlier removal) is critical for stability.

— Editorial Team