Multiple Regression in Python: scikit-learn

Multiple Linear Regression in Python: Modeling with Multiple Predictors

Multiple linear regression extends simple regression by incorporating multiple independent variables. The model fits a hyperplane in multidimensional space to predict the target variable Y using predictors X₁ through Xₙ. Each coefficient βᵢ reflects the isolated impact of its respective factor while holding all other variables constant. Using the Advertising dataset, we’ll predict sales based on TV, Radio, and Newspaper ad budgets.

Mathematical Formulation

The model equation is:

Y = β₀ + β₁X₁ + β₂X₂ + ⋯ + βₙXₙ + ε

β₀ is the intercept, βᵢ are the regression coefficients, and ε represents residuals. Optimal β values are computed using ordinary least squares (OLS):

β̂ = (XᵀX)⁻¹Xᵀy

Here, X is the extended feature matrix (with a column of ones), and y is the target vector. Key assumptions include linear relationships and homoscedasticity of residuals.

Data Preparation

Split the data into train/test sets (80/20) to evaluate generalization performance and prevent overfitting. Apply One-Hot Encoding for categorical features and use StandardScaler to standardize numerical features—this improves coefficient interpretability.

Key preparation steps:

• Check for missing values and outliers.
• Encode categorical variables.
• Split the dataset: train_test_split(X, y, test_size=0.2, random_state=42).
• Scale features: StandardScaler().fit_transform(X_train).

Implementation on the Advertising Dataset

Dataset: 200 observations, predicting Sales from TV, Radio, and Newspaper advertising budgets.

Import and Load Data

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error, r2_score, mean_absolute_error

plt.style.use('seaborn-v0_8-whitegrid')
sns.set_palette("husl")

df = pd.read_csv("Advertising.csv")
if 'Unnamed: 0' in df.columns:
    df = df.drop('Unnamed: 0', axis=1)
df.columns = ['TV', 'Radio', 'Newspaper', 'Sales']
print(df.head())
print(f"Data size: {df.shape}")

First few rows:

| TV | Radio | Newspaper | Sales |

|-------|-------|-----------|-------|

| 230.1 | 37.8 | 69.2 | 22.1 |

| 44.5 | 39.3 | 45.1 | 10.4 |

EDA and Correlations

Summary statistics:

| | TV | Radio | Newspaper | Sales |

|-------|--------|-------|-----------|-------|

| mean | 147.04 | 23.26 | 30.55 | 14.02 |

| std | 85.85 | 14.85 | 21.78 | 5.22 |

Correlation with Sales: TV (0.78), Radio (0.58), Newspaper (0.23).

Scatter plots reveal strong linear relationships between TV-Sales and Radio-Sales, but a weak link for Newspaper.

Model Training

X = df[['TV', 'Radio', 'Newspaper']]
y = df['Sales']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

model = LinearRegression()
model.fit(X_train, y_train)

print(f"Intercept: {model.intercept_:.4f}")
for name, coef in zip(X.columns, model.coef_):
    print(f"{name}: {coef:.4f}")

Results:

• Intercept: 2.9791
• TV: 0.0447
• Radio: 0.1892
• Newspaper: 0.0028

Interpretation: For every $1,000 increase in TV ads, sales rise by ~44.7 units; Radio yields ~189.2 additional units. Newspaper has negligible impact.

Model Evaluation

y_pred = model.predict(X_test)
mse = mean_squared_error(y_test, y_pred)
rmse = np.sqrt(mse)
mae = mean_absolute_error(y_test, y_pred)
r2 = r2_score(y_test, y_pred)
print(f"MSE: {mse:.2f}, RMSE: {rmse:.2f}, MAE: {mae:.2f}, R²: {r2:.4f}")

Metrics:

• MSE: 3.17
• RMSE: 1.78
• MAE: 1.46
• R²: 0.8994

An R² of 0.90 indicates the model explains 90% of the variance in sales. RMSE and MAE provide error magnitude in actual sales units.

Residual Diagnostics

• Residual histogram: symmetric around zero.
• Residuals vs predictions: no visible patterns (homoscedasticity).
• Scatter plot of actual vs predicted values shows points clustering tightly along the diagonal line.

Key Takeaways

• Multiple regression isolates predictor effects while controlling for confounders.
• The near-zero coefficient for Newspaper suggests it adds little predictive value.
• R² = 0.90 reflects strong explanatory power, but always check VIF for multicollinearity.
• Always analyze residuals to validate assumptions of linearity and normality.
• Scaling isn’t required for OLS but enhances coefficient interpretability.

— Editorial Team