Statistics without ML: key methods in Python

Statistical Methods for Data Analysis Without Machine Learning

Statistical tools let you quickly evaluate data without complex machine learning models. Weighted averages account for the importance of observations, standard deviation and variance reveal heterogeneity, while Poisson and binomial distributions forecast rare events. These methods using Python with numpy and pandas are ideal for mid/senior developers analyzing business metrics.

Weighted Average: Correcting for Outlier Distortions

The regular average is vulnerable to outliers. With salaries of $10 (director) and $0.10 (janitor, weight 10), the average is $5.05, which doesn't reflect reality. Weighted average solves this:

import numpy as np

salary = np.array([10.0, 0.10])
weight = np.array([1, 10])

result = np.sum(salary * weight) / np.sum(weight)
print(f"Weighted average: ${result:.2f}")

Output: $1.00. Applications:

• Analyzing revenue by sales channels accounting for traffic.
• Evaluating metrics with varying event frequencies.
• Grouping customers by segments with weights.

Standard Deviation and Variance: Assessing Stability

Standard deviation measures spread around the mean. For salaries [500, 400, 450, 550, 5000]:

import pandas as pd
import numpy as np

data = {
    "Employee": ["Ivan", "Maria", "Oleg", "Anna", "CEO"],
    "Salary": [500, 400, 450, 550, 5000]
}

df = pd.DataFrame(data)
n = len(df['Salary'])
mean_ = df['Salary'].mean()
s = np.sqrt(np.sum((df['Salary'] - mean_) ** 2) / (n - 1))

print(f"Standard deviation: {s:.2f}")

Result: 2024.41 — indicator of a strong outlier. Variance (square of deviation) is useful for comparing datasets:

import numpy as np

sales = np.array([90, 95, 92, 93, 91, 200, 210])
mean_sales = np.mean(sales)
dispersion = np.sum((sales - mean_sales) ** 2) / (len(sales) - 1)

print("Variance:", dispersion)

Spread by days (weekdays vs weekends) requires data segmentation.

Pearson Correlation: Linear Relationships Without Causality

Pearson correlation coefficient evaluates linear dependence. Formula:

import numpy as np

X = np.array([1, 2, 3, 4, 5])  # study hours
Y = np.array([50, 55, 60, 65, 70])  # test scores

x_mean = np.mean(X)
y_mean = np.mean(Y)

corXY = np.sum((X - x_mean) * (Y - y_mean))
corr_sqrt = np.sqrt(np.sum((X - x_mean)**2) * np.sum((Y - y_mean)**2))

res = corXY / corr_sqrt
print(f"Pearson correlation: {res:.2f}")

r=1.00 — perfect correlation. The square (r²) gives the proportion of explained variation (in regression — coefficient of determination). Remember: correlation ≠ causation (example: golf and mortality in the elderly).

Chi-Square Test: Checking Expectations vs Reality

Chi-square compares observed (O) and expected (E) values:

import numpy as np

O = np.array([1, 2, 3, 4, 7, 9, 11, 13, 14])  # observed
E = np.array([1, 2, 3, 4, 5, 6, 7, 8, 9])    # expected

xi = np.sum((O - E) ** 2 / E)
print(f"Chi-square statistic: {xi:.2f}")

Conditions: same length, E > 0. Larger value — greater discrepancy. Used for validating forecasts.

Poisson Distribution: Modeling Rare Events

For events with constant intensity λ (average per interval). Probability of k events:

import math

λ = 3   # srednesutochnoe count requests
k = 5   

prob = (λ ** k) * math.exp(-λ) / math.factorial(k)
print(f"Probability 5 requests: {prob:.4f}")

Result: 0.1008. Assumptions: independence, constant probability, one event at a time. Applications:

• Forecasting support calls.
• Estimating website orders.
• Analyzing security incidents.
• Risks in finance.

Binomial Distribution: Successes in Trials

For n trials with success probability p. Probability of k successes:

import math

n = 100
k = 70
p = 0.7

b_coef = math.factorial(n) / (math.factorial(k) * math.factorial(n - k))
prob = b_coef * (p ** k) * ((1 - p) ** (n - k))
print(f"Probability 70 otkrytiy: {prob:.6f}")

Examples: email opens, A/B tests, quality control.

Exponential Distribution: Intervals Until Events

Probability of an event after time x at λ:

import math

lambd = 4    # vyzovov/hour
x = 1 / 3    # 20 min

prob = lambd * math.exp(-lambd * x)
print(f"Probability vyzova cherez 20 min: {prob:.4f}")

For queues, equipment downtime.

Key Takeaways

• Weighted average and deviation detect distortions without ML.
• Correlation assesses relationships, r² — explained variation.
• Poisson and binomial — for forecasting events in business.
• Chi-square validates hypotheses with minimal data.
• All methods implementable with numpy/pandas, no GPU needed.

— Editorial Team