x̄ -> Regression Models in Machine Learning

Regression Models in Machine Learning

Regression models are used to predict continuous values based on input features. These models establish a relationship between independent variables (X) and a dependent variable (Y).

---

1. Types of Regression Models

a) Linear Regression

Equation:

y = w_1x_1 + w_2x_2 + ... + w_nx_n + b

Finds the best-fit line that minimizes the error (Residual Sum of Squares).

Python Example:

from sklearn.linear_model import LinearRegression from sklearn.datasets import make_regression X, y = make_regression(n_samples=100, n_features=1, noise=10) lr = LinearRegression() lr.fit(X, y) print("Coefficients:", lr.coef_) print("Intercept:", lr.intercept_)

Use case: Predicting housing prices, sales forecasts.

---

b) Ridge Regression (L2 Regularization)

Adds a penalty for large coefficients to avoid overfitting.

Loss = Σ(y - ŷ)^2 + α Σw_i^2

Python Example:

from sklearn.linear_model import Ridge ridge = Ridge(alpha=1.0) ridge.fit(X, y) print("Ridge Coefficients:", ridge.coef_)

Use case: When you have multicollinearity (highly correlated features).

---

c) Lasso Regression (L1 Regularization)

Adds a penalty to encourage sparsity (some coefficients become zero).

Loss = Σ(y - ŷ)^2 + α Σ|w_i|

Python Example:

from sklearn.linear_model import Lasso lasso = Lasso(alpha=0.1) lasso.fit(X, y) print("Lasso Coefficients:", lasso.coef_)

Use case: Feature selection, reducing complexity.

---

d) Polynomial Regression

Extends Linear Regression by adding polynomial features.

y = w_0 + w_1x + w_2x^2 + ... + w_nx^n

Python Example:

from sklearn.preprocessing import PolynomialFeatures from sklearn.pipeline import make_pipeline poly_model = make_pipeline(PolynomialFeatures(degree=3), LinearRegression()) poly_model.fit(X, y)

Use case: Modeling curved relationships, price elasticity.

---

e) Logistic Regression (for Classification)

Used for binary classification problems (not regression).

Python Example:

from sklearn.linear_model import LogisticRegression logistic = LogisticRegression() logistic.fit(X, (y > 0)) # Converting to binary classification

Use case: Fraud detection, medical diagnoses.

---

2. Choosing the Right Regression Model

Model	Use When	Regularization?
Linear Regression	Relationship is linear	❌ No
Ridge Regression	Many correlated features	✅ L2 (shrinks coefficients)
Lasso Regression	Need feature selection	✅ L1 (some coefficients = 0)
Polynomial Regression	Relationship is non-linear	❌ No
Logistic Regression	Binary classification	✅ L2 by default

This work is licensed under a Creative Commons Attribution 4.0 International License.