Linear Regression is the simplest form of regression, built on the least-squares method Adrien-Marie Legendre published in 1805.[1] Its core assumption is elegance: the relationship between your input (X) and your output (Y) can be described by a straight line.

The algorithm finds the Line of Best Fit by minimising the Sum of Squared Errors — making the total distance between the data points and the line as small as possible. While basic, it is incredibly powerful for interpretation: the slope tells you exactly how much Y changes for every unit increase in X.

🏠 Use case — real estate pricing. Generally, as size increases, price increases consistently. A 2,000 sq ft house is usually double the price of a 1,000 sq ft house (all else equal). The relationship is additive and linear.

What happens when the data curves, accelerates, or fluctuates? Fit a straight line to curved data and you get a high error rate — underfitting.

Polynomial Regression upgrades the linear equation with exponents (X², X³). A quadratic term creates a U-shape; a cubic creates an S-shape. That flexibility captures complex growth patterns, biological phenomena, and physics trajectories.

🦠 Use case — epidemic growth. A virus spreads exponentially (1 person infects 2, who infect 4, who infect 8). A straight line would massively underestimate the danger; squared and cubed terms capture the rapid acceleration of cases.

Logistic Regression is not used to predict continuous numbers like price or temperature, but categories: Yes/No, True/False, Spam/Not Spam.

It predicts the probability of an event. Because probability must live between 0 and 1, a straight line fails (it runs to infinity). Instead, the sigmoid function squashes the output into an S-curve that stays neatly inside [0, 1].

📧 Use case — spam detection. We don’t want “this email is 500% spam” — mathematically impossible. We want “there is a 99% probability this is spam.” Logistic Regression provides that score, which you threshold (e.g. > 50% = spam).

Sometimes a model memorises the noise instead of learning the pattern — overfitting. It happens often when many variables are correlated (multicollinearity).

Ridge adds a penalty to the size of the coefficients: minimise error plus the square of the coefficients (the L2 penalty). Coefficients shrink toward zero — but rarely reach it. The model becomes simpler and smoother, and stops reacting wildly to small changes.

🧬 Use case — genetic analysis. With 10,000 genes predicting one trait, many genes are correlated. A standard model assigns massive positive/negative weights that cancel out. Ridge keeps all 10,000 genes but shrinks their impact so no single gene dominates artificially.

Lasso (Least Absolute Shrinkage and Selection Operator) shrinks coefficients like Ridge, but with the L1 penalty it can push them all the way to zero.

That means Lasso performs feature selection: it looks at your data, decides which variables are useless, and deletes them from the equation. The final model is far easier to interpret — it only includes the factors that matter.

🥗 Use case — nutritional science. Given 500 food items, most (water, lettuce, spices) have zero impact on weight gain. You want a model that says “Sugar” and “Fat” matter and ignores the rest — Lasso sets the coefficient for “Salt” to exactly zero.

What if you have correlations (Ridge’s strength) and want to eliminate useless variables (Lasso’s strength)? Enter Elastic Net.

It combines L1 and L2 penalties, balancing Lasso’s aggressive feature elimination with Ridge’s stability. It is the safe bet when you have a messy, high-dimensional dataset and don’t know which regularisation to pick.

📊 Use case — financial forecasting. Economic indicators are highly correlated (inflation moves with interest rates) and some are pure noise. Elastic Net groups correlated variables together like Ridge, then selects or rejects the whole group like Lasso.