Machine Learning Champions

The Kaggle king, uncrowned.

While brain-inspired deep learning wins the headlines, the quiet trophy case of tabular prediction belongs to XGBoost — eXtreme Gradient Boosting. It is fast, regularized, and unfussy about messy data. This is why it remains the first model serious practitioners train on a new table.

↓ Why it wins ↓ Skip to the rules

A decade after its release — and well into the 3.x era with GPU and external-memory training — the crown is contested by LightGBM and CatBoost, but never quite taken. High speed. Smart logic. Robust by default.

Ali Reza Rashidi7 min readUpdated 6 August 2026

01—The arena

The digital olympics of tables#

Thousands of competitors, one recurring winner. The reasons are more engineering than magic.

⚡ High speedParallelized tree building that uses the whole machine.

🧠 Smart logicSecond-order gradients aim every new tree at the residual errors.

🛡 RegularizedBuilt-in penalties keep the ensemble honest on new data.

🕳 Missing-proofLearns the best default direction for missing values — no imputation required.

02—The reasons

How the king actually wins#

Four ideas: industrial engineering, iterative correction, honest optimization, and knowing its sport.

Picture a digital arena where thousands of data scientists battle for prize money to predict the future. That is Kaggle — and for structured data, the gold medal keeps going to the same elegant tool. XGBoost builds a team of “weak” models that correct each other’s mistakes, and it does it fast:

The engineering

Hardware parallelization: it uses every core while others walk single-file.
Handles missing data natively: it learns which way to route the holes, no imputation ceremony.
Regularization built in: it resists memorizing the answer key (overfitting) by design.

The honest caveats (2026 edition)

LightGBM & CatBoost are real rivals: often faster, sometimes better — benchmark all three.
Deep learning still owns unstructured data: pixels, audio, and long text are not its sport.

The champion’s profile

XGBoostClassic boosting

How to read this: classic boosting had the accuracy idea first; XGBoost made it industrial — speed, missing values, and anti-overfit as engineering, not hope.

Imagine guessing a house price. You don’t get it right immediately — you start with a guess, then friends correct your errors one by one. That is boosting:

Base model (Bob) looks at the neighborhood average.

Guess: $200,000.

Correction 1 (Alice) sees the swimming pool Bob missed.

+ $50,000.

Correction 2 (Charlie) notices the old roof.

− $10,000.

Final prediction: $240,000.

Each new tree fits the residual errors of the team so far.

XGBoost computes the gradient (the slope of the error) for every prediction and adds trees that step downhill. With second-order math and regularization, each step is unusually well-aimed.

Error vs. ensemble size

Training errorValidation error

How to read this: training error falls forever; validation error tells the truth. Early stopping picks the valley floor — before the mountain starts climbing again.

This is why ‘more trees’ is not a strategy.Learning rate × tree count is a budget, not a dial to max out.

# Regression — continuous values

House prices ($)
Stock value
Temperature

? Classification — categories

Spam vs. not spam
Churn (leave vs. stay)
Fraud vs. legitimate

Where it shines

How to read this: XGBoost is a champion, but not for every sport. It dominates structured data and generally loses to deep learning on unstructured data.

03—The rules

Best practices#

The settings that separate a leaderboard model from an overfit bonfire.

👍 Do this

Tune learning rate (eta): lower rates (e.g. 0.01–0.05) with more trees usually win.
Use early stopping: halt when validation score stops improving.
Check feature importance: know which columns actually matter.
Benchmark LightGBM & CatBoost too: loyalty is for sports, not baselines.

👎 Don’t do this

Don’t ignore outliers: extreme values can still skew splits — clean first.
Don’t over-deepen: max_depth > 10 is rarely needed; start at 3–6.
Don’t forget encoding: XGBoost needs numbers — convert text first.

04—Choosing

How I would use it#

XGBoost is not the answer to every problem — it is the answer to a specific, extremely common one: medium-sized tables where accuracy and speed both matter.

New tabular problem? Baseline with XGBoost.

If a fancier model can’t beat it, ship the tree.

Tune depth and learning rate before anything exotic.

Small steps, many trees, early stop.

Read the feature importances.

They are a data audit disguised as a chart.

Images, audio, long text? Different kingdom.

Deep learning rules the unstructured lands.

The honest crown: XGBoost wins not because it is brilliant, but because it is disciplined — fast enough to iterate, regularized enough to trust, and simple enough to debug at 3 a.m.

05—Grounding

Sources#

The family tree is real: every claim in this piece descends from one of these papers. The charts are illustrative profiles, as marked.

Friedman, J. H. (2001). “Greedy Function Approximation: A Gradient Boosting Machine.” The Annals of Statistics, 29(5), 1189–1232. The origin of gradient boosting itself — fitting each new tree to the residual errors of the ensemble so far. doi.org/10.1214/aos/1013203451
Chen, T., & Guestrin, C. (2016). “XGBoost: A Scalable Tree Boosting System.” KDD ’16, pp. 785–794. The paper behind the library: second-order gradients, built-in regularization, sparsity-aware split finding, and the systems engineering that made it fast. arxiv.org/abs/1603.02754
Ke, G., Meng, Q., Finley, T., Wang, T., Chen, W., Ma, W., Ye, Q., & Liu, T.-Y. (2017). “LightGBM: A Highly Efficient Gradient Boosting Decision Tree.” NeurIPS 2017, 3146–3154. The strongest challenger — histogram-based training that wins on speed at large scale. papers.nips.cc
Prokhorenkova, L., Gusev, G., Vorobev, A., Dorogush, A. V., & Gulin, A. (2018). “CatBoost: Unbiased Boosting with Categorical Features.” NeurIPS 2018, 6638–6648. The other challenger — ordered boosting and native categorical handling. papers.nips.cc
Grinsztajn, L., Oyallon, E., & Varoquaux, G. (2022). “Why Do Tree-Based Models Still Outperform Deep Learning on Typical Tabular Data?” NeurIPS 2022 Datasets and Benchmarks. The evidence behind the Choosing section: on medium-sized tabular data, tuned gradient-boosted trees remain the benchmark to beat. arxiv.org/abs/2207.08815

XGBoost, why It Rules Kaggle!

Why Active Learning in ML

All Types of Regression

The Kaggle king, uncrowned.

The digital olympics of tables#