Artificial Intelligence & Machine Learning
Gradient Boosting
Definition
Gradient Boosting is an ensemble learning method that builds a strong predictive model by sequentially adding "weak" models (typically decision trees). Each new tree is trained to correct the errors made by the previous trees.
Why It Matters
Gradient boosting machines are among the most powerful and widely used algorithms for structured data problems (like data in tables or spreadsheets). They frequently win machine learning competitions like those on Kaggle.
Contextual Example
The first tree in a gradient boosting model might make a rough prediction. The second tree is then trained on the "residual errors" of the first tree, learning to predict what the first tree got wrong. This process is repeated, with each tree improving the overall model.
Common Misunderstandings
- It is a type of "boosting" ensemble method, which is different from "bagging" methods like Random Forest.
- Popular implementations include XGBoost, LightGBM, and CatBoost.