Ensemble Learning & Advanced ML Algorithms(Loan Default Prediction)

Gradient Boosting and XGBoost

Learning Outcome

Topic Name-Recall(Slide3)

Hook/Story/Analogy(Slide 4)

Transition from Analogy to Technical Concept(Slide 5)

The Math of the Sculptor

The Combination: 

Final_Prediction = Model_1(Base) + Model_2(Error)

To get our final prediction, we simply add the outputs together

The Problem: Standard Gradient Boosting is Fragile

Overfitting (Too Much Learning)

Slow Speed

Problem 1:

Problem 2:

• Gradient Boosting keeps adding trees again and again

• After a point, it starts learning noise (wrong patterns) instead of real data

• Like sculptor adding extra fake details → model becomes less general

• Trees are built one after another (sequentially)

• Next tree starts only after previous one finishes

• So it is hard to parallelize → very slow

The Evolution: Enter XGBoost

What is it?

Extreme Gradient Boosting

Takes the math of Gradient Boosting and injects hardware optimization, mathematical safety nets, and extreme speed.

Why "Extreme"? :

It enhances Gradient Boosting with speed, optimization, and better accuracy.

Why XGBoost is Best

Built-in Regularization

L1 & L2 Penalties

XGBoost controls model complexity and prevents overfitting by penalizing complex trees.

Acts as a strict manager 

Sparsity Awareness

Handles Missing Data

XGBoost handles missing data automatically by learning the best path

No imputation required!

Does magic under the hood

XGBoost speeds up training by using parallel processing on multi-core systems.

Hardware Parallelization

Extreme Speed

Pros & Cons Cheat Sheet

Summary

What does the second tree predict in Gradient Boosting?

What does the second tree predict in Gradient Boosting?