XGBoost stands for Extreme Gradient Boosting and is an open-source machine learning library. It offers standard machine learning algorithms that use the so-called boosting algorithm. The library is extremely efficient with memory and computing power and thus delivers high performance when training the models.

What is Boosting?

In machine learning, not only individual models are used. In order to improve the performance of the entire program, several individual models are sometimes combined into a so-called ensemble. A random forest, for example, consists of many individual decision trees whose results are combined into one result.

Boosting describes the procedure of combining multiple models into an ensemble. Using the example of decision trees, the training data is used to train a tree. For all the data for which the first decision tree gives bad or wrong results, a second decision tree is formed. This is then trained using only the data that the first one misclassified. This chain is continued and the next tree in turn uses the information that led to wrong results in the first two trees.

The ensemble of all these decision trees can then provide good results for the entire data set since each individual model compensates for the weaknesses of the others. This is also referred to as combining many “weak learners” into one “strong learner”.

What is Gradient Boosting?

Gradient boosting, in turn, is a subset of many, different boosting algorithms. The basic idea behind it is that the next model should be built in such a way that it further minimizes the loss function of the ensemble.

In the simplest cases, the loss function simply describes the difference between the model’s prediction and the actual value. Suppose we train an AI to predict a house price. The loss function could then simply be the mean squared error between the actual price of the house and the predicted price of the house. Ideally, the function approaches zero over time and our model can predict correct prices.

New models are added as long as prediction and reality no longer differ, i.e. the loss function has reached the minimum. Each new model tries to predict the error of the previous model.

Let’s go back to our example with house prices. Let’s assume a property has a living area of 100m², four rooms, and a garage and costs 200,000€. The gradient boosting process would then look like this:

1. Training a regression to predict the purchase price with the features of living space, the number of rooms, and the garage. This model predicts a purchase price of 170,000 € instead of the actual 200,000 €, so the error is 30,000 €.
2. Training another regression that predicts the error of the previous model with the features of living space, number of rooms, and garage. This model predicts a deviation of 23,000 € instead of the actual 30,000 €. The remaining error is therefore 7,000 €.

These steps are repeated until the remaining error is as small as possible or even zero.

What are the Advantages and Disadvantages of Boosting in General?

The general advantage of boosting is that many weak learners are combined into one strong and powerful model. Despite a large number of small models, these boosting algorithms are usually easier to compute than comparable neural networks. However, this does not necessarily mean that they also produce worse results. In some cases, ensemble models can even beat the more complex networks in terms of accuracy. Thus, they are also interesting candidates for text or image classification.

Furthermore, boosting algorithms, such as AdaBoost, also tend to overfit less. This simply means that they not only perform well with the training dataset but also classify well with new data with high accuracy. It is believed that the multilevel model computation of boosting algorithms is not as prone to dependencies as the layers in a neural network, since the models are not optimized contiguously as is the case with backpropagation in the model.

Due to the stepwise training of single models, boosting models often have a relatively slow learning rate and therefore need more iterations to deliver good results. Furthermore, they require very good data sets, since the models react very sensitively to noise and this should be removed in the data preprocessing.

What are the Advantages of using XGBoost?

The XGBoost library provides the ability to train large data sets with proven Machine Learning algorithms while optimizing computational performance. In addition, XGBoost offers these benefits:

• Open source platform for various programming languages, such as Python or R.
• A large number of users and developers continuously develop the platform.
• Algorithms for a variety of use cases, such as classification or regression.
• Flexible and high-performance training of machine learning models, which can also be moved to the cloud.

Which Applications can be solved with XGBoost?

The XGBoost library was developed to provide high computing power during training and to parallelize and accelerate the training. This is ensured by the already explained gradient boosting, which combines many, so-called weak learners into one powerful model.

These are several decision trees that are not combined into an ensemble as in the case of a random forest but are combined with each other according to the gradient boosting algorithm in order to ensure optimal performance. Accordingly, the same fields of application can be implemented as with conventional decision trees.

The decision trees are used for classifications or regressions depending on the target variable. If the last value of the tree can be mapped to a continuous scale, we speak of a regression tree. On the other hand, if the target variable belongs to a category, we speak of a classification tree.

Due to its simple structure, this type of decision-making is very popular and is used in a wide variety of fields:

• Business management: Opaque cost structures can be illustrated with the help of a tree structure and make clear which decisions entail how many costs.
• Medicine: Decision trees help patients to find out whether they should seek medical help.
• Machine Learning and Artificial Intelligence: In this area, decision trees are used to learn classification or regression tasks and then make predictions.

This is what you should take with you

• XGBoost stands for Extreme Gradient Boosting and is an open-source Machine Learning library.
• XGBoost offers common machine learning algorithms that use the so-called boosting algorithm. This is used to combine multiple decision trees into a high-performance ensemble model.
• The advantages of XGBoost are the efficient use of computing power and the good results that the models deliver in many use cases.
• Nevertheless, neural networks have already been able to beat the already good results of XGBoost models in many comparative experiments.

Other Articles on the Topic of XGBoost

• XGBoost Documentation