The Naive Bayes Algorithm is a classification method based on the so-called Bayes Theorem. In essence, it assumes that the occurrence of a feature is completely uncorrelated with the occurrence of another feature within the class.

The algorithm is naive because it considers the features completely independent of each other and all contribute to the probability of the class. A simple example of this: A car is characterized by having four wheels, being about 4-5 meters long, and being able to drive. All three of these features independently contribute to this object being a car.

How does the Algorithm work?

The Naive Bayes algorithm is based on the Bayes theorem. It describes a formula for calculating the conditional probability P(A|B) or in words: What is the probability that event A occurs when event B has occurred? As an example: What is the probability that I have Corona (= event A) if my rapid test is positive (= event B)?

According to Bayes, this conditional probability can be calculated using the following formula:

\(\) \[P(A|B) = \frac{P(B|A) * P(A)}{P(B)} \]

• P(B|A) = probability that event B occurs if event A has already occurred
• P(A) = probability that event A occurs
• P(B) = probability that event B occurs

Why should we use this formula? Let us return to our example with the positive test and the Corona disease. I cannot know the conditional probability P(A|B) and can only find it out via an elaborate experiment. The inverse probability P(B|A), on the other hand, is easier to find out. In words, it means: How likely is it that a person suffering from Corona has a positive rapid test?

This probability can be found out relatively easily by having demonstrably ill persons perform a rapid test and then calculating the ratio of how many of the tests were actually positive. The probabilities P(A) and P(B) are similarly easy to find out. The formula then makes it easy to calculate the conditional probability P(A|B).

If we have only one feature, this already explains the complete Naive Bayes algorithm. With a feature for the conditional probability P(x | K) for different classes is calculated and the class with the highest probability wins. For our example, this means that the identical conditional probabilities P(the person is sick | test is positive) and P(the person is healthy | test is negative) are calculated using Bayes’ theorem and the classification is done for the class with the higher probability.

If our dataset consists of more than one feature, we proceed similarly and compute the conditional probability for each combination of feature x and class K. We then multiply all probabilities for one feature. The class K that then has the highest product of probabilities is the corresponding class of the dataset.

What are the Advantages and Disadvantages of the Naive Bayes Algorithm?

The Naive Bayes Algorithm is a popular starting point for a classification application since it is very easy and fast to train and can deliver good results in some cases. If the assumption of independence of the individual features is given, it even performs better than comparable classification models, such as logistic regression, and requires less data to train.

Although the Naive Bayes Algorithm can achieve good results with only a few data, we need so much data that each class appears at least once in the training data set. Otherwise, the classifier will return a probability of 0 as a result of the category in the test dataset. Moreover, in reality, it is very unlikely that all input variables are completely independent of each other, which is also very difficult to test.

How can you improve the Naive Bayes algorithm?

There are several ways to improve the performance of the Naive Bayes algorithm on a data set. The most common methods are presented below.

• Feature engineering: Like any machine learning model, the Naive Bayes algorithm depends heavily on the quality of the input data. A good selection of the required features can improve the accuracy of the model and reduce the risk of overfitting. We can use feature engineering techniques for this, such as feature extraction or feature scaling.
• Smoothing: If a data set contains no data for a certain combination of features, the so-called zero-frequency problem can occur. Poor predictions are then made for these rare categories because the model was unable to recognize sufficient structures. The Naive Bayes algorithm therefore uses smoothing to prevent situations in which a zero probability is predicted. For example, in “add-one smoothing”, one unit of the feature is added to the frequency to ensure better generalization of the model to new data.
• Ensemble methods: In ensemble training, multiple Naive Bayes models are combined and used for joint classification. The accuracy of the joint result of the models is usually higher than the result of a single model. Depending on how the models are trained and combined, there are different variants. One possibility, for example, is to use an AdaBoost approach in which the next model is only trained on the data that the previous model classified incorrectly.
• Parameter tuning: Naive Bayes also offers a selection of hyperparameters that can be adapted to the data set to improve the performance of the model. For example, the smoothing parameter can be adjusted or different sets of features can be tested.
• Dealing with unbalanced data: The Naive Bayes algorithm depends on whether the number of data sets per class is balanced or not. If this is not the case, there may be a bias towards the majority class. Methods such as oversampling or undersampling can be used to prevent these errors. With oversampling, for example, the number of data records in a minority class is increased to create a balance between the classes. Individual instances can be duplicated or slightly modified to create new instances of the minority class.
• Treatment of continuous features: The normal Naive Bayes algorithm assumes that the input features are categorical. However, this is not the case in many real-world applications and many datasets also contain continuous features. To train a Naive Bayes model, these features must first be converted into categorical data. There are different methods of how this so-called discretization can be done. For example, the data can be divided into equal intervals or subdivided using quantiles. Although some of the information content of the data set is always lost during discretization, it could not become part of the Naive Bayes model without this step.

These methods mean that the performance of the Naive Bayesian model can be further improved and a model that is as robust as possible can be trained.

What is the difference between Multinomial Naive Bayes and Bernoulli Naive Bayes?

Multinomial and Bernoulli Naive Bayes are two frequently used variants of the original Naive Bayes algorithm, which are mainly used in text classification. They differ mainly in how they represent the input data numerically. While Multinomial Naive Bayes is based on the assumption that the word components can be represented by the pure number or frequency in which they occur, Bernoulli Naive Bayes assumes that the input data is best represented by binary features. These binary features measure, for example, whether a word occurs in a document or not.

Multinomial Naive Bayes uses the so-called bag-of-words as input data. It counts how often each word occurs in the document. The model then estimates the conditional probability of each word depending on the class, using a multinomial distribution. Bernoulli Naive Bayes, on the other hand, uses binary input data and features that indicate whether a particular word occurs in the document or not. Then, analogous to Multinomial Naive Bayes, the conditional probability of a feature is estimated as a function of the class variable, but using a Bernoulli distribution.

This structure results in a further difference that relates to the handling of missing features. With Multinomial Naive Bayes, a missing word is assigned the frequency number zero, which can lead to problems with zero probabilities. The Bernoulli classifier, on the other hand, treats the missing word as a separate feature and handles it accordingly. This means that no problems arise here.

The choice of algorithm depends heavily on the task and, above all, the type of input features. Multinomial Naive Bayes is often used for text classifications that work with discrete numbers of words and calculate the classification based on a complex interplay of the individual words. Bernoulli Naive Bayes, on the other hand, is used for binary features where the prediction is more dependent on the presence or absence of individual words, such as spam recognition or sentiment analysis.

What Applications use the Naive Bayes Algorithm?

In the field of machine learning, Naive Bayes is used as a classification model, i.e. to classify a data set into a certain class. There are various concrete applications for these models for which Naive Bayes is also used:

Natural Language Processing

In this area, the model can be used to assign a section of text to a specific class. E-mail programs, for example, are interested in classifying incoming emails as “spam” or “not spam”. For this purpose, the conditional probabilities of individual words are then calculated and matched with the class. The same procedure can also be used to classify social media comments as “positive” or “negative”.

Although Naive Bayes provides a fast and simple approach for these applications in the text domain, there are other models, such as Transformers, that provide much better results. This is because the Naive Bayes model does not take into account word order or some arrangement. For example, if I say “I don’t like this product.” it is probably not a positive product review just because the word “like” is in it.

Classification of Credit Risks

For banks, loan default is an immense risk, as they lose large sums of money if a customer can no longer pay the loan. That’s why a lot of work is put into models that can calculate the individual default risk depending on the customer. In the end, this is also a classification in which the customer is assigned to either the “loan repayment” or “loan default” group. For this purpose, some specific characteristics are used, such as loan amount, income, or the number of previous loans. With the help of Naive Bayes, a reliable classification model can be trained from this.

Prediction of Medical Treatment

In medicine, a doctor has to decide which treatment and which drugs are most promising for the individual patient and his clinical picture and have the highest probability to make the patient healthy again. To support this, a Naive Bayes classification model can be trained, which calculates the probability that the client will recover or not, depending on characteristics of the health condition, such as blood pressure, well-being, or symptoms, as well as the possible treatment (medication). The results of the model can in turn be used by the physician in his decision.

This is what you should take with you

• The Naive Bayes Algorithm is a simple method to classify data.
• It is based on Bayes’ theorem and is naive because it assumes that all input variables and their expression are independent of each other.
• The Naive Bayes Algorithm is relatively quick and easy to train, but in many cases, it does not give good results because the assumption of independence of the variables is violated.

Other Articles on the Topic of Naive Bayes

• Scikit-Learn provides some examples and programming instructions for the Naive Bayes algorithm in Python.