27 Ensemble Learning Interview Questions (SOLVED) For Data Scientists

• Ensemble learning is a machine learning method that uses multiple learning algorithms to obtain a better predictive performance than could be obtained from any of the constituent learning algorithms alone.
• Ensembles tend to yield better results when there is significant diversity among the models. The ensemble methods, therefore, seek to promote diversity among the models they combine.

The idea of stacking is to learn several different weak learners and combine them by training a meta-model to output predictions based on the multiple predictions returned by these weak models.

If a stacking ensemble is composed of L weak learners, then to fit the model the following steps are followed:

• Split the training data into two folds.
• Choose L weak learners and fit them to the data of the first fold.
• For each of the L weak learners, make predictions for observations in the second fold.
• Fit the meta-model on the second fold, using predictions made by the weak learners as inputs.

The process of stacking is shown in the figure below:

• Random forest is an ensemble learning method that works by constructing a multitude of decision trees. A random forest can be constructed for both classification and regression tasks.
• Random forest outperforms decision trees, and it also does not have the habit of overfitting the data as decision trees do.
• A decision tree trained on a specific dataset will become very deep and cause overfitting. To create a random forest, decision trees can be trained on different subsets of the training dataset, and then the different decision trees can be averaged with the goal of decreasing the variance.

• Yes, Random forest is a tree-based ensemble algorithm with each tree depending on a collection of random variables. Random forest is using bagging as the ensemble method and decision tree as the individual model.
• Random forests can be used for classification, or regression and other tasks that operate by constructing a multitude of decision trees at a training time.
• For classification tasks, the output of the random forest is the class selected by most trees.
• For regression tasks, the mean or average prediction of the individual tress is returned.
• Random forests correct for decision trees' habit of overfitting to their training set.

• Although it provides a better outcome many times, it is not true that it will always perform better.
• There are several ensemble methods, each with its own advantages/disadvantages, and choosing one to use depends on the problem at hand.
• If there are models with high variance, then it will benefit from bagging. If the model is biased, it is better to use boosting.
• If the work is in probabilistic setting, the ensemble methods may not work because it is known that boosting delivers poor probability estimates.

In ensemble learning theory, we call weak learners (or base models) models that can be used as building blocks for designing more complex models by combining several of them. Most of the time, these basics models perform not so well by themselves either because they have a high bias (low degree of freedom models, for example) or because they have too much variance to be robust (high degree of freedom models, for example).

Let's drop just a few:

Person recognition

• Person recognition is the problem of verifying the identity of a person using the characteristics of that person, typically for security applications. It includes iris recognition, fingerprint recognition, face recognition, and behavior recognition.
• It has problems involving multiple types of features, difficulty in collecting good data, and different misclassification costs (e.g. denying system access to a legitimate user vs. allowing access to an illegitimate user).

Medicine

• Medical applications include analyzing X-ray images, human genome analysis, and examining sets of medical test data to look for anomalies.
• It has problems of having limited training and test examples, imbalanced datasets, too many attributes, and different misclassification costs (i.e., false negatives significantly worse than false positives).

• Bagging mostly aims at reducing variance.
• Boosting is mainly focused on reducing bias. The base models that are considered for boosting are models with a low variance but high bias.

• Bagging can be parallelized. The different models are fitted independently from each other.
• Boosting can not be parallelized, and it can become too expensive to fit sequentially several complex models.

• Meta-learning refers to learning about learning. In machine learning, it most commonly refers to machine learning algorithms that learn from the output of other machine learning algorithms.
• Meta-learning can be used in ensemble learning, e.g., fitting a meta-model on the out-of-fold predictions from the k-fold cross-validation.

• The idea of an ensemble method is just to pool the strengths of multiple models by combining the models.
• If none of the constituent models can handle the outliers themselves, it may be useful to have an outlier-removal step. It depends on what the ensemble is doing, and how the outliers came about.

Super learner can be thought of as a specific configuration of stacking specifically to k-fold cross-validation. The procedure can be summarized as follows: 1. Select a k-fold split of the training dataset. 2. Select m base-models or model configurations. 3. For each base model:

• Evaluate using k-fold cross-validation.
• Store all out-of-fold predictions.
• Fit the model on the full training dataset and store.

4. Fit a meta-model on the out-of-fold prediction.
5. Evaluate the model on a holdout dataset or use the model to make predictions.

The dataflow of the super learner algorithm is shown below:

Ada-boost or Adaptive Boosting is one of the ensembles boosting classifiers, it means that it will combine multiple poorly performing classifiers (or weak classifiers) to obtain a high accuracy strong classifier.

It works by following the next steps:

1. Initially, it selects a training subset randomly.
2. It iteratively trains the AdaBoost machine learning model by selecting the training set based on the accurate prediction of the last training.
3. It assigns the higher weight to wrong classified observations so that in the next iteration these observations will get a high probability for classification.
4. Also, It assigns the weight to the trained classifier in each iteration according to the accuracy of the classifier. The more accurate classifier will get high weight.
5. This process iterates until the complete training data fits without any error or until it reaches the specified maximum number of estimators.
6. To classify, it performs a vote across all of the learning algorithms built.

• Ensemble classifiers should not be used when the model needs to be interpretable and explainable.
• Ensembles can be highly competitive and provide good results, but in situations where the models are too difficult to implement, maintain, modify, or port, it is not a good idea to use ensemble models.
• The model that is closest to the true data generating process will always be the best, and will beat most ensemble methods.

• Real AdaBoost: The term real refers to the fact that the algorithm uses real-valued classifiers. This real value can be seen as the probability that a given input pattern belongs to a class. The way that this algorithm differs from the original AdaBoost algorithm is that it consists of computing and using estimates of the probabilities that each training pattern belongs to a class, under the current weight distribution whereas the Standard AdaBoost classifies the input patterns and computes the weighted error rate.
• Logit Boost: It consists of using adaptive Newton steps to fit an additive logistic model. Instead of minimizing the exponential loss, Logit Boost minimizes the logistic loss.
• Emphasis Boost: It uses a weighted emphasis (WE) function. Each input pattern is weighted according to a criterion, through the WE function, in such a way that the training process focuses on the 'critical' patterns or the quadratic error of each pattern. The WE function is given by:

• Boosting is an ensemble meta-algorithm.
• It is used to reduce bias, and also variance in supervised learning.
• In boosting, models are learned sequentially with early models fitting the simple models to the data and then analyzing the data for errors. So, consecutive decision trees are fit at every step, with the goal of solving for the net error from the prior tree. Each decision tree dictates what feature the next one will work on (it is more like teamwork rather than all trees working independently). When an input is misclassified, its weight is increased so that the next decision tree is more likely to classify it correctly.

• OOB score is calculated on data that was not necessarily used in the analysis of the model.
• Validation score is a part of the original dataset which is set apart before training the model.

• The OOB score is calculated using only a subset of Decision Trees not containing the OOB sample in their bootstrap training dataset.
• The Validation score is calculated using all the Decision Trees of the ensemble.

• A Weak learner produces a classifier that is only slightly more accurate than random classification.

• A Strong learner produces a classifier that achieves arbitrarily good accuracy.

They are used to construct boosting ensembles, which goal is to develop a large number of weak learners for a predictive learning problem, then best combine them in order to achieve a strong learner. This is a good goal as weak learners are easy to prepare but not desirable, and strong learners are hard to prepare and highly desirable.

Data Sampling and Selection: Diversity

• Defining different errors on any given sample is of paramount importance in ensemble-based systems.
• Diversity in ensembles can be achieved through several strategies, although using different subsets of the training data is the most common approach.

Training Member Classifiers

• At the core of any ensemble-based system is the strategy used to train individual ensemble members.
• Out of the numerous competing algorithms that have been developed for training ensemble classifiers; bagging, boosting, stack generalization, and hierarchical MoE remain the most commonly employed approaches.

Combining Ensemble Members

• The last step in any ensemble-based system is the mechanism used to combine the individual classifiers. The strategy used in this step depends, in part, on the type of classifiers used as ensemble members.
• The strategies are based on whether class labels are combined, or continuous outputs are combined.

All three are so-called "meta-algorithms": approaches to combine several machine learning techniques into one predictive model in order to decrease the variance (bagging), bias (boosting) or improving the predictive force (stacking alias ensemble).

• Bagging (stands for Bootstrap Aggregating) is a way to decrease the variance of your prediction by generating additional data for training from your original dataset using combinations with repetitions to produce multisets of the same cardinality/size as your original data. By increasing the size of your training set you can't improve the model predictive force, but just decrease the variance, narrowly tuning the prediction to expected outcome.

• Boosting is a two-step approach, where one first uses subsets of the original data to produce a series of averagely performing models and then "boosts" their performance by combining them together using a particular cost function (=majority vote). Unlike bagging, in the classical boosting the subset creation is not random and depends upon the performance of the previous models: every new subsets contains the elements that were (likely to be) misclassified by previous models.

• Stacking is a similar to boosting: you also apply several models to your original data. The difference here is, however, that you don't have just an empirical formula for your weight function, rather you introduce a meta-level and use another model/approach to estimate the input together with outputs of every model to estimate the weights or, in other words, to determine what models perform well and what badly given these input data.