24 Decision Tree Interview Questions Every Data Scientist Must Know

• Decision trees is a tool that uses a tree-like model of decisions and their possible consequences. If an algorithm only contains conditional control statements, decision trees can model that algorithm really well.
• Decision trees are a non-parametric, supervised learning method.
• Decision trees are used for classification and regression tasks.
• The diagram below shows an example of a decision tree (the dataset used is the Titanic dataset to predict whether a passenger survived or not):

A decision tree is a flowchart-like structure in which:

• Each internal node represents the test on an attribute (e.g. outcome of a coin flip).
• Each branch represents the outcome of the test.
• Each leaf node represents a class label.
• The paths from the root to leaf represent the classification rules.

• Random forest is an ensemble learning method that works by constructing a multitude of decision trees. A random forest can be constructed for both classification and regression tasks.
• Random forest outperforms decision trees, and it also does not have the habit of overfitting the data as decision trees do.
• A decision tree trained on a specific dataset will become very deep and cause overfitting. To create a random forest, decision trees can be trained on different subsets of the training dataset, and then the different decision trees can be averaged with the goal of decreasing the variance.

• It is simple to understand and interpret. It can be visualized easily.
• It does not require as much data preprocessing as other methods.
• It can handle both numerical and categorical data.
• It can handle multiple output problems.

• If the Gini Index of the data is 0 then it means that all the elements belong to a specific class. When this happens it is said to be pure.
• When all of the data belongs to a single class (pure) then the leaf node is reached in the tree.
• The leaf node represents the class label in the tree (which means that it gives the final output).

• Linear regression is used to predict continuous outputs where there is a linear relationship between the features of the dataset and the output variable.
• Decision trees work by splitting the dataset, in a tree-like structure, into smaller and smaller subsets and make predictions based on which subset the new example falls into.

• Linear regression is used for regression problems where it predicts something with infinite possible answers such as the price of a house.
• Decision trees can be used to predict both regression and classification problems.

• Linear regression is prone to underfitting the data. Switching to polynomial regression will sometimes help in countering underfitting.
• Decision trees are prone to overfit the data. Pruning helps with the overfitting problem.

• In the case of regression trees, the CART algorithm looks for splits that minimize the Least Square Deviation (LSD).
• Least squares is an approach in regression analysis to approximate the solution of systems where the number of equations is higher than the number of unknowns (overdetermined systems). It does this by minimizing the sum of the squares of the residuals made in the results of every single equation.
• In the case of decision trees, the LSD metric minimizes the sum of the squared distances between observed values and the predicted values. The difference between the observed and predicted values is called residual.

We can deal with the overfitted decision tree by reducing its complexity through the Pruning technique, which is changing the model by deleting the child nodes of a branch node.

The pruning processes can occur in:

• Bottom-up fashion: Start at the lowest node in the tree and follow recursively upwards to determine the relevance of each node. If the relevance for the classification is not given, the node is dropped or replaced by a leaf.

• Top-down fashion: Starts at the root of the tree. Following the structure below, a relevance check is carried out which decides whether a node is relevant for the classification. Here, an entire sub-tree (regardless of its relevance) can be dropped.

There is also a popular pruning algorithm called reduced error pruning, in which starting at the leaves, each node is replaced with its most popular class. If the prediction accuracy is not affected then the change is kept.

A stopping criteria is needed when a decision tree learner runs, otherwise, it would overfit the data making it unable to give accurate predictions for new data. Some ways to define stopping criteria are as follows:

• Assigning a minimum count in the number of training instances assigned to each leaf node. If the count is less than the minimum then the split is not accepted and the node is taken as a final leaf node.
• Assigning a pre-specified depth of the tree so that the decision tree is not so complex that it overfits the data.
• Stopping when predictor values for all nodes are identical. When this happens, no rule can be generated to split the node.

• Gini Index is also known as Gini Impurity.
• It calculates the amount of probability of a specific feature that is classified incorrectly when selected randomly.
• Gini index varies between 0 and 1. A Gini index of 0 means that all of the elements belong to a specific class, and 1 indicates a random distribution of elements across various classes. A value of 0.5 shows an equal distribution of elements over some classes.

• The formula for Gini Index is given below:

  $GI = 1 - \sum_{i=1}^n (P_i)^2$

  where, $P_i$ denotes the probability of an element being classified for a distinct class.

• In decision trees, the features possessing the least value of Gini Index get preferred over others.

• Greedy Splitting is also called Recursive Binary Splitting.
• In this procedure all the features are considered and different split points are tried and tested using a cost function. The split with the best cost (or lowest cost) is selected.
• All input variables and all possible split points are evaluated and chosen in a greedy manner (choosing the lowest value of cost possible).

• Boosting is an ensemble meta-algorithm.
• It is used to reduce bias, and also variance in supervised learning.
• In boosting, models are learned sequentially with early models fitting the simple models to the data and then analyzing the data for errors. So, consecutive decision trees are fit at every step, with the goal of solving for the net error from the prior tree. Each decision tree dictates what feature the next one will work on (it is more like teamwork rather than all trees working independently). When an input is misclassified, its weight is increased so that the next decision tree is more likely to classify it correctly.

Pruning involves cutting back the tree. After a tree has been built, it might have overfit the data. There are many ways to prune a tree, some of which are:

• Minimum error: The tree is pruned back to the point where the cross-validation error is minimum.
• Smallest tree: The tree is pruned back slightly further than the minimum error.

Pre-pruning is also known as early-stopping. In this method, the overfitting of the data may also be prevented by stopping the tree-building process early (before it produces leaf nodes with very small samples). Pre-pruning can underfit the data by stopping too early. A method to pre-prune a tree is described below:

• At each stage of splitting the tree, the cross-validation error can be checked. If the error does not decrease significantly enough then the process can be stopped.
• Pruning and pre-pruning can both be used together, separately, or not at all. Post-pruning is more mathematically rigorous when compared to pre-pruning.

After a tree has been built (and in the absence of early stopping discussed below) it may be overfitted. Pruning reduces the size of decision trees by removing parts of the tree that do not provide power to classify instances.

• Pruning the tree after it learns can increase its performance.
• Pruning is a technique that reduces the size of the decision trees by removing sections of the tree that are non-critical and redundant to classify instances.
• If the tree is too large then there is a risk of overfitting, but if the tree is too small then it will not be able to generalize the problem. Pruning helps to keep the tree in a reduced size without affecting the performance in a bad way.

We can use cross-validation to see how the error in the tree changes with the size of the tree. Beyond some size, the cross-validation error gradually increases, which can be a sign of overfitting.