50 Data Scientist Interview Questions (ANSWERED with PDF) To Crack Next ML Interview

• Decision trees is a tool that uses a tree-like model of decisions and their possible consequences. If an algorithm only contains conditional control statements, decision trees can model that algorithm really well.
• Decision trees are a non-parametric, supervised learning method.
• Decision trees are used for classification and regression tasks.
• The diagram below shows an example of a decision tree (the dataset used is the Titanic dataset to predict whether a passenger survived or not):

Essentially, Machine Learning is a method of teaching computers to make and improve predictions or behaviors based on some data. Machine Learning introduces a class of algorithms which is data-driven, i.e. unlike "normal" algorithms it is the data that "tells" what the "good answer" is. Machine learning creates a model based on sample data and use the model to make some prediction.

More rigid explanation: Machine Learning is a field of computer science, probability theory, and optimization theory which allows complex tasks to be solved for which a logical/procedural approach would not be possible or feasible.

Dimensionality Reduction is typically choosing a basis or mathematical representation within which you can describe most but not all of the variance within your data, thereby retaining the relevant information, while reducing the amount of information necessary to represent it.

There are a variety of techniques for doing this including but not limited to PCA, ICA, and Matrix Feature Factorization.

These will take existing data and reduce it to the most discriminative components. These all allow you to represent most of the information in your dataset with fewer, more discriminative features.

• Random forest is an ensemble learning method that works by constructing a multitude of decision trees. A random forest can be constructed for both classification and regression tasks.
• Random forest outperforms decision trees, and it also does not have the habit of overfitting the data as decision trees do.
• A decision tree trained on a specific dataset will become very deep and cause overfitting. To create a random forest, decision trees can be trained on different subsets of the training dataset, and then the different decision trees can be averaged with the goal of decreasing the variance.

1. Measuring the distance of the observed y-values from the predicted y-values at each value of x.
2. Squaring each of these distances.
3. Calculating the mean of each of the squared distances.

MSE = (1/n) * Σ(actual – forecast)2

4. The smaller the Mean Squared Error, the closer you are to finding the line of best fit
5. How bad or good is this final value always depends on the context of the problem, but the main goal is that its value is so minimal as possible.

• Support vectors are the data points nearest to the hyperplane, the points of a data set that, if removed, would alter the position of the dividing hyperplane.
• Using these support vectors, we maximize the margin of the classifier.
• For computing predictions, only the support vectors are used.

The curse of dimensionality refers to various phenomena that arise when analyzing and organizing data in high-dimensional spaces that do not occur in low-dimensional settings. The common theme of these problems is that when the dimensionality increases, the volume of the space increases so fast that the available data become sparse.

Let's say you have a straight line 100 yards long and you dropped a penny somewhere on it. It wouldn't be too hard to find. You walk along the line and it takes two minutes.

Now let's say you have a square 100 yards on each side and you dropped a penny somewhere on it. It would be pretty hard, like searching across two football fields stuck together. It could take days.

Now a cube 100 yards across. That's like searching a 30-story building the size of a football stadium. Ugh.

The difficulty of searching through the space gets a lot harder as you have more dimensions. You might not realize this intuitively when it's just stated in mathematical formulas, since they all have the same "width". That's the curse of dimensionality.

• The Principal Component Analysis (PCA) is the process of computing principal components and using them to perform a change of basis on the data.
• The Principal Component of a collection of points in a real coordinate space are a sequence of p unit vectors, where the i-th vector is the direction of a line that best fits the data while being orthogonal to the i - 1 vectors. The best-fitting line is defined as the line that minimizes the average squared distance from the points to the line.
• PCA is commonly used in dimensionality reduction by projecting each data point onto only the first few principal components to obtain lower-dimensional data while preserving as much of the data's variation as possible.

• Overfitting refers to a model that models the training data too well.
• Overfitting happens when a model learns the detail and noise in the training data to the extent that it negatively impacts the performance of the model on new data. This means that the noise or random fluctuations in the training data is picked up and learned as concepts by the model. The problem is that these concepts do not apply to new data and negatively impact the model's ability to generalize.

• Underfitting refers to a model that can neither model the training data nor generalizes to new data.
• An underfit machine learning model is not a suitable model and will be obvious as it will have poor performance on the training data.
• Underfitting is often not discussed as it is easy to detect given a good performance metric. The remedy is to move on and try alternate machine learning algorithms.

• High Bias can cause an algorithm to miss the relevant relations between features and target outputs (underfitting).
• High Variance may result from an algorithm modeling random noise in the training data (overfitting).

• The Bias-Variance tradeoff is a central problem in supervised learning. Ideally, a model should be able to accurately capture the regularities in its training data, but also generalize well to unseen data.
• It is called a tradeoff because it is typically impossible to do both simultaneously:
  • Algorithms with high variance will be prone to overfitting the dataset, but
  • Algorithms with high bias will underfit the dataset.

• The 68-95-99.7 rule, is also known as the Empirical Rule.
• It is a shorthand used to remember the percentage of values that lie within an interval estimate in a Normal Distribution.
• It is shown below:

The rule states that 68%, 95%, and 99.7% of the values lie within one, two, and three standard deviations of the mean, respectively.

• A Cost Function is something we want to minimize. For example, our cost function might be the sum of squared errors over the training set.
• Gradient Descent is a method for finding the minimum of a function of multiple variables.

• Supervised learning is when the data you feed your algorithm with is tagged or labelled, to help your logic make decisions.

  Example: a hypothetical non-machine learning algorithm for face detection in images would try to define what a face is (round skin-like-colored disk, with dark area where you expect the eyes etc). A machine learning algorithm would not have such coded definition, but would "learn-by-examples": you'll show several images of faces and not-faces and a good algorithm will eventually learn and be able to predict whether or not an unseen image is a face.

• Unsupervised learning are types of algorithms that try to find correlations without any external inputs other than the raw data (your examples are not labeled, i.e. you don't say anything). In such a case the algorithm itself cannot "invent" what a face is, but it can try to cluster the data into different groups, e.g. it can distinguish that faces are very different from landscapes, which are very different from horses.

Although the task we are targeting in logistic regression is a classification, logistic regression does not actually individually classify things for you: it just gives you probabilities (or log odds ratios in the logit form).

The only way logistic regression can actually classify stuff is if you apply a rule to the probability output. For example, you may round probabilities greater than or equal to 50% to 1, and probabilities less than 50% to 0, and that’s your classification.

• In general, the loss needs to be differentiable, with some caveats. \ Consider a neural network with parameters $\theta \in R^d$, which is usually a vector of weights and biases. The gradient descent algorithm seeks to find parameters $\theta_{min}$ which minimize the loss function:

Gradient descent is performed by the update rule:

yielding new parameters $\theta_n$ which should give a smaller loss $L(\theta_n)$. The quantity $\gamma$ is the learning rate. \ The gradient descent rule requires the gradient $\nabla L(\theta_{n-1})$ to be defined, so the loss function must be differentiable. \ In most texts on calculus or mathematical analysis you'll find the result that if a function is differentiable at a point x, it is also continuous at x. Obviously, there is no hope that we could perform this procedure without knowing the gradient. \ In principle, differentiability is sufficient to run gradient descent. That said, unless L is convex, gradient descent offers no guarantees of convergence to a global minimizer. In practice, neural network loss functions are rarely convex anyway.

• An unfortunate technicality that has to be mentioned is that strictly speaking if you use the ReLU activation function, the neural network function f becomes non-differentiable.

The applicability of batch or stochastic gradient descent depends on the error manifold expected.

• Batch gradient descent computes the gradient using the whole dataset. This is great for convex, or relatively smooth error manifolds. In this case, we move somewhat directly towards an optimum solution, either local or global. Additionally, batch gradient descent, given an annealed learning rate, will eventually find the minimum located in its basin of attraction.
• Stochastic gradient descent (SGD) computes the gradient using a single sample. Most applications of SGD use a minibatch of several samples. SGD works better than batch gradient descent for error manifolds that have lots of local maxima/minima. In this case, the somewhat noisier gradient calculated using the reduced number of samples tends to jerk the model out of local minima into a region that hopefully is more optimal. Single samples are noisy, while mini-batches tend to average a little of the noise out. Thus, the amount of jerk is reduced when using mini-batches. A good balance is struck when the minibatch size is small enough to avoid some of the poor local minima but large enough that it doesn't avoid the global minima or better-performing local minima.

One benefit of SGD is that it's computationally a whole lot faster. Large datasets often can't be held in RAM, which makes vectorization much less efficient. Rather, each sample or batch of samples must be loaded, worked with, the results stored, and so on. Minibatch SGD, on the other hand, is usually intentionally made small enough to be computationally tractable. Usually, this computational advantage is leveraged by performing many more iterations of SGD, making many more steps than conventional batch gradient descent. This usually results in a model that is very close to that which would be found via batch gradient descent, or better.

• Supervised Learning analyses the training data and produces a generalized formula.
• In Reinforcement Learning basic reinforcement is defined in the model Markov’s Decision process.

• In Supervised Learning, each example will have input objects and output with desired values.
• In Reinforcement Learning, the agent interacts with the environment in discrete steps, and receives a reward for ever observation. The goal is to collect as many rewards as possible to make more observations.

• In KNN, K represents the number of nearest neighbors you want to select to predict the class of a given item, which is coming as an unseen dataset for the model. We can relate the bias-variance tradeoff with this value.
• Choosing a small value of K can cause that noise to have a higher influence on the result, which will also lead to a large variance in the predictions and an overfitted model.
• Choosing a large value of K, the accuracy in the training set will increase and it will lead to an under-fitted model. As a result, the error in the test set will go up. So, choosing K to a large value may lead to a model with a large bias(error).

Hence, there is a tradeoff between overfitting and underfitting and we have to maintain a balance while choosing the value of K in KNN, K should not be too small or too large.

When something unlikely happens, we say it's big news. Also, when we say something predictable, it's not really interesting. So to quantify this `interestingness, the function should satisfy

• if the probability of the event is 1 (predictable), then the function gives 0
• if the probability of the event is close to 0, then the function should give a high number
• if probability 0.5 event happens it give one bit of information.

One natural measure that satisfies the constraints is

I(X) = -log_2(p)

where p is the probability of the event X. And the unit is in bit, the same bit computer uses. 0 or 1.

A computer program is said to learn from experience with respect to some class of tasks if its performance in these tasks, with respect to some performance measure, improves with experience.

A High Variance model is due to a complex model and can be easily identified when you see:

1. Low training error
2. High validation error or high test error

To fix a high variance model, you can:

1. Get more training data
2. Reduce input features
3. Increase the regularization term

• Binary targets: In this case, the observed value is drawn from -1 to 1. The loss function for this case is shown as follows:

  $L = log(1 + exp(-y.\hat y))$

  Where, $\hat y$ is the predicted output. y is the observed output.

  This type of loss function implements a machine learning method known as logistic regression.

• Categorical targets: If y are the probabilities of k classes, and rth class is the ground truth class, the loss function for a single instance is defined as follows:

  $L = -log(\hat y_r)$

  This type of loss function implements multinomial logistic regression, and it is called cross-entropy loss.

• Binary logistic regression is similar to multinomial logistic regression with the value of k set to 2.

• The polynomial kernel is a kernel function commonly used with SVMs and other kernelized models, that represents the similarity of vectors in a feature space over polynomials of the original variables, allowing learning of non-linear models.
• The polynomial kernel looks not only at the given features of input samples to determine their similarity, but also combinations of these. In regression analysis, these combinations of features are called interaction features.
• For degree d polynomials, the polynomial kernel is defined as:

where x and y are vectors in the input space.

• Polynomial kernels are used for Natural Language Processing

There are three types of gradient descent methods based on the amount of data used to calculate the gradient:

• Batch gradient descent - In batch gradient descent, to calculate the gradient of the cost function, we calculate the error for each example in the training dataset and then take the sum. The model is updated only after all examples have been evaluated.
• Stochastic gradient descent - In SGD, we use one training sample at each iteration instead of using the whole dataset to sum all for every step, that is — SGD performs a parameter update for each observation. So instead of looping over each observation, it just needs one to perform the parameter update. In SGD, before for-looping, we need to randomly shuffle the training examples.
• Mini-batch gradient descent - Mini-batch gradient descent uses n data points (instead of one sample in SGD) at each iteration.

Some challenges faced when using a supervised regression model are:

• Nonlinearities: Real-world data points are more complex and do not follow a linear relationship. Sometimes a non-linear model is better at fitting the dataset. So, it is a challenge to find the perfect equation for the dataset.
• Multicollinearity: Multicollinearity is a phenomenon where one predictor variable in a multiple regression model can be linearly predicted from the others with a substantial degree of accuracy. If there is a problem of multicollinearity then even the slightest change in the independent variable causes the output to change erratically. Thus, the accuracy of the model is affected and it undermines the quality of the whole model.
• Outliers: Outliers can change and make huge impact on the machine learning model. This happens because the regression model tries to fit the outliers into the model as well. The problem of outliers is shown in the figure below:

Machine Learning

• Some see it as a branch of applied probability and statistics involving models with curvature and the application of those principles in digital computing.
• Some see it as an extension of the work of James Watt and Le Roy MacColl the application of the feedback control concept to digital control.
• Some see it as the natural result of the pioneering AI work of Norbert Wiener and John Von Neumann, where the adaptive qualities of nature, including neurochemistry, are simulated with the intention of birthing artificial life.

Data Mining

• Each book, and sometimes each chapter within the same book, seems to have its distinct conception of the verb mining.
• Unfolding the metaphor contained in the two words of the term, data mining is digging up data, and perhaps that's a satisfactory definition for the most general use of the term.

Pattern Recognition

• The term pattern recognition is perhaps the most ambiguous because neither of the two words arose in a scientific context.
• Pattern recognition is the use of machine learning algorithms to identify patterns. It classifies data based on statistical information or knowledge gained from patterns and their representation.

Overlap and Associations

• When mining data, we may be looking for a particular kind of structure in a sea of data and have a particular search strategy to narrow the search and make it manageable for computing resources available. The test used during the search may be called pattern recognition.
• In machine learning, we may train a network of artificial cells to assist in locating data or features in data that are meaningful to the stakeholders in the project. That would be using ML for data mining projects.

• Although they share some similarities, it would be difficult to declare any two of the three to be synonymous. The three arose out of different kinds of research and from different orientations.

The two branches of dimensionality reduction are:

• Linear Projection: It involves linearly projecting data from a high-dimensional space to a low-dimensional space. It includes techniques such as:
  • principle component analysis (PCA),
  • singular value decomposition, and
  • random projection.
• Manifold Learning: It is also known as nonlinear dimensionality reduction. It involves techniques such as isomap, which learns the curved distance between the points rather than the Euclidean distance. Other techniques involve:
  • multi-dimensional scaling,
  • locally linear embedding,
  • t-distributed stochastic neighbor embedding,
  • dictionary learning,
  • random trees embedding, and
  • independent component analysis.

• The basic definition of entropy is a measure of disorder. The equation of entropy is as follows:

  $E(S) = \sum_{i=1}^c - p_i \log_2 p_i$

  where, $p_i$ is the probability of frequentist probability of the element i in the data.

• In machine learning models, the goal is to decrease uncertainties. Thus, the models should have less entropy.

• The reduction of entropy is defined as information gain. It is shown below:

• Information gain is just the subtraction of entropy of Y given X from the entropy of just Y. E(Y|X) corresponds to the information of Y that we already know. So, E(Y|X) is not new information for the model.

• A confusion matrix is a technique for summarizing the performance of a classification algorithm.
• A confusion matrix will give a better idea of how the classification model works, and what type of errors it is making.
• A confusion matrix is shown in the figure below:

The matrix gives the following outputs:

• True Positive: For correctly predicted positive event values.
• False Positive: For incorrectly predicted positive event values.
• True Negative: For correctly predicted negative event values.
• False Negative: For incorrectly predicted negative event values.

In probability theory, the Central Limit Theorem (CLT) states that the distribution of a sample variable approximates a normal distribution (i.e., a “bell curve”) as the sample size becomes larger, assuming that all samples are identical in size, and regardless of the population's actual distribution shape.

Put another way, CLT is a statistical premise that, given a sufficiently large sample size from a population with a finite level of variance, the mean of all sampled variables from the same population will be approximately equal to the mean of the whole population.

Mathematically, the Central Limit Theorem (CLT) is a statement about the cumulative distribution function of the random variable

where the $X_i$ are independent identically distributed random variables with mean $\mu$ and standard deviation $\sigma$. The CLT asserts that for each $a$, $-\infty < a < \infty$,

as $n \to \infty$.

• Linear regression output as probabilities In linear regression, the outcome (dependent variable) is continuous. It can have any one of an infinite number of possible values. In logistic regression, the outcome (dependent variable) has only a limited number of possible values.

• Outcome In linear regression, the outcome (dependent variable) is continuous. It can have any one of an infinite number of possible values. In logistic regression, the outcome (dependent variable) has only a limited number of possible values.
• The dependent variable Logistic regression is used when the response variable is categorical in nature. For instance, yes/no, true/false, red/green/blue, 1st/2nd/3rd/4th, etc. Linear regression is used when your response variable is continuous. For instance, weight, height, number of hours, etc.
• Equation Linear regression gives an equation which is of the form Y = mX + C, means equation with degree 1. However, logistic regression gives an equation which is of the form Y = eX + e-X
• Coefficient interpretation In linear regression, the coefficient interpretation of independent variables are quite straightforward (i.e. holding all other variables constant, with a unit increase in this variable, the dependent variable is expected to increase/decrease by xxx). However, in logistic regression, depends on the family (binomial, Poisson, etc.) and link (log, logit, inverse-log, etc.) you use, the interpretation is different.
• Error minimization technique Linear regression uses ordinary least squares method to minimise the errors and arrive at a best possible fit, while logistic regression uses maximum likelihood method to arrive at the solution. Linear regression is usually solved by minimizing the least squares error of the model to the data, therefore large errors are penalized quadratically. Logistic regression is just the opposite. Using the logistic loss function causes large errors to be penalized to an asymptotically constant. Consider linear regression on categorical {0, 1} outcomes to see why this is a problem. If your model predicts the outcome is 38, when the truth is 1, you've lost nothing. Linear regression would try to reduce that 38, logistic wouldn't (as much)2.

Long story short, I would say that:

A loss function is a part of a cost function which is a type of an objective function.

These are not very strict terms and they are highly related. However:

• Loss function is usually a function defined on a data point, prediction and label, and measures the penalty. For example:
  • square loss $l(f(x_i|\theta),y_i) = \left (f(x_i|\theta)-y_i \right )^2$, used in linear regression
  • hinge loss $l(f(x_i|\theta), y_i) = \max(0, 1-f(x_i|\theta)y_i)$, used in SVM
  • 0/1 loss $l(f(x_i|\theta), y_i) = 1 \iff f(x_i|\theta) \neq y_i$, used in theoretical analysis and definition of accuracy
• Cost function is usually more general. It might be a sum of loss functions over your training set plus some model complexity penalty (regularization). For example:
  • Mean Squared Error $MSE(\theta) = \frac{1}{N} \sum_{i=1}^N \left (f(x_i|\theta)-y_i \right )^2$
  • SVM cost function $SVM(\theta) = \|\theta\|^2 + C \sum_{i=1}^N \xi_i$ (there are additional constraints connecting $\xi_i$ with $C$ and with training set)
• Objective function is the most general term for any function that you optimize during training. For example, a probability of generating training set in maximum likelihood approach is a well defined objective function, but it is not a loss function nor cost function (however you could define an equivalent cost function). For example:
  • MLE is a type of objective function (which you maximize)
  • Divergence between classes can be an objective function but it is barely a cost function, unless you define something artificial, like 1-Divergence, and name it a cost

All that being said, thse terms are far from strict, and depending on context, research group, background, can shift and be used in a different meaning. With the main (only?) common thing being "loss" and "cost" functions being something that want wants to minimise, and objective function being something one wants to optimise (which can be both maximisation or minimisation).

If you have high variance problem:

1. You can get more training examples because a larger the dataset is more probable to get a higher predictions.
2. Try smaller sets of features (because you are overfitting)
3. Try increasing lambda, so you can not overfit the training set as much. The higher the lambda, the more the regularization applies, for Linear Regression with regularization.

• Homoskedasticity occurs when the variance of the error term in a regression model is constant.
• Heteroskedasticity happens when the standard errors of a variable, monitored over a specific amount of time, are non-constant.

We can identify each case by plotting the residual values vs the fitted values of a linear regression model.

The importance of each one case relies on the context of the ordinary least squares (OLS) estimator, which is a common way to perform linear regression. In OLS, we must satisfy the assumption of homoskedasticity in order to get an efficient estimation.

All three are so-called "meta-algorithms": approaches to combine several machine learning techniques into one predictive model in order to decrease the variance (bagging), bias (boosting) or improving the predictive force (stacking alias ensemble).

• Bagging (stands for Bootstrap Aggregating) is a way to decrease the variance of your prediction by generating additional data for training from your original dataset using combinations with repetitions to produce multisets of the same cardinality/size as your original data. By increasing the size of your training set you can't improve the model predictive force, but just decrease the variance, narrowly tuning the prediction to expected outcome.

• Boosting is a two-step approach, where one first uses subsets of the original data to produce a series of averagely performing models and then "boosts" their performance by combining them together using a particular cost function (=majority vote). Unlike bagging, in the classical boosting the subset creation is not random and depends upon the performance of the previous models: every new subsets contains the elements that were (likely to be) misclassified by previous models.

• Stacking is a similar to boosting: you also apply several models to your original data. The difference here is, however, that you don't have just an empirical formula for your weight function, rather you introduce a meta-level and use another model/approach to estimate the input together with outputs of every model to estimate the weights or, in other words, to determine what models perform well and what badly given these input data.

the Interquartile Range tells you the spread of the middle half of your distribution.

Quartiles segment any distribution that’s ordered from low to high into four equal parts. The interquartile range (IQR) contains the second and third quartiles, or the middle half of your data set.

The interquartile range is the best measure of variability for skewed distributions or data sets with outliers. Because it’s based on values that come from the middle half of the distribution, it’s unlikely to be influenced by outliers.

This depends on your loss function. In many circumstances it makes sense to give more weight to points further away from the mean:

• If being off by 10 is more than twice as bad as being off by 5. In such cases, RMSE is a more appropriate measure of error.
• If being off by 10 is just twice as bad as being off by 5, then MAE is more appropriate.

Another situation when you want to use (R)MSE instead of MAE: when your observations' conditional distribution is asymmetric and you want an unbiased fit. The (R)MSE is minimized by the conditional mean, the MAE by the conditional median. So if you minimize the MAE, the fit will be closer to the median and biased.

The PCA object is very useful but has certain limitations for large datasets. The biggest limitation is that PCA only supports batch processing, which means all of the data to be processed must fit in the main memory.

A better alternative to use with large dataset is IncrementalPCA, this object uses a different form of processing and allows for partial computations which almost exactly match the results of PCA while processing the data in a minibatch fashion.

IncrementalPCA has the parameter batch_size to specify the number of samples to use for each batch and only stores estimates of component and noise variances, in order to update explained_variance_ratio_ incrementally. The memory usage depends on the number of samples per batch, rather than the number of samples to be processed in the dataset.

Therefore depending on the size of the input data, this algorithm can be much more memory efficient than a PCA, and allows sparse input.