25 Linear Regression Interview Questions Every Machine Learning Engineer Must Know

Linear Regression is a supervised machine learning algorithm where the predicted output is continuous and has a constant slope. It’s used to predict values within a continuous range, (e.g. sales, price) rather than trying to classify them into categories (e.g. cat, dog).

• Non-linear functions have variables with powers greater than 1. Like $x^2$. If these non-linear functions are graphed, they do not produce a straight line (their direction changes constantly).
• Linear functions have variables with only powers of 1. They form a straight line if it is graphed.

• Non-linear regression analysis tries to model a non-linear relationship between the independent and dependent variables.
• A simple non-linear relationship is shown below:

• Linear regression analysis tries to model a linear relationship between the independent and dependent variables.
• A simple linear relationship is shown below:

1. Measuring the distance of the observed y-values from the predicted y-values at each value of x.
2. Squaring each of these distances.
3. Calculating the mean of each of the squared distances.

MSE = (1/n) * Σ(actual – forecast)2

4. The smaller the Mean Squared Error, the closer you are to finding the line of best fit
5. How bad or good is this final value always depends on the context of the problem, but the main goal is that its value is so minimal as possible.

The common pattern for overfitting can be seen on learning curve plots, where model performance on the training dataset continues to improve (e.g. loss or error continues to fall) and performance on the test or validation set improves to a point and then begins to get worse.

So an overfit model will have extremely low training error but a high testing error.

• Simple linear regression uses traditional slope-intercept form. 𝑥 represents our input data and 𝑦 represents our prediction.

  𝑦 = 𝑚𝑥+𝑏

• A more complex, multi-variable linear equation might look like this, where 𝑤 represents the coefficients, or weights, our model will try to learn.

  𝑓(𝑥,𝑦,𝑧) = 𝑤1𝑥+𝑤2𝑦+𝑤3𝑧

  The variables 𝑥, 𝑦, 𝑧 represent the attributes, or distinct pieces of information, we have about each observation. For sales predictions, these attributes might include a company’s advertising spend on radio, TV, and newspapers.

  𝑆𝑎𝑙𝑒𝑠 = 𝑤1𝑅𝑎𝑑𝑖𝑜+𝑤2𝑇𝑉+𝑤3*𝑁𝑒𝑤𝑠

• The Mean Squared Error measures the variance of the residuals and is used when we want to punish the outliers in the dataset. It's defined as:

  $MSE = \frac{1}{N} \sum_{i=1}^N(y_i - \hat{y})^2$

• The Mean Absolute Error measures the average of the residuals in the dataset. Is used when we don’t want outliers to play a big role. It can also be useful if we know that our distribution is multimodal, and it’s desirable to have predictions at one of the modes, rather than at the mean of them. It's defined as:

  $MAE = \frac{1}{N} \sum_{i=1}^n |y_i -\hat{y}|$

• Linear regression is used to predict continuous outputs where there is a linear relationship between the features of the dataset and the output variable.
• Decision trees work by splitting the dataset, in a tree-like structure, into smaller and smaller subsets and make predictions based on which subset the new example falls into.

• Linear regression is used for regression problems where it predicts something with infinite possible answers such as the price of a house.
• Decision trees can be used to predict both regression and classification problems.

• Linear regression is prone to underfitting the data. Switching to polynomial regression will sometimes help in countering underfitting.
• Decision trees are prone to overfit the data. Pruning helps with the overfitting problem.

The Gradient Descent works by starting with random values for each coefficient in the linear regression model.

• After this, the sum of the squared errors is calculated for each pair of input and output values (loss function), using a learning rate as a scale factor.
• For each iteration, the coefficients are updated in the direction towards minimizing the error,
• then we keep repeating the iteration process until a minimum sum squared error is achieved or no further improvement is possible.

The constant term in regression analysis is the value at which the regression line crosses the y-axis. The constant is also known as the y-intercept.

The intercept term signifies the independent variable’s shift from the origin and ensures that the model would be unbiased.

The Constant Absorbs the Bias for the Regression Model.

If we omit the intercept term, then the model is forced to go through the origin and the slope would become steeper and biased. Hence, we should not remove the intercept term unless we are completely sure that it is 0 according to theory and expectations.

Hypothesis testing is used in a linear regression model to test if the β1 parameter in the linear equation is statistically significant. In other words, to check if the linear relationship that we obtained was not caused just by random chance.

The way to use hypothesis testing is described as follows:

1. We start establishing the null hypothesis (H0) that β1 is not significant, i.e. there is no linear relationship between independent variables and the dependent variable. The alternative hypothesis (H1) is then that β1 is not zero.

   • H0 : β1=0
   • HA : β1≠0

2. We compute the test statistic which could it be the T-test or the Z-test depending on how many samples the dataset has.
3. We compute the corresponding p-value.
4. If the p-value turns out to be less than 0.05, we can reject the null hypothesis and state that β1 is indeed significant at the 95% confidence level.

With this, we can validate that our model coefficients are not obtained just by random chance.

A way to perform the variable selection is trying out different models, each containing a different subset of the predictors. For instance, if the number of predictors is 2, then we can consider 4 models:

1. A model containing no variables.
2. A model containing X1 only.
3. A model containing X2 only.
4. A model containing both X1 and X2.

We can then select the best model out of all of the models that we have considered by computing some statistics like Adjusted R-squared. However, if the number of predictors is high, we must use some more elaborated methods for feature selection, like:

• stepwise regression,
• forward selection, and
• backward elimination.

R-squared (R²) is a statistical measure of how close the data are to the fitted regression line. It is also known as the coefficient of determination, or the coefficient of multiple determination for multiple regression.

R-squared takes values between 0 and 1, with 0 indicating that the proposed model does not improve prediction over the mean model and 1 indicating the perfect prediction. However, one drawback of R-squared is that its values can increase if we add predictors to the regression model, leading to a possible overfitting.

To address this issue, we can use Adjusted R-squared: a modified version of R-squared that has been adjusted for the number of predictors in the model. The adjusted R-squared increases when the new term improves the model more than would be expected by chance, and it decreases when a predictor improves the model by less than expected.

• Mean absolute error (MAE): calculates the absolute difference between actual and predicted values. It can be used when we want that our model be robust to outliers, but this metric has the disadvantage of not being differentiable so we can't use it if we want to apply optimizers like Gradient descent.

• Mean squared error (MSE): calculates the squared difference between actual and predicted value. We can use this metric if we want to give bigger penalization to outliers and apply optimizers who require differentiation. MSE is a differentiable function that makes it easy to perform mathematical operations in comparison to a non-differentiable function like MAE.

• Root mean squared error (RMSE): This is simply the square root of mean squared error. This metric is not so robust to outliers as the mean absolute error but it has the advantage to be differentiable so we can use it if we want to apply gradient descent to minimize losses.

When to use one depends on your loss function:

• When to use MAE: If being off by ten is just twice as bad as being off by 5. it is better to use the MAE if you don't want your performance metric to be overly sensitive to outliers.
• When to use RMSE: In many circumstances, it makes sense to give more weight to points further away from the mean - that is, being off by 10 is more than twice as bad as being off by 5. In such cases, RMSE is a more appropriate measure of error.

We make a few assumptions when we use linear regression to model the relationship between a response and a predictor. These assumptions are essential conditions that should be met before we draw inferences regarding the model estimates or before we use a model to make a prediction.

There are four principal assumptions which justify the use of linear regression models for purposes of inference or prediction:

(i) Linearity and Additivity of the relationship between dependent and independent variables:

• (a) The expected value of dependent variable is a straight-line function of each independent variable, holding the others fixed.
• (b) The slope of that line does not depend on the values of the other variables.
• (c) The effects of different independent variables on the expected value of the dependent variable are additive.

(ii) Statistical Independence of the errors (in particular, no correlation between consecutive errors in the case of time series data)

(iii) Homoscedasticity (constant variance) of the errors

• (a) versus time (in the case of time series data)
• (b) versus the predictions
• (c) versus any independent variable

(iv) Normality of the error distribution.

• Linear regression output as probabilities In linear regression, the outcome (dependent variable) is continuous. It can have any one of an infinite number of possible values. In logistic regression, the outcome (dependent variable) has only a limited number of possible values.

• Outcome In linear regression, the outcome (dependent variable) is continuous. It can have any one of an infinite number of possible values. In logistic regression, the outcome (dependent variable) has only a limited number of possible values.
• The dependent variable Logistic regression is used when the response variable is categorical in nature. For instance, yes/no, true/false, red/green/blue, 1st/2nd/3rd/4th, etc. Linear regression is used when your response variable is continuous. For instance, weight, height, number of hours, etc.
• Equation Linear regression gives an equation which is of the form Y = mX + C, means equation with degree 1. However, logistic regression gives an equation which is of the form Y = eX + e-X
• Coefficient interpretation In linear regression, the coefficient interpretation of independent variables are quite straightforward (i.e. holding all other variables constant, with a unit increase in this variable, the dependent variable is expected to increase/decrease by xxx). However, in logistic regression, depends on the family (binomial, Poisson, etc.) and link (log, logit, inverse-log, etc.) you use, the interpretation is different.
• Error minimization technique Linear regression uses ordinary least squares method to minimise the errors and arrive at a best possible fit, while logistic regression uses maximum likelihood method to arrive at the solution. Linear regression is usually solved by minimizing the least squares error of the model to the data, therefore large errors are penalized quadratically. Logistic regression is just the opposite. Using the logistic loss function causes large errors to be penalized to an asymptotically constant. Consider linear regression on categorical {0, 1} outcomes to see why this is a problem. If your model predicts the outcome is 38, when the truth is 1, you've lost nothing. Linear regression would try to reduce that 38, logistic wouldn't (as much)2.

• Ordinary least squares fit a linear model by minimizing the residual sum of squares between the observed targets in the dataset and the targets predicted by the linear approximation. Mathematically it solves a problem of the form:

  $\min_{w} || X w - y||_2^2$

• The Lasso regression fits a linear model that estimates sparse coefficients. It is useful in some contexts due to its tendency to prefer solutions with fewer non-zero coefficients, effectively reducing the number of features upon which the given solution is dependent. Mathematically, it consists of a linear model with an added regularization term:

  $\min_{w} { \frac{1}{2n} ||X w - y||_2 ^ 2 + \alpha ||w||_1}$

• It is required for the independent and dependent variables to be linear for linear regression models, but the independent and dependent variables are not required to have a linear relationship in logistic functions.

• The Linear Regression models assume that the error terms are normally distributed (bell-shaped graph) whereas there are no error terms in Logistic Regression because it is assumed to follow a Bernoulli distribution.

• Linear regression has a continuous output. Logistic regression does not have a continuous output, rather the output is a probability between 0 and 1. A linear regression may have an output that can go beyond 0 and 1.

This depends on your loss function. In many circumstances it makes sense to give more weight to points further away from the mean:

• If being off by 10 is more than twice as bad as being off by 5. In such cases, RMSE is a more appropriate measure of error.
• If being off by 10 is just twice as bad as being off by 5, then MAE is more appropriate.

Another situation when you want to use (R)MSE instead of MAE: when your observations' conditional distribution is asymmetric and you want an unbiased fit. The (R)MSE is minimized by the conditional mean, the MAE by the conditional median. So if you minimize the MAE, the fit will be closer to the median and biased.

• Normalization transforms your data into a range between 0 and 1
• Standardization transforms your data such that the resulting distribution has a mean of 0 and a standard deviation of 1

Normalization/standardization are designed to achieve a similar goal, which is to create features that have similar ranges to each other. We want that so we can be sure we are capturing the true information in a feature and that we don't overweigh a particular feature just because its values are much larger than other features.

If all of your features are within a similar range of each other then there's no real need to standardize/normalize. If, however, some features naturally take on values that are much larger/smaller than others then normalization/standardization is called to fix it.

If you're going to be normalizing at least one variable/feature, I would do the same thing to all of the others as well.